End of Google Summer of Code Project

I would just like to wrap up this blog with the end of the Google Summer of Code SDL2 meta-build system project.

Review of Project Goals

In review, my goals were as follows:

• Create a meta-build system that allows for simple generation of methods to build SDL2 on different platforms
• Replicate as much of the SDL2 build process as possible
• Keep the meta-build system portable and lightweight
• Make sure extending it as simple as can be, for future parts added to the project

The medium for meta-building I chose was Premake, which uses Lua as a configuration language for setting up projects in a functional way. Given the simplicity of both Lua and the premake API, it seemed ideal. Equally important, Premake also has the entire system packed into a single executable, rendering it completely portable.

Accomplishments

Given my goals, I feel I accomplished all of them in one way or another. The following is a list of accomplishments from my perspective (development perspective):

• The meta-build system provides a method of rapidly and conveniently generating new projects related to the SDL2 build tree (such as a new test or SDL2 library)
• Provides a cross-platform way to generate various SDL2 build methods using the same meta-build interface (everything runs through premake4.lua for MinGW, Windows, Linux, Mac OS X, and Xcode)
• It provides the possibility of as complex dependency checking as CMake or Autotools with cross-platform convenience of Premake (with some tweaking, of course)
• It provides tested capabilities of generating GNU Makefiles; Visual Studio C++ (2008, 2010, and 2012) project and solution files; and Xcode (3 and 4) project and workspace environments

Features Support (Final)

The Premake project supports the following:

• Cross-platform project and solution generation for various operating systems and build environments
• Build environments include GNU Make (MinGW/Cygwin/Linux); Xcode 3 and 4 (both OS X and iOS); and Visual Studio 2008, 2010, and 2012 (Windows)
• Supported operating systems include Windows, Mac OS X, and Linux, with side targets for MinGW and iOS
• Support for building the SDL2 library and the side SDL2main and SDL2test libraries
• Support for building all of the projects in the test suite implemented at the time of this project
• Support for building all of the iOS demo projects
• Partial Cygwin support
• Interface for implementing CMake and Autotool-esque dependency checking routines (slightly demonstrated in the SDL2 project targeted at Linux)

Post-Mortem

As a reflection, I've noted many things that have gone wrong with this project, or just varied significantly from the original vision. I had very little build experience coming into this project (created a few of my own premake setups, created a few makefiles in the past to my own need, used Visual Studio for a couple of years, and used Apache Ant in the past). My Autotools experienced was limited to running a configure script.

Nevertheless, despite the huge barriers I had to overcome in order to understand what was needed to adequately replicate the complex build setup of the SDL2 library, I rose to the challenge and had a lot of fun creating the project. I gained a huge amount of build system experience and I hope that my work helps SDL2 in its efforts and all those who use it.

The project ended up going well beyond what I originally expected, though some of my stretch goals were missed. I wanted to implement iOS support as a stretch, plus build all of the SDL2 extra libraries (like SDL2_mixer, etc.) The second one proved to be well out of scope and there was simply not enough time to implement them. iOS ended up being one of the last major hurdles, and I feel it was well worth it. I hoped to have setup a good environment if others wish to add support for those extra libraries in the future.

Beyond that, the project moved to Linux support and I ended up creating extra goals that were not originally in the project description. Linux, itself, was a stretch goal because I originally signed on with my mentors' expectations I was focusing on Visual Studio and Xcode. Those weren't enough for  me, though. After getting them working, I decided to turn my focus toward Linux and Cygwin. Dependency handling in Linux greatly challenged me, which is why I ended up developing the goal to replicate some sort of complex dependency checking system like CMake and Autotools have. I implemented a primitive one and demonstrated it worked. I hope others or myself can extend it in the future, perhaps even leading to it becoming part of Premake. As far as Cygwin support goes, I regret it ended up not happening. Cygwin currently has linker issues as is covered both in this blog and in the changelog I kept throughout the project.

One of the most unexpected parts of the project was how much I ended up committing to the Premake source code. I noted in the description of my project that Premake may lead to barriers in implementing a system as complex as SDL2 (especially with forward compatibility in mind), but how to approach solving the problems those barriers caused was the question. One could work around the code in hackish or unappealing ways (which was my original approach), or directly modify the Premake source code (which I ultimately decided to do later, when I reached insurpassable barriers). What was interesting was all of the changes I personally made to Premake ended up being committed to the stable branch of Premake, making the current Premake used in the SDL2 project one patch short of being the live version currently maintained (the one patch adding core iOS support).

Final Words

Overall, I really enjoyed my time on this project. I am very excited to see how the community reacts to this project and I can only hope for the best. This was a great experience and I would recommend Google Summer of Code to anyone dedicated and willing to have a ton of fun, meet new people, and learn a huge amount of information in a topic of their choice, with the plus of getting paid for it.

Signing out.

Ben Henning

Linux, Cygwin, iOS, and moving forward (CMake/autotools on premake?)

Howdy,

It's been a while since my last post, but I want to take some time to let the community know the latest progress. My work on Google Summer of Code is almost up. I intend on writing one more post after this to wrap things up.

Over the last few weeks, I implemented support for Linux (experimental), Cygwin (rather slim), and iOS. I've implemented a few functions that allow me to do a lot of the same dependency checking as CMake and autotools, as I will discuss below. I have also cleaned up the dependency system some and intend on continuing to make the meta-build system as clean and easy-to-understand as possible.

iOS

For starters, I wanted iOS support in the meta-build system almost since the beginning of the project. It seemed like a very interesting system to target, but not without its potential difficulties. Because of desired iOS support, I need to make several modifications to the premake source, all of which were patched into the current stable branch of premake. One of the patches another fellow wrote which provided basic iOS functionality. There was still a lot of other things to modify in order to get iOS to work properly.

Nevertheless, it is working now. I've created a separate Xcode directory specifically for iOS. The same demos used in the current iOS project are being referenced to and tested with the generated system. There are a few minor things related to the logo, startup image, and codesigning that will be more challenging to solve, and may remain unsolved by the end of this project. The goal was getting basic iOS support without much effort on behalf of the developers using the system, and I think I successfully achieved that.

Cygwin

Cygwin was proving especially challenging. Upon implementing MinGW (check my last post), I really wanted to implement Cygwin. MinGW proved incredibly challenging because of a bug in premake that I needed to fix, but Cygwin is even more complicated. For those who are not aware, MinGW is setup to create Windows applications using a POSIX environment (GCC, Make, SH, etc.) Cygwin is designed to write POSIX/Unix applications on Windows. That means that the sources that are compiled for each of these similar environments are very different. The Cygwin target actually compiles the Linux version of SDL, not the Windows version.

The next hurdle was me not being able to successfully build the current SDL project using autotools. Autotools refused to build with Cygwin unless its cross-compiled GCC could produce a Windows executable. Well, of course GCC can produce Windows executables, so this error is a bit non-intuitive. It's essentially saying that there can't be a dependency on Cygwin, which is why the authors of the cross-compiler implementing a -mno-cygwin flag. However, this flag is only available on gcc-3, which is no longer shipped with Cygwin. This approach to writing Cygwin-independent applications is now obsolete. It's recommended to use MinGW-w64 instead.

I decided to provide a stripped implementation that would work on Cygwin. It basically just has threading and file support. Everything major like video and audio has been completely stripped due to time and complexity. This functionality is completely possible to implement, but I just felt like Cygwin was the least important target, so I didn't invest too much time into it. The executables produced by the make files generated from the meta-build system will be dependent on cygwin1.dll.

Linux

Currently only tested on Linux Mint 15, I've began to implement Linux support. I had this a while back (I started Linux support to help with Cygwin support), but it's now reaching a much better level. The sheer magnitude of achieving complete Linux support is an entire project in itself (just take a look at the CMake or autotools config files sometime). Therefore, my disclaimer here is the Linux target is completely experimental, incomplete, and likely containing issues that will only be found after it's been tested with many different flavors of Linux.

My goal was not to implement complete Linux support but, rather, to give a good starter for me or other people to work off of in the future. The current meta-build systems allowed me to implement Linux support with ease, but the ability to test for the monolithic list of dependencies SDL is capable of having is well beyond premake. I will discuss this in greater detail in the next section.

I say that SDL is capable of having these dependencies because it's modular. It can have a half a dozen audio devices or just one. SDL can be built on a minimalist configuration file that basically just loads dummy drivers for the entire system. My goal wasn't just to get some shell of SDL working, it was to reproduce the current functionality I am able to build with autotools itself. With that, any features beyond that autotools configures SDL for will not be supported with the results of this GSoC project. That's not to say it wouldn't take just a few minutes to implement them, though. I daresay testing takes much longer than implementing new features in this system.

One final note is about the largest difference between the autotools Linux setup and the one from premake: building on Linux should produce a shared library, just as on Windows. However, due to how GCC handles linking to shared libraries, the best approach to linking is to have the shared libraries in a public location (such as the /usr/lib folder, hence its existence). The problem is premake generates makefiles without custom targets. I am not able to implement an 'install' target for the shared SDL2 library, so I cannot link to it there. My only solutions are to do post-build copies of the library to the correct public location (which seems hackish and non-intuitive for the developer), keep the shared library bound to the same directory as the executable (incredibly inconvenient), or to simply produce a static library. I opted for the last option for times' sake. This is definitely an excellent aspect to improve on in the future.

Dependency Checking: Evolved from CMake and Autotools

My system is setup to produce dependency directives in project files that depend on some named function. In another file, a function is paired with its name and registered in a table. I decided to use functions for dependency handling because dependency checking can depend on a huge variety of factors, many of which aren't continuously considered. Functions can do essentially anything, so they were the ideal choice.

What do these functions do? Well, they are uniquely expected to determine whether their target dependency is on the the current system. Beyond that, they may or may not need to provide directories for header files, directories for static libraries, or a list of libraries to link to. The question yet remains...how do we find whether we have the dependency?

CMake and Autotools use the functionality for finding a library using common, public locations of where the library may be residing. Premake has this functionality as well in its os.findlib function. Where CMake and Autotools start to shine over premake is their ability to check dependencies by generating basic source files and seeing whether they compile, link, or run, depending on what's being checked. I decided that this functionality is extremely interesting and very useful, so I've implemented my own basic system for supporting a handful of the features of CMake, specifically.

The hope is to allow much better support for dependency checking than just looking for the library. Source files allow checking of compiler flags, checking for the size of certain datatypes, and whether built-in library functions exist. The possibilities are nearly limitless, so I am excited to see how these features are used in the future. I intend on demonstrating their use partially in some of the Linux dependency functions. I must note, however, that I only slightly emphasized cross-platform support. Right now it specifically focuses on GCC, but Microsoft compiler, Clang, Borland, or whatever else would be possible to add, if desired. The system would just need to be reworked slightly.

Moving Forward

Over the next week I hope to make the meta-build system as easy to understand, use, and extend as possible. I did not implement quite as much as I could have on this project, but I am hoping I laid a good foundation for a meta-build system for SDL. This week's work will comprise of the terribly-exciting work of cleaning things up and making them presentable. Nevertheless, I ask anyone interested to download the meta-build system, try using one or more of the generated targets, and post any feedback you may have.

I would also to make one little note that I did merge my repository with the latest SDL changes (based on the timestamps of the commit) to reflect the ease of implementing the new and changed features. Take a look at the commit logs for more.

Thanks.

More Configurations, MinGW, and Midterm for GSoC

Howdy,

Halfway done with Google Summer of Code. I have a few quick updates to discuss and then I have a request for the SDL community.

Complex Configurations (pt. 2)

I discussed in my last post how I added support for complex configurations by allowing SDL projects to set preprocessor definitions that would be embedded with a configuration header file that drives the entire SDL build process. This is the same way taken by ./configure and related Linux systems. The system has been implemented at this point and has allowed me to do some interesting stuff.

First of all, the modularity of SDL is much more customizable now than it was previously. I can disable DirectX or OpenGL support with ease. If the meta-build system cannot find one or both of these dependencies, it will appropriately configure SDL to neither compile its related modules nor link to the resulting modules. It becomes rather convenient. I theorized in the beginning of my project that SDL should be able to be built and used on Windows without DirectX support. Recently, I have successfully built and ran all the tests without DirectX support. It works surprisingly well. There are a feel hiccups in the test suites, but that's expected given this was mostly untested behavior prior to this meta-build project.

I hope to expand the configurations systems to new depths by providing command-line arguments for easily enabling and disabling whole modules of SDL, without having to have any previous knowledge about its configuration systems or the meta-build system itself. As this project evolves to more a more modular design, this customization will prove to be very convenient and beneficial. SDL can literally be built dozens of different ways on some systems and I would really like to try and capture that with as much ease as possible. The only thing the end user would need to be aware of is which dependencies SDL has, which can be easily documented.

Full Test Suite

I forgot to mention this in my last post, but I did add support for building and linking the entire SDL test suite to every platform supported. Originally, the Visual Studio portion of SDL only had 10 tests supported. The Xcode version had a few more than that. It turns out there are 42 tests part of the SDL test suite. I was a bit shocked when I noticed I wasn't targeting most of them, so I worked to add support for the whole lot. I've written a lot of convenient build scripts (as I will discuss later) for making building and running these tests as painless as possible.

Mac OS X: Multiple Architectures

I am beginning to look into multiple architecture support, starting with Mac OS X. Currently, the Xcode projects should support building to architectures i386 and x86_64. I am planning on implementing Universal binaries for Mac OS X and any platform that has GCC, so including Windows through MinGW and Cygwin, as well as any Linux platforms. This is continuing the current support for "fat" binaries that SDL offers.

MinGW Support

I decided to be a bit adventurous this week. I implemented MinGW support using GNU makefiles. I hope to expand this project to full make support, but I needed to start with something that would not involve a lot of changes to the current project. MinGW is targeted for building native Windows applications using GCC and the GNU toolchain. Essentially, that implies the Windows setup to project generation is not much different between Visual Studio and MinGW. In some parts this is true, but in others...not so much.

The major differences are in the path separators and the different compilers (obviously). I had to create a MinGW mode (using the --mingw command line option) to signify using POSIX paths, etc. Beyond that, I ran into a lot of trouble over properly linking to the main function of the test applications. I had to use the linker argument -lmingw32 to GCC, though premake was giving me a lot of grief by making it only possible to put it in the wrong place. At the end of the day, I had to modify premake and change the order it generates its linker arguments in the resulting makefiles.

The only other problem with the MinGW support is GCC cannot compile the DirectX and Windows SDK header files correctly. There are MinGW alternatives to DirectX, but these are not the easiest things to work with. I've encountered some problems working with them and I know a bunch of other people have, too. As a result, I am not going to support DirectX with the MinGW generation option. I do plan on adding an override command line argument to the meta-build system for generating MinGW projects that still link to DirectX regardless of it being MinGW, leaving it up to the fate of the developer to try and get the thing to link properly.

Google Summer of Code Midterm!

I have to say I'm a bit surprised that it's already the halfway point. Time does fly when you're really focused on a project. Nevertheless, I am hoping the SDL community is able to help me at this point by pulling my repository and tinkering with it. Please, if you have time, try to give the meta-build system a whirl. The currently supported platforms are Windows using Visual Studio and MinGW and Mac OS X using Xcode. Here are a few things to note:

Obviously the system is very experimental. That's why it needs to be tested. If something goes wrong, feel free to email me, comment on this blog, or post it on the mailing list. I will see it and get back to you very soon. I appreciate any problems or feedback you have.

If you want to just give a quick contribution to the project, simply build the SDL project and test suites for whichever choice platform and IDE. If you run the tests, it can give a quick feedback about whether SDL seems to run mostly normally on your system using the build configurations setup by the meta-build system. I have created several build scripts for each platform for handling various types of builds, as well as testing those builds using the extensive SDL test suite.

If you want to give a more elaborate contribution, I would really appreciate you testing binary compatibility or interoperability by either building or linking any SDL projects you have or are using against the libraries produced by the meta-build projects. This would give excellent feedback as to whether I am correctly generating the projects and that they are behaving as they should so far.

I really appreciate any and all help so far.

Thanks,

Ben

Scrubbing Project Syntax & Complex Configurations

Greetings,

After a six day outing to South Dakota, I'm happy to return and get right back to work on SDL's meta-build system!

Cleaning up Project Syntax

Before, the project syntax was rather icky. It involved a lot of table manipulation, assignment, listing, you name it. I just didn't like it. I didn't plan on keeping it from the beginning. In fact, I wanted a syntax closer to premake's.

With premake, you just do a bunch of function calls in a specific order. However, it doesn't feel like you're making function calls. That's because in Lua, you don't need surrounding parentheses when making a function call if you're passing it one parameter and that parameter is either a table or a string. Conveniently, everything in premake takes either a string or a table. So, in all their examples, they demonstrate passing string literals and inline tables. This gives almost a functional programming feel (which is obviously the end goal on Lua's part). It's just so much cleaner and more elegant. I wanted this.

Hence, I created a series of new functions which maintained a minimal inner state where all the project files were able to call these functions and build projects functionally, rather than uses data structures. Take this example of how  looked before the new system: 

SDL_project = {
 name = "SDL2",
 kind = "SharedLib",
 language = "C++",
 dependencyTree = { },
 uuid = os.uuid(),
 sourcedir = "../src",
 -- as dependencies...?
 customLinks = { }
}

-- statically link on mac osx
if os.get() == "macosx" then
 SDL_project.kind = "StaticLib"
elseif os.get() == "windows" then
 table.insert(SDL_project.customLinks, "winmm")
 table.insert(SDL_project.customLinks, "imm32")
 table.insert(SDL_project.customLinks, "oleaut32")
 table.insert(SDL_project.customLinks, "version")
end

projects["SDL2"] = SDL_project

-- dependency functions must return the following:
-- <foundDep> <name> [includes] [libs] [inputs]
function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

-- TODO: convert this to be functional (like premake), so the syntax isn't as
-- repetitive

-- format is { dependencyLambda }
-- if not in table, it will be excluded from the project
-- if dependency lambda is nil, it will always be included
local dep = SDL_project.dependencyTree;
-- setup dependency tree for SDL 2
dep["/"] = { nil }
dep["/atomic/"] = { nil }
dep["/audio/"] = { nil }
...
dep["/video/"] = { nil }
dep["/video/dummy/"] = { nil }
-- platform-specific implementations
if os.get() == "windows" then
 dep["/audio/directsound/"] = { nil }
 dep["/audio/winmm/"] = { nil }
...
 dep["/render/opengl/"] = { nil }
 -- added exclusion filter to thread/generic to avoid double linking warnings
 -- and incorrect linking
 dep["/thread/generic/"] = { nil, files = { "SDL_syscond.c", "SDL_sysmutex_c.h" } }
 dep["/thread/windows/"] = { nil }
 dep["/timer/windows/"] = { nil }
 dep["/video/windows/"] = { nil }
elseif os.get() == "macosx" then
 dep["/audio/coreaudio/"] = { nil }
 dep["/file/cocoa/"] = { nil }
...
 dep["/video/cocoa/"] = { nil }
 dep["/video/x11/"] = { nil }
else
 print("SDL2 via premake is not supported on platform: " .. os.get())
end

That was the old file, shortened. It's obviously disgusting and ugly. At least, it is compared to the newer version:

function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

function winmmDep()
 print("Checking winmm dependencies...")
 local libpath = os.findlib("winmm")
 local foundLib = libpath ~= nil
 if not foundLib then return false, "winmm" end
 return true, "winmm", { }, { libpath }, { "winmm" }
end

SDL_project "SDL2"
 SDL_kind "SharedLib"
 SDL_language "C++"
 SDL_sourcedir "../src"
 -- dependency tree for SDL2
 SDL_dependency ""
  -- unnamed means it's not a dependency
  -- no OS means platform-independent

  -- this is a minimal setup that should
  -- essentially work on every target platform
  SDL_paths
  {
   "/",
   "/atomic/",
   "/audio/",
   "/audio/disk/",
   "/audio/dummy/",
   "/cpuinfo/",
   "/events/",
   "/file/",
   "/haptic/",
   "/joystick/",
   "/power/",
   "/render/",
   "/render/software/",
   "/stdlib/",
   "/thread/",
   "/timer/",
   "/video/",
   "/video/dummy/"
  }
 -- windows dependencies
 SDL_dependency "windows"
  SDL_os "windows"
  SDL_links { "imm32", "oleaut32", "version" }
  SDL_paths
  {
   "/core/windows/",
   "/haptic/windows/",
   "/joystick/windows/",
   "/libm/",
   "/loadso/windows/",
   "/power/windows/",
   "/render/opengl/",
   "/thread/windows/",
   "/timer/windows/",
   "/video/windows/"
  }
  SDL_files
  {
   -- these files have to be specified uniquely to avoid double
   -- and incorrect linking
   "/thread/generic/SDL_syscond.c",
   "/thread/generic/SDL_sysmutex_c.h"
  }
 -- winmm dependency
 SDL_dependency "winmm"
  SDL_os "windows"
  SDL_depfunc(winmmDep)
  SDL_paths { "/audio/winmm/" }
 -- directx dependency
 SDL_dependency "directx"
  SDL_os "windows"
  SDL_depfunc(directXDep)
  SDL_paths
  {
   "/audio/directsound/",
   "/audio/xaudio2/",
   "/render/direct3d/"
  }
 SDL_dependency "macosx"
  SDL_os "macosx"
  SDL_paths
  {
   "/audio/coreaudio/",
   "/file/cocoa/",
   "/haptic/darwin/",
   "/joystick/darwin/",
   "/loadso/dlopen/",
   "/power/macosx/",
   "/render/opengl/",
   "/thread/pthread/",
   "/timer/unix/",
   "/video/cocoa/",
   "/video/x11/"
  }

The beauty of this new code goes without explanation, but I will point some things out. I didn't have to trim it at all in my paste here, because it's hardly longer than the trimmed other version. The code length is shorter and there's less to type. I also increased the length of this version by separating the winmm dependencies from the general windows dependencies. The flexibility here is self-explanatory.

Complex SDL Configurations

I mentioned or hinted a while back at wanting more sophisticated means of building SDL. An example I gave was the ability to build SDL without any sort of native renderer, and having it still work. Ie, the windows solution file for SDL should not depend on OpenGL or DirectX at all. If someone doesn't have DirectX installed, they should still be able to build SDL. Granted, they will be missing a significant number of features, but that doesn't mean it shouldn't build, link, and run correctly.

However, simply omitting a few libraries from SDL is not enough to stop it from expecting those links. If I do not include /render/opengl, it will compile fine, but it won't link fine. Why? Because SDL is expecting the functions in /render/opengl to be there. How do we stop SDL from expecting something we don't want to be there?

Enter configure and autotools.

SDL's "assumption system" is based on a single configuration header file, which differs from operating system to operating system (on windows, it's SDL_config_windows.h). On a Unix environment, one can use the configure script to check for system dependencies and generate an appropriate config header file that allows SDL to build properly. This is very nice, but it was not taken into consideration for Windows or Mac OS X.

I've implemented a new function in the functionally-driven project definition system for setting up custom defines that can be pasted into a generated config file. For example, the following configuration directive is for setting up the defines needed to enable DirectX support in SDL:

  SDL_config
  {
   ["SDL_AUDIO_DRIVER_DSOUND"] = 1,
   ["SDL_AUDIO_DRIVER_XAUDIO2"] = 1,
   ["SDL_JOYSTICK_DINPUT"] = 1,
   ["SDL_HAPTIC_DINPUT"] = 1,
   ["SDL_VIDEO_RENDER_D3D"] = 1
  }

The only thing next to do is to decide whether to generate a config file or try to use compiler-level defines to simply set flags. Due to various defines in the SDL_config_windows.h file, I cannot just set compiler-level defines, unless I change that file to use #ifndef when setting some of the defines (it does it for many of the other defines, which is nice).

After that is decided, I'll be able to support a lot more flexibility when building SDL than was possible before on both Windows and Mac OS X. Also, if the build system is extended to other platforms, it will conveniently similar to the processing of configure and autotools for generating a makefile, if developers are interested in that.

Until next time,

Ben

Bringing support to Mac OS X

Greetings readers,

After getting acceptable support for Visual Studio 2008, 2010, and 2012 on Windows, it's time to turn attention over to Mac OS X and Xcode.

Premake and Xcode

Not being at all familiar with Xcode, I had to spend a lot of time last week simply wrapping my head around its many differences (as opposed to Visual Studio, Eclipse, etc.) I still have a lot to learn with this IDE, but I think I figured out enough to work through this part of the project. The first step in this process was cloning my GSoC repository on Mountain Lion and making sure the hand-rolled Xcode projects would build, link, and run properly. They did, as the SDL buildbot said they should.

Next, I basically just copied over the Visual Studio meta-build premake setup I've been working on and used it to generate Xcode 4 projects. Needless to say, it obviously did not build. After getting rid of the windows-specific dependencies and cross referencing with the files the manual Xcode project included, I was able to reproduce a buildable, linkable, and runnable SDL within a matter of hours over just a few days.

However, like everything else I've had to deal with when using premake, there was one concern I had. After reading some changes to premake over the last couple years, I thought it would generate an Xcode project for each of my defined projects in the generated lua file. Regardless, I was happy to find out it generated a workspace file which works essentially identically to that of a VS solution file.

The main pieces left to work out are various configurations on Xcode (such as having multiple build schemas), properly handling architecture differences (the generated builds for 64-bit but SDL was previously built for 32-bit), and ensuring the new setup for SDL's Xcode projects is acceptable.

Consolidating Differences

The next challenge was consolidating the differences between the VS and Xcode solution files. As per my mentor's suggestion, making major directory changes in a repository is often ill-advised, so if I must do it, I should do it asap.

What directory changes? Well, I originally created my premake solution for VS in SDL's VisualC folder. To be similar, I also created a similar directory in SDL's Xcode folder recently. This is obviously not a happy solution to keeping things simple using premake, so my most recent work prior to this blog post had been creating a premake folder in the root SDL folder, removing the other premake folders, and consolidating all build configurations into a single, unified script. I also created subdirectories for VisualC and Xcode within that premake folder. That last part was quickly thrown together, so more work will be done to ensure the projects are generated and built relative to those folders, which they currently are not.

Consolidating the two files has also led to increased hardcoding and, thus, some more complexity. A lot of future work will be to cleanup the central premake file, simplify the dependency tree and project definitions using a lighter syntax (like premake uses for its project and solution definitions), and determining a smart way to do cross-platform dependency checking. Xcode and premake are smart enough to make the hassle of porting from VS to Xcode very minimal (especially with a reference Xcode project), but other complexities have arisen that need to be taken care of, especially if any sort of template system is going to be implemented or if even more platforms are targeted.

Until the next post,

Ben

Binary Compatibility

Hello!

As I continue to anticipate starting development for the XCode project generation, I instead turned toward working on binary compatibility verification for the current Visual Studio solutions. This proved to be an interesting test of determination and perplexity.

What is Binary Compatibility & Why need it?

Binary compatibility for libraries means that if I build an executable against some dynamic library and then that library is updated, my executable should be able to work with the new shared library without requiring recompilation or relinking. It becomes immediately obvious some of the problems that can occur when upgrading the shared library. Nevertheless, why am I talking about needing to test binary compatibility when working on a meta-build system when I'm not changing the library source code at all?

This may or may not be evident. We strive to achieve binary compatibility between applications built in the past and the SDL shared library file built by the generated solutions. Since the generated project leads to different build and link flags (see below for why that is), it's possible that the resulting SDL2.dll file from the generated projects work perfectly fine with the test suites, but perhaps not fine with applications already built using different compiler/linker flags.

Thus, the goal is to figure out what flags are necessary to achieve binary compatibility. Well...why can't I just replicate the existing build and linker options when generating the solution files using premake?

Premake's "Specific" Pitfalls

Premake allows for incredible flexibility when defining how a particular project should be built over many systems. However, it has its own pitfalls. As a general consequence of abstracting varied systems, premake mostly supports features that exist across all or most of the platforms being targeted by a given premake file. As a result, it makes certain assumptions about premake flags set, plus it doesn't allow for complete flexibility when creating IDE project files, such as for Visual Studio. As a result, it's very difficult to create warning-free projects that perfectly replicate all the build and link settings as the original SDL project without modifying premake.

There are many workarounds, but for now, the goal is simply working with the current situation instead of making things more complicated. Also, several assumptions premake has made makes it difficult to correctly set /MTd versus /MDd in Visual Studio, which is frustrating for me. This was worked around by switching to premake 5 dev, which seems to handle this situation better.

Results

Right now, the generated Visual Studio project files build a SDL2.dll file which is completely horizontally compatible with the test programs built using the manual Visual Studio project files. The converse, however, does not work correctly (failing with an error saying the C runtime library is being loaded incorrectly). Nevertheless, this is not as desired of a feature, so it working is not as important. As perfectly replicating the build and link options in the premake-generated solution becomes more important, I will go back and ensure this compatibility is correct, as well.

"Meta-"meta-build system

I just want to quickly note that the premake setup is now being used to traverse dependencies and generate a premake file, which is then executed and the resulting projects are generated by premake. This "meta-meta-build system" was discussed briefly in an older post. It allows for forward compatibility with premake 5 dev (which is useful, as described above) and maintains backward compatibility with premake 4. It should also ease troubleshooting any problems, since the exact text being fed to preamke can be dumped to a lua file and viewed manually.

Until the next time,

Ben

First Week: Finding a solution to dependencies in premake

Greetings readers,
This post is about my first week of experiences with the GSoC project of recreating the meta-build system for Simple DirectMedia Layer 2.0 (SDL). I'll briefly go over the events. For a more detailed and less wordy explanation of what happened, check out the RSS feed on the side or look at the push history of my bitbucket repository:

https://bitbucket.org/gsocben/sdl-gsoc-2013

First Day

Upon starting my first week of this meta-build system for SDL, I immediately ran into some roadblocks. Besides having to setup various aspects of my improved development environment (which was interesting, to say the least), I realized portions of this project I hadn't previously considered which were standing in my way.

I spent the first day building a completely project-specific LUA script for building the SDL2 project (builds SDL2.dll) on Visual Studio 2010. This proved to be non-straightforward, because the SDL source folder contains a huge conglomeration of modules, some of which are platform-specific and others which are platform-independent. More importantly, the exact order of dependencies is rather confusing and unknown without thoroughly reading through the source code. In particular, I noticed there were 'dummy' folders for modules like audio, video, etc. I assumed these folders were replacements for the audio, video, etc. modules if they could not bind to a library like OpenGL or DirectX. I proved to be right, although the idea of not depending on DirectX was not a previous concern for SDL VS solution authors.

I decided to rethink my approach to doing the build system. I spent a bit of time brainstorming various aspects of the build system and this is what I came up with:

1. Consolidation: a single "code" base which reflects desired configurations for all target platforms
2. Automated project generation (metabuild): nice for when dependencies change, the source tree changes, or the user decides on altering configurations
3. Configuration reflection: rather than templates, having a way to store manually-specified settings per-project if projects are to be regenerated
4. Dependency resolution: ability to resolve dependencies that are absolutely necessary, along with those which are optional (see below)
5. Dependency tree with source modules: cross-platform means of marking the dependencies a specific source folder or source file requires, thus allowing for more complex permutations of projects to be handled and significantly more flexibility, at minimal overhead for the developers
6. Default projects: SDL should come with default projects which *may* work out of the box; if they fail, the user may generate a new project using the build system and possibly receive more verbose information as to why it may not build, plus possible options to ignore non-required dependencies (such as the DirectSound example above)
7. Portability: CMake's greatest drawback is its need to be installed to use; Premake or other portable systems are the choice, if they are capable of handling the other features here

I will not go into detail about each specific point. If they end up becoming part of this project, I will emphasize them, or others, in future posts. As far as #7 goes, that was me implying the possibility of not using premake as a solution for this GSoC project. I'm still standing by using premake, although my approach may vary tremendously from my original vision.

Project Dependencies

What do you do when you have a project which has a varied number of dependencies, changing from platform to platform? More importantly, what do you do when these dependencies are optional and how do you communicate that to the user?

These questions, among others, plagued my thoughts for a while this week. I decided on coming up with a simple approach for handling project dependencies, with the thought of expanding it. But first, let's investigate some code of a typical Premake project LUA file (this is the first iteration of the SDL2 project, as available on bitbucket):

 solution "SDL"
  configurations { "Debug" }

 project "SDL2"
  targetname "SDL2"
  kind "SharedLib"
  language "C"
  flags { "NoRTTI", "NoExceptions" }
  includedirs { "../../include", "$(DXSDK_DIR)\Include" }
  libdirs { "$(DXSDK_DIR)\Lib\x86" }
  
  files
  {
   "../../src/*.c",
   "../../src/*.h",
   "../../src/atomic/*.c",
   "../../src/atomic/*.h",
   "../../src/audio/*.c",
   "../../src/audio/*.h",
   "../../src/audio/directsound/*.c",
   "../../src/audio/directsound/*.h",
   "../../src/audio/disk/*.c",
   "../../src/audio/disk/*.h",
   "../../src/audio/dummy/*.c",
   "../../src/audio/dummy/*.h",
   "../../src/audio/winmm/*.c",
   "../../src/audio/winmm/*.h",
   "../../src/audio/xaudio2/*.c",
   "../../src/audio/xaudio2/*.h",
   "../../src/core/windows/*.c",
   "../../src/core/windows/*.h",
   "../../src/cpuinfo/*.c",
   "../../src/cpuinfo/*.h",
   "../../src/events/*.c",
   "../../src/events/*.h",
...
   "../../src/video/windows/*.c",
   "../../src/video/windows/*.h"
  }
  
  excludes
  {
   "*/*psp*/*"
  }
  
  configuration "Debug"
   defines { "_DEBUG", "_WINDOWS" }
   buildoptions { "/GS-", "/Gy-", "/MDd" }
   linkoptions { "/INCREMENTAL:NO" }
   links { "winmm", "imm32", "oleaut32", "version" }

As can be seen, this is a rather annoying and 'verbose' means of maintaining a project. More importantly, the dependencies are hardcoded for the include paths and the library links. So what do we do? How do we handle an abundance of various test SDL projects (totaling 16 projects in the current SDL solution), on top of both their internal and external dependencies?

Honestly, I didn't want to have to write separate project LUA files for each project of the SDL solution. More importantly, I didn't want anyone else to have to maintain and add on to such a system. How to solve this problem? To be horribly cliche...

"All problems in computer science can be solved by another level of indirection" 

-David Wheeler

I decided to abstract the similar functionality that each build system depends on. It's sort of the idea of not recreating similar code when it can be created once for many situations. In order to facilitate a single system to handle many build systems, I had to come up with the idea of using a dependency tree for each project. I also decided to use the idea of automatic folder-based inclusion with specific file exclusion.

Here is what the latest LUA file looks like for generating the SDL2 project in VS2008-VS2012:

SDL_project = {
 name = "SDL2",
 kind = "SharedLib",
 language = "C",
 dependencyTree = { },
 uuid = os.uuid(),
 sourcedir = "../../src",
 -- as dependencies...?
 customLinks = { "winmm", "imm32", "oleaut32", "version" }
}

projects["SDL2"] = SDL_project

-- dependency functions must return the following:
--   [includes] [libs] [inputs]
function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

-- TODO: convert this to be functional (like premake), so the syntax isn't as
-- repetitive

-- format is { dependencyLambda }
-- if not in table, it will be excluded from the project
-- if dependency lambda is nil, it will always be included
local dep = SDL_project.dependencyTree;
-- setup dependency tree for SDL 2
dep["/"] = { nil }
dep["/atomic/"] = { nil }
dep["/audio/"] = { nil }
dep["/audio/directsound/"] = { nil }
dep["/audio/disk/"] = { nil }
dep["/audio/dummy/"] = { nil }
dep["/audio/winmm/"] = { nil }
dep["/audio/xaudio2/"] = { directXDep }
dep["/core/windows/"] = { nil }
dep["/cpuinfo/"] = { nil }
dep["/events/"] = { nil }
dep["/file/"] = { nil }
dep["/file/cocoa/"] = { nil }
dep["/haptic/"] = { nil }
dep["/haptic/windows/"] = { nil }
dep["/joystick/"] = { nil }
dep["/joystick/windows/"] = { nil }
dep["/libm/"] = { nil }
dep["/loadso/windows/"] = { nil }
dep["/power/"] = { nil }
dep["/power/windows/"] = { nil }
dep["/render/"] = { nil }
dep["/render/direct3d/"] = { nil }
dep["/render/opengl/"] = { nil }
dep["/render/software/"] = { nil }
dep["/stdlib/"] = { nil }
dep["/thread/"] = { nil }
-- added exclusion filter to thread/generic to avoid double linking warnings
-- and incorrect linking
dep["/thread/generic/"] = { nil, files = { "SDL_syscond.c", "SDL_sysmutex_c.h" } }

This is actually the entire file, minus comments. This, combined with the automatic system defined in premake4.lua, generates a working project file for VS2008, VS2010, and VS2012 for correctly building and linking SDL2.dll on Win32 combined with a DirectX dependency.

My hope was to create a dependency system wherein I could easily manage dependencies per-project (including inter-project dependencies, like how the test suite depends on SDL and SDLtest). I'm able to define a function which uses my own utility functions to search for dependencies. The next evolution of this system will be to support an array of dependency functions, based on handling the three following features:

1. Some dependencies are required; if they are not met, the project cannot be built or linked
2. Other dependencies are recommended before some dependencies, meaning there must be a certain order to handling dependencies
3. If multiple dependencies are met, the user should be prompted on which to use or, in the very least, an option to specify usage and override default functionality

The above allow for a robust and reliable resolution of dependencies, but in a way that lets users customize the build well beyond what SDL currently supports. Going back to the earlier example, a user should very well be able to build SDL without DirectX (regardless of how limiting it may be, but that's why the software rasterizer exists). Whether this works or not is mere theory and needs testing, but the functionality to achieve this build should be available to the end-programmer, without them having to understand the innerworkings of SDL.

Hacking And Abusing Premake

It turns out that creating a dynamic dependency-handling system in Premake is more possible than I originally expected. A lot of what I did was more series of a "what-if"s that turned out to be plausible. Intentionally, Premake has a nice, relaxed syntactical structure to it which, thanks to LUA, calls various functions which handle internal states about your build configurations, solution settings, and projects. This is all statically written, though. In order to dynamically create solutions and projects, I had to hope Premake worked the way I needed it to, and that invoking its functions using variables and changing its state arbitrarily would keep things working as expected. It turns out it did.

I would not recommend this at all, because it involved a lot of trial and error, among other difficulties. If you wish to see what I did to exploit Premake and dynamically create solutions and projects, check out the latest premake4.lua on bitbucket ) (commit beef0b8). It's a bit too messy and long to paste in here, but it handles generating every single project in the SDL solution quite nicely. I hope to replace it with a generation system (as mentioned below) later.

All assumptions have a possibility of leading to disastrous results. Upon testing my dynamic solution and project code with Premake 5 (development branch), it turned out the author had change the way he was handling the internal Premake script context. This completely broke my system and I have yet to find a solution for fixing it. Although my solution seems to work excellently on Premake 4, it seems to not be forward compatible with Premake 5. There are various options I have moving forward, one of which I explore here and others which I may explore later, if they become practical.

Moving Forward

Upon the first week of GSoC, I managed to assemble a basic build system that correctly replicates the current Visual Studio SDL solutions for VS2008, VS2010, and VS2012. There is much more testing to be done, features that need to be cleaned up, etc. Beyond that, the hackery of Premake mentioned above will becoming increasingly difficult to maintain and likely not forward-compatible. In order to mitigate this upcoming problem, it would be nice to build a sort of meta-meta-build system that generates the LUA files instructing Premake what to do. It would essentially just be a scanner handling dependencies and allowing LUA to do its job naturally, without coercion.

Beyond that, adding support for XCode is absolutely vital and a focus for upcoming weeks. Due to my lack of owning a Macintosh machine, this area of development will be halted for the time being. It will become my first priority upon being able to start it.

Finally, I like to think about long-stretch goals every once in a while, so mentioning them here would be nice, too. Ultimately, I would be happy with this project if I can move it to target other build  environments, such as the Android, MinGW, and Cygwin environments on Windows. Again, it's more consolidation of build settings. It would also be very nice to support some sort of project template system, where SDL developers are able to modify settings in a global template file and the premake system would be able to notice these differences and apply them appropriately to generated projects. My own spin on this suggestion is to allow users to change settings in the generated project, wherein the meta-meta-build system would scan it and be aware of the changes from the default, thus reapplying them and leading to a more manageable system of settings handling (though, with its own cons, of course).

Thus concludes my eventful first week of GSoC.

Until the next time,

Ben