First Week: Finding a solution to dependencies in premake

Greetings readers,
This post is about my first week of experiences with the GSoC project of recreating the meta-build system for Simple DirectMedia Layer 2.0 (SDL). I'll briefly go over the events. For a more detailed and less wordy explanation of what happened, check out the RSS feed on the side or look at the push history of my bitbucket repository:

https://bitbucket.org/gsocben/sdl-gsoc-2013

First Day

Upon starting my first week of this meta-build system for SDL, I immediately ran into some roadblocks. Besides having to setup various aspects of my improved development environment (which was interesting, to say the least), I realized portions of this project I hadn't previously considered which were standing in my way.

I spent the first day building a completely project-specific LUA script for building the SDL2 project (builds SDL2.dll) on Visual Studio 2010. This proved to be non-straightforward, because the SDL source folder contains a huge conglomeration of modules, some of which are platform-specific and others which are platform-independent. More importantly, the exact order of dependencies is rather confusing and unknown without thoroughly reading through the source code. In particular, I noticed there were 'dummy' folders for modules like audio, video, etc. I assumed these folders were replacements for the audio, video, etc. modules if they could not bind to a library like OpenGL or DirectX. I proved to be right, although the idea of not depending on DirectX was not a previous concern for SDL VS solution authors.

I decided to rethink my approach to doing the build system. I spent a bit of time brainstorming various aspects of the build system and this is what I came up with:

1. Consolidation: a single "code" base which reflects desired configurations for all target platforms
2. Automated project generation (metabuild): nice for when dependencies change, the source tree changes, or the user decides on altering configurations
3. Configuration reflection: rather than templates, having a way to store manually-specified settings per-project if projects are to be regenerated
4. Dependency resolution: ability to resolve dependencies that are absolutely necessary, along with those which are optional (see below)
5. Dependency tree with source modules: cross-platform means of marking the dependencies a specific source folder or source file requires, thus allowing for more complex permutations of projects to be handled and significantly more flexibility, at minimal overhead for the developers
6. Default projects: SDL should come with default projects which *may* work out of the box; if they fail, the user may generate a new project using the build system and possibly receive more verbose information as to why it may not build, plus possible options to ignore non-required dependencies (such as the DirectSound example above)
7. Portability: CMake's greatest drawback is its need to be installed to use; Premake or other portable systems are the choice, if they are capable of handling the other features here

I will not go into detail about each specific point. If they end up becoming part of this project, I will emphasize them, or others, in future posts. As far as #7 goes, that was me implying the possibility of not using premake as a solution for this GSoC project. I'm still standing by using premake, although my approach may vary tremendously from my original vision.

Project Dependencies

What do you do when you have a project which has a varied number of dependencies, changing from platform to platform? More importantly, what do you do when these dependencies are optional and how do you communicate that to the user?

These questions, among others, plagued my thoughts for a while this week. I decided on coming up with a simple approach for handling project dependencies, with the thought of expanding it. But first, let's investigate some code of a typical Premake project LUA file (this is the first iteration of the SDL2 project, as available on bitbucket):

 solution "SDL"
  configurations { "Debug" }

 project "SDL2"
  targetname "SDL2"
  kind "SharedLib"
  language "C"
  flags { "NoRTTI", "NoExceptions" }
  includedirs { "../../include", "$(DXSDK_DIR)\Include" }
  libdirs { "$(DXSDK_DIR)\Lib\x86" }
  
  files
  {
   "../../src/*.c",
   "../../src/*.h",
   "../../src/atomic/*.c",
   "../../src/atomic/*.h",
   "../../src/audio/*.c",
   "../../src/audio/*.h",
   "../../src/audio/directsound/*.c",
   "../../src/audio/directsound/*.h",
   "../../src/audio/disk/*.c",
   "../../src/audio/disk/*.h",
   "../../src/audio/dummy/*.c",
   "../../src/audio/dummy/*.h",
   "../../src/audio/winmm/*.c",
   "../../src/audio/winmm/*.h",
   "../../src/audio/xaudio2/*.c",
   "../../src/audio/xaudio2/*.h",
   "../../src/core/windows/*.c",
   "../../src/core/windows/*.h",
   "../../src/cpuinfo/*.c",
   "../../src/cpuinfo/*.h",
   "../../src/events/*.c",
   "../../src/events/*.h",
...
   "../../src/video/windows/*.c",
   "../../src/video/windows/*.h"
  }
  
  excludes
  {
   "*/*psp*/*"
  }
  
  configuration "Debug"
   defines { "_DEBUG", "_WINDOWS" }
   buildoptions { "/GS-", "/Gy-", "/MDd" }
   linkoptions { "/INCREMENTAL:NO" }
   links { "winmm", "imm32", "oleaut32", "version" }

As can be seen, this is a rather annoying and 'verbose' means of maintaining a project. More importantly, the dependencies are hardcoded for the include paths and the library links. So what do we do? How do we handle an abundance of various test SDL projects (totaling 16 projects in the current SDL solution), on top of both their internal and external dependencies?

Honestly, I didn't want to have to write separate project LUA files for each project of the SDL solution. More importantly, I didn't want anyone else to have to maintain and add on to such a system. How to solve this problem? To be horribly cliche...

"All problems in computer science can be solved by another level of indirection" 

-David Wheeler

I decided to abstract the similar functionality that each build system depends on. It's sort of the idea of not recreating similar code when it can be created once for many situations. In order to facilitate a single system to handle many build systems, I had to come up with the idea of using a dependency tree for each project. I also decided to use the idea of automatic folder-based inclusion with specific file exclusion.

Here is what the latest LUA file looks like for generating the SDL2 project in VS2008-VS2012:

SDL_project = {
 name = "SDL2",
 kind = "SharedLib",
 language = "C",
 dependencyTree = { },
 uuid = os.uuid(),
 sourcedir = "../../src",
 -- as dependencies...?
 customLinks = { "winmm", "imm32", "oleaut32", "version" }
}

projects["SDL2"] = SDL_project

-- dependency functions must return the following:
--   [includes] [libs] [inputs]
function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

-- TODO: convert this to be functional (like premake), so the syntax isn't as
-- repetitive

-- format is { dependencyLambda }
-- if not in table, it will be excluded from the project
-- if dependency lambda is nil, it will always be included
local dep = SDL_project.dependencyTree;
-- setup dependency tree for SDL 2
dep["/"] = { nil }
dep["/atomic/"] = { nil }
dep["/audio/"] = { nil }
dep["/audio/directsound/"] = { nil }
dep["/audio/disk/"] = { nil }
dep["/audio/dummy/"] = { nil }
dep["/audio/winmm/"] = { nil }
dep["/audio/xaudio2/"] = { directXDep }
dep["/core/windows/"] = { nil }
dep["/cpuinfo/"] = { nil }
dep["/events/"] = { nil }
dep["/file/"] = { nil }
dep["/file/cocoa/"] = { nil }
dep["/haptic/"] = { nil }
dep["/haptic/windows/"] = { nil }
dep["/joystick/"] = { nil }
dep["/joystick/windows/"] = { nil }
dep["/libm/"] = { nil }
dep["/loadso/windows/"] = { nil }
dep["/power/"] = { nil }
dep["/power/windows/"] = { nil }
dep["/render/"] = { nil }
dep["/render/direct3d/"] = { nil }
dep["/render/opengl/"] = { nil }
dep["/render/software/"] = { nil }
dep["/stdlib/"] = { nil }
dep["/thread/"] = { nil }
-- added exclusion filter to thread/generic to avoid double linking warnings
-- and incorrect linking
dep["/thread/generic/"] = { nil, files = { "SDL_syscond.c", "SDL_sysmutex_c.h" } }

This is actually the entire file, minus comments. This, combined with the automatic system defined in premake4.lua, generates a working project file for VS2008, VS2010, and VS2012 for correctly building and linking SDL2.dll on Win32 combined with a DirectX dependency.

My hope was to create a dependency system wherein I could easily manage dependencies per-project (including inter-project dependencies, like how the test suite depends on SDL and SDLtest). I'm able to define a function which uses my own utility functions to search for dependencies. The next evolution of this system will be to support an array of dependency functions, based on handling the three following features:

1. Some dependencies are required; if they are not met, the project cannot be built or linked
2. Other dependencies are recommended before some dependencies, meaning there must be a certain order to handling dependencies
3. If multiple dependencies are met, the user should be prompted on which to use or, in the very least, an option to specify usage and override default functionality

The above allow for a robust and reliable resolution of dependencies, but in a way that lets users customize the build well beyond what SDL currently supports. Going back to the earlier example, a user should very well be able to build SDL without DirectX (regardless of how limiting it may be, but that's why the software rasterizer exists). Whether this works or not is mere theory and needs testing, but the functionality to achieve this build should be available to the end-programmer, without them having to understand the innerworkings of SDL.

Hacking And Abusing Premake

It turns out that creating a dynamic dependency-handling system in Premake is more possible than I originally expected. A lot of what I did was more series of a "what-if"s that turned out to be plausible. Intentionally, Premake has a nice, relaxed syntactical structure to it which, thanks to LUA, calls various functions which handle internal states about your build configurations, solution settings, and projects. This is all statically written, though. In order to dynamically create solutions and projects, I had to hope Premake worked the way I needed it to, and that invoking its functions using variables and changing its state arbitrarily would keep things working as expected. It turns out it did.

I would not recommend this at all, because it involved a lot of trial and error, among other difficulties. If you wish to see what I did to exploit Premake and dynamically create solutions and projects, check out the latest premake4.lua on bitbucket ) (commit beef0b8). It's a bit too messy and long to paste in here, but it handles generating every single project in the SDL solution quite nicely. I hope to replace it with a generation system (as mentioned below) later.

All assumptions have a possibility of leading to disastrous results. Upon testing my dynamic solution and project code with Premake 5 (development branch), it turned out the author had change the way he was handling the internal Premake script context. This completely broke my system and I have yet to find a solution for fixing it. Although my solution seems to work excellently on Premake 4, it seems to not be forward compatible with Premake 5. There are various options I have moving forward, one of which I explore here and others which I may explore later, if they become practical.

Moving Forward

Upon the first week of GSoC, I managed to assemble a basic build system that correctly replicates the current Visual Studio SDL solutions for VS2008, VS2010, and VS2012. There is much more testing to be done, features that need to be cleaned up, etc. Beyond that, the hackery of Premake mentioned above will becoming increasingly difficult to maintain and likely not forward-compatible. In order to mitigate this upcoming problem, it would be nice to build a sort of meta-meta-build system that generates the LUA files instructing Premake what to do. It would essentially just be a scanner handling dependencies and allowing LUA to do its job naturally, without coercion.

Beyond that, adding support for XCode is absolutely vital and a focus for upcoming weeks. Due to my lack of owning a Macintosh machine, this area of development will be halted for the time being. It will become my first priority upon being able to start it.

Finally, I like to think about long-stretch goals every once in a while, so mentioning them here would be nice, too. Ultimately, I would be happy with this project if I can move it to target other build  environments, such as the Android, MinGW, and Cygwin environments on Windows. Again, it's more consolidation of build settings. It would also be very nice to support some sort of project template system, where SDL developers are able to modify settings in a global template file and the premake system would be able to notice these differences and apply them appropriately to generated projects. My own spin on this suggestion is to allow users to change settings in the generated project, wherein the meta-meta-build system would scan it and be aware of the changes from the default, thus reapplying them and leading to a more manageable system of settings handling (though, with its own cons, of course).

Thus concludes my eventful first week of GSoC.

Until the next time,

Ben