Linux, Cygwin, iOS, and moving forward (CMake/autotools on premake?)

Howdy,

It's been a while since my last post, but I want to take some time to let the community know the latest progress. My work on Google Summer of Code is almost up. I intend on writing one more post after this to wrap things up.

Over the last few weeks, I implemented support for Linux (experimental), Cygwin (rather slim), and iOS. I've implemented a few functions that allow me to do a lot of the same dependency checking as CMake and autotools, as I will discuss below. I have also cleaned up the dependency system some and intend on continuing to make the meta-build system as clean and easy-to-understand as possible.

iOS

For starters, I wanted iOS support in the meta-build system almost since the beginning of the project. It seemed like a very interesting system to target, but not without its potential difficulties. Because of desired iOS support, I need to make several modifications to the premake source, all of which were patched into the current stable branch of premake. One of the patches another fellow wrote which provided basic iOS functionality. There was still a lot of other things to modify in order to get iOS to work properly.

Nevertheless, it is working now. I've created a separate Xcode directory specifically for iOS. The same demos used in the current iOS project are being referenced to and tested with the generated system. There are a few minor things related to the logo, startup image, and codesigning that will be more challenging to solve, and may remain unsolved by the end of this project. The goal was getting basic iOS support without much effort on behalf of the developers using the system, and I think I successfully achieved that.

Cygwin

Cygwin was proving especially challenging. Upon implementing MinGW (check my last post), I really wanted to implement Cygwin. MinGW proved incredibly challenging because of a bug in premake that I needed to fix, but Cygwin is even more complicated. For those who are not aware, MinGW is setup to create Windows applications using a POSIX environment (GCC, Make, SH, etc.) Cygwin is designed to write POSIX/Unix applications on Windows. That means that the sources that are compiled for each of these similar environments are very different. The Cygwin target actually compiles the Linux version of SDL, not the Windows version.

The next hurdle was me not being able to successfully build the current SDL project using autotools. Autotools refused to build with Cygwin unless its cross-compiled GCC could produce a Windows executable. Well, of course GCC can produce Windows executables, so this error is a bit non-intuitive. It's essentially saying that there can't be a dependency on Cygwin, which is why the authors of the cross-compiler implementing a -mno-cygwin flag. However, this flag is only available on gcc-3, which is no longer shipped with Cygwin. This approach to writing Cygwin-independent applications is now obsolete. It's recommended to use MinGW-w64 instead.

I decided to provide a stripped implementation that would work on Cygwin. It basically just has threading and file support. Everything major like video and audio has been completely stripped due to time and complexity. This functionality is completely possible to implement, but I just felt like Cygwin was the least important target, so I didn't invest too much time into it. The executables produced by the make files generated from the meta-build system will be dependent on cygwin1.dll.

Linux

Currently only tested on Linux Mint 15, I've began to implement Linux support. I had this a while back (I started Linux support to help with Cygwin support), but it's now reaching a much better level. The sheer magnitude of achieving complete Linux support is an entire project in itself (just take a look at the CMake or autotools config files sometime). Therefore, my disclaimer here is the Linux target is completely experimental, incomplete, and likely containing issues that will only be found after it's been tested with many different flavors of Linux.

My goal was not to implement complete Linux support but, rather, to give a good starter for me or other people to work off of in the future. The current meta-build systems allowed me to implement Linux support with ease, but the ability to test for the monolithic list of dependencies SDL is capable of having is well beyond premake. I will discuss this in greater detail in the next section.

I say that SDL is capable of having these dependencies because it's modular. It can have a half a dozen audio devices or just one. SDL can be built on a minimalist configuration file that basically just loads dummy drivers for the entire system. My goal wasn't just to get some shell of SDL working, it was to reproduce the current functionality I am able to build with autotools itself. With that, any features beyond that autotools configures SDL for will not be supported with the results of this GSoC project. That's not to say it wouldn't take just a few minutes to implement them, though. I daresay testing takes much longer than implementing new features in this system.

One final note is about the largest difference between the autotools Linux setup and the one from premake: building on Linux should produce a shared library, just as on Windows. However, due to how GCC handles linking to shared libraries, the best approach to linking is to have the shared libraries in a public location (such as the /usr/lib folder, hence its existence). The problem is premake generates makefiles without custom targets. I am not able to implement an 'install' target for the shared SDL2 library, so I cannot link to it there. My only solutions are to do post-build copies of the library to the correct public location (which seems hackish and non-intuitive for the developer), keep the shared library bound to the same directory as the executable (incredibly inconvenient), or to simply produce a static library. I opted for the last option for times' sake. This is definitely an excellent aspect to improve on in the future.

Dependency Checking: Evolved from CMake and Autotools

My system is setup to produce dependency directives in project files that depend on some named function. In another file, a function is paired with its name and registered in a table. I decided to use functions for dependency handling because dependency checking can depend on a huge variety of factors, many of which aren't continuously considered. Functions can do essentially anything, so they were the ideal choice.

What do these functions do? Well, they are uniquely expected to determine whether their target dependency is on the the current system. Beyond that, they may or may not need to provide directories for header files, directories for static libraries, or a list of libraries to link to. The question yet remains...how do we find whether we have the dependency?

CMake and Autotools use the functionality for finding a library using common, public locations of where the library may be residing. Premake has this functionality as well in its os.findlib function. Where CMake and Autotools start to shine over premake is their ability to check dependencies by generating basic source files and seeing whether they compile, link, or run, depending on what's being checked. I decided that this functionality is extremely interesting and very useful, so I've implemented my own basic system for supporting a handful of the features of CMake, specifically.

The hope is to allow much better support for dependency checking than just looking for the library. Source files allow checking of compiler flags, checking for the size of certain datatypes, and whether built-in library functions exist. The possibilities are nearly limitless, so I am excited to see how these features are used in the future. I intend on demonstrating their use partially in some of the Linux dependency functions. I must note, however, that I only slightly emphasized cross-platform support. Right now it specifically focuses on GCC, but Microsoft compiler, Clang, Borland, or whatever else would be possible to add, if desired. The system would just need to be reworked slightly.

Moving Forward

Over the next week I hope to make the meta-build system as easy to understand, use, and extend as possible. I did not implement quite as much as I could have on this project, but I am hoping I laid a good foundation for a meta-build system for SDL. This week's work will comprise of the terribly-exciting work of cleaning things up and making them presentable. Nevertheless, I ask anyone interested to download the meta-build system, try using one or more of the generated targets, and post any feedback you may have.

I would also to make one little note that I did merge my repository with the latest SDL changes (based on the timestamps of the commit) to reflect the ease of implementing the new and changed features. Take a look at the commit logs for more.

Thanks.

First Week: Finding a solution to dependencies in premake

Greetings readers,
This post is about my first week of experiences with the GSoC project of recreating the meta-build system for Simple DirectMedia Layer 2.0 (SDL). I'll briefly go over the events. For a more detailed and less wordy explanation of what happened, check out the RSS feed on the side or look at the push history of my bitbucket repository:

https://bitbucket.org/gsocben/sdl-gsoc-2013

First Day

Upon starting my first week of this meta-build system for SDL, I immediately ran into some roadblocks. Besides having to setup various aspects of my improved development environment (which was interesting, to say the least), I realized portions of this project I hadn't previously considered which were standing in my way.

I spent the first day building a completely project-specific LUA script for building the SDL2 project (builds SDL2.dll) on Visual Studio 2010. This proved to be non-straightforward, because the SDL source folder contains a huge conglomeration of modules, some of which are platform-specific and others which are platform-independent. More importantly, the exact order of dependencies is rather confusing and unknown without thoroughly reading through the source code. In particular, I noticed there were 'dummy' folders for modules like audio, video, etc. I assumed these folders were replacements for the audio, video, etc. modules if they could not bind to a library like OpenGL or DirectX. I proved to be right, although the idea of not depending on DirectX was not a previous concern for SDL VS solution authors.

I decided to rethink my approach to doing the build system. I spent a bit of time brainstorming various aspects of the build system and this is what I came up with:

1. Consolidation: a single "code" base which reflects desired configurations for all target platforms
2. Automated project generation (metabuild): nice for when dependencies change, the source tree changes, or the user decides on altering configurations
3. Configuration reflection: rather than templates, having a way to store manually-specified settings per-project if projects are to be regenerated
4. Dependency resolution: ability to resolve dependencies that are absolutely necessary, along with those which are optional (see below)
5. Dependency tree with source modules: cross-platform means of marking the dependencies a specific source folder or source file requires, thus allowing for more complex permutations of projects to be handled and significantly more flexibility, at minimal overhead for the developers
6. Default projects: SDL should come with default projects which *may* work out of the box; if they fail, the user may generate a new project using the build system and possibly receive more verbose information as to why it may not build, plus possible options to ignore non-required dependencies (such as the DirectSound example above)
7. Portability: CMake's greatest drawback is its need to be installed to use; Premake or other portable systems are the choice, if they are capable of handling the other features here

I will not go into detail about each specific point. If they end up becoming part of this project, I will emphasize them, or others, in future posts. As far as #7 goes, that was me implying the possibility of not using premake as a solution for this GSoC project. I'm still standing by using premake, although my approach may vary tremendously from my original vision.

Project Dependencies

What do you do when you have a project which has a varied number of dependencies, changing from platform to platform? More importantly, what do you do when these dependencies are optional and how do you communicate that to the user?

These questions, among others, plagued my thoughts for a while this week. I decided on coming up with a simple approach for handling project dependencies, with the thought of expanding it. But first, let's investigate some code of a typical Premake project LUA file (this is the first iteration of the SDL2 project, as available on bitbucket):

 solution "SDL"
  configurations { "Debug" }

 project "SDL2"
  targetname "SDL2"
  kind "SharedLib"
  language "C"
  flags { "NoRTTI", "NoExceptions" }
  includedirs { "../../include", "$(DXSDK_DIR)\Include" }
  libdirs { "$(DXSDK_DIR)\Lib\x86" }
  
  files
  {
   "../../src/*.c",
   "../../src/*.h",
   "../../src/atomic/*.c",
   "../../src/atomic/*.h",
   "../../src/audio/*.c",
   "../../src/audio/*.h",
   "../../src/audio/directsound/*.c",
   "../../src/audio/directsound/*.h",
   "../../src/audio/disk/*.c",
   "../../src/audio/disk/*.h",
   "../../src/audio/dummy/*.c",
   "../../src/audio/dummy/*.h",
   "../../src/audio/winmm/*.c",
   "../../src/audio/winmm/*.h",
   "../../src/audio/xaudio2/*.c",
   "../../src/audio/xaudio2/*.h",
   "../../src/core/windows/*.c",
   "../../src/core/windows/*.h",
   "../../src/cpuinfo/*.c",
   "../../src/cpuinfo/*.h",
   "../../src/events/*.c",
   "../../src/events/*.h",
...
   "../../src/video/windows/*.c",
   "../../src/video/windows/*.h"
  }
  
  excludes
  {
   "*/*psp*/*"
  }
  
  configuration "Debug"
   defines { "_DEBUG", "_WINDOWS" }
   buildoptions { "/GS-", "/Gy-", "/MDd" }
   linkoptions { "/INCREMENTAL:NO" }
   links { "winmm", "imm32", "oleaut32", "version" }

As can be seen, this is a rather annoying and 'verbose' means of maintaining a project. More importantly, the dependencies are hardcoded for the include paths and the library links. So what do we do? How do we handle an abundance of various test SDL projects (totaling 16 projects in the current SDL solution), on top of both their internal and external dependencies?

Honestly, I didn't want to have to write separate project LUA files for each project of the SDL solution. More importantly, I didn't want anyone else to have to maintain and add on to such a system. How to solve this problem? To be horribly cliche...

"All problems in computer science can be solved by another level of indirection" 

-David Wheeler

I decided to abstract the similar functionality that each build system depends on. It's sort of the idea of not recreating similar code when it can be created once for many situations. In order to facilitate a single system to handle many build systems, I had to come up with the idea of using a dependency tree for each project. I also decided to use the idea of automatic folder-based inclusion with specific file exclusion.

Here is what the latest LUA file looks like for generating the SDL2 project in VS2008-VS2012:

SDL_project = {
 name = "SDL2",
 kind = "SharedLib",
 language = "C",
 dependencyTree = { },
 uuid = os.uuid(),
 sourcedir = "../../src",
 -- as dependencies...?
 customLinks = { "winmm", "imm32", "oleaut32", "version" }
}

projects["SDL2"] = SDL_project

-- dependency functions must return the following:
--   [includes] [libs] [inputs]
function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

-- TODO: convert this to be functional (like premake), so the syntax isn't as
-- repetitive

-- format is { dependencyLambda }
-- if not in table, it will be excluded from the project
-- if dependency lambda is nil, it will always be included
local dep = SDL_project.dependencyTree;
-- setup dependency tree for SDL 2
dep["/"] = { nil }
dep["/atomic/"] = { nil }
dep["/audio/"] = { nil }
dep["/audio/directsound/"] = { nil }
dep["/audio/disk/"] = { nil }
dep["/audio/dummy/"] = { nil }
dep["/audio/winmm/"] = { nil }
dep["/audio/xaudio2/"] = { directXDep }
dep["/core/windows/"] = { nil }
dep["/cpuinfo/"] = { nil }
dep["/events/"] = { nil }
dep["/file/"] = { nil }
dep["/file/cocoa/"] = { nil }
dep["/haptic/"] = { nil }
dep["/haptic/windows/"] = { nil }
dep["/joystick/"] = { nil }
dep["/joystick/windows/"] = { nil }
dep["/libm/"] = { nil }
dep["/loadso/windows/"] = { nil }
dep["/power/"] = { nil }
dep["/power/windows/"] = { nil }
dep["/render/"] = { nil }
dep["/render/direct3d/"] = { nil }
dep["/render/opengl/"] = { nil }
dep["/render/software/"] = { nil }
dep["/stdlib/"] = { nil }
dep["/thread/"] = { nil }
-- added exclusion filter to thread/generic to avoid double linking warnings
-- and incorrect linking
dep["/thread/generic/"] = { nil, files = { "SDL_syscond.c", "SDL_sysmutex_c.h" } }

This is actually the entire file, minus comments. This, combined with the automatic system defined in premake4.lua, generates a working project file for VS2008, VS2010, and VS2012 for correctly building and linking SDL2.dll on Win32 combined with a DirectX dependency.

My hope was to create a dependency system wherein I could easily manage dependencies per-project (including inter-project dependencies, like how the test suite depends on SDL and SDLtest). I'm able to define a function which uses my own utility functions to search for dependencies. The next evolution of this system will be to support an array of dependency functions, based on handling the three following features:

1. Some dependencies are required; if they are not met, the project cannot be built or linked
2. Other dependencies are recommended before some dependencies, meaning there must be a certain order to handling dependencies
3. If multiple dependencies are met, the user should be prompted on which to use or, in the very least, an option to specify usage and override default functionality

The above allow for a robust and reliable resolution of dependencies, but in a way that lets users customize the build well beyond what SDL currently supports. Going back to the earlier example, a user should very well be able to build SDL without DirectX (regardless of how limiting it may be, but that's why the software rasterizer exists). Whether this works or not is mere theory and needs testing, but the functionality to achieve this build should be available to the end-programmer, without them having to understand the innerworkings of SDL.

Hacking And Abusing Premake

It turns out that creating a dynamic dependency-handling system in Premake is more possible than I originally expected. A lot of what I did was more series of a "what-if"s that turned out to be plausible. Intentionally, Premake has a nice, relaxed syntactical structure to it which, thanks to LUA, calls various functions which handle internal states about your build configurations, solution settings, and projects. This is all statically written, though. In order to dynamically create solutions and projects, I had to hope Premake worked the way I needed it to, and that invoking its functions using variables and changing its state arbitrarily would keep things working as expected. It turns out it did.

I would not recommend this at all, because it involved a lot of trial and error, among other difficulties. If you wish to see what I did to exploit Premake and dynamically create solutions and projects, check out the latest premake4.lua on bitbucket ) (commit beef0b8). It's a bit too messy and long to paste in here, but it handles generating every single project in the SDL solution quite nicely. I hope to replace it with a generation system (as mentioned below) later.

All assumptions have a possibility of leading to disastrous results. Upon testing my dynamic solution and project code with Premake 5 (development branch), it turned out the author had change the way he was handling the internal Premake script context. This completely broke my system and I have yet to find a solution for fixing it. Although my solution seems to work excellently on Premake 4, it seems to not be forward compatible with Premake 5. There are various options I have moving forward, one of which I explore here and others which I may explore later, if they become practical.

Moving Forward

Upon the first week of GSoC, I managed to assemble a basic build system that correctly replicates the current Visual Studio SDL solutions for VS2008, VS2010, and VS2012. There is much more testing to be done, features that need to be cleaned up, etc. Beyond that, the hackery of Premake mentioned above will becoming increasingly difficult to maintain and likely not forward-compatible. In order to mitigate this upcoming problem, it would be nice to build a sort of meta-meta-build system that generates the LUA files instructing Premake what to do. It would essentially just be a scanner handling dependencies and allowing LUA to do its job naturally, without coercion.

Beyond that, adding support for XCode is absolutely vital and a focus for upcoming weeks. Due to my lack of owning a Macintosh machine, this area of development will be halted for the time being. It will become my first priority upon being able to start it.

Finally, I like to think about long-stretch goals every once in a while, so mentioning them here would be nice, too. Ultimately, I would be happy with this project if I can move it to target other build  environments, such as the Android, MinGW, and Cygwin environments on Windows. Again, it's more consolidation of build settings. It would also be very nice to support some sort of project template system, where SDL developers are able to modify settings in a global template file and the premake system would be able to notice these differences and apply them appropriately to generated projects. My own spin on this suggestion is to allow users to change settings in the generated project, wherein the meta-meta-build system would scan it and be aware of the changes from the default, thus reapplying them and leading to a more manageable system of settings handling (though, with its own cons, of course).

Thus concludes my eventful first week of GSoC.

Until the next time,

Ben