Linux, Cygwin, iOS, and moving forward (CMake/autotools on premake?)

Howdy,

It's been a while since my last post, but I want to take some time to let the community know the latest progress. My work on Google Summer of Code is almost up. I intend on writing one more post after this to wrap things up.

Over the last few weeks, I implemented support for Linux (experimental), Cygwin (rather slim), and iOS. I've implemented a few functions that allow me to do a lot of the same dependency checking as CMake and autotools, as I will discuss below. I have also cleaned up the dependency system some and intend on continuing to make the meta-build system as clean and easy-to-understand as possible.

iOS

For starters, I wanted iOS support in the meta-build system almost since the beginning of the project. It seemed like a very interesting system to target, but not without its potential difficulties. Because of desired iOS support, I need to make several modifications to the premake source, all of which were patched into the current stable branch of premake. One of the patches another fellow wrote which provided basic iOS functionality. There was still a lot of other things to modify in order to get iOS to work properly.

Nevertheless, it is working now. I've created a separate Xcode directory specifically for iOS. The same demos used in the current iOS project are being referenced to and tested with the generated system. There are a few minor things related to the logo, startup image, and codesigning that will be more challenging to solve, and may remain unsolved by the end of this project. The goal was getting basic iOS support without much effort on behalf of the developers using the system, and I think I successfully achieved that.

Cygwin

Cygwin was proving especially challenging. Upon implementing MinGW (check my last post), I really wanted to implement Cygwin. MinGW proved incredibly challenging because of a bug in premake that I needed to fix, but Cygwin is even more complicated. For those who are not aware, MinGW is setup to create Windows applications using a POSIX environment (GCC, Make, SH, etc.) Cygwin is designed to write POSIX/Unix applications on Windows. That means that the sources that are compiled for each of these similar environments are very different. The Cygwin target actually compiles the Linux version of SDL, not the Windows version.

The next hurdle was me not being able to successfully build the current SDL project using autotools. Autotools refused to build with Cygwin unless its cross-compiled GCC could produce a Windows executable. Well, of course GCC can produce Windows executables, so this error is a bit non-intuitive. It's essentially saying that there can't be a dependency on Cygwin, which is why the authors of the cross-compiler implementing a -mno-cygwin flag. However, this flag is only available on gcc-3, which is no longer shipped with Cygwin. This approach to writing Cygwin-independent applications is now obsolete. It's recommended to use MinGW-w64 instead.

I decided to provide a stripped implementation that would work on Cygwin. It basically just has threading and file support. Everything major like video and audio has been completely stripped due to time and complexity. This functionality is completely possible to implement, but I just felt like Cygwin was the least important target, so I didn't invest too much time into it. The executables produced by the make files generated from the meta-build system will be dependent on cygwin1.dll.

Linux

Currently only tested on Linux Mint 15, I've began to implement Linux support. I had this a while back (I started Linux support to help with Cygwin support), but it's now reaching a much better level. The sheer magnitude of achieving complete Linux support is an entire project in itself (just take a look at the CMake or autotools config files sometime). Therefore, my disclaimer here is the Linux target is completely experimental, incomplete, and likely containing issues that will only be found after it's been tested with many different flavors of Linux.

My goal was not to implement complete Linux support but, rather, to give a good starter for me or other people to work off of in the future. The current meta-build systems allowed me to implement Linux support with ease, but the ability to test for the monolithic list of dependencies SDL is capable of having is well beyond premake. I will discuss this in greater detail in the next section.

I say that SDL is capable of having these dependencies because it's modular. It can have a half a dozen audio devices or just one. SDL can be built on a minimalist configuration file that basically just loads dummy drivers for the entire system. My goal wasn't just to get some shell of SDL working, it was to reproduce the current functionality I am able to build with autotools itself. With that, any features beyond that autotools configures SDL for will not be supported with the results of this GSoC project. That's not to say it wouldn't take just a few minutes to implement them, though. I daresay testing takes much longer than implementing new features in this system.

One final note is about the largest difference between the autotools Linux setup and the one from premake: building on Linux should produce a shared library, just as on Windows. However, due to how GCC handles linking to shared libraries, the best approach to linking is to have the shared libraries in a public location (such as the /usr/lib folder, hence its existence). The problem is premake generates makefiles without custom targets. I am not able to implement an 'install' target for the shared SDL2 library, so I cannot link to it there. My only solutions are to do post-build copies of the library to the correct public location (which seems hackish and non-intuitive for the developer), keep the shared library bound to the same directory as the executable (incredibly inconvenient), or to simply produce a static library. I opted for the last option for times' sake. This is definitely an excellent aspect to improve on in the future.

Dependency Checking: Evolved from CMake and Autotools

My system is setup to produce dependency directives in project files that depend on some named function. In another file, a function is paired with its name and registered in a table. I decided to use functions for dependency handling because dependency checking can depend on a huge variety of factors, many of which aren't continuously considered. Functions can do essentially anything, so they were the ideal choice.

What do these functions do? Well, they are uniquely expected to determine whether their target dependency is on the the current system. Beyond that, they may or may not need to provide directories for header files, directories for static libraries, or a list of libraries to link to. The question yet remains...how do we find whether we have the dependency?

CMake and Autotools use the functionality for finding a library using common, public locations of where the library may be residing. Premake has this functionality as well in its os.findlib function. Where CMake and Autotools start to shine over premake is their ability to check dependencies by generating basic source files and seeing whether they compile, link, or run, depending on what's being checked. I decided that this functionality is extremely interesting and very useful, so I've implemented my own basic system for supporting a handful of the features of CMake, specifically.

The hope is to allow much better support for dependency checking than just looking for the library. Source files allow checking of compiler flags, checking for the size of certain datatypes, and whether built-in library functions exist. The possibilities are nearly limitless, so I am excited to see how these features are used in the future. I intend on demonstrating their use partially in some of the Linux dependency functions. I must note, however, that I only slightly emphasized cross-platform support. Right now it specifically focuses on GCC, but Microsoft compiler, Clang, Borland, or whatever else would be possible to add, if desired. The system would just need to be reworked slightly.

Moving Forward

Over the next week I hope to make the meta-build system as easy to understand, use, and extend as possible. I did not implement quite as much as I could have on this project, but I am hoping I laid a good foundation for a meta-build system for SDL. This week's work will comprise of the terribly-exciting work of cleaning things up and making them presentable. Nevertheless, I ask anyone interested to download the meta-build system, try using one or more of the generated targets, and post any feedback you may have.

I would also to make one little note that I did merge my repository with the latest SDL changes (based on the timestamps of the commit) to reflect the ease of implementing the new and changed features. Take a look at the commit logs for more.

Thanks.

Scrubbing Project Syntax & Complex Configurations

Greetings,

After a six day outing to South Dakota, I'm happy to return and get right back to work on SDL's meta-build system!

Cleaning up Project Syntax

Before, the project syntax was rather icky. It involved a lot of table manipulation, assignment, listing, you name it. I just didn't like it. I didn't plan on keeping it from the beginning. In fact, I wanted a syntax closer to premake's.

With premake, you just do a bunch of function calls in a specific order. However, it doesn't feel like you're making function calls. That's because in Lua, you don't need surrounding parentheses when making a function call if you're passing it one parameter and that parameter is either a table or a string. Conveniently, everything in premake takes either a string or a table. So, in all their examples, they demonstrate passing string literals and inline tables. This gives almost a functional programming feel (which is obviously the end goal on Lua's part). It's just so much cleaner and more elegant. I wanted this.

Hence, I created a series of new functions which maintained a minimal inner state where all the project files were able to call these functions and build projects functionally, rather than uses data structures. Take this example of how  looked before the new system: 

SDL_project = {
 name = "SDL2",
 kind = "SharedLib",
 language = "C++",
 dependencyTree = { },
 uuid = os.uuid(),
 sourcedir = "../src",
 -- as dependencies...?
 customLinks = { }
}

-- statically link on mac osx
if os.get() == "macosx" then
 SDL_project.kind = "StaticLib"
elseif os.get() == "windows" then
 table.insert(SDL_project.customLinks, "winmm")
 table.insert(SDL_project.customLinks, "imm32")
 table.insert(SDL_project.customLinks, "oleaut32")
 table.insert(SDL_project.customLinks, "version")
end

projects["SDL2"] = SDL_project

-- dependency functions must return the following:
-- <foundDep> <name> [includes] [libs] [inputs]
function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

-- TODO: convert this to be functional (like premake), so the syntax isn't as
-- repetitive

-- format is { dependencyLambda }
-- if not in table, it will be excluded from the project
-- if dependency lambda is nil, it will always be included
local dep = SDL_project.dependencyTree;
-- setup dependency tree for SDL 2
dep["/"] = { nil }
dep["/atomic/"] = { nil }
dep["/audio/"] = { nil }
...
dep["/video/"] = { nil }
dep["/video/dummy/"] = { nil }
-- platform-specific implementations
if os.get() == "windows" then
 dep["/audio/directsound/"] = { nil }
 dep["/audio/winmm/"] = { nil }
...
 dep["/render/opengl/"] = { nil }
 -- added exclusion filter to thread/generic to avoid double linking warnings
 -- and incorrect linking
 dep["/thread/generic/"] = { nil, files = { "SDL_syscond.c", "SDL_sysmutex_c.h" } }
 dep["/thread/windows/"] = { nil }
 dep["/timer/windows/"] = { nil }
 dep["/video/windows/"] = { nil }
elseif os.get() == "macosx" then
 dep["/audio/coreaudio/"] = { nil }
 dep["/file/cocoa/"] = { nil }
...
 dep["/video/cocoa/"] = { nil }
 dep["/video/x11/"] = { nil }
else
 print("SDL2 via premake is not supported on platform: " .. os.get())
end

That was the old file, shortened. It's obviously disgusting and ugly. At least, it is compared to the newer version:

function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

function winmmDep()
 print("Checking winmm dependencies...")
 local libpath = os.findlib("winmm")
 local foundLib = libpath ~= nil
 if not foundLib then return false, "winmm" end
 return true, "winmm", { }, { libpath }, { "winmm" }
end

SDL_project "SDL2"
 SDL_kind "SharedLib"
 SDL_language "C++"
 SDL_sourcedir "../src"
 -- dependency tree for SDL2
 SDL_dependency ""
  -- unnamed means it's not a dependency
  -- no OS means platform-independent

  -- this is a minimal setup that should
  -- essentially work on every target platform
  SDL_paths
  {
   "/",
   "/atomic/",
   "/audio/",
   "/audio/disk/",
   "/audio/dummy/",
   "/cpuinfo/",
   "/events/",
   "/file/",
   "/haptic/",
   "/joystick/",
   "/power/",
   "/render/",
   "/render/software/",
   "/stdlib/",
   "/thread/",
   "/timer/",
   "/video/",
   "/video/dummy/"
  }
 -- windows dependencies
 SDL_dependency "windows"
  SDL_os "windows"
  SDL_links { "imm32", "oleaut32", "version" }
  SDL_paths
  {
   "/core/windows/",
   "/haptic/windows/",
   "/joystick/windows/",
   "/libm/",
   "/loadso/windows/",
   "/power/windows/",
   "/render/opengl/",
   "/thread/windows/",
   "/timer/windows/",
   "/video/windows/"
  }
  SDL_files
  {
   -- these files have to be specified uniquely to avoid double
   -- and incorrect linking
   "/thread/generic/SDL_syscond.c",
   "/thread/generic/SDL_sysmutex_c.h"
  }
 -- winmm dependency
 SDL_dependency "winmm"
  SDL_os "windows"
  SDL_depfunc(winmmDep)
  SDL_paths { "/audio/winmm/" }
 -- directx dependency
 SDL_dependency "directx"
  SDL_os "windows"
  SDL_depfunc(directXDep)
  SDL_paths
  {
   "/audio/directsound/",
   "/audio/xaudio2/",
   "/render/direct3d/"
  }
 SDL_dependency "macosx"
  SDL_os "macosx"
  SDL_paths
  {
   "/audio/coreaudio/",
   "/file/cocoa/",
   "/haptic/darwin/",
   "/joystick/darwin/",
   "/loadso/dlopen/",
   "/power/macosx/",
   "/render/opengl/",
   "/thread/pthread/",
   "/timer/unix/",
   "/video/cocoa/",
   "/video/x11/"
  }

The beauty of this new code goes without explanation, but I will point some things out. I didn't have to trim it at all in my paste here, because it's hardly longer than the trimmed other version. The code length is shorter and there's less to type. I also increased the length of this version by separating the winmm dependencies from the general windows dependencies. The flexibility here is self-explanatory.

Complex SDL Configurations

I mentioned or hinted a while back at wanting more sophisticated means of building SDL. An example I gave was the ability to build SDL without any sort of native renderer, and having it still work. Ie, the windows solution file for SDL should not depend on OpenGL or DirectX at all. If someone doesn't have DirectX installed, they should still be able to build SDL. Granted, they will be missing a significant number of features, but that doesn't mean it shouldn't build, link, and run correctly.

However, simply omitting a few libraries from SDL is not enough to stop it from expecting those links. If I do not include /render/opengl, it will compile fine, but it won't link fine. Why? Because SDL is expecting the functions in /render/opengl to be there. How do we stop SDL from expecting something we don't want to be there?

Enter configure and autotools.

SDL's "assumption system" is based on a single configuration header file, which differs from operating system to operating system (on windows, it's SDL_config_windows.h). On a Unix environment, one can use the configure script to check for system dependencies and generate an appropriate config header file that allows SDL to build properly. This is very nice, but it was not taken into consideration for Windows or Mac OS X.

I've implemented a new function in the functionally-driven project definition system for setting up custom defines that can be pasted into a generated config file. For example, the following configuration directive is for setting up the defines needed to enable DirectX support in SDL:

  SDL_config
  {
   ["SDL_AUDIO_DRIVER_DSOUND"] = 1,
   ["SDL_AUDIO_DRIVER_XAUDIO2"] = 1,
   ["SDL_JOYSTICK_DINPUT"] = 1,
   ["SDL_HAPTIC_DINPUT"] = 1,
   ["SDL_VIDEO_RENDER_D3D"] = 1
  }

The only thing next to do is to decide whether to generate a config file or try to use compiler-level defines to simply set flags. Due to various defines in the SDL_config_windows.h file, I cannot just set compiler-level defines, unless I change that file to use #ifndef when setting some of the defines (it does it for many of the other defines, which is nice).

After that is decided, I'll be able to support a lot more flexibility when building SDL than was possible before on both Windows and Mac OS X. Also, if the build system is extended to other platforms, it will conveniently similar to the processing of configure and autotools for generating a makefile, if developers are interested in that.

Until next time,

Ben

Google Summer of Code: Simple DirectMedia Layer Meta-build System

Greetings,

Besides the rather long title, I'm here to post the introduction of my Google Summer of Code (GSoC) Simple DirectMedia Layer (SDL) meta-build system project. This blog exists to display my efforts on GSoC projects, my first being the SDL meta-build system. I'll briefly go over the project and, in subsequent days/weeks/months, I will continue to post my efforts as I construct a meta-build system.

Overview of Meta-Build Systems

First of all, what is a meta-build system? Imagine creating a new project (regardless of the language, but we can specifically reference C for the sake of SDL) and you're working mainly with one platform...say GNU/Linux on a 64-bit architecture. You're using makefiles with gcc, knowing various aspects of compiling 64-bit C applications using gcc (such as a long being 8 bytes). But eventually, you decide that you want your application to be buildable on multiple platforms, say Win64. So you have a few options: you could create a Visual Studio project to house your project (recreating the efforts of your earlier makefiles), or you could install MinGW or Cygwin and try to hack your earlier makefiles into working on Windows in those stripped environments, to name a few. Nevertheless, a very annoying pattern arises: every platform, architecture, and even compiler that you want to target will require unique parameters and build settings. Many platform-based build systems allow for multiple configurations to handle different compilers and, with a little work, also different architectures.

The problem is targeting many platforms without having to continue redoing the build systems for each one. This can be done in a decent manner if you are using the same Integrated Development Environment across each platform (such as Eclipse, which is platform-independent). Configuring the project files to properly build across many platforms can be a real pain, though. Beyond that, many people who may use your project likely prefer their own setup (ie, some people will prefer using Visual Studio on windows to trying makefiles with Cygwin or MinGW).

What option do we have then?

The answer is a meta-build system. It is thusly named because it builds/generators build files. Generating build files seems like a neat concept, but we need to be critical the build system is adequate for what we want. It has to satisfy various criteria:

1. It must be platform-independent, at least to each platform our application is going to target
2. It must be easily extensible, incase we need to implement per-project build options (such as searching for dependencies)
3. Ideally, it should be portable so it can be packaged with the project's source code on some Version Control System (like SVN or GIT)

I won't go into depth about various meta-build systems, but I am going to highlight premake here. From my experience, premake massively satisfies the above conditions. It's platform-independent (source code is available), it's extensible (uses lua to create build generation scripts), and it's portable (doesn't require any additional dependencies to support lua) and works out of the box. It can also target most major desired platforms.

Premake has its cons, though this can and will be mitigated throughout the project. There are other meta-build systems available, but I feel as though premake is the best option for SDL (especially given its size and magnitude), plus it's proved to be a dependable option. It's the system of choice for SDL.

SDL and Premake

As mentioned, I am deciding to use premake as the meta-build system for SDL. SDL currently maintains various Visual Studio projects for building on windows, an XCode project for MacOSX, and cmake files for other build targets. There is a need to consolidate all of the different build targets into a single system, to increase maintainability (change one to change all, versus having to actually go and change all), portability (cmake requires installation prior to use, making it very inconvenient), and flexibility (again, lua). Integrating a lightweight meta-build system was the answer, so premake was the choice.

The goal of this project is to create a single premake setup which slowly replaces each of the targeted platforms of SDL, possibly with the chance of adding more platforms per-choice of the user. SDL currently targets a plethora of platforms, including Linux 32/64, Windows using MSVC, Windows using Cygwin or MinGW, MacOSX and iOS, and android, to name some. As a result, supporting each individual platform is a goal in itself, and will require overcoming one of premake's cons: searching for dependencies.

In the coming weeks, I hope to post more regarding the dependencies of each of SDL's build targets, thus categorizing and organizing what the replacement premake lua script will need to be able to support. Additional research will need to be taken into how the Visual Studio and XCode projects are setup, plus what sort of dependencies the CMake scripts look for and how that can be replaced by lua with premake.

That's all I have for now. Feel free to comment or post any questions on the SDL Developer Mailing List, or as a comment on this blog.

Cheers.