Monday, August 26, 2013

Linux, Cygwin, iOS, and moving forward (CMake/autotools on premake?)


It's been a while since my last post, but I want to take some time to let the community know the latest progress. My work on Google Summer of Code is almost up. I intend on writing one more post after this to wrap things up.

Over the last few weeks, I implemented support for Linux (experimental), Cygwin (rather slim), and iOS. I've implemented a few functions that allow me to do a lot of the same dependency checking as CMake and autotools, as I will discuss below. I have also cleaned up the dependency system some and intend on continuing to make the meta-build system as clean and easy-to-understand as possible.


For starters, I wanted iOS support in the meta-build system almost since the beginning of the project. It seemed like a very interesting system to target, but not without its potential difficulties. Because of desired iOS support, I need to make several modifications to the premake source, all of which were patched into the current stable branch of premake. One of the patches another fellow wrote which provided basic iOS functionality. There was still a lot of other things to modify in order to get iOS to work properly.

Nevertheless, it is working now. I've created a separate Xcode directory specifically for iOS. The same demos used in the current iOS project are being referenced to and tested with the generated system. There are a few minor things related to the logo, startup image, and codesigning that will be more challenging to solve, and may remain unsolved by the end of this project. The goal was getting basic iOS support without much effort on behalf of the developers using the system, and I think I successfully achieved that.


Cygwin was proving especially challenging. Upon implementing MinGW (check my last post), I really wanted to implement Cygwin. MinGW proved incredibly challenging because of a bug in premake that I needed to fix, but Cygwin is even more complicated. For those who are not aware, MinGW is setup to create Windows applications using a POSIX environment (GCC, Make, SH, etc.) Cygwin is designed to write POSIX/Unix applications on Windows. That means that the sources that are compiled for each of these similar environments are very different. The Cygwin target actually compiles the Linux version of SDL, not the Windows version.

The next hurdle was me not being able to successfully build the current SDL project using autotools. Autotools refused to build with Cygwin unless its cross-compiled GCC could produce a Windows executable. Well, of course GCC can produce Windows executables, so this error is a bit non-intuitive. It's essentially saying that there can't be a dependency on Cygwin, which is why the authors of the cross-compiler implementing a -mno-cygwin flag. However, this flag is only available on gcc-3, which is no longer shipped with Cygwin. This approach to writing Cygwin-independent applications is now obsolete. It's recommended to use MinGW-w64 instead.

I decided to provide a stripped implementation that would work on Cygwin. It basically just has threading and file support. Everything major like video and audio has been completely stripped due to time and complexity. This functionality is completely possible to implement, but I just felt like Cygwin was the least important target, so I didn't invest too much time into it. The executables produced by the make files generated from the meta-build system will be dependent on cygwin1.dll.


Currently only tested on Linux Mint 15, I've began to implement Linux support. I had this a while back (I started Linux support to help with Cygwin support), but it's now reaching a much better level. The sheer magnitude of achieving complete Linux support is an entire project in itself (just take a look at the CMake or autotools config files sometime). Therefore, my disclaimer here is the Linux target is completely experimental, incomplete, and likely containing issues that will only be found after it's been tested with many different flavors of Linux.

My goal was not to implement complete Linux support but, rather, to give a good starter for me or other people to work off of in the future. The current meta-build systems allowed me to implement Linux support with ease, but the ability to test for the monolithic list of dependencies SDL is capable of having is well beyond premake. I will discuss this in greater detail in the next section.

I say that SDL is capable of having these dependencies because it's modular. It can have a half a dozen audio devices or just one. SDL can be built on a minimalist configuration file that basically just loads dummy drivers for the entire system. My goal wasn't just to get some shell of SDL working, it was to reproduce the current functionality I am able to build with autotools itself. With that, any features beyond that autotools configures SDL for will not be supported with the results of this GSoC project. That's not to say it wouldn't take just a few minutes to implement them, though. I daresay testing takes much longer than implementing new features in this system.

One final note is about the largest difference between the autotools Linux setup and the one from premake: building on Linux should produce a shared library, just as on Windows. However, due to how GCC handles linking to shared libraries, the best approach to linking is to have the shared libraries in a public location (such as the /usr/lib folder, hence its existence). The problem is premake generates makefiles without custom targets. I am not able to implement an 'install' target for the shared SDL2 library, so I cannot link to it there. My only solutions are to do post-build copies of the library to the correct public location (which seems hackish and non-intuitive for the developer), keep the shared library bound to the same directory as the executable (incredibly inconvenient), or to simply produce a static library. I opted for the last option for times' sake. This is definitely an excellent aspect to improve on in the future.

Dependency Checking: Evolved from CMake and Autotools

My system is setup to produce dependency directives in project files that depend on some named function. In another file, a function is paired with its name and registered in a table. I decided to use functions for dependency handling because dependency checking can depend on a huge variety of factors, many of which aren't continuously considered. Functions can do essentially anything, so they were the ideal choice.

What do these functions do? Well, they are uniquely expected to determine whether their target dependency is on the the current system. Beyond that, they may or may not need to provide directories for header files, directories for static libraries, or a list of libraries to link to. The question yet do we find whether we have the dependency?

CMake and Autotools use the functionality for finding a library using common, public locations of where the library may be residing. Premake has this functionality as well in its os.findlib function. Where CMake and Autotools start to shine over premake is their ability to check dependencies by generating basic source files and seeing whether they compile, link, or run, depending on what's being checked. I decided that this functionality is extremely interesting and very useful, so I've implemented my own basic system for supporting a handful of the features of CMake, specifically.

The hope is to allow much better support for dependency checking than just looking for the library. Source files allow checking of compiler flags, checking for the size of certain datatypes, and whether built-in library functions exist. The possibilities are nearly limitless, so I am excited to see how these features are used in the future. I intend on demonstrating their use partially in some of the Linux dependency functions. I must note, however, that I only slightly emphasized cross-platform support. Right now it specifically focuses on GCC, but Microsoft compiler, Clang, Borland, or whatever else would be possible to add, if desired. The system would just need to be reworked slightly.

Moving Forward

Over the next week I hope to make the meta-build system as easy to understand, use, and extend as possible. I did not implement quite as much as I could have on this project, but I am hoping I laid a good foundation for a meta-build system for SDL. This week's work will comprise of the terribly-exciting work of cleaning things up and making them presentable. Nevertheless, I ask anyone interested to download the meta-build system, try using one or more of the generated targets, and post any feedback you may have.

I would also to make one little note that I did merge my repository with the latest SDL changes (based on the timestamps of the commit) to reflect the ease of implementing the new and changed features. Take a look at the commit logs for more.


Friday, August 2, 2013

More Configurations, MinGW, and Midterm for GSoC


Halfway done with Google Summer of Code. I have a few quick updates to discuss and then I have a request for the SDL community.

Complex Configurations (pt. 2)

I discussed in my last post how I added support for complex configurations by allowing SDL projects to set preprocessor definitions that would be embedded with a configuration header file that drives the entire SDL build process. This is the same way taken by ./configure and related Linux systems. The system has been implemented at this point and has allowed me to do some interesting stuff.

First of all, the modularity of SDL is much more customizable now than it was previously. I can disable DirectX or OpenGL support with ease. If the meta-build system cannot find one or both of these dependencies, it will appropriately configure SDL to neither compile its related modules nor link to the resulting modules. It becomes rather convenient. I theorized in the beginning of my project that SDL should be able to be built and used on Windows without DirectX support. Recently, I have successfully built and ran all the tests without DirectX support. It works surprisingly well. There are a feel hiccups in the test suites, but that's expected given this was mostly untested behavior prior to this meta-build project.

I hope to expand the configurations systems to new depths by providing command-line arguments for easily enabling and disabling whole modules of SDL, without having to have any previous knowledge about its configuration systems or the meta-build system itself. As this project evolves to more a more modular design, this customization will prove to be very convenient and beneficial. SDL can literally be built dozens of different ways on some systems and I would really like to try and capture that with as much ease as possible. The only thing the end user would need to be aware of is which dependencies SDL has, which can be easily documented.

Full Test Suite

I forgot to mention this in my last post, but I did add support for building and linking the entire SDL test suite to every platform supported. Originally, the Visual Studio portion of SDL only had 10 tests supported. The Xcode version had a few more than that. It turns out there are 42 tests part of the SDL test suite. I was a bit shocked when I noticed I wasn't targeting most of them, so I worked to add support for the whole lot. I've written a lot of convenient build scripts (as I will discuss later) for making building and running these tests as painless as possible.

Mac OS X: Multiple Architectures

I am beginning to look into multiple architecture support, starting with Mac OS X. Currently, the Xcode projects should support building to architectures i386 and x86_64. I am planning on implementing Universal binaries for Mac OS X and any platform that has GCC, so including Windows through MinGW and Cygwin, as well as any Linux platforms. This is continuing the current support for "fat" binaries that SDL offers.

MinGW Support

I decided to be a bit adventurous this week. I implemented MinGW support using GNU makefiles. I hope to expand this project to full make support, but I needed to start with something that would not involve a lot of changes to the current project. MinGW is targeted for building native Windows applications using GCC and the GNU toolchain. Essentially, that implies the Windows setup to project generation is not much different between Visual Studio and MinGW. In some parts this is true, but in others...not so much.

The major differences are in the path separators and the different compilers (obviously). I had to create a MinGW mode (using the --mingw command line option) to signify using POSIX paths, etc. Beyond that, I ran into a lot of trouble over properly linking to the main function of the test applications. I had to use the linker argument -lmingw32 to GCC, though premake was giving me a lot of grief by making it only possible to put it in the wrong place. At the end of the day, I had to modify premake and change the order it generates its linker arguments in the resulting makefiles.

The only other problem with the MinGW support is GCC cannot compile the DirectX and Windows SDK header files correctly. There are MinGW alternatives to DirectX, but these are not the easiest things to work with. I've encountered some problems working with them and I know a bunch of other people have, too. As a result, I am not going to support DirectX with the MinGW generation option. I do plan on adding an override command line argument to the meta-build system for generating MinGW projects that still link to DirectX regardless of it being MinGW, leaving it up to the fate of the developer to try and get the thing to link properly.

Google Summer of Code Midterm!

I have to say I'm a bit surprised that it's already the halfway point. Time does fly when you're really focused on a project. Nevertheless, I am hoping the SDL community is able to help me at this point by pulling my repository and tinkering with it. Please, if you have time, try to give the meta-build system a whirl. The currently supported platforms are Windows using Visual Studio and MinGW and Mac OS X using Xcode. Here are a few things to note:

Obviously the system is very experimental. That's why it needs to be tested. If something goes wrong, feel free to email me, comment on this blog, or post it on the mailing list. I will see it and get back to you very soon. I appreciate any problems or feedback you have.

If you want to just give a quick contribution to the project, simply build the SDL project and test suites for whichever choice platform and IDE. If you run the tests, it can give a quick feedback about whether SDL seems to run mostly normally on your system using the build configurations setup by the meta-build system. I have created several build scripts for each platform for handling various types of builds, as well as testing those builds using the extensive SDL test suite.

If you want to give a more elaborate contribution, I would really appreciate you testing binary compatibility or interoperability by either building or linking any SDL projects you have or are using against the libraries produced by the meta-build projects. This would give excellent feedback as to whether I am correctly generating the projects and that they are behaving as they should so far.

I really appreciate any and all help so far.