Scrubbing Project Syntax & Complex Configurations

Greetings,

After a six day outing to South Dakota, I'm happy to return and get right back to work on SDL's meta-build system!

Cleaning up Project Syntax

Before, the project syntax was rather icky. It involved a lot of table manipulation, assignment, listing, you name it. I just didn't like it. I didn't plan on keeping it from the beginning. In fact, I wanted a syntax closer to premake's.

With premake, you just do a bunch of function calls in a specific order. However, it doesn't feel like you're making function calls. That's because in Lua, you don't need surrounding parentheses when making a function call if you're passing it one parameter and that parameter is either a table or a string. Conveniently, everything in premake takes either a string or a table. So, in all their examples, they demonstrate passing string literals and inline tables. This gives almost a functional programming feel (which is obviously the end goal on Lua's part). It's just so much cleaner and more elegant. I wanted this.

Hence, I created a series of new functions which maintained a minimal inner state where all the project files were able to call these functions and build projects functionally, rather than uses data structures. Take this example of how  looked before the new system: 

SDL_project = {
 name = "SDL2",
 kind = "SharedLib",
 language = "C++",
 dependencyTree = { },
 uuid = os.uuid(),
 sourcedir = "../src",
 -- as dependencies...?
 customLinks = { }
}

-- statically link on mac osx
if os.get() == "macosx" then
 SDL_project.kind = "StaticLib"
elseif os.get() == "windows" then
 table.insert(SDL_project.customLinks, "winmm")
 table.insert(SDL_project.customLinks, "imm32")
 table.insert(SDL_project.customLinks, "oleaut32")
 table.insert(SDL_project.customLinks, "version")
end

projects["SDL2"] = SDL_project

-- dependency functions must return the following:
-- <foundDep> <name> [includes] [libs] [inputs]
function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

-- TODO: convert this to be functional (like premake), so the syntax isn't as
-- repetitive

-- format is { dependencyLambda }
-- if not in table, it will be excluded from the project
-- if dependency lambda is nil, it will always be included
local dep = SDL_project.dependencyTree;
-- setup dependency tree for SDL 2
dep["/"] = { nil }
dep["/atomic/"] = { nil }
dep["/audio/"] = { nil }
...
dep["/video/"] = { nil }
dep["/video/dummy/"] = { nil }
-- platform-specific implementations
if os.get() == "windows" then
 dep["/audio/directsound/"] = { nil }
 dep["/audio/winmm/"] = { nil }
...
 dep["/render/opengl/"] = { nil }
 -- added exclusion filter to thread/generic to avoid double linking warnings
 -- and incorrect linking
 dep["/thread/generic/"] = { nil, files = { "SDL_syscond.c", "SDL_sysmutex_c.h" } }
 dep["/thread/windows/"] = { nil }
 dep["/timer/windows/"] = { nil }
 dep["/video/windows/"] = { nil }
elseif os.get() == "macosx" then
 dep["/audio/coreaudio/"] = { nil }
 dep["/file/cocoa/"] = { nil }
...
 dep["/video/cocoa/"] = { nil }
 dep["/video/x11/"] = { nil }
else
 print("SDL2 via premake is not supported on platform: " .. os.get())
end

That was the old file, shortened. It's obviously disgusting and ugly. At least, it is compared to the newer version:

function directXDep()
 print("Checking DirectX dependencies...")
 local foundInc, incpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Include")
 local foundLib, libpath = find_dependency_dir_windows("DXSDK_DIR", "C:/Program Files;C:/Program Files (x86)", "DirectX", "Lib/x86")
 if not foundInc or not foundLib then return false, "DirectX" end
 return true, "DirectX", { incpath }, { libpath }, { }
end

function winmmDep()
 print("Checking winmm dependencies...")
 local libpath = os.findlib("winmm")
 local foundLib = libpath ~= nil
 if not foundLib then return false, "winmm" end
 return true, "winmm", { }, { libpath }, { "winmm" }
end

SDL_project "SDL2"
 SDL_kind "SharedLib"
 SDL_language "C++"
 SDL_sourcedir "../src"
 -- dependency tree for SDL2
 SDL_dependency ""
  -- unnamed means it's not a dependency
  -- no OS means platform-independent

  -- this is a minimal setup that should
  -- essentially work on every target platform
  SDL_paths
  {
   "/",
   "/atomic/",
   "/audio/",
   "/audio/disk/",
   "/audio/dummy/",
   "/cpuinfo/",
   "/events/",
   "/file/",
   "/haptic/",
   "/joystick/",
   "/power/",
   "/render/",
   "/render/software/",
   "/stdlib/",
   "/thread/",
   "/timer/",
   "/video/",
   "/video/dummy/"
  }
 -- windows dependencies
 SDL_dependency "windows"
  SDL_os "windows"
  SDL_links { "imm32", "oleaut32", "version" }
  SDL_paths
  {
   "/core/windows/",
   "/haptic/windows/",
   "/joystick/windows/",
   "/libm/",
   "/loadso/windows/",
   "/power/windows/",
   "/render/opengl/",
   "/thread/windows/",
   "/timer/windows/",
   "/video/windows/"
  }
  SDL_files
  {
   -- these files have to be specified uniquely to avoid double
   -- and incorrect linking
   "/thread/generic/SDL_syscond.c",
   "/thread/generic/SDL_sysmutex_c.h"
  }
 -- winmm dependency
 SDL_dependency "winmm"
  SDL_os "windows"
  SDL_depfunc(winmmDep)
  SDL_paths { "/audio/winmm/" }
 -- directx dependency
 SDL_dependency "directx"
  SDL_os "windows"
  SDL_depfunc(directXDep)
  SDL_paths
  {
   "/audio/directsound/",
   "/audio/xaudio2/",
   "/render/direct3d/"
  }
 SDL_dependency "macosx"
  SDL_os "macosx"
  SDL_paths
  {
   "/audio/coreaudio/",
   "/file/cocoa/",
   "/haptic/darwin/",
   "/joystick/darwin/",
   "/loadso/dlopen/",
   "/power/macosx/",
   "/render/opengl/",
   "/thread/pthread/",
   "/timer/unix/",
   "/video/cocoa/",
   "/video/x11/"
  }

The beauty of this new code goes without explanation, but I will point some things out. I didn't have to trim it at all in my paste here, because it's hardly longer than the trimmed other version. The code length is shorter and there's less to type. I also increased the length of this version by separating the winmm dependencies from the general windows dependencies. The flexibility here is self-explanatory.

Complex SDL Configurations

I mentioned or hinted a while back at wanting more sophisticated means of building SDL. An example I gave was the ability to build SDL without any sort of native renderer, and having it still work. Ie, the windows solution file for SDL should not depend on OpenGL or DirectX at all. If someone doesn't have DirectX installed, they should still be able to build SDL. Granted, they will be missing a significant number of features, but that doesn't mean it shouldn't build, link, and run correctly.

However, simply omitting a few libraries from SDL is not enough to stop it from expecting those links. If I do not include /render/opengl, it will compile fine, but it won't link fine. Why? Because SDL is expecting the functions in /render/opengl to be there. How do we stop SDL from expecting something we don't want to be there?

Enter configure and autotools.

SDL's "assumption system" is based on a single configuration header file, which differs from operating system to operating system (on windows, it's SDL_config_windows.h). On a Unix environment, one can use the configure script to check for system dependencies and generate an appropriate config header file that allows SDL to build properly. This is very nice, but it was not taken into consideration for Windows or Mac OS X.

I've implemented a new function in the functionally-driven project definition system for setting up custom defines that can be pasted into a generated config file. For example, the following configuration directive is for setting up the defines needed to enable DirectX support in SDL:

  SDL_config
  {
   ["SDL_AUDIO_DRIVER_DSOUND"] = 1,
   ["SDL_AUDIO_DRIVER_XAUDIO2"] = 1,
   ["SDL_JOYSTICK_DINPUT"] = 1,
   ["SDL_HAPTIC_DINPUT"] = 1,
   ["SDL_VIDEO_RENDER_D3D"] = 1
  }

The only thing next to do is to decide whether to generate a config file or try to use compiler-level defines to simply set flags. Due to various defines in the SDL_config_windows.h file, I cannot just set compiler-level defines, unless I change that file to use #ifndef when setting some of the defines (it does it for many of the other defines, which is nice).

After that is decided, I'll be able to support a lot more flexibility when building SDL than was possible before on both Windows and Mac OS X. Also, if the build system is extended to other platforms, it will conveniently similar to the processing of configure and autotools for generating a makefile, if developers are interested in that.

Until next time,

Ben

Bringing support to Mac OS X

Greetings readers,

After getting acceptable support for Visual Studio 2008, 2010, and 2012 on Windows, it's time to turn attention over to Mac OS X and Xcode.

Premake and Xcode

Not being at all familiar with Xcode, I had to spend a lot of time last week simply wrapping my head around its many differences (as opposed to Visual Studio, Eclipse, etc.) I still have a lot to learn with this IDE, but I think I figured out enough to work through this part of the project. The first step in this process was cloning my GSoC repository on Mountain Lion and making sure the hand-rolled Xcode projects would build, link, and run properly. They did, as the SDL buildbot said they should.

Next, I basically just copied over the Visual Studio meta-build premake setup I've been working on and used it to generate Xcode 4 projects. Needless to say, it obviously did not build. After getting rid of the windows-specific dependencies and cross referencing with the files the manual Xcode project included, I was able to reproduce a buildable, linkable, and runnable SDL within a matter of hours over just a few days.

However, like everything else I've had to deal with when using premake, there was one concern I had. After reading some changes to premake over the last couple years, I thought it would generate an Xcode project for each of my defined projects in the generated lua file. Regardless, I was happy to find out it generated a workspace file which works essentially identically to that of a VS solution file.

The main pieces left to work out are various configurations on Xcode (such as having multiple build schemas), properly handling architecture differences (the generated builds for 64-bit but SDL was previously built for 32-bit), and ensuring the new setup for SDL's Xcode projects is acceptable.

Consolidating Differences

The next challenge was consolidating the differences between the VS and Xcode solution files. As per my mentor's suggestion, making major directory changes in a repository is often ill-advised, so if I must do it, I should do it asap.

What directory changes? Well, I originally created my premake solution for VS in SDL's VisualC folder. To be similar, I also created a similar directory in SDL's Xcode folder recently. This is obviously not a happy solution to keeping things simple using premake, so my most recent work prior to this blog post had been creating a premake folder in the root SDL folder, removing the other premake folders, and consolidating all build configurations into a single, unified script. I also created subdirectories for VisualC and Xcode within that premake folder. That last part was quickly thrown together, so more work will be done to ensure the projects are generated and built relative to those folders, which they currently are not.

Consolidating the two files has also led to increased hardcoding and, thus, some more complexity. A lot of future work will be to cleanup the central premake file, simplify the dependency tree and project definitions using a lighter syntax (like premake uses for its project and solution definitions), and determining a smart way to do cross-platform dependency checking. Xcode and premake are smart enough to make the hassle of porting from VS to Xcode very minimal (especially with a reference Xcode project), but other complexities have arisen that need to be taken care of, especially if any sort of template system is going to be implemented or if even more platforms are targeted.

Until the next post,

Ben

Binary Compatibility

Hello!

As I continue to anticipate starting development for the XCode project generation, I instead turned toward working on binary compatibility verification for the current Visual Studio solutions. This proved to be an interesting test of determination and perplexity.

What is Binary Compatibility & Why need it?

Binary compatibility for libraries means that if I build an executable against some dynamic library and then that library is updated, my executable should be able to work with the new shared library without requiring recompilation or relinking. It becomes immediately obvious some of the problems that can occur when upgrading the shared library. Nevertheless, why am I talking about needing to test binary compatibility when working on a meta-build system when I'm not changing the library source code at all?

This may or may not be evident. We strive to achieve binary compatibility between applications built in the past and the SDL shared library file built by the generated solutions. Since the generated project leads to different build and link flags (see below for why that is), it's possible that the resulting SDL2.dll file from the generated projects work perfectly fine with the test suites, but perhaps not fine with applications already built using different compiler/linker flags.

Thus, the goal is to figure out what flags are necessary to achieve binary compatibility. Well...why can't I just replicate the existing build and linker options when generating the solution files using premake?

Premake's "Specific" Pitfalls

Premake allows for incredible flexibility when defining how a particular project should be built over many systems. However, it has its own pitfalls. As a general consequence of abstracting varied systems, premake mostly supports features that exist across all or most of the platforms being targeted by a given premake file. As a result, it makes certain assumptions about premake flags set, plus it doesn't allow for complete flexibility when creating IDE project files, such as for Visual Studio. As a result, it's very difficult to create warning-free projects that perfectly replicate all the build and link settings as the original SDL project without modifying premake.

There are many workarounds, but for now, the goal is simply working with the current situation instead of making things more complicated. Also, several assumptions premake has made makes it difficult to correctly set /MTd versus /MDd in Visual Studio, which is frustrating for me. This was worked around by switching to premake 5 dev, which seems to handle this situation better.

Results

Right now, the generated Visual Studio project files build a SDL2.dll file which is completely horizontally compatible with the test programs built using the manual Visual Studio project files. The converse, however, does not work correctly (failing with an error saying the C runtime library is being loaded incorrectly). Nevertheless, this is not as desired of a feature, so it working is not as important. As perfectly replicating the build and link options in the premake-generated solution becomes more important, I will go back and ensure this compatibility is correct, as well.

"Meta-"meta-build system

I just want to quickly note that the premake setup is now being used to traverse dependencies and generate a premake file, which is then executed and the resulting projects are generated by premake. This "meta-meta-build system" was discussed briefly in an older post. It allows for forward compatibility with premake 5 dev (which is useful, as described above) and maintains backward compatibility with premake 4. It should also ease troubleshooting any problems, since the exact text being fed to preamke can be dumped to a lua file and viewed manually.

Until the next time,

Ben