Lumiera
The new emerging NLE for GNU/Linux

Lumiera uses a build system based on SCons

SCons is an open source software construction tool based on Python build definition scripts. Within these build scripts, we define a data structure to describe the parts and dependencies of our software. When executed, SCons evaluates those definitions and the actual files in the source tree to derive a build strategy, which is then performed.

SCons core concepts

this section is based on the introductory pages on the SCons Wiki

SCons Environment

When SCons starts building the project, it creates its own environment with dependency trees, helper functions, builders and other stuff. The SCons environment is created in memory and some parts of it are saved to disk to speed up things on the next start. The definition of the build happens within this artificial build environment. This often confuses people who used Makefiles, where environment is actually the System Environment.

System Environment

the familiar operating system container with environment variables such as PATH, HOME etc. It is usually accessible via os.environ mapping in Python and therefore in SCons too. SCons doesn’t import any settings from System Environment automatically (like flags for compilers, or paths for tools), because it’s designed to be a cross platform tool with predictable behaviour. That’s why, if you rely on any system PATHs or environment variables — you need to extract those settings explicitly in your build definition.

SConstruct

when SCons executes, it performs a build definition python script written by the user. By convention, this main script is called SConstruct and is located in the root of the source tree. It is a full featured Python module executed within a specifically prepared environment.

SConscript

these files are also SCons scripts, but they are placed in subdirectories of the project. Typically they are used to organize hierarchical builds and are included from the main SConstruct file from the project root. Often, all of the actual build definitions reside in SConscript files in the sub-trees of the project.

Builder

The SCons buildsystem revolves around the metaphor of a Builder. This is a SCons object that you explicitly invoke from the scripts to define that there is something to build, transforming a source into a target. So the target depends on the sources, and typically those source nodes were created by previous builder invocations. The power of SCons is in the fact that dependencies are tracked automatically. When source files change, the system is able to derive which targets to rebuild.

Scanner

when defining a builder, SCons relies on modular scanner components to “understand” the source of the build step. They may scan source files to discover additional dependencies referenced inside. Thus, SCons comes with built-in knowledge about the source files and artefacts to be created by a typical build, and further types can be added through plug-ins.

Tool

any further, external component that adds Builders, Scanners and other helpers to SCons environments for use within scripts. There are special tools for configuring the platform to detect libraries and further requirements. Relying on those tools. the build environment will be outfitted to reflect the needs of the specific build. Sub-environments with special configuration may be created.

Target

any node or “build step” encountered through the definition of the build is a target. The actual build will be triggered by requesting a target, which typically might be just an executable known to reside at some location in the tree. Special alias targets may be defined, based on other targets, to trigger off standard build situations. Especially, a default target may be defined.


Organisation of the Lumiera SCons build

Within our build system, we build upon the power of the Python programming language to create abstractions tailored to the needs of our project. Located in the admin/scons subdirectory, you’ll find a collection of Python modules to provide these building blocks.

  • the LumieraEnvironment is created as a subclass of the standard SCons build environment; it is outfitted with pre-configured custom builders for executables, libraries, extension module, Lumiera plug-in and icon resources.

  • all these custom builders implement a set of conventions and directory locations within the tree. These are defined (and can be adjusted) in the Setup.py module. This way, each builder automatically places the generated artefacts at standard build and installation locations.

  • for defining individual targets and builder invocations, we rely on build helpers to process whole source sub trees rather than individual files. Mostly, just placing a source file into the appropriate sub tree is sufficient to get it compiled, linked and installed in a standard way.

Sub-trees

the source tree

All sourcecode of the core application resides below src/. Building these components is controlled by the SConscript residing in this application source root. By convention, this is also the root for header includes — all headers should be included relative to src/.

the three layers

Within this application core tree, there are sub-trees for the main layers comprising the application. Each of these sub-trees will be built into a shared library and then linked against the application framework and common services residing in src/common. These common services in turn are also built into a shared library liblumieracommon.so, as is the collection of helper classes and support facilities, known as our support library liblumierasupport.so. Besides, there is a sub-tree for core plug-ins and helper tools.

the GTK Gui

one of the sub-trees, residing in src/stage forms the upper layer or user-interaction layer. Contrary to the lower layers, the Stage Layer (GUI) is optional and the application is fully operational without GUI. Thus, the GTK Gui is built and loaded as Lumiera a plug-in.

unit tests

Since our development is test-driven, about half of the overall code can be found in unit- and integration tests, residing below test/. There is a separate SConscript file, to define the various kinds of test artefacts to be created.

  • plain-C tests are defined in test-collections, grouped thematically into several subdirectories. Here, each translation unit provides a separate main() function and is linked into a stand-alone executable (yet still linked against the appropriate shared libraries of the main application layers)

  • the tests covering C++ components are organised into test-suites, residing in separate sub-trees. Currently (as of 5/2015), we link each sub-tree into a shared test library. Here individual translation units define individual test case classes. At the end, all these unit tests are linked together with a testrunner main() into the test-suite executable.

research

There is a separate subtree for research and experiments. The rationale being to avoid re-building most of the core application when it comes to experimenting and trying out new technologies.

icons and resources

the data/ subtree holds resources, configuration files and icons for the GUI. Most of our icons are defined as SVG graphics. The build process creates a helper executable (rsvg_convert) to render these vector graphics with the help of lib Cairo into icon collections of various sizes.

documentation

Largely, the documentation is written in Asciidoc and provided online at the documentation section of our website. The plain-text sources of this documentation tree is shipped alongside with the code. Besides, we build Doxygen API documentation there, and we create design and technical specs and drawings in SVG and in UML.

the target directory

This is where the results of the build process are created. Lumiera is organised into a self contained folder structure. As long as the relative locations, as found within target/, are kept intact, the Application will be able to start up and find all its resources. Consequently, there is no need to “install” Lumiera (and the “install” target just copies this folder structure into the standard installation locations of a typical Unix system)

Unfortunately SCons is a bit weird regarding the object files created during the build process. So, for the time being, we’re just building in-tree. Apologies for that.

Invoking the Build

All of the build process is launched through the scons python script, usually installed into /usr/bin when installing the SCons package onto the system. Just invoking

scons -h

prints a summary of all custom options, targets and toggles defined for our build.

Targets

  • build is the default target: it creates the shared libs, the application, core plug-ins and the GUI.

  • testcode additionally builds the research and unit test code

  • check builds test code and runs our test-suites

  • research builds just the research tree

  • doc builds documentation (currently just Doxygen)

  • all builds the Application, test-suites and documentation

  • install builds and installs the Lumiera Application

By convention, invoking scons -c <TARGET> will clean up everything the given target would build. Thus, invoking scons -c / is the most global clean operation: it will clean up al build artefacts and will un-install Lumiera (recall: every defined node, or directory is also a target).

Configure checks

SCons does not support the concept of a separate “configure” step. The necessary dependency detection is performed before each build. Currently, we expect all dependencies to be installed first-class into the system. We do not (yet) support custom libraries in strange locations. Please use your package manager.

Caching and MD5 sums

SCons stores MD5 sums of all source files, all configure checks and all the command lines used to invoke compilers and external tools. The decision, what needs to be rebuilt is based entirely on these checksums. For one, this means that configure checks are re-run only when necessary. It also means that changes to some compiler switches will automatically cause all affected parts of the application to be re-built. And of course it means, that you only ever compile what is necessary.

With SCons, there is no need for the usual “cleaning paranoia”. Similarly, there is no need for CCache (but using DistCC rocks !). Unfortunately, calculating those MD5 sums requires some time on each build, even if the net result is that nothing will happen at all.

Configuration options

We provide several custom configuration options (run scons -h to get a summary). All of these options are sticky: once set, the build system will recall them in a file .optcache and apply them the same way in subsequent builds. It is fine to edit .optcache with a text editor.