Brubuild is a brand new build system written from scratch in pure Ruby. It is designed to be a replacement for Make, automake, autoconf, libtool and similar tools on some common platforms. It is currently targeted at projects that are written in C/C++ and has been tested on Linux and Mac/OSX.

1. Installing Ruby and Tokyo Cabinet
2. How to build demo projects
3. Rationale
4. Features
5. How to build your projects
6. Architecture
7. Limitations
8. Odds and Ends

Brubuild needs Ruby 1.9.X or later which itself has a couple of prerequisites: the development libraries for zlib, bzip2, ffi, yaml and libreadline6 (packages zlib1g-dev, libbz2-dev, libffi-dev, libyaml-dev, and libreadline6-dev on debian-based systems). It also needs Tokyo Cabinet (a "NoSQL" database), including its Ruby bindings.

If you are a Ruby veteran and are already familiar with the process of installing it, just make sure the above pre-requisites are satisfied.

If you are new to Ruby, a small shell script prereq.sh is provided to simplify the process of installing the prerequisites on Debian-based Linux systems. Just change to the directory where you want the archives unpacked and built (e.g. cd ~/src) and give it an (optional) argument which is the install location (default: /usr/local). For example:

mkdir ~/src; cd ~/src
<path>/prereq.sh

After this, you should be able verify the Ruby version with:

ruby -v

First, retrieve the Brubuild sources with:

git clone https://github.com/amberarrow/brubuild

The projects subdirectory under brubuild has a few directories for building various open-source projects. The current list is:

• HelloWorld
• snappy
• tokyocabinet
• openssl

Each directory has a README file describing how to build it.

HelloWorld has a small C++ program and a C library and is part of the Brubuild sources. Build it like this (details in projects/HelloWorld/README):

cd brubuild/projects/HelloWorld
ruby -w hello_world.rb -s $PWD/../../HelloWorld -o /var/tmp/brubuild/HelloWorld

Snappy is a fast C++ compression library available at http://code.google.com/p/snappy/ Building it is similar (details in projects/snappy/README):

cd brubuild/projects/snappy
ruby -w snappy.rb -s <snappy-src-dir> -o /var/tmp/brubuild/snappy -b rel

Tokyo Cabinet is a fast "NoSQL" database written in C. It is available at http://fallabs.com/tokyocabinet/tokyocabinet-1.4.48.tar.gz Building it is also similar (details in projects/tokyocabinet/README):

cd brubuild/projects/tokyocabinet
ruby -w tc.rb -s <tc-src-dir> -o /var/tmp/brubuild/tokyocabinet -b rel

OpenSSL is a toolkit for the SSL protocols written in C. It is available at http://www.openssl.org/ Building it is similar (details in projects/openssl/README):

cd brubuild/projects/openssl
ruby -w openssl.rb -s <openssl-src-dir> -o /var/tmp/brubuild/openssl -b rel -l static

The motivation for writing Brubuild is to transcend the limitations of a tool like Make (and its numerous siblings and look-alikes) with its declarative semantics that severely limit its usability, flexibility and programmability. Over the years, Make has been extended by a patchwork of enhancements to provide conditionals, loops and function calls with highly idiosyncratic syntax that makes it awkward to use for even a project of moderate complexity. Additionally, a vast array of equally awkward tools such as automake, autoconf and libtool have sprung up around it resulting in substantial increases in complexity. Various websites discuss the pain associated with these tools, for example:

• http://www.conifersystems.com/whitepapers/gnu-make/
• http://voices.canonical.com/jussi.pakkanen/tag/pain/
• http://titusd.co.uk/2010/08/29/rake-builder/

In contrast, project configuration files in Brubuild are Ruby scripts so the full power of a clean, well-designed, object-oriented, fully dynamic programming language is available along with a wealth of standard libraries. A specific design goal, therefore, was to eschew declarative data files (e.g. XML files, Makefiles, properties files) completely and use code files for everything with the sole exception of the persistence database.

This approach provides a number of benefits, among them:

• Fine-grained control over the specific options used to compile, assemble, or link individual files. In contrast, conventional Makefiles use pattern rules which makes it very difficult to control options at the level of individual files resulting in many unnecessary options being used.

  Another use case is a situation where a build is using a number of warning flags along with the -Werror option (which causes warnings to be treated as errors). When a new version of the compiler arrives, it causes build failures due to more stringent checking of warning-conditions. With Brubuild, in such cases, we can selectively disable -Werror on the specific file(s) which cause build failure rather than for the entire build.

• Detection of erroneous or suboptimal compiler or linker option combinations. For example, a common mistake is to use the -shared option when linking an executable. The resulting file will not be runnable because what was created was a dynamic library! Another example is omitting the required option -fPIC when building dynamic libraries. Yet another is using the -fPIC which compiling files for inclusion in static libraries: though this is not a fatal error, it causes some performance degradation since it introduces an unnecessary extra indirection to every variable and function reference.

• Persistence enables Brubuild to detect changes in options and trigger a rebuild; in contrast, if Make is run once with one set of say, CFLAGS supplied on the command line and immediately re-run with a different set, it will not rebuild any of the targets since it has no persistence mechanism to detect the change in options.

• Better logging and explanations of why a particular target was rebuilt. With Make, it can sometimes be difficult to deduce the logic behind some actions (e.g. why was foo.o not rebuilt ? Why was baz.o rebuilt ?) even with debugging enabled, because of the complex way the decision threads its way through implicit rules, static pattern rules, single and double colon rules and multiple flavors of variable evaluation semantics.

• Decoupling the location of the source directory from the directory where the objects are generated and each from the location of the build tool itself. This allows Brubuild to operate in a completely non-intrusive manner since it make no changes whatsover to the source directories.

Here is a brief feature summary of Brubuild features:

• All build files are Ruby programs so arbitrary customization and tweaking of the build process should, in theory, be easy.
• Fine-grained control over compiler and linker options used on individual files.
• Location of build files is decoupled from the location of the sources and both are decoupled from the directory where generated objects are placed.
• Uses a thread pool for parallel builds.
• Fast -- in our informal time trials, Brubuild is as fast as and often faster than Make with the same parallel build factor.
• Supports 3 build types: debug, optimized, release. The last is the same as the penultimate but it strips symbols from the object files.
• Supports 2 link types: dynamic and static.
• Objects of each build and link type use different file extensions so they can all simultaneously coexist; hence, when you switch from one type to another, you don't have to rebuild the entire project.
• Has detailed knowledge of GCC options and checks the option set for errors or inconsistencies.
• Uses standard Ruby logging.
• Prints histogram of build times to log file
• Prints histogram of dependency counts to log file.
• Shows a commandline progress indicator with the number of remaining targets to build.
• Uses a "NoSQL" database (Tokyo Cabinet) for persistence, so targets will be rebuilt if there is any change to the options used to build them.
• Tested on Linux and Mac/OSX.

There is some code in features.rb for querying the environment for things like the presence of a header file, endianness of the CPU, compiler version etc. This code is very preliminary and is being worked on.

The easiest way is to emulate some of the demo projects. Some knowledge of Ruby is obviously required. It is important to remember that Build.src_root and Build.obj_root are critically important variables holding respectively, the root of the source directory (which is not changed in any way) and the root of the object directory where all generated objects are placed.

We hope to simplify this process soon, but for now we suggest the following steps to port your project, say xyz, to Brubuild:

• Create two Ruby files by copying the corresponding files from HelloWorld (or one of the other demo projects): xyz_config.rb and xyz.rb; the first should contain global configuration (such as compiler and linker flags) across the entire project. If some files need different options, you can make these adjustments later in the specific subprojects that contain these files.

  The second should define one class per subproject (discussed below) derived from the class named Bundle. At least one such class must be present.

• In the first file, xyz_config.rb, change these functions suitably to define the corresponding set of global options (no changes should normally be needed in any of the other functions):

  Function	Description
  init_cpp_options	Pre-processor
  init_cc_options	C compiler
  init_cxx_options	C++ compiler
  init_ld_cc_lib_options	Linker, building library of C files
  init_ld_cc_exec_options	Linker, building executable of C files
  init_ld_cxx_lib_options	Linker, building library of C++ files
  init_ld_cxx_exec_options	Linker, building executable of C++ files

• A subproject is, roughly speaking, a subdirectory of the main project with source files that form libraries and executables. The dir_root instance variable holds the name of this subdirectory (which can just be '.' if all sources are in the main project directory). Brubuild will automatically scan all files and directories under the root for C or C++ files.

• The derivative Bundle class should then define include and exclude lists (to limit the files and directories that are searched), libraries and executables that can be built and the list of default targets to be built. The setup method must be present and is automatically invoked at the appropriate time; additional methods that are invoked within setup may be defined as needed. The initialize method, after invoking super, should define these instance variables:

  Instance variable	Description
  dir_root	Root of the subproject, relative to the root of the main project.
  include	List of directories to search for sources; if omitted, all directories will be searched.
  exclude	List of directories to exclude from search for sources; if omitted, nothing is excluded (note that this list may also contain specific filenames that should be ignored).
  libraries	List of libraries to be built.
  executables	List of executables to be built.
  targets	List of default targets to be built.

  The list of methods in the derivative class is described in the table below:

  Function	Description
  initialize	Perform necessary initialization as described above.
  setup	Perform necessary setup for this subproject; all the remaining methods are invoked by this one.
  create_dirs	Create necessary subdirectories under the object root.
  discover_targets	Recursively traverse the source root (consistent with the include and exclude lists discussed above) finding all source files.
  add_lib_targets	Register library targets.
  add_exe_targets	Register executable targets.
  adjust_options	Tweak options for invidual targets if necessary; should be called towards the end of setup (see below).
  add_default_targets	Register targets that will be built by default.

• As an example, suppose your project has subdirectories A, B, C and D where the first two have source files that are aggregated into libraries libA and libB and you want Brubuild to ignore completely the last two; assume further that you also have some source files in the main project directory that need to be compiled and built into an executable. You would proceed as follows:

  • Define three subclasses of Bundle in xyz.rb:

  
        class LibA < Bundle  ...  end
        class LibB < Bundle  ...  end
        class Xyz  < Bundle  ...  end
    

  • Within each class, define the libraries and executables to be built from files in that directory. Within the last you should also add C and D to the exclude list to prevent scanning of those directories.

• Within the setup method, after all targets for this subproject have been discovered or registered, you can customize options for individual files. Typically, this is done in a method named adjust_options. For example, to add the -Wtype-limits warning option when compiling foo.cc and to remove the -Wshadow warning option (which is presumably part of the global project defaults defined in xyz_conf.rb:

  
    add_target_options( :target  => ['foo', :obj],
                        :options => ['-Wtype-limits'] )
    delete_target_options( :target  => ['foo', :obj],
                           :options => ['-Wshadow'] )
    

  Similarly, to add a different include path on Mac versus Linux (pre-processor options, typically -I, -D and -U, need to be explicitly tagged as such) :

  
    opt = [ @@system.darwin? ? '-I/opt/freetype/include/freetype2'
                             : '-I/usr/include/freetype2' ]
    add_target_options( :target  => ['MyFontManager', :obj],
                        :type    => :cpp,
                        :options => opt )
    

The main class is Build which also serves as the encapsulation namespace. Its definition therefore is distributed across multiple files with the core definitions in build.rb; the command line parsing is also done in this file. Here is a table that summarizes the functionality embodied in each file:

We plan to add much more detail here shortly but in the interim there are comments on all non-trivial parts of the code so it should be very readable if you know some Ruby.

Recall that each project, say xyz has two associated files -- a configuration file xyz_config.rb and the main file that defines all the subprojects, xyz.rb.

The high level algorithm and call sequence is as follows:

System information such as the type OS, CPU, number of cores, RAM size etc. is initialized at load time by the call to Build.init_system when system.rb is loaded. You can examine this information by running this file in isolation:

ruby -w system.rb

Main execution begins with the call to Build.start at the very end of your project file, e.g. hello_world.rb. This function does the following:

• Initialize logger

• Parse commandline arguments

• Create threadpool

• Invoke Build.setup located in your project configuration file which, in turn, does the following:

  • Create necessary subdirectories (e.g. bin, lib, include) under the root of the object directory.
  • Invoke setup method of each bundle (i.e. subproject) to initialize the subproject (e.g. define the libraries and executables that may be built and the default targets that should be built).

• (At this point, project initialization/configuration is complete and the core build process begins). Replace target list to be built by the user-specified list on the command line, if any.

• Open the persistence database.

• Discover header file dependencies (either from the database or by running the pre-processor)

• Enqueue all out-of-date targets in the job queue for the thread pool and wait for all jobs to complete.

• Shut down thread pool

• Save information about the current build to the database

• Log "Build finished" message and exit.

• Currently limited to C/C++ (with some assembler) projects in Linux and Unix-like environments.
• There is no equivalent of make install; we hope to remedy this soon.
• There is also no equivalent of make clean but this is less of an issue since you can get the desired effect by simply removing the entire object directory.

• When you start the build you'll see the message Detecting dependencies ... for a few seconds; thereafter you should see a progress indicator in the form of a countdown spinner showing the number of targets that remain to be built. Note that this spinner may reach zero and bounce back up to a small number and count down to zero again; this is normal and no cause for alarm.

Finally, Brubuild is still in its infancy and has only been lightly tested, so it is likely that it will undergo significant changes in the weeks ahead.

We welcome feedback, so please feel free to send your comments to amberarrow on gmail.

Thanks!