SCons A Demo Implementation using SciTools

Introduction 

Navid has been busy setting up a demonstration environment using SCons and the ScienceTools v8r2.

Location of Installation

SCons is installed here at SLAC:  /afs/slac/g/glast/applications/SCons/0.97/bin/scons
There's a copy of SciTools in Navid's area:  /nfs/farm/g/glast/u06/golpa/ST-v8r2-scons/ScienceTools
If you desire to run SCons, please copy the ScienceTools from Navid's area into a place you have write-access.  Once you have a copy, enter the top level of the ScienceTools directory where the SConstruct file is located and run:

scons -h

This will provide a list of available options as well as how to specify compile options such as debug and optimize.  If you desire to build you would just type:

scons

Interesting Details and things we may need to discuss

There is a top-level SConstruct file which contains configuration/build information to be used "globally".  This includes options for the compilation such as debug and optimization flags and tells SCons the list of packages associated with this project and how to find the SConscript files for each package.  See the Gory Details of the SConstruct file for more information.  There is another file at the top-level called externals.scons which is used to set up the location of the external libraries.  More on this point further down in the External Libraries section.

Location of SCons specific files and package structure

Each package contains its own SConscript file which act as SCons' version of CMT's requirements file.  You can see an example for the facilities package further down on this page.  The SConstruct file communicates the location of the SConscript file for each package.

The virjpk subdirectories are not in place or assumed in this organization.  There is a note out to Riccardo to check if that arrangement is acceptable to MRStudio.  Doing without the package-tag subdirectories allows us to utilize CVS directly for our checkouts, rather than devising a way to recreate CMT checkouts.  Using this organization, users would obtain code and build via the following commands:
cvs co -r v8r2 ScienceTools
cd ScienceTools
scons

(These changes need some minor CVS modification that'll stay compatible with our current CMT) 

External Libraries

By default, SCons provides a mechanism similar to automake that allows users to specify the location of external libraries and their headers using with-LIBRARY, with-LIBRARY-lib and --with-LIBRARY-include 

There is now a with-GLAST-EXT option where a user can specify a GLAST_EXT style directory layout for our libraries. Any of those libraries can still be overridden with the with-LIBRARY[-lib | -include] options.

The scons-specific file:  externals.scons sets up the external libraries and adds the appropriate -L options to the baseEnv and specifies the specific libraries from the externals to include in a build.  The external library version numbers are now part of externals.scons in order to support with-GLAST-EXT option.

 Building Against A Release

SCons  will not accept having two versions of the same library - so we must devise a method to over-ride the one located in the shared release.  A potential example:

A local version of package X is installed and we wish to build against a release stored at SLAC on u30.  We start a scons build pointing at the SConstruct file located on u30 with an override directory pointing at the local copy of the package you want to override.  The SCons compile would proceed for your local package pointing at the includes and libraries in u30.

Gory Details of the SConstruct file

The SConstruct file begins by setting up the global environment for the build.  This is done by updating SCons' environment variable "baseEnv".  Here are the contents of the SConstruct file:

import os,platform
#########################
#   Global Environment  #
#########################
baseEnv = Environment();
if ARGUMENTS.get('debug',0):
        baseEnv.Append(CCFLAGS = "-g")
if ARGUMENTS.get('optimized',0):
        baseEnv.Append(CCFLAGS = "-O2")
if ARGUMENTS.get('CC',0):
        baseEnv.Replace(CC = ARGUMENTS.get('CC'))
if ARGUMENTS.get('CXX',0):
        baseEnv.Replace(CXX = ARGUMENTS.get('CXX'))
if ARGUMENTS.get('CCFLAGS',0):
        baseEnv.Append(CCFLAGS = ARGUMENTS.get('CCFLAGS'))
if ARGUMENTS.get('CXXFLAGS',0):
        baseEnv.Append(CXXFLAGS = ARGUMENTS.get('CXXFLAGS'))
helpString = """
Usage:

        scons [target] [compile options]

Compile Options:
        Optimized:              Set to 1 to compile with optimization. Default 0.
        CC:                     Set to the compiler to use for C source code.
        CXX:                    Set to the compiler to use for C++ source code.
        CCFLAGS:                Set to additional flags passed to the C compiler.
        CXXFLAGS:               Set to additional flags passed to the C++ compiler.
"""
Export('baseEnv')

The next section pertains to externals:

#########################
#  External Libraries   #
#########################
helpString += SConscript('externals.scons')

Help(helpString)

The following sets up the directories which will contain the libraries, binaries, and include files: 

#########################
#  Project Environment  #
#########################
libDir = os.path.join(os.path.abspath('.'),'lib')
binDir = os.path.join(os.path.abspath('.'),'bin')
incDir = os.path.join(os.path.abspath('.'),'include')
testDir = binDir

baseEnv.Append(LIBPATH = [libDir])

Export('libDir','binDir','incDir','testDir')

Create a list of packages for this SCons project and define the registerObject function which will be used by each package in their respective SConscript file to define the list of libraries, applications, public headers, etc that are desired for the global compilation.

packages = [

def registerObjects(package, objects):

        libs = []

Export('registerObjects')

Lastly we tell SCons where to find the SConscript file for each package:

for pkg in packages:
        SConscript(os.path.join(pkg,"SConscript"))

An Example SConscript File

Here is the SConscript file for the facilities package.  Each package will have its own SConscript file, similar to the requirements file from CMT: 

import glob,os

Import('baseEnv')
Import('registerObjects')
env = baseEnv.Copy()

facilitiesLib = env.StaticLibrary('facilities', glob.glob(os.path.join('src','*.cxx')))

This line creates a variable called facilitiesList.  It says to create a static library called facilities and these are the source files

registerObjects('facilities', { 'libraries': [facilitiesLib], 'includes': glob.glob(os.path.join('facilities','*.h'))}

We set up the compiler options, macros, etc in a SCons "environment".  In this case, the environment is imported from "baseEnv" which is set up in the SConstruct and the external.scons files at the top-level of ScienceTools.  It is imperative that the SConscript files COPY the baseEnv, and update the copy, rather than updating baseEnv directly.  The reason is that any changes to baseEnv will propagate to all packages, even retroactively.

The registerObjects function is called to setup the items we wish to submit from facilities to be part of the global build.  This is all just preparation, no building it done yet, we are just storing the information needed by scons to perform the build.  The first argument is what the package is called, in this case, "facilities".  The second argument is a hash of arrays.  The hash name is the type of object you are registering (libraries, include files, binaries, test apps, etc.), and the value of the hash is a list of those objects.

After all the SConscript files have been called, and registerObjects updated for each package, scons is ready to compute the dependencies and start the build in the correct order.

 Dependency Computation

A feature of SCons, called a Tool, needs to be used to have SCons compute library dependencies for us.  To do this, each package that creates a Static/Shared Library needs to create a python file called <package_Name>Lib.py. This file needs to define 2 functions generate(env, **kw) and exists(env). The exists() function only returns 1 while the generate function adds its own library to the list of libraries needed to be linked and libraries it DIRECTLY depends on. A sample for flux package is:

def generate(env, **kw):
    env.AppendUnique(LIBS = ['flux'])
    env.Tool('xmlBaseLib', toolpath = ['#xmlBase'])
    env.Tool('astroLib', toolpath = ['#astro'])
    env.AppendUnique(LIBS = env['clhepLibs'])
    env.AppendUnique(LIBS = env['cfitsioLibs'])

If an application depends on this flux library they'll have to import the flux library (and all its dependencies) by issuing the command env.Tool('fluxLib', toolpath = ['#flux']). 

The ordering of these libraries still needs to be addressed. Currently ST won't compile because the ordering of those libraries is not correct. There's an issue with shared library which requires that shared libraries be built in a separate environment (minor inconvenience). There also needs to be an option for supporting a package building multiple libraries with different dependencies.

 CVS

A copy of our CVS was made and is located at /nfs/farm/g/glast/u33/golpa/cvs. This copy of our CVS repository contains some changes to allow for both CMT checkouts and straight CVS checkouts which work with SCons. To do a SCons checkout:

set  your CVSROOT to /nfs/farm/g/glast/u33/golpa/cvs

check out SCons version of ST by using: cvs co -r ScienceTools-v8r2 ScienceTools-scons

change dir to ScienceTools-scons and build the code using: /afs/slac/g/glast/applications/SCons/0.97/bin/scons with-GLAST-EXT=/afs/slac/g/glast/ground/GLAST_EXT/rh9_gcc32

This is a work in progress still so the compilation will fail because I introduced some errors when importing the SCons files into the CVS copy.

Please try not to make changes to this CVS repository. 

Interesting Python and SCons filesystem manipulation routines

os.path.join joins 'src' and '*.cxx' together in the correct way ie \ for windows and / for linux
glob.glob then takes the src/*.cxx and gets a list of files that match that pattern
I have added a listFiles() function that'll list the files in the "virtual" environment that SCons creates.

Overriding packages

There are two ways two override packages. The first method you are doing a full build in your own environment with multiple versions of the same package. In the second method you are building only a few packages (again with potentially multiple versions) against an already built build version of the application.

1. Doing a full build with multiple versions of the packages.
   With this method there's nothing "special" you need to do. As long as your packages have the format packageName-distinctionString, then SCons will pick up only one version of this package. For example let's say you wish to compile ST-v8r2 with multiple versions of astro. You would check out ST-v8r2 as before which will check out the standard version of astro. Now you wish to compile a different version of astro. You simply check out the version of astro into a directory with slightly modified name that follows the convention above. For example you check it out as astro-1. When SCons sees this package it automatically knows that it's a different version of astro and compiles astro-1 instead of astro. If you have multiple several versions (for example astro, astro-1, astro-2,etc.), then SCons will pick the last one in lexical order (in this case astro-2).
2. Building a partial set of packages against a full build
   In this case you run SCons with the argument override=somePath where somePath is the full path to the override directory. For example let's say the full build is located in /ST-v8r2 and you have overridden astro in ~/ST-v8r2-override then, while located in /ST-v8r2, you execute scons with-GLAST-EXT=/path/to/glast-ext override=~/ST-v8r2-override. This will build the new astro in ~/ST-v8r2-override against other libraries located in /ST-v8r2. Of course ~/ST-v8r2-override can contain multiple packages including multiple versions of the same package. In the latter case the same rules as the previous option apply.
   In the likely event you don't wish to be located in /ST-v8r2 when compiling the override packages you can also do this build by adding an additional argument to scons. This argument is -C /ST-v8r2. This will tell SCons to change dir to /ST-v8r2 before executing and return back to your current dir after execution.

Todo

Clean up the way source files are defined. Having a bunch of glob.glob and os.path.join functions creates a lot of clutter...Done

Deal with CMT style overriding of packages...Done

Define Swig builder so ST can build swig libraries...Done

Setup CVS to allow SCons style checkouts...Done