Page History
Include Page | ||||
---|---|---|---|---|
|
Table of Contents |
---|
Include Page | ||||
---|---|---|---|---|
|
Introduction
Below is some more detailed/advanced information about the offline build system. The offline build system is based on SCons. It is implemented as a set of scripts that import functionality from SConsTools
(software developed internally). SConsTools introduces a package and release structure to support the build process. The top level directory is for the release and contains the SConstruct
file. Each package is in a sub-directory to the release. Packages have a SConscript
file. The structure of a release directory is explained in this page about Packages and Releases. SConsTools has a number of features including:
- Targets are automatically identified and built based on source files in the package directories
- Libraries that targets need from other packages are automatically added during the linking of the targets
- A release wide include directory is set up. Package headers are prefixed by package names to avoid file name collisions
- A release wide location for applications and libraries is created
- Release wide directories for
data
andweb
are set up with links to packagedata
andweb
files. - Circular dependencies between packages are identified. Errors are raised to stop the build in these cases.
...
There are a number of targets that will be created based on what is in the package. For example, suppose one has the following files split between two packages in a release:
Code Block |
---|
release/
release/packageA
release/packageA/src/A1.cpp
release/packageA/src/A2.cpp
release/packageA/src/pythonA.py
release/packageA/include/A.h
release/packageA/app/programA1.cpp
release/packageA/app/programA2.cpp
release/packageA/app/appA
release/packageB/src/B1.cpp
release/packageB/src/B2.cpp
release/packageB/include/B.h
release/packageB/src/pythonB1.py
release/packageB/src/pythonB2.py
|
...
Scripts need to start with the appropriate #! line for the language used. For python, you should start you script with
Code Block |
---|
#!<at:var at:name="PYTHON" />@PYTHON@ |
to use the same version of python that the release uses.
The way functions are used from other packages to simple #include the appropriate header When C and C++ code includes a header file, it needs to be qualified by the package name. To use functions in the packageA library, files in packageB (as well as packageA itself) would do
Code Block |
---|
#include "packageA/fileA.h"
|
The way to access python modules is simply
Code Block |
---|
import packageA.pythonA
import packageB.pythonB1
import packageB.pythonB2
|
...
After you run scons
on the above files, you will see three new directories:
Code Block |
---|
release/include
release/build
release/arch
|
You may also see
release/data
release/web
depending on the packages being built. These directories belong to the build system - you should never put anything in these directories. They will be rebuilt each time scons
runs if need be.
In release/include
you will find two links to the package include directories:
Code Block |
---|
release/include/packageA --> release/packageA/include
release/include/packageB --> release/packageB/include
|
...
In the arch
directory, you will find all the executables, scripts, libaries libraries (built as shared object libraries) and python modules for each package.
The build
directory includes all intermediate files. Note - no compiled code is put in the package directories. It all goes into the release/build
directory.
Package data
and web
directories
The same mechanism used to share package header files is also done for package data
and web
directories. That is, if one had the files:
Code Block |
---|
release/packageA/data/data_fileA
release/packageB/web/introB.html
|
after running scons
, the following directories would be created:
Code Block |
---|
release/data
release/web
|
and within those directories, the following softlinks
Code Block |
---|
release/data/packageA -> release/packageA/data
release/web/packageB -> release/packageB/web
|
As with the directory release/include
, do not create or put anything in release/data
or release/web
as they are cleaned out and recreated during an scons
build.
As with release/include
, release/data
and release/web
are created before any targets are built.
Useful Switches and Build Options
The top level SConstruct
file implements several features that may be useful. Running
Code Block |
---|
scons -h
|
from the release directory displays all the available options. A few notable things one might do with the switches:
...
When getting your code working it can be convenient to stop compiling after the first error - less it scroll away after all the subsequent errors. One can do
Code Block |
---|
scons CCFLAGS=-Wfatal-errors
|
...
A useful tool for checking python code is pylint
. scons
will run pylint
and report any errors it finds on your python code if you do
Code Block |
---|
scons pylint
|
from the release directory.
...
Most packages require no additional options beyond those in the default configuration. If a package requires additional build options, these can often be added by calling the standardSConscript()
function in the SConscript file in the package directory. For instance, suppose a psana - Original Documentation user is developing a module in a package called MyPackage
which needs to use functions from the Gnu Scientific Library. They would add the following line:
Code Block |
---|
standardSConscript(LIBS='gsl gslcblas')
|
To the file MyPackage/SConscript
. Here is a complete list of options you can set with standardSConscipt()
:
Code Block | ||||
---|---|---|---|---|
| ||||
LIBS - list of additional libraries needed by this package
LIBPATH - list of directories for additional libraries
BINS - dictionary of executables and their corresponding source files
TESTS - dictionary of test applications and their corresponding source files
SCRIPTS - list of scripts in app/ directory
UTESTS - names of the unit tests to run, if not given then all tests are unit tests
PYEXTMOD - name of the Python extension module, package name used by default
CCFLAGS - additional flags passed to C/C++ compilers
NEED_QT - set to True to enable Qt support
|
...
Suppose you have the files:
Code Block |
---|
mycode/myheader.h
mycode/mysource.cpp
mycode/libmylib.so
|
That is you have a dynamic library built out of some source code. If you want to call functions in mylib from packageA, you could modify packageA/SConscript to have
Code Block |
---|
standardSConscript(CCFLAGS="-I/reg/neh/home/username/mycode", LIBPATH='/reg/neh/home/username/mycode', LIBS='mylib')
|
Then targets in packageA could do
Code Block |
---|
#include "myheader.h"
|
To find the header file (given the use of CCFLAGS above) and the library mylib will be added to the link line.
While targets in packageA that use mylib can now be built correctly, to run them you will need to make sure that the operating system can find the shared object library mylib. This is done by setting the environment variable LD_LIBRARY_PATH to include the directory where mylib is located. This could be done in the shell initialization file (such as .bashrc, or .cshrc, depending on which shell you use).
Using a Library through the External Package Mechanism
What if you want to make an external library available to all the packages in your release? It would be tedious to modify each new package you develop as in the section above. A better solution is to create an external package that interfaces to the library. In your release directory, do the following:
newpkg MyCode
edit the file
~/release/MyCode/SConscript
This SConscript calls the funciton standardSConscript(), but this is not what we want. We want to use standardExternalPackage(). Change the file to look like:
Code Block | ||||
---|---|---|---|---|
| ||||
# Do not delete following line, it must be present in
# SConscript file for any SIT project
Import('*')
import os
from SConsTools.standardExternalPackage import standardExternalPackage
#
# For the standard external packages which contain includes, libraries,
# and applications it is usually sufficient to call standardExternalPackage()
# giving some or all parameters.
#
PREFIX = os.path.expanduser('~username/mycode')
INCDIR = "include"
LIBDIR = "lib"
PKGLIBS = "mylib"
standardExternalPackage('MyCode', **locals()) |
This is the only file that need by in the MyCode package. After doing scons, any package in the release will be able to call functions in mycode by doing
#include "MyCode/myheader.h"
The build system will identify the dependency on the MyCode package. Given the PKGLIBS parameter, it will then add mylib to the link line. libmylib.so will be found because PREFIX and LIBDIR together give the directory where the MyCode shared object libraries are. This directory is not added to the LD_LOAD_LIBRARY path or explicitly used when building. By default, standardExternalPackage() will make soft links from the $SIT_ARCH/lib directory of your release to all the shared object libraries in the external package lib directory. When you ran sit_setup from your release directory, it placed the $SIT_ARCH/lib directory of your release first in the $LD_LIBRARY path. This is the mechanism by which MyCode libraries are made available to all packages in your release.
The above SConscript file for an external package is the simplest case. MyCode has a common directory structure: a base directory with an include and lib directory as sub-directories. Thus the PREFIX, INCDIR and LIBDIR parameters work well to describe it to SConsTools. If the include and lib directory were in different places, one could omit the PREFIX parameter and give absolute paths for INCDIR and LIBDIR:
INCDIR = os.path.expanduser('~username/mycode/include')
LIBDIR = os.path.expanduser('~username/mycode/lib')
Many external packages are more than just a library. They have stand alone programs or tools to run. You can add these using the BINDIR parameter. If an external package like MyCode were to in turn depend on another external package, that package could be defined in the same SConScript file and the DEPS parameter could be used when defining MyCode with standardExternalPackage. An example of using these options can be seen in SConscript file for the hdf5 package. One can do
addpkg hdf5
Within your release and take a look at hdf5/SConscript.Need to add this section
Building Part of the Release
...
To upgrade the version of a release, see Typical Common development tasks.
Presentations
Below are slides from a talk given during the 2014 LCSL users meeting that covers the release/package structure of SConsTools, as well as how psana modules: Software_Development_Environment.pdf