Workflow for checking out, building and deploying code

1. Check out the workspace and the code base (as per The Subversion code repository#checkout)
2. (Optionally) Edit the projects.mk file to change the projects (as per #CompControl)
3. (Optionally) Edit the compilation.mk file to change targets (as per #CompControlCompdotMK)
4. make all

Builds, Releases and Distributables

For clarity, we define the following terms, which will be used throughout this document:

• constituent — One code file.
• package — A set of constituents to be compiled (and possibly linked). This set is contained in a single directory hierarchy.
• project — A set of code packages with some logical connection between them. This set of packages is contained in a single directory hierarchy.
• workspace — The directory containing a set of projects and the core development system (makefiles, etc).
• build — A compiled and linked set of projects in a workspace.
• release — A self consistent set of libraries, executables, header files and documentation in a self contained directory structure.
• distribution — A release, packaged into a file suitable for installation by user groups. (e.g., a tarball or an RPM)
• module — A relocatable ELF object built for loading on an embedded system. See RCE dynamic Linker for more information.
• runnable module — A module executable in the context of an RTEMS task.

The DAT group periodically announces the distribution of the Cluster Computing Toolkit in a well-known location. (Confluence, probably).

A DAT distribution consists of the following parts:

1. A distribution of the build tools (cross-compilers, debuggers, etc) for a platform, built on Redhat Linux.
   • gcc, g++, gdb, assembler, linker et cetera.
2. A distribution of RTEMS cross-compiled with the above build tools.
   • Base operating system, processor support code, telnet, shell, NFS et cetera.
3. A distribution of the Core, compiled against the RTEMS distribution, using the build tools.
   • Interfaces to OS functions, DAT shell commands, board support code

Typically, the build tools and the RTEMS distribution are tightly coupled: patches to the build tools are dependent on the RTEMS version and RTEMS is built using the tools. The DAT Core distribution is then dependent on both. In addition, the DAT build system uses advanced features from GNU make, and requires a gmake version higher than 3.80. (To check yours, type make --version at your command line. Default for RHEL4 is 3.80, RHEL5 is 3.81.)

Users of the DAT code will install all three distributions and then compile and link against them. Developers of the DAT Core codebase will work against the RTEMS and the build tool distributions.

RTEMS and the Tool Chain

Information on building RTEMS and the build tools can be found here.

Core Development Process

The DAT development environment is documented here. Several environment variables are defined by the setup, but the one used by the build system is

• RTEMS_ROOT — path to an RTEMS distribution
  This environment must be in place prior to building.

The core DAT codebase is held in a subversion repository on SLAC AFS: SVNROOT = file:///afs/slac/g/cci/repositories/ctkrepo. Each project in the repository is laid out in the format

project_A/
  trunk/
  branches/
  tags/
project_B/
  ...

where the trunk is the main line of development (roughly equivalent to the CVS HEAD), tags contains various milestones and branches contains development forks. More information on subversion is found at subversion.tigris.org and in the book Version Control with Subversion. More information on the repository is here.

To start development on the DAT core, the developer first checks out a copy of the workspace directory from $SVNROOT/workspace/trunk or from the specific release tag they need to work with. The workspace contains only the toplevel Makefile, the core build system (make/), core header files (datCode.hh, bldInfo.h, bldInfo.hh) and examples of the makefile fragments packages.mk and compilation.mk. Now, check out the core projects listed on the repository page and type make all at the command line.

Controlling Compilation

The makefile fragments workspace/projects.mk and workspace/compilation.mk control compilation and development in your workspace.

projects.mk simply determines which projects will be compiled.

# List of projects (low level first)
projects := configuration
projects += oldPpi
projects += ppi
projects += service
projects += platform
projects += tool
projects += bootstrap

compilation.mk is more interesting: it determines what platforms is compiled.

# Platforms
platforms := i386-linux
platforms += ppc-rtems-rce405

# Compilation switches
codetypes :=        # clear the name, just in case
codetypes += opt    # turn on optimization compilation (opt only)
codetypes += dbg    # turn on debug compilation (debug only)
#codetypes += plain  # just compile it

verbose  := false   # Turn off verbose messages

With the fragments typing make all will build four targets: i386-linux-dbg, i386-linux-opt, ppc-rtems-rce405-dbg and ppc-rtems-rce405-opt. As of 2012/05, the valid platforms are all listed under workspace/make/sw/flags.mk:

1. ppc-linux — Desktop Linux box with PowerPC.
2. ppc-eabi-rceG1 — 1st gen RCE (pre-COB, PPC405) without RTEMS.
3. ppc-rtems-rceG1 — Same as above except has RTEMS.
4. ppc-eabi-cmG1 — 1st gen CM (pre-COB, PPC405) without RTEMS.
5. ppc-rtems-cmG1 — Same as above except has RTEMS.
6. ppc-eabi-rce405 — 2nd gen RCE (COB, PPC405) without RTEMS.
7. ppc-rtems-rce405 — Same as above except has RTEMS.
8. ppc-eabi-cm405 — 2nd gen CM (COB, PPC405) without RTEMS.
9. ppc-rtems-cm405 — Same as above except has RTEMS.
10. ppc-eabi-rce440 — 2nd gen RCE (COB, PPC440) without RTEMS.
11. ppc-rtems-rce440 — Same as above except has RTEMS.
12. ppc-eabi-cm440 — 2nd gen CM (COB, PPC440) without RTEMS.
13. ppc-rtems-cm440 — Same as above except has RTEMS.
14. ppc-rtems-virtex4 — Virtex 4 without RCE firmware.
15. ppc-rtems-virtex5 — Same as above but for Virtex5.
16. i386-linux — Desktop Linux with 32-bit Intel CPU.
17. x86_64-linux — Desktop Linux with 64-bit Intel CPU.
18. sparc-solaris — Desktop Solaris box with SPARC.

All the variables may be overridden from the command line.

make all verbose=true codetypes=plain

Projects and Packages

The directories directly under the workspace are called projects. The level under your project is called a package. The base setup for both is similar, starting with template files located in the make/ directory in the workspace.

To create a new software project in the workspace:

1. Create the directory under your workspace
2. Copy the following files to your project
   1. workspace/make/share/Makefile.project.template
   2. workspace/make/sw/flags.mk.template
   3. workspace/make/share/packages.mk.template
3. Rename Makefile.project.template to Makefile
4. Rename packages.mk.template to packages.mk
5. Rename flags.mk.template to flags.mk
6. Add the project name to the list of projects in your workspace/projects.mk file and add a use statement in the versions section.

Your project is now ready, and it is time to add packages.

To create a new software package in a project:

1. Create the package directory under your project
2. Copy the following files to your package
   1. workspace/make/sw/Makefile.package.template
   2. workspace/make/constituents.mk.template
3. Rename Makefile.package.template to Makefile
4. Rename constituents.mk.template to constituents.mk
5. Add the new package name to the list of packages in your project/packages.mk file

Your package is now ready, time to add code constituents.

Coding and Constituents

Controlling the compilation of code files is the job of the constituents.mk file, created from the template in the previous section. As an example, here is a hypothetical package that sets up a module to be run under RTEMS, plus a test executable for Linux. (Makefile fragments are suppressed at each level.)

workspace/
   myProject/
      myPackage/
         foo.hh
         bar.hh
         src/
            afile.cc
            gen2/
               gen2code.cc
            bfile.cc
         test/
            testMyModule.cc
            i86-linux/
               testLinux.cc

A valid constituents.mk file for the above package could look like this:

modlibnames := myModule
libsrcs_myModule := src/afile.cc
libsrcs_myModule += src/bfile.cc

ifeq ($(tgt_board),rce440)
libsrcs_myModule += src/gen2/gen2code.cc
endif

ifeq ($(tgt_os),rtems)
modnames := testMyModule
branch_testMyModule := test
majorv_testMyModule := 1
minorv_testMyModule := 7
modsrcs_testMyModule := test/testMyModule.cc
modlibs_testMyModule := myProject/myModule
endif

ifeq ($(tgt_os),linux)
tgtnames := testMyModule
tgtsrcs_testMyModule := test/i86-linux/testLinux.cc
tgtlibs_testMyModule := myProject/myModule
tgtslib_testMyModule := /usr/lib/rt
endif

The constituents file is broken into logical pieces:

1. Common code for all platforms that defines the source files and generated module libraries
2. Board dependent code which is only used for certain boards
3. OS dependent sections that generate the proper outputs for specific operating systems.
   The syntax for the constituents.mk file is described below.

Releases

The linked libraries, executables, documentation and public header files are deliverables for the DAT group, and thus must be able to be released to external users such as ATLAS or LSST. The following steps are required to make a release of DAT:

1. Each core project must be tagged in the SVN repository (preferrably with the same tag)
   • For example: svn cp $SVNROOT/rce/trunk $SVNROOT/rce/tags/TAGNAME
2. The workspace must be tagged in SVN
3. A new copy of the workspace is checked out from the tag
4. And the core projects are also checked out from the tag
5. Build everything with make all
6. Create a base installation with make install
   • This will create an install/ directory under workspace
   • Optionally, the user can specify where the installation is made by defining the MAKE variable DESTDIR on the command line: make install DESTDIR=/foo/bar/baz

The created directory will contain the following files and directories:

1. README — Describes the release (tags, compilers, etc.)
2. CHANGELOG — The list of SVN changes since the previous release
3. bin/ — Binaries for all built platforms (debug and optimized)
4. doc/ — Documentation (generated)
5. include/ — public core headers (and dependencies)
6. lib/ — Static and Dynamic libraries
7. mod/ — Modules
8. workspace/ — An example workspace for development

Once we have a release generated, we can package and place it in our "well known location". (Packaging at this point consists of generating a tarball of the release.)

Conventions

In order to standardize the DAT codebase, the DAT group has adopted several coding and structural conventions:

1. Document, document, document. Comments and README files where appropriate
   • We use Doxygen for automatic documentation generation.
2. All include files placed at package level are public, and all public include files are placed at package level
   • Private include files are all in package subdirectories
3. Library names in a workspace must be distinct. (Two different libfoo.so files is verboten.)
4. For conditional compilation, we use our own defined symbols, not those defined by the compilers
5. All code shall have the standard SLAC copyright/license
   • quarks package currently has boilerplate templates
6. The DAT group as a whole can and will reject code that does not conform to these policies.

Appendices

Syntax in constituents.mk

First off, we define several symbols usable by make for conditional compilation, as well as parallel symbols for C and C++.

In a package's constituents.mk we have four categories of items to be built: modules, module libraries, static libraries, and targets. The variables for specifying the items to be built are:

For modules, there are make variables that control compilation and output. These are keyed on the entries in modnames. The items built will follow the form <modname>.<majorv>.<minorv>.<branch>.so.

For targets, the make variables are keyed on tgtnames:

For module and static libraries, the make variables are keyed on either modlibnames or libnames, and have the same syntax:

Defined symbols for make and C/C++

Build Environment (Description from before the reorg of 2011/04/06)

• Currently, all tools are deployed on AFS
  • We use GNU tools set up for cross-compiling for the PowerPC architecture
  • As of 2011/03/24, these tools are installed under /afs/slac/g/cci/package/gnu/. See the setup.{sh,csh} files mentioned above for more information. This isn't actually true, gnu is a softlink to /reg/g/common...
• The Data Acquisition Toolkit (DAT) uses SVN for configuration management:
  • SVNROOT == file:///afs/slac.stanford.edu/g/cci/svnrepo

Checking out the code from the repository

Checkout top level release directory (svn co $SVNROOT/release/trunk release) and cd into it.
This directory contains the Makefile infrastructure.

Checkout the core of the RCE system:

svn co ${SVNROOT}/rce/trunk rce
svn co ${SVNROOT}/rceusr/trunk rceusr
svn co ${SVNROOT}/rceapp/trunk rceapp

The ideas behind these three projects are as follows:

• rce/ provides common infrastructure that should be re-usable for all software that runs on the RCE. It is like a C++ extension of the RTEMS BSP (e.g. network, thread wrappers, ethernet/PGP PIC drivers).
• rceusr contains higher level code that can be reused (e.g. libraries to support a console application, or a particular use of PGP).
• rceapp contains the highest level executables.

Building

There are several files of importance to building executables:

1. toplevel/projects.mk contains the list of projects that will be built
2. toplevel/compilation.mk contains the compilation settings
3. toplevel/project/packages.mk contains the list of the project packages that will be built.
4. toplevel/project/package/constituents.mk contains the detailed build instructions for each package.
5. Each level also includes its own Makefile. Under most circumstances, these are copies of the templates under toplevel/make/sw
   • These files may need to be modified in order to allow non-standard compilation. For example, if your package contains a python interface that needs to be compiled with swig, you would add the line include $(RELEASE_DIR)/make/sw/python.mk to toplevel/project/package/Makefile
6. The support files for make are under toplevel/make.

To build, we use the GNU make supplied with RedHat. The Makefile system provides several default architectures, the purpose of which should be self-explanatory:

• i386-linux
• ppc-rtems-rce405

Appending -opt to the architecture name will build optimized, and -dbg will build with debugging information.

There is at least one command line tag we can supply to make which provides more information: make <target> verbose=y will display all commands used to build, rather than just progress messages.

The make commands can be issued at the top level to build the specified architecture for all projects in the release, at the project level to build all packages for that project or at the package level to build one package only. Note that:

gmake rceapp.console.ppc-rtems-rce405

and

cd rceapp/console
gmake ppc-rtems-rce405

are equivalent.

Note that we use shared libraries on Linux, but with the much-improved-but-not-perfect $ORIGIN approach for locating shared libraries, so there is no more LD_LIBRARY_PATH or RPATH to worry about.