...
Just like the CMT Release Manager, the SCons Release Manager is split into several components. These components are the batch submission system, the workflow system, and the release manager system. Unlike the CMT Release Manager, the SCons Release Manager is not written in perl. It is written in C++ relying mostly on the Qt library. Additionally, it depends on the Boost Spirit library, and the stx-exparser library. The Qt library is used for most platform independent interactions. The Boost spirit and stx-exparser are used on the linux side only for linux components only (in particular, for a Workflow application only).
Qt
A major component of the SCons infrastructure is the Qt library. For linux and the mac the source code is obtained from the trolltech website and compiled. In both cases it's compiled as a static library. For windows the installation is a bit more complicated. The windows installation doesn't come with the mysql plugins pre-compiled. As a result it needs to be compiled after installing Qt for windows. From the trolltech webpage one needs to download the LGPL/Free version for windows. There are several choices to download. The appropriate one is the Framework Only version for Windows. This version is complied with mingw so the machine it's installed on will need mingw. The installer for Qt will take care of installing it but it should NOT be installed on any of the machines that are used by the ReleaseManager as SCons might get confused with mingw and visual c++ both installed.
...
The Batch Submission System consists of two applications. The lsf Daemon and the lsf callback application. The daemon is setup to run on fermilnx-v03 (a virtual machine) via trscron (trscron is used instead of regular cron because the LSF processes need AFS tokens and normal cron doesn't provide them). The daemon runs for an hour and is then retriggered.
It checks the database for new jobs. Jobs are submitted to lsf LSF in suspend mode. For each job, there are three lsf LSF tasks submitted. The first tasks task is the actual command to be executed. The second task is a call to the lsf callback application with the condition that it be executed as soon as the first task starts execution. The third task is a call to the lsf callback application with the condition that it be executed as soon as the first task finishes execution. All lsf LSF tasks are submitted and initially put into suspend mode. If all three tasks are submitted successfully, the jobs are changed from suspend to pending mode so they actually start execution. If at any time anything goes wrong, the lsf LSF tasks are killed and the run in the table is marked as failed. The job is put back into the queue to be attempted at a later time as a new run.
The lsf callback application is called once when the lsf LSF task starts and once when it finishes. When started, it updates the run table with the information about the start time. Additionally, it checkes the callback table to determine if the higher layers wish to be notified when the job has started. If so, it calls the callback command to notify the higher levels. When called for a task that has finished. The callback system checks the lsf LSF output, parses some of the information, like the return code, and stores that information in the run, job, and output tables. Additionally, it checks for any callback information for the higher layers.
...
| Code Block |
|---|
+---------------------------+
| Tables_in_rd_releasemgr |
+---------------------------+
| build |
| buildPackage |
| deletedBuild |
| entry |
| extLib |
| os |
| outputMessage |
| package |
| settings |
| subPackage |
| subPackageCompileFailures |
| variant |
| versionType |
+---------------------------+
|
...
- build – This contains the platform dependent information for a particular build.
- buildPackage – This contains the platform independent information for a particular build.
- deletedBuild – This table contains has same structure as the build table but contains the entries for builds that have been completely removed by the eraseBuild program.
- entry - this table was migraged from glastdb and hold the log entries from the entry - this table was migraged from glastdb and hold the log entries from the operation of all the RM processes. Logs are kept for 5 days and then removed to prevent the database from filling up.
- extLib – This contains the external libraries used for a particular build.
- os – This contains a list of operating systems currently supported.
- outputMessage – This contains the checkout or compile output for a particular build.
- package – This contains a list of checkout packages currently supported.
- settings – This contains both global and build specific settings for the Release Manager.
- subPackage – This contains a list of subPackages and their compile status for a particular build.
- subPackageCompileFailures – This contains the location of specific compile failures in the compile output for a particular subPackage.
- variant – This contains a list of supported variants.
- versionType – This contains a list of supported versionTypes.
...
- checkoutBuild – This executable checks out code from cvs in preparation to building it.
- cleanBuild – This executable erases parts of a build that are not needed for execution (source code and temporary files).
- compileBuild – This executable builds the code checked out with checkoutBuild.
- createDoxygen – This executable generates doxygen output for a build.
- deleteBuild – This executable is an interactive program to trigger the RM to erase a build. By default, the entry will remain in the database.
ex. deleteBuild --package ScienceTools --versionType Integration --version 3366 --os Windows-i386-32bit-vc90 --variant Debug --complete
versionType may be Integration "Release Candidate", or Release
The --complete option erases the build and also removes the entry from the database. This is used when one wishes to re-trigger a build. - eraseBuild – This executable erases everything belonging to a build and marks the build as hidden in the database. This is called by deleteBuild and most likely never need be called directly by a user. As this is the program called by deleteBuild to actually do the remove, it also supports the --complete option.
- finishBuild – This executable marks the build as having finished.
- releaseManagerDaemon – This executable runs in the background submitting new builds to the workflow.
- testBuild – This executable runs the unit tests for a particular build.
- triggerBuild – This executable can be used to trigger a new build.
- createReleaseBuild – This executable creates the tar.gz/zip files of the release and externals for the installer.
...
To re-enable tagging, just undo the changes made above.
Miscellaneous Details
This section provides links to various sub-pages describing random bits of SCons lore.