Overview
This is a description of the back end of the system tests. They consist of a perl module, several perl scripts and some root macros. The bulk of the original development was by Karl Young. You can find a copy of the documentation he wrote attached to this page (you'll notice that these pages bear some similarities to his work!)
The purpose of the GLAST SAS system tests is to provide and end to end test of the offline software. They are typically run for each tagged release of the GlastRelease, EngineeringModel and BeamtestRelease packages.
The components are outlined in the following diagram.
The text below describes how to run the system tests manually. In principle, the Release Manager runs these tests automatically after a new release is generated, comparing that release to the previous one. For reasons not yet understood (by Julie and Navid), this job consistently fails to copy the files from the working directory to the archive directory. (In fact, typically the first step in running the system tests in manual mode is to cancel all the jobs which have been automatically submitted for that package.) When this problem is fixed much of the following will be unecessary.
The code for the system tests currently resides in /nfs/farm/g/glast/u17/systests/src. (TBD: commit the current working version of the code to CVS!)
The system test area, $SYSTESTS, is owned by glastsys and the tests are run via this account (most necessary environment variables are set up in the scripts. The password for this account is kept in escrow and should be kept up-to-date there.
NB: if you have logged in as glastsys, which you need to do to get write-access, the envvar $SYSTESTS is set to /nfs/farm/g/glast/u17/systests. Also, you need to login to a SLAC linux machine; practically speaking, this means a noric. This is because the output directory is named with the operating system of the machine that submits the batch jobs, and the batch job looks for the name if its operating system. (This probably isn't necessary, since that directory contains platform-independent files.)
The main executive script is runSysTests.pl in the the exec subdirectory, and is supported by utility routines in SysTests.pm. It runs the suite of shell scripts that it finds in the EMtests, GRtests and BTtests subdirectories of src/, tracks the results and updates Oracle database tables. Each of the shell scripts that it runs consists of at least two parts: it first runs either a Gleam or LatIntegration job, and then runs a RootAnalysis macro, which reads the output files produced in the Gleam step and fills a collection of histograms. Once the histograms have been created, the executive script sets the standard and threshold for each histogram, that is, the version that this release is to be compared against, and the statistical threshold that defines whether the test has passed or failed. Finally the executive script runs a root macro which calculates several quantities from each histogram (number or entries, rms, KS probability that the histogram differs from its standard etc) and updates the database.
The EMtests, GRtests and BTtests directories contain one directory for each test type used for the EngineeringModel, GlastRelease and BeamtestRelease packages respectively. The exec directory contains the perl scripts that run the tests. The output of the tests (root files and run log and summary) ends up in subdirectories of GlastRelease, EngMod or BeamtestRelease.
Running the system tests
The system tests are configured using a configuration file. An annotated example can be found in /nfs/farm/g/glast/u17/systests/src/exec/Config_example.txt.
Running all parts of the system tests
To run the system tests from scratch, i.e. running Gleam to create the full root trees, RootAnalysis to create the histograms and finally a step to set the standard version, threshold and calculate the metadata:
Go to the directory which will contain the output. For example, to run the test for v11r3 of GlastRelease:
> cd $SYSTESTS/GlastRelease/v11r3.
In this directory, you will find (or not) the file Config.txt, which was put there (or not) by the Release Manager. If it isn't there, copy one from a previously run suite of systests. Modify this file as desired. For example, you might want to change the reference release, or add the subset command. The reference chosen must have previously run successfully.
To start the tests, do:
> $SYSTESTS/src/exec/runSysTests.pl [ configfile ]
where configfile is the name of your configuration file. If configfile is not supplied, Config.txt will be used.
You will see lots of console output, and then a batch job will be submitted for each test.
The output for this run will appear in the directories:
> $SYSTESTS/GlastRelease/v11r3/testname/linux
where testname is AllGamma, VerticalMuon1GeV, etc.
In slightly more detail, runSysTests.pl launches a batch job, which runs a series of shell scripts, say GRtests/AllGamma/systest_AllGamma, as an example. This job runs Gleam using the jobOptions.txt file in the same directory, then runs the RUN_linux root macro in that directory to generate the plots. (This last step is the only thing that needs to be changed to convert to "Tracy's" system tests.) Throughout this process, the Oracle database is updated as the tests progress.
It is important that only one set of system tests for each package is running at any given time. (This can probably be fixed.) You can run a set of system tests for EngModel and GlastRelease simultaneously, but you cannot run two sets for different GlastRelease versions, because the tests will try to store the output files in the same working directory. After you start the system tests it is usually a good idea to look at the file testerr (in the same directory that you ran from). We fairly often get a sporadic error when connecting to the Oracle database. If this happens, you should just kill the jobs with bkill 0 and start them again.
To examine the output of the job, go here.
Skipping the Gleam step (does this still work?)
If the full root trees already exist and are stored in the systests db and you wish to regenerate the histograms you can do:
> $SYSTESTS/src/exec/runRootOnly.pl [ configfile ]
This will access the database to find the locations of the files containing the root trees and will regenerate the histograms. This is very convenient if the RootAnalysis macros have been altered.
Re-running the comparisons
Once the gleam and root jobs have run, and the output files and histograms have been created, the comparison can be re-run quickly. In fact new comparisons between releases, or other special runs can be done in the same way:
Generate the appropriate Config.txt file in the directory /src/exec, and edit as desired:
> cd $SYSTESTS/src/exec
> cp $SYSTESTS/GlastRelease/v12r4/Config.txt Config_GRv12r4.txt
> emacs Config_GRv12r4.txt &
> insert_metadata.pl Config_GRv12r4.txt