Overview
This is a description of the backend of the system tests. They consist of a perl module, several perl scripts and some root macros. The bulk of the original development was by Karl Young. You can find a copy of the documentation he wrote attached to this page (you'll notice that these pages bear some similarities to his work!)
The purpose of the GLAST SAS system tests is to provide and end to end test of the offline software. They are typically run for each tagged release of the GlastRelease, EngineeringModel and BeamtestRelease packages.
The components are outlined in the following diagram.
The text below describes how to run the system tests manually. In principle, the Release Manager runs these tests automatically after a new release is generated, comparing that release to the previous one. For reasons not yet understood (by Julie and Navid), this job consistently fails to copy the files from the working directory to the archive directory. (In fact, typically the first step in running the system tests in manual mode is to cancel all the jobs which have been automatically submitted for that package.) When this problem is fixed much of the following will be unecessary.
The code for the system tests currently resides in /nfs/farm/g/glast/u17/systests/src. (TBD: commit the current working version of the code to CVS!)
The main executive script is runSysTests.pl in the the exec subdirectory, and is supported by utility routines in SysTests.pm. It runs the suite of shell scripts that it finds in the EMtests, GRtests and BTtests subdirectories of src/, tracks the results and updates Oracle database tables. Each of the shell scripts that it runs consists of at least two parts: it first runs either a Gleam or LatIntegration job, and then runs a RootAnalysis macro, which reads the output files produced in the Gleam step and fills a collection of histograms. Once the histograms have been created, the executive script sets the standard and threshold for each histogram, that is, the version that this release is to be compared against, and the statistical threshold that defines whether the test has passed or failed. Finally the executive script runs a root macro which calculates several quantities from each histogram (number or entries, rms, KS probability that the histogram differs from its standard etc) and updates the database.
The EMtests, GRtests and BTtests directories contain one directory for each test type used for the EngineeringModel, GlastRelease and BeamtestRelease packages respectively. The exec directory contains the perl scripts that run the tests. The output of the tests (root files and run log and summary) ends up in subdirectories of GlastRelease, EngMod or BeamtestRelease.
Running the system tests
The system tests are configured using a configuration file. An anotated example can be found in /nfs/farm/g/glast/u17/systests/src/exec/Config_example.txt.
Running all parts of the system tests
To run the system tests from scratch, i.e. runing Gleam to create the full root trees, RootAnalysis to create the histograms and finally a step to set the standard version, threshold and calculate the metadata:
Go to the directory which will contain the output. For example, to run the test for v11r3 of GlastRelease, go to:
/nfs/farm/g/glast/systests/GlastRelease/v11r3.
In this directory, you will find the file Config.txt, which was put there by the Release Manager. If it isn't there, copy one from a previously run suite of systests. Modify this file as desired. For example, you might want to change the reference release, or add the subset command. The reference chosen must have previously run successfully.
To start the tests, do:
/nfs/farm/g/glast/u17/systests/src/exec/runSysTests.pl [_configfile]_
where configfile is the name of your configuration file. If configfile is not supplied, Config.txt will be used.
You will see lots of console output, and then a batch job will be submitted for each test.
The output for this run will appear in the directories:
/nfs/farm/g/glast/u17/systests/GlastRelease/v11r3/testname/linux
where testname is AllGamma, VerticalMuon1GeV, etc.
In slightly more detail, runSysTests.pl launches a batch job, which runs a series of shell scripts, say GRtests/AllGamma/systest_AllGamma, as an example. This job runs Gleam using the jobOptions.txt file in the same directory, then runs the RUN_linux root macro in that directory to generate the plots. Throughout this process, the Oracle database is updated as the tests progress.
It is important that only one set of system tests for each package is running at any given time. You can run a set of system tests for EngModel and GlastRelease simultaneously, but you cannot run two sets for different GlastRelease versions, because the tests will try to store the output files in the same working directory. After you start the system tests it is usually a good idea to look at the file testerr (in the same directory that you ran from). We fairly often get a sporadic error when connecting to the Oracle database. If this happens, you should just kill the jobs with bkill 0 and start them again.
To examine the output of the job, go here.
Skipping the Gleam step (does this still work?)
If the full root trees already exist and are stored in the systests db and you wish to regenerate the histograms you can do:
/nfs/farm/glast/g/u17/systests/src/exec/runRootOnly.pl [_configfile]_
This will access the database to find the locations of the files containing the root trees and will regenerate the histograms. This is very convenient if the RootAnalysis macros have been altered.
Making the histograms appear (no longer needed)
At the moment, the step to calculate the metadata and insert it into the database is not automatically run (due to a mismatch in oracle versions needed for the ROOT/perl step). The histograms cannot be viewed until the metadata is in the database. To run this manually, you need to login as glastsys,
cd to u17/systests/src/exec directory
copy the Config.txt that you need to the current directory
i.e. cp ../../EngModel/v5r0703p5/Config.txt .
and run
./insert_metadata.pl Config.txt