...
Running the system tests
The system tests are configured using a configuration file. An annotated example can be found in /nfs/farm/g/glast/u17/systests/src/exec/Config_example.txt.
Running all parts of the system tests
...
Glossary
- package name: The software package to be tested, e.g., GlastRelease.
- package directory: Each package has a setup directory containing test files, e.g., $SYSTESTS/src/GRtests, and a separate result directory, e.g., $SYSTESTS/GlastRelease.
- version name: A name for the version of the the package to be tested, e.g., v17r25p20 or v17r35p20_geom. Note that this name usually matches the release version, but can include additional information that indicates details of how the system test was run.
- version result directory: Each version of a package that is tested has a separate result directory in the package result directory, e.g., $SYSTESTS/GlastRelease/v17r35p20.
- test name: Several tests are run for each version of a package, e.g., AllGamma, VerticalProton1GeV. These typically specify different MC inputs to Gleam.
- test directory: Each test has a setup directory in the package setup directory. New tests can be added by simply creating a directory and placing the appropriate files and inputs in the package directory. The tests have result directories in the version result directory, e.g., $SYSTESTS/GlastRelease/v17r35p20/AllGamma or $SYSTESTS/GlastRelease/v17r35p20/VerticalProton1GeV
- config file: Common adjustments to how the system tests are run can usually be made in a configuration file. Typically, this is placed in the version result directory and called Config.txt. Other locations and names may be used if specified when running the test script.
Running all parts of the system tests
Instructions for running the system tests from scratch, i.e., run Gleam to create the full root trees, RootAnalysis to create the histograms, set the standard version and threshold for comparison, calculate the metadata, and run the comparison:
For this example, we setup a test for GR v17r35p20. The package is GlastRelease, the version name we choose is v17r35p20, and we are running all the tests, AllGamma, BackGndMixDC2, VerticalGamma100MeV, VerticalProton1GeV, etc. We want the default comparison to be to version name, v15r47p12gr11.
Panel |
---|
title | System Test Run ExampleSetup |
---|
|
Log into a machine with access to the batch farm. > ssh noric.slac.stanford.edu The glastsys password is stored in escrow. > su glastsys Create the version result directory in the package result directory. If space on the system test disk, u17, is slim, delete the mc.root, digi.root, recon.root, and result.root files for old tests. > cd $SYSTESTS/GlastRelease > mkdir $SYSTESTS/ Grab a recent config file from another version directory and edit for this run. See the example below. > cp ../v17r35p14/Config.txt ./ |
Anchor |
---|
| configexample |
---|
| configexample |
---|
|
Panel |
---|
| file |
packag GlastRelease pversion v17r35p20 sversion v15r47p12gr11 ksthresh 0.5 builddir /nfs/farm/g/glast/u30/builds/rhel4_gcc34opt/GlastRelease/GlastRelease-v17r35p20 cmtpath /nfs/farm/g/glast/u30/builds/rhel4_gcc34opt/GlastRelease/GlastRelease-v17r35p20 cmtconfig rhel4_gcc34opt executable /nfs/farm/g/glast/u30/builds/rhel4_gcc34opt/GlastRelease/GlastRelease-v17r35p20/Gleam/*/rhel4_gcc34opt/Gleam.exe systests /nfs/farm/g/glast/u17/systests/src/GRtests output /nfs/farm/g/glast/u17/systests/ glastext /afs/slac/g/glast/ground/GLAST_EXT/rhel4_gcc34opt |
Now run the tests. These will be submitted to the batch queue.
Panel |
---|
title | Running the System Tests |
---|
|
> $SYSTESTS/src/exec/runSysTests.sh Optionally, you can specify a different configuration file name. > $SYSTESTS/src/exec/runSysTests.sh Config_ |
...
somethingelse.txt
|
panel Note that rerunning the tests in the same directory with a different configuration file will overwrite the existing outputs and log files. |
You will see lots of console output, but the jobs are submitted to the batch queue and do not require you to stay logged in to the terminal. You can come back and find top level errors in the version result directory in testerr. The output files and logs will be in the test result directories, e.g., $SYSTESTS/GlastRelease/v17r35p20/<test_name>/linux/
You can check on the runs using bjobs and looking at the output files. If there is a problem with the tests or you want to end the run early, you can kill the jobs with bkill 0. It is fine to do this and start them again. A run will completely overwrite a previous run with the same version name unless you set special options in the config file.
To examine the output of the job see http://glast-ground.slac.stanford.edu/SystemTests/summary.jsp. You will see completed jobs appear there, but no histograms or comparison results until you do the next step.
Once all the runs have finished, you will need to load the histograms and results to the database for web viewing. Panel |
---|
title | Loading the histograms and comparison results to the database |
---|
|
> cd $SYSTESTS/src/exec > source setup.csh > python insert_metadata_version_sequence.py -b <version_name> |
...
Only one set of system tests for each package can run at the same time. You can run a set of system tests for EngModel and GlastRelease simultaneously, but you cannot run two sets for different GlastRelease versions simultaneously because the tests will try to store the output files in the same working directory. (This can be fixed, but needs more work to separate out some directory structure assumptions properly.)
Running the System Tests for SCons builds
The only things that change for SCons builds (on the surface, anyway) are the configuration file and the script used to call the tests. Essentially, the environment is set in the configuration file and propagated through the test scripts as needed.
Other optional modifications to the configuration file described here can be used as normal.
> $SYSTESTS/src/exec/runSysTests.sh
becomes
> $SYSTESTS/src/exec/runSysTests_SCons.sh
The SCons version of the config file looks like this:
Panel |
---|
title | Config File Example for SCons |
---|
|
packag GlastRelease pversion v17r35p20 sversion v15r47p12gr11 ksthresh 0.5 bldtype redhat4-i686-32bit-gcc34 instdir /nfs/farm/g/glast/u52/ReleaseManagerBuild/redhat4-i686-32bit-gcc34/Optimized/GlastRelease/17-35-24-gr17 systests /nfs/farm/g/glast/u17/systests/src/GRtests output /nfs/farm/g/glast/u17/systests/ glastext /afs/slac/g/glast/ground/GLAST_EXT/rhel4_gcc34opt |
While cmt and scons builds exist in parallel, versions names for tests of scons builds will end in -scons.
Tips and tricks for the savvy tester
Skipping the Gleam step (MC generation, digi, recon, and merit)
If the mc, digi, recon, and merit trees already exist and are populated and you wish to simply regenerate the system test histograms, then adjust the config file to select a root only run.
...
This will access the database to find the locations of the root files that were already produced by Gleam and will regenerate the system test histograms. This is very convenient for testing and updating the RootAnalysis macros.
Re-running the comparisons/setting the default reference version
Once the gleam and root jobs have run, and the output files and system test histograms have been created, the comparison can be re-run quickly. In fact, new comparisons between releases or other special runs can be done in the same way. Note that you can check the statistical comparison of and overlay plots for any existing version anytime without doing thisby using selection boxes on the statistics panel page or plots page of the system test web summaryinterface. This step simply sets the default comparison, i.e. the one that shows up in the summary with the plot failure counts and comes up automatically for a version.
Panel |
---|
title | Histogram Comparison Example |
---|
|
> cd $SYSTESTS/src/exec > source setup.csh First, set the standard (reference) test to be used for the default comparison. > runSetStandard.sh <version_name> <standard_version_name> TheThen, run the comparison and load the results to the database. > python insert_metadata_version_sequence.py -b <version_name> |
Running a subset of the tests
This is incredibly useful for debugging or rerunning tests that encountered an issue.
Run the tests by the normal method, but use a modified configuration file.
Panel |
---|
title | Config File Example for a subset of the tests |
---|
|
packag GlastRelease pversion v17r35p20 sversion v15r47p12gr11 ksthresh 0.5 builddir /nfs/farm/g/glast/u30/builds/rhel4_gcc34opt/GlastRelease/GlastRelease-v17r35p20 cmtpath /nfs/farm/g/glast/u30/builds/rhel4_gcc34opt/GlastRelease/GlastRelease-v17r35p20 cmtconfig rhel4_gcc34opt executable /nfs/farm/g/glast/u30/builds/rhel4_gcc34opt/GlastRelease/GlastRelease-v17r35p20/Gleam/*/rhel4_gcc34opt/Gleam.exe systests /nfs/farm/g/glast/u17/systests/src/GRtests output /nfs/farm/g/glast/u17/systests/ glastext /afs/slac/g/glast/ground/GLAST_EXT/rhel4_gcc34opt subset AllGamma, VerticalGamma100MeV, VerticalGamma100GeV |
Note the additional and optional last line. If the config file includes the subset keyword, then the list of comma-delimited tests following will be run without the others.
If I call this Config_subset.txt then I just run the tests giving that file as the argument, ../../src/exec/runSysTests.sh Config_subset.txt.
If I do this in an existing test result directory, then existing files for the selected tests will be overwritten.
Deleting a version
You probably do not need to do this.
You can always delete an outdated/uninteresting or garbage version of the system tests by simply removing the version result directory. The version will still be listed in the database, but links to the files will be broken.
There is no pressing need to actually do this, but if you do want to eliminate a version of the tests from the database as well, then this might do the trick.
Note that you do not want to do this for any version that you want to keep the histograms around for reference. If the version to be deleted from the db has been used as a reference version for other versions, this will make the java interface unhappy for those versions. Deletion form the database has generally only been used when a run was particularly ill-contrived, on throwaway test/debug runs, or if the package version itself was a disaster for some reason.
Panel |
---|
title | Deleting a version of the tests from the database |
---|
|
> cd $SYSTESTS/src/exec > source setup.csh > runDeleteVersion.sh <version_name> |
Setting JobOptions for L1proc or Pass8
The JobOptions parameters for the current v17 version of L1proc are slightly different than the settings for Pass8. Some of which is due to the different G4 versions in use. To switch back and forth, before running system tests, you'll want to "source setMode*.csh" for the appropriate version you are planning to run. There are also two versions of the ROOT scripts for L1proc and P8, TupleFragment.cxx and ReconTkrFragment.cxx. This script handles switching these as well.
The relevant files are in systests/src/exec:
setModeLevel1.csh, setModePass8.csh, which both call setModeVersion.csh.
Adding new Histograms to the scripts
Sometimes it is necessary to add or modify histograms to the existing ROOT scripts when new analyses are requested. The copy of the source to modify is located in $SYSTESTS/src/root. Note that some of the scripts have both a L1 and P8 version associated with them.
When running system tests, with new histograms for the first time, you could use the initial run as the reference version to itself - otherwise when running the comparison, there will be errors about missing variables if a previous run is used as reference.
System Test Reports