Page History

• Overview
• Instructions for Running
• Reports
• Tips and Tricks for Running
• ReportsAdvanced Comparison Tips

Overview

...

Running all parts of the system tests

Quick glossary:

• package name: The software package to be tested, e.g., GlastRelease.
• package directory: Each package has a setup directory containing test files, e.g., $SYSTESTS/src/GRtests, and a separate result directory, e.g., $SYSTESTS/GlastRelease.
• version name: A name for the version of the the package to be tested, e.g., v17r25p20 or v17r35p20_geom. Note that this name usually matches the release version, but can include additional information that indicates details of how the system test was run.
• version result directory: Each version of a package that is tested has a separate result directory in the package result directory, e.g., $SYSTESTS/GlastRelease/v17r35p20.
• test name: Several tests are run for each version of a package, e.g., AllGamma, VerticalProton1GeV. These typically specify different MC inputs to Gleam.
• test directory: Each test has a setup directory in the package setup directory. New tests can be added by simply creating a directory and placing the appropriate files and inputs in the package directory. The tests have result directories in the version result directory, e.g., $SYSTESTS/GlastRelease/v17r35p20/AllGamma or $SYSTESTS/GlastRelease/v17r35p20/VerticalProton1GeV
• config file: Common adjustments to how the system tests are run can usually be made in a configuration file. Typically, this is placed in the version result directory and called Config.txt. Other locations and names may be used if specified when running the test script.

Instructions for running the To run the system tests from scratch, i.e. running , run Gleam to create the full root trees, RootAnalysis to create the histograms and finally a step to , set the standard version and threshold for comparison, threshold and calculate the metadata, and run the comparison:

Go to the directory which will contain the output. For example, to run the test for v11r3 of GlastRelease:

...

For this example, we setup a test for GR v17r35p20. The package is GlastRelease, the version name we choose is v17r35p20, and we are running all the tests, AllGamma, BackGndMixDC2, VerticalGamma100MeV, VerticalProton1GeV, etc. We want the default comparison to be to version name, v15r47p12gr11.

...

In this directory, you will find (or not) the file Config.txt, which was put there (or not) by the Release Manager. If it isn't there, copy one from a previously run suite of systests. Modify this file as desired. For example, you might want to change the reference release, or add the subset command. The reference chosen must have previously run successfully.

To start the tests, do:

...

...

...

...

You will see lots of console output, and then a batch job will be submitted for each test.

The output for this run will appear in the directories:

  > $SYSTESTS/GlastRelease/v11r3/testname/linux

where testname is AllGamma, VerticalMuon1GeV, etc. 

but the jobs are submitted to the batch queue and do not require you to stay logged in to the terminal. You can come back and find top level errors in the version result directory in testerr. The output files and logs will be in the test result directories, e.g., $SYSTESTS/GlastRelease/v17r35p20/<test_name>/linux/

You can check on the runs using bjobs and looking at the output files. If there is a problem with the tests or you want to end the run early, you can kill the jobs with bkill 0. It is fine to do this and start them again. A run will completely overwrite a previous run with the same version name unless you set special options in the config file.

To examine the output of the job see http://glast-ground.slac.stanford.edu/SystemTests/summary.jsp. You will see completed jobs appear there, but no histograms or comparison results until you do the next step.

Once all the runs have finished, you will need to load the histograms and results to the database for web viewing.

Note that the above step can be run multiple times without penalty before all the tests have completed to check on shorter test results as they become available.

More Details

In slightly more detail, runSysTests.sh sets up the run environment and calls runSysTests.pl. This In slightly more detail, runSysTests.pl launches a batch job, which runs a series of shell scripts, say , e.g., $SYSTESTS/src/GRtests/AllGamma/systest_AllGamma, as an example. This job runs Gleam using the jobOptions.txt file in the same test setup directory, and then runs the RUN_linux root macro in that the test setup directory to generate the plots. (This  last step is the only thing that needs to be changed to convert to "Tracy's" system tests.) histograms. Throughout this process, the Oracle database is updated as the tests progress .(except for loading the histograms and comparison results due to changes in db access - hopefully fixed soon).

Only It is important that only one set of system tests for each package is running at any given time. (This can probably be fixed.) can run at the same time. You can run a set of system tests for EngModel and GlastRelease simultaneously, but you cannot run two sets for different GlastRelease versions , simultaneously because the tests will try to store the output files in the same working directory. After you start the system tests it is usually a good idea to look at the file testerr (in the same directory that you ran from). We fairly often get a sporadic error when connecting to the Oracle database. If this happens, you should just kill the jobs with bkill 0 and start them again.To examine the output of the job, go here.(This can be fixed, but needs more work to separate out some directory structure assumptions properly.)

Skipping the Gleam step (MC generation, digi, recon, and merit)

...

This will access the database to find the locations of the root files that were already produced by GLEAM Gleam and will regenerate the system test histograms. This is very convenient for testing and updating the RootAnalysis macros. Anchortipstips

Re-running the comparisons

 Once Once the gleam and root jobs have run, and the output files and system test histograms have been created, the comparison can be re-run quickly. In fact, new comparisons between releases , or other special runs can be done in the same way:

Generate the appropriate Config.txt file in the directory /src/exec, and edit as desired:

...

. Note that you can check the statistical comparison of any existing version anytime without doing this using the statistics panel of the system test web summary. This step simply sets the default comparison, i.e. the one that shows up in the summary with the plot failure counts.

...

System Test Reports