...
Only one set of system tests for each package can run at the same time. You can run a set of system tests for EngModel and GlastRelease simultaneously, but you cannot run two sets for different GlastRelease versions simultaneously because the tests will try to store the output files in the same working directory. (This can be fixed, but needs more work to separate out some directory structure assumptions properly.)
| Anchor | |||
|---|---|---|---|
|
Tips and tricks for the savvy tester
- Rerunning the root analysis on existing Gleam outputs
- Re-running the comparisons/setting the default reference version
- Running a subset of the tests
- Deleting a version of the tests from the database
...
Skipping the Gleam step (MC generation, digi, recon, and merit)
If the mc, digi, recon, and merit trees already exist and are populated and you wish to simply regenerate the system test histograms, then adjust the config file to select a root only run.
...
| title | Root Only Example |
|---|
...
|
Running the System Tests for SCons builds
The only things that change for SCons builds (on the surface, anyway) are the configuration file and the script used to call the tests. Essentially, the environment is set in the configuration file and propagated through the test scripts as needed.
Other optional modifications to the configuration file described here can be used as normal.
> $SYSTESTS/src/exec/runSysTests.sh
becomes
> $SYSTESTS/src/exec/
...
runSysTests_SCons.sh
...
This will access the database to find the locations of the root files that were already produced by Gleam and will regenerate the system test histograms. This is very convenient for testing and updating the RootAnalysis macros.
...
Re-running the comparisons/setting the default reference version
Once the gleam and root jobs have run, and the output files and system test histograms have been created, the comparison can be re-run quickly. In fact, new comparisons between releases or other special runs can be done in the same way. Note that you can check the statistical comparison and overlay plots for any existing version anytime without doing this by using selection boxes on the statistics page or plots page of the system test web interface. This step simply sets the default, i.e. the one that shows up in the summary with the plot failure counts and comes up automatically for a version.
| Panel | ||
|---|---|---|
| ||
> cd $SYSTESTS/src/exec |
...
Running a subset of the tests
This is incredibly useful for debugging or rerunning tests that encountered an issue.
Run the tests by the normal method, but use a modified configuration file.
| Panel | ||
|---|---|---|
| ||
packag GlastRelease |
Note the additional and optional last line. If the config file includes the subset keyword, then the list of comma-delimited tests following will be run without the others.
If I call this Config_subset.txt then I just run the tests giving that file as the argument, ../../src/exec/runSysTests.sh Config_subset.txt.
If I do this in an existing test result directory, then existing files for the selected tests will be overwritten.
...
Deleting a version
You probably do not need to do this.
You can always delete an outdated/uninteresting or garbage version of the system tests by simply removing the version result directory. The version will still be listed in the database, but links to the files will be broken.
There is no pressing need to actually do this, but if you do want to eliminate a version of the tests from the database as well, then this might do the trick.
Note that you do not want to do this for any version that you want to keep the histograms around for reference. If the version to be deleted from the db has been used as a reference version for other versions, this will make the java interface unhappy for those versions. Deletion form the database has generally only been used when a run was particularly ill-contrived, on throwaway test/debug runs, or if the package version itself was a disaster for some reason.
...
| title | Deleting a version of the tests from the database |
|---|
...
The SCons version of the config file looks like this:
| Panel | ||
|---|---|---|
| ||
packag GlastRelease |
While cmt and scons builds exist in parallel, versions names for tests of scons builds will end in -scons.
| Anchor | ||||
|---|---|---|---|---|
|
Tips and tricks for the savvy tester
- Rerunning the root analysis on existing Gleam outputs
- Re-running the comparisons/setting the default reference version
- Running a subset of the tests
- Deleting a version of the tests from the database
- Setting up to use L1 or Pass8 jobOptions
- Adding new Histograms to the ROOT scripts
| Anchor | ||||
|---|---|---|---|---|
|
Skipping the Gleam step (MC generation, digi, recon, and merit)
If the mc, digi, recon, and merit trees already exist and are populated and you wish to simply regenerate the system test histograms, then adjust the config file to select a root only run.
| Panel | ||
|---|---|---|
| ||
> cd $SYSTESTS/GlastRelease/<version_name> |
This will access the database to find the locations of the root files that were already produced by Gleam and will regenerate the system test histograms. This is very convenient for testing and updating the RootAnalysis macros.
| Anchor | ||||
|---|---|---|---|---|
|
Re-running the comparisons/setting the default reference version
Once the gleam and root jobs have run, and the output files and system test histograms have been created, the comparison can be re-run quickly. In fact, new comparisons between releases or other special runs can be done in the same way. Note that you can check the statistical comparison and overlay plots for any existing version anytime without doing this by using selection boxes on the statistics page or plots page of the system test web interface. This step simply sets the default, i.e. the one that shows up in the summary with the plot failure counts and comes up automatically for a version.
| Panel | ||
|---|---|---|
| ||
> cd $SYSTESTS/src/exec |
| Anchor | ||||
|---|---|---|---|---|
|
Running a subset of the tests
This is incredibly useful for debugging or rerunning tests that encountered an issue.
Run the tests by the normal method, but use a modified configuration file.
| Panel | ||
|---|---|---|
| ||
packag GlastRelease |
Note the additional and optional last line. If the config file includes the subset keyword, then the list of comma-delimited tests following will be run without the others.
If I call this Config_subset.txt then I just run the tests giving that file as the argument, ../../src/exec/runSysTests.sh Config_subset.txt.
If I do this in an existing test result directory, then existing files for the selected tests will be overwritten.
| Anchor | ||||
|---|---|---|---|---|
|
Deleting a version
You probably do not need to do this.
You can always delete an outdated/uninteresting or garbage version of the system tests by simply removing the version result directory. The version will still be listed in the database, but links to the files will be broken.
There is no pressing need to actually do this, but if you do want to eliminate a version of the tests from the database as well, then this might do the trick.
Note that you do not want to do this for any version that you want to keep the histograms around for reference. If the version to be deleted from the db has been used as a reference version for other versions, this will make the java interface unhappy for those versions. Deletion form the database has generally only been used when a run was particularly ill-contrived, on throwaway test/debug runs, or if the package version itself was a disaster for some reason.
| Panel | ||
|---|---|---|
| ||
> cd $SYSTESTS/src/exec |
| Anchor | ||||
|---|---|---|---|---|
|
Setting JobOptions for L1proc or Pass8
The JobOptions parameters for the current v17 version of L1proc are slightly different than the settings for Pass8. Some of which is due to the different G4 versions in use. To switch back and forth, before running system tests, you'll want to "source setMode*.csh" for the appropriate version you are planning to run. There are also two versions of the ROOT scripts for L1proc and P8, TupleFragment.cxx and ReconTkrFragment.cxx. This script handles switching these as well.
The relevant files are in systests/src/exec:
setModeLevel1.csh, setModePass8.csh, which both call setModeVersion.csh.
| Anchor | ||||
|---|---|---|---|---|
|
Adding new Histograms to the scripts
Sometimes it is necessary to add or modify histograms to the existing ROOT scripts when new analyses are requested. The copy of the source to modify is located in $SYSTESTS/src/root. Note that some of the scripts have both a L1 and P8 version associated with them.
...
When running system tests, with new histograms for the first time, you could use the initial run as the reference version to itself - otherwise when running the comparison, there will be errors about missing variables if a previous run is used as reference.
| Anchor | ||||
|---|---|---|---|---|
|