3.3 Functional Spec for System Tests

Introduction

The System Tests provide performance diagnostics and the ability to browse the results. They also provide comparisons to standards and can flag suspicious comparisons. Diagnostics are created by running multiple configurations of Gleam (via different jobOptions files, specifying source type etc etc) and then running Root macros to analyze the resulting MC/digi/recon trees, producing Root histogram files. The rest of Systests consists of web display of results allowing the user to browse versions, select standards, see comparisons and view plots. It could also generate reports and notify responsible parties of problems.

The System Tests technology should be reusable for testing code releases as well as for providing high level diagnostics for data taking. The latter can be run in the processing pipeline after the full reconstruction of the data.

Requirements

• be triggerable (eg from a Release definition) and run to completion without human intervention
• run multiple configurations of applications (not just GlastRelease - eg G4 tests). Allow for multiple batch jobs for a given configuration (in case individual job CPU limits make for inadequate statistics).
• allow for multiple code packages to be monitored as well as real data processing
• run Root analysis macros to produce plots
• allow robust comparisons of distributions, and maintain a list of standards to compare to
• maintain bookkeeping to track all the generated output, comparisons and link to the appropriate code version used
• provide web access to the histogram and ntuple files
• provide web displays to view the plots ergonomically (not a plot at a time)
• compare to standards (overlays)
• identify likely problem areas to examine in more detail
• generate web report as archival record

Goals

• minimize duplication of code in Gleam and Root configurations. Use some degree of "templating" to customize the differences in very similar configurations
• connect the system tests for GlastRelease to large simulation runs (eg allGamma and background runs) via the Pipeline bookkeeping.
• provide web administration of histogram definitions and display groupings
• allow triggering System Tests from the web by authorized users
• allow users to submit System Tests to private code versions with temporary storage of the output? This could be useful for testing incoming releases or versions that are a little more experimental.

External "actors"

1. batch system
   1. submit jobs
   2. retrieve batch log
   3. perhaps monitor job progress
   4. be notified about job completion
2. email notification of responsible parties about problems
3. afs & nfs space
   1. available space
   2. cleanup of old systests output
   3. archive to tape?
4. user services authentication and privileges
5. Release Manager - query builds db for available versions and the locations of the exe's on disk

Actions

1. discover location of executable belonging to the desired code version
2. checkout version of Systests
3. run set of configurations of the desired applications
4. run Root macros on the resulting Root trees
5. perform comparisons (eg KS tests) against the standards
6. update the bookkeeping with the output files and comparisons
7. create a report based on the comparisons etc
8. provide a web display for the user to browse
9. archive files?