here's a first idea on what the new schema might look like:

Release Table:

ReleaseID

Package

Version

VersionLabel

SystemTest Table:

SystemTestID

ReleaseID

Status

Output

StandardID

TestName

Result

Events

CPU

MEM

Metadata Table:

SystemTestID

Histogram

Label

Value

Threshold

OutputFile Table:

SystemTestID

Label

Value

Release Table contains an entry for each RM checked out package.
SystemTest table contains all the test done for a specific entry in Release Table
Metadata table contains all the metadata of a specific test in SystemTest
OutputFile contains the files written out for a specific SystemTest.

I'm not sure how the triggering of it will work. I am considering making it a script that's part of the RM set of scripts which get executed in order.

  • No labels

4 Comments

  1. Richard Dubois

    It would be nice to have a 'diff' to the current schema; could you add that as a comment, please? Is this the entire new schema you have in mind?

    For the record, we did actually write up the existing schema at:

    https://oraweb.slac.stanford.edu:8080/pls/slacquery/LATDOCSORT.asp?doc_num=876

    As I read this, it is entirely tied to the RM builds. How would it accomodate real data, via the pipeline? In that case, the "ReleaseID" would (could) be the ID of a task process id - or something like that.

    Is there any versioning record? If (for example) one reruns the Root macros on a given tests run, the previous results go to the bit bucket?

    Is there a Standard_Tests table in this revision or does StandardID point into another test in SystemTest? A reasonable idea, but going via a relational table at least permits storing information about that comparison - something you might not want to tie in to the SystemTest table. Admittedly we don't store anything about the comparison now. We do store per-histogram quantities in the Systest_Histogram table, including means, sigmas & KS test results... and we allowed thresholds per histogram in the Standard_Test_Thresholds, even though we now set a global threshold. Anyhow, something to think about in terms of what we store about the test comparisons and thresholds.

  2. Julie McEnery

    If we choose to extend the system tests to real data, we could consider adding a second table somewhat analogous to the
    system test table, but with with systest_id replaced with run_id (or whatever is used to track data). The per histogram quantities are
    stored in the metadata table in this schema, the threshold column allows one to set a threshold per histogram, per metadata quantity
    (in case there are more than one statistical test),

  3. Richard Dubois

    Isn't that a bit wasteful? - duplicating an entire table to change the pointer to the parent? Nothing ties ReleaseId to RM (it is not a real pointer). In the old schema, we called it TesterId (or the like) to indicate it could point to different parent types. We could include that parent type as a string in the SystemTest table to ensure fidelity...

  4. Tony Johnson

    One thing I found a little inconvenient in using the existing schema is that there is no table that corresponds to the tests themselves (e.g. AllGamma). There is of course the SYSTEM_TESTS table, but entries there correspond to the result of running a test for a given release version. I would like to see an additional table which assigns an ID to each test, and change the SYSTEM_TESTS table to contain references to that new table.