Introduction
While the SSC maintains its server for GLAST Level 1 data to distribute publicly, the LAT team needs access to all levels of LAT-produced data and simulations. These are expected to be used for detailed instrument studies as well as providing access to event details for Level 1 events that strike our fancy.
The pipeline organizes its products via a metadata database. Files can be located by the "task" name, "run" identifier and data type. Individual events are not referred to in the database. It is assumed for now that the pipeline is entirely responsible for populating the database.
Requirements
What Data
The data server's job is to give access to all the various tasks and data types available in the database. This will include flight integration data as well as MC simulations, and (obviously) flight data.
Formats
The server must deal with the data formats in use by the LAT team: Level 0 is converted to "digis" and then processed through reconstruction; output is primarily stored in Root format. It is not clear yet whether server access to L0 event data is required. Level 1 data ends up as FITS at the SSC and are the input to Science Tools.
Data Products to Handle
If they are stored in the pipeline's dataset catalogue, the server ought to be able to fetch them for the user. It is not clear yet what all the dataset types are or what search criteria will be appropriate.
The user should be able to select outputs as ntuples, combinations of MC/Digi/Recon trees and FITS L1 files.
The FITS Level 1 data is actually extracted from the Root Merit ntuple. The server should be able to perform the L1 Root to FITS conversion on demand. The same holds for the pointing and livetime histories.
Additional data types might include housekeeping.
Queries
Queries take on two characters:
- apply cuts to all events in a particular set (as a result of the query) of files
- find individual events by their run/event tags
Item 2 is pretty obvious. Item 1 will likely depend somewhat on practicalities. It is straightforward to apply cuts on the Merit ntuple columns; it is much less so to apply detailed cuts on the various Root MC/Digi/Recon trees. One should at least be able to cut on Merit columns. We can think about whether the user can supply a macro to select from the trees.
Delivery of Results
The resulting events should be gathered into output files and made available to the requestor via ftp and the user notified. Whether the events are put into a single file, multiple files with an archive or multiple files will likely depend on operating system limitations (eg 2 GB file size limit on unix - still there?). Note that ftp space is just nfs disk at SLAC, so SLAC users have direct access to the files.
The server will be responsible for cleaning up its ftp space as needed to make way for new queries. This space should be sized to match the observed user load; it is expected to be on the order of 200-500 GB to start. There must also be sufficient disk scratch space to buffer files that had to be retrieved from the tape archives.
Response Time
The SSC thought long and hard about its response requirements. Given that the reconstruction files can be quite large, it is not possible to achieve the same throughput for them as for ntuples. I can only think of "as fast as reasonably achieved".
How quickly we want the results back will determine the implementation - parallel queries, spatial indexes for events etc.
Audit Trail
The user should be supplied with a "bill of lading" indicating what their request was and some summary of the results (eg numbers of events, filename and size). Should we keep a record of them? Could others re-use them?
User Authentication
It must be possible to authenticate users of the server. It probably will not be open to the public.
Documentation
The server should be equipped with online documentation for user and operators manuals. The latter should be a reference document for the server as well as containing "how-to-fix" information.
Server Maintenance
The server should be able to give a status report on its health and recent usage.