Page History

...

• short-term goal - support calibration data for detectors moving across LCLS experiments.
• long-term goal - calibration store for LCLS-II.

It is assumed that hardware Hardware configuration of modern detectors will detector can be associated with detector index coming from DAQunique index hardwired on detector controller chip. This index will be used to recognize detector and extract hardly retrievable calibration data such as parameters for geometry, gain factors, masks etc. in case of transition of the detector from one experiment to another.

...

• Instrument / Experiment independent - detector can be moved between experiments, hatches, and instruments.
• Portability - calibration data for particular detector should be portable as a self consistent file or (part of) calibration directory.
• Interfaces - API, Command Line (CL), GUI for central management; management can ONLY be done through the API/CL/GUI in order to prevent adding/removing unknown files without history.
• Time stamp (sec) - is THE ONLY value for validity range check; the same rules are applied to the time stamp like for to the run number in current calibration system.
• File name - the <detname> in the calibration store is equivalent to <dettype>-<detid> and is THE ONLY choice for the file name in the calibration store.
• Aliases to the calibration file names and specific data can be used to simplify access.
• Versions - support versions of calibration data.
• Links to predecessor and successor - support access to predecessor and successor if this info is available.

...

To accommodate requirements Detector Calibration Store (DCS) suppose to be implemented as a combination of FS & DB file system (FS) and data base (DB) files (hdf5 for uniformity). Functionality of the DCS from low bottom to top level can be listed as follows.

• All calibration constants for particular detector id (detid - index of the detector version) will be kept in a single hdf5 file named as <dettype>-<detid>.h5
  • Schema of this file contains a few levels which accounts account for constant types, time stamp ranges, versions of constants.
  • Beside main functionality each level contains dictionary of parameters and dictionary of history records.
• These files are grouped in the directory for detector type; ex.: cspad, cspad2x2, pnccd, etc.
  • The same directory contains files with aliases <aliases1>.als which map human readable detector name-and-version with appropriate <dettype>-<detid>.h5 file.
• All detector type folders in regular case are collected under calib directory, although it is assumed that calibration files can be used directly.
• The calib directory may be nested in three locations;
  • repository - experiment-independent space with FS back-up, contains most complete calibration files in terms of time stamp and versions,
  • experimental work-space - experiment-dependent space, contains squeezed version of calibration files with time stamp and versions essential for particular experiment,
  • local work-space - experiment-independent arbitrary spaceany local directory, with squeezed files similar to experimental work-space.

 The repository and experimental works-pace have predictable location in the file system, local work-space may have an arbitrary path. Direct file access pick up calibration file from any place.

•  Two-level API, CL and GUIlevels of interaction with DCS:
  
  • interaction with calibration data in particular calibration file,
    • add/get/remove constants for type, time stamp, version, etc.
    • add/remove/edit metadata.
  • data exchange between files in between repository, experimental and local work-spaces.
    • file difference
    • transfer constants for type, time stamp, version, etc.
  • • use web-service mechanism
• Access Control Lists (ACL) depends on file location and is assumed to be
  
  • repository - limited to dedicated persons
  • experimental work-spaces - members of experimental group
  • local work-spaces - all
  
  
  

...

• The calib directory is a top level DCS directory.
• Detector type <dettype> folder is folders are used to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file.
• Unique part of the detector name <detid> is used to assign calibration file to the particular detector hardware configuration version. If the detector hardware configuration is changing a new calibration file is created with new <detid>.
• Information about predecessor and successor (if available) can be saved under the root level of the calibration file.
• Current sources (ex: 'Camp.0:pnCCD.1' ) should could be presented in the dictionary of aliases.

...

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

The schemeschema (structure) of the DCS files <dettype>-<detid>.h5

/                                                                   # top-root-level contains info for the unique detector version
 dettype (str)                                                      # detector type name, ex: cspad, pnccd, etc
 detid (str)                                                        # unique detector id, ex: 01234
 tscfile (float)                                                    # time stamp of creation of this structure
 predecessor (str)                                                  # name of the previous detector if available or None
 successor (str)                                                    # name of the next detector if available or None
 dictpars (dict)                                                    # dictionary of parameters associated with this version of detector
 history (dict)                                                     # dictionary of history records paired as (tstamp:record)
 <ctype>/                                                           # folder for calibration type, ex.: pedestals, rms, mask, etc
          ctype (str)                                               # calibration type name
          dictpars (dict)                                           # dictionary of parameters associated with calibration type
          history (dict)                                            # dictionary of history records paired as (tstamp:record)
          <tstamp-range>/                                           # folder for time stamp validity range
          <tstamp>[-<tstamp-end>]/                                  # folder for validity range. If <tstamp-end> is not specified - then valid to the end
                                  tsbegin (float)                   # time stamp for the beginning of the validity range
                                  tsend (float)                     # time stamp for the end of the validity range
                                  defaultv                          # reference to the default calibration, ex:  <vers-tstamp2>
                                  dictpars (dict)                   # dictionary of parameters associated with validity range
                                  history (dict)                    # dictionary of history records paired as (tstamp:record)
                                  <vers-tstamp1>/                   # folder for version created on tstamp1
                                                 tsvers (float)     # tstamp1 of this version production 
                                                 calib (ndarray)    # calibration data
                                                 history (dict)     # dictionary of history records paired as (tstamp:record)
                                                 dictpars (dict)    # dictionary of parameters, 
                                                                    #   ex: array size, number of dimensions, shape, data type, experiment, run, comments, author

                                  <vers-tstamp2>/                   # folder for version created on tstamp2
                                                 tsvers (float)
                                                 calib (ndarray)
                                                 history (dict)
                                                 dictpars (dict)
           <tstamp2>[-<tstamp2-end>]/                               # folder for the next validity range.
 pedestals/                                                         # folder for the next calibration type, pedestals
 rms/                                                               # folder for the next calibration type, rms

...

Schema features

• Detector name consists of a common part <dettype> and unique part <detid>.
• Alias to the detector name should be kept in separate dictionary outside the file scheme.
• Each detector may have optional predecessor, successor, and other parameters kept in the dictionary of parameters.
• Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
• Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
• The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of parameters associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
• The number of calibration versions for each time stamp range is unlimited. Default calibration is defined by the reference default version defaultv.

...