...

• short-term goal - support in a single place for calibration data  detectors moving across LCLS experiments.
• perspective goal long-term goal - calibration store for LCLS-II

It is assumed that hardware configuration of modern detectors will be associated with configuration detector index coming from DAQ. This index will be used to extract hardly retrievable calibration data such as geometry, gain factors etc. in case of transition of the detector from one experiment to another.

Requirements

• Instrument independent - detector can be moved between instruments and hatches. / Experiment independent - detector can be moved between experiments, hatches, and instruments.
• Portability - calibration data for particular detector should be portable as a self consistent file or (part of) calibration directory.
• API + Interfaces - API, Command Line (CL), GUI for central management; management can ONLY be done through the API/CL/GUI in order to prevent adding/removing unknown files without history.
  
• Time stamp (sec) - is THE ONLY value for validity range check; the same rules are applied to the time stamp, like for run number in current calibration system.
• The File name - the <detname> is equivalent to <dettype>-<detid> and is THE ONLY choice for the file name in the calibration store.
• Aliases to the calibration file names and specific data  data can be used to simplify access.
• Support Versions - support for calibration data versions.
• Support links to predecessor and successor.

Architecture

To accommodate requirements Detector Calibration Store (DCS) suppose to be implemented as a combination of FS & DB files (hdf5 for uniformity). Functionality of the DCS from low to top level can be listed as follows.

• All calibration constants for particular version of the detector will be kept in a single hdf5 file named as <dettype>-<detid>.h5
  • Schema of this file contains a few levels which accounts for constant types, time stamp ranges, versions of constants.
  • Beside main functionality each level contains dictionary of tagged parameters and dictionary of history records.
• These files are grouped in the directory for detector type; ex.: cspad, cspad2x2, pnccd, etc.
  • The same directory contains files with aliases <aliases1>.als which map human readable detector name-and-version with appropriate <dettype>-<detid>.h5 file.
• All detector type folders in regular case are collected under calib directory, although it is assumed that calibration files can be used directly.
• The calib directory may be nested in three locations;
  • repository - experiment-independent space with FS back-up, contains most complete calibration files in terms of time stamp and versions,
  • experimental workspacework-space - experiment-dependent space, contains squeezed version of calibration files with time stamp and versions essential for particular experiment,
  • local workspacework-space - experiment-independent arbitrary space, with squeezed files similar to experimental workspacework-space.

 The repository and experimental workspaceworks-pace have predictable location in the file system, local workspacework-space may have an arbitrary path. Direct file access also pick up calibration file from any place.

•  Two-level API for command line , CL and GUI
  
  • interaction with data in particular calibration file,
    • add/get/remove constants for type, time stamp, version, etc.
  • data exchange between files in repository, experimental and local workspaces.
    • file difference
    • transfer constants for type, time stamp, version, etc.
• Access and ACLrepository
  

...

• Control Lists (ACL) depends on file location and is assumed to be
  
  • repository - limited to dedicated persons
  • experimental work-spaces - members of experimental group
  • local work-spaces - all
    
    

Calibration directory and files

...

• /reg/g/psdm/detector/calib

Experimental

...

work-space

experiment-dependent path

• /reg/d/psdm/<INS>/<experiment>/calib

Local

...

work-space

local calib path

• <path>/calib/
  assuming further structure of the calibration directory <dettype>/<dettype>-<detid>.h5

...

Everything in lower case:

<path>/calib/                                   # top calib directory
             <dettype>/                         # detector type folders
             cspad2x2/
             pnccd/
             epix100a/
             cspad/
                       <files>                  # level of files
                       <aliases1>.als           # file with a map of aliases to detector names
                       ...
                       <aliasesN>.als           # one of these files can be used to map "sources" to detector names
                       <dettype>-<detid>.h5     # files with detector-dependent calibration data
                       <dettype>-<detid>.h5-<detid>.h5

The calib directory is a top level DCS directory.

Detector type <dettype>  <dettype>  is used as a folder to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. Unique part of the detector name <detid> is used to assign calibration file to the particular detector version. If the detector is changing the new calibration file is created with new <detid>. Information about predecessor and successor (if available) can be saved under the root level of the calibration file. Current sources (ex: 'Camp.0:pnCCD.1' ) should be presented in the dictionary of aliases.

...

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

The scheme(structure) of the calibration data, which presumably will be implemented in DCS files <dettype>-<detid>.h5

/                                                                   # top-root-level contains info for the unique detector version
 dettype (str)                                                      # detector type name, ex: cspad, pnccd, etc
 detid (str)                                                        # unique detector id, ex: 01234
 tscfile (float)                                                    # time stamp of creation of this structure
 predecessor (str)                                                  # name of the previous detector if available or None
 successor (str)                                                    # name of the next detector if available or None
 tags (dict)                                                        # dictionary of tags associated with this version of detector
 history (dict)                                                     # dictionary of history records paired as (tstamp:record)
 <ctype>/                                                           # folder for calibration type, ex.: pedestals, rms, mask, etc
          ctype (str)                                               # calibration type name
          tags (dict)                                               # dictionary of tags associated with calibration type
          history (dict)                                            # dictionary of history records paired as (tstamp:record)
          <tstamp-range>/                                           # folder for time stamp validity range
          <tstamp>[-<tstamp-end>]/                                  # folder for validity range. If <tstamp-end> is not specified - then valid to the end
                                  tsbegin (float)                   # time stamp for the beginning of the validity range
                                  tsend (float)                     # time stamp for the end of the validity range
                                  defaultv                          # reference to the default calibration, ex:  <vers-tstamp2>
                                  tags (dict)                       # dictionary of tags associated with validity range
                                  history (dict)                    # dictionary of history records paired as (tstamp:record)
                                  <vers-tstamp1>/                   # folder for version created on tstamp1
                                                 tsvers (float)     # tstamp1 of this version production 
                                                 calib (ndarray)    # calibration data
                                                 history (dict)     # dictionary of history records paired as (tstamp:record)
                                                 tags (dict)        # dictionary of tags, 
                                                                    #   ex: array size, number of dimensions, shape, data type, experiment, run, comments, author

                                  <vers-tstamp2>/                   # folder for version created on tstamp2
                                                 tsvers (float)
                                                 calib (ndarray)
                                                 history (dict)
                                                 tags (dict)
           <tstamp2>[-<tstamp2-end>]/                               # folder for the next validity range.
 pedestals/                                                         # folder for the next calibration type, pedestals
 rms/                                                               # folder for the next calibration type, rms

...

• Detector name consists of a common part <dettype> part <dettype> and unique part <detid>.
• Alias to the detector name should be kept in separate dictionary outside the file scheme.
• Each detector may have optional predecessor, successor, and other tags in the dictionary.
• Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
• Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
• The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
• The number of calibration versions for each time stamp range is not limited. Default calibration is defined by the reference default_version.

...

API

Parameters

dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

...