Content
Purpose
- short-term goal - support in a single place detectors moving across LCLS experiments.
- perspective goal - calibration store for LCLS-II
It is assumed that hardware configuration of modern detectors will be associated with configuration index coming from DAQ. This index will be used to extract hardly retrievable calibration data such as geometry, gain factors etc. in case of transition of the detector from one experiment to another.
Requirements
- Instrument independent - detector can be moved between instruments and hatches.
- Experiment independent - detector can be moved between experiments.
- Portability - calibration data for particular detector should be portable as a self consistent file or (part of) calibration directory.
- API + GUI for central management; management can ONLY be done through the API/GUI in order to prevent adding/removing unknown files without history.
- Time stamp (sec) is THE ONLY value for validity range check; the same rules are applied to the time stamp, like for run number in current calibration system.
- The <detname> is equivalent to <dettype>-<detid> and is THE ONLY choice for the file name in the calibration store.
- Aliases to the calibration file names can be used to simplify access.
- Support calibration versions.
- Support links to predecessor and successor.
Architecture
To accommodate requirements calibration store suppose to be implemented as a combination of FS & DB files (hdf5 for uniformity).
Calibration directory and files
Default calibration directory in LCLS FS
instrument and experiment-independent path
- /reg/g/psdm/detector/calib
in contrast to the current instrument and experiment-dependent path
- /reg/d/psdm/<INS>/<experiment>/calib
In local space
define calib path
- <path>/calib/
assuming further structure of the calibration directory <dettype>/<dettype>-<detid>.h5
or use detector calibration file directly
- <path>/<dettype>-<detid>.h5
Structure of calibration directory
Everything in lower case:
<path>/calib/
<dettype>/
cspad2x2/
pnccd/
epix100a/
cspad/
<files>
<aliases1>.als # file with a map of aliases to detector names
...
<aliasesN>.als # one of these files can be used to map "sources" to detector names
<dettype>-<detid>.h5
<dettype>-<detid>.h5
Detector type <dettype> is used as a folder to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. Unique part of the detector name <detid> is used to assign calibration file to the particular detector version. If the detector is changing the new calibration file is created with new <detid>. Information about predecessor and successor (if available) can be saved under the root level of the calibration file. Current sources (ex: 'Camp.0:pnCCD.1' ) should be presented in the dictionary of aliases.
Calibration file structure
Calibration data hierarchical scheme
Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM
The scheme of the calibration data, which presumably will be implemented in <dettype>-<detid>.h5
/ # top-root-level contains info for the unique detector version
dettype (str) # detector type name, ex: cspad, pnccd, etc
detid (str) # unique detector id, ex: 01234
tscfile (uint) # time stamp of creation of this structure
predecessor (str) # name of the previous detector if available or None
successor (str) # name of the next detector if available or None
tags (dict) # dictionary of tags associated with this version of detector
<ctype>/ # folder for calibration type
ctype (str) # calibration type name
tags (dict) # dictionary of tags associated with calibration type
<tstamp-range>/ # folder for time stamp validity range
<tstamp>[-<tstamp-end>]/ # folder for validity range. If <tstamp-end> is not specified - then valid to the end
tsbegin (uint) # time stamp for the beginning of the validity range
tsend (uint) # time stamp for the end of the validity range
defaultv # reference to the default calibration, ex: <vers-tstamp2>
tags (dict) # dictionary of tags associated with validity range
<vers-tstamp1>/ # folder for version created on tstamp1
tsvers (uint) # tstamp1 of this version production
calib (ndarray) # calibration data
history (str) # production history record
tags (dict) # dictionary of tags,
# ex: array size, number of dimensions, shape, data type, experiment, run, comments, author
<vers-tstamp2>/ # folder for version created on tstamp2
tsvers (uint)
calib (ndarray)
history (str)
tags (dict)
<tstamp2>[-<tstamp2-end>]/ # folder for the next validity range.
pedestals/ # folder for the next calibration type, pedestals
rms/ # folder for the next calibration type, rms
Scheme features
- Detector name consists of a common part <dettype> and unique part <detid>.
- Alias to the detector name should be kept in separate dictionary outside the scheme.
- Each detector may have optional predecessor, successor, and other tags in the dictionary.
- Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
- Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
- The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
- The number of calibration versions for each time stamp range is not limited. Default calibration is defined by the reference default_version.
Metadata in tags
Below are the lists of metadata fields which potentially can be used to define detector configuration, calibration parameters etc.
Detector metadata
| Field name | Description | More details, example |
|---|---|---|
| dettype | detector type | CSPAD, CSPAD2X2, EPIX100A, etc. |
| detname | detector unique name | (if any) ex.: Camera1 |
| detalias | alias name | if it is hard to memorize the entire name, ex.: 'cspad1' |
| detidx | detector index | integer number which codes the hardware version |
| detidxalias | symbolic alias of the index | can be used if it is hard to memorize the index integer number |
| detcompidx:001 | list of component indexes | just in case if we are going to retrieve calibration parameters for separate components |
| detidxprev | detector index for previous version | detector index for previous version (if available) for the purpose of old calibration search |
| detidxnext | detector index for next version | detector index for next version (if available) for the purpose of new calibration search |
| dettsec | time-stamp | time stamp associated with beginning of the validity range for new configuration |
| detcom:001 | comments for this hardware version | as it says {key:comment} |
| dettag:001 | other tags | just in case if something is forgotten in this table |
Calibration parameters' metadata
| Field name | Description | More details, example |
|---|---|---|
| calibtype | calibration type | ex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc. |
| tsec | time stamp | beginning of the validity range |
| exp | original experiment | (if available) where calibration constants were obtained |
| runnum | original run number | (if available) where calibration constants were obtained |
| runbegin | begin run number | (if available) for validity range |
| runend | end run number | (if available) for validity range |
| source | original DAQ data source | data source from DAQ, ex.: 'CxiDs2.0:Cspad.0' |
| srcalias | data source alias | ex.: 'cspad' |
| calibvers | version tag | in order to access using symbolic name or some alias |
| calibversalias | version alias | if it is hard to memorize version tag |
| com:001 | comments for this version | as it says {key:comment} |
| tag:001 | other tags | just in case if something is forgotten in this table |
In case of numpy array their metadata are stored with an object.
Text file needs in n-d array metadata
n-d array metadata
| Field name | Description | More details, example |
|---|---|---|
| dtype | data type | int, float, double, etc |
| ndims | number of dimensions (N) | as it says |
| dim:1 | size of dim.1 | ... |
| dim:2 | size of dim.2 | ... |
| ... | ... | ... |
| dim:N | size of dim.N | ... |
API
Parameters
dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.
ctype (str) - calibration type: pedestals, status, rms, mask, gain, bkgd, common_mode, geometry
detid (str) - detector unique id number, ex: 123456
detname (str) - detector unique name, combination of <dettype>-<detid>
detalias (str) - detector alias name assigned to the detname
cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.
version (int) - time stamp of the version creation
versind (int) - consecutive version index assigned/mapped to the version production time stamp.
Initialization
# Import
from PSCalib.DCStore import DCStore # inport DCStore (Detector Calibration Store) object
# Initialization
CPATH_DEF = '/reg/g/psdm/detector/calib' # default calib directory
cpath = '<path>/calib' # local calib directory
cpath = '<path>/<dettype>-<detid>.h5' # direct to file
cpath = env.calibDir() # '/reg/d/psdm/<INS>/<experiment>/calib' - accept current directory
detname = 'pnccd-12345678' # standard name includes detector type, dash, and n-digit id number
detname = 'Camera1' # alias
cs = DCStore(cpath, detname) # creates a DCStore object.
""" get calibration store object
Input parameters:
cpath [str] - path to the hdf5 file or calibration directory.
If cpath is a path to hdf5 file - next parameter is ignored.
If cpath=None - default path is used.
If cpath is specified as a path to directory (or default) then detname can be an alias.
detname [str] - name/alias of the detector
"""
UI acess methods
ctype = pedestals # status, rms, mask, gain, bkgd, common_mode, geometry, etc tsp = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt vers = None # for default or versind or version time stamp. # generic access method: obj = cs.get(ctype, tsp, vers)
Base class
Is reserved to support common methods of all project classes. For now it stands for manipulations with tags but not limited to.
DCBase - base class for this project ==================================== tags = obj.tags() # returns (dict) dictionary of tags associated with each object tag = obj.tag(k) # returns tag value for key k log = obj.history() # returns (str) history records associated with current object obj.set_tags(d) # set (dict) dictionary of tags for object obj.add_tag(k,v) # add (k,v) tag to the dictionary of tags for object obj.del_tags() # delete all tags from the dictionary obj.del_tag(k) # delete tag with key k obj.add_record(rec) # add (str) hystory record
Access methods
DCStore methods
===============
tscfile = cs.tscfile() # (int) time stamp of the file creation
dettype = cs.dettype() # (str) detector type
detid = cs.detid() # (str) detector id
detname = cs.detname() # (str) detector name of self object
predecessor = cs.predecessor() # (str) detname of predecessor or None
successor = cs.successor() # (str) detname of successor or None
ctypes = cs.ctypes() # (list) calibration types in the file
cto = cs.ctypeobj(ctype) # (DCType ~ h5py.Group) calibration type object
_________________________________________
nda = cs.get(ctype, tsp, vers) # (str) ctype - calibration type
# (...) tsp - parameter to get time stamp (evt, runnum, ts_sec)
_________________________________________# (int) vers - version of calibration, None - use default
DCType methods
==============
ctype = cto.ctype() # (str) of ctype name
tsranges = cto.ranges() # (list) of time ranges for ctype
tsro = cto.rangeobj(tsrange) # (DCRange ~ h5py.Group) time stamp validity range object
DCRange methods
===============
tsbegin = tsro.begin() # (int) time stamp beginning validity range
tsend = tsro.end() # (int) time stamp ending validity range
versions = tsro.versions() # (list of uint) versions of calibrations
versodef = tsro.versdef() # (DCVersion ~ h5py.Group) reference to the default version in the time-range object
verso = tsro.versobj(vers) # (DCVersion ~ h5py.Group) specified version in the time-range object
DCVersion methods
=================
tsvers = verso.tsprod() # (int) time stamp of the version production
calibdata = verso.calib() # (np.array) calibration array
Management methods
DCStore methods =============== cs.set_tscfile(ts) # set (int) time stamp of the file creation cs.set_dettype(dettype) # set (str) detector type cs.set_detid(detid) # set (str) detector id cs.set_detname(detname) # set (str) detector name of self object cs.set_predecessor(pred) # set (str) detname of predecessor or None cs.set_successor(succ) # set (str) detname of successor or None cs.add_ctype(ctype) # add (str) calibration type to the DCStore object cs.del_ctype(ctype) # delete ctype (str) from the DCStore object cs.save(path) # save current calibration in the file specified by path, if path is Null - update current file. DCType methods ============== cto.add_range(tsr) # add (str) of time ranges for ctype cto.del_range(tsr) # delete range from the DCType object DCRange methods =============== tsro.set_begin(tsbegin) # set (int) time stamp beginning validity range tsro.set_end(tsend) # set (int) time stamp ending validity range tsro.add_version(vers) # set (DCVersion ~ h5py.Group) versions of calibrations tsro.set_versdef(vers) # set (DCVersion ~ h5py.Group) versions of calibrations tsro.del_version(vers) # delete version DCVersion methods ================= verso.set_tsprod(tsprod) # set (int) time stamp of the version production verso.add_calib(nda) # set (np.array) calibration array
TBD
Open questions
- What is included in the detector Id version? FEE version, controller version, etc?
- when constants are updated (file open to write) they are not available... Lock to resolve.
Data flow
- who produces and supply constants
- who is allowed to submit constants
- who is allowed to access constants
- ACL inside API or using OS
- ACL for all or particular detector/type/ etc.
References
Overview
Content Tools