Page History
Content
Table of Contents |
---|
Purpose
- short-term goal - support in a single place calibration data for detectors moving across LCLS experiments.
- perspective goal long-term goal - calibration store for LCLS-II.
It is assumed that hardware Hardware configuration of modern detectors will detector can be associated with configuration index coming from DAQunique index hardwired on detector controller chip. This index will be used to recognize detector and extract hardly retrievable calibration data such as parameters for geometry, gain factors, masks etc. in case of transition of the detector from one experiment to another.
Requirements
- Instrument independent - detector can be moved between instruments and hatches. / Experiment independent - detector can be moved between experiments, hatches, and instruments.
- Portability - calibration data for particular detector should be portable as a self consistent file or (part of) calibration directory.
- API + Interfaces - API, Command Line (CL), GUI for central management; management can ONLY be done through the API/CL/GUI in order to prevent adding/removing unknown files without history.
- Time stamp (sec) - is THE ONLY value for validity range check.; the same rules are applied to the time stamp like to the run number in current calibration system.
- File name - the <detname> The <detname> is equivalent to <dettype>-<detid> and is THE ONLY choice for the file name in the calibration store is <dettype>-<detid>.
- Aliases to the calibration file names and specific data can be used to simplify access.
- Versions - support versions of calibration data.
- Links to predecessor and successor - support access to predecessor and successor if this info is available.
Architecture
To accommodate requirements calibration store Detector Calibration Store (DCS) suppose to be implemented as a combination of FS & DB file system (FS) and data base (DB) files (hdf5 for uniformity).
Calibration directory and files
Default calibration directory in LCLS FS
instrument and experiment-independent path
- /reg/g/psdm/detector/calib
in contrast to the current instrument and experiment-dependent path
- /reg/d/psdm/<INS>/<experiment>/calib
In local space
define calib path
- <path>/calib/
assuming further structure of the calibration directory <dettype>/<dettype>-<detid>.h5
or use detector calibration file directly
- <path>/<dettype>-<detid>.h5
Structure of calibration directory
Everything in lower case:
Functionality of the DCS from bottom to top level can be listed as follows.
- All calibration constants for particular detector id (
detid
- index of the detector version) will be kept in a single hdf5 file named as<dettype>-<detid>.h5
- Schema of this file contains a few levels which account for constant types, time stamp ranges, versions of constants.
- Beside main functionality each level contains dictionary of parameters and dictionary of history records.
- These files are grouped in the directory for detector type; ex.:
cspad, cspad2x2, pnccd
, etc.- The same directory contains files with aliases
<aliases1>.als
which map human readable detector name-and-version with appropriate<dettype>-<detid>.h5
file.
- The same directory contains files with aliases
- All detector type folders in regular case are collected under
calib
directory, although it is assumed that calibration files can be used directly. - The
calib
directory may be nested in three locations;- repository - experiment-independent space with FS back-up, contains most complete calibration files in terms of time stamp and versions,
- experimental work-space - experiment-dependent space, contains squeezed version of calibration files with time stamp and versions essential for particular experiment,
- local work-space - experiment-independent any local directory, with squeezed files similar to experimental work-space.
The repository and experimental works-pace have predictable location in the file system, local work-space may have an arbitrary path. Direct file access pick up calibration file from any place.
- Two-levels of interaction with DCS:
- interaction with calibration data in particular calibration file,
- add/get/remove constants for type, time stamp, version, etc.
- add/remove/edit metadata.
- data exchange between files between repository, experimental and local work-spaces.
- file difference
- transfer constants for type, time stamp, version, etc.
- use web-service mechanism
- interaction with calibration data in particular calibration file,
- Access Control Lists (ACL) depends on file location and is assumed to be
- repository - limited to dedicated persons
- experimental work-spaces - members of experimental group
- local work-spaces - all
Calibration directory and files
Repository
instrument and experiment-independent path to calibration directory and file
- <PATH>/detector/calib/
<dettype>/<dettype>-<detid>.h5
Experiment calibration directory
experiment-dependent path to calibration directory and file
- <PATH>/<INS>/<experiment>/calib/
<dettype>/<dettype>-<detid>.h5
Local calibration directory
local calib path to calibration directory and file
- <path-to-calib>/calib/
<dettype>/<dettype>-<detid>.h5
Direct path to calibration file
direct path to calibration file
- <path-to-calib>/<dettype>-<detid>.h5
Structure of calibration directory
Everything in lower case:
Code Block |
---|
<path-to-calib>/calib/ # top calib directory
<dettype>/ # detector type folders
cspad2x2/
pnccd/
epix100a/
cspad/
<files> # level of files
<aliases1>.als # file with a map of aliases to detector names
...
<aliasesN>.als # one of these files can be used to map "sources" to detector names
<dettype>-<detid>.h5 # files with detector-dependent calibration data
<dettype>-<detid>.h5 |
- The
calib
directory is a top level DCS directory. - Detector type
<dettype>
folders are used to organize files under thecalib
directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. - Unique part of the detector name
<detid>
is used to assign calibration file to the particular detector hardware configuration version. If the detector hardware configuration is changing a new calibration file is created with new<detid>
. - Information about predecessor and successor (if available) can be saved under the root level of the calibration file.
- Current sources (ex:
'Camp.0:pnCCD.1'
) could be presented in the dictionary of aliases.
Calibration file structure
Calibration data hierarchical scheme
Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM
The schema (structure) of the DCS files <dettype>-<detid>.h5
Code Block |
---|
DCStore DCType |
Code Block |
<path>/calib/ <dettype>/ cspad2x2/ pnccd/ epix100a/ cspad/ <files> DCRange DCVersion / <aliases1>.als # file with a map of aliases to detector names ... # top-root-level contains info for the unique detector <aliasesN>.alsversion dettype (str) # one of these files can be used to map "sources" to detector names <dettype>-<detid>.h5 # detector type <dettype>-<detid>.h5 |
Detector type <dettype> is used as a folder to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. Unique part of the detector name <detid> is used to assign calibration file to the particular detector version. If the detector is changing the new calibration file is created with new <detid>. Information about predecessor and successor (if available) can be saved under the root level of the calibration file. Current sources (ex: 'Camp.0:pnCCD.1' ) should be presented in the dictionary of aliases.
Calibration file structure
Calibration data hierarchical scheme
Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM
The scheme of the calibration data, which presumably will be implemented in <dettype>-<detid>.h5
Code Block |
---|
/ name, ex: cspad, pnccd, etc detid (str) # unique detector id, ex: 01234 tscfile (float) # time stamp of creation of this structure predecessor (str) # name of the previous detector if available or None successor (str) # top-root-level contains# infoname forof the uniquenext detector versionif available or None dettypedictpars (strdict) # dictionary #of detectorparameters typeassociated name,with ex:this cspad,version pnccd,of etcdetector detidhistory (strdict) # dictionary of #history uniquerecords detectorpaired id,as ex: 01234 tscfile (uint)(tstamp:record) <ctype>/ # folder time stamp of creation of this structure predecessor (str)for calibration type, ex.: pedestals, rms, mask, etc ctype (str) # name of the previous detector if# availablecalibration ortype Nonename successor (str) dictpars (dict) # namedictionary of theparameters nextassociated detectorwith if available or None tags (dict)calibration type history (dict) # dictionary of tagshistory associatedrecords withpaired this version of detectoras (tstamp:record) <ctype>/ <tstamp-range>/ # folder for calibrationtime type stamp validity range ctype (str) <tstamp>[-<tstamp-end>]/ # folder for validity range. If <tstamp-end> is not specified - then #valid calibrationto typethe nameend tags (dict) tsbegin (float) # time stamp for #the dictionarybeginning of the tagsvalidity associatedrange with calibration type <tstamp-range>/ tsend (float) # foldertime stamp for the end timeof stampthe validity range <tstamp>[-<tstamp-end>]/ defaultv # folder for validity range. If <tstamp-end> is not specified - then# validreference to the enddefault calibration, ex: <vers-tstamp2> tsbegindictpars (uintdict) # timedictionary stampof forparameters theassociated beginningwith of the validity range tsendhistory (uintdict) # time stamp for the end of the validity range # dictionary of history records paired as (tstamp:record) defaultv <vers-tstamp1>/ # reference to the default calibration,# ex:folder for <vers-tstamp2> version created on tstamp1 tags (dict) tsvers (float) # dictionarytstamp1 of tagsthis associatedversion withproduction validity range <vers-tstamp1>/ calib (ndarray) # folder for version created on tstamp1calibration data tsvershistory (uintdict) # tstamp1dictionary of history thisrecords versionpaired productionas (tstamp:record) calibdictpars (ndarraydict) # calibrationdictionary data of parameters, tags (dict) # dictionary of tags, # ex: array size, number of dimensions, shape, data type, experiment, run, comments, author # <vers-tstamp2>/ ex: array size, number of dimensions, shape, data type, experiment, run, comments, author # folder for version created on tstamp2 <vers-tstamp2>/ # folder for version created ontsvers tstamp2(float) tsverscalib (uintndarray) history calib (ndarraydict) tagsdictpars (dict) <tstamp2>[-<tstamp2-end>]/ # folder for the next validity range. pedestals/ # folder for the next calibration type, pedestals rms/ # folder for the next calibration type, rms |
...
Schema features
- Detector name consists of a common part <dettype> part
<dettype>
and unique part<detid>
. - Alias to the detector name should be kept in separate dictionary outside the file scheme.
- Each detector may have optional predecessor, successor, and other tags parameters kept in the dictionary of parameters.
- Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
- Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
- The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags parameters associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
- The number of calibration versions for each time stamp range is not limitedunlimited. Default calibration is defined by the reference default _ version
defaultv
.
...
Metadata in
...
dictpars
Below are the lists of metadata fields which potentially can be used to define detector and access calibrationsdetector configuration, calibration parameters etc.
Detector metadata
Field name | Description | More details, example |
---|---|---|
dettype | detector type | CSPAD, CSPAD2X2, EPIX100A, etc. |
detname | detector unique name | (if any) ex.: Camera1 |
detalias | alias name | if it is hard to memorize the entire name, ex.: 'cspad1' |
detidx | detector index | integer number which codes the hardware version |
detidxalias | symbolic alias of the index | can be used if it is hard to memorize the index integer number |
detcompidx:001 | list of component indexes | just in case if we are going to retrieve calibration parameters for separate components |
detidxprev | detector index for previous version | detector index for previous version (if available) for the purpose of old calibration search |
detidxnext | detector index for next version | detector index for next version (if available) for the purpose of new calibration search |
dettsec | time-stamp | time stamp associated with beginning of the validity range for new configuration |
detcom:001 | comments for this hardware version | as it says {key:comment} |
detpar:001 | other |
parameters | just in case if something is forgotten in this table |
Calibration parameters' metadata
Field name | Description | More details, example |
---|---|---|
calibtype | calibration type | ex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc. |
tsec | time stamp | beginning of the validity range |
exp | original experiment | (if available) where calibration constants were obtained |
runnum | original run number | (if available) where calibration constants were obtained |
runbegin | begin run number | (if available) for validity range |
runend | end run number | (if available) for validity range |
source | original DAQ data source | data source from DAQ, ex.: 'CxiDs2.0:Cspad.0' |
srcalias | data source alias | ex.: 'cspad' |
calibvers | version |
par | in order to access using symbolic name or some alias | |
calibversalias | version alias | if it is hard to memorize version |
par | ||
com:001 | comments for this version | as it says {key:comment} |
par:001 | other |
parameters | just in case if something is forgotten in this table |
In case of numpy array their metadata are stored with an object.
Text file needs in n-d array metadata
n-d array metadata
Field name | Description | More details, example |
---|
DTYPE | (str) data type | int, float, double, etc. |
NDIMS | (int) number of dimensions (N) |
ex.: 3 | ||
DIM:1 | (int) size of dim.1 | ex. |
: 185 |
DIM:2 | (int) size of dim.2 | ex. |
: 388 | ||
... | ... | ... |
DIM:N | (int) size of dim.N | ex. |
: 2 |
API
Parameters
dettype
(str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a
, etc.
...
version
(int) - time stamp of the version creation
versind
(int) - consecutive version index assigned/mapped to the version production time stamp.
Initialization
Code Block |
---|
# Import from PSCalib.DCStore import DCStore # inport DCStore (Detector Calibration Store) object # Initialization CPATH_DEFREPO = '/reg/g/psdm<PATH>/detector/calib' # default calib repository cdir = env.calibDir() # '<PATH>/<INS>/<experiment>/calib' - accept current directory cpathpath = cdir + <dettype> fname = '<path>/calib[path/]pnccd-12345678' # standard name includes detector type, dash, and n-digit # local calib directory cpathid number fname = '<path>/<dettype>-<detid>.h5'[path/]Camera1' # direct to# filealias cpathcs = env.calibDirDCStore(fname) # '/reg/d/psdm/<INS>/<experiment>/calib' - accept current directory detname = 'pnccd-12345678' creates a DCStore object. """ get calibration store object Input parameters: fname [str] - file name/alias of the detector """ |
- If path is missing - use repository.
- If calibration file is not found - throw raise IOError('File %s is not available' % fname)
UI acess methods
Code Block |
---|
ctype = pedestals # standardstatus, namerms, includesmask, detectorgain, typebkgd, dashcommon_mode, and n-digit id number detnamegeometry, etc tsp = 'Camera1'tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt vers = None # for default or versind or version time stamp. # generic access method: obj = cs.get(ctype, tsp, vers=None) |
Base class DCBase
Is reserved to support common methods of all project classes. For now it stands for manipulations with dictpars but not limited to.
Code Block |
---|
o = DCBase() # acess methods dictpars = o.dictpars() # alias cs = DCStore(cpath, detname) # creates a DCStore object. """ get calibration store object Input parameters: cpath [str] - path to the hdf5 file or calibration directory. # returns (dictpars) dictionary of dictpars associated Ifwith cpath is a path to hdf5 file - next parameter is ignored. If cpath=None - default path is used. each object par = o.par(k) # returns par Ifvalue cpathfor iskey specifiedk log as a path to directory (or default) then detname can be an alias. detname [str] - name/alias of the detector """ |
UI - acess methods
Code Block |
---|
p = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt
nda = cs.get(ctype, p) # where ctype stands for pedestals, status, rms, geometry, etc.
# or specific access methods:
nda = cs.pedestals(p)
nda = cs.pixel_status(p)
nda = cs.pixel_rms(p)
nda = cs.pixel_mask(p)
nda = cs.pixel_gain(p)
nda = cs.pixel_bkgd(p)
nda = cs.common_mode(p)
geo = cs.geometry(p)
|
Base class
Code Block |
---|
DCBase - base class for this project ==================================== tags = obj.tags()= o.history(fmt) # returns (str) history records preceded by the time stamp (default fmt='%Y-%m-%dT%H:%M:%S%Z') as a text d = o.histdict() # returns (dict) history dictionary associated with current object # management methods o.set_dictpars(d) # set (d) dictionary of parameters for object o.add_par(k,v) # add (dict)k,v) par to the dictionary of tags associated with each object tag = obj.tag parameters for object o.del_dictpars() # delete all parameters from the dictionary o.del_par(k) # returnsdelete tagpar value forwith key k objo.set_tagshistory(d) # set (dict) as a history dictionary of tagsthe forcurrent object objo.add_taghistory(krec,v ts) # add (str) record with # add (k,v) tag(float) time stamp to the history dictionary of tags for object obj.del_tags() (ts:rec). If ts is None - call current time is used as a key. |
Class DCStore
Code Block |
---|
cs = DCStore(cpath, detname) # delete all tags from the dictionary obj.del_tag(k# (str) path to calib directory or file, (str) detector name # acess methods nda = cs.get(ctype, tsp, vers) # delete(str) tagctype with key k |
Access methods
Code Block |
---|
DCStore methods =============== tscfile- calibration type = cs.tscfile() # (int) time stamp of the file creation dettype =# cs(...dettype() tsp - parameter to get time stamp (evt, runnum, ts_sec) # (str) detector type detid = cs.detid() # (str) detector id detnameint) vers - version of calibration, None - use default tscfile = cs.detnametscfile() # (int) time stamp of the # (str) detector name of self object predecessor file creation dettype = cs.predecessordettype() # (str) detector detnametype detid of predecessor or None successor = cs.successordetid() # (str) detname of successor or None ctypesdetector id detname = cs.ctypesdetname() # (str) detector name of self object predecessor = cs.predecessor() # (liststr) calibrationdetname typesof inpredecessor theor fileNone ctosuccessor = cs.ctypeobjsuccessor(ctype) # (DCType ~ h5py.Group) calibration type object _________________________________________ nda str) detname of successor or None ctypes = cs.get(ctype, tsp, vers)ctypes() # (strlist) calibration ctypetypes -in calibrationthe type file cto = cs.ctypeobj(ctype) # (DCType ~ h5py.Group) calibration type object # management methods cs.set_tscfile(ts) # set (int) time stamp of the file creation cs.set_dettype(dettype) # (...) tsp - parameter to get time# stampset (evt, runnum, ts_sec) _________________________________________# (int) vers - version of calibration, None - use default DCType methods ============== ctype = cto.ctype(str) detector type cs.set_detid(detid) # set (str) detector id cs.set_detname(detname) # #set (str) ofdetector ctype name tsranges of = cto.ranges(self object cs.set_predecessor(pred) # set # (liststr) detname of timepredecessor rangesor for ctype tsroNone cs.set_successor(succ) = cto.rangeobj(tsrange) # set # (DCRange ~ h5py.Group(str) timedetname stampof validitysuccessor rangeor object DCRange methods =============== tsbeginNone cs.add_ctype(ctype) = tsro.begin() # add (str) calibration type to the # (intDCStore object cs.del_ctype(ctype) time stamp beginning validity range tsend = tsro.end() # delete ctype (str) from the DCStore object cs.save(path) # (int) time stamp ending validity range versions = tsro.versions() # save current calibration in the file specified #by (listpath, ofif uint)path versionsis ofNull calibrations versodef- update current file. |
Class DCType
Code Block |
---|
cto = tsro.versdefDCType(dettype) # (DCVersion ~ h5py.Groupstr) reference to the default version in the time-range object versodetector type # acess methods ctype = tsrocto.versobjctype(vers) # (DCVersion ~ h5py.Group) specified version in the time-range object DCVersion methods ================= tsvers = verso.tsprod() str) of ctype name tsranges = cto.ranges() # (list) of time ranges for ctype tsro = cto.rangeobj(tsrange) # (intDCRange ~ h5py.Group) time stamp validity of the version production calibdatarange object # management methods cto.add_range(tsr) = verso.calib() # #add (np.arraystr) calibration array |
Management methods
Code Block |
---|
DCStore methods =============== cs.set_tscfile(tsof time ranges for ctype cto.del_range(tsr) # setdelete (int)range timefrom stamp of the DCType object |
Class DCRange
Code Block |
---|
tsro = DCRange(tsrangefile creation cs.set_dettype(dettype) # set (str) detector type cs.set_detid(detid) time stamp validity range tsro = DCRange(tsbegin, tsend) # set (strint,int) detector id cs.set_detname(detname) time stamp validity range # acess methods tsbegin # set (str) detector name of self object cs.set_predecessor(pred= tsro.begin() # set (strint) detnametime ofstamp predecessorbeginning orvalidity None cs.set_successor(succrange tsend = tsro.end() # set (strint) detnametime ofstamp successorending orvalidity None cs.add_ctype(ctyperange versions = tsro.versions() # (list of float) versions of calibrations versodef # add (str) calibration type to the DCStore object cs.del_ctype(ctype) = tsro.versdef() # (DCVersion ~ h5py.Group) reference to the default version in the time-range object verso # delete ctype= tsro.versobj(strvers) from the DCStore# object cs.save(path) (DCVersion ~ h5py.Group) specified version in the time-range object # management methods tsro.set_begin(tsbegin) # saveset current(int) calibrationtime instamp thebeginning file specified by path, if path is Null - update current file. DCType methods ============== cto.add_range(tsr) validity range tsro.set_end(tsend) # addset (strint) oftime timestamp rangesending forvalidity ctyperange ctotsro.deladd_rangeversion(tsrvers) # deleteset range(DCVersion from the DCType object DCRange methods ===============~ h5py.Group) versions of calibrations tsro.set_beginversdef(tsbeginvers) # set (int)DCVersion time stamp beginning validity range~ h5py.Group) versions of calibrations tsro.setdel_endversion(tsendvers) # set (int) time stamp ending validity range tsro.add_versiondelete version |
Class DCVersion
Code Block |
---|
verso = DCVersion(vers) # set (DCVersionstr) ~ h5py.Group) versions of calibrations tsro.set_versdef(vers)version name # acess methods tsvers = verso.tsprod() # set (DCVersion ~ h5py.Group) versions of calibrations tsro.del_version(vers) (int) time stamp of the version production calibdata = verso.calib() # # delete version DCVersion methods ================= (np.array) calibration array # management methods verso.set_tsprod(tsprod) # set (int) time stamp of the version production verso.add_calib(nda) # set (np.array) calibration array |
TBD
Open questions
- What is included in the detector Id version? FEE version, controller version, etc?
- when constants are updated (file open to write) they are not available... Lock to resolve.
Data flow
- who produces and supply constants
- who is allowed to submit constants
- who is allowed to access constants
- ACL inside API or using OS
- ACL for all or particular detector/type/ etc.
References
set (np.array) calibration array |
TBD
Open questions
- logic for default version
- always last added constants
- set specified version
- what to do with default if new version is added?
- when constants are updated (file open to write) they are not available... Lock to resolve.
Data flow
- who produces and supply constants
- who is allowed to submit constants
- who is allowed to access constants
- ACL inside API or using OS
- ACL for all or particular detector/type/ etc.
2016-10-27 Mtg with cpo
- table of aliases with records: <alias> <source> <begin> <end>
- repository
- <PATH>
/detector/calib
"g" or "d" - epix100a full name - 6 numbers of hardware versions, firmware version is a 7th number
- time double to two integers or int64 (32bit-sec, 32-bit-nsec)
- geometry needs to be saved as a text
- repository file has
geometry
andpixel_gain
- like constants - do not search in
<experiment>/calib
- access through network for AMI (cpo: TCP with hardwired IP addresses, m:web service/reddis)
References
- Calibration Store
- Path to data and calibration constants for psana in LCLS1
- Calibration Store
- Unix time
- ISO 8601 time format
...