short-term goal - support in a single place calibration data for detectors moving across LCLS experiments.
perspective goal long-term goal - calibration store for LCLS-II.

It is assumed that hardware Hardware configuration of modern detectors will detector can be associated with configuration index coming from DAQunique index hardwired on detector controller chip. This index will be used to recognize detector and extract hardly retrievable calibration data such as parameters for geometry, gain factors, masks etc. in case of transition of the detector from one experiment to another.

Requirements

Instrument independent - detector can be moved between instruments and hatches. / Experiment independent - detector can be moved between experiments, hatches, and instruments.
Portability - calibration data for particular detector should be portable as a self consistent file or (part of) calibration directory.
API + Interfaces - API, Command Line (CL), GUI for central management; management can ONLY be done through the API/CL/GUI in order to prevent adding/removing unknown files without history.
Time stamp (sec) - is THE ONLY value for validity range check.; the same rules are applied to the time stamp like to the run number in current calibration system.
File name - the <detname> The <detname> is equivalent to <dettype>-<detid> and is THE ONLY choice for the file name in the calibration store is <dettype>-<detid>.
Aliases to the calibration file names and specific data can be used to simplify access.
Versions - support versions of calibration data.
Links to predecessor and successor - support access to predecessor and successor if this info is available.

Architecture

To accommodate requirements calibration store Detector Calibration Store (DCS) suppose to be implemented as a combination of FS & DB file system (FS) and data base (DB) files (hdf5 for uniformity).

Calibration directory and files

Default calibration directory in LCLS FS

instrument and experiment-independent path

/reg/g/psdm/detector/calib

in contrast to the current instrument and experiment-dependent path

/reg/d/psdm/<INS>/<experiment>/calib

In local space

define calib path

<path>/calib/
assuming further structure of the calibration directory <dettype>/<dettype>-<detid>.h5

or use detector calibration file directly

<path>/<dettype>-<detid>.h5

Structure of calibration directory

Everything in lower case:

Code Block

<path>/calib/
             <dettype>/
             cspad2x2/
             pnccd/
             epix100a/
             cspad/
                       <files>
                       <aliases1>.als           # file with a map of aliases to detector names
                       ...
                       <aliasesN>.als           # one of these files can be used to map "sources" to detector names
                       <dettype>-<detid>.h5
                       <dettype>-<detid>.h5

Detector type <dettype> is used as a folder to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. Unique part of the detector name <detid> is used to assign calibration file to the particular detector version. If the detector is changing the new calibration file is created with new <detid>. Information about predecessor and successor (if available) can be saved under the root level of the calibration file. Current sources (ex: 'Camp.0:pnCCD.1' ) should be presented in the dictionary of aliases.

Calibration file structure

Calibration data hierarchical scheme

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

The scheme of the calibration data, which presumably will be implemented in <dettype>-<detid>.h5

Functionality of the DCS from bottom to top level can be listed as follows.

All calibration constants for particular detector id (detid - index of the detector version) will be kept in a single hdf5 file named as <dettype>-<detid>.h5
- Schema of this file contains a few levels which account for constant types, time stamp ranges, versions of constants.
- Beside main functionality each level contains dictionary of parameters and dictionary of history records.
These files are grouped in the directory for detector type; ex.: cspad, cspad2x2, pnccd, etc.
- The same directory contains files with aliases <aliases1>.als which map human readable detector name-and-version with appropriate <dettype>-<detid>.h5 file.
All detector type folders in regular case are collected under calib directory, although it is assumed that calibration files can be used directly.
The calib directory may be nested in three locations;
- repository - experiment-independent space with FS back-up, contains most complete calibration files in terms of time stamp and versions,
- experimental work-space - experiment-dependent space, contains squeezed version of calibration files with time stamp and versions essential for particular experiment,
- local work-space - experiment-independent any local directory, with squeezed files similar to experimental work-space.

The repository and experimental works-pace have predictable location in the file system, local work-space may have an arbitrary path. Direct file access pick up calibration file from any place.

Two-levels of interaction with DCS:
- interaction with calibration data in particular calibration file,
  - add/get/remove constants for type, time stamp, version, etc.
  - add/remove/edit metadata.
- data exchange between files between repository, experimental and local work-spaces.
  - file difference
  - transfer constants for type, time stamp, version, etc.
- - use web-service mechanism
Access Control Lists (ACL) depends on file location and is assumed to be
- repository - limited to dedicated persons
- experimental work-spaces - members of experimental group
- local work-spaces - all

Calibration directory and files

Repository

instrument and experiment-independent path to calibration directory and file

<PATH>/detector/calib/
<dettype>/<dettype>-<detid>.h5

Experiment calibration directory

experiment-dependent path to calibration directory and file

<PATH>/<INS>/<experiment>/calib/
<dettype>/<dettype>-<detid>.h5

Local calibration directory

local calib path to calibration directory and file

<path-to-calib>/calib/
<dettype>/<dettype>-<detid>.h5

Direct path to calibration file

direct path to calibration file

<path-to-calib>/<dettype>-<detid>.h5

Structure of calibration directory

Everything in lower case:

Code Block

<path-to-calib>/calib/                          # top calib directory
             <dettype>/                         # detector type folders
             cspad2x2/
             pnccd/
             epix100a/
             cspad/
                       <files>                  # level of files
                       <aliases1>.als           # file with a map of aliases to detector names
                       ...
                       <aliasesN>.als           # one of these files can be used to map "sources" to detector names
                       <dettype>-<detid>.h5     # files with detector-dependent calibration data
                       <dettype>-<detid>.h5

The calib directory is a top level DCS directory.
Detector type <dettype> folders are used to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file.
Unique part of the detector name <detid> is used to assign calibration file to the particular detector hardware configuration version. If the detector hardware configuration is changing a new calibration file is created with new <detid>.
Information about predecessor and successor (if available) can be saved under the root level of the calibration file.
Current sources (ex: 'Camp.0:pnCCD.1' ) could be presented in the dictionary of aliases.

Calibration file structure

Calibration data hierarchical scheme

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

The schema (structure) of the DCS files <dettype>-<detid>.h5

Code Block

DCStore  DCType                   DCRange         DCVersion    

/                                                                   # top-root-level contains info for the unique detector version
 dettype (str)                                                      # detector type name, ex: cspad, pnccd, etc
 detid (str)                                                        # unique detector id, ex: 01234
 tscfile (float)

Code Block

/          # time stamp of creation of this structure
 predecessor (str)                                                # top-root-level contains# infoname forof the uniqueprevious detector version if available or None
 dettypesuccessor (str)                                                    # name #of detectorthe typenext name,detector ex:if cspad,available pnccd,or etcNone
 detiddictpars (strdict)                                                    # dictionary of parameters #associated uniquewith detectorthis id,version ex:of 01234detector
 tscfilehistory (uintdict)                                                     # timedictionary stampof ofhistory creationrecords ofpaired this structure
 predecessor (str)as (tstamp:record)
 <ctype>/                                                      # name of the previous detector# iffolder availablefor orcalibration None
 successor (str)   type, ex.: pedestals, rms, mask, etc
          ctype (str)                                      # name of the next detector if available or None
# tagscalibration (dict)type name
          dictpars  (dict)                                           # dictionary of tagsparameters associated with this version of detectorcalibration type
 <ctype>/         history (dict)                                            # dictionary of history records #paired folder for calibration typeas (tstamp:record)
          ctype (str)<tstamp-range>/                                           # folder for time stamp # calibration type name
     validity range
     tags (dict)    <tstamp>[-<tstamp-end>]/                                  # folder for validity range. If <tstamp-end> is not #specified dictionary- ofthen tagsvalid associatedto withthe calibrationend
 type
          <tstamp-range>/                       tsbegin (float)                   # folder for time stamp validityfor range
the beginning of the validity range
     <tstamp>[-<tstamp-end>]/                             tsend (float)    # folder for validity range. If <tstamp-end> is not specified - then valid to the end
   # time stamp for the end of the validity range
                      tsbegin (uint)           defaultv         # time stamp for the beginning of the validity range
        # reference to the default calibration, ex:  <vers-tstamp2>
                  tsend (uint)               dictpars (dict)      # time stamp for the end of the validity range
    # dictionary of parameters associated with validity range
                       defaultv           history (dict)              # reference to the default calibration, ex:# dictionary <vers-tstamp2>
of history records paired as (tstamp:record)
                             tags (dict)    <vers-tstamp1>/                   # dictionaryfolder for ofversion tagscreated associatedon withtstamp1
 validity range
                                  <vers-tstamp1>/             tsvers (float)     # tstamp1 folderof forthis version createdproduction on tstamp1
                                                 tsverscalib (uintndarray)    # calibration #data
 tstamp1 of this version production 
                                           history (dict)     calib# (ndarray)dictionary of history records #paired calibration dataas (tstamp:record)
                                                 dictpars tags (dict)        # dictionary of tagsparameters, 
                                                                    #   ex: array size, number of dimensions, shape, data type, experiment, run, comments, author

                                  <vers <vers-tstamp2>/                   # folder for version created on tstamp2 tstamp2
                                                 tsvers (float)
                                                 tsverscalib (uintndarray)
                                                 calib history (ndarraydict)
                                                 tagsdictpars (dict)
           <tstamp2>[-<tstamp2-end>]/                               # folder for the next validity range.
 pedestals/                                                         # folder for the next calibration type, pedestals
 rms/                                                               # folder for the next calibration type, rms

...

Schema features

Detector name consists of a common part <dettype> part <dettype> and unique part <detid>.
Alias to the detector name should be kept in separate dictionary outside the file scheme.
Each detector may have optional predecessor, successor, and other tags parameters kept in the dictionary of parameters.
Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags parameters associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
The number of calibration versions for each time stamp range is not limitedunlimited. Default calibration is defined by the reference default _ version defaultv.

Metadata in

...

dictpars

Below are the lists of metadata fields which potentially can be used to define detector configuration, calibration parameters etc.

Detector metadata

Field name	Description	More details, example
dettype	detector type	CSPAD, CSPAD2X2, EPIX100A, etc.
detname	detector unique name	(if any) ex.: Camera1
detalias	alias name	if it is hard to memorize the entire name, ex.: 'cspad1'
detidx	detector index	integer number which codes the hardware version
detidxalias	symbolic alias of the index	can be used if it is hard to memorize the index integer number
detcompidx:001	list of component indexes	just in case if we are going to retrieve calibration parameters for separate components
detidxprev	detector index for previous version	detector index for previous version (if available) for the purpose of old calibration search
detidxnext	detector index for next version	detector index for next version (if available) for the purpose of new calibration search
dettsec	time-stamp	time stamp associated with beginning of the validity range for new configuration
detcom:001	comments for this hardware version	as it says {key:comment}

dettag

detpar:001

other

Calibration parameters' metadata

Field name	Description	More details, example
calibtype	calibration type	ex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc.
tsec	time stamp	beginning of the validity range
exp	original experiment	(if available) where calibration constants were obtained
runnum	original run number	(if available) where calibration constants were obtained
runbegin	begin run number	(if available) for validity range
runend	end run number	(if available) for validity range
source	original DAQ data source	data source from DAQ, ex.: 'CxiDs2.0:Cspad.0'
srcalias	data source alias	ex.: 'cspad'
calibvers	version

tag

par	in order to access using symbolic name or some alias
calibversalias	version alias	if it is hard to memorize version

tag

par
com:001	comments for this version	as it says {key:comment}

tag

par:001

other

n-d array metadata

Field name	Description	More details, example

dtype

DTYPE

(str) data type

int, float, double, etc.

ndims

NDIMS

(int) number of dimensions (N)

as it saysdim

ex.: 3
DIM:1	(int) size of dim.1	ex.

..

: 185

dim

DIM:2

(int) size of dim.2

ex.

..

: 388
...	...	...

dim

DIM:N

(int) size of dim.N

ex.

..

: 2

API

Parameters

dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

...

versind (int) - consecutive version index assigned/mapped to the version production time stamp.

Initialization

Code Block

# Import
from PSCalib.DCStore import DCStore       # inport DCStore (Detector Calibration Store) object 

# Initialization
CPATH_DEFREPO = '<PATH>/reg/g/psdm/detector/calib'  # default calib directory
cpath = '<path>/calib'                    # localdefault calib directory
cpath = '<path>/<dettype>-<detid>.h5'     # direct to file
cpathrepository

cdir = env.calibDir()                     # '/reg/d/psdm<PATH>/<INS>/<experiment>/calib' - accept current directory
path = cdir + <dettype>

detnamefname = '[path/]pnccd-12345678'                # standard name includes detector type, dash, and n-digit id number
detnamefname = '[path/]Camera1'                  # alias

cs = DCStore(fname)                       # creates a DCStore object.
""" get calibration store object
    Input parameters:
    fname [str] - file name/alias of the #detector alias

cs = DCStore(cpath, detname)              # creates a DCStore object.
""" get calibration store object
    Input parameters:
    cpath [str] - path to the hdf5 file or calibration directory. 
    
"""

If path is missing - use repository.
If calibration file is not found - throw raise IOError('File %s is not available' % fname)

UI acess methods

Code Block

ctype = pedestals # status, rms, mask, gain, bkgd, common_mode, geometry, etc
tsp = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt
vers = None # for default or versind or version time stamp.

# generic access method:
obj = cs.get(ctype, tsp, vers=None)

Base class DCBase

Is reserved to support common methods of all project classes. For now it stands for manipulations with dictpars but not limited to.

Code Block

o = DCBase()

# acess methods
dictpars = o.dictpars()            # returns If(dictpars) cpathdictionary isof adictpars pathassociated towith hdf5each fileobject
par - next parameter is ignored.
= o.par(k)                    If# cpath=Nonereturns -par defaultvalue pathfor iskey used.k
log  = o.history(fmt)              # Ifreturns cpath(str) ishistory specifiedrecords aspreceded aby paththe totime directorystamp (or default) then detname can be an alias.  
    detname [str] - name/alias of the detector 
"""

UI - acess methods

Code Block

p = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt

# generic access method:
obj = cs.get(ctype, tsp, vers)

# or specific access methods:
nda = cs.pedestals(p)
nda = cs.pixel_status(p)
nda = cs.pixel_rms(p)
...
geo = cs.geometry(p)

Base class

Is reserved to support common methods of all project classes. For now it stands for manipulations with tags but not limited to.

Code Block

DCBase - base class for this project
====================================
tags = obj.tags()default fmt='%Y-%m-%dT%H:%M:%S%Z') as a text 
d    = o.histdict()                # returns (dict) history dictionary associated with current object

# management methods
o.set_dictpars(d)                  # set (d) dictionary of parameters for object
o.add_par(k,v)                     # add (dict)k,v) par to the dictionary of tags associated with each object
tag  = obj.tag parameters for object
o.del_dictpars()                   # delete all parameters from the dictionary
o.del_par(k)                       # returnsdelete tagpar value forwith key k

objo.set_tagshistory(d)                    # set (dict) as a history dictionary of tagsthe forcurrent object
objo.add_taghistory(krec,v ts)             # add (str) record with  # add (k,v) tag(float) time stamp to the history dictionary of tags for object
obj.del_tags()                 (ts:rec). If ts is None - call current time is used as a key.

Class DCStore

Code Block

cs = DCStore(cpath, detname)    # delete all tags from the dictionary
obj.del_tag(k# (str) path to calib directory or file, (str) detector name

# acess methods
nda   = cs.get(ctype, tsp, vers)    # delete(str) tagctype with key k

Access methods

Code Block

DCStore methods
===============
tscfile- calibration type
       = cs.tscfile()               # (int) time stamp of the file creation
dettype     =# cs(...dettype() tsp - parameter to get time stamp (evt, runnum, ts_sec) 
       # (str) detector type
detid       = cs.detid()                 # (str) detector id
detnameint) vers - version of calibration, None - use default 
tscfile     = cs.detnametscfile()         # (int) time stamp of the # (str) detector name of self object
predecessor file creation
dettype     = cs.predecessordettype()           # (str) detector detnametype
detid of predecessor or None
successor   = cs.successordetid()             # (str) detname of successor or None
ctypesdetector id
detname      = cs.ctypesdetname()         # (str) detector name of self object
predecessor = cs.predecessor()     # (liststr) calibrationdetname typesof inpredecessor theor fileNone
ctosuccessor         = cs.ctypeobjsuccessor(ctype)         # (DCType ~ h5py.Group) calibration type object
_________________________________________
nda    str) detname of successor or None
ctypes      = cs.get(ctype, tsp, vers)ctypes()          # (strlist) calibration ctypetypes -in calibrationthe type
file
cto         = cs.ctypeobj(ctype)   # (DCType ~ h5py.Group) calibration type object

# management methods
cs.set_tscfile(ts)                 # set (int) time stamp of the file creation  
cs.set_dettype(dettype)     # (...) tsp - parameter to get time# stampset (evt, runnum, ts_sec) 
_________________________________________# (int) vers - version of calibration, None - use default 


DCType methods
==============
ctype       = cto.ctype(str) detector type
cs.set_detid(detid)                # set (str) detector id
cs.set_detname(detname)            #    #set (str) ofdetector ctype name
tsranges of   = cto.ranges(self object
cs.set_predecessor(pred)           #  set  # (liststr) detname of timepredecessor rangesor for ctype
tsroNone
cs.set_successor(succ)        = cto.rangeobj(tsrange)    # set # (DCRange ~ h5py.Group(str) timedetname stampof validitysuccessor rangeor object


DCRange methods
===============
tsbeginNone
cs.add_ctype(ctype)       = tsro.begin()        # add (str) calibration type to the # (intDCStore object
cs.del_ctype(ctype) time stamp beginning validity range
tsend       = tsro.end()   # delete ctype (str) from the DCStore object
cs.save(path)       # (int) time stamp ending validity range
versions    = tsro.versions()    # save current calibration in the file specified #by (listpath, ofif uint)path versionsis ofNull calibrations
versodef- update current file.

Class DCType

Code Block

cto = tsro.versdefDCType(dettype)              # (DCVersion ~ h5py.Groupstr) reference to the default version in the time-range object
versodetector type

# acess methods
ctype       = tsrocto.versobjctype(vers)           # (DCVersion ~ h5py.Group) specified version in the time-range object


DCVersion methods
=================
tsvers      = verso.tsprod()     str) of ctype name
tsranges   = cto.ranges()          # (list) of time ranges for ctype
tsro       = cto.rangeobj(tsrange) # (intDCRange ~ h5py.Group) time stamp validity of the version production
calibdatarange object

# management methods
cto.add_range(tsr)   = verso.calib()             # #add (np.arraystr) calibration array

Management methods

Code Block
DCStore methods =============== cs.set_tscfile(tsof time ranges for ctype cto.del_range(tsr) # setdelete (int)range timefrom stamp of the DCType object

Class DCRange

Code Block

tsro = DCRange(tsrangefile creation  
cs.set_dettype(dettype)            # set (str) detector type
cs.set_detid(detid)     time stamp validity range
tsro = DCRange(tsbegin, tsend)           # set (strint,int) detector id
cs.set_detname(detname)    time stamp validity range

# acess methods
tsbegin        # set (str) detector name of self object
cs.set_predecessor(pred= tsro.begin()           # set (strint) detnametime ofstamp predecessorbeginning orvalidity None
cs.set_successor(succrange
tsend       = tsro.end()             # set (strint) detnametime ofstamp successorending orvalidity None
cs.add_ctype(ctyperange
versions    = tsro.versions()      # (list of float) versions of calibrations
versodef    # add (str) calibration type to the DCStore object
cs.del_ctype(ctype)           = tsro.versdef()       # (DCVersion ~ h5py.Group) reference to the default version in the time-range object
verso     # delete ctype= tsro.versobj(strvers) from the DCStore# object
cs.save(path)          (DCVersion ~ h5py.Group) specified version in the time-range object

# management methods
tsro.set_begin(tsbegin)            # saveset current(int) calibrationtime instamp thebeginning file specified by path, if path is Null - update current file.


DCType methods
==============
cto.add_range(tsr) validity range
tsro.set_end(tsend)                # addset (strint) oftime timestamp rangesending forvalidity ctyperange
ctotsro.deladd_rangeversion(tsrvers)                 # deleteset range(DCVersion from the DCType object


DCRange methods
===============~ h5py.Group) versions of calibrations
tsro.set_beginversdef(tsbeginvers)             # set (int)DCVersion time stamp beginning validity range~ h5py.Group) versions of calibrations
tsro.setdel_endversion(tsendvers)                # set (int) time stamp ending validity range
tsro.add_versiondelete version

Class DCVersion

Code Block

verso = DCVersion(vers)             # set (DCVersionstr) ~ h5py.Group) versions of calibrations
tsro.set_versdef(vers)version name

# acess methods
tsvers      = verso.tsprod()       # set (DCVersion ~ h5py.Group) versions of calibrations
tsro.del_version(vers)    (int) time stamp of the version production
calibdata   = verso.calib()        # # delete version


DCVersion methods
=================
(np.array) calibration array

# management methods
verso.set_tsprod(tsprod)           # set (int) time stamp of the version production
verso.add_calib(nda)               # set (np.array) calibration array

TBD

Open questions

What is included in the detector Id version? FEE version, controller version, etc?
when constants are updated (file open to write) they are not available... Lock to resolve.

Data flow

who produces and supply constants
who is allowed to submit constants
who is allowed to access constants
ACL inside API or using OS
ACL for all or particular detector/type/ etc.

References

 set (np.array) calibration array

TBD

Open questions

logic for default version
- always last added constants
- set specified version
- what to do with default if new version is added?

when constants are updated (file open to write) they are not available... Lock to resolve.

Data flow

who produces and supply constants
who is allowed to submit constants
who is allowed to access constants
ACL inside API or using OS
ACL for all or particular detector/type/ etc.

2016-10-27 Mtg with cpo

table of aliases with records: <alias> <source> <begin> <end>
repository
<PATH>/detector/calib "g" or "d"
epix100a full name - 6 numbers of hardware versions, firmware version is a 7th number
time double to two integers or int64 (32bit-sec, 32-bit-nsec)
geometry needs to be saved as a text
repository file has geometry and pixel_gain - like constants
do not search in <experiment>/calib
access through network for AMI (cpo: TCP with hardwired IP addresses, m:web service/reddis)

References

...

Page tree

Page History

Versions Compared

Old Version 30

New Version Current

Key

Requirements

Architecture

Calibration directory and files

Default calibration directory in LCLS FS

In local space

Structure of calibration directory

Calibration file structure

Calibration data hierarchical scheme

Calibration directory and files

Repository

Experiment calibration directory

Local calibration directory

Direct path to calibration file

Structure of calibration directory

Calibration file structure

Calibration data hierarchical scheme

Schema features

Metadata in

dictpars

Detector metadata

Calibration parameters' metadata

n-d array metadata

API

Parameters

Initialization

UI acess methods

Base class DCBase

UI - acess methods

Base class

Class DCStore

Access methods

Class DCType

Management methods

Class DCRange

Class DCVersion

TBD

References

TBD

2016-10-27 Mtg with cpo

References