Detector type <dettype> is used as a folder to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. Unique part of the detector name <detid> is used to assign calibration file to the particular detector version. If the detector is changing the new calibration file is created with new <detid>. Information about predecessor and successor (if available) can be saved under the root level of the calibration file. Current sources (ex: 'Camp.0:pnCCD.1' ) should be presented in the dictionary of aliases.

Calibration file structure

Calibration data hierarchical scheme

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

...

Detector name consists of a common part <dettype> and unique part <detid>.
Alias to the detector name should be kept in separate dictionary outside the scheme.
Each detector may have optional predecessor, successor, and other tags in the dictionary.
Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
The number of calibration versions for each time stamp range is not limited. Default calibration is defined by the reference default_version.

*

Base class

Code Block

DCBase - base class for this project
====================================
tags = obj.tags()                  # (dict) dictionary of tags associated with each object
tag  = obj.tag(k)                  # returns tag value for key k

obj.set_tags(d)                    # set (dict) dictionary of tags for object
obj.add_tag(k,v)                   # add (k,v) tag to the dictionary of tags for object
obj.del_tags()                     # delete all tags from the dictionary
obj.del_tag(k)                     # delete tag with key k

Access methods

Data and metadata in tags

Below are the lists of fields which potentially can be used to define detector and access calibrations

Detector metadata

Field name	Description	More details, example
dettype	detector type	CSPAD, CSPAD2X2, EPIX100A, etc.
detname	detector unique name	(if any) ex.: Camera1
detalias	alias name	if it is hard to memorize the entire name, ex.: 'cspad1'
detidx	detector index	integer number which codes the hardware version
detidxalias	symbolic alias of the index	can be used if it is hard to memorize the index integer number
detcompidx:001	list of component indexes	just in case if we are going to retrieve calibration parameters for separate components
detidxprev	detector index for previous version	detector index for previous version (if available) for the purpose of old calibration search
detidxnext	detector index for next version	detector index for next version (if available) for the purpose of new calibration search
dettsec	time-stamp	time stamp associated with beginning of the validity range for new configuration
detcom:001	comments for this hardware version	as it says {key:comment}
dettag:001	other tags	just in case if something is forgotten in this table

Calibration parameters' metadata

Field name	Description	More details, example
calibtype	calibration type	ex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc.
tsec	time stamp	beginning of the validity range
exp	original experiment	(if available) where calibration constants were obtained
runnum	original run number	(if available) where calibration constants were obtained
runbegin	begin run number	(if available) for validity range
runend	end run number	(if available) for validity range
source	original DAQ data source	data source from DAQ, ex.: 'CxiDs2.0:Cspad.0'
srcalias	data source alias	ex.: 'cspad'
calibvers	version tag	in order to access using symbolic name or some alias
calibversalias	version alias	if it is hard to memorize version tag
com:001	comments for this version	as it says {key:comment}
tag:001	other tags	just in case if something is forgotten in this table

In case of numpy array their metadata are stored with an object.
Text file needs in n-d array metadata

n-d array metadata

Field name	Description	More details, example
dtype	data type	int, float, double, etc
ndims	number of dimensions (N)	as it says
dim:1	size of dim.1	...
dim:2	size of dim.2	...
...	...	...
dim:N	size of dim.N	...

API

Parameters

dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

ctype (str) - calibration type: pedestals, status, rms, mask, gain, bkgd, common_mode, geometry

detid (str) - detector unique id number, ex: 123456

detname (str) - detector unique name, combination of <dettype>-<detid>

detalias (str) - detector alias name assigned to the detname

cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.

version (int) - time stamp of the version creation

Initialization

Code Block

# Import
from PSCalib.DCStore import DCStore       # inport DCStore (Detector Calibration Store) object 

# Initialization
CPATH_DEF = '/reg/g/psdm/detector/calib'  # default calib directory
cpath = '<path>/calib'

Code Block

DCStore methods
===============
tscfile     = cs.tscfile()               # (int) time stamp of the file creation
dettype     = cs.dettype()               # (str) detector type
detid       = cs.detid()                 # (str) detector id
detname     = cs.detname()               # (str) detector name of self object
predecessor = cs.predecessor()           # (str) detname of predecessor or None
successor   = cs.successor()             # (str) detname of successor or None
ctypes      = cs.ctypes()                # (list) calibration types in the file
cto         = cs.ctypeobj(ctype)         # (DCType ~ h5py.Group) calibration type object
_________________________________________
nda         = cs.get(ctype, tsp, vers)   # (str) ctype - calibration type
                                         # (...) tsp - parameter to get time stamp (evt, runnum, ts_sec) 
_________________________________________# (int) vers - version of calibration, None - use default 


DCType methods
==============
ctype       = cto.ctype()                # (str) of ctype name
tsranges    = cto.ranges()           # local calib directory
cpath # (list) of time ranges for ctype
tsro        = cto.rangeobj(tsrange= '<path>/<dettype>-<detid>.h5'     # direct to file
cpath = env.calibDir()      # (DCRange ~ h5py.Group) time stamp validity range object


DCRange methods
===============
tsbegin     = tsro.begin()      # '/reg/d/psdm/<INS>/<experiment>/calib' - accept current directory

detname = 'pnccd-12345678'         # (int) time stamp beginning validity range
tsend # standard name includes detector type, = tsro.end()     dash, and n-digit id number
detname = 'Camera1'            # (int) time stamp ending validity range
versions    = tsro.versions()    # alias

cs = DCStore(cpath, detname)        # (list of uint) versions of calibrations
versodef# creates a DCStore = tsro.versdef()    object.
""" get calibration store object
    Input parameters:
    # (DCVersion ~ h5py.Group) referencecpath [str] - path to the defaulthdf5 versionfile inor the time-range object
versocalibration directory. 
       = tsro.versobj(vers)         # (DCVersion ~ h5py.Group) specified version in the time-range object


DCVersion methods
=================
tsversIf cpath is a path to hdf5 file - next parameter is ignored.
      = verso.tsprod()           If cpath=None #- (int)default timepath stampis ofused.
 the version production
calibdata   =  verso.calib()          If cpath is specified # (np.array) calibration array

Management methods

as a path to directory (or default) then detname can be an alias.  
    detname [str] - name/alias of the detector 
"""

UI - acess methods

Code Block

p = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt

nda = cs.get(ctype, p) # where ctype stands for pedestals, status, rms, geometry, etc.

# or specific access methods:
nda = cs.pedestals(p)
nda = cs.pixel_status(p)
nda = cs.pixel_rms(p)
nda = cs.pixel_mask(p)
nda = cs.pixel_gain(p)
nda = cs.pixel_bkgd(p)
nda = cs.common_mode(p)

geo = cs.geometry(p)

Base class

Code Block

DCBase - base class for this project
====================================
tags = obj.tags(

Code Block

DCStore methods
===============
cs.set_tscfile(ts)                 # set (int) time stamp of the file creation  
cs.set_dettype(dettype)            # set (str) detector type
cs.set_detid(detid)                # set (str) detector id
cs.set_detname(detname)            # set (str) detector name of self object
cs.set_predecessor(pred)           # set (str) detname of predecessor or None
cs.set_successor(succ)                  # set (strdict) detnamedictionary of tags successorassociated with oreach None
cs.add_ctype(ctypeobject
tag  = obj.tag(k)                  # addreturns (str)tag calibrationvalue typefor to the DCStore object
cs.del_ctype(ctypekey k

obj.set_tags(d)                    # deleteset ctype (strdict) dictionary fromof thetags DCStorefor object


DCType methods
==============
ctoobj.add_rangetag(tsrk,v)                   # add (strk,v) oftag time rangesto the dictionary of tags for ctypeobject
ctoobj.del_rangetags(tsr)                     # delete all rangetags from the DCType object


DCRange methods
===============
tsro.set_begin(tsbegindictionary
obj.del_tag(k)            # set (int) time stamp beginning validity range
tsro.set_end(tsend)   # delete tag with key k

Access methods

Code Block

DCStore methods
===============
tscfile     = cs.tscfile()               # set (int) time stamp endingof the validityfile range
tsro.add_version(vers)creation
dettype     = cs.dettype()               # set (DCVersionstr) ~ h5py.Group) versions of calibrations
tsro.set_versdef(vers)detector type
detid       = cs.detid()                 # set (DCVersion ~ h5py.Group) versions of calibrations
tsro.del_version(vers(str) detector id
detname     = cs.detname()               # delete version


DCVersion methods
=================
verso.set_tsprod(tsprod (str) detector name of self object
predecessor = cs.predecessor()           # set (intstr) timedetname stamp of thepredecessor versionor production
verso.add_calib(ndaNone
successor   = cs.successor()             #  # set (np.array) calibration array

*

Data and metadata in tags

Below are the lists of fields which potentially can be used to define detector and access calibrations

Detector metadata

Field name	Description	More details, example
dettype	detector type	CSPAD, CSPAD2X2, EPIX100A, etc.
detname	detector unique name	(if any) ex.: Camera1
detalias	alias name	if it is hard to memorize the entire name, ex.: 'cspad1'
detidx	detector index	integer number which codes the hardware version
detidxalias	symbolic alias of the index	can be used if it is hard to memorize the index integer number
detcompidx:001	list of component indexes	just in case if we are going to retrieve calibration parameters for separate components
detidxprev	detector index for previous version	detector index for previous version (if available) for the purpose of old calibration search
detidxnext	detector index for next version	detector index for next version (if available) for the purpose of new calibration search
dettsec	time-stamp	time stamp associated with beginning of the validity range for new configuration
detcom:001	comments for this hardware version	as it says {key:comment}
dettag:001	other tags	just in case if something is forgotten in this table

Calibration parameters' metadata

Field name	Description	More details, example
calibtype	calibration type	ex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc.
tsec	time stamp	beginning of the validity range
exp	original experiment	(if available) where calibration constants were obtained
runnum	original run number	(if available) where calibration constants were obtained
runbegin	begin run number	(if available) for validity range
runend	end run number	(if available) for validity range
source	original DAQ data source	data source from DAQ, ex.: 'CxiDs2.0:Cspad.0'
srcalias	data source alias	ex.: 'cspad'
calibvers	version tag	in order to access using symbolic name or some alias
calibversalias	version alias	if it is hard to memorize version tag
com:001	comments for this version	as it says {key:comment}
tag:001	other tags	just in case if something is forgotten in this table

In case of numpy array their metadata are stored with an object.
Text file needs in n-d array metadata

n-d array metadata

Field name	Description	More details, example
dtype	data type	int, float, double, etc
ndims	number of dimensions (N)	as it says
dim:1	size of dim.1	...
dim:2	size of dim.2	...
...	...	...
dim:N	size of dim.N	...

API

Parameters

dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

ctype (str) - calibration type: pedestals, status, rms, mask, gain, bkgd, common_mode, geometry

detid (str) - detector unique id number, ex: 123456

detname (str) - detector unique name, combination of <dettype>-<detid>

detalias (str) - detector alias name assigned to the detname

cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.

version (int) - time stamp of the version creation

Initialization

(str) detname of successor or None
ctypes      = cs.ctypes()                # (list) calibration types in the file
cto         = cs.ctypeobj(ctype)         # (DCType ~ h5py.Group) calibration type object
_________________________________________
nda         = cs.get(ctype, tsp, vers)   # (str) ctype - calibration type
                                         # (...) tsp - parameter to get time stamp (evt, runnum, ts_sec) 
_________________________________________# (int) vers - version of calibration, None - use default 


DCType methods
==============
ctype       = cto.ctype()                # (str) of ctype name
tsranges    = cto.ranges()               # (list) of time ranges for ctype
tsro        = cto.rangeobj(tsrange)      # (DCRange ~ h5py.Group) time stamp validity range object


DCRange methods
===============
tsbegin     = tsro.begin()               # (int) time stamp beginning validity range
tsend       = tsro.end()                 # (int) time stamp ending validity range
versions    = tsro.versions()            # (list of uint) versions of calibrations
versodef    = tsro.versdef()             # (DCVersion ~ h5py.Group) reference to the default version in the time-range object
verso       = tsro.versobj(vers)         # (DCVersion ~ h5py.Group) specified version in the time-range object


DCVersion methods
=================
tsvers      = verso.tsprod()             # (int) time stamp of the version production
calibdata   = verso.calib()              # (np.array) calibration array

Management methods

Code Block

DCStore methods
===============
cs.set_tscfile(ts)                 # set (int) time stamp of the file creation  
cs.set_dettype(dettype)            # set (str) detector type
cs.set_detid(detid)                # set (str) detector id
cs.set_detname(detname)

Code Block

# Import
from PSCalib.DCStore import DCStore       # inport DCStore (Detector Calibration Store) object 

# Initialization
CPATH_DEF = '/reg/g/psdm/detector/calib'  # default calib directory
cpath = '<path>/calib'                     # local calib directory
cpath = '<path>/<dettype>-<detid>.h5' set (str) detector name of self object
cs.set_predecessor(pred)        # direct to file
cpath# =set env.calibDir(str) detname of predecessor    or None
cs.set_successor(succ)             # '/reg/d/psdm/<INS>/<experiment>/calib' - accept current directory

detname = 'pnccd-12345678'set (str) detname of successor or None
cs.add_ctype(ctype)                # standardadd name(str) includescalibration detector type, dash,to andthe n-digit id number
detname = 'Camera1'DCStore object
cs.del_ctype(ctype)                # delete ctype (str) from the DCStore # aliasobject


cs = DCStore(cpath, detnameDCType methods
==============
cto.add_range(tsr)              # creates a DCStore object.
""" get calibration store object
    Input parameters:
    cpath [str] - path to the hdf5 file or calibration directory. 
# add (str) of time ranges for ctype
cto.del_range(tsr)                 # delete range from the DCType object


DCRange methods
===============
tsro.set_begin(tsbegin)  If cpath is a path to hdf5 file - next parameter# is ignored.
     set (int) time stamp beginning validity range
tsro.set_end(tsend)             If cpath=None - default# pathset is used.
    (int) time stamp ending validity range
tsro.add_version(vers)             # Ifset cpath(DCVersion is specified as a path to directory (or default) then detname can be an alias.  
    detname [str] - name/alias of the detector 
"""

UI - acess methods

...

~ h5py.Group) versions of calibrations
tsro.set_versdef(vers)             # set (DCVersion ~ h5py.Group) versions of calibrations
tsro.del_version(vers)             # delete version


DCVersion methods
=================
verso.set_tsprod(tsprod)           # set (int) time stamp of the version production
verso.add_calib(nda)               # set (np.array) calibration array

TBD

Open questions

What is included in the detector Id version? FEE version, controller version, etc?
when constants are updated (file open to write) they are not available... Lock to resolve.

...

Page tree

Versions Compared

Old Version 27

New Version 28

Key

Calibration file structure

Calibration data hierarchical scheme

Base class

Access methods

Data and metadata in tags

Detector metadata

Calibration parameters' metadata

n-d array metadata

API

Parameters

Initialization

Management methods

UI - acess methods

Base class

Access methods

Data and metadata in tags

Detector metadata

Calibration parameters' metadata

n-d array metadata

API

Parameters

Initialization

Management methods

UI - acess methods

TBD

Page tree

Page History

Versions Compared

Old Version 27

New Version 28

Key

Calibration file structure

Calibration data hierarchical scheme

Base class

Access methods

Data and metadata in tags

Detector metadata

Calibration parameters' metadata

n-d array metadata

API

Parameters

Initialization

Management methods

UI - acess methods

Base class

Access methods

Data and metadata in tags

Detector metadata

Calibration parameters' metadata

n-d array metadata

API

Parameters

Initialization

Management methods

UI - acess methods

TBD