Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Content

Table of Contents

Purpose

  • short-term goal - support in a single place calibration data for detectors moving across LCLS experiments.
  • perspective goal long-term goal - calibration store for LCLS-II.

It is assumed that hardware Hardware configuration of modern detectors will detector can be associated with configuration index coming from DAQunique index hardwired on detector controller chip. This index will be used to recognize detector and extract hardly retrievable calibration data such as parameters for geometry, gain factors, masks etc. in case of transition of the detector from one experiment to another.

Architecture

Requirements

  • instrument Instrument / Experiment independent - detector can be moved between instruments and hatches
  • experiment independent - detector can be moved between experiments
  • experiments, hatches, and instruments.
  • Portability - portability - calibration data for particular detector should be portable as a self consistent file
  • combination of FS & DB files (hdf5 for uniformity)
  • or (part of) calibration directory.
  • Interfaces - API, Command Line (CL), GUI for central management; management can ONLY be done through the API/CL/GUI management model - calibration data should be managed through API only in order to prevent adding/removing unknown files without history
  • API + GUI for central management
  • deployment  can ONLY be done through the API/GUI.
  • time Time stamp (sec) - is THE ONLY value for validity range check; the same rules are applied to the time stamp like to the run number in current calibration system.
  • File name - the <detname> in the calibration store is equivalent to <dettype>-<detid> and is THE ONLY choice for the official file name. This name is mapped from other aliases.

Location of calibration directory and files

Default calibration directory in LCLS  FS

instrument and experiment-independent path

  • /reg/g/psdm/detector/calib

in contrast to the current instrument and experiment-dependent path

  • /reg/d/psdm/<INS>/<experiment>/calib

In local space

define calib path

  • <path>/calib/
    assuming further structure of the calibration directory <dettype>/<dettype>-<detid>.h5

or use detector calibration file directly

  • <path>/<dettype>-<detid>.h5

Structure of calibration directory

Everything in lower case:

Code Block
<path>/calib/
             <dettype>/
             cspad2x2/
             pnccd/
             epix100a/
             cspad/
                       aliases                # file with a map of aliases to detector names
                       <calib-files>
                       <dettype>-<detid>.h5
                       <dettype>-<detid>.h5

Detector type <dettype>  is used as a folder to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file. Unique part of the detector name <detid> is used to assign calibration file to the particular detector version. If the detector is changing the new calibration file is created with new <detid>. Information about predecessor and successor (if available) can be saved under the root level of the calibration file. Current sources (ex: 'Camp.0:pnCCD.1' ) should be presented in the dictionary of aliases.

Calibration data hierarchical scheme

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

The scheme of the calibration data, which presumably will be implemented in <dettype>-<detid>.h5

  • .
  • Aliases to the calibration file names and specific data can be used to simplify access.
  • Versions - support versions of calibration data.
  • Links to predecessor and successor - support access to predecessor and successor if this info is available.

Architecture

To accommodate requirements Detector Calibration Store (DCS) suppose to be implemented as a combination of file system (FS) and data base (DB) files (hdf5 for uniformity). Functionality of the DCS from bottom to top level can be listed as follows.

  • All calibration constants for particular detector id (detid - index of the detector version) will be kept in a single hdf5 file named as <dettype>-<detid>.h5
    • Schema of this file contains a few levels which account for constant types, time stamp ranges, versions of constants.
    • Beside main functionality each level contains dictionary of parameters and dictionary of history records.
  • These files are grouped in the directory for detector type; ex.: cspad, cspad2x2, pnccd, etc.
    • The same directory contains files with aliases <aliases1>.als which map human readable detector name-and-version with appropriate <dettype>-<detid>.h5 file.
  • All detector type folders in regular case are collected under calib directory, although it is assumed that calibration files can be used directly.
  • The calib directory may be nested in three locations;
    • repository - experiment-independent space with FS back-up, contains most complete calibration files in terms of time stamp and versions,
    • experimental work-space - experiment-dependent space, contains squeezed version of calibration files with time stamp and versions essential for particular experiment,
    • local work-space - experiment-independent any local directory, with squeezed files similar to experimental work-space.

 The repository and experimental works-pace have predictable location in the file system, local work-space may have an arbitrary path. Direct file access pick up calibration file from any place.

  •  Two-levels of interaction with DCS:
    • interaction with calibration data in particular calibration file,
      • add/get/remove constants for type, time stamp, version, etc.
      • add/remove/edit metadata.
    • data exchange between files between repository, experimental and local work-spaces.
      • file difference
      • transfer constants for type, time stamp, version, etc.
      • use web-service mechanism
  • Access Control Lists (ACL) depends on file location and is assumed to be
    • repository - limited to dedicated persons
    • experimental work-spaces - members of experimental group
    • local work-spaces - all


Calibration directory and files

Repository

instrument and experiment-independent path to calibration directory and file

  • <PATH>/detector/calib/
                                              <dettype>/<dettype>-<detid>.h5

Experiment calibration directory

experiment-dependent path to calibration directory and file

  • <PATH>/<INS>/<experiment>/calib/
                                                                 <dettype>/<dettype>-<detid>.h5

Local calibration directory

local calib path to calibration directory and file

  • <path-to-calib>/calib/
                        <dettype>/<dettype>-<detid>.h5

Direct path to calibration file

direct path to calibration file

  • <path-to-calib>/<dettype>-<detid>.h5

Structure of calibration directory

Everything in lower case:

Code Block
<path-to-calib>/calib/         
Code Block
/                                                                   # top-root-level contains info for the unique detector version
 dettype (str)                 # top calib directory
             <dettype>/                     # detector type name, ex:# cspad,detector pnccd,type etcfolders
 detid (str)           cspad2x2/
             pnccd/
             epix100a/
             cspad/
      # unique detector id, ex: 01234
 tscfile (uint)          <files>                  # level of files
                      # time<aliases1>.als stamp of creation of this structure
 predecessor (str)   # file with a map of aliases to detector names
                       ...
               # name of the previous detector if available or<aliasesN>.als None
 successor (str)        # one of these files can be used to map "sources" to detector names
                         <dettype>-<detid>.h5      # name of the nextfiles with detector-dependent if available or Nonecalibration data
 tags (dict)                     <dettype>-<detid>.h5


  • The calib directory is a top level DCS directory.
  • Detector type <dettype> folders are used to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file.
  • Unique part of the detector name <detid> is used to assign calibration file to the particular detector hardware configuration version. If the detector hardware configuration is changing a new calibration file is created with new <detid>.
  • Information about predecessor and successor (if available) can be saved under the root level of the calibration file.
  • Current sources (ex: 'Camp.0:pnCCD.1' ) could be presented in the dictionary of aliases.

Calibration file structure

Calibration data hierarchical scheme

Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

The schema (structure) of the DCS files <dettype>-<detid>.h5

Code Block
DCStore  DCType                                   # dictionary of tags associated with this version of detector
 <ctype>/                                                           # folder for calibration type
          tags (dict)                                               # dictionary of tags associated with calibration type
          <tstamp-range>/                     DCRange         DCVersion    

/         # folder for time stamp validity range
          <tstamp>[-<tstamp-end>]/                                  # folder for validity range. If <tstamp-end> is not# specified top-root-level thencontains validinfo tofor the unique detector endversion
 dettype (str)                                tsbegin  (uint)                    # timedetector stamptype forname, theex: beginningcspad, ofpnccd, theetc
 validity range
detid (str)                                  tsend (uint)                      # timeunique stampdetector forid, theex: end01234
 oftscfile the(float) validity range
                                  defaultv                # time stamp of creation of this structure
 predecessor (str) # reference to the default calibration, ex:  <vers-tstamp2>
                                  tags (dict)      # name of the previous detector if available or None
 successor (str)      # dictionary of tags associated with validity range
                                  <vers-tstamp1>/     # name of the next detector if available or None
 dictpars (dict)   # folder for version created on tstamp1
                                           # dictionary of parameters associated with tsprodthis (uint)version of detector
 history (dict)   #  tstamp1 of this version production 
                                           # dictionary of history records paired calibas (ndarray)tstamp:record)
 <ctype>/    # calibration data
                                                 tags (dict)   # folder for calibration type, # dictionary of tags, 
ex.: pedestals, rms, mask, etc
          ctype (str)                                               # calibration type name
        #  dictpars ex:(dict) array size, number of dimensions, shape, data type, experiment, run, comments, author
                               # dictionary of <vers-tstamp2>/parameters associated with calibration type
          history (dict)    # folder for version created on tstamp2
                                  # dictionary of history records paired as (tstamp:record)
        tstamp_prod (uint)
  <tstamp-range>/                                           # folder for time stamp calibvalidity (ndarray)range
          <tstamp>[-<tstamp-end>]/                                  # folder for validity range. tags (dict)
 If <tstamp-end> is not specified - then valid to the end
                  <tstamp2>[-<tstamp2-end>]/                tsbegin (float)                   # foldertime stamp for the beginning nextof the validity range.
 pedestals/                                  tsend (float)                     # #time folderstamp for the nextend of calibrationthe type,validity pedestalsrange
 rms/                                 defaultv                          # reference to the # folder for the next calibration type, rms

Scheme features

  • Detector name consists of a common part <dettype> and unique part <detid>.
  • Alias to the detector name should be kept in separate dictionary outside the scheme.
  • Each detector may have optional predecessor, successor, and other tags in the dictionary.
  • Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
  • Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
  • The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
  • The number of calibration versions for each time stamp range is not limited. Default calibration is defined by the reference default_version.

*

Access methods

Code Block
DCStore methods
===============
tscfile     = cs.tscfile()default calibration, ex:  <vers-tstamp2>
                                  dictpars (dict)                   # dictionary of parameters associated with validity range
                                  history (dict)                    # dictionary of history records paired as (tstamp:record)
                                  <vers-tstamp1>/                   # folder for version created on tstamp1
                                                 tsvers (float)     # tstamp1 of this version production 
                                                 calib (ndarray)    # calibration data
                                                 history (dict)     # dictionary of history records paired as (tstamp:record)
                                                 dictpars (dict)    # dictionary of parameters, 
                                                                    #   ex: array size, number of dimensions, shape, data type, experiment, run, comments, author

                                  <vers-tstamp2>/                   # folder for version created on tstamp2
                                                 tsvers (float)
                                                 calib (ndarray)
                                                 history (dict)
                                                 dictpars (dict)
           <tstamp2>[-<tstamp2-end>]/                               # folder for the next validity range.
 pedestals/                                                         # (int)folder timefor stampthe ofnext thecalibration filetype, creation
dettypepedestals
 rms/           = cs.dettype()               # (str) detector type
detid       = cs.detid()                 # (str) detector id
detname     = cs.detname()               # (str) detector name of self object
predecessor = cs.predecessor()           # (str) detname of predecessor or None
successor   = cs.successor()             # (str) detname of successor or None
tags        = cs.tags()                  # (dict) dictionary of tags associated with this detector
ctypes      = cs.ctypes()                # (list) calibration types in the file
ctg         = cs.ctypegrp(ctype)         # (DCType ~ h5py.Group) calibration type object

DCType methods
==============
tags        = ctg.tags()                 # (dict) dictionary of tags associated with this detector
tsranges    = ctg.ranges()               # (list) of time ranges for ctype
tsrg        = ctg.rangegrp(tsrange)      # (DCRange ~ h5py.Group) time stamp validity range object


DCRange methods
===============
tags        = tsrg.tags()                # (dict) dictionary of tags associated with time stamp range
tsbegin     = tsrg.begin()               # (int) time stamp beginning validity range
tsend       = tsrg.end()                 # (int) time stamp ending validity range
versions    = tsrg.versions()            # (list) versions of calibrations
versdef     = tsrg.versdef()             # (DCVersion ~ h5py.Group) reference to the default version in the time-range group
versg       = tsrg.versgrp(vers)         # (DCVersion ~ h5py.Group) specified version in the time-range group

DCVersion methods
=================
tags        = versg.tags()               # (dict) dictionary of tags associated with time stamp range
tsversprod  = versg.tsprod()             # (int) time stamp of the version production
calibdata   = versg.calib()              # (np.array) calibration array

Management methods

# folder for the next calibration type, rms

Schema features

  • Detector name consists of a common part <dettype> and unique part <detid>.
  • Alias to the detector name should be kept in separate dictionary outside the file scheme.
  • Each detector may have optional predecessor, successor, and other parameters kept in the dictionary of parameters.
  • Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
  • Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
  • The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of parameters associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
  • The number of calibration versions for each time stamp range is unlimited. Default calibration is defined by the reference default version defaultv.

Metadata in dictpars

Below are the lists of metadata fields which potentially can be used to define detector configuration, calibration parameters etc.

Detector metadata

Field nameDescriptionMore details, example
dettypedetector typeCSPAD, CSPAD2X2, EPIX100A, etc.
detnamedetector unique name(if any) ex.: Camera1
detaliasalias nameif it is hard to memorize the entire name, ex.: 'cspad1'
detidxdetector indexinteger number which codes the hardware version
detidxaliassymbolic alias of the indexcan be used if it is hard to memorize the index integer number
detcompidx:001list of component indexesjust in case if we are going to retrieve calibration parameters for separate components
detidxprevdetector index for previous versiondetector index for previous version (if available) for the purpose of old calibration search
detidxnextdetector index for next versiondetector index for next version (if available) for the purpose of new calibration search
dettsectime-stamptime stamp associated with beginning of the validity range for new configuration
detcom:001comments for this hardware versionas it says {key:comment}
detpar:001other parametersjust in case if something is forgotten in this table

Calibration parameters' metadata

Field nameDescriptionMore details, example
calibtypecalibration typeex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc.
tsectime stampbeginning of the validity range
exporiginal experiment(if available) where calibration constants were obtained
runnumoriginal run number(if available) where calibration constants were obtained
runbeginbegin run number(if available) for validity range
runendend run number(if available) for validity range
sourceoriginal DAQ data sourcedata source from DAQ, ex.: 'CxiDs2.0:Cspad.0'
srcaliasdata source aliasex.: 'cspad'
calibversversion parin order to access using symbolic name or some alias
calibversaliasversion aliasif it is hard to memorize version par
com:001comments for this versionas it says {key:comment}
par:001other parametersjust in case if something is forgotten in this table

In case of numpy array their metadata are stored with an object.
Text file needs in n-d array metadata

n-d array metadata

Field nameDescriptionMore details, example
DTYPE(str) data typeint, float, double, etc.
NDIMS(int) number of dimensions (N)ex.: 3
DIM:1(int) size of dim.1ex.: 185
DIM:2(int) size of dim.2ex.: 388
.........
DIM:N(int) size of dim.Nex.: 2

API

Parameters

dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

ctype (str) - calibration type: pedestals, status, rms, mask, gain, bkgd, common_mode, geometry

detid (str) - detector unique id number, ex: 123456

detname (str) - detector unique name, combination of <dettype>-<detid>

detalias (str) - detector alias name assigned to the detname

cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.

version (int) - time stamp of the version creation

versind (int) - consecutive version index assigned/mapped to the version production time stamp.


Initialization

Code Block
# Import
from PSCalib.DCStore import DCStore       # inport DCStore (Detector Calibration Store) object 

# Initialization
REPO = '<PATH>/detector/calib'
Code Block
DCStore methods
===============
cs.set_tscfile(ts)                 # set (int) time stamp of the file creation  
cs.set_dettype(dettype)            # set (str) detector type
cs.set_detid(detid)                # set (str) detector id
cs.set_detname(detname)            # setdefault (str) detector name of self object
cs.set_predecessor(predcalib repository

cdir = env.calibDir()           # set (str) detname of predecessor or None
cs.set_successor(succ)   # '<PATH>/<INS>/<experiment>/calib' - accept   current directory
path = cdir # set (str) detname of successor or None
cs.set_tags(d)+ <dettype>

fname = '[path/]pnccd-12345678'           # standard name includes detector type, dash, and n-digit id number
fname = '[path/]Camera1'  # set (dict) dictionary of tags associated with this detector
cs.add_tag(k,v)       # alias

cs = DCStore(fname)          # add (k,v) add tag to the dictionary of tags associated with this detector
cs.add_ctype(ctype)         # creates a DCStore object.
""" get calibration store object
    Input parameters:
   # addfname ([str)] - calibrationfile typename/alias toof the DCStore object
cs.del_tags()                      # delete tags
cs.del_tag(k)                      # delete tag with key k
cs.del_ctype(ctype)                # delete ctype from the DCStore object


DCType methods
==============
ctg.set_tags(d)      detector 
"""
  • If path is missing - use repository.
  • If calibration file is not found - throw raise IOError('File %s is not available' % fname)

UI acess methods

Code Block
ctype = pedestals # status, rms, mask, gain, bkgd, common_mode, geometry, etc
tsp = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt
vers = None # for default or versind or version time stamp.

# generic access method:
obj = cs.get(ctype, tsp, vers=None)

Base class DCBase

Is reserved to support common methods of all project classes. For now it stands for manipulations with dictpars but not limited to.

Code Block
o = DCBase()

# acess methods
dictpars = o.dictpars()              # setreturns (dictdictpars) dictionary of tagsdictpars associated forwith DCTypeeach object
ctg.add_tagpar  = o.par(k,v)                    # add (k,v) tag to the dictionary of tags for DCType object
ctg.add_range(tsr)  returns par value for key k
log  = o.history(fmt)               # addreturns (str) ofhistory timerecords rangespreceded for ctype
ctg.del_tags()                     # delete tags
ctg.del_tag(kby the time stamp (default fmt='%Y-%m-%dT%H:%M:%S%Z') as a text 
d    = o.histdict()                # returns (dict) history dictionary #associated deletewith tag with key k
ctg.del_range(tsr)current object

# management methods
o.set_dictpars(d)                  # set delete(d) ctypedictionary fromof theparameters DCTypefor object


DCRange methods
===============
tsrg.set_tags(do.add_par(k,v)                     # setadd (dictk,v) dictionarypar ofto tagsthe associateddictionary withof timeparameters stampfor rangeobject
tsrgo.adddel_tagdictpars(k,v)                   # add delete all parameters from the dictionary
o.del_par(k,v) tag to the dictionary of tags for DCType object
tsrg.set_begin(tsbegin)                      # delete par with key k
o.set_history(d)                   # set (int) time stamp beginning validity range
tsrg.set_end(tsenddict) as a history dictionary of the current object
o.add_history(rec, ts)             # add (str) #record setwith (intfloat) time stamp to the history dictionary (ts:rec). If ts is None - call current time is used as a key. 

Class DCStore

Code Block
cs = DCStore(cpath, detnameending validity range
tsrg.add_version(vers)             # set (DCVersion ~ h5py.Group) versions of calibrations
tsrg.set_versdef(vers)       # (str) path to calib directory #or setfile, (DCVersionstr) ~ h5py.Group) versions of calibrations
tsrg.del_version(vers)detector name

# acess methods
nda       = cs.get(ctype, tsp, vers)   # (str) ctype #- deletecalibration version
tsrg.del_tags()type
                    # delete tags
tsrg.del_tag(k)             # (...) tsp - parameter to get #time deletestamp tag with key k


DCVersion methods
=================
versg.set_tags(d)(evt, runnum, ts_sec) 
                                   # set (dictint) vers - dictionaryversion of tags associated with DCVersion object
versg.add_tag(k,v)calibration, None - use default 
tscfile     = cs.tscfile()           # add (k,vint) tagtime stamp toof the file creation
dettype dictionary of tags for DCVersion= object
versg.set_tsprod(tsprodcs.dettype()         #  # set (int) time stamp of the version production
versg.add_calib(nda(str) detector type
detid       = cs.detid()           # (str) detector id
detname #  set (np.array) calibration= array
versg.del_tagscs.detname()         # (str) detector   name of self object
predecessor = cs.predecessor()     # delete tags
versg.del_tag(k(str) detname of predecessor or None
successor   = cs.successor()          # delete(str) tagdetname with key k

*

 

Data and metadata in tags

Below are the lists of fields which potentially can be used to define detector and access calibrations

Detector metadata

Field nameDescriptionMore details, example
dettypedetector typeCSPAD, CSPAD2X2, EPIX100A, etc.
detnamedetector unique name(if any) ex.: Camera1
detaliasalias nameif it is hard to memorize the entire name, ex.: 'cspad1'
detidxdetector indexinteger number which codes the hardware version
detidxaliassymbolic alias of the indexcan be used if it is hard to memorize the index integer number
detcompidx:001list of component indexesjust in case if we are going to retrieve calibration parameters for separate components
detidxprevdetector index for previous versiondetector index for previous version (if available) for the purpose of old calibration search
detidxnextdetector index for next versiondetector index for next version (if available) for the purpose of new calibration search
dettsectime-stamptime stamp associated with beginning of the validity range for new configuration
detcom:001comments for this hardware versionas it says {key:comment}
dettag:001other tagsjust in case if something is forgotten in this table

Calibration parameters' metadata

Field nameDescriptionMore details, example
calibtypecalibration typeex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc.
tsectime stampbeginning of the validity range
exporiginal experiment(if available) where calibration constants were obtained
runnumoriginal run number(if available) where calibration constants were obtained
runbeginbegin run number(if available) for validity range
runendend run number(if available) for validity range
sourceoriginal DAQ data sourcedata source from DAQ, ex.: 'CxiDs2.0:Cspad.0'
srcaliasdata source aliasex.: 'cspad'
calibversversion tagin order to access using symbolic name or some alias
calibversaliasversion aliasif it is hard to memorize version tag
com:001comments for this versionas it says {key:comment}
tag:001other tagsjust in case if something is forgotten in this table

In case of numpy array their metadata are stored with an object.
Text file needs in n-d array metadata

n-d array metadata

Field nameDescriptionMore details, example
dtypedata typeint, float, double, etc
ndimsnumber of dimensions (N)as it says
dim:1size of dim.1...
dim:2size of dim.2...
.........
dim:Nsize of dim.N...

API

Parameters

dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

ctype (str) - calibration type: pedestals, status, rms, mask, gain, bkgd, common_mode, geometry

detid (str) - detector unique id number, ex: 123456

detname (str) - detector unique name, combination of <dettype>-<detid>

detalias (str) - detector alias name assigned to the detname

cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.

 

 

Initialization

of successor or None
ctypes      = cs.ctypes()          # (list) calibration types in the file
cto         = cs.ctypeobj(ctype)   # (DCType ~ h5py.Group) calibration type object

# management methods
cs.set_tscfile(ts)                 # set (int) time stamp of the file creation  
cs.set_dettype(dettype)            # set (str) detector type
cs.set_detid(detid)                # set (str) detector id
cs.set_detname(detname)            # set (str) detector name of self object
cs.set_predecessor(pred)           # set (str) detname of predecessor or None
cs.set_successor(succ)             # set (str) detname of successor or None
cs.add_ctype(ctype)                # add (str) calibration type to the DCStore object
cs.del_ctype(ctype)                # delete ctype (str) from the DCStore object
cs.save(path)                      # save current calibration in the file specified by path, if path is Null - update current file.

Class DCType

Code Block
cto = DCType(dettype)              # (str) detector type

# acess methods
ctype      = cto.ctype()           # (str) of ctype name
tsranges   = cto.ranges()          # (list) of time ranges for ctype
tsro       = cto.rangeobj(tsrange) # (DCRange ~ h5py.Group) time stamp validity range object

# management methods
cto.add_range(tsr)          
Code Block
# Import
from PSCalib.DCStore import DCStore       # inportadd DCStore (Detectorstr) Calibrationof Store)time objectranges 

#for Initializationctype
CPATH_DEF = '/reg/g/psdm/detector/calib'  # default calib directory
cpath = '<path>/calib'cto.del_range(tsr)                 # delete range from the DCType object

Class DCRange

Code Block
tsro = DCRange(tsrange)  # local calib directory
cpath = '<path>/<dettype>-<detid>.h5'     # direct to file
cpath = env.calibDir()     (str) time stamp validity range
tsro = DCRange(tsbegin, tsend)     # (int,int) time stamp validity range

# acess methods
tsbegin     = tsro.begin()         # '/reg/d/psdm/<INS>/<experiment>/calib' - accept current directory

detname = 'pnccd-12345678'(int) time stamp beginning validity range
tsend       = tsro.end()           # standard(int) nametime includesstamp detectorending type, dash, and n-digit id number
detname = 'Camera1'    validity range
versions    = tsro.versions()      # (list of float) versions of calibrations
versodef  # alias

cs = DCStore(cpath, detnametsro.versdef()       # (DCVersion ~ h5py.Group) reference to the #default createsversion ain DCStore object.
""" get calibration store object
the time-range object
verso    Input parameters:
  = tsro.versobj(vers) cpath [str] -# path(DCVersion to the hdf5 file or calibration directory. 
  ~ h5py.Group) specified version in the time-range object

# management methods
tsro.set_begin(tsbegin)            # set (int) time Ifstamp cpathbeginning is a path to hdf5 file - next parameter is ignored.
validity range
tsro.set_end(tsend)                # set (int) time stamp ending    validity range
tsro.add_version(vers)  If cpath=None - default path is used.
     # set (DCVersion ~ h5py.Group) versions of calibrations
tsro.set_versdef(vers)      If cpath is specified as a path to# directoryset (or defaultDCVersion ~ h5py.Group) thenversions detname can be an alias.  
of calibrations
tsro.del_version(vers)         detname [str] - name/alias of the# detectordelete version
"""

...

Class DCVersion

Code Block
pverso = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt

nda = cs.get(ctype, p) # where ctype stands for pedestals, status, rms, geometry, etc.

# or specific access methods:
nda = cs.pedestals(p)
nda = cs.pixel_status(p)
nda = cs.pixel_rms(p)
nda = cs.pixel_mask(p)
nda = cs.pixel_gain(p)
nda = cs.pixel_bkgd(p)
nda = cs.common_mode(p)

geo = cs.geometry(p)

 

TBD

Open questions

  • What is included in the detector Id version?  FEE version, controller version, etc?
  • when constants are updated (file open to write) they are not available... Lock to resolve.

Data flow

  • who produces and supply constants
  • who is allowed to submit constants
  • who is allowed to access constants
  • ACL inside API or using OS
  • ACL for all or particular detector/type/ etc.

References

 DCVersion(vers)            # (str) version name

# acess methods
tsvers      = verso.tsprod()       # (int) time stamp of the version production
calibdata   = verso.calib()        # (np.array) calibration array

# management methods
verso.set_tsprod(tsprod)           # set (int) time stamp of the version production
verso.add_calib(nda)               # set (np.array) calibration array

TBD

Open questions

  • logic for default version
    • always last added constants
    • set specified version
    • what to do with default if new version is added?


  • when constants are updated (file open to write) they are not available... Lock to resolve.

Data flow

  • who produces and supply constants
  • who is allowed to submit constants
  • who is allowed to access constants
  • ACL inside API or using OS
  • ACL for all or particular detector/type/ etc.

2016-10-27 Mtg with cpo

  • table of aliases with records: <alias> <source> <begin> <end>
  • repository
  • <PATH>/detector/calib  "g" or "d"
  • epix100a full name  - 6 numbers of hardware versions, firmware version is a 7th number
  • time double to two integers or int64 (32bit-sec, 32-bit-nsec)
  • geometry needs to be saved as a text
  • repository file has geometry and pixel_gain - like constants
  • do not search in <experiment>/calib
  • access through network for AMI (cpo: TCP with hardwired IP addresses, m:web service/reddis)

References

...