...

• short-term goal - support in a single place calibration data for detectors moving across LCLS experiments.
• perspective goal long-term goal - calibration store for LCLS-II.

It is assumed that hardware Hardware configuration of modern detectors will detector can be associated with configuration index coming from DAQunique index hardwired on detector controller chip. This index will be used to recognize detector and extract hardly retrievable calibration data such as parameters for geometry, gain factors, masks etc. in case of transition of the detector from one experiment to another.

Requirements

• Instrument independent - detector can be moved between instruments and hatches. / Experiment independent - detector can be moved between experiments, hatches, and instruments.
• Portability - calibration data for particular detector should be portable as a self consistent file or (part of) calibration directory.
• API + Interfaces - API, Command Line (CL), GUI for central management; management can ONLY be done through the API/CL/GUI in order to prevent adding/removing unknown files without history.
• Time stamp (sec) - is THE ONLY value for validity range check; the same rules are applied to the time stamp , like for to the run number in current calibration system.
• The <detname> is equivalent to <dettype>-<detid> and is THE ONLY choice for the file name in the calibration storeFile name - the <detname> in the calibration store is <dettype>-<detid>.
• Aliases to the calibration file names and specific data can be used to simplify access.
• Support calibration versionsVersions - support versions of calibration data.
• Support links Links to predecessor and successor - support access to predecessor and successor if this info is available.

Architecture

To accommodate requirements calibration store Detector Calibration Store (DCS) suppose to be implemented as a combination of FS & DB file system (FS) and data base (DB) files (hdf5 for uniformity).

Calibration directory and files

Default calibration directory in LCLS  FS

instrument and experiment-independent path

• /reg/g/psdm/detector/calib

in contrast to the current instrument and experiment-dependent path

• /reg/d/psdm/<INS>/<experiment>/calib

In local space

define calib path

. Functionality of the DCS from bottom to top level can be listed as follows.

• All calibration constants for particular detector id (detid - index of the detector version) will be kept in a single hdf5 file named as <path>/calib/
  assuming further structure of the calibration directory <dettype>/<dettype>-<detid>.h5

  or use detector calibration file directly

  • <path>/<dettype>-<detid>.h5

  Structure of calibration directory

  Everything in lower case:

  • • Schema of this file contains a few levels which account for constant types, time stamp ranges, versions of constants.
    • Beside main functionality each level contains dictionary of parameters and dictionary of history records.
  • These files are grouped in the directory for detector type; ex.: cspad, cspad2x2, pnccd, etc.
    • The same directory contains files with aliases <aliases1>.als which map human readable detector name-and-version with appropriate <dettype>-<detid>.h5 file.
  • All detector type folders in regular case are collected under calib directory, although it is assumed that calibration files can be used directly.
  • The calib directory may be nested in three locations;
    • repository - experiment-independent space with FS back-up, contains most complete calibration files in terms of time stamp and versions,
    • experimental work-space - experiment-dependent space, contains squeezed version of calibration files with time stamp and versions essential for particular experiment,
    • local work-space - experiment-independent any local directory, with squeezed files similar to experimental work-space.

   The repository and experimental works-pace have predictable location in the file system, local work-space may have an arbitrary path. Direct file access pick up calibration file from any place.

  •  Two-levels of interaction with DCS:
    
    • interaction with calibration data in particular calibration file,
      • add/get/remove constants for type, time stamp, version, etc.
      • add/remove/edit metadata.
    • data exchange between files between repository, experimental and local work-spaces.
      • file difference
      • transfer constants for type, time stamp, version, etc.
    • • use web-service mechanism
  • Access Control Lists (ACL) depends on file location and is assumed to be
    
    • repository - limited to dedicated persons
    • experimental work-spaces - members of experimental group
    • local work-spaces - all
    
    
    

  Calibration directory and files

  Repository

  instrument and experiment-independent path to calibration directory and file

  • <PATH>/detector/calib/
                                              <dettype>/<dettype>-<detid>.h5

  Experiment calibration directory

  experiment-dependent path to calibration directory and file

  • <PATH>/<INS>/<experiment>/calib/
                                                                 <dettype>/<dettype>-<detid>.h5

  Local calibration directory

  local calib path to calibration directory and file

  • <path-to-calib>/calib/
                        <dettype>/<dettype>-<detid>.h5

  Direct path to calibration file

  direct path to calibration file

  • <path-to-calib>/<dettype>-<detid>.h5

  Structure of calibration directory

  Everything in lower case:

  Code Block
  <path-to-calib>/calib/ # top calib directory <dettype>/ # detector type folders
  Code Block
  <path>/calib/ <dettype>cspad2x2/ cspad2x2pnccd/ pnccdepix100a/ epix100acspad/ cspad/ <files> # level of <files>files <aliases1>.als # file with a map of aliases to detector names ... <aliasesN>.als # one of these files can be used to map "sources" to detector names <dettype>-<detid>.h5 # files with detector-dependent calibration data <dettype>-<detid>.h5

  
  

  • The calib directory is a top level DCS directory.
  • Detector type

  ...

  • <dettype> folders are used to organize files under the calib directory. In case of direct access to the hdf5 file the detector type is duplicated in the name of the file.
  • Unique part of the detector name <detid> is used to assign calibration file to the particular detector hardware configuration version. If the detector hardware configuration is changing

  ...

  • a new calibration file is created with new <detid>.
  • Information about predecessor and successor (if available) can be saved under the root level of the calibration file.
  • Current sources (ex: 'Camp.0:pnCCD.1' )

  ...

  • could be presented in the dictionary of aliases.

  Calibration file structure

  ...

  Access to calibration data is based on time stamp. Time stamp internally is presented in Unix time sec which can be easily converted forth and back to human readable format YYYY-MM-DDTHH:MM:SS+HH:MM

  The scheme of the calibration data, which presumably will be implemented in schema (structure) of the DCS files <dettype>-<detid>.h5

  Code Block

  DCStore DCType DCRange DCVersion / # top-root-level contains info for the unique detector version dettype (str) # detector type name, ex: cspad, pnccd, etc detid (str) # unique detector id, ex: 01234 tscfile (float) # time stamp of creation of this structure predecessor (str) # name of the previous detector if available or None successor (str) # name of the next detector if available or None tagsdictpars (dict) # dictionary of tagsparameters associated with this version of detector history (dict) # dictionary of history records paired as (tstamp:record)  <ctype>/ # folder for calibration type, ex.: pedestals, rms, mask, etc ctype (str) # calibration type name tagsdictpars (dict) # dictionary of tagsparameters associated with calibration type history (dict) # dictionary of history records paired as (tstamp:record) <tstamp-range>/ # folder for time stamp validity range <tstamp>[-<tstamp-end>]/ # folder for validity range. If <tstamp-end> is not specified - then valid to the end tsbegin (float) # time stamp for the beginning of the validity range tsend (float) # time stamp for the end of the validity range defaultv # reference to the default calibration, ex: <vers-tstamp2> tagsdictpars (dict) # dictionary of tagsparameters associated with validity range history (dict) # dictionary of history records paired as (tstamp:record) <vers-tstamp1>/ # folder for version created on tstamp1 tsvers (float) # tstamp1 of this version production calib (ndarray) # calibration data history (dict) # dictionary of history records paired as (tstamp:record)   tagsdictpars (dict) # dictionary of tagsparameters, # ex: array size, number of dimensions, shape, data type, experiment, run, comments, author  <vers-tstamp2>/ # folder for version created on tstamp2 tsvers (float) calib (ndarray)  history (dict) tagsdictpars (dict) <tstamp2>[-<tstamp2-end>]/ # folder for the next validity range. pedestals/ # folder for the next calibration type, pedestals rms/ # folder for the next calibration type, rms

  ...

  Schema features

  • Detector name consists of a common part <dettype> part <dettype> and unique part <detid>.
  • Alias to the detector name should be kept in separate dictionary outside the file scheme.
  • Each detector may have optional predecessor, successor, and other tags parameters kept in the dictionary of parameters.
  • Calibration type folders contain info about calibrations of particular type, ex: pedestals, rms, status, mask, background, etc.
  • Each calibration type contains a set of time stamp ranges defining calibration validity range. If the second time stamp of the range is missing it is considered as infinity (by the end).
  • The time stamp range folder contains tstamp_begin, tstamp_end (int) values, dictionary of tags parameters associated with this folder, reference to the default calibration, and folders with calibration versions, distinguished by there production time stamp.
  • The number of calibration versions for each time stamp range is not limitedunlimited. Default calibration is defined by the reference default _ version defaultv.

  Metadata in

  ...

  dictpars

  Below are the lists of metadata fields which potentially can be used to define detector configuration, calibration parameters etc.

  Detector metadata

  Field name	Description	More details, example
  dettype	detector type	CSPAD, CSPAD2X2, EPIX100A, etc.
  detname	detector unique name	(if any) ex.: Camera1
  detalias	alias name	if it is hard to memorize the entire name, ex.: 'cspad1'
  detidx	detector index	integer number which codes the hardware version
  detidxalias	symbolic alias of the index	can be used if it is hard to memorize the index integer number
  detcompidx:001	list of component indexes	just in case if we are going to retrieve calibration parameters for separate components
  detidxprev	detector index for previous version	detector index for previous version (if available) for the purpose of old calibration search
  detidxnext	detector index for next version	detector index for next version (if available) for the purpose of new calibration search
  dettsec	time-stamp	time stamp associated with beginning of the validity range for new configuration
  detcom:001	comments for this hardware version	as it says {key:comment}

  dettag

  detpar:001

  other

  tags

  parameters

  just in case if something is forgotten in this table

  Calibration parameters' metadata

  Field name	Description	More details, example
  calibtype	calibration type	ex.: geometry, pixel_status, pixel_gain, pedestals, common_mode, etc.
  tsec	time stamp	beginning of the validity range
  exp	original experiment	(if available) where calibration constants were obtained
  runnum	original run number	(if available) where calibration constants were obtained
  runbegin	begin run number	(if available) for validity range
  runend	end run number	(if available) for validity range
  source	original DAQ data source	data source from DAQ, ex.: 'CxiDs2.0:Cspad.0'
  srcalias	data source alias	ex.: 'cspad'
  calibvers	version

  tag

  par	in order to access using symbolic name or some alias
  calibversalias	version alias	if it is hard to memorize version

  tag

  par
  com:001	comments for this version	as it says {key:comment}

  tag

  par:001

  other

  tags

  parameters

  just in case if something is forgotten in this table

  In case of numpy array their metadata are stored with an object.
  Text file needs in n-d array metadata

  n-d array metadata

  Field name	Description	More details, example

  dtype

  DTYPE

  (str) data type

  int, float, double, etc.

  ndims

  NDIMS

  (int) number of dimensions (N)

  as it saysdim

  ex.: 3
  DIM:1	(int) size of dim.1	ex.

  ..

  : 185

  dim

  DIM:2

  (int) size of dim.2

  ex.

  ..

  : 388
  ...	...	...

  dim

  DIM:N

  (int) size of dim.N

  ex.

  ..

  : 2

  API

  Parameters

  dettype (str) - detector type: cspad, cspad2x2, pnccd, fccd, opal, epix100a, etc.

  ...

  detname (str) - detector unique name, combination of <dettype>-<detid>

  detalias (str) - detector alias name assigned to the detname

  cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.

  version (int) - time stamp of the version creation

  versind (int) - consecutive version index assigned/mapped to the version production time stamp.

   

  Initialization

  of <dettype>-<detid>

  detalias (str) - detector alias name assigned to the detname

  cpath (str) - path to the calibration directory (ex: cpath='<path>/calib') or direct hdf5 file name (ex: cpath='<path>/<detname>-<detid>.h5'). Default calibration directory CPATH_DEF='/reg/g/psdm/detector/calib'.

  version (int) - time stamp of the version creation

  versind (int) - consecutive version index assigned/mapped to the version production time stamp.

  
  

  Initialization

  Code Block
  # Import from PSCalib.DCStore import DCStore
  Code Block
  # Import from PSCalib.DCStore import DCStore # inport DCStore (Detector Calibration Store) object # Initialization CPATH_DEF = '/reg/g/psdm/detector/calib' # default calib directory cpath = '<path>/calib' # local calib directory cpath = '<path>/<dettype>-<detid>.h5' # direct to file cpath = env.calibDir() # '/reg/d/psdm/<INS>/<experiment>/calib' - accept current directory detname = 'pnccd-12345678' # standardinport nameDCStore includes(Detector detectorCalibration type, dash, and n-digit id number detnameStore) object # Initialization REPO = 'Camera1' <PATH>/detector/calib' # alias cs = DCStore(cpath, detname) # creates a DCStore object. """ get calibration store object # default calib repository cdir = env.calibDir() Input parameters: cpath [str] - path to the hdf5 file or calibration directory. # '<PATH>/<INS>/<experiment>/calib' - accept current directory path = cdir + <dettype> fname = '[path/]pnccd-12345678' # Ifstandard cpathname isincludes a path to hdf5 file - next parameter is ignored. detector type, dash, and n-digit id number fname = '[path/]Camera1' # Ifalias cs cpath=None - default path is used. DCStore(fname) If cpath is specified# ascreates a path to directory (or default) then detname can be an alias.  detnameDCStore object. """ get calibration store object Input parameters: fname [str] - file name/alias of the detector """

  • If path is missing - use repository.
  • If calibration file is not found - throw raise IOError('File %s is not available' % fname)

  UI acess methods

  Code Block
  ctype = pedestals # status, rms, mask, gain, bkgd, common_mode, geometry, etc tsp = tstamp parameter to identify constants, which can be retrieved from evt.run() - run number, evt vers = None # for default or versind or version time stamp. # generic access method: obj = cs.get(ctype, tsp, vers=None)

  Base class DCBase

  Is reserved to support common methods of all project classes. For now it stands for manipulations with tags dictpars but not limited to.

  Code Block
  o = DCBase() # acess methods dictpars = o.dictpars()
  Code Block
  DCBase - base class for this project ==================================== tags = obj.tags() # returns (dictdictpars) dictionary of tagsdictpars associated with each object tagpar = objo.tagpar(k) # returns tagpar value for key k log = objo.history(fmt) # returns (str) history records preceded by the time stamp (default fmt='%Y-%m-%dT%H:%M:%S%Z') as a text d = objo.histdict() # returns (dict) history dictionary associated with current object obj# management methods o.set_tagsdictpars(d) # set (dictd) dictionary of tagsparameters for object objo.add_tagpar(k,v) # add (k,v) tagpar to the dictionary of tagsparameters for object objo.del_tagsdictpars() # delete all tagsparameters from the dictionary objo.del_tagpar(k) # delete tagpar with key k objo.set_history(d) # set (dict) as a history dictionary of the current object objo.add_history(rec, ts) # add (str) record with (float) time stamp to the history dictionary (ts:rec). If ts is None - call current time is used as a key.

  Class DCStore

  Code Block

  cs = DCStore(cpath, detname) # (str) path to calib directory or file, (str) detector name # acess methods ===================== tscfile = cs.tscfile() nda = cs.get(ctype, tsp, vers) # (str) ctype - calibration type # (int) time stamp of the file creation dettype = cs.dettype() # (str) detector type detid # (...) tsp - = cs.detid() parameter to get time stamp (evt, runnum, ts_sec) # (str) detector id detname = cs.detname() # (str) detector name of self object predecessor = cs.predecessor() # (strint) vers - detnameversion of predecessor or None successor calibration, None - use default tscfile = cs.successortscfile() # (strint) time detnamestamp of successorthe orfile Nonecreation ctypes dettype = cs.ctypesdettype() # (list) calibration types in the file cto str) detector type detid = cs.detid() = cs.ctypeobj(ctype) # (DCTypestr) ~ h5py.Group) calibration type object ___________________________________ ndadetector id detname = cs.get(ctype, tsp, vers)detname() # (str) ctypedetector name -of calibrationself typeobject predecessor = cs.predecessor() # (str) detname of predecessor or None successor = cs.successor() # (str) detname of successor or None ctypes #= cs.ctypes(...) tsp - parameter to get time stamp (evt, runnum, ts_sec) ___________________________________ # (int) vers - version of calibration, None - use default DCStore management methods ==========================list) calibration types in the file cto = cs.ctypeobj(ctype) # (DCType ~ h5py.Group) calibration type object # management methods cs.set_tscfile(ts) # set (int) time stamp of the file creation cs.set_dettype(dettype) # set (str) detector type cs.set_detid(detid) # set (str) detector id cs.set_detname(detname) # set (str) detector name of self object cs.set_predecessor(pred) # set (str) detname of predecessor or None cs.set_successor(succ) # set (str) detname of successor or None cs.add_ctype(ctype) # add (str) calibration type to the DCStore object cs.del_ctype(ctype) # delete ctype (str) from the DCStore object cs.save(path) # save current calibration in the file specified by path, if path is Null - update current file.

  Class DCType

  Code Block

  cto = DCType(dettype) # (str) detector type # acess methods ==================== ctype = cto.ctype() # (str) of ctype name tsranges = cto.ranges() # (list) of time ranges for ctype tsro = cto.rangeobj(tsrange) # (DCRange ~ h5py.Group) time stamp validity range object DCType# management methods =========================methods cto.add_range(tsr) # add (str) of time ranges for ctype cto.del_range(tsr) # delete range from the DCType object

  Class DCRange

  Code Block

  tsro = DCRange acess methods =====================(tsrange) # (str) time stamp validity range tsro = DCRange(tsbegin, tsend) # (int,int) time stamp validity range # acess methods tsbegin = tsro.begin() # (int) time stamp beginning validity range tsend = tsro.end() # (int) time stamp ending validity range versions = tsro.versions() # (list of float) versions of calibrations versodef = tsro.versdef() # (DCVersion ~ h5py.Group) reference to the default version in the time-range object verso = tsro.versobj(vers) # (DCVersion ~ h5py.Group) specified version in the time-range object DCRange# management methods ========================== tsro.set_begin(tsbegin) # set (int) time stamp beginning validity range tsro.set_end(tsend) # set (int) time stamp ending validity range tsro.add_version(vers) # set (DCVersion ~ h5py.Group) versions of calibrations tsro.set_versdef(vers) # set (DCVersion ~ h5py.Group) versions of calibrations tsro.del_version(vers) # delete version

  Class DCVersion

  Code Block

  verso = DCVersion(vers) # (str) version acess methods =======================name # acess methods tsvers = verso.tsprod() # (int) time stamp of the version production calibdata = verso.calib() # (np.array) calibration array DCVersion# management methods ============================ verso.set_tsprod(tsprod) # set (int) time stamp of the version production verso.add_calib(nda) # set (np.array) # set (np.array) calibration array

  TBD

  Open questions

  • What is included in the detector Id version?  FEE version, controller version, etc?
  • when constants are updated (file open to write) they are not available... Lock to resolve.

  Data flow

  • who produces and supply constants
  • who is allowed to submit constants
  • who is allowed to access constants
  • ACL inside API or using OS
  • ACL for all or particular detector/type/ etc.

  References

  calibration array

  TBD

  Open questions

  • logic for default version
    • always last added constants
    • set specified version
    • what to do with default if new version is added?

  
  

  • when constants are updated (file open to write) they are not available... Lock to resolve.

  Data flow

  • who produces and supply constants
  • who is allowed to submit constants
  • who is allowed to access constants
  • ACL inside API or using OS
  • ACL for all or particular detector/type/ etc.
    
    

  2016-10-27 Mtg with cpo

  • table of aliases with records: <alias> <source> <begin> <end>
  • repository
  • <PATH>/detector/calib  "g" or "d"
  • epix100a full name  - 6 numbers of hardware versions, firmware version is a 7th number
  • time double to two integers or int64 (32bit-sec, 32-bit-nsec)
  • geometry needs to be saved as a text
  • repository file has geometry and pixel_gain - like constants
  • do not search in <experiment>/calib
  • access through network for AMI (cpo: TCP with hardwired IP addresses, m:web service/reddis)

  References

  • Calibration Store
  • Path to data and calibration constants for psana in LCLS1Calibration Store
  • Unix time
  • ISO 8601 time format

  ...