...

This document is for developing a new schema for LCLS Hdf5 files - most important of which is to provide . The main motivation is

• use a src/type vs a type/src hierarchy

...

• use DAQ aliases
• Use the name Step in place of CalibCycle

There are several other things that seem good to do as well. These have been listed below. A few alternatives were considered. If you have any comments or suggestions, please feel to use confluence to add comments to the bottom of the document, add to the document, or email me at davidsch@slac.stanford.edu.

This document is just about changing the group hierarchy and names for data. Issues like aligning data are orthogonal to this issue and not covered here.

Implementing Schema

Initially, this schema could sit alongside the current schema and use softlinks to the actual data. This would not brake anybodies code. However I am interested in developing something robust enough that people are happy with for which we could replace the previous current schema. Hence the The below schema should be readable by frameworks as well as users browsing hdf5 files.This document is just about changing the group hierarchy and names for data. Issues like aligning data are orthogonal to this issue and not covered here.

Current Schema

Here is an example of the current schema. Click on the box to expand the schema.

*** DAQ configure
/Configure:0000         
/Configure:0000/Alias::ConfigV1/Control   
/Configure:0000/Bld::BldDataEBeamV7/EBeam 
/Configure:0000/TimeTool::ConfigV2/XppEndstation.0:Opal1000.1 
/Configure:0000/Camera::FrameFexConfigV1/XppEndstation.0:Opal1000.1 
/Configure:0000/ControlData::ConfigV3/Control 
/Configure:0000/CsPad2x2::ConfigV2/XppGon.0:Cspad2x2.0 
/Configure:0000/CsPad::ConfigV5/XppGon.0:Cspad.0 
/Configure:0000/Epics::ConfigV1/EpicsArch.0:NoDevice.0 
/Configure:0000/Epics::EpicsPv/EpicsArch.0:NoDevice.0 
/Configure:0000/Epics::EpicsPv/EpicsArch.0:NoDevice.0/Attenuator_transmission {Soft Link}
/Configure:0000/Epics::EpicsPv/EpicsArch.0:NoDevice.0/BEAM:LCLS:ELEC:Q 
/Configure:0000/EvrData::ConfigV7/NoDetector.0:Evr.0 
/Configure:0000/EvrData::ConfigV7/NoDetector.0:Evr.1 
/Configure:0000/EvrData::IOConfigV2/Control 
/Configure:0000/Ipimb::ConfigV2/NH2-SB1-IPM-01 
/Configure:0000/Ipimb::ConfigV2/XppEnds_Ipm0 
/Configure:0000/L3T::ConfigV1/Event 
/Configure:0000/Lusi::IpmFexConfigV2/NH2-SB1-IPM-01 
Configure:0000/Lusi::IpmFexConfigV2/XppEnds_Ipm0 
/Configure:0000/Opal1k::ConfigV1/XppEndstation.0:Opal1000.1 
/Configure:0000/Partition::ConfigV1/Control 
*** Run/CalibCycle
/Configure:0000/Run:0000 
/Configure:0000/Run:0000/CalibCycle:0000 
/Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataEBeamV7/EBeam 
/Configure:0000/Run:0000/CalibCycle:0000/Camera::FrameV1/XppEndstation.0:Opal1000.1 
/Configure:0000/Run:0000/CalibCycle:0000/ControlData::ConfigV3/Control 
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0 
/Configure:0000/Run:0000/CalibCycle:0000/CsPad::ElementV2/XppGon.0:Cspad.0 
/Configure:0000/Run:0000/CalibCycle:0000/Epics::EpicsPv/EpicsArch.0:NoDevice.0/Attenuator_transmission {Soft Link}
/Configure:0000/Run:0000/CalibCycle:0000/Epics::EpicsPv/EpicsArch.0:NoDevice.0/BEAM:LCLS:ELEC:Q 
/Configure:0000/Run:0000/CalibCycle:0000/EvrData::ConfigV7/NoDetector.0:Evr.0 
/Configure:0000/Run:0000/CalibCycle:0000/EvrData::ConfigV7/NoDetector.0:Evr.1 
/Configure:0000/Run:0000/CalibCycle:0000/EvrData::DataV4/NoDetector.0:Evr.0 
/Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV2/NH2-SB1-IPM-01 
/Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV2/XppEnds_Ipm0 
/Configure:0000/Run:0000/CalibCycle:0000/L3T::DataV2/Event 
/Configure:0000/Run:0000/CalibCycle:0000/Lusi::IpmFexV1/NH2-SB1-IPM-01 
/Configure:0000/Run:0000/CalibCycle:0000/Lusi::IpmFexV1/XppEnds_Ipm0 
*** CalibStore
/Configure:0000/CalibStore 
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.0
/Configure:0000/CalibStore/pdscalibdata::CsPadPedestalsV1/XppGon.0:Cspad.0 

...

Note, the shared types should not show up in the translation. Psana breaks them upper-processes them and puts the sub types in the event.

Calib Store

We also need to introduce simpler type names for the calibStore types:

...

The drawback is a higher risk of a name collision (see problems below). For instance if there is both config and regular event data occurring during the event, then the Translator will try to put them in the same group. When it fails, it will have to make a messy name to distinguish them.

A new Group for EventData

Since I Config group there is a Config group, it seems like a good idea , it seems natural to group the non-config datahave a group for EventData, i.e:

/Data/Run/Step:0000/Config/UsdUsbConfig        # the config data
/Data/Run/Step:0000/EventData/UsdUsbData     # the event data

However maybe some will find this new group gets in the way.

Just starting with a Run group

Trying to have Have the hierarchy start here

/Run
/Run/Config
/Run/EpicsConfig
/Run/Step:0000
/Run/Step:0000/Config/srcAlias/UsdUsbConfig        # the config data
/Run/Step:0000/srcAlias/UsdUsbData                     # the event data

The drawback is that this is not how the xtc data is formed. In xtc files, the beginRun transition is preceded by the Configure transition. Collapsing the information from both transitions into a Run group is probably reasonable, but makes it more awkward to recover information that belonged to one xtc transition and not the other (users generally don't care about this, but it when psana reads hdf5 it is importanta framework like psana does),

Problems/Issues/Surprises

...

Presently, there should not be a collision. If one happens, it is treated as a fatal error.a collision. If one happens, it is treated as a fatal error.

They don't happen because currently there is a near 1-1 mapping between the psana event keys from which the Translator gets the data, to the group names. This mapping uses the distinct pairs of All the C++ type names are distinct, and all the DAQ sources map to distinct stringsand  DAQ source names in the event keys.

That means one can always add a new TypeName parallel to the list of existing typenamesType without colliding with existing types, as long as the Translator uses a fully qualified C++ typename for the group name.

Likewise for DAQ sources - but we currently do simplify some of these, in particular the messy sources that have a distinct ipaddress in them from each stream.

An example of a collision would be

• Daq Alias called noSrc
• user does evt.put(myndarray,'mykey')

The Translator already uses the string noSrc for user data without a source - collision.

...

With corner cases like that, users, and frameworks, may find they need to know exactly what type they are dealing with. This will be stored in hdf5 attributes to the groups.

...