Data file structure quality check
Datagram and XTC structure check
Stand-alone C++ application which uses the pdsdata
library.
Assumes the loop over all Drgams in a single XTC file/stream with iterations over all XTC containers. Retrieved information is tested for consistency, as explained below.
Datagram header records in log-file:
Datagram:1 Transition:Configure/ 4 seqtype:0 2012-02-06 15:11:23.529 fiducial:01ffff ticks:000000 pos 0x0 Datagram:2 Transition:BeginRun/ 6 seqtype:0 2012-02-06 19:47:19.529 fiducial:01ffff ticks:000000 pos 0x9b004 Datagram:3 Transition:BeginCalibCycle/ 8 seqtype:0 2012-02-06 19:47:19.653 fiducial:01ffff ticks:000000 pos 0x9b0cc Datagram:4 Transition:Enable/10 seqtype:0 2012-02-06 19:47:19.654 fiducial:01ffff ticks:000000 pos 0x9b294 Datagram:10 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:19.952 fiducial:00f6fe ticks:05092c pos 0x2f5d864 Datagram:20 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:20.452 fiducial:00f7b2 ticks:050dd2 pos 0x8cdbe58 Datagram:30 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:20.951 fiducial:00f866 ticks:050ab4 pos 0xea5d6a8 Datagram:40 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:21.451 fiducial:00f91a ticks:050cac pos 0x147dbc9c Datagram:72 Depth:0 Type:Xtc Damage bit 16: ContainsIncomplete Datagram:72 Depth:1 Type:Xtc Damage bit 16: ContainsIncomplete Datagram:72 Depth:2 Type:CspadElement Damage bit 15: IncompleteContribution Datagram:100 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:24.450 fiducial:00fd52 ticks:050d7e pos 0x378db6cc Datagram:123 Depth:0 Type:Xtc Damage bit 16: ContainsIncomplete Datagram:123 Depth:1 Type:Xtc Damage bit 16: ContainsIncomplete Datagram:123 Depth:2 Type:CspadElement Damage bit 15: IncompleteContribution Datagram:200 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:29.448 fiducial:01045a ticks:050f30 pos 0x71fdace8 Datagram:272 Depth:0 Type:Xtc Damage bit 1: DroppedContribution Datagram:272 Depth:1 Type:Xtc Damage bit 1: DroppedContribution Datagram:272 Depth:2 Type:Any Damage bit 1: DroppedContribution Datagram:284 Depth:0 Type:Xtc Damage bit 14: UserDefined
XTC header records in log-file (if needed):
Datagram:100 Transition:L1Accept/12 seqtype:0 2012-02-06 19:47:24.450 fiducial:00fd52 ticks:050d7e pos 0x378db6cc Depth:0 XTC header: dmg:0 payload:9803432 extent:9803452 Src:Level:3 Event:ProcInfo(172.21.26.100, pid=18143) TypeId:ver:0 id:1:Xtc Depth:1 XTC header: dmg:0 payload:500 extent:520 Src:Level:2 Segment:ProcInfo(172.21.26.39, pid=6099) TypeId:ver:0 id:1:Xtc Depth:2 XTC header: dmg:0 payload:32 extent:52 Src:Level:1 Source:DetInfo(CxiDg2.0:Ipimb.1) TypeId:ver:2 id:22:IpimbData Depth:2 XTC header: dmg:0 payload:28 extent:48 Src:Level:1 Source:DetInfo(CxiDg2.0:Ipimb.1) TypeId:ver:1 id:31:IpmFex Depth:2 XTC header: dmg:0 payload:32 extent:52 Src:Level:1 Source:DetInfo(CxiDg4.0:Ipimb.0) TypeId:ver:2 id:22:IpimbData Depth:2 XTC header: dmg:0 payload:28 extent:48 Src:Level:1 Source:DetInfo(CxiDg4.0:Ipimb.0) TypeId:ver:1 id:31:IpmFex Depth:2 XTC header: dmg:0 payload:32 extent:52 Src:Level:1 Source:DetInfo(CxiSc1.0:Ipimb.0) TypeId:ver:2 id:22:IpimbData Depth:2 XTC header: dmg:0 payload:28 extent:48 Src:Level:1 Source:DetInfo(CxiSc1.0:Ipimb.0) TypeId:ver:1 id:31:IpmFex Depth:2 XTC header: dmg:0 payload:32 extent:52 Src:Level:1 Source:DetInfo(CxiDg1.0:Ipimb.0) TypeId:ver:2 id:22:IpimbData Depth:2 XTC header: dmg:0 payload:28 extent:48 Src:Level:1 Source:DetInfo(CxiDg1.0:Ipimb.0) TypeId:ver:1 id:31:IpmFex Depth:2 XTC header: dmg:0 payload:32 extent:52 Src:Level:1 Source:DetInfo(CxiDg2.0:Ipimb.0) TypeId:ver:2 id:22:IpimbData Depth:2 XTC header: dmg:0 payload:28 extent:48 Src:Level:1 Source:DetInfo(CxiDg2.0:Ipimb.0) TypeId:ver:1 id:31:IpmFex Depth:1 XTC header: dmg:0 payload:0 extent:20 Src:Level:2 Segment:ProcInfo(172.21.26.26, pid=20346) TypeId:ver:0 id:1:Xtc Depth:1 XTC header: dmg:0 payload:132 extent:152 Src:Level:2 Segment:ProcInfo(172.21.26.102, pid=16207) TypeId:ver:0 id:1:Xtc Depth:2 XTC header: dmg:0 payload:32 extent:52 Src:Level:6 Reporter:BldInfo(FEEGasDetEnergy) TypeId:ver:0 id:14:FEEGasDetEnergy Depth:2 XTC header: dmg:0 payload:60 extent:80 Src:Level:6 Reporter:BldInfo(EBeam) TypeId:ver:1 id:15:EBeamBld Depth:1 XTC header: dmg:0 payload:614436 extent:614456 Src:Level:2 Segment:ProcInfo(172.21.26.26, pid=20340) TypeId:ver:0 id:1:Xtc Depth:2 XTC header: dmg:0 payload:614416 extent:614436 Src:Level:1 Source:DetInfo(CxiSc1.0:Tm6740.0) TypeId:ver:1 id:2:Frame Depth:1 XTC header: dmg:0 payload:36 extent:56 Src:Level:2 Segment:ProcInfo(172.21.26.25, pid=17284) TypeId:ver:0 id:1:Xtc Depth:2 XTC header: dmg:0 payload:16 extent:36 Src:Level:1 Source:DetInfo(NoDetector.0:Evr.0) TypeId:ver:3 id:19:EvrData Depth:1 XTC header: dmg:0 payload:0 extent:20 Src:Level:2 Segment:ProcInfo(172.21.26.27, pid=13359) TypeId:ver:0 id:1:Xtc Depth:1 XTC header: dmg:0 payload:4594084 extent:4594104 Src:Level:2 Segment:ProcInfo(172.21.26.139, pid=5224) TypeId:ver:0 id:1:Xtc Depth:2 XTC header: dmg:0 payload:4594064 extent:4594084 Src:Level:1 Source:DetInfo(CxiDs1.0:Cspad.0) TypeId:ver:2 id:28:CspadElement Depth:1 XTC header: dmg:0 payload:4594084 extent:4594104 Src:Level:2 Segment:ProcInfo(172.21.26.95, pid=3697) TypeId:ver:0 id:1:Xtc Depth:2 XTC header: dmg:0 payload:4594064 extent:4594084 Src:Level:1 Source:DetInfo(CxiDsd.0:Cspad.0) TypeId:ver:2 id:28:CspadElement
Accumulate statistics for transitions
QCStatistics::printTransIdCounter() Total number of known TransitionId : 13 Unknown : 0 Reset : 0 Map : 0 Unmap : 0 Configure : 1 Unconfigure : 0 BeginRun : 1 EndRun : 1 BeginCalibCycle : 1 EndCalibCycle : 1 Enable : 1 Disable : 1 L1Accept : 1589
In most cases the number of "Begin"-like transitions should be the same as the number of "End"-like transitions.
Check correctness of the datagram time sequence
In perfect world it should be
t(Dg1) < t(Dg2) < t(Dg3) < ... < t(DgN)
search for datadrams with wrong time sequence, like
t(Dgi) > t(Dg[i+1])
Content of each record:
struct BrokenTimeRecord { unsigned ndg; uint32_t damage_value; Pds::ClockTime time_dg; Pds::ClockTime time_dg_prev; };
Print summary for broken time records:
QCStatistics::printBrokenTimeRecords() Total number of broken time records : 100 BrokenTimeRecord: Dgram: 1 t:1969-12-31 16:00:00.997 t-previous:16:00:00.000 damage:0 BrokenTimeRecord: Dgram: 2 t:2012-02-05 10:27:28.948 t-previous:10:27:28.997 damage:0 BrokenTimeRecord: Dgram: 3 t:2012-02-05 20:12:35.074 t-previous:20:12:35.948 damage:0 ...
Size consistency of XTC containers
- Check that the XTC container size does not exceed certain maximal value.
- The total size of internal XTC containers should be consistent (and not exceed) with size of overlapping XTC container.
- Accumulate XTC container size error records; content of each record:
struct SizeErrorRecord { unsigned ndg; unsigned depth; int remaining; uint32_t damage_ext; uint32_t damage_int; Pds::TypeId::Type typeid_ext; Pds::TypeId::Type typeid_int; };
Print in summary the XTC size error records:
QCStatistics::printXTCSizeErrorRecords() Total number of size error records : 91 Size Error Record: Dgram: 1 depth:1 remaining:0 damage ext:0 damage int:0 name ext:Xtc name int:Xtc Size Error Record: Dgram: 1 depth:1 remaining:0 damage ext:0 damage int:0 name ext:Xtc name int:Xtc Size Error Record: Dgram: 1 depth:1 remaining:0 damage ext:0 damage int:0 name ext:Xtc name int:Xtc
Accumulate statistics for data types
Count occurrence of each data type in XTC containers.
QCStatistics::printTypeIdCounter() Total number of known TypeId : 58 Ind: TypeId name : Entries Vers payload Depth Src:Level Src:Names : min/max min/max : #zeros min/max min/max 0: Any : 2 0/0 0/0 : 2 2/2 6/6 BldInfo(FEEGasDetEnergy) 1: Xtc : 14365 0/1 0/9816324 : 2619 0/2 1/3 ProcInfo(172.21.26.100, pid=18143) ProcInfo(172.21.26.25, pid=17284) ProcInfo(172.21.26.39, pid=6099) ProcInfo(172.21.26.26, pid=20340) ProcInfo(172.21.26.95, pid=3697) ProcInfo(172.21.26.26, pid=20346) ProcInfo(172.21.26.102, pid=16207) DetInfo(BldEb.0:NoDevice.0) ProcInfo(172.21.26.139, pid=5224) ProcInfo(172.21.26.27, pid=13359) 2: Frame : 1589 1/1 614416/614416 : 0 2/2 1/1 DetInfo(CxiSc1.0:Tm6740.0) 3: AcqWaveform : 0 4: AcqConfig : 0 5: TwoDGaussian : 0 6: Opal1kConfig : 0 7: FrameFexConfig : 1 1/1 28/28 : 0 2/2 1/1 DetInfo(CxiSc1.0:Tm6740.0) 8: EvrConfig : 2 5/5 192/192 : 0 2/2 1/1 DetInfo(NoDetector.0:Evr.0) 9: TM6740Config : 1 2/2 12/12 : 0 2/2 1/1 DetInfo(CxiSc1.0:Tm6740.0) 10: RunControlConfig : 2 1/1 24/24 : 0 2/2 0/0 ProcInfo(0.0.0.0, pid=24604) 11: pnCCDframe : 0 12: pnCCDconfig : 0 13: Epics : 20997 1/1 24/496 : 0 2/2 1/1 DetInfo(EpicsArch.0:NoDevice.0) 14: FEEGasDetEnergy : 1588 0/0 32/32 : 0 2/3 6/6 BldInfo(FEEGasDetEnergy) 15: EBeamBld : 1590 1/1 60/60 : 0 2/3 6/6 BldInfo(EBeam) 16: PhaseCavity : 0 17: PrincetonFrame : 0 18: PrincetonConfig : 0 19: EvrData : 1589 3/3 16/16 : 0 2/2 1/1 DetInfo(NoDetector.0:Evr.0) 20: FrameFccdConfig : 0 21: FccdConfig : 0 22: IpimbData : 7945 2/2 32/32 : 0 2/2 1/1 DetInfo(CxiDg2.0:Ipimb.1) DetInfo(CxiDg4.0:Ipimb.0) DetInfo(CxiSc1.0:Ipimb.0) DetInfo(CxiDg1.0:Ipimb.0) DetInfo(CxiDg2.0:Ipimb.0) 23: IpimbConfig : 5 2/2 60/60 : 0 2/2 1/1 DetInfo(CxiDg1.0:Ipimb.0) DetInfo(CxiDg2.0:Ipimb.0) DetInfo(CxiDg2.0:Ipimb.1) DetInfo(CxiDg4.0:Ipimb.0) DetInfo(CxiSc1.0:Ipimb.0) 24: EncoderData : 0 25: EncoderConfig : 0 26: EvrIOConfig : 0 27: PrincetonInfo : 0 28: CspadElement : 3178 1/2 4594064/4594064 : 0 2/2 1/1 DetInfo(CxiDs1.0:Cspad.0) DetInfo(CxiDsd.0:Cspad.0) 29: CspadConfig : 2 3/3 287844/287844 : 0 2/2 1/1 DetInfo(CxiDsd.0:Cspad.0) DetInfo(CxiDs1.0:Cspad.0) 30: IpmFexConfig : 5 2/2 520/520 : 0 2/2 1/1 DetInfo(CxiDg1.0:Ipimb.0) DetInfo(CxiDg2.0:Ipimb.0) DetInfo(CxiDg2.0:Ipimb.1) DetInfo(CxiDg4.0:Ipimb.0) DetInfo(CxiSc1.0:Ipimb.0) 31: IpmFex : 7945 1/1 28/28 : 0 2/2 1/1 DetInfo(CxiDg2.0:Ipimb.1) DetInfo(CxiDg4.0:Ipimb.0) DetInfo(CxiSc1.0:Ipimb.0) DetInfo(CxiDg1.0:Ipimb.0) DetInfo(CxiDg2.0:Ipimb.0) 32: DiodeFexConfig : 0 33: DiodeFex : 0 34: PimImageConfig : 1 1/1 8/8 : 0 2/2 1/1 DetInfo(CxiSc1.0:Tm6740.0) 35: SharedIpimb : 0 36: AcqTDCConfig : 0 37: AcqTDCData : 0 38: Index : 0 39: XampsConfig : 0 40: XampsElement : 0 41: Cspad2x2Element : 0 42: SharedPIM : 0 43: Cspad2x2Config : 0 44: FexampConfig : 0 45: FexampElement : 0 46: Gsc16aiConfig : 0 47: Gsc16aiData : 0 48: PhasicsConfig : 0 49: TimepixConfig : 0 50: TimepixData : 0 51: CspadCompressedElement : 0 52: OceanOpticsConfig : 0 53: OceanOpticsData : 0 54: EpicsConfig : 0 55: FliConfig : 0 56: FliFrame : 0 57: QuartzConfig : 0
Accumulate statistics of damages
- Acumulate counters for
- damage bits
- damage data type
- damage bits vs data type
- Accumulate damage records; content of each record:
struct DamageRecord { unsigned ndg; unsigned depth; uint32_t damage_value; Pds::TypeId::Type type_id; };
Summary table for damage bit counters
QCStatistics::printDamageBitCounter() Damage counter Damage bit:1 : DroppedContribution 6 Damage bit:12 : OutOfOrder 0 Damage bit:13 : OutOfSynch 0 Damage bit:14 : UserDefined 4 Damage bit:15 : IncompleteContribution 9 Damage bit:16 : ContainsIncomplete 18
Summary table for damage bit vs data types counters
QCStatistics::printDamageCounter() Total number of known TypeId : 58 Ind: TypeId name : Entries DmgTot Bits:1 12 13 14 15 16 0: Any : 2 2 2 0 0 0 0 0 1: Xtc : 14365 26 4 0 0 4 0 18 2: Frame : 1589 0 0 0 0 0 0 0 3: AcqWaveform : 0 4: AcqConfig : 0 5: TwoDGaussian : 0 6: Opal1kConfig : 0 7: FrameFexConfig : 1 0 0 0 0 0 0 0 8: EvrConfig : 2 0 0 0 0 0 0 0 9: TM6740Config : 1 0 0 0 0 0 0 0 10: RunControlConfig : 2 0 0 0 0 0 0 0 11: pnCCDframe : 0 12: pnCCDconfig : 0 13: Epics : 20997 0 0 0 0 0 0 0 14: FEEGasDetEnergy : 1588 0 0 0 0 0 0 0 15: EBeamBld : 1590 0 0 0 0 0 0 0 16: PhaseCavity : 0 17: PrincetonFrame : 0 18: PrincetonConfig : 0 19: EvrData : 1589 0 0 0 0 0 0 0 20: FrameFccdConfig : 0 21: FccdConfig : 0 22: IpimbData : 7945 0 0 0 0 0 0 0 23: IpimbConfig : 5 0 0 0 0 0 0 0 24: EncoderData : 0 25: EncoderConfig : 0 26: EvrIOConfig : 0 27: PrincetonInfo : 0 28: CspadElement : 3178 9 0 0 0 0 9 0 29: CspadConfig : 2 0 0 0 0 0 0 0 30: IpmFexConfig : 5 0 0 0 0 0 0 0 31: IpmFex : 7945 0 0 0 0 0 0 0 32: DiodeFexConfig : 0 33: DiodeFex : 0 34: PimImageConfig : 1 0 0 0 0 0 0 0 35: SharedIpimb : 0 36: AcqTDCConfig : 0 37: AcqTDCData : 0 38: Index : 0 39: XampsConfig : 0 40: XampsElement : 0 41: Cspad2x2Element : 0 42: SharedPIM : 0 43: Cspad2x2Config : 0 44: FexampConfig : 0 45: FexampElement : 0 46: Gsc16aiConfig : 0 47: Gsc16aiData : 0 48: PhasicsConfig : 0 49: TimepixConfig : 0 50: TimepixData : 0 51: CspadCompressedElement : 0 52: OceanOpticsConfig : 0 53: OceanOpticsData : 0 54: EpicsConfig : 0 55: FliConfig : 0 56: FliFrame : 0 57: QuartzConfig : 0
Print in summary all damage records
QCStatistics::printDamageRecords() Total number of damage records : 33 Damage Record: Dgram: 72 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 72 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 72 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 123 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 123 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 123 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 272 depth:0 type:Xtc damage: 2 bit: 1: DroppedContribution Damage Record: Dgram: 272 depth:1 type:Xtc damage: 2 bit: 1: DroppedContribution Damage Record: Dgram: 272 depth:2 type:Any damage: 2 bit: 1: DroppedContribution Damage Record: Dgram: 284 depth:0 type:Xtc damage: 81920 bit:14: UserDefined bit:16: ContainsIncomplete Damage Record: Dgram: 284 depth:1 type:Xtc damage: 81920 bit:14: UserDefined bit:16: ContainsIncomplete Damage Record: Dgram: 284 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 332 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 332 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 332 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 479 depth:0 type:Xtc damage: 2 bit: 1: DroppedContribution Damage Record: Dgram: 479 depth:1 type:Xtc damage: 2 bit: 1: DroppedContribution Damage Record: Dgram: 479 depth:2 type:Any damage: 2 bit: 1: DroppedContribution Damage Record: Dgram: 700 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 700 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 700 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 790 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 790 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 790 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 923 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 923 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 923 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 1025 depth:0 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 1025 depth:1 type:Xtc damage: 65536 bit:16: ContainsIncomplete Damage Record: Dgram: 1025 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution Damage Record: Dgram: 1305 depth:0 type:Xtc damage: 81920 bit:14: UserDefined bit:16: ContainsIncomplete Damage Record: Dgram: 1305 depth:1 type:Xtc damage: 81920 bit:14: UserDefined bit:16: ContainsIncomplete Damage Record: Dgram: 1305 depth:2 type:CspadElement damage: 32768 bit:15: IncompleteContribution
Data quality check
psana
-based analysis for selected types of data with known properties.- This is data-specific check, which can not be universal for all data types.
Indexing?
The same procedure of the loop over datagrams with iteration over XTC containers can be used for XTC data indexing..., but need to read all files for streams and chunks within run using the same algorithm like in psana
.
Command line interface
xtcqc [-h] [-l <logfile>] [-b <basedir>] <fname1> [<fname2> [<fname3> ...]]
Optional parameters:
-h
- help, showing usage of the command line parameters.
-l
- log-file; if specified, then output will be re-directed to this file, otherwise on stdout.
-b
- base-directory, i.e.: /reg/d/psdm/<INSTRUMENT>/<experiment>/xtc
; if specified, than all xtc file names will be defined w.r.t. this directory.
Non-optional parameters:
<fname1> [<fname2> [<fname3> ...]] - all xtc file names should be listed without options.
Unpleasant feature of getopt
:
The options -l
and -b
require the file name as a trailed optional parameter. If the optional parameter is mistakenly missed, then entire input will be corrupted.
Database for XTC Quality Check
Whenever the xtc file is processed, the results are saved in the database
/reg/d/psdm/<INSTRUMENT>/<experiment>/xtcqc/<xtc-like-file-name>.qc
Web-based control and access
Igor is going to produce web-based interface, like for translator, with functionality as follows.
- Show list of files with processed/not-processed flag,
- Control on job submission for quality check,
- Browser for QC results of particular file/run.