Data file structure quality check

Datagram and XTC structure check

Stand-alone C++ application which uses the pdsdata library.
Assumes the loop over all Drgams in a single XTC file/stream with iterations over all XTC containers. Retrieved information is tested for consistency, as explained below.

Datagram header records in log-file:

Datagram:1    Transition:Configure       /4  seqtype:0 2012-02-05 10:27:28.997 fiducial:1ffff  ticks:0      pos:0
Datagram:2    Transition:BeginRun        /6  seqtype:0 2012-02-05 20:12:35.948 fiducial:1ffff  ticks:0      pos:9b004
Datagram:3    Transition:BeginCalibCycle /8  seqtype:0 2012-02-05 20:12:36.74  fiducial:1ffff  ticks:0      pos:9b0cc
Datagram:4    Transition:Enable          /10 seqtype:0 2012-02-05 20:12:36.75  fiducial:1ffff  ticks:0      pos:9b294
Datagram:5    Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.108 fiducial:985d   ticks:50f06  pos:9b35c
Datagram:6    Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.158 fiducial:986f   ticks:50d7e  pos:98ea30
Datagram:7    Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.208 fiducial:9881   ticks:50b16  pos:127eea8
Datagram:8    Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.258 fiducial:9893   ticks:50e5e  pos:1b6f320
Datagram:9    Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.308 fiducial:98a5   ticks:50a36  pos:245f834
Datagram:10   Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.358 fiducial:98b7   ticks:50f06  pos:2d4fcac
Datagram:20   Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:36.858 fiducial:996b   ticks:50f3e  pos:86b2b30
Datagram:30   Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:37.358 fiducial:9a1f   ticks:50e18  pos:e018c10
Datagram:40   Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:37.858 fiducial:9ad3   ticks:50980  pos:1397ba94
Datagram:100  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:40.858 fiducial:9f0b   ticks:50ec0  pos:351d68c0
Datagram:200  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:45.858 fiducial:a613   ticks:50a60  pos:6cfc3518
Datagram:300  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:50.858 fiducial:ad1b   ticks:50f5a  pos:a4db020c
Datagram:400  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:12:55.859 fiducial:b423   ticks:50b08  pos:dcb99d40
Datagram:500  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:13:00.860 fiducial:bb2b   ticks:50956  pos:14986a34
Datagram:600  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:13:05.861 fiducial:c233   ticks:50bbe  pos:4c773728
Datagram 697    Damage Depth:0 Type:Xtc  bit:1 : DroppedContribution
Datagram 697    Damage Depth:1 Type:Xtc  bit:1 : DroppedContribution
Datagram 697    Damage Depth:2 Type:Any  bit:1 : DroppedContribution
Datagram 698    Damage Depth:0 Type:Xtc  bit:1 : DroppedContribution
Datagram 698    Damage Depth:1 Type:Xtc  bit:1 : DroppedContribution
Datagram 698    Damage Depth:2 Type:Any  bit:1 : DroppedContribution
Datagram:700  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:13:10.863 fiducial:c93b   ticks:508a0  pos:845603dc
Datagram:800  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:13:15.864 fiducial:d043   ticks:50a0c  pos:bc34d0d0
Datagram:900  Transition:L1Accept        /12 seqtype:0 2012-02-05 20:13:20.865 fiducial:d74b   ticks:50b5c  pos:f4136c04
Datagram:1000 Transition:L1Accept        /12 seqtype:0 2012-02-05 20:13:25.866 fiducial:de53   ticks:50b32  pos:2bf23994

XTC header records in log-file (if needed):

Datagram:100  Transition:L1Accept/12  seqtype:0  2012-02-06 19:47:24.450  fiducial:00fd52  ticks:050d7e  pos 0x378db6cc 
Depth:0  XTC header: dmg:0  payload:9803432  extent:9803452  Src:Level:3  Event:ProcInfo(172.21.26.100, pid=18143)  TypeId:ver:0 id:1:Xtc
Depth:1  XTC header: dmg:0  payload:500  extent:520  Src:Level:2  Segment:ProcInfo(172.21.26.39, pid=6099)  TypeId:ver:0 id:1:Xtc
Depth:2  XTC header: dmg:0  payload:32  extent:52  Src:Level:1  Source:DetInfo(CxiDg2.0:Ipimb.1)  TypeId:ver:2 id:22:IpimbData
Depth:2  XTC header: dmg:0  payload:28  extent:48  Src:Level:1  Source:DetInfo(CxiDg2.0:Ipimb.1)  TypeId:ver:1 id:31:IpmFex
Depth:2  XTC header: dmg:0  payload:32  extent:52  Src:Level:1  Source:DetInfo(CxiDg4.0:Ipimb.0)  TypeId:ver:2 id:22:IpimbData
Depth:2  XTC header: dmg:0  payload:28  extent:48  Src:Level:1  Source:DetInfo(CxiDg4.0:Ipimb.0)  TypeId:ver:1 id:31:IpmFex
Depth:2  XTC header: dmg:0  payload:32  extent:52  Src:Level:1  Source:DetInfo(CxiSc1.0:Ipimb.0)  TypeId:ver:2 id:22:IpimbData
Depth:2  XTC header: dmg:0  payload:28  extent:48  Src:Level:1  Source:DetInfo(CxiSc1.0:Ipimb.0)  TypeId:ver:1 id:31:IpmFex
Depth:2  XTC header: dmg:0  payload:32  extent:52  Src:Level:1  Source:DetInfo(CxiDg1.0:Ipimb.0)  TypeId:ver:2 id:22:IpimbData
Depth:2  XTC header: dmg:0  payload:28  extent:48  Src:Level:1  Source:DetInfo(CxiDg1.0:Ipimb.0)  TypeId:ver:1 id:31:IpmFex
Depth:2  XTC header: dmg:0  payload:32  extent:52  Src:Level:1  Source:DetInfo(CxiDg2.0:Ipimb.0)  TypeId:ver:2 id:22:IpimbData
Depth:2  XTC header: dmg:0  payload:28  extent:48  Src:Level:1  Source:DetInfo(CxiDg2.0:Ipimb.0)  TypeId:ver:1 id:31:IpmFex
Depth:1  XTC header: dmg:0  payload:0  extent:20  Src:Level:2  Segment:ProcInfo(172.21.26.26, pid=20346)  TypeId:ver:0 id:1:Xtc
Depth:1  XTC header: dmg:0  payload:132  extent:152  Src:Level:2  Segment:ProcInfo(172.21.26.102, pid=16207)  TypeId:ver:0 id:1:Xtc
Depth:2  XTC header: dmg:0  payload:32  extent:52  Src:Level:6  Reporter:BldInfo(FEEGasDetEnergy)  TypeId:ver:0 id:14:FEEGasDetEnergy
Depth:2  XTC header: dmg:0  payload:60  extent:80  Src:Level:6  Reporter:BldInfo(EBeam)  TypeId:ver:1 id:15:EBeamBld
Depth:1  XTC header: dmg:0  payload:614436  extent:614456  Src:Level:2  Segment:ProcInfo(172.21.26.26, pid=20340)  TypeId:ver:0 id:1:Xtc
Depth:2  XTC header: dmg:0  payload:614416  extent:614436  Src:Level:1  Source:DetInfo(CxiSc1.0:Tm6740.0)  TypeId:ver:1 id:2:Frame
Depth:1  XTC header: dmg:0  payload:36  extent:56  Src:Level:2  Segment:ProcInfo(172.21.26.25, pid=17284)  TypeId:ver:0 id:1:Xtc
Depth:2  XTC header: dmg:0  payload:16  extent:36  Src:Level:1  Source:DetInfo(NoDetector.0:Evr.0)  TypeId:ver:3 id:19:EvrData
Depth:1  XTC header: dmg:0  payload:0  extent:20  Src:Level:2  Segment:ProcInfo(172.21.26.27, pid=13359)  TypeId:ver:0 id:1:Xtc
Depth:1  XTC header: dmg:0  payload:4594084  extent:4594104  Src:Level:2  Segment:ProcInfo(172.21.26.139, pid=5224)  TypeId:ver:0 id:1:Xtc
Depth:2  XTC header: dmg:0  payload:4594064  extent:4594084  Src:Level:1  Source:DetInfo(CxiDs1.0:Cspad.0)  TypeId:ver:2 id:28:CspadElement
Depth:1  XTC header: dmg:0  payload:4594084  extent:4594104  Src:Level:2  Segment:ProcInfo(172.21.26.95, pid=3697)  TypeId:ver:0 id:1:Xtc
Depth:2  XTC header: dmg:0  payload:4594064  extent:4594084  Src:Level:1  Source:DetInfo(CxiDsd.0:Cspad.0)  TypeId:ver:2 id:28:CspadElement

Accumulate statistics for transitions

=========================  
  Quality check summary
=========================  
Total number of dgrams=13743

QCStatistics::printTransIdStatistics()
Total number of known TransitionId : 13
Unknown         : 0         Reset           : 0         
Map             : 0         Unmap           : 0         
Configure       : 1         Unconfigure     : 0         
BeginRun        : 1         EndRun          : 1         
BeginCalibCycle : 1         EndCalibCycle   : 1         
Enable          : 2         Disable         : 2         
L1Accept        : 13734     

In most cases the number of "Begin"-like transitions should be the same as the number of "End"-like transitions.

Check correctness of the datagram time sequence

In perfect world it should be

t(Dg1) < t(Dg2) < t(Dg3) < ... < t(DgN)

search for datadrams with wrong time sequence, like

t(Dgi) > t(Dg[i+1])

Content of each record:

struct BrokenTimeRecord {
  unsigned       ndg;
  uint32_t       damage_value;
  Pds::ClockTime time_dg;
  Pds::ClockTime time_dg_prev;
};

Print summary for broken time records:

QCStatistics::printBrokenTimeRecords()
Total number of broken time records : 100
BrokenTimeRecord: Dgram:     1  t:1969-12-31 16:00:00.997  t-previous:16:00:00.000  damage:0
BrokenTimeRecord: Dgram:     2  t:2012-02-05 10:27:28.948  t-previous:10:27:28.997  damage:0
BrokenTimeRecord: Dgram:     3  t:2012-02-05 20:12:35.074  t-previous:20:12:35.948  damage:0
...

Size consistency of XTC containers

  • Check that the XTC container size does not exceed certain maximal value.
  • The total size of internal XTC containers should be consistent (and not exceed) with size of overlapping XTC container.
  • Accumulate XTC container size error records; content of each record:
    struct SizeErrorRecord {
      unsigned ndg;
      unsigned depth;
      int      remaining;
      uint32_t damage_ext;
      uint32_t damage_int;
      Pds::TypeId::Type typeid_ext;
      Pds::TypeId::Type typeid_int;
    };
    

Print in summary the XTC size error records:

QCStatistics::printXTCSizeErrorRecords()
Total number of size error records : 91
Size Error Record: Dgram:     1  depth:1  remaining:0  damage ext:0  damage int:0  name ext:Xtc  name int:Xtc
Size Error Record: Dgram:     1  depth:1  remaining:0  damage ext:0  damage int:0  name ext:Xtc  name int:Xtc
Size Error Record: Dgram:     1  depth:1  remaining:0  damage ext:0  damage int:0  name ext:Xtc  name int:Xtc

Accumulate statistics for data types

Count occurrence of each data type in XTC containers.

QCStatistics::printTypeIdCounter()
Total number of known TypeId : 58
Ind: TypeId name             : Entries Vers       payload                 Depth     Src:Level    Src:Names
                             :         min/max    min/max     : #zeros    min/max   min/max 
  0: Any                     : 2         0/0        0/0       : 2           2/2       6/6        BldInfo(FEEGasDetEnergy)
  1: Xtc                     : 14365     0/1        0/9816324 : 2619        0/2       1/3        ProcInfo(172.21.26.100, pid=18143)
                                                                                                 ProcInfo(172.21.26.25, pid=17284)
                                                                                                 ProcInfo(172.21.26.39, pid=6099)
                                                                                                 ProcInfo(172.21.26.26, pid=20340)
                                                                                                 ProcInfo(172.21.26.95, pid=3697)
                                                                                                 ProcInfo(172.21.26.26, pid=20346)
                                                                                                 ProcInfo(172.21.26.102, pid=16207)
                                                                                                 DetInfo(BldEb.0:NoDevice.0)
                                                                                                 ProcInfo(172.21.26.139, pid=5224)
                                                                                                 ProcInfo(172.21.26.27, pid=13359)
  2: Frame                   : 1589      1/1   614416/614416  : 0           2/2       1/1        DetInfo(CxiSc1.0:Tm6740.0)
  3: AcqWaveform             : 0     
  4: AcqConfig               : 0     
  5: TwoDGaussian            : 0     
  6: Opal1kConfig            : 0     
  7: FrameFexConfig          : 1         1/1       28/28      : 0           2/2       1/1        DetInfo(CxiSc1.0:Tm6740.0)
  8: EvrConfig               : 2         5/5      192/192     : 0           2/2       1/1        DetInfo(NoDetector.0:Evr.0)
  9: TM6740Config            : 1         2/2       12/12      : 0           2/2       1/1        DetInfo(CxiSc1.0:Tm6740.0)
 10: RunControlConfig        : 2         1/1       24/24      : 0           2/2       0/0        ProcInfo(0.0.0.0, pid=24604)
 11: pnCCDframe              : 0     
 12: pnCCDconfig             : 0     
 13: Epics                   : 20997     1/1       24/496     : 0           2/2       1/1        DetInfo(EpicsArch.0:NoDevice.0)
 14: FEEGasDetEnergy         : 1588      0/0       32/32      : 0           2/3       6/6        BldInfo(FEEGasDetEnergy)
 15: EBeamBld                : 1590      1/1       60/60      : 0           2/3       6/6        BldInfo(EBeam)
 16: PhaseCavity             : 0     
 17: PrincetonFrame          : 0     
 18: PrincetonConfig         : 0     
 19: EvrData                 : 1589      3/3       16/16      : 0           2/2       1/1        DetInfo(NoDetector.0:Evr.0)
 20: FrameFccdConfig         : 0     
 21: FccdConfig              : 0     
 22: IpimbData               : 7945      2/2       32/32      : 0           2/2       1/1        DetInfo(CxiDg2.0:Ipimb.1)
                                                                                                 DetInfo(CxiDg4.0:Ipimb.0)
                                                                                                 DetInfo(CxiSc1.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg1.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg2.0:Ipimb.0)
 23: IpimbConfig             : 5         2/2       60/60      : 0           2/2       1/1        DetInfo(CxiDg1.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg2.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg2.0:Ipimb.1)
                                                                                                 DetInfo(CxiDg4.0:Ipimb.0)
                                                                                                 DetInfo(CxiSc1.0:Ipimb.0)
 24: EncoderData             : 0     
 25: EncoderConfig           : 0     
 26: EvrIOConfig             : 0     
 27: PrincetonInfo           : 0     
 28: CspadElement            : 3178      1/2  4594064/4594064 : 0           2/2       1/1        DetInfo(CxiDs1.0:Cspad.0)
                                                                                                 DetInfo(CxiDsd.0:Cspad.0)
 29: CspadConfig             : 2         3/3   287844/287844  : 0           2/2       1/1        DetInfo(CxiDsd.0:Cspad.0)
                                                                                                 DetInfo(CxiDs1.0:Cspad.0)
 30: IpmFexConfig            : 5         2/2      520/520     : 0           2/2       1/1        DetInfo(CxiDg1.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg2.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg2.0:Ipimb.1)
                                                                                                 DetInfo(CxiDg4.0:Ipimb.0)
                                                                                                 DetInfo(CxiSc1.0:Ipimb.0)
 31: IpmFex                  : 7945      1/1       28/28      : 0           2/2       1/1        DetInfo(CxiDg2.0:Ipimb.1)
                                                                                                 DetInfo(CxiDg4.0:Ipimb.0)
                                                                                                 DetInfo(CxiSc1.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg1.0:Ipimb.0)
                                                                                                 DetInfo(CxiDg2.0:Ipimb.0)
 32: DiodeFexConfig          : 0     
 33: DiodeFex                : 0     
 34: PimImageConfig          : 1         1/1        8/8       : 0           2/2       1/1        DetInfo(CxiSc1.0:Tm6740.0)
 35: SharedIpimb             : 0     
 36: AcqTDCConfig            : 0     
 37: AcqTDCData              : 0     
 38: Index                   : 0     
 39: XampsConfig             : 0     
 40: XampsElement            : 0     
 41: Cspad2x2Element         : 0     
 42: SharedPIM               : 0     
 43: Cspad2x2Config          : 0     
 44: FexampConfig            : 0     
 45: FexampElement           : 0     
 46: Gsc16aiConfig           : 0     
 47: Gsc16aiData             : 0     
 48: PhasicsConfig           : 0     
 49: TimepixConfig           : 0     
 50: TimepixData             : 0     
 51: CspadCompressedElement  : 0     
 52: OceanOpticsConfig       : 0     
 53: OceanOpticsData         : 0     
 54: EpicsConfig             : 0     
 55: FliConfig               : 0     
 56: FliFrame                : 0     
 57: QuartzConfig            : 0     

Accumulate statistics of damages

  • Acumulate counters for
    • damage bits
    • damage data type
    • damage bits vs data type
  • Accumulate damage records; content of each record:
    struct DamageRecord {
      unsigned ndg;
      unsigned depth;
      uint32_t damage_value;
      Pds::TypeId::Type type_id;
    };
    

Summary table for damage bit counters

QCStatistics::printDamageBitCounter()   Damage counter
Damage bit:1  : DroppedContribution            6
Damage bit:12 : OutOfOrder                     0
Damage bit:13 : OutOfSynch                     0
Damage bit:14 : UserDefined                    4
Damage bit:15 : IncompleteContribution         9
Damage bit:16 : ContainsIncomplete            18

Summary table for damage bit vs data types counters

QCStatistics::printDamageCounter()
Total number of known TypeId : 58
Ind: TypeId name             : Entries  DmgTot  Bits:1       12      13      14      15      16      
  0: Any                     : 2             2       2       0       0       0       0       0
  1: Xtc                     : 14365        26       4       0       0       4       0      18
  2: Frame                   : 1589          0       0       0       0       0       0       0
  3: AcqWaveform             : 0      
  4: AcqConfig               : 0      
  5: TwoDGaussian            : 0      
  6: Opal1kConfig            : 0      
  7: FrameFexConfig          : 1             0       0       0       0       0       0       0
  8: EvrConfig               : 2             0       0       0       0       0       0       0
  9: TM6740Config            : 1             0       0       0       0       0       0       0
 10: RunControlConfig        : 2             0       0       0       0       0       0       0
 11: pnCCDframe              : 0      
 12: pnCCDconfig             : 0      
 13: Epics                   : 20997         0       0       0       0       0       0       0
 14: FEEGasDetEnergy         : 1588          0       0       0       0       0       0       0
 15: EBeamBld                : 1590          0       0       0       0       0       0       0
 16: PhaseCavity             : 0      
 17: PrincetonFrame          : 0      
 18: PrincetonConfig         : 0      
 19: EvrData                 : 1589          0       0       0       0       0       0       0
 20: FrameFccdConfig         : 0      
 21: FccdConfig              : 0      
 22: IpimbData               : 7945          0       0       0       0       0       0       0
 23: IpimbConfig             : 5             0       0       0       0       0       0       0
 24: EncoderData             : 0      
 25: EncoderConfig           : 0      
 26: EvrIOConfig             : 0      
 27: PrincetonInfo           : 0      
 28: CspadElement            : 3178          9       0       0       0       0       9       0
 29: CspadConfig             : 2             0       0       0       0       0       0       0
 30: IpmFexConfig            : 5             0       0       0       0       0       0       0
 31: IpmFex                  : 7945          0       0       0       0       0       0       0
 32: DiodeFexConfig          : 0      
 33: DiodeFex                : 0      
 34: PimImageConfig          : 1             0       0       0       0       0       0       0
 35: SharedIpimb             : 0      
 36: AcqTDCConfig            : 0      
 37: AcqTDCData              : 0      
 38: Index                   : 0      
 39: XampsConfig             : 0      
 40: XampsElement            : 0      
 41: Cspad2x2Element         : 0      
 42: SharedPIM               : 0      
 43: Cspad2x2Config          : 0      
 44: FexampConfig            : 0      
 45: FexampElement           : 0      
 46: Gsc16aiConfig           : 0      
 47: Gsc16aiData             : 0      
 48: PhasicsConfig           : 0      
 49: TimepixConfig           : 0      
 50: TimepixData             : 0      
 51: CspadCompressedElement  : 0      
 52: OceanOpticsConfig       : 0      
 53: OceanOpticsData         : 0      
 54: EpicsConfig             : 0      
 55: FliConfig               : 0      
 56: FliFrame                : 0      
 57: QuartzConfig            : 0      

Print in summary all damage records

QCStatistics::printDamageRecords()
Total number of damage records : 33
Damage Record: Dgram:    72  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:    72  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:    72  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:   123  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   123  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   123  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:   272  depth:0  type:Xtc                       damage:     2  bit: 1: DroppedContribution
Damage Record: Dgram:   272  depth:1  type:Xtc                       damage:     2  bit: 1: DroppedContribution
Damage Record: Dgram:   272  depth:2  type:Any                       damage:     2  bit: 1: DroppedContribution
Damage Record: Dgram:   284  depth:0  type:Xtc                       damage: 81920  bit:14: UserDefined
                                                                                    bit:16: ContainsIncomplete
Damage Record: Dgram:   284  depth:1  type:Xtc                       damage: 81920  bit:14: UserDefined
                                                                                    bit:16: ContainsIncomplete
Damage Record: Dgram:   284  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:   332  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   332  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   332  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:   479  depth:0  type:Xtc                       damage:     2  bit: 1: DroppedContribution
Damage Record: Dgram:   479  depth:1  type:Xtc                       damage:     2  bit: 1: DroppedContribution
Damage Record: Dgram:   479  depth:2  type:Any                       damage:     2  bit: 1: DroppedContribution
Damage Record: Dgram:   700  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   700  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   700  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:   790  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   790  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   790  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:   923  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   923  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:   923  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:  1025  depth:0  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:  1025  depth:1  type:Xtc                       damage: 65536  bit:16: ContainsIncomplete
Damage Record: Dgram:  1025  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution
Damage Record: Dgram:  1305  depth:0  type:Xtc                       damage: 81920  bit:14: UserDefined
                                                                                    bit:16: ContainsIncomplete
Damage Record: Dgram:  1305  depth:1  type:Xtc                       damage: 81920  bit:14: UserDefined
                                                                                    bit:16: ContainsIncomplete
Damage Record: Dgram:  1305  depth:2  type:CspadElement              damage: 32768  bit:15: IncompleteContribution

Data quality check

  • psana-based analysis for selected types of data with known properties.
  • This is data-specific check, which can not be universal for all data types.

Indexing?

The same procedure of the loop over datagrams with iteration over XTC containers can be used for XTC data indexing..., but need to read all files for streams and chunks within run using the same algorithm like in psana.

Command line interface

xtcqc [-h] [-l <logfile>] [-b <basedir>] <fname1> [<fname2> [<fname3> ...]]

Optional parameters:
-h - help, showing usage of the command line parameters.
-l - log-file; if specified, then output will be re-directed to this file, otherwise on stdout.
-b - base-directory, i.e.: /reg/d/psdm/<INSTRUMENT>/<experiment>/xtc; if specified, than all xtc file names will be defined w.r.t. this directory.

Non-optional parameters:
<fname1> [<fname2> [<fname3> ...]] - all xtc file names should be listed without options. At least one xtc file name should be provided.

Unpleasant feature of getopt:
The options -l and -b require the file name as a trailed optional parameter. If the optional parameter is mistakenly missed, then entire input will be corrupted.

Database for XTC Quality Check

Whenever the xtc file is processed, the results are saved in the database

 
/reg/d/psdm/<INSTRUMENT>/<experiment>/xtcqc/<xtc-like-file-name>.qc

Web-based control and access

Igor is going to produce web-based interface, like for translator, with functionality as follows.

  • Show list of files with processed/not-processed flag,
  • Control on job submission for quality check,
  • Browser for QC results of particular file/run;
    • show log file content - minimal version,
    • show statistics in the database - full-scale version.
  • No labels