You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Data file quality check

Datagram and XTC structure check

Stand-alone C++ application which uses the pdsdata library.
Assumes the loop over all Drgams in a single XTC file with iterations over all XTC containers. Retrieved information is tested for consistency, as explained below.

Accumulate statistics for transitions

QCStatistics::printTransIdCounter()
Total number of known TransitionId-s:13
Unknown         : 0         Reset           : 0         
Map             : 0         Unmap           : 0         
Configure       : 1         Unconfigure     : 0         
BeginRun        : 1         EndRun          : 0         
BeginCalibCycle : 1         EndCalibCycle   : 0         
Enable          : 1         Disable         : 0         
L1Accept        : 96

Accumulate statistics for damages

Damage bit  1 : DroppedContribution
Damage bit 12 : OutOfOrder 
Damage bit 13 : OutOfSynch 
Damage bit 14 : UserDefined
Damage bit 15 : IncompleteContribution
Damage bit 16 : ContainsIncomplete

Check correctness of the datagram time sequence

In perfect world it should be 

t(Dg1) < t(Dg2) < t(Dg3) < ... < t(DgN)

search for datadrams with wrong time sequence, like

t(Dgi) > t(Dg[i+1])

Size consistency of XTC containers

  • Check that the XTC container size does not exceed certain maximal value.
  • The total size of internal XTC containers should be consistent (and not exceed) with size of overlapping XTC container.

Accumulate statistics for data types

Count occurrence of each data type in XTC containers.

QCStatistics::printTypeIdCounter()
Total number of known TypeId-s:58
Ind: TypeId name             : Entries  Vers.  payload min/max:#zeros  srclog min/max    srcphy min/max       level min/max 
  0: Any                     : 0     
  1: Xtc                     : 901       0/1        0/9385644 : 57         1685/31619     167772160/2887064203        1/3       
  2: Frame                   : 96        1/1   614416/614416  : 0         12875/12875     587203584/587203584         1/1       
  3: AcqWaveform             : 0     
  4: AcqConfig               : 0     
  5: TwoDGaussian            : 0     
  6: Opal1kConfig            : 0     
  7: FrameFexConfig          : 1         1/1       28/28      : 0         12875/12875     587203584/587203584         1/1       
  8: EvrConfig               : 2         5/5      192/192     : 0         11052/11052           256/256               1/1       
  9: TM6740Config            : 1         2/2       12/12      : 0         12875/12875     587203584/587203584         1/1       
 10: RunControlConfig        : 2         1/1       24/24      : 0          8644/8644              0/0                 0/0       
 11: pnCCDframe              : 0     
 12: pnCCDconfig             : 0     
 13: Epics                   : 16167     1/1       24/496     : 0          1685/12881     150994944/150994944         1/1       
 14: FEEGasDetEnergy         : 97        0/0       32/32      : 0             0/0                 2/2                 6/6       
 15: EBeamBld                : 97        1/1       60/60      : 0             0/0                 0/0                 6/6       
 16: PhaseCavity             : 0     
 17: PrincetonFrame          : 0     
 18: PrincetonConfig         : 0     
 19: EvrData                 : 96        3/3       16/16      : 0         11052/11052           256/256               1/1       
 20: FrameFccdConfig         : 0     
 21: FccdConfig              : 0     
 22: IpimbData               : 480       2/2       32/32      : 0         31619/31619     452986880/587204608         1/1       
 23: IpimbConfig             : 5         2/2       60/60      : 0         31619/31619     452986880/587204608         1/1       
 24: EncoderData             : 0     
 25: EncoderConfig           : 0     
 26: EvrIOConfig             : 0     
 27: PrincetonInfo           : 0     
 28: CspadElement            : 192       2/2  4163384/4594064 : 0         12855/26765     536873472/603982336         1/1       
 29: CspadConfig             : 2         3/3   287844/287844  : 0         12855/26765     536873472/603982336         1/1       
 30: IpmFexConfig            : 5         2/2      520/520     : 0         31619/31619     452986880/587204608         1/1       
 31: IpmFex                  : 480       1/1       28/28      : 0         31619/31619     452986880/587204608         1/1       
 32: DiodeFexConfig          : 0     
 33: DiodeFex                : 0     
 34: PimImageConfig          : 1         1/1        8/8       : 0         12875/12875     587203584/587203584         1/1       
 35: SharedIpimb             : 0     
 36: AcqTDCConfig            : 0     
 37: AcqTDCData              : 0     
 38: Index                   : 0     
 39: XampsConfig             : 0     
 40: XampsElement            : 0     
 41: Cspad2x2Element         : 0     
 42: SharedPIM               : 0     
 43: Cspad2x2Config          : 0     
 44: FexampConfig            : 0     
 45: FexampElement           : 0     
 46: Gsc16aiConfig           : 0     
 47: Gsc16aiData             : 0     
 48: PhasicsConfig           : 0     
 49: TimepixConfig           : 0     
 50: TimepixData             : 0     
 51: CspadCompressedElement  : 0     
 52: OceanOpticsConfig       : 0     
 53: OceanOpticsData         : 0     
 54: EpicsConfig             : 0     
 55: FliConfig               : 0     
 56: FliFrame                : 0     
 57: QuartzConfig            : 0

Data quality check

  • psana-based analysis for selected types of data with known properties.
  • This is data-specific check, which can not be universal for all data types.

Indexing?

The same procedure of the loop over datagrams with iteration over XTC containers can be used for XTC data indexing..., but need to read all files for streams and chunks within run using the same algorithm like in psana.

Command line interface

xtcqc -f <file-name>.xtc -h

Database for XTC Quality Check

Whenever the xtc file is processed, the results are saved in the database

 
/reg/d/psdm/<INSTRUMENT>/<experiment>/xtcqc/<xtc-like-file-name>.qc

Web-based control and access

Igor is going to produce web-based interface, like for translator, with functionality as follows.

  • Show list of files with processed/not-processed flag,
  • Control on job submission for quality check,
  • Browser for QC results of particular file/run.
  • No labels