You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Data file quality check

Datagram and XTC structure check

Stand-alone C++ application which uses the pdsdata library.
Assumes the loop over all Drgams in a single XTC file with iterations over all XTC containers. Retrieved information is tested for consistency, as explained below.

Accumulate statistics for transitions

Print transition Id counters
  Total number of transition Id:13
Unknown         : 0         Reset           : 0         
Map             : 0         Unmap           : 0         
Configure       : 1         Unconfigure     : 0         
BeginRun        : 1         EndRun          : 1         
BeginCalibCycle : 1         EndCalibCycle   : 1         
Enable          : 1         Disable         : 1         
L1Accept        : 3520

Accumulate statistics for damages

Damage bit  1 : DroppedContribution
Damage bit 12 : OutOfOrder 
Damage bit 13 : OutOfSynch 
Damage bit 14 : UserDefined
Damage bit 15 : IncompleteContribution
Damage bit 16 : ContainsIncomplete

Check correctness of the datagram time sequence

In perfect world it should be 

t(Dg1) < t(Dg2) < t(Dg3) < ... < t(DgN)

search for datadrams with wrong time sequence, like

t(Dgi) > t(Dg[i+1])

Size consistency of XTC containers

  • Check that the XTC container size does not exceed certain maximal value.
  • The total size of internal XTC containers should be consistent (and not exceed) with size of overlapping XTC container.

Accumulate statistics for data types

Count occurrence of each data type in XTC containers.

Data quality check

  • psana-based analysis for selected types of data with known properties.
  • This is data-specific check, which can not be universal for all data types.

Indexing?

The same procedure of the loop over datagrams with iteration over XTC containers can be used for XTC data indexing..., but need to read all files for streams and chunks within run using the same algorithm like in psana.

Command line interface 

b. xtcqc -f <file-name>.xtc [-h]

Database for XTC Quality Check

Whenever the xtc file is processed, the results are saved in the database

 
/reg/d/psdm/<INSTRUMENT>/<experiment>/xtcqc/<xtc-like-file-name>.qc

Web-based control and access

Igor is going to produce web-based interface, like for translator, with functionality as follows.

  • Show list of files with processed/not-processed flag,
  • Control on job submission for quality check,
  • Browser for QC results of particular file/run.
  • No labels