• To support chunking in psana2 we propose to reuse the SlowUpdate Enable transition
  • When a DAQ segment level wants to chunk it will send a chunk-request to the control level
  • In response, the control level will somehow mark a future SlowUpdate Enable as a chunking "opportunity"
  • Those nodes that wish to chunk may do so when that SlowUpdate Enable is received
  • That SlowUpdate Enable should be the last datagram in the file
  • In that SlowUpdate Enable, segment levels will record a fake epics variable with the new chunk's filename
  • Mona will handle that fake epics variable as she would any other
  • We may want to have the fake epics variable name start with "_" (underscore) so we can hide it from "detnames -e"
  • Having the filename variables in Mona's epics-env-store will allow these filenames to be persisted to support future "event jumping"
  • We could imagine having all filenames (including first chunk) handled in this manner, but that hasn't been done for old data, so not possible.
  • need to check that file-registering with the database is done correctly?  Any issue with opening file's in different state-machine states?

Final Design

  • When a node gets to threshold (e.g. 500GB) then it send a request-for-disable-enable
  • Any node within an error bar of the threshold (e.g. 250GB) will use that enable to chunk, without making a request.  The enable could be sent by a scan
  • Enable contains the information about the new chunk filename, and is the last datagram in the file
  • Do we handle scans correctly?

Psana 2

Overview

Currently, we only support chunking on bigdata. The premises are:

  1. We start with -c00.xtc2 or -c00.xtc2.inprogress 
  2. When chunking happens,
    1. SlowUpdate with "_NEW_CHUNK_ID_SNN" (where NN is the stream id) is inserted to the smd file. 
    2. "_NEW_CHUNK_ID_SNN" stores id of the next chunk for look up -cKK (where KK is the chunk id)
  3. Bigdata after this SlowUpdate gets written onto a new chunk file
  4. Offsets of smddata after this SlowUpdate gets reset to 0

See our lcls2 repo. tests/test_data/chunking for an example.

Detail

With this, only bigdata nodes need to be refactored. We introduced a concept of cutoff_flag_array to identify how many times can we do "chunk" read based on the information that we have from smalldata. Here's an example of cutoff_flag_array (spreadsheet).

Here's how cutoff_flag_array is identified (1 means cutoff)

  • If this is a Transition (we don't need to read Transition from bigdata - they are part of smd files)
  • For L1Accept, cutoff_flag is set when
    • it's the first L1
    • It's offset is not connected to this "chunk" read

Once cutoff flags are all identified, anything that comes after SlowUpdate will be in the next read giving us an opportunity to check if we need to switch reading bigdata from a new chunk file.

DAQ/psana Communication Details

Two alternatives:

  • add another epics variable (1Hz)
    • evt._dgrams[nDgram].detname[nSegment].drpClassName.attr1.attr2...
  • add new epics-like data ("new detector") with this information (some preference?). 
    • more work for psana, easier for the daq
    • should we use a special "fixed" NamesIndex (like psdaq/drp/drp.hh) for this new data type
      • cpo thinks the answer is that we should use a fixed NamesIndex for this new data type (252 is the next one available) "CHUNKINFO"
      • enum name should match the xtc data name below
    • "detector" would be something different ("chunk") in the raw data python interface: evt._dgrams[nDgram].detname[nSegment].drpClassName.attr1.attr2...
    • how often does this new data type show up?  every SlowUpdate?  cpo hopes just when we chunk.
  • possible complexity: names have to be declared on configure transition.  getting those into every configure might be tricky.  might be high level hooks. (runinfo may solve this)

Default plan:

  • Dan: clone the RunInfo pattern in the file writer adding the new data type
  • Mona: mock up xtcwriter to add this new datatype: clone RunInfoDef pattern, and then add the data to SlowUpdate


Conversation with Two Chris's

Oct. 29, 2021

Two options:

  • a coordinated "chunking opportunity" (e.g. chunk on file size 500GB+-100GB) with Disable/Enable issued by control level, requested by a drp node
    • can the control level avoid chaos with multiple "simultaneous" requests? (e.g. race-conditions from requests 0.25s apart?)
      Answer: if the state has been Running for only a few seconds, ignore the new chunk request
    • more infrastructure needed
    • "deadtime" behavior different than the other case (stop events)
    • when we run a scan-sequence drp's should chunk on the Disable in the scan.  What if the node also sends a chunk request to control? (would it interrupt the scan?) 
      • what happens if we're running a scan-sequence and a drp sends a chunk request while in the Running state?  Do we interrupt the sequence? Answer: no, we don't interrupt the sequence?  control-level ignore the chunk-requests during a scan, perhaps with a warning message.  How does control figure out we're in a scan?  Chris Ford thinks there should be a way
  • a chaotic chunking opportunity on SlowUpdate
    • no control-level involvement (except maybe registering the file) 
    • potentially chaotic dead time as different node decide to chunk at different times

Meeting Notes: Chris Ford, Dan Damiani, and Matt

Nov. 1, 2021

  • LCLS-I Background:
    • chunking threshold is 50% of specified size
    • triggers are disabled while chunking
    • users have been happy with LCLS-I chunking
    • difference: LCLS-I slices grow fairly evenly, so they chunk simultaneously
  • choice: coordinated or chaotic?
    • we don't want to design a feature that depends on the reliability of "deadtime"
    • we could create a stress test separately, if needed
    • prefer coordinated approach
  • what  about chunking during a scan?
    • if chunk request received while Running, an extra Disable/Enable would occur, then scan would resume
    • to deal with race conditions, chunk requests received within a few seconds of  the last Enable are ignored
    • if chunk request received while not Running, the chunk request would be safely dropped
    • chunk requests during a scan should be rare, as chunking opportunities occur naturally through periodic Enable/Disable


  • No labels