Deadtime Behavior

From a slack thread with Matt on Jan. 31, 2024:

Matt, does the 0x80 in this line in control.py mean that SlowUpdate will not be sent if any readout group has deadtime?

            # Force SlowUpdate to respect deadtime                                     
            if not self.pva.pv_put(pv, (0x80 | ControlDef.transitionId['SlowUpdate'])):

Matt writes: The coupled deadtime idea didn't work.  So, this should make the transition respect deadtime per readout group (some might get it and some might not).

Mona writes about psana2: eventbuilder requires that all non-L1 have to be complete (i.e. all streams need to see SlowUpdate).  This suggests to cpo that we need to eliminate the 0x80 bit and set xpm-pause_threshold low enough that we always guarantee at least one SlowUpdate can be received by all detectors.  Matt agrees with this, but points out that the system must drain that SlowUpdate before another is sent down.

Brainstorming

Issue: SlowUpdate currently preempts L1Accept, so integrating detectors miss their normalization data

...

meb tells teb, teb tells drp which L1 meb and which buffer to use (for L1)

for SU (and other transitions) meb tells drp which transition-buffer to use
   - msg is sent as soon as meb is done with the previous SU or
     any other transition (i.e. when previous transition copied to shmem)
effectively, the drp has a list of free transition buffers

...

  - need to tell worker to poll all the variables 
  - could the epicsarch worker attach the payload to a high-rate l1accept
    at some slow rate (allows the slowupdate to flow from worker to ebreceiver)
  - EbReceiver would translate the low-rate L1Accept into a slowupdate transition
    (using the +1ns hack) and save SlowUpdate to xtc
  - could keep the L1Accept (a little ugly) could truncate it in epicsarch.
    The If the L1Accept is selected for monitoring, the MEB needs the L1Accept header at least to complete the event-build!
    Mona doesn't need the L1Accept, cpo thinks.
  - SlowUpdates won't necessarily be accompanied by their "parent" L1Accepts in the MEB

two SU broadcasts:
- EbReceiver going to multiple meb's
- psana's SMD0

what if there are two generators of slowupdate?  What if they're generated on the same timestamp?

like a "locally generated occurrence" (babar synchrequest from 1 ROM)

...

- can things get out of time order in the meb if locally-generated slowupdate is not event-built?

• Ric: Yes, I think they can.  It leads to the same problem that was dealt with by inventing the common readout group.
  
  • The EBs are often empty
  • When a new contribution arrives, the only time reference available is that of the most recently built event or previous contribution.  For the case of an empty EB, the code can thus recognize if the new contribution starts a new event or is out of time order.
  • For the case of the proposed SlowUpdate, it may make sense to think of it as belonging to its own readout group (perhaps a fictitious one).  The single contribution can then be determined to be a complete event and is flushed to the TransitionCache.
    • Another possibility might be to have SlowUpdates bypass the EB and go directly to the TransitionCache, but that would take some redesign of the system.
  • Various delays in the system (e.g., network and scheduling, etc.) can conspire to cause the first contribution of an event older than the SlowUpdate to arrive in the MEB after the SlowUpdate has been flushed.  All contributions from a given source arrive in the EBs in time order, but nothing coordinates their arriving in time order relative to other sources.  Thus, contributions from different readout groups could arrive in any order were it not for our requirement that the common readout group triggers together with any other readout group.
• Since SlowUpdate data is imprecisely timestamped (PV read times may differ from the parent L1Accept timestamp by random amounts), is it a significant problem if the SlowUpdates appear in MEB shmem out of order with respect to the L1Accepts?
  • This might affect reproducibility between online monitoring and offline psana

changes required:
- remove slowupdate generation from control.py
- change epicsarch to do the L1/SU generation
- mona changes the psana broadcast of SU in some sort of backwardly compatible way
- ric intuition: drp executable significant changes? EbReceiver?