...

A potential issue: (lower priority) this method of running psana (and shmem) do not have scalable ways of loading the calibration constants: each core will access the database.  Ideally we would fix.

• needed for epixhr
  • we should make psana (det.calib and det.image) usable in drp
• happens in the worker cores (right after dma engine, before time-ordered messages for teb/meb):
  • worry about GIL with threads
• may need easier tools to append to xtc from python (extracted features)
• configured from configdb?
• can consider compiling python (e.g. with cython) for speed improvement
  • can gain 20%-30%? (potentially small improvement compared to GIL)
  • cpo: treat as an optimization
• really depending on mikhail's area-detector xface being able to handle variable number of segments
• problem: currently we can't make a DataSource
  • Mona can probably implement one
  • problem: How do we scale to large numbers of cores fetching calibration constants? (this is a problem for shmem too)

Thoughts on a DRP DataSource

• dgrams flow through in memory, one at a time (a little like shmem)
• the dgrams are in circular buffers (a little like shmem) ("pebble": per-event-buffers-with-boundaries-listed-explicitly)
• like shared memory, we don't actually free the dgram memory when we're done
• shmem doesn't work, because shmem drops events if the code can't keep up
• need a new pattern similar to shmem

Usage of Python

• (optional, e.g. if we run 'drp -p "myroi.py <arg1> <arg2>"') run in each worker thread (one subprocess per worker):
  
  • send detector raw-data datagram to psana-python subprocess via a zmq bi-directional pipe (perhaps).  Performance matters
  • receive potentially two datagrams back:
    • datagram with fex data
    • datagram with trigger data (e.g. number of photons) to send to the teb for both the trigger/monitoring decision
  • to save memory the received fex datagram would replace the raw datagram in pebble memory (requires a memory copy)
• each teb process would have optional python to process the trigger-datagram produced by the worker (the python may not be psana, depending on whether or not we send all the phase 2 transitions to the teb? ... don't remember)
• we would start with fex and do the trigger later

Possible approach in time-order:

• standalone C++ with two threads, each thread having a python subprocess communicating with a bidirectional pipe/queue/shmem (whatever is most efficient).  send "hello world" back and forth.
• embed the above in a drp executable
• start passing dgrams instead of "hello world"
• switch to psana-python

An early-ish goal is the TXI 2M epixHR.  ROI doesn't need corrections, but photon finding does.  Can we do photon-finding @25kHz?  Suppose 25kHz distributed over 4 nodes (240 cores total) equivalent of 100Hz, but currently 2M epix's run on one core at 5Hz (what Valerio sees with peakfinder8 in MFX?).  So a factor of 20 discrepancy.

Second Discussion

Oct 14, 2021 with Mikhail, Mona, Valerio

...