Initial Discussion

want:
- give the drp a psana-python script
- drive that psana-python script by calling psana_set_dgram(Dgram*) (would replace the file reading)

ds = DataSource(dgramDsource=True)
myrun = next(ds.runs())
for event in myrun.events():
pass

xtcreader C++ calls: psana_set_dgram(Dgram*) (goes into dgram.cc?)

idea:

- multiprocess drp (not multithreaded) (Valerio), talk with Elliott (like legion)
  o have standalone multi-threaded or multi-process C++ code run python scripts
- (Mona) use xtcreader to represent one of those processes for development
modified to call a python script, perhaps like:
https://github.com/slac-lcls/lcls2/blob/c4fa38db1799b5c2acf6e4908daf50403c1bf616/psdaq/drp/BEBDetector.cc#L80
- xtcreader C++ calls psana-python script
  o it make a new DataSource (DgramDataSource?)
  o as drp C++ receives a new dgram, it passes it to the DgramDataSource (instead of reading from file)

ds = DataSource(dgramDsource=True)

C++ call: psana_set_dgram(Dgram*) (would replace the file reading)

the above Dgram is passed (somehow) to psana/src/dgram.cc (does
file-reading) who creates the python "Dgram"

maybe use the "buffer"/view interface?

two options:
(1) shmem we copy every dgram so that the python reference counting works in a standard way. could do the same thing here. decouples the psana-memory-management from drp-memory-management
(2) we don't copy the dgram, more efficient but we can't delete the dgram in a normal way, and we can't save information from old events

my inclination is to do (1)

A potential issue: (lower priority) this method of running psana (and shmem) do not have scalable ways of loading the calibration constants: each core will access the database.  Ideally we would fix.

• needed for epixhr
  • we should make psana (det.calib and det.image) usable in drp
• happens in the worker cores (right after dma engine, before time-ordered messages for teb/meb):
  • worry about GIL with threads
• may need easier tools to append to xtc from python (extracted features)
• configured from configdb?
• can consider compiling python (e.g. with cython) for speed improvement
  • can gain 20%-30%? (potentially small improvement compared to GIL)
  • cpo: treat as an optimization
• really depending on mikhail's area-detector xface being able to handle variable number of segments
• problem: currently we can't make a DataSource
  • Mona can probably implement one
  • problem: How do we scale to large numbers of cores fetching calibration constants? (this is a problem for shmem too)

Thoughts on a DRP DataSource

• dgrams flow through in memory, one at a time (a little like shmem)
• the dgrams are in circular buffers (a little like shmem) ("pebble": per-event-buffers-with-boundaries-listed-explicitly)
• like shared memory, we don't actually free the dgram memory when we're done
• shmem doesn't work, because shmem drops events if the code can't keep up
• need a new pattern similar to shmem

Usage of Python

• (optional, e.g. if we run 'drp -p "myroi.py <arg1> <arg2>"') run in each worker thread (one subprocess per worker):
  
  • send detector raw-data datagram to psana-python subprocess via a zmq bi-directional pipe (perhaps).  Performance matters
  • receive potentially two datagrams back:
    • datagram with fex data
    • datagram with trigger data (e.g. number of photons) to send to the teb for both the trigger/monitoring decision
  • to save memory the received fex datagram would replace the raw datagram in pebble memory (requires a memory copy)
• each teb process would have optional python to process the trigger-datagram produced by the worker (the python may not be psana, depending on whether or not we send all the phase 2 transitions to the teb? ... don't remember)
• we would start with fex and do the trigger later

Possible approach in time-order:

• standalone C++ with two threads, each thread having a python subprocess communicating with a bidirectional pipe/queue/shmem (whatever is most efficient).  send "hello world" back and forth.
• embed the above in a drp executable
• start passing dgrams instead of "hello world"
• switch to psana-python

An early-ish goal is the TXI 2M epixHR.  ROI doesn't need corrections, but photon finding does.  Can we do photon-finding @25kHz?  Suppose 25kHz distributed over 4 nodes (240 cores total) equivalent of 100Hz, but currently 2M epix's run on one core at 5Hz (what Valerio sees with peakfinder8 in MFX?).  So a factor of 20 discrepancy.

Second Discussion

Oct 14, 2021 with Mikhail, Mona, Valerio

Executive Summary: now prefer option 3 below, which feels like the simplest, although not highest performance.

Current picture:

kcu1500 -> PGPReader -> N*Worker -> Ric's TEB -> Collector
(TEB/MEB stuff on another node)

Ideally, replace Worker threads with Worker processes
BUT, RIC's (complicated) infiniband stuff is all multithreaded

Option 1: (the original idea discussed above)

One process for C++ and python: Mona calls Py_CallScript("dummy.py")
which returns when it hits iterator iterator is awoken with
psana_set_dgram(dg)

Advantage:
- only 1 process
- no new DRP

Disadvantage:
- weird

Option 2:

Python is low performance.  Maybe replace all of it with multiprocess zmq?

Disadvantages:
- two independent DRP's to maintain
- rewrite Ric's TEB/MEB, easier with ZMQ
  o ZMQ has some uncontrollable aspects (e.g. hidden threads)
- also have to rewrite the shmem server
- a lot of work

Advantage:
- looks more like shmem
- standard multiprocess python

Option 3:

kcu1500 -> PGPReader -> N*Worker -> Ric's TEB -> Collector
(TEB/MEB stuff on another node)

each Worker has a bidirectional pipe to a separate psana process
(behaves like shmem again) where dgrams go back-and-forth: psana is
given a "raw" dgram, psana returns a "fex" dgram, e.g. with list-of-photons

Worker <---> psana

Could use zmq? bidirectional pipe? or something similar.

Advantage:
- only have to support 1 DRP

Disadvantage:
- 2 processes with communication (also have that in option 2)

work to be done:

- valerio: drp communicate with multiple python processes
- mona:
  o routines to modify dgrams from python
    - look at dgramCreate.pyx
      https://github.com/slac-lcls/lcls2/blob/master/psana/psana/peakFinder/dgramCreate.pyx
      also the test test_py2xtc.py
      goal: receive the raw-dgram, and return a fex-dgram with the raw (how do we do this return?)
      data removed (almost always) and put in fex data.
  o think about calib-fetch scaling (lower-priority)
    - ideally independent of mpi. zmq approach? how do we
      determine "supervisor"?

Json2XTC Option

drpDG -> psana calls createsjson -> drp calls Json2Xtc

- need to make sure json2xtc can separate the Names (configure-only)
  from Shapes/Data (l1accept)
- cpo worries about performance of the extra json step
- cpo votes to try dgramcreate approach (with 75% confidence level)