Page History
Initial Discussion
want:
- give the drp a psana-python script
- drive that psana-python script by calling psana_set_dgram(Dgram*) (would replace the file reading)
...
A potential issue: (lower priority) this method of running psana (and shmem) do not have scalable ways of loading the calibration constants: each core will access the database. Ideally we would fix.
Second Discussion
Oct 14, 2021 with Mikhail, Mona, Valerio
Executive Summary: now prefer option 1 below, which feels like the simplest, although not highest performance.
Current picture:
kcu1500 -> PGPReader -> N*Worker -> Ric's TEB -> Collector
^^^^^^^^^-- on another node
Ideally, replace Worker threads with Worker processes
BUT, RIC's (complicated) infiniband stuff is all multithreaded
Option 1: (the original idea discussed above)
One process for C++ and python: Mona calls Py_CallScript("dummy.py")
which returns when it hits iterator iterator is awoken with
psana_set_dgram(dg)
Advantage:
- only 1 process
- no new DRP
Disadvantage:
- weird
Option 2:
Python is low performance. Maybe replace all of it with multiprocess zmq?
Disadvantages:
- two independent DRP's to maintain
- rewrite Ric's TEB/MEB, easier with ZMQ
o ZMQ has some uncontrollable aspects (e.g. hidden threads)
- also have to rewrite the shmem server
- a lot of work
Advantage:
- looks more like shmem
- standard multiprocess python
Option 3:
kcu1500 -> PGPReader -> N*Worker -> Ric's TEB -> Collector
^^^^^^^^^-- on another node
each Worker has a bidirectional pipe to a separate psana process
(behaves like shmem again) where dgrams go back-and-forth: psana is
given a "raw" dgram, psana returns a "fex" dgram, e.g. with list-of-photons
Worker <---> psana
Could use zmq? or something similar.
Advantage:
- only have to support 1 DRP
Disadvantage:
- 2 processes with communication (also have that in option 2)
work to be done:
- valerio: drp communicate with multiple python processes
- mona:
o routines to modify dgrams from python
- look at dgramCreate.pyx
https://github.com/slac-lcls/lcls2/blob/master/psana/psana/peakFinder/dgramCreate.pyx
also the test test_py2xtc.py
goal: receive the raw-dgram, and return a fex-dgram with the raw
data removed (almost always) and put in fex data.
o think about calib-fetch scaling (lower-priority)
- ideally independent of mpi. zmq approach? how do we
determine "supervisor"?