A brainstorming session on Aug. 16, 2021 with Mona, Mikhail, Silke, cpo.


conclusions:
- we should probably implement (1) below because it's easy
- (2) is more difficult and depends on the specific problem, so we wait until there is a concrete need

possibilities:
1) layer on top of the filter_fn where user provides a list of timestamps
o advantage of this: scales (as many BD cores as you want)
o disadvantage of this: you can't jump back in time (still receive events in time order). BUT, I know of now LCLS analysis that needs to really jump back in time (large-timescale XPCS is not done at LCLS, ghost-correction is another case)
o users could buffer the data in memory (if not too large) to jump "back in time"
o advantage: minimal changes to psana
o another example: timetool subtracting closest laser-off event (mona says if they could be marked as "broadcast" by DAQ that would happen naturally). doesn't allow for picking "future" background event?
o problematic case: ghost-correction (one event depends on another)

Possible interface: DataSource(exp=,run=,timestamps=)

2) something more complex to jump backward in time:
o we can't make this run at 1MHz
o harder
o needed by large-timescale XPCS (Marcin analysis) and ghost-correction
o how to handle the "broadcast"

total events: 3,4,5,6,7,8,9

for ghost-correction (evt4 = evt4-0.1*evt3):
BD1 wants to analyze 3,4 (gets sent 2,3,4 (window1))
BD2 wants to analyze 5,6 (gets sent 4,5,6 (window2))
....

possible general idea:
- for small windows of jumping: smd0 sends out "windows" of overlapping events
- how to handle a large jump backwards or forwards? (e.g. last event back to first event). everyone needs the smd0 jump-table which is too large for memory @1MHz. guess: 256bytes/event*1MHz = 256MB/s, so 10 minute run for 16 streams is 153GB*16=2.4TB of small data.

assume for the moment only jumping within a run.

  • No labels