conversation with Wilko/Murali on Sept. 28, 2022

In future transfers to remote sites will happen in parallel to improve performance, resulting in temporary "holes" in files.  We discussed three options to handle this in psana:

(1) "smart" transfer using the smd file to control which events psana will read (this means the transfer software understand events)

easier for psana, but more difficult for transfer software to understand events (vs. bytes)

time----->

big file
   1-10     11-20         (transfer process 1)
   100-110  111-120  (transfer process 2)
small file
   1-10     11-20

(2) introduce another mechanism to communicate to smd0 which offsets
    can be safely read (psana is translating between bytes/events)

SMD0 queries a web service to get valid offsets: stop at the first "hole"?

easier for transfer software, but more difficult for psana SMD0 needing to query web service and translate between bytes/events

(3) psana reads files remotely? and then stores at remote site for reprocessing?

  • No labels