MPI.Reduce with Fake Steps

Psana2 client-server model results in uneven no. of events distributed to BigData cores. Especially in the last batch, one or more BigData cores might not receive the data. This results in MPI hang if users try to call MPI.Reduce. This problem can be mitigated by allowing broadcasting no. of times each BigData core performs Reduce operations through an insertion of the "fake" step transition set. Below is an explanation which shows how this is done by Psana2.

Environment variables 

PS_SMD_N_EVENTS=N

where N is no. of events that Smd0 sends to any EventBuilder cores at a time.

PS_FAKESTEP_FLAG=1

when the value is set, psana2 will insert fake step transition set at the end of each chunk IF AND ONLY IF: the last event of this chunk is either L1Accept or SlowUpdate. This is to preserve the order of the allowed state machine.

Fake step transition set

To preserve the order of the produced state machine:

BeginStep, Enable, L1Accept, L1Accept, ..., SlowUpdate, L1Accept, ... Disable, EndStep

, Psana2 inserts these four transitions at the end of the chunk (if allowed - see note above).

Disable, EndStep, BeginStep, Enable

Example

For a run with two stream files and two steps, with PS_SMD_N_EVENTS=5 and PS_FAKESTEP_FLAG=1

• First insert happens after the first five events (after the third L1)
• No insert after the second five events (not ending with an L1 or SlowUpdate)
• Second insert happens after the third five events (after the eight L1)
• No insert in the last batch (not ending with an L1 or SlowUpdate)

To preserve the consistency of the inserted transition, only one "winning" stream is used to determined the location for the insert. In other streams, the insertion rules apply according to what happen to the first stream.

Testing

An example test script and data are available as part of psana2 repo. In tests folder, run 

mpirun -n 5 python run_fakestep.py