This script lives in /sdf/group/lcls/ds/ana/tutorials/psana1_examples/mpiGather.py and demonstrates how to save small pieces of information on every event (in parallel) and then gather those pieces together onto one core.

Run this with "mpirun -n 2 python mpiGather.py":

from psana import *
import numpy as np
ds = DataSource('exp=xpptut15:run=54:smd')
det = Detector('cspad')

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

intensity = []
for nevent,evt in enumerate(ds.events()):
    if nevent%size!=rank: continue # different ranks look at different events
    img = det.image(evt)
    intensity.append(img.sum())
    if nevent>=3: break

allIntensities = comm.gather(intensity) # get intensities from all ranks

if rank==0:
    allIntensities = np.concatenate((allIntensities[:])) # put in one long list
    print(allIntensities)

MPI.Finalize()

Note that this example uses the lower-case "gather" method (not "Gather").  This means it can transmit python objects (via "pickle") which is more flexible and easier, but it is less performant than the capitalized "Gatherv" approach, which works more efficiently (but is more difficult to use).  An example with Gatherv is here.

  • No labels