A less common pattern for parallelization is the "cube" pattern:  e.g. average image (or other variable) vs. pump-probe time.  This can be too memory intensive with the standard MPIDataSource parallelization (over events) because every core will have a copy of the full cube.  Instead the pattern is to give each core only one (or a few) time bins and make sure that core only fetches/analyzes data for those events.  The script below is an example of this parallelization pattern.

from psana import *
import numpy as np

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

dsource = DataSource('exp=xpptut15:run=54:smd')
cspaddet = Detector('cspad')

mysum = np.zeros([32,185,388])
myevts = 0
for nevt,evt in enumerate(dsource.events()):
   # replace this line with your calculation (using "rank")
   # of whether or not this event is in this rank's time bin.
   # e.g. rank0 may get the first time bin, rank1 the second, etc.
   mytimebin=True
   if not mytimebin: continue
   calib = cspaddet.calib(evt)
   if calib is None: continue
   mysum += calib
   myevts += 1
   # early exit for faster debugging
   if nevt>3: break

# each mpi rank save its own average
print 'Rank',rank,'saw',myevts,'events'
if myevts>0:
   outfile = open('mybinavg%d.npy'%rank,'wb')
   np.save(outfile,mysum/myevts)
   outfile.close()
  • No labels