A less common pattern for parallelization is the "cube" pattern: e.g. average image (or other variable) vs. pump-probe time. This can be too memory intensive with the standard MPIDataSource parallelization (over events) because every core will have a copy of the full cube. Instead the pattern is to give each core only one (or a few) time bins and make sure that core only fetches/analyzes data for those events. The script below is an example of this parallelization pattern.
from psana import * import numpy as np from mpi4py import MPI comm = MPI.COMM_WORLD rank = comm.Get_rank() size = comm.Get_size() dsource = DataSource('exp=xpptut15:run=54:smd') cspaddet = Detector('cspad') mysum = np.zeros([32,185,388]) myevts = 0 for nevt,evt in enumerate(dsource.events()): # replace this line with your calculation (using "rank") # of whether or not this event is in this rank's time bin. # e.g. rank0 may get the first time bin, rank1 the second, etc. mytimebin=True if not mytimebin: continue calib = cspaddet.calib(evt) if calib is None: continue mysum += calib myevts += 1 # early exit for faster debugging if nevt>3: break # each mpi rank save its own average print 'Rank',rank,'saw',myevts,'events' if myevts>0: outfile = open('mybinavg%d.npy'%rank,'wb') np.save(outfile,mysum/myevts) outfile.close()
Overview
Content Tools