Software version: ps-4.3.2 (Non-blocking MPIs)

Cluster: srcf

MPI Version: openmpi 4.1.0

Data size: 6.6 GB for one smd and bigdata files. Note that the original data are from tmoc00018 run 463 which has 5 set of smd and bigdata files (1 timing and 4 hsds). One of the hsd stream was duplicated to generate up to 60 sets of smd and bigdata files.

Data location: /cds/data/drpsrcf/users/monarin/tmoc00118/xtc

MPI Settings: Rank 0 (Smd0 )runs on its own node using 32 as maximum no. of threads. Other ranks are spread out with no. of maximum ranks per node = 50. Note that these no. of thread (32) and spreading factor (50) were based on empirical runs. 

Reading performance

This test only captures reading performance when running psana2 to stream data down to bigdata nodes. The plot below shows strong scaling on 4 - 60 streams of datasets using 18 to 2177 cores (2 - 45 nodes). The best performance observed when reading 60 streams is 500 kHz using 45 nodes.

Potential overheads

• Reading bandwidth on a single node is ~12 GB/s. 
• MPI Intra-node communication was not done over IB

More info: Reading performance

Reading with analysis performance

TBA