Mar. 2, 2021
Code:
import datetime as dt
t = dt.datetime.now()
print('Start at',t.strftime('%H:%M:%S'))
from psana import *
import sys
myrun=sys.argv[1]
t = dt.datetime.now()
print('Import done at',t.strftime('%H:%M:%S'))
dsource = MPIDataSource('exp=cxilu9218:run='+myrun+':smd:dir=/cds/data/drpsrcf/cxi/cxilu9218/xtc')
jungfraudet = Detector('jungfrau4M')
t = dt.datetime.now()
print('Detector done at',t.strftime('%H:%M:%S'))
for nevt,evt in enumerate(dsource.events()):
calib = jungfraudet.calib(evt)
t = dt.datetime.now()
print('Job done at',t.strftime('%H:%M:%S'))
Results for cxilu9218 run 237:
| Nic Speed (Gb/s) | Nodes | Cores/Node | Import Time (s) | Detector Creation Time (s) | Analysis Time (MM:SS) |
|---|---|---|---|---|---|
| 10 | 1 | 120 | 6 | 36 | 26:57 |
| 10 | 1 | 64 | 10 | 23 | 24:37 |
| 10 | 2 | 64 | 8 | 22 | 13:17 |
| 10 | 2 | 120 | 6 | 41 | 14:02 |
| 1 | 1 | 64 | 15 | 3 | 24:34 |
| 1 | 1 | 120 | 8 | 40 | 26:42 |
A similar test on psanaq:
| Nodes | Cores/Node | Import Time (s) | Detector Creation Time (s) | Analysis Time (HH:MM:SS) |
|---|---|---|---|---|
| 2 | 12 | 16 | 9 | 2:05:53 |
Conclusions
- Unlike Jan. 27 measurements below the hyper threading cores didn't seem to help. Feels like "operator error" on my part?
- Nodes with 10Gb NICs vs. 1Gb NICs didn't make a significant difference
- Best configuration seems to be 2 nodes with 64 cores/node.
- "import psana" times fairly small. Detector creation times (fetching calibration constants?) larger.
Jan. 27, 2021
Code:
from psana import *
dsource = MPIDataSource('exp=cxilu9218:run=237:smd:dir=/cds/drpsrcf/cxi/cxilu921
8/xtc/')
jungfraudet = Detector('jungfrau4M')
smldata = dsource.small_data('/cds/drpsrcf/cxi/cxilu9218/scratch/run237.h5',gath
er_interval=100)
partial_run_sum = None
for nevt,evt in enumerate(dsource.events()):
calib = jungfraudet.calib(evt)
if calib is None: continue
jungfrau_sum = calib.sum() # number
smldata.event(jungfrau_sum=jungfrau_sum)
Results:
| Nodes | Cores/Node | Time (HH:MM:SS) |
|---|---|---|
| 2 | 64 | 16:06 |
| 1 | 64 | 27:31 |
| 1 | 32 | 1:43:29 |
| 2 | 128 | 16:11 |
| 1 | 120 | 16:45 |
| 2 | 240 | 19:17 |
Redo these tests on March 6, 2021. Can't reproduce the benefit of the hyper threading cores:
JobID Elapsed NodeList NCPUS ------------ ---------- --------------- ---------- 1707 00:28:49 drp-srcf-cmp028 120 1707.batch 00:28:49 drp-srcf-cmp028 120 1708 00:27:15 drp-srcf-cmp028 64 1708.batch 00:27:15 drp-srcf-cmp028 64 1710 00:30:46 drp-srcf-cmp028 120 1710.batch 00:30:46 drp-srcf-cmp028 120
Overview
Content Tools