Page History
Content
Table of Contents |
---|
2024-06-
...
17 Test for mpi det.calib
...
standard
Dataset
ds = MPIDataSource('exp=xcsl1030422:run=237')
det = Detector('XcsEndstation.0:Epix10ka2M.0')
ds.break_after(nevents*size)
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
|
Results
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
Ratio: 1.33
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Code Block | ||||
---|---|---|---|---|
| ||||
def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None """ Returns calibrated epix10ka data - gets constants - gets raw data - evaluates (code - pedestal - offset) - applys common mode correction if turned on - apply gain factor Parameters - det (psana.Detector) - Detector object - evt (psana.Event) - Event object - cmpars (tuple) - common mode parameters = None - use pars from calib directory = cmpars=(<alg>, <mode>, <maxcorr>) alg is not used mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST i.e: cmpars=(7,0,100) or (7,2,100) - **kwa - used here and passed to det.mask_comb - nda_raw - substitute for det.raw(evt) - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default - mask - user defined mask passed as optional parameter """ t00 = time() logger.debug('In calib_epix10ka_any') nda_raw = kwa.get('nda_raw', None) raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16 if raw is None: return None t01 = time() cmp = det.common_mode(evt) if cmpars is None else cmpars gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384) peds = det.pedestals(evt) # - 4d pedestals if gain is None: return None # gain = np.ones_like(peds) # - 4d gains if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains t02 = time() store = ue.dic_store.get(det.name, None) if store is None: logger.debug('create store for detector %s' % det.name) store = ue.dic_store[det.name] = ue.Storage() # do ONCE this initialization logger.debug(info_ndarr(raw, '\n raw ')def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None """ Returns calibrated epix10ka data - gets constants - gets raw data - evaluates (code - pedestal - offset) - applys common mode correction if turned on - apply gain factor Parameters - det (psana.Detector) - Detector object - evt (psana.Event) - Event object - cmpars (tuple) - common mode parameters = None - use pars from calib directory = cmpars=(<alg>, <mode>, <maxcorr>) alg is not used mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST i.e: cmpars=(7,0,100) or (7,2,100) - **kwa - used here and passed to det.mask_comb - nda_raw - substitute for det.raw(evt) - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default - mask - user defined mask passed as optional parameter """ t00 = time() logger.debug('In calib_epix10ka_any') nda_raw = kwa.get('nda_raw', None) raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16 if raw is None: return None t01 = time() cmp = det.common_mode(evt) if cmpars is None else cmpars gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384) peds = det.pedestals(evt) # - 4d pedestals if gain is None: return None # gain = np.ones_like(peds) # - 4d gains if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains t02 = time() store = ue.dic_store.get(det.name, None) if store is None: logger.debug('create store for detector %s' % det.name) store = ue.dic_store[det.name] = ue.Storage() # do ONCE this initialization logger.debug(info_ndarr(raw, '\n raw ')\ +info_ndarr(gain, '\n gain')\ +info_ndarr(peds, '\n peds')) store.gfac = divide_protected(np.ones_like(gain), gain) store.arr1 = np.ones_like(raw, dtype=np.int8) logger.debug(info_ndarr(store.gfac, '\n gfac ')) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1 # L - center t03 = time() gfac = store.gfac gmaps = ue.gain_maps_epix10ka_any(det, raw) if gmaps is None: return None gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps factor = np.select(gmaps,\ +info_ndarr(gain, (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\ '\n gain')\ +info_ndarr(peds, '\n peds')) gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec store.gfac = divide_protected(np.ones_like(gain), gain) #==================== TEST RETURN MAP OFstore.arr1 PIXEL GAIN MODES = np.ones_like(raw, dtype=np.int8) #return map_pixel_gain_mode1(gmaps) #==================== pedest = np.select(gmaps,\ logger.debug(info_ndarr(store.gfac, '\n gfac ')) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial #store.gf6 (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\ = np.ones_like(raw, dtype=np.int32) * 1 # L - center t03 = time() gfac = store.gfac peds[4,:], peds[5,:], peds[6,:]), default=0) gmaps = ue.gain_maps_epix10ka_any(det, raw) t04if = time() gmaps is None: return None store.counter += 1 if not store.counter%100:gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps factor = logger.debug(ue.info_gain_mode_arrays1(gmaps)\ np.select(gmaps,\ +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\ t05 = time() if store.mask is None: store.mask = det.mask_total(evt, **kwa) mask = store.mask t06 = time() logger.debug('common-mode correction pars cmp: %s' % str(cmp))gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec #==================== TEST RETURN MAP OF PIXEL GAIN MODES #return map_pixel_gain_mode1(gmaps) #==================== ifpedest cmp is not None: = np.select(gmaps,\ mode, cormax = int(cmp[1]), cmp[2] npixmin = cmp[3] if len(cmp)>3 else 10 (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\ if mode>0: t0_sec_cm = time() #t2_sec_cmpeds[4,:], peds[5,:], peds[6,:]), default=0) t04 = time() store.counter += 1 arr1 = store.arr1if #not np.ones_like(mask, dtype=np.uint8)store.counter%100: grhm = np.select((gr0, gr1, gr3, gr4), (arr1, arr1, arr1, arr1), default=0) logger.debug(ue.info_gain_mode_arrays1(gmaps)\ gmask = np.bitwise_and(grhm, mask) if mask is not None else grhm +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest t05 = time() if gmaskstore.ndimmask ==is 2None: gmask.shape store.mask = det.mask_total(1,gmask.shape[-2],gmask.shape[-1]) evt, **kwa) mask = store.mask #logger.debug(info_ndarr(arr1, '\n arr1')) t06 = time() #loggerlogger.debug(info_ndarr(grhm, 'XXXX grhm')) logger.debug(info_ndarr(gmask, 'gmask')\'common-mode correction pars cmp: %s' % str(cmp)) if cmp is not None: mode, cormax + '\n per panel statistics of cm-corrected pixels: %s'%= int(cmp[1]), cmp[2] npixmin = cmp[3] if len(cmp)>3 else 10 if mode>0: t07 = t0_sec_cm = time() str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)#t2_sec_cm = time() #logger.debug('common-mode mask massaging (sec)arr1 = %store.6f'arr1 %# (time()-t2_sec_cm)) # 5msec np.ones_like(mask, dtype=np.uint8) #shgrhm = np.select((nsegsgr0, 352gr1, 384) gr3, gr4), (arr1, arr1, arr1, arr1), hrows = 176 # int(352/2default=0) forgmask s in range(arrf.shape[0]): = np.bitwise_and(grhm, mask) if mask is not None else grhm if mode & 4: # in banks: (352/2,384/8)=(176,48) pixelsgmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]) #logger.debug(info_ndarr(arr1, '\n ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin) arr1')) #logger.debug(info_ndarr(grhm, 'XXXX grhm')) logger.debug(info_ndarr(gmask, 'gmask')\ ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) if mode & 1: # in rows per bank: 384/8 = 48 pixels # 190ms + '\n per panel statistics of cm-corrected pixels: %s'% ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin) str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)) #logger.debug('common-mode mask massaging if mode & 2: # in cols per bank: 352/2 = 176 pixels # 150ms(sec) = %.6f' % (time()-t2_sec_cm)) # 5msec t08 = time() #sh ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin= (nsegs, 352, 384) hrows = 176 ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin) # int(352/2) for s in range(arrf.shape[0]): logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp))) t09 = time() res = arrf * factor if maskmode is& None4: else# arrfin * factor * mask # gain correction banks: (352/2,384/8)=(176,48) pixels t07 = time() logger.debug('TOTAL consumed time (sec) = %.6f' % (t07-t00)\ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin) +info_ndarr(factor, '\n calib_epix10ka factor')\ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) t10 = time() +info_ndarr(pedest, '\n calib_epix10ka pedest')) if mode times& =1: t00, t01, t02, t03, t04, t05, t06, t07 # in rows per bank: 384/8 = 48 pixels # 190ms return res, times |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: apply common mode correction if on
List of commands
Code Block |
---|
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
t01 t02 t03 t04 t05 t06 t07
1-core: 0.333 0.025 0.002 10.906 1.438 0.001 140.304
80-core: 0.673 0.039 0.003 17.535 2.406 0.003 178.890
ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin)
t11 = time()
if mode & 2: # in cols per bank: 352/2 = 176 pixels # 150ms
ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin)
ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin)
logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp)))
t12 = time()
res = arrf * factor if mask is None else arrf * factor * mask # gain correction
t13 = time()
logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\
+info_ndarr(factor, '\n calib_epix10ka factor')\
+info_ndarr(pedest, '\n calib_epix10ka pedest'))
return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13) |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: begin common mode algorithm
- t08: define mask for common mode
- t09: beginning of the loop over panels for cm
- t10: cmc in banks: (352/2,384/8)=(176,48) pixels
- t11: cmc in rows per bank: 384/8 = 48 pixels
- t12: cmc in cols per bank: 352/2 = 176 pixels
- t13: apply gain and mask to output array
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
The same as in previous, then
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan004
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
t01 t02 t03 t04 t05 t06 t07 t08 t09 t10 t11 t12 t13
1-core: 0.326 0.021 0.001 10.921 1.338 0.001 0.023 3.960 319.467 3.471 9.426 8.430 1.558
0.294 0.020 0.001 11.150 1.275 0.001 0.023 3.968 319.375 3.443 9.424 8.438 1.607
0.316 0.021 0.001 11.103 1.291 0.001 0.024 3.974 319.814 3.460 9.436 8.443 1.567
80-core,rank 0: 0.373 0.024 0.002 12.639 1.674 0.001 0.017 4.744 384.576 4.154 11.336 10.108 2.100
0.385 0.025 0.002 12.725 1.558 0.001 0.017 4.579 382.630 4.192 11.463 10.118 2.160
0.396 0.024 0.002 13.547 1.691 0.001 0.018 4.621 381.572 4.092 11.252 10.070 2.106
Ratio: 1.22. 1.2. - 1.14. 1.29 - - 1.16. - 1.20 1.20. 1.20. 1.34
References
- Scaling behavior of psana1 - Part 1 - det.calib method in multicore processing with mpi
- Scaling behavior of psana1 - Part 2 - test with command perf stat
- Scaling behavior of psana1 - Part 3 - test for reduced memory consumption
...