Page History
...
2024-06-17 Test for mpi det.calib standard
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Dataset
ds = MPIDataSource('exp=xcsl1030422:run=237')
det = Detector('XcsEndstation.0:Epix10ka2M.0')
ds.break_after(nevents*size)
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python | ||||
Code Block | ||||
title | Detector/examples/test-scaling-mpi-epix10ka2m.py | |||
collapse | true | 1
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
|
Results
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
Ratio: 1.33
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Code Block | ||||
---|---|---|---|---|
| ||||
def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None """ Returns calibrated epix10ka data - gets constants - gets raw data - evaluates (code - pedestal - offset) - applys common mode correction if turned on - apply gain factor Parameters - det (psana.Detector) - Detector object - evt (psana.Event) - Event object - cmpars (tuple) - common mode parameters = None - use pars from calib directory = cmpars=(<alg>, <mode>, <maxcorr>) alg is not used mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST i.e: cmpars=(7,0,100) or (7,2,100) - **kwa - used here and passed to det.mask_comb - nda_raw - substitute for det.raw(evt) - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default - mask - user defined mask passed as optional parameter """ t00 = time() def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None """ Returns calibrated epix10ka data - gets constants - gets raw data - evaluates (code - pedestal - offset) - applys common mode correction if turned on - apply gain factor Parameters - det (psana.Detector) - Detector object - evt (psana.Event) - Event object - cmpars (tuple) - common mode parameters = None - use pars from calib directory = cmpars=(<alg>, <mode>, <maxcorr>) alg is not used mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST i.e: cmpars=(7,0,100) or (7,2,100) - **kwa - used here and passed to det.mask_comb - nda_raw - substitute for det.raw(evt) - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default - mask - user defined mask passed as optional parameter """ t00 = time() logger.debug('In calib_epix10ka_any') nda_raw = kwa.get('nda_raw', None) raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16 if raw is None: return None t01 = time() cmp = det.common_mode(evt) if cmpars is None else cmpars gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384) peds = det.pedestals(evt) # - 4d pedestals if gain is None: return None # gain = np.ones_like(peds) # - 4d gains if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains t02 = time() store = ue.dic_store.get(det.name, None) if store is None: logger.debug('create store for detector %s' % det.nameIn calib_epix10ka_any') nda_raw = kwa.get('nda_raw', None) storeraw = ue.dic_store[det.name] = ue.Storage() # do ONCE this initialization logger.debug(info_ndarr(raw, '\n raw ')\ +info_ndarr(gain, '\n gain')\ raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16 if raw is None: return None t01 = time() cmp = det.common_mode(evt) if cmpars is None else cmpars gain = det.gain(evt) # - 4d gains +info_ndarr(peds7, '\n peds')) <nsegs>, 352, 384) peds = det.pedestals(evt) # store.gfac- = divide_protected(np.ones_like(gain), gain) store.arr14d pedestals if gain is None: return None # gain = np.ones_like(raw, dtype=np.int8) peds) # - 4d gains logger.debug(info_ndarr(store.gfac, '\n gfac ')) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains t02 = time() store = ue.dic_store.get(det.name, None) if store is None: #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1 # L - centerlogger.debug('create store for detector %s' % det.name) store = ue.dic_store[det.name] = ue.Storage() t03 = time() # do gfacONCE =this store.gfacinitialization gmaps = ue.gain_maps_epix10ka_any(det, raw) logger.debug(info_ndarr(raw, '\n raw ')\ if gmaps is None: return None gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps +info_ndarr(gain, '\n gain')\ factor = np.select(gmaps,\ +info_ndarr(peds, '\n peds')) store.gfac = divide_protected(np.ones_like(gain), gain) store.arr1 = (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\np.ones_like(raw, dtype=np.int8) logger.debug(info_ndarr(store.gfac, '\n gfac ')) gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec #==================== TEST RETURN MAP OF PIXEL GAIN MODES# 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial #return map_pixel_gain_mode1(gmaps) #store.gf6 #==================== pedest = np.select(gmaps,\ = np.ones_like(raw, dtype=np.int32) * 1 # L - center t03 = time() gfac = store.gfac gmaps (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\ peds[4,:], peds[5,:], peds[6,:]), default=0) t04 = time() store.counter += 1 if not store.counter%100: logger.debug(ue.info_gain_mode_arrays1(gmaps)\ +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest t05 = time() if store.mask is None: = ue.gain_maps_epix10ka_any(det, raw) if gmaps is None: return None gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps factor = np.select(gmaps,\ (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\ gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec #==================== TEST RETURN MAP OF PIXEL GAIN MODES #return map_pixel_gain_mode1(gmaps) #==================== pedest = np.select(gmaps,\ store.mask = det.mask_total(evt, **kwa) mask = store.mask (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\ t06 = time() logger.debug('common-mode correction pars cmp: %s' % str(cmp)) if cmp is not None: peds[4,:], peds[5,:], peds[6,:]), default=0) mode, cormaxt04 = inttime(cmp[1]), cmp[2]) store.counter += 1 npixmin = cmp[3] if len(cmp)>3 else 10 if mode>0 not store.counter%100: t0_sec_cm = time()logger.debug(ue.info_gain_mode_arrays1(gmaps)\ #t2_sec_cm = time() arr1 = store.arr1 # np.ones_like(mask+'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) arrf = np.array(raw & ue.M14, dtype=np.uint8float32) - pedest t05 grhm = np.select((gr0,time() gr1, if gr3,store.mask gr4), (arr1, arr1, arr1, arr1), default=0)is None: gmaskstore.mask = npdet.bitwisemask_andtotal(grhmevt, mask**kwa) if mask is notmask None else grhm = store.mask t06 = time() if gmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]logger.debug('common-mode correction pars cmp: %s' % str(cmp)) if cmp is #logger.debug(info_ndarr(arr1, '\n arr1')) not None: mode, cormax = #logger.debug(info_ndarr(grhm, 'XXXX grhm')) int(cmp[1]), cmp[2] npixmin = cmp[3] if logger.debug(info_ndarr(gmask, 'gmask')\ len(cmp)>3 else 10 if mode>0: t07 = t0_sec_cm + '\n= time() per panel statistics of #t2_sec_cm-corrected pixels: %s'%= time() arr1 = store.arr1 # np.ones_like(mask, dtype=np.uint8) grhm = str(np.sum(gmask, axis=(1,2), dtype=np.uint32select((gr0, gr1, gr3, gr4), (arr1, arr1, arr1, arr1), default=0) gmask = np.bitwise_and(grhm, mask) if gmaskmask is not None else None))grhm if #logger.debug('common-mode mask massaging (sec) = %.6f' % (time()-t2_sec_cm)) # 5msecgmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]) #sh = (nsegs, 352, 384#logger.debug(info_ndarr(arr1, '\n arr1')) hrows = 176 # int(352/2#logger.debug(info_ndarr(grhm, 'XXXX grhm')) for s in range(arrf.shape[0]): logger.debug(info_ndarr(gmask, 'gmask')\ if mode & 4: # in banks: (352/2,384/8)=(176,48) pixels + '\n per panel statistics of cm-corrected pixels: %s'% ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin) str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)) ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) if mode & 1: # in rows per bank: 384/8 = 48 pixels # 190ms#logger.debug('common-mode mask massaging (sec) = %.6f' % (time()-t2_sec_cm)) # 5msec t08 = time() #sh = (nsegs, 352, 384) hrows = 176 ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin# int(352/2) for s in range(arrf.shape[0]): t09 = time() if mode & 24: # in cols per bankbanks: (352/2 = 176 pixels # 150ms,384/8)=(176,48) pixels ue.common_mode_2d_hsplit_colsnbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin) ue.common_mode_cols2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) logger.debug('TIME common-mode correctiont10 = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp))) time() if mode & 1: res# =in arrfrows *per factor if mask is None else arrf * factor * mask # gain correction bank: 384/8 = 48 pixels # 190ms t07 = time( ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin) logger.debug('TOTAL consumed time (sec) = %.6f' % (t07-t00)\ t11 = time() if mode & 2: # +info_ndarr(factor, '\n calib_epix10ka factor')\ in cols per bank: 352/2 = 176 pixels # 150ms +info_ndarr(pedest, '\n calib_epix10ka pedest')) ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin) times = t00, t01, t02, t03, t04, t05, t06, t07 return res, times |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: apply common mode correction if on
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
OR:
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan050
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin)
logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp)))
t12 = time()
res = arrf * factor if mask is None else arrf * factor * mask # gain correction
t13 = time()
logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\
+info_ndarr(factor, '\n calib_epix10ka factor')\
+info_ndarr(pedest, '\n calib_epix10ka pedest'))
return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13) |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: begin common mode algorithm
- t08: define mask for common mode
- t09: beginning of the loop over panels for cm
- t10: cmc in banks: (352/2,384/8)=(176,48) pixels
- t11: cmc in rows per bank: 384/8 = 48 pixels
- t12: cmc in cols per bank: 352/2 = 176 pixels
- t13: apply gain and mask to output array
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
The same as in previous, then
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan004
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
t01 t02 t03 t04 t05 t06 t07 t08 t09 t10 t11 t12 t13
1-core: 0.326 0.021 0.001 10.921 1.338 0.001 0.023 3.960 319.467 3.471 9.426 8.430 1.558
0.294 0.020 0.001 11.150 1.275 0.001 0.023 3.968 319.375 3.443 9.424 8.438 1.607
0.316 0.021 0.001 11.103 1.291 0.001 0.024 3.974 319.814 3.460 9.436 8.443 1.567
80-core,rank 0: 0.373 0.024 0.002 12.639 1.674 0.001 0.017 4.744 384.576 4.154 11.336 10.108 2.100
0.385 0.025 0.002 12.725 1.558 0.001 0.017 4.579 382.630 4.192 11.463 10.118 2.160
0.396 0.024 0.002 13.547 1.691 0.001 0.018 4.621 381.572 4.092 11.252 10.070 2.106
Ratio: 1.22. t01 t02 t03 t04 t05 t06 t071-core: 1.2. - 1.14. 1.29 - - 0.333 0.025 0 1.002 1016.906 1.438 0.001 140.30480-core: - 0 1.673 0.039 0.003 17.535 2.406 0.003 178.89020 1.20. 1.20. 1.34