Page History
...
2024-06-17 Test for mpi det.calib standard
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Dataset
ds = MPIDataSource('exp=xcsl1030422:run=237')
det = Detector('XcsEndstation.0:Epix10ka2M.0')
ds.break_after(nevents*size)
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
|
Results
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
Ratio: 1.33
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Code Block | ||||
---|---|---|---|---|
| ||||
def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None
"""
Returns calibrated epix10ka data
- gets constants
- gets raw data
- evaluates (code - pedestal - offset)
- applys common mode correction if turned on
- apply gain factor
Parameters
- det (psana.Detector) - Detector object
- evt (psana.Event) - Event object
- cmpars (tuple) - common mode parameters
= None - use pars from calib directory
= cmpars=(<alg>, <mode>, <maxcorr>)
alg is not used
mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST
i.e: cmpars=(7,0,100) or (7,2,100)
- **kwa - used here and passed to det.mask_comb
- nda_raw - substitute for det.raw(evt)
- mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default
- mask - user defined mask passed as optional parameter
"""
t00 = time()
logger.debug('In calib_epix10ka_any')
nda_raw = kwa.get('nda_raw', None)
raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16
if raw is None: return None
t01 = time()
cmp = det.common_mode(evt) if cmpars is None else cmpars
gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384)
peds = det.pedestals(evt) # - 4d pedestals
if gain is None: return None # gain = np.ones_like(peds) # - 4d gains
if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains
t02 = time()
store = ue.dic_store.get(det.name, None)
if store is None:
logger.debug('create store for detector %s' % det.name)
store = ue.dic_store[det.name] = ue.Storage()
# do ONCE this initialization
logger.debug(info_ndarr(raw, '\n raw ') | ||||
Code Block | ||||
| ||||
def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None """ Returns calibrated epix10ka data - gets constants - gets raw data - evaluates (code - pedestal - offset) - applys common mode correction if turned on - apply gain factor Parameters - det (psana.Detector) - Detector object - evt (psana.Event) - Event object - cmpars (tuple) - common mode parameters = None - use pars from calib directory = cmpars=(<alg>, <mode>, <maxcorr>) alg is not used mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST i.e: cmpars=(7,0,100) or (7,2,100) - **kwa - used here and passed to det.mask_comb - nda_raw - substitute for det.raw(evt) - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default - mask - user defined mask passed as optional parameter """ t00 = time() logger.debug('In calib_epix10ka_any') nda_raw = kwa.get('nda_raw', None) raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16 if raw is None: return None t01 = time() cmp = det.common_mode(evt) if cmpars is None else cmpars gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384) peds = det.pedestals(evt) # - 4d pedestals if gain is None: return None # gain = np.ones_like(peds) # - 4d gains if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains t02 = time() store = ue.dic_store.get(det.name, None) if store is None: logger.debug('create store for detector %s' % det.name) store = ue.dic_store[det.name] = ue.Storage() # do ONCE this initialization logger.debug(info_ndarr(raw, '\n raw ')\ +info_ndarr(gain, '\n gain')\ +info_ndarr(peds, '\n peds')) store.gfac = divide_protected(np.ones_like(gain), gain) store.arr1 = np.ones_like(raw, dtype=np.int8) logger.debug(info_ndarr(store.gfac, '\n gfac ')) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1 # L - center t03 = time() gfac = store.gfac gmaps = ue.gain_maps_epix10ka_any(det, raw) if gmaps is None: return None gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps factor = np.select(gmaps,\ (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\ +info_ndarr(gain, '\n gain')\ gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec #==================== TEST RETURN MAP OF PIXEL GAIN MODES +info_ndarr(peds, '\n #return map_pixel_gain_mode1(gmaps peds')) #==================== pedest = np.select(gmaps,\ store.gfac = divide_protected(np.ones_like(gain), gain) store.arr1 = np.ones_like(raw, dtype=np.int8) (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\ logger.debug(info_ndarr(store.gfac, '\n gfac ')) peds[4,:], peds[5,:], peds[6,:]), default=0) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' t04 = time() #store.gf4 = store.counter += 1 if not store.counter%100:np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial logger.debug(ue.info_gain_mode_arrays1(gmaps)\ #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1 # L - center t03 +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps))= time() arrfgfac = np.array(raw & ue.M14, dtype=np.float32) - pedeststore.gfac t05gmaps = time() ue.gain_maps_epix10ka_any(det, raw) if store.maskgmaps is None: return None store.mask = det.mask_total(evt, **kwa) mask = store.maskgr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps t06factor = time() np.select(gmaps,\ logger.debug('common-mode correction pars cmp: %s' % str(cmp)) if cmp is not None: mode, cormax = int(cmp[1]), cmp[2] (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\ npixmin = cmp[3] if len(cmp)>3 else 10 if mode>0: gfac[4,:], gfac[5,:], t07 = t0_sec_cm = time()gfac[6,:]), default=1) # 2msec #==================== TEST RETURN #t2_sec_cm = time(MAP OF PIXEL GAIN MODES #return map_pixel_gain_mode1(gmaps) #==================== arr1pedest = store.arr1 # np.ones_likeselect(maskgmaps, dtype=np.uint8) \ grhm = np.select((gr0, gr1, gr3, gr4), (arr1peds[0,:], arr1peds[1,:], arr1peds[2,:], arr1), default=0) peds[3,:],\ gmask = np.bitwise_and(grhm, mask) if mask is not None else grhm if gmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]) peds[4,:], peds[5,:], peds[6,:]), default=0) t04 = time() store.counter += 1 if not store.counter%100: #loggerlogger.debug(ue.info_gain_mode_ndarr(arr1, '\n arr1'))arrays1(gmaps)\ #logger.debug(info_ndarr(grhm, 'XXXX grhm')) logger.debug(info_ndarr(gmask, 'gmask')\+'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest t05 = time() + '\n per panel statistics of cm-corrected pixels: %s'%if store.mask is None: store.mask = det.mask_total(evt, **kwa) mask = store.mask t06 = time() logger.debug('common-mode correction pars cmp: %s' % str(np.sum(gmask, axis=(1,2), dtype=np.uint32)cmp)) if gmaskcmp is not None else None)) : #logger.debug('common-mode, maskcormax massaging= int(seccmp[1]) = %.6f' % (time()-t2_sec_cm)) # 5msec, cmp[2] npixmin = cmp[3] if len(cmp)>3 else 10 if mode>0: t08t07 = t0_sec_cm = time() #sh#t2_sec_cm = time(nsegs, 352, 384) hrowsarr1 = 176store.arr1 # int(352/2np.ones_like(mask, dtype=np.uint8) forgrhm s in range(arrf.shape[0]): t09 = time() = np.select((gr0, gr1, gr3, gr4), (arr1, arr1, arr1, arr1), default=0) gmask if mode & 4: # in banks: (352/2,384/8)=(176,48) pixels= np.bitwise_and(grhm, mask) if mask is not None else grhm if gmask.ndim == 2: ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin) gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]) #logger.debug(info_ndarr(arr1, '\n arr1')) ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) #logger.debug(info_ndarr(grhm, 'XXXX grhm')) t10 = time() logger.debug(info_ndarr(gmask, 'gmask')\ if mode & 1: #+ in'\n rows per bank: 384/8 = 48 pixels # 190ms panel statistics of cm-corrected pixels: %s'% ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin) t11 = time() str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)) #logger.debug('common-mode mask ifmassaging mode & 2: # in cols per bank: 352/2 = 176 pixels # 150ms(sec) = %.6f' % (time()-t2_sec_cm)) # 5msec t08 = time() #sh ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin= (nsegs, 352, 384) hrows = 176 # int(352/2) for s ue.common_mode_colsin range(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin.shape[0]): t09 = time() logger.debug('TIME common-mode correctionif =mode %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp))) & 4: # in banks: (352/2,384/8)=(176,48) pixels t12 = time() res = arrf * factor if mask is None else arrf * factor * mask # gain correction t13 ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin) ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) t10 = time() logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\ +info_ndarr(factor, '\n calib_epix10ka factor')\ if mode & 1: # in rows per bank: 384/8 = 48 pixels # 190ms ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin) +info_ndarr(pedest, '\n t11 calib_epix10ka pedest'))= time() return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13) |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: begin common mode algorithm
- t08: define mask for common mode
- t09: beginning of the loop over panels for cm
- t10: cmc in banks: (352/2,384/8)=(176,48) pixels
- t11: cmc in rows per bank: 384/8 = 48 pixels
- t12: cmc in cols per bank: 352/2 = 176 pixels
- t13: apply gain and mask to output array
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
OR:
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan004
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
t01 t02 t03 t04 t05 t06 t07 t08 t09 t10 t11 t12 t13
1-core: 0.326 0.021 0.001 10.921 1.338 0.001 0.023 3.960 319.467 3.471 9.426 8.430 1.558
0.294 0.020 0.001 11.150 1.275 0.001 0.023 3.968 319.375 3.443 9.424 8.438 1.607
0.316 0.021 0.001 11.103 1.291 0.001 0.024 3.974 319.814 3.460 9.436 8.443 1.567
80-core: 0.373 0.024 0.002 12.639 1.674 0.001 0.017 4.744 384.576 4.154 11.336 10.108 2.100
0.385 0.025 0.002 12.725 1.558 0.001 0.017 4.579 382.630 4.192 11.463 10.118 2.160
if mode & 2: # in cols per bank: 352/2 = 176 pixels # 150ms
ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin)
ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin)
logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp)))
t12 = time()
res = arrf * factor if mask is None else arrf * factor * mask # gain correction
t13 = time()
logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\
+info_ndarr(factor, '\n calib_epix10ka factor')\
+info_ndarr(pedest, '\n calib_epix10ka pedest'))
return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13) |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: begin common mode algorithm
- t08: define mask for common mode
- t09: beginning of the loop over panels for cm
- t10: cmc in banks: (352/2,384/8)=(176,48) pixels
- t11: cmc in rows per bank: 384/8 = 48 pixels
- t12: cmc in cols per bank: 352/2 = 176 pixels
- t13: apply gain and mask to output array
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
The same as in previous, then
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan004
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
t01 t02 t03 t04 t05 t06 t07 t08 t09 t10 t11 t12 t13
1-core: 0.326 0.021 0.001 10.921 1.338 0.001 0.023 3.960 319.467 3.471 9.426 8.430 1.558
0.294 0.020 0.001 11.150 1.275 0.001 0.023 3.968 319.375 3.443 9.424 8.438 1.607
0.316 0.021 0.001 11.103 1.291 0.001 0.024 3.974 319.814 3.460 9.436 8.443 1.567
80-core,rank 0: 0.373 0.024 0.002 12.639 1.674 0.001 0.017 4.744 384.576 4.154 11.336 10.108 2.100
0.385 0.025 0.002 12.725 1.558 0.001 0.017 4.579 382.630 4.192 11.463 10.118 2.160
0.396 0.024 0.002 13.547 1.691 0.001 0.018 4.621 381.572 4.092 11.252 10.070 2.106
Ratio: 1.22. 1.2. - 1.14. 1.29 - - 1.16. - 1.20 1.20. 1.20. 1.34 0.396 0.024 0.002 13.547 1.691 0.001 0.018 4.621 381.572 4.092 11.252 10.070 2.106