Page History
...
2024-06-17 Test for mpi det.calib standard
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Dataset
ds = MPIDataSource('exp=xcsl1030422:run=237')
det = Detector('XcsEndstation.0:Epix10ka2M.0')
ds.break_after(nevents*size)
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
|
Results
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
Per event time, sec, for rank 0:
hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014
hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022
Ratio: 1.33
2024-06-18 Test for mpi det.calib local with inserted timeing points
Code of local calib_epix10ka_any
Code Block | ||||
---|---|---|---|---|
| ||||
def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None
"""
Returns calibrated epix10ka data
- gets constants
- gets raw data
- evaluates (code - pedestal - offset)
- applys common mode correction if turned on
- apply gain factor
Parameters
- det (psana.Detector) - Detector object
- evt (psana.Event) - Event object
- cmpars (tuple) - common mode parameters
= None - use pars from calib directory
= cmpars=(<alg>, <mode>, <maxcorr>)
alg is not used
mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST
i.e: cmpars=(7,0,100) or (7,2,100)
- **kwa - used here and passed to det.mask_comb
- nda_raw - substitute for det.raw(evt)
- mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default
- mask - user defined mask passed as optional parameter
"""
t00 = time()
logger.debug('In calib_epix10ka_any')
nda_raw = kwa.get('nda_raw', None)
raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16
if raw is None: return None
t01 = time()
cmp = det.common_mode(evt) if cmpars is None else cmpars
gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384)
peds = det.pedestals(evt) # - 4d pedestals
if gain is None: return None # gain = np.ones_like(peds) # - 4d gains
if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains
t02 = time()
store = ue.dic_store.get(det.name, None)
if store is None:
logger.debug('create store for detector %s' % det.name)
store = ue.dic_store[det.name] = ue.Storage()
# do ONCE this initialization
logger.debug(info_ndarr(raw, '\n raw ')\
+info_ndarr(gain, '\n gain')\
| ||||
Code Block | ||||
| ||||
def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None """ Returns calibrated epix10ka data - gets constants - gets raw data - evaluates (code - pedestal - offset) - applys common mode correction if turned on - apply gain factor Parameters - det (psana.Detector) - Detector object - evt (psana.Event) - Event object - cmpars (tuple) - common mode parameters = None - use pars from calib directory = cmpars=(<alg>, <mode>, <maxcorr>) alg is not used mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST i.e: cmpars=(7,0,100) or (7,2,100) - **kwa - used here and passed to det.mask_comb - nda_raw - substitute for det.raw(evt) - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default - mask - user defined mask passed as optional parameter """ t00 = time() logger.debug('In calib_epix10ka_any') nda_raw = kwa.get('nda_raw', None) raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16 if raw is None: return None t01 = time() cmp = det.common_mode(evt) if cmpars is None else cmpars gain = det.gain(evt) # - 4d gains (7, <nsegs>, 352, 384) peds = det.pedestals(evt) # - 4d pedestals if gain is None: return None # gain = np.ones_like(peds) # - 4d gains if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains t02 = time() store = ue.dic_store.get(det.name, None) if store is None: logger.debug('create store for detector %s' % det.name) store = ue.dic_store[det.name] = ue.Storage() # do ONCE this initialization logger.debug(info_ndarr(raw, '\n raw ')\ +info_ndarr(gain, '\n gain')\ +info_ndarr(peds, '\n peds')) store.gfac = divide_protected(np.ones_like(gain), gain) store.arr1 = np.ones_like(raw, dtype=np.int8) logger.debug(info_ndarr(store.gfac, '\n gfac ')) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1 # L - center t03 = time() gfac = store.gfac gmaps = ue.gain_maps_epix10ka_any(det, raw) if gmaps is None: return None gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps factor = np.select(gmaps,\ (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\+info_ndarr(peds, '\n peds')) store.gfac = divide_protected(np.ones_like(gain), gain) store.arr1 gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec #==================== TEST RETURN MAP OF PIXEL GAIN MODES #return map_pixel_gain_mode1(gmaps) #==================== pedest = np.select(gmaps,\= np.ones_like(raw, dtype=np.int8) logger.debug(info_ndarr(store.gfac, '\n gfac ')) # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L' #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1 # L - center (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\ t03 = time() gfac = store.gfac gmaps = ue.gain_maps_epix10ka_any(det, raw) if gmaps is None: return None peds[4,:]gr0, gr1, peds[5,:]gr2, gr3, peds[6,:]), default=0)gr4, gr5, gr6 = gmaps t04factor = time() np.select(gmaps,\ store.counter += 1 if not store.counter%100: logger.debug(ue.info_gain_mode_arrays1(gmaps)\ (gfac[0,:], gfac[1,:], +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) gfac[2,:], gfac[3,:],\ arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest t05 = time() if store.mask is None: store.mask = det.mask_total(evt, **kwa) mask = store.maskgfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec t06 = time() logger.debug('common-mode correction pars cmp: %s' % str(cmp)) if cmp is not None:#==================== TEST RETURN MAP OF PIXEL GAIN MODES #return map_pixel_gain_mode1(gmaps) #==================== pedest mode, cormax = int(cmp[1]), cmp[2]np.select(gmaps,\ npixmin = cmp[3] if len(cmp)>3 else 10 if mode>0: (peds[0,:], t07 = t0_sec_cm = time() peds[1,:], peds[2,:], peds[3,:],\ #t2_sec_cm = time() arr1 = store.arr1 # np.ones_like(mask, dtype=np.uint8) peds[4,:], peds[5,:], peds[6,:]), default=0) t04 = time() grhm = np.select((gr0, gr1, gr3, gr4), (arr1, arr1, arr1, arr1), default=0) gmask = np.bitwise_and(grhm, mask) if mask is not None else grhm store.counter += 1 if not store.counter%100: logger.debug(ue.info_gain_mode_arrays1(gmaps)\ if gmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]+'\n'+ue.info_pixel_gain_mode_statistics1(gmaps)) arrf = np.array(raw & #logger.debug(info_ndarr(arr1, '\n arr1'))ue.M14, dtype=np.float32) - pedest t05 = time() #logger.debug(info_ndarr(grhm, 'XXXX grhm')) if store.mask is None: store.mask = loggerdet.debug(infomask_ndarrtotal(gmaskevt, 'gmask')\**kwa) mask = store.mask t06 = time() + '\n per panel statistics of cm-corrected pixels: %s'% logger.debug('common-mode correction pars cmp: %s' % str(cmp)) if cmp is not None: mode, cormax = int(cmp[1]), cmp[2] npixmin = str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)) cmp[3] if len(cmp)>3 else 10 if mode>0: #logger.debug('common-mode mask massaging (sec) = %.6f' % (time()-t2t07 = t0_sec_cm = time() #t2_sec_cm)) #= 5msectime() t08arr1 = time() store.arr1 # np.ones_like(mask, dtype=np.uint8) #shgrhm = np.select(nsegs(gr0, 352gr1, 384) gr3, gr4), (arr1, arr1, arr1, hrowsarr1), = 176 # int(352/2default=0) forgmask s in range(arrf.shape[0]): t09 = time() = np.bitwise_and(grhm, mask) if mask is not None else grhm if mode & 4: # in banks: (352/2,384/8)=(176,48) pixelsgmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1]) #logger.debug(info_ndarr(arr1, '\n arr1')) ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin #logger.debug(info_ndarr(grhm, 'XXXX grhm')) ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) logger.debug(info_ndarr(gmask, 'gmask')\ t10 = time() + if'\n mode & 1: # in rows per bank: 384/8 = 48 pixels # 190ms per panel statistics of cm-corrected pixels: %s'% ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin) str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)) #logger.debug('common-mode mask t11massaging (sec) = %.6f' % (time()-t2_sec_cm)) # 5msec t08 = if mode & 2: # in cols per bank: 352/2time() #sh = (nsegs, 352, 384) hrows = 176 pixels # 150msint(352/2) for s ue.common_mode_colsin range(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin) .shape[0]): t09 = time() if ue.common_mode_cols(arrf[s,hrows:,:], mask=mode & 4: # in banks: (352/2,384/8)=(176,48) pixels ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp))) t12 ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin) t10 = time() res = arrfif *mode factor& if1: mask# isin Nonerows elseper arrf * factor * mask # gain correction bank: 384/8 = 48 pixels # 190ms t13 = time() logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin) t11 = time() +info_ndarr(factor, '\n calib_epix10ka factor')\ if mode & 2: # in cols per bank: 352/2 = 176 +info_ndarr(pedest, '\n calib_epix10ka pedest')) pixels # 150ms return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13) |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: begin common mode algorithm
- t08: define mask for common mode
- t09: beginning of the loop over panels for cm
- t10: cmc in banks: (352/2,384/8)=(176,48) pixels
- t11: cmc in rows per bank: 384/8 = 48 pixels
- t12: cmc in cols per bank: 352/2 = 176 pixels
- t13: apply gain and mask to output array
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$
In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
OR:
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan050
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin)
ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin)
logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp)))
t12 = time()
res = arrf * factor if mask is None else arrf * factor * mask # gain correction
t13 = time()
logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\
+info_ndarr(factor, '\n calib_epix10ka factor')\
+info_ndarr(pedest, '\n calib_epix10ka pedest'))
return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13) |
Timing points
- t00: at the entrance
- t01: det.raw
- t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
- t03: access constants at store - 1-st event consumption
- t04: factor and pedestals for gain ranges using np.select
- t05: apply data-bit mask to raw and subtract pedestals
- t06: access mask - SHOULD BE DONE ONCE
- t07: begin common mode algorithm
- t08: define mask for common mode
- t09: beginning of the loop over panels for cm
- t10: cmc in banks: (352/2,384/8)=(176,48) pixels
- t11: cmc in rows per bank: 384/8 = 48 pixels
- t12: cmc in cols per bank: 352/2 = 176 pixels
- t13: apply gain and mask to output array
List of commands
Code Block | ||||
---|---|---|---|---|
| ||||
The same as in previous, then
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2 |
Results
on sdfmilan004
mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
per event time difference betwee points in msec
t01 t02 t03 t04 t05 t06 t07 t08 t09 t10 t11 t12 t13
1-core: 0.326 0.021 0.001 10.921 1.338 0.001 0.023 3.960 319.467 3.471 9.426 8.430 1.558
0.294 0.020 0.001 11.150 1.275 0.001 0.023 3.968 319.375 3.443 9.424 8.438 1.607
0.316 0.021 0.001 11.103 1.291 0.001 0.024 3.974 319.814 3.460 9.436 8.443 1.567
80-core,rank 0: 0.373 0.024 0.002 12.639 1.674 0.001 0.017 4.744 384.576 4.154 11.336 10.108 2.100
0.385 0.025 0.002 12.725 1.558 0.001 0.017 4.579 382.630 4.192 11.463 10.118 2.160
0.396 0.024 0.002 13.547 1.691 0.001 0.018 4.621 381.572 4.092 11.252 10.070 2.106
Ratio: 1.22. t01 t02 1.2. t03 t04 t05 - t06 t07 t08 1.14. t09 t10 t11 t12 t131-core: 1.29 - - 1.000 0.035 0.003 27.343 3.861 0.002 0.024 7 1.988 55316.949 7.998 16.733 12.358 4.70480-core: 0.373 0 1.024 0.002 12.639 20 1.674 020.001 0.017 4 1.744 38420.576 4.154 11.336 10.108 2.100 1.34