Content

Table of Contents

2024-06-

...

17 Test for mpi det.calib

...

standard

Dataset

ds = MPIDataSource('exp=xcsl1030422:run=237')
det = Detector('XcsEndstation.0:Epix10ka2M.0')
ds.break_after(nevents*size)

List of commands

Code Block

title	Detector/examples/test-scaling-mpi-epix10ka2m.pyList of test commands
collapse	true

> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$


In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel

mpirun -n   1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n  80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1

Results

ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1

ana-4.0.62-py3 [dubrovin@sdfmilan219:~/LCLS/con-py3]$ mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1

Per event time, sec, for rank 0:

hostname:sdfmilan219 rank:000 cpu:000 cmt:16p-v1 proc time (sec) mean: 0.1559 +/- 0.0020 rms: 0.0032 +/- 0.0014

hostname:sdfmilan219 rank:000 cpu:014 cmt:16p-v1 proc time (sec) mean: 0.2079 +/- 0.0032 rms: 0.0109 +/- 0.0022

Ratio: 1.33

2024-06-18 Test for mpi det.calib local with inserted timeing points

Code of local calib_epix10ka_any

Code Block

title	Detector/examples/test-scaling-mpi-epix10ka2m.py
collapse	true

  def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None
    """
    Returns calibrated epix10ka data

    - gets constants
    - gets raw data
    - evaluates (code - pedestal - offset)
    - applys common mode correction if turned on
    - apply gain factor

    Parameters

    - det (psana.Detector) - Detector object
    - evt (psana.Event)    - Event object
    - cmpars (tuple) - common mode parameters
          = None - use pars from calib directory
          = cmpars=(<alg>, <mode>, <maxcorr>)
            alg is not used
            mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST
            i.e: cmpars=(7,0,100) or (7,2,100)
    - **kwa - used here and passed to det.mask_comb
      - nda_raw - substitute for det.raw(evt)
      - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default
      - mask - user defined mask passed as optional parameter
    """

    t00 = time()

    logger.debug('In calib_epix10ka_any')

    nda_raw = kwa.get('nda_raw', None)
    raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16
    if raw is None: return None

    t01 = time()

    cmp  = det.common_mode(evt) if cmpars is None else cmpars
    gain = det.gain(evt)      # - 4d gains  (7, <nsegs>, 352, 384)
    peds = det.pedestals(evt) # - 4d pedestals
    if gain is None: return None # gain = np.ones_like(peds)  # - 4d gains
    if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains

    t02 = time()

    store = ue.dic_store.get(det.name, None)

    if store is None:
        logger.debug('create store for detector %s' % det.name)
        store = ue.dic_store[det.name] = ue.Storage()

        # do ONCE this initialization
        logger.debug(info_ndarr(raw,  '\n  raw ')def calib_epix10ka_any(det, evt, cmpars=None, **kwa): # cmpars=(7,2,10,10), mbits=None, mask=None, nda_raw=None
    """
    Returns calibrated epix10ka data

    - gets constants
    - gets raw data
    - evaluates (code - pedestal - offset)
    - applys common mode correction if turned on
    - apply gain factor

    Parameters

    - det (psana.Detector) - Detector object
    - evt (psana.Event)    - Event object
    - cmpars (tuple) - common mode parameters
          = None - use pars from calib directory
          = cmpars=(<alg>, <mode>, <maxcorr>)
            alg is not used
            mode =0-correction is not applied, =1-in rows, =2-in cols-WORKS THE BEST
            i.e: cmpars=(7,0,100) or (7,2,100)
    - **kwa - used here and passed to det.mask_comb
      - nda_raw - substitute for det.raw(evt)
      - mbits - deprecated parameter of the det.mask_comb(...), det.mask_v2 is used by default
      - mask - user defined mask passed as optional parameter
    """

    t00 = time()

    logger.debug('In calib_epix10ka_any')

    nda_raw = kwa.get('nda_raw', None)
    raw = det.raw(evt) if nda_raw is None else nda_raw # shape:(352, 384) or suppose to be later (<nsegs>, 352, 384) dtype:uint16
    if raw is None: return None

    t01 = time()

    cmp  = det.common_mode(evt) if cmpars is None else cmpars
    gain = det.gain(evt)      # - 4d gains  (7, <nsegs>, 352, 384)
    peds = det.pedestals(evt) # - 4d pedestals
    if gain is None: return None # gain = np.ones_like(peds)  # - 4d gains
    if peds is None: return None # peds = np.zeros_like(peds) # - 4d gains

    t02 = time()

    store = ue.dic_store.get(det.name, None)

    if store is None:
        logger.debug('create store for detector %s' % det.name)
        store = ue.dic_store[det.name] = ue.Storage()

        # do ONCE this initialization
        logger.debug(info_ndarr(raw,  '\n  raw ')\
                    +info_ndarr(gain, '\n  gain')\
                    +info_ndarr(peds, '\n  peds'))

        store.gfac = divide_protected(np.ones_like(gain), gain)
        store.arr1 = np.ones_like(raw, dtype=np.int8)

        logger.debug(info_ndarr(store.gfac,  '\n  gfac '))

        # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L'
        #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial
        #store.gf6 = np.ones_like(raw, dtype=np.int32) * 1    # L - center

    t03 = time()

    gfac = store.gfac

    gmaps = ue.gain_maps_epix10ka_any(det, raw)
    if gmaps is None: return None
    gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps

    factor = np.select(gmaps,\
                    +info_ndarr(gain,   (gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\
'\n  gain')\
                    +info_ndarr(peds, '\n  peds'))

        gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec
store.gfac = divide_protected(np.ones_like(gain), gain)
    #==================== TEST RETURN MAP OFstore.arr1 PIXEL GAIN MODES
= np.ones_like(raw, dtype=np.int8)

      #return map_pixel_gain_mode1(gmaps)
    #====================

    pedest = np.select(gmaps,\
           logger.debug(info_ndarr(store.gfac,  '\n  gfac '))

        # 'FH','FM','FL','AHL-H','AML-M','AHL-L','AML-L'
        #store.gf4 = np.ones_like(raw, dtype=np.int32) * 0.25 # 0.3333 # M - perefierial
        #store.gf6      (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\
= np.ones_like(raw, dtype=np.int32) * 1    # L - center

    t03 = time()

    gfac = store.gfac

     peds[4,:], peds[5,:], peds[6,:]), default=0)
gmaps = ue.gain_maps_epix10ka_any(det, raw)
    t04if = time()

gmaps is None: return None
    store.counter += 1
    if not store.counter%100:gr0, gr1, gr2, gr3, gr4, gr5, gr6 = gmaps

    factor =   logger.debug(ue.info_gain_mode_arrays1(gmaps)\
np.select(gmaps,\
                  +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps))

    arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest

(gfac[0,:], gfac[1,:], gfac[2,:], gfac[3,:],\
        t05 = time()

    if store.mask is None:
       store.mask = det.mask_total(evt, **kwa)
    mask = store.mask

    t06 = time()

    logger.debug('common-mode correction pars cmp: %s' % str(cmp))gfac[4,:], gfac[5,:], gfac[6,:]), default=1) # 2msec

    #==================== TEST RETURN MAP OF PIXEL GAIN MODES
    #return map_pixel_gain_mode1(gmaps)
    #====================

    ifpedest cmp is not None:
= np.select(gmaps,\
        mode, cormax = int(cmp[1]), cmp[2]
      npixmin = cmp[3] if len(cmp)>3 else 10   (peds[0,:], peds[1,:], peds[2,:], peds[3,:],\
      if mode>0:
        t0_sec_cm = time()
        #t2_sec_cmpeds[4,:], peds[5,:], peds[6,:]), default=0)

    t04 = time()

    store.counter += 1
  arr1 = store.arr1if #not np.ones_like(mask, dtype=np.uint8)store.counter%100:
        grhm = np.select((gr0,  gr1,  gr3,  gr4), (arr1, arr1, arr1, arr1), default=0)
logger.debug(ue.info_gain_mode_arrays1(gmaps)\
             gmask = np.bitwise_and(grhm, mask) if mask is not None else grhm
     +'\n'+ue.info_pixel_gain_mode_statistics1(gmaps))

    arrf = np.array(raw & ue.M14, dtype=np.float32) - pedest

    t05 = time()

    if gmaskstore.ndimmask ==is 2None: gmask.shape
       store.mask = det.mask_total(1,gmask.shape[-2],gmask.shape[-1])

evt, **kwa)
    mask = store.mask

   #logger.debug(info_ndarr(arr1, '\n  arr1')) t06 = time()

        #loggerlogger.debug(info_ndarr(grhm, 'XXXX grhm'))
        logger.debug(info_ndarr(gmask, 'gmask')\'common-mode correction pars cmp: %s' % str(cmp))

    if cmp is not None:
      mode, cormax      + '\n  per panel statistics of cm-corrected pixels: %s'%= int(cmp[1]), cmp[2]
      npixmin = cmp[3] if len(cmp)>3 else 10
      if mode>0:

        t07 = t0_sec_cm = time()

  str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None)#t2_sec_cm = time()
        #logger.debug('common-mode mask massaging (sec)arr1 = %store.6f'arr1 %# (time()-t2_sec_cm)) # 5msec
np.ones_like(mask, dtype=np.uint8)
        #shgrhm = np.select((nsegsgr0,  352gr1, 384)
 gr3,  gr4), (arr1, arr1, arr1, arr1), hrows = 176 # int(352/2default=0)
        forgmask s in range(arrf.shape[0]):

  = np.bitwise_and(grhm, mask) if mask is not None else grhm
        if mode & 4: # in banks: (352/2,384/8)=(176,48) pixelsgmask.ndim == 2: gmask.shape = (1,gmask.shape[-2],gmask.shape[-1])

        #logger.debug(info_ndarr(arr1, '\n   ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin)
arr1'))
        #logger.debug(info_ndarr(grhm, 'XXXX grhm'))
        logger.debug(info_ndarr(gmask, 'gmask')\
                 ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin)

          if mode & 1: # in rows per bank: 384/8 = 48 pixels # 190ms
 + '\n  per panel statistics of cm-corrected pixels: %s'%
                           ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin)

str(np.sum(gmask, axis=(1,2), dtype=np.uint32) if gmask is not None else None))
        #logger.debug('common-mode mask massaging if mode & 2: # in cols per bank: 352/2 = 176 pixels # 150ms(sec) = %.6f' % (time()-t2_sec_cm)) # 5msec

        t08 = time()

        #sh    ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin= (nsegs, 352, 384)
        hrows = 176  ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin)

# int(352/2)
        for s in range(arrf.shape[0]):

          logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp)))

   t09 = time()

     res = arrf * factor if maskmode is& None4: else# arrfin * factor * mask # gain correction

banks: (352/2,384/8)=(176,48) pixels
      t07 = time()

    logger.debug('TOTAL consumed time (sec) = %.6f' % (t07-t00)\ue.common_mode_2d_hsplit_nbanks(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], nbanks=8, cormax=cormax, npix_min=npixmin)
                +info_ndarr(factor, '\n  calib_epix10ka factor')\ue.common_mode_2d_hsplit_nbanks(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], nbanks=8, cormax=cormax, npix_min=npixmin)

          t10 = time()

    +info_ndarr(pedest, '\n  calib_epix10ka pedest'))

  if mode times& =1: t00, t01, t02, t03, t04, t05, t06, t07
# in rows per bank: 384/8 = 48 pixels # 190ms
       return res, times

Timing points

t00: at the entrance
t01: det.raw
t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
t03: access constants at store - 1-st event consumption
t04: factor and pedestals for gain ranges using np.select
t05: apply data-bit mask to raw and subtract pedestals
t06: access mask - SHOULD BE DONE ONCE
t07: apply common mode correction if on

List of commands

Code Block

> s3dflogin
> psana
srun --partition milano --account lcls:prjdat21 -n 1 --time=01:00:00 --exclusive --pty /bin/bash
...
[dubrovin@sdfmilan050:~/LCLS/con-py3]$


In other window
===============
> s3dflogin
ssh -Y sdfmilan050
cd LCLS/con-py3
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/setup_testrel

mpirun -n  1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 1
mpirun -n  1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2

Results

on sdfmilan050

mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2

mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2

per event time difference betwee points in msec

t01 t02 t03 t04 t05 t06 t07

1-core: 0.333 0.025 0.002 10.906 1.438 0.001 140.304

80-core: 0.673 0.039 0.003 17.535 2.406 0.003 178.890

     ue.common_mode_rows_hsplit_nbanks(arrf[s,], mask=gmask[s,], nbanks=8, cormax=cormax, npix_min=npixmin)

          t11 = time()

          if mode & 2: # in cols per bank: 352/2 = 176 pixels # 150ms
            ue.common_mode_cols(arrf[s,:hrows,:], mask=gmask[s,:hrows,:], cormax=cormax, npix_min=npixmin)
            ue.common_mode_cols(arrf[s,hrows:,:], mask=gmask[s,hrows:,:], cormax=cormax, npix_min=npixmin)

        logger.debug('TIME common-mode correction = %.6f sec for cmp=%s' % (time()-t0_sec_cm, str(cmp)))

        t12 = time()
        res = arrf * factor if mask is None else arrf * factor * mask # gain correction

    t13 = time()

    logger.debug('TOTAL consumed time (sec) = %.6f' % (t13-t00)\
                +info_ndarr(factor, '\n  calib_epix10ka factor')\
                +info_ndarr(pedest, '\n  calib_epix10ka pedest'))

    return res, (t00, t01, t02, t03, t04, t05, t06, t07, t08, t09, t10, t11, t12, t13)

Timing points

t00: at the entrance
t01: det.raw
t02: gain and pedestals - NEEDS TO BE CALLED ONCE, DO NOT NEED IN gain USE factor ONLY!!!
t03: access constants at store - 1-st event consumption
t04: factor and pedestals for gain ranges using np.select
t05: apply data-bit mask to raw and subtract pedestals
t06: access mask - SHOULD BE DONE ONCE
t07: begin common mode algorithm
t08: define mask for common mode
t09: beginning of the loop over panels for cm
t10: cmc in banks: (352/2,384/8)=(176,48) pixels
t11: cmc in rows per bank: 384/8 = 48 pixels
t12: cmc in cols per bank: 352/2 = 176 pixels
t13: apply gain and mask to output array

List of commands

Code Block

title	List of test commands
collapse	true

The same as in previous, then

mpirun -n  1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2
mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2

Results

on sdfmilan004

mpirun -n 1 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2

mpirun -n 80 python Detector/examples/test-scaling-mpi-epix10ka2m.py 2

per event time difference betwee points in msec

t01 t02 t03 t04 t05 t06 t07 t08 t09 t10 t11 t12 t13

1-core: 0.326 0.021 0.001 10.921 1.338 0.001 0.023 3.960 319.467 3.471 9.426 8.430 1.558

0.294 0.020 0.001 11.150 1.275 0.001 0.023 3.968 319.375 3.443 9.424 8.438 1.607

0.316 0.021 0.001 11.103 1.291 0.001 0.024 3.974 319.814 3.460 9.436 8.443 1.567

80-core,rank 0: 0.373 0.024 0.002 12.639 1.674 0.001 0.017 4.744 384.576 4.154 11.336 10.108 2.100

0.385 0.025 0.002 12.725 1.558 0.001 0.017 4.579 382.630 4.192 11.463 10.118 2.160

0.396 0.024 0.002 13.547 1.691 0.001 0.018 4.621 381.572 4.092 11.252 10.070 2.106

Ratio: 1.22. 1.2. - 1.14. 1.29 - - 1.16. - 1.20 1.20. 1.20. 1.34

References

...

Page tree

Versions Compared

Old Version 3

New Version Current

Key

Content

2024-06-

17 Test for mpi det.calib

standard

Dataset

List of commands

Results

2024-06-18 Test for mpi det.calib local with inserted timeing points

Code of local calib_epix10ka_any

Timing points

List of commands

Results

Timing points

List of commands

Results

References

Page tree

Page History

Versions Compared

Old Version 3

New Version Current

Key

Content

2024-06-

17 Test for mpi det.calib

standard

Dataset

List of commands

Results

2024-06-18 Test for mpi det.calib local with inserted timeing points

Code of local calib_epix10ka_any

Timing points

List of commands

Results

Timing points

List of commands

Results

References