Contents

2014-01-22 Meeting minutes

Hi everyone,

Here is a short summary of what I heard today for how we should start
with the pnCCD.

For pnCCD algorithms:
common-mode, pedestals, hot-pixels, quadrant rotations, hit-finders,
support in mikhail's calibManager

For pnCCD online displays: (using matplotlib for now)
shot by shot raw data
shot by shot calibrated data
projections of the above
region-of-interest
strip-charts of interesting quantities
(also display calibration values like noise-map,pedestal-map)

After this we will work on the acqiris as well (acqiris
constant-fraction algos already exist in psana).

Attached below is a 12 line python program that plots a real pnCCD
image and an x-projection (amoc0113 also has pnccd data we can look
at).  You can run it on a psana node by saving it to pnccd.py and
doing "sit_setup" and then "ipython pnccd.py".  This sort of code
should work online too (although we may have to change matplotlib
settings) as well as with calibrated images.

Display group (dan, mikhail, me) meets Thursday at 10:30.  Analysis
group (sebastian, ankush(?), phil, mikhail, me) meets Friday at 1.

See you then...

chris

Script form Chris

Use interactive psana framework ~cpo/ipsana/shm.py:

from psana import *

events = DataSource('shmem=1_1_XCS.0').events()
src = Source('DetInfo(XcsBeamline.1:Tm6740.5)')
import matplotlib.pyplot as plt

plt.ion()
fig = plt.figure('pulnix')
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])  # x0, y0, h, w

for i in range(100):

    evt = events.next()
    frame = evt.get(Camera.FrameV1, src)

    ax.cla()
    ax.imshow(frame.data16())
    fig.canvas.draw()

Walking and talking about unlimited pipeline (processing)

CASS Heritage

Online monitor

Data for tests

On 2014-01-27 Sebastian Carron kindly provide us with data files for pnCCD experiment amoa1214:

• Dark Run: 169, rear sensors gain 1/64, front 1/1, Imaging mode                                  exp=amoa1214:run=169
• Run With Hits:  170  Low hit rate though, so you will have to use a hit finder of sorts   exp=amoa1214:run=170

Calibration of pnCCD

New modules for "old-style" calibration:

• pdscalibdata/include/PnccdBaseV1.h                   - baseclass for pnCCD parameters, defines Segs, Rows, Cols, Size
• pdscalibdata/include/PnccdPedestalsV1.h          -  loads pedestals from file, returns ndarray of pedestals
• pdscalibdata/include/PnccdCommonModeV1.h  -  the same for common mode
• pdscalibdata/include/PnccdPixelGainV1.h            - the same for pixel gain
• pdscalibdata/include/PnccdPixelStatusV1.h         - the same for pixel status
• PSCalib::PnccdCalibPars                                       - wrapper for all pnCCD types

Interface

Example can be found in PSCalib/test/ex_calib_file_finder.cpp:

// Assume that file is located in /reg/d/psdm/AMO/amotut13/calib/PNCCD::CalibV1/Camp.0:pnCCD.1/pedestals/1-end.data

  const std::string calibDir   = "/reg/d/psdm/AMO/amotut13/calib";
  const std::string groupName  = "PNCCD::CalibV1";
  const std::string source     = "Camp.0:pnCCD.1";
  unsigned long     runNumber  = 10;
  unsigned          print_bits = 255;

  PSCalib::PnccdCalibPars *calibpars = new PSCalib::PnccdCalibPars(calibDir, groupName, source, runNumber, print_bits);  

  calibpars->printCalibPars();
  calibpars->printCalibParsStatus();
  calibpars->printInputPars();

  ndarray<pdscalibdata::PnccdPedestalsV1::pars_t, 3>   peds = calibpars -> pedestals();
  ndarray<pdscalibdata::PnccdCommonModeV1::pars_t, 1>  cmod = calibpars -> common_mode();
  ndarray<pdscalibdata::PnccdPixelStatusV1::pars_t, 3> stat = calibpars -> pixel_status();
  ndarray<pdscalibdata::PnccdPixelGainV1::pars_t, 3>   gain = calibpars -> pixel_gain();

Pros

• Simple format for calibration files - just a text file with pre-defined number of values for each type:

973.941639 881.189675 1050.211 773.263749 899.241302 981.805836 1150.72615 993.084175 1121.15488 1029.76319 1220.14927 903.278339 1097.49944 1066.94949 1263.71044 1053.53872 1194.35915 935.320988 1317 ...

Cons

• Too simple calibration file format, does not allow any metadata or comments.
• Detector-dependent objects and parameters "knows" about parameters' array type and shape:
  • PSCalib::PnccdCalibPars which depends on PnccdPedestalsV1, PnccdCommonModeV1, ..., PnccdBaseV1

  • pdscalibdata::PnccdPedestalsV1::pars_t      = float

    pdscalibdata::PnccdCommonModeV1::pars_t     = uint16_t 

    pdscalibdata::PnccdPixelStatusV1::pars_t    = uint16_t

    pdscalibdata::PnccdPixelGainV1::pars_t      = float

  • const std::string groupName = "PNCCD::CalibV1";                      - do we really need it ?   

How to improve interface for "new-style" calibration

We need to define calibration array shape, type of parameters, and where to find them under calib directory: groupName="PNCCD::CalibV1", source="Camp.0:pnCCD.1";

• Use universal calibration object (for all detectors!):
  PSCalib::CalibPars *calibpars = new PSCalib::CalibPars(calibDir, source, runNumber, print_bits);
  where source defines the detector type
  
  
• Use universal (for all detectors!) modules for each calibration type:
  pdscalibdata/include/DetPedestalsV1.h          -  loads pedestals from file, returns ndarray of pedestals
  pdscalibdata/include/DetCommonModeV1.h  -  the same for common mode
  pdscalibdata/include/DetPixelGainV1.h            - the same for pixel gain
  pdscalibdata/include/DetPixelStatusV1.h         - the same for pixel status
  
  
• Use universally (for all detectors!) pre-defined types of parameters :
  

  pdscalibdata::DetPedestalsV1::pars_t   = float

  pdscalibdata::DetCommonModeV1::pars_t  = uint16_t 

  pdscalibdata::DetPixelStatusV1::pars_t = uint16_t

  pdscalibdata::DetPixelGainV1::pars_t   = float

• Define the groupName="PNCCD::CalibV1" from dictionary for source="Camp.0:pnCCD.1" - this means that we will never change the groupName !
  
  

• Save/retrieve ndarray type/shape from file header, for example:
  
  

  # RULES:
  # Lines starting with # in the beginning of the file are considered as comments or pseudo-comments for metadata
  # Lines without # with space-separated values are used for input of parameters
  # Empty lines are ignored
  
  # Optional fields:
  # TITLE:      This is a file with pedestals
  # DATE_TIME:  2014-01-30 10:21:23
  # AUTHOR:     someone
  # EXPERIMENT: amotut13
  # DETECTOR:   Camp.0:pnCCD.1
  # CALIB_TYPE: pedestals
  
  # Mandatory fields to define the ndarray<TYPE,NDIMS> and its shape as unsigned shape[NDIMS] = {DIM1,DIM2,DIM3}
  # TYPE:       float
  # NDIMS:      3
  # DIM1:       4
  # DIM2:       255
  # DIM3:       255
  
  973.941639 881.189675 1050.211 773.263749 899.241302 981.805836 1150.72615 993.084175 1121.15488 1029.76319 1220.14927 903.278339 1097.49944 1066.94949 1263.71044 1053.53872 1194.35915 935.320988 1317 ...

psana modules for pnCCD

New module ImgAlgos.PnccdNDArrProducer

• Get from the event store Psana::PNCCD::FramesV1,
• Put in the event store   ndarray<T,3>, where shape=[4][512][512], T=uint16_t, int, float, double, int16_t

Performance: ~13 ms/event

Modified module ImgAlgos.PnccdImageProducer

• Get  from the event store Psana::PNCCD::FullFrameV1 or ndarray<T,3> for source and key parameters
• Put in the event store   ndarray<T,2>, where shape=[1024+gap][1024], T= input type

Performance: ~30 ms/event (copy involves inverse iteration for 180 degree rotation of two bottom frames)

Old sequence of image averaging

          ImgAlgos.PnccdImageProducer - get Psana::PNCCD::FullFrameV1, put  ndarray<uint16_t, 2>
          ImgAlgos.NDArrAverage             - averages ndarray<T, 2>, save in file

New sequence of image averaging

          ImgAlgos.PnccdNDArrProducer - get Psana::PNCCD::FramesV1,     put  ndarray<T, 3>
          ImgAlgos.PnccdImageProducer - get ndarray<T,3>, put  ndarray<T, 2>
          ImgAlgos.NDArrAverage             - averages ndarray<T, 2>, save in file

"Natural order" for common mode correction in pnCCD ndarray

pnCCD image has intensity "strips" in both dimensions;


[4][512][512] array for single event and averaged over 1000 events:

At large number of events common mode should be averaged out. For 1000 events horizontal intensity "stripes" have gone.

This proves that common mode should be evaluated for horizontal stripes.

Corrections in module ImgAlgos.NDArrCalib

List of parameters in configuration file

Pedestals subtraction

Calibration type: pedestals

Threshold

Apply common threshold = 100 ADU:

peak in spectrum at 0 corresponds settings for gap

peak in spectrum at 50 corresponds settings for under threshold pixels

Gain

Calibration type: pixel_gain

Apply common gain factor = 0.5:

Mask

Calibration type: pixel_status (0-good, 1,2,4,...-bad)

Set bad pixels (1) in the half of frame[1]:

2014-02-06 pnccd with common mode

Dark run: exp=amoa1214:run=7, all plots are shown for Camp.0:pnCCD.0 event 5

Pedestals are generated for ndarray using the same dark run

Raw data

Pedestals subtracted

Pedestals and common mode subtracted

Using intensity distribution for pedestals set

/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/common_mode/1-end.data

Average over 128 pixels

Common mode subtraction improves the width of intensity distribution.

References

• Psana Module Examples
• Psana Module Catalog
• Calibration management tool
• CSPAD Calibration