Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Content

Table of Contents

2014-01-22 Meeting minutes

...

 

pnCCD overview

Large area pnCCD DAQ and Elictronics, Lothar Struder & Robert Hartmann

 

Data for tests

On 2014-01-27 Sebastian Carron kindly provide us with data files for pnCCD experiment amoa1214:

  • Dark Run: 169, rear sensors gain 1/64, front 1/1, Imaging mode                                  exp=amoa1214:run=169
  • Run With Hits:  170  Low hit rate though, so you will have to use a hit finder of sorts   exp=amoa1214:run=170

Psana modules for pnCCD

Module ImgAlgos.PnccdNDArrProducer

  • Get from the event store Psana::PNCCD::FramesV1,
  • Put in the event store   ndarray<const T,3>, where shape=[4][512][512], T=uint16_t, int, float, double, int16_t

Performance: ~13 ms/event

Modified module ImgAlgos.PnccdImageProducer

  • Get  from the event store Psana::PNCCD::FullFrameV1 or ndarray<const T,3> or ndarray<const T,3>
    for source and key parameters
  • Put in the event store   ndarray<const T,2>, where shape=[1024+gap][1024], T= input type

Performance: ~30 ms/event (copy involves inverse iteration for 180 degree rotation of two bottom frames)

Sequence of modules for raw image averaging

  •           ImgAlgos.PnccdImageProducer - get Psana::PNCCD::FullFrameV1, put  ndarray<const uint16_t, 2>
  •           ImgAlgos.NDArrAverage            - averages ndarray<const T, 2>, save in file

Sequence of modules for calibrated image or ndarray averaging

  •           ImgAlgos.PnccdNDArrProducer - get Psana::PNCCD::FramesV1,     put  ndarray<const T, 3>
  •           ImgAlgos.NDArrCalib - getinput (raw) ndarray<const T, Ndim>, put  calibrated ndarray<const T, Ndim>
  •           ImgAlgos.PnccdImageProducer - get ndarray<const T,3>, put  ndarray<const T, 2>
  •           ImgAlgos.NDArrAverage            - averages ndarray<const T, 2> or ndarray<const T,3>, save in file

Image Added

Common mode correction in pnCCD ndarray

pnCCD image has intensity "strips" in both dimensions;

[4][512][512] array for single event and averaged over 1000 events:

Image AddedImage Added

At large number of events common mode should be averaged out. For 1000 events horizontal intensity "stripes" have gone.

This proves that common mode should be evaluated for horizontal stripes.

Due to specifics of pnCCD DAQ readout, it is recommended to evaluate common mode for consecutive groups of 128 pixel in the  [4][512][512] array.

 

Data corrections in module ImgAlgos.NDArrCalib

Module description: Module ImgAlgos::NDArrCalib 

List of parameters in configuration file

Module ImgAlgos::NDArrCalib is controlled by the list of parameters in the configuration file for psana, for example:

Code Block
[ImgAlgos.NDArrCalib]
source = DetInfo(Camp.0:pnCCD.0)
key_in = pnccd-ndarr  
key_out = calibrated  
do_peds = yes 
do_cmod = no 
do_stat = no 
do_mask = no 
do_bkgd = no 
do_gain = no  
do_nrms = no 
do_thre = yes 
fname_bkgd =  
fname_mask =
masked_value = 0
threshold_nrms = 0  
threshold = 100.0  
below_thre_value = 50
bkgd_ind_min = 0
bkgd_ind_max = 1000
bkgd_ind_inc = 2
print_bits = 11

 

 

2014-02-10 Test of the module ImgAlgos.NDArrCalib

Dark run: exp=amoa1214:run=7, all plots are shown for Camp.0:pnCCD.0 event 5

pedestals and  pixel_rms are generated by the calibman for arrays of shape=[4,512,512] using the same run.

Raw data

All do_* = no - that means no correction is applied

Image AddedImage Added

Pedestals

do_peds = yes 

file with pedestals is loaded automatically from

/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/pedestals/1-end.data

Image AddedImage Added

Common mode

do_peds = yes 
do_cmod = yes

files with pedestals and common_mode are loaded automatically from

/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/pedestals/1-end.data

/reg/d/psdm/AMO/amoa1214

chris

 

Script form Chris

Use interactive psana framework ~cpo/ipsana/shm.py:

Code Block
from psana import *

events = DataSource('shmem=1_1_XCS.0').events()
src = Source('DetInfo(XcsBeamline.1:Tm6740.5)')
import matplotlib.pyplot as plt

plt.ion()
fig = plt.figure('pulnix')
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8])  # x0, y0, h, w

for i in range(100):

    evt = events.next()
    frame = evt.get(Camera.FrameV1, src)

    ax.cla()
    ax.imshow(frame.data16())
    fig.canvas.draw()

Walking and talking about unlimited pipeline (processing)

Image RemovedImage Removed

CASS Heritage

Image RemovedImage Removed

Online monitor

Image RemovedImage Removed

Data for tests

On 2014-01-27 Sebastian Carron kindly provide us with data files for pnCCD experiment amoa1214:

  • Dark Run: 169, rear sensors gain 1/64, front 1/1, Imaging mode                                  exp=amoa1214:run=169
  • Run With Hits:  170  Low hit rate though, so you will have to use a hit finder of sorts   exp=amoa1214:run=170

Calibration of pnCCD

New modules for "old-style" calibration:

  • pdscalibdata/include/PnccdBaseV1.h                   - baseclass for pnCCD parameters, defines Segs, Rows, Cols, Size
  • pdscalibdata/include/PnccdPedestalsV1.h            - loads pedestals from file, returns ndarray of pedestals
  • pdscalibdata/include/PnccdCommonModeV1.h     - the same for common mode
  • pdscalibdata/include/PnccdPixelGainV1.h             - the same for pixel gain
  • pdscalibdata/include/PnccdPixelRmsV1.h             - the same for pixel rms
  • pdscalibdata/include/PnccdPixelStatusV1.h          - the same for pixel status
  • PSCalib::PnccdCalibPars                                      - wrapper for all pnCCD types

Detector-dependent interface

Example can be found in PSCalib/test/ex_calib_file_finder.cpp:

...

/calib/PNCCD::CalibV1/Camp.0:pnCCD.

...

0/

...

common_mode/1-end.data

...

where common mode parameters were set preliminary as:

echo "1 300 50 256 0.2" > /reg/d/psdm/AMO/

...

amoa1214/calib

...

/PNCCD::CalibV1

...

/Camp.0:pnCCD.0/common_mode/1-end.data

Average over each consecutive group of 256 pixels

Image AddedImage Added

Common mode subtraction improves the width of intensity distribution.


Pixel status

do_stat = yes

masked_value=0

Calibration type: pixel_status (0-good, 1,2,4,...-bad)

File with pixel status mask was produced in Calibration Manager ROI Mask application

/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.

...

0/pixel_status/7-7.data

Set bad pixels (1) in the half of frame[1]:

Image AddedImage Added

 

Mask

Region of interest (ROI) mask can be generated by the Mask Editor.

do_mask = yes

fname_mask = pnccd-test-mask.txt

masked_value=0

Image AddedImage Added

Background

do_bkgd = yes

fname_bkgd = pnccd-test-mask.txt

For this test the file with pedestals is used:

Image AddedImage Added

 Gain

do_gain = yes  

file with gain factors is loaded automatically from

/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/pixel_gain/7-7.data

 

 

Fot this test all gains for [4,512,512] pixels were set to 0.5

Image AddedImage Added

 

N*RMS Threshold

do_nrms = yes

below_thre_value = 0

threshold_nrms = 0.5
file with rms values is loaded automatically from
/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/pixel_rms/1-end.data
 

Image AddedImage Added

Common Threshold

do_thre = yes

below_thre_value = 0

threshold = 100 

Image AddedImage Added

Test for amob33314 run 167

Evaluate parameters in Calibration Manager

for types

  • pedestals
  • pixel_status for rms threshold = 15 ADU
  • pixel_rms

Camp.0.pnCCD.0

Image AddedImage Added

Camp.0.pnCCD.1

Image AddedImage Added

Raw data average and rms

Image AddedImage Added

Pedestals subtracted average and rms

Image AddedImage Added

Pedestals and common mode subtracted average and rms

echo "1 50 10 128 0.2" > /reg/d/psdm/AMO/amob3313/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/common_mode/167-167.data

Image AddedImage Added

Image AddedImage Added

 

 Support of pnCCD in Calibration Manager

In dark run processing in  Calibration Manager produces pedestals and pixel_rms. Then, if thresholds on rms and averaged intensity are set correctly, the pixel_status can be also produced and deployed under the calib directory.  Calibration manager works with arrays of shape=[4,512,512]. Embedded ROI Mask Editor can be used to generate the ROI mask for pnCCD.

Get latest version of calibman

Run Calibration Manager from current release (for release version ≥ ana-0.10.12):

Code Block
ssh -Y psana
cd <your-favorite-NON-RELEASE-directory>
sit_setup
calibman

 

If CalibManager or other packages were recently updated and these updates are wanted to be used:

Code Block
ssh -Y psana
cd <your-favorite-directory>
newrel ana-current <release-directory>
cd <release-directory>
sit_setup 

addpkg CalibManager HEAD;
addpkg <package-name-2> HEAD;
addpkg <package-name-3> HEAD;
...
scons;

calibman

 

Image of pnCCD

pnCCD image can be reconstructed from data by two methods using

  1. Module ImgAlgos::PnccdImageProducer and
  2. Module ImgAlgos::NDArrImageProducer

Examples of the first method are available in psana - Module Examples

Second method is available since release ana-0.13.14 (package PSCalib V00-02-40).

It allows to reconstruct image using generic Detector Geometry technique with geometry file. Current version of pnCCD has a solid structure without moving parts, hence the main part of the geometry file is unchanged. Example of the geometry file for pnCCD can be found in /reg/g/psdm/detector/alignment/pnccd/amo-pnccd.1-2015-01-23/calib/PNCCD::CalibV1/Camp.0:pnCCD.1/geometry/0-end.data, which essential part is

Code Block
# HDR PARENT IND      OBJECT   IND    X0[um]  Y0[um]  Z0[um]   ROT-Z ROT-Y ROT-X    TILT-Z   TILT-Y   TILT-X
DETPNCCD:V1    0    PNCCD:V1     0         0       0       0       0     0     0    0.0      0.0      0.0
DETPNCCD:V1    0    PNCCD:V1     1     76725   38325       0     180     0     0    0.0      0.0      0.0
DETPNCCD:V1    0    PNCCD:V1     2     76725   77925       0     180     0     0    0.0      0.0      0.0
DETPNCCD:V1    0    PNCCD:V1     3         0   39600       0       0     0     0    0.0      0.0      0.0
#IP:V1         0    DETPNCCD:V1  0     38400   39000   10000     -90     0     0    0.0      0.0      0.0

The last commented line in this file can be un-commented in order to rotate and translate detector relative to IP.

Note

  • The name of the segment PNCCD:V1 should be unchanged. Other names DETPNCCD:V1 and IP:V1 are optional.
  • Four segment coordinates are given in µm and it is assumed that each segment has 512x512 pixels of size 75x75µm².
  • All coordinates are given in matrix-like frame, where X axis goes from top to bottom, Y - from left to right.
  • Gap is set to 16 pixels and may be a subject of calibration.

 

 

Example of the configuration file psana-amoa1214-r0108-pnccd-NDArrImageProducer.cfg

Code Block
[psana]                                                              
# psana -m EventKeys -n 3 exp=amoa1214:run=108
#calib-dir = /reg/d/psdm/AMO/amoa1214/calib
files = exp=amoa1214:run=108
events = 10
 
modules = ImgAlgos.PnccdNDArrProducer \
          ImgAlgos.NDArrCalib \
          ImgAlgos.NDArrImageProducer \
          ImgAlgos.NDArrAverage:nda \
          ImgAlgos.NDArrAverage:img-geo \
          ImgAlgos.Tahometer
 
[ImgAlgos.PnccdNDArrProducer]
source  = DetInfo(Camp.0:pnCCD.1)
key_in  =
key_out = pnccd-ndarr
outtype = asdata
print_bits = 0
 
[ImgAlgos.NDArrCalib]
source = DetInfo(Camp.0:pnCCD.1)
key_in = pnccd-ndarr
key_out = calibrated
do_peds = yes
do_cmod = yes
do_stat = yes
do_mask = no
do_bkgd = no
do_gain = no
do_nrms = no
do_thre = no
#fname_mask = pnccd-test-mask.txt
#fname_bkgd = pnccd-test-bkgd.txt
masked_value     =    0
threshold_nrms   =  4.0
threshold        =  100
below_thre_value =    0
bkgd_ind_min     = 10000
bkgd_ind_max     = 10200
bkgd_ind_inc     =    1
print_bits       =    1
 
[ImgAlgos.NDArrAverage:nda]
source       = DetInfo(Camp.0:pnCCD.1)
key          = calibrated
avefile      = pnccd-nda-ave
rmsfile      = pnccd-nda-rms
#maskfile     = pnccd-nda-msk
#hotpixfile   = pnccd-nda-hot
thr_rms_ADU  = 0
thr_min_ADU  = 2
thr_max_ADU  = 65000
print_bits   = 29

[ImgAlgos.NDArrImageProducer]
#calibdir = ./calib
calibdir = /reg/g/psdm/detector/alignment/pnccd/amo-pnccd.1-2015-01-23/calib
source       

Pros

  • Simple format for calibration files - just a text file with pre-defined number of values for each type:
Code Block
973.941639 881.189675 1050.211 773.263749 899.241302 981.805836 1150.72615 993.084175 1121.15488 1029.76319 1220.14927 903.278339 1097.49944 1066.94949 1263.71044 1053.53872 1194.35915 935.320988 1317 ...

Cons

  • Too simple calibration file format, does not allow any metadata or comments.
  • Detector-dependent objects and parameters "knows" about parameters' array type and shape:
    • PSCalib::PnccdCalibPars which depends on PnccdPedestalsV1, PnccdCommonModeV1, ..., PnccdBaseV1
    • pdscalibdata::PnccdPedestalsV1::pars_t      = float
      pdscalibdata::PnccdCommonModeV1::pars_t     = uint16_t 
      pdscalibdata::PnccdPixelStatusV1::pars_t    = uint16_t
      pdscalibdata::PnccdPixelGainV1::pars_t      = float
    • const std::string groupName = "PNCCD::CalibV1";       - do we really need it ?   

Detector-independent interface

  • Interface is declared in the abstract base class PSCalib::CalibPars
  • Access to all detector-dependent classes is hidden in the static factory class PSCalib::CalibParsStore
Note

Factory is implemented for pnCCD only. CSPAD and CSPAD2x2 will be added soon.

Code Block
#include "PSCalib/CalibPars.h"
#include "PSCalib/CalibParsStore.h"

// Instatiation
//Here we assume that code is working inside psana module where evt and env variables are defined through input parameters of call-back methods.
//Code below instateates calibpars object using factory static method PSCalib::CalibParsStore::Create:

std::string calib_dir = env.calibDir(); // or "/reg/d/psdm/<INS>/<experiment>/calib"
std::string  group = std::string(); // or something like "PNCCD::CalibV1";
const std::string source = "Camp.0:pnCCD.1";
const std::string key = ""; // key for raw data
Pds::Src src; env.get(source, key, &src);
PSCalib::CalibPars* calibpars = PSCalib::CalibParsStore::Create(calib_dir, group, src, PSCalib::getRunNumber(evt));

// Access methods
calibpars->printCalibPars();
const PSCalib::CalibPars::pedestals_t*    peds_data = calibpars->pedestals();
const PSCalib::CalibPars::pixel_gain_t*   gain_data = calibpars->pixel_gain();
const PSCalib::CalibPars::pixel_rms_t*    rms_data  = calibpars->pixel_rms();
const PSCalib::CalibPars::pixel_status_t* stat_data = calibpars->pixel_status();
const PSCalib::CalibPars::common_mode_t*  cmod_data = calibpars->common_mode();

 

New approach to calibration files with header

In order to get rid of detector dependent types of calibration parameters we need to add metadata in the calibration file. All metadata can be listed in the header of the calibration files, for example, using keyward mapping (dictionary):

Code Block
# RULES:
# Lines starting with # in the beginning of the file are considered as comments or pseudo-comments for metadata
# Lines without # with space-separated values are used for input of parameters
# Empty lines are ignored

# Optional fields:
# TITLE:      This is a file with pedestals
# DATE_TIME:  2014-01-30 10:21:23
# AUTHOR:     someone
# EXPERIMENT: amotut13
# DETECTOR:   Camp.0:pnCCD.1
# CALIB_TYPE: pedestals

# Mandatory fields to define the ndarray<TYPE,NDIMS> and its shape as unsigned shape[NDIMS] = {DIM1,DIM2,DIM3}
# TYPE:       float
# NDIMS:      3
# DIM1:       4
# DIM2:       255
# DIM3:       255

973.941639 881.189675 1050.211 773.263749 899.241302 981.805836 1150.72615 993.084175 1121.15488 1029.76319 1220.14927 903.278339 1097.49944 1066.94949 1263.71044 1053.53872 1194.35915 935.320988 1317 ...

 

psana modules for pnCCD

New module ImgAlgos.PnccdNDArrProducer

  • Get from the event store Psana::PNCCD::FramesV1,
  • Put in the event store   ndarray<T,3>, where shape=[4][512][512], T=uint16_t, int, float, double, int16_t

Performance: ~13 ms/event

Modified module ImgAlgos.PnccdImageProducer

  • Get  from the event store Psana::PNCCD::FullFrameV1 or ndarray<T,3> for source and key parameters
  • Put in the event store   ndarray<T,2>, where shape=[1024+gap][1024], T= input type

Performance: ~30 ms/event (copy involves inverse iteration for 180 degree rotation of two bottom frames)

Old sequence of image averaging

          ImgAlgos.PnccdImageProducer - get Psana::PNCCD::FullFrameV1, put  ndarray<uint16_t, 2>
          ImgAlgos.NDArrAverage            - averages ndarray<T, 2>, save in file

Image Removed
 

New sequence of image averaging

Note

For demonstration only! Just in order to confirm that we produce the same image from different objects. In real case image needs to be produced at the final stage.

          ImgAlgos.PnccdNDArrProducer - get Psana::PNCCD::FramesV1,     put  ndarray<T, 3>
          ImgAlgos.PnccdImageProducer - get ndarray<T,3>, put  ndarray<T, 2>
          ImgAlgos.NDArrAverage            - averages ndarray<T, 2>, save in file

Image Removed

"Natural order" for common mode correction in pnCCD ndarray

pnCCD image has intensity "strips" in both dimensions;

[4][512][512] array for single event and averaged over 1000 events:

Image RemovedImage Removed

At large number of events common mode should be averaged out. For 1000 events horizontal intensity "stripes" have gone.

This proves that common mode should be evaluated for horizontal stripes.

 

Data corrections in module ImgAlgos.NDArrCalib

Midule description: Module ImgAlgos::NDArrCalib 

List of parameters in configuration file

Code Block
[ImgAlgos.NDArrCalib]
source = DetInfo(Camp.0:pnCCD.01)
key_in     = pnccd-ndarr = calibrated
key_out = calibrated  
do_peds = yes 
do_cmod = no 
do_stat = no 
do_mask = no 
do_bkgd = no 
do_gain = no  
do_nrms = no 
do_thre = yes 
fname_bkgd =  
fname_mask =
masked_value = 0
threshold_nrms = 0  
threshold = 100.0  
below_thre_value = 50
bkgd_ind_min = 0
bkgd_ind_max = 1000
bkgd_ind_inc = 2
print_bits = 11

 

Pedestals subtraction

Calibration type: pedestals

Image RemovedImage Removed

 

 

Threshold

Apply common threshold = 100 ADU:

Image RemovedImage Removed

peak in spectrum at 0 corresponds settings for gap

peak in spectrum at 50 corresponds settings for under threshold pixels

Gain

Calibration type: pixel_gain

Apply common gain factor = 0.5:

Image RemovedImage Removed

 

Pixel status mask

Calibration type: pixel_status (0-good, 1,2,4,...-bad)

Set bad pixels (1) in the half of frame[1]:

Image RemovedImage Removed

 

2014-02-06 pnccd with common mode

Dark run: exp=amoa1214:run=7, all plots are shown for Camp.0:pnCCD.0 event 5

Pedestals are generated for ndarray using the same dark run

Raw data

Image RemovedImage Removed

Pedestals subtracted

Image RemovedImage Removed

Pedestals and common mode subtracted

Using intensity distribution for pedestals set

/reg/d/psdm/AMO/amoa1214/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/common_mode/1-end.data

Note

1 300 50

Average over 128 pixels

Image RemovedImage Removed

Common mode subtraction improves the width of intensity distribution.

Support of pnCCD in Calibration Manager

In dark run processing in  Calibration Manager produces pedestals and pixel_rms. Then, if thresholds on rms and averaged intensity are set correctly, the pixel_status can be also produced and deployed under the calib directory.  Calibration manager works with arrays of shape=[4,512,512]. Embedded ROI Mask Editor can be used to generate the ROI mask for pnCCD.

Get latest version of packages for psana and calibman

Code Block
ssh -Y psana
cd <your-favorite-directory>
newrel ana-current <release-directory>
cd <release-directory>
sit_setup 

addpkg ImgAlgos HEAD;
addpkg pdscalibdata HEAD;
addpkg PSCalib HEAD;
addpkg CalibManager HEAD;
scons;

 

References

pnccd-img-geo
print_bits   = 1

[ImgAlgos.NDArrAverage:img-geo]
source       = DetInfo(Camp.0:pnCCD.1)
key          = pnccd-img-geo
avefile      = pnccd-img-geo-ave
rmsfile      = pnccd-img-geo-rms
#maskfile     = pnccd-img-geo-msk
#hotpixfile   = pnccd-img-geo-hot
thr_rms_ADU  = 0
thr_min_ADU  = 2
thr_max_ADU  = 65000
print_bits   = 29

[ImgAlgos.Tahometer]
dn         = 100
print_bits = 7

uses psana modules

  • ImgAlgos.PnccdNDArrProducer  - gets pnCCD data and put it as ndarray in the event store
  • ImgAlgos.NDArrCalib - applys calibrations to ndarray
  • ImgAlgos.NDArrImageProducer - produces calibrated image
  • ImgAlgos.NDArrAverage:nda - averages calibrated ndarray
  • ImgAlgos.NDArrAverage:img-geo - averages calibratedimage

This script can be executed by the command

Code Block
psana -c psana-amoa1214-r0108-pnccd-NDArrImageProducer.cfg

which saves averaged and RMS files for calibrated ndarray and image, which can be plotted by the command plims <file-name>. For eample

Code Block
plims pnccd-nda-ave-amoa1214-r0108.dat
plims pnccd-img-geo-ave-amoa1214-r0108.dat

plot images with intensity distributions for ndarray and image, respectively:

Image AddedImage Added

Masks and other calibration files for pnCCD and image can be generated with calibman.

 

References