Psocake SFX tutorial

"Data analysis == Piece of cake"

Sections in this tutorial

It is always a good idea for the people doing analysis to be able to look at their detector images and probe intensity values. Given that a typical LCLS experiment has millions of snapshots to choose from, it is also critical that you can quickly select images of interest and set regions of interest using masks. By the end of this tutorial, you will be able to browse images, jump to images of interest, generate masks, change the x,y,z positions of your detector, find peaks in your images and index crystal diffraction patterns.

Citation for psocake (and other psana-based programs):
@Article{Thayer2017,
author="Thayer, J. and Damiani, D. and Ford, C. and Dubrovin, M. and Gaponenko, I. and O'Grady, C. P. and Kroeger, W. and Pines, J. and Lane, T. J. and Salnikov, A. and Schneider, D. and Tookey, T. and Weaver, M. and Yoon, C. H. and Perazzo, A.",
title="Data systems for the Linac coherent light source",
journal="Advanced Structural and Chemical Imaging",
year="2017", month="Jan", day="14", volume="3", number="1", pages="3", issn="2198-0926",
doi="10.1186/s40679-016-0037-7", url="https://doi.org/10.1186/s40679-016-0037-7"}
 
@article{Damiani:zw5004, 
author = "Damiani, D. and Dubrovin, M. and Gaponenko, I. and Kroeger, W. and Lane, T. J. and Mitra, A. 
and O'Grady, C. P. and Salnikov, A. and Sanchez-Gonzalez, A. and Schneider, D. and Yoon, C. H.", 
title = "{Linac Coherent Light Source data analysis using {it psana}}", 
journal = "Journal of Applied Crystallography", 
year = "2016", volume = "49", number = "2", pages = "672--679", month = "Apr", 
doi = {10.1107/S1600576716004349}, url = {http://dx.doi.org/10.1107/S1600576716004349}, }

Starting psocake in SFX mode

If you are on a psana machine, set up your environment by adding these lines to .bashrc (or your start up script):

# Psocake (Run 18 or newer)
source /reg/g/psdm/etc/psconda.sh -py3
# Psocake (Run 17 or older)
source /reg/g/psdm/etc/psconda.sh

# CrystFEL
source /reg/g/cfel/crystfel/crystfel-0.9.1/setup-sh

# CCP4 (version may change)
source /reg/common/package/ccp4-7.1/bin/ccp4.setup-sh

# XDS
export PATH=/reg/common/package/XDS-INTEL64_Linux_x86_64:$PATH

For this tutorial, we will look at experiment cxic00318, run 124, detector jungfrau4M, event 33.

$ psocake exp=cxic00318:run=124 -d jungfrau4M -m sfx

Or you can also use the -e and -r arguments for the experiment and the run number:

$ psocake -e cxic00318 -r 124 -d jungfrau4M -m sfx

$ psocake -e cxic00318 -r 124 -d jungfrau4M -m sfx -a ffb

#######################################
# Available area detectors: 
# ('CxiDs1.0:Jungfrau.0', 'jungfrau4M', '')
#######################################

CxiDs1.0:Jungfrau.0 is the full detector name. jungfrau4M is the simpler DAQ alias. In most cases, Psocake can understand both naming conventions.

To check psocake version:

$ psocake --version

Don’t worry if you don’t remember these arguments. You can view argument options using --help:

$ psocake --help

Psocake should have generated directories and files in the experiment directory. At LCLS, all experiments are stored here: /reg/d/psdm/<instrument>/<experiment>. Let's take a moment and check out our directory structure. Either open a new terminal (Remember to 'ssh psana') or use the current terminal ('Ctrl+z' to suspend psocake that is running then 'bg' to run psocake in the background), type the following command:

$ ls /reg/d/psdm/cxi/cxic00318
calib  hdf5  results  scratch  stats  usrdaq  xtc

calib: This is where all psana calibration is stored. Detector geometry, pedestals, gain, common mode constants, and bad pixelmap.

xtc: This is where all your raw data is stored. XTC is an efficient format for storing large data which can be read using psana. Note you have 4 months to analyse your data before xtcs are moved off to tape.

scratch: This is where psocake saves all the files like .cxi and .stream. This directory is not backed up, so important files need to be backed up in /results.

results (or res): This is the results directory which is backed up on tape. After completing your analysis, your results/data should be moved here.

Mask making

In this section, let's learn how to mask out pixels that should not be used for analysis (such as dead pixels), mask out the jet streak at the centre of the detector, and mask out the water ring (just for fun!).

Note: the Image Panel must be in the default "greyscale" colormap for the mask colors to display properly.

1) In the mask panel, "Use psana mask" checkbox is turned on by default. This will mask out the following pixels that should not be used for analysis; calib, status, edge, central, unbonded pixels, unbonded pixel neighbor pixels. These masked pixels are shown as green on the image panel. Red circle indicates the detector centre.

2) To mask out jet streak near the centre of the detector, click on "Use streak mask". This will mask out strong intensities originating from the liquid jet. The streak mask is dynamic, i.e. recalculated for each image.

3) To make a donut mask over the water ring, click on "Use user-defined mask". Select "Toggle" in Masking mode. This will bring up a red circle, polygon and square mask widgets.

Move the red circle widget to the centre of the detector by dragging it. Resize it by dragging the diamond handle on the perimeter. Once you are happy with the area to be masked out, click "Stamp circular mask" button on the mask panel.

Increase the size of the red circle widget again by dragging the diamond handle on the perimeter. Click "Stamp circular mask" button on the mask panel. Because we are in the "toggle" mode, previously mask pixels inside our widget is unmasked, i.e. gets toggled. The area that does not overlap with the previous mask get masked out.

To save the user-defined mask, click on "Save static mask" button on the mask panel which will save the mask in your designated psocake run folder. This will combine user-defined mask (red) and the psana mask (green) into a single mask. You should see the following message in the terminal:

Saving mask to /reg/d/psdm/cxi/cxic00318/scratch/<username>/psocake/r0124
*** deploy user-defined mask as mask.txt and mask.npy as DAQ shape ***
*** deploy user-defined mask as mask_natural_shape.npy as natural shape ***
Saving Cheetah static mask in:  /reg/d/psdm/cxi/cxic00318/scratch/<username>/psocake/r0124/staticMask.h5

mask.npy has the shape (8, 512, 1024) for this jungfrau4M detector. We refer to this shape as an unassembled 3D ndarray which can be used in psana python scripts.

mask.txt is in a 2D text format which is compatible with detector related files in the LCLS calibration folder.

You can load this mask in psocake by clicking the "Load mask" button and selecting mask.npy.

To delete the mask on the screen, select "Unmask" under Masking mode. Drag one of the red mask widgets over the detector and click "Stamp circular mask".

Peak finding

In this section, we will find peaks on the detector image.

Default setting in the Peak Finder panel is to "Show peaks found". You should notice peaks being highlighted in the Image panel. If you are happy with the peaks found, you are done. Otherwise, you can interactively tune peak finding parameters by expanding the "Parameters" field. Details of the peak finding algorithm is given here: Hit and Peak Finding Algorithms. 

$ squeue -u <username>

$ cat /reg/d/psdm/cxi/cxic00318/scratch/<username>/psocake/r0124/status_peaks.txt
{"numHits": 19, "hitRate(%)": 38.0, "fracDone(%)": 100.0}

We found 19 hits from 50 events. The hit rate is 38%. Fraction of events processed is 100%.

Indexing crystals

First things first, crystal indexing requires an accurate detector geometry. Latest CXI geometry files can be found here: Geometry history

• In the diffraction geometry panel, set "Detector distance" to 145mm. 
• In the indexing panel, tick "Indexing on". If indexing succeeds, the integrated (predicted) peaks will be marked with magenta circles. These triple rings represent the integration radii. A magenta triangle means wait. If indexing fails, you will see a magenta X.
• If you see magenta circles and magenta unitcell appear, congratulations! You have indexed your first pattern using psocake.
• Try incrementing this distance in 1mm step till the unit cell parameters are as close as possible to lys.cell. The optimum detector distance is around 158mm.
  Everytime the "Detector distance" value is changed by the user, psocake converts the psana geometry (in /reg/d/psdm/cxi/cxitut13/calib/CsPad\:\:CalibV1/CxiDs1.0\:Cspad.0/geometry/10-11.data) to a CrystFEL geom file (in /reg/d/psdm/cxi/cxitut13/scratch/<username>/psocake/<runNumber>/.temp.geom).

Indexing panel uses CrystFEL to index the diffraction patterns, so the input parameters in the indexing panel should be familiar to you if you've used indexamajig before.

CrystFEL geometry: This geometry file is automatically converted from our psana geometry to CrystFEL geometry for you. Feel free to look inside .temp.geom. If you have a CrystFEL geometry file that you know is good, you can simply type it in. Psocake will never modify this file even if you change the "detector distance" in the diffraction geometry panel. (Just don't name your geometry .temp.geom, it will get overridden). You can also deploy the CrystFEL geometry as a psana geometry by clicking "Deploy CrystFEL geometry" in the indexing panel.

Integration radii: These 3 numbers define the radius of two concentric rings about each Bragg spot. Inner ring is used to integrate the Bragg spot and the outer ring is used to estimate the background. Try adjusting these numbers and see what is being integrated on screen. It should be large enough to fit a Bragg spot inside the inner ring.

PDB: If you have a CrystFEL unitcell, you can constrain the indexing algorithms to look for this unit cell.

Indexing method: Default is mosflm-noretry, dirax. "retry" is used to speed up mosflm (it can take few seconds).

Tolerance: These 4 numbers define how much wriggle room you want for indexing. 5, 5, 5 are the tolerance level for unitcell axes a, b, c. 1.5 is the tolerance level for the angles alpha, beta, gamma.

Extra CrystFEL parameters: You can enter extra parameters for indexamajig in this field. It will be appended at the end of the command line, e.g. --profile will turn on the processing timing information.

Let's try to index another diffraction pattern at event 44.

• In the experiment parameters panel, set Event Number to 44.
• You should see the magenta triangle appear again. Wait few seconds and hopefully you will have indexed another pattern.

Hopefully, you have indexed this diffraction pattern. Notice that the unitcell parameters are a bit off compared to what is expected. Let's load a CrystFEL unitcell file to help the indexer along.

• In the Indexing panel, set the PDB field to: /reg/d/psdm/cxi/cxitut13/scratch/psocake/lys.cell
  

CrystFEL unit cell file version 1.0
lattice_type = tetragonal
centering = P
unique_axis = c
a = 77.05 A
b = 77.05 A
c = 37.21 A
al = 90 deg
be = 90 deg
ga = 90 deg

182 in #evr1#

182 in #evr1# and 173 in #evr1# 

# Phenix
source /reg/common/package/phenix/phenix-1.10.1-2155/phenix_env.sh
# CCP4
source /reg/common/package/ccp4/ccp4-7.0/bin/ccp4.setup-sh
# XDS is available here
/reg/common/package/XDS-INTEL64_Linux_x86_64/

ebeam = ebeamDet.get(evt)
pulseEnergy = ebeam.ebeamL3Energy()  # MeV

es = psana.DataSource.env().epicsStore()
calculatedNumberOfPhotons = get_es_value(es, 'SIOC:SYS0:ML00:AO580', NoneCheck=False, exceptReturn=0) * 1e12  # photons