E. Galtier, P. McGehee, H.J. Lee, and B. Nagler
Conference Poster: MEC - HEDLA.pdf
Introduction
This page describes all the common online/real-time and offline analysis tools available at MEC, as well as some basics such as instructions on how to use these tools, how the data is managed and stored at LCLS-MEC, how to transfer the data, how to use the computing farms at LCLS etc. The online and offline analysis tools are described below.
Online Analysis
Online data analysis tools we use during FEL beam-time
The Data Acquisition (DAQ) software we use is made in house by the LCLS Data System group. To view the data in real time while it is recorded, the LCLS DAQ provides a tool called Analysis Monitoring Interface (AMI), which can perform basic real-time analysis such as displaying detector images at specified acquisition rate, making projections of the detector image on-the-fly onto specified axis, making the trending plots of some diode read-out, so on. Your experiment POC will operate or show users how to operate DAQ and AMI during the beam-time.
Here below we show some typical examples of use of AMI, please contact the MEC team in advance (more than 2 month) of the beam-time if there are analysis tools that are essential for your beam time and are missing from the common list of tools below.
How the interface looks like as a whole, you will see each individual detector, such as "MecTargetChamber-0|Cspad", "MecTargetChamber-0|Princeton-2", under the list in AMI. Clicking on the detector that you want to visualizes the data will bring up a separate GUI or Window as shown next.
Visualizing raw images from detectors, such as CSPAD, Princeton camera, OPAL optical cameras. Here you can do simple things such as projections of the image, simple mathematical operations such as additions and multiplications, or saving a background file as reference. For example in "ChB" one could save an acquired image, which is then subtract from for every single event in "ChA", and the results is displayed in “ChC” in real time.`
- For CSPAD and Epix detectors, a background subtraction procedure is provided by the “calibman” tool (see calibman webpage). The AMI will automatically pick up those background calibrations and display the background subtracted images.
- Making projections of detector images onto X or Y axis, for spatial and spectrally resolved signals, histograming of the images, some statistically functions such as integrals, RMS, mean ,etc.
All "scalar" data such as read-out from some diodes such as IPM2, IPM3 (they are called Beam Line Device ,Bld in the data structures), gas detector energies for beam intensities, some EPICS PV variables for photon energy for example, can be accessed from the "Env" GUI in AMI. AMI will give you a full list of all the recorded data (shown on the right), and you can select the one you need. Some simple arithmetic (shown in the middle) can be then be performed on the data. Usually it's the trending plot ("v Time") that is important for aligning purposes.
If you need some customized analysis tools which are not existing in AMI, you need contact the MEC team at the very beginning of beam-time preparations. We can discuss with LCLS DAQ group or Data Analysis group to see if these requests can fit into the beam-time preparation schedule. If it gets implemented, it can either be within the AMI software, or a standalone python script called ipsana (psana python interface).
Offline Analysis
Offline data analysis, data extraction, etc.
For analysis of the data, LCLS provides a psana python environment, which we encourage users to use. Users can find a number of psana-python "building block" examples here. It is important to emphasize that users should practice writing analysis scripts well before the experiment starts. To practice, all the examples on the above confluence page can be run by the users.
The raw recorded data are always saved in XTC format and they are saved at: /reg/d/psdm/mec/mecxxxxx/xtc/, where xxxxx is the experiment id, as shown below. You can access the raw data files from any psana server machines (ssh to pslogin first, and then ssh to psana), or psexport machines. Since data generated by a run can be large, it is split up in 4 stream, each with its own xtc files. In these filenames, s00, s01, s02, s03, stand for stream numbering in the data system, the r0001, r0002 stand for different runs.
A subsection of the experiments at MEC only produce very low data rates, since they are low repetition rate or single shot experiments, with a limited number of shots in each run. For the user's convenience, it is possible to extract the data from such runs and save them in a user-friendly and human-readable format. This can be done in the Data Manager (also known as the Electronic Logbook, or Elog), under the ‘Workflow’ tab in ‘Batch Analysis’. The data images of cameras will then be saved in .tiff file, and a text file with all relevant scalar variables will be saved there as well. This data will be saved on the data servers in the location /reg/d/psdm/mec/mecxxxx/scratch/, with each run saved in a separate subdirectory :
This method works well for runs with a very limited number of shots (e.g. experiment with the MEC long pulse laser), but it becomes impractical when large amounts of data are being generated at higher repetition rates, in which case psana should be used.
Data can be transferred to out-side of LCLS using psexport machines, with some applications such as FileZilla, WinSCP, Cyberduck, etc