Page History
Table of Contents |
---|
The XTCAV (X-band Transverse deflecting mode CAVity) is a detector that is used to determine the x-ray power vs. time of each LCLS shot. It can also be used to give information about the relative energies and arrival times of x-ray pulses that are closely spaced in time (fs delays). Unfortunately, some data analysis is necessary to extract this "power profile" from the raw detector signal. This documentation will go through how to do this.
...
You may notice there are vertical streaks in this image. Those streaks reflect the energy some electrons lost when they generated x-rays in the XFEL undulator! If we compare this image to a comparable image that did not generate x-rays ("lasing off" run, more on that later) then we can "reconstruct" the x-ray power as a function of time. For example, in the next figure, panel (a) shows a lasing off shot, and panel (b) shows a similar electron bunch that lased, a lasing on shot. Finally, panel (c) shows the power reconstruction produced by comparing these two shots.
Contents
Table of Contents |
---|
Quickstart
All XTCAV analysis is a three step process involving "dark", "lasing off", and "lasing on" runs. The first two are references; the last is how we commonly refer to runs that contain data of interest. Simple scripts are provided as part of the psana python environment for processing each.
Dark Runs
Dark runs are used to simply make pedestals for the (aka "darks") for the XTCAV camera and other diagnostics. These runs are recorded without any electrons going through the accelerator. We take the average of a series of dark run images to get the general pixel values of the camera background. We can then subtract this "average image" from the lasing on and off runs to get a closer approximation of the actual energy of the electron bunches.
...
Code Block | ||
---|---|---|
| ||
dark background reference Experiment: amox23616 Run: 104 Valid shots to process: 400 100.0 % done, 400 / 400 Maximum number of images processed |
Lasing Off Runs
Lasing off runs provide a reference an estimate of what the electron beam would have looked like if it did not lose any energy to generate x-rays. To provide this reference, a "lasing off" run is generated that has electons but no x-rays. The electrons are "kicked" so that they cannot generate x-rays (lase) as they travel through the undulator. These electron bunches are captured on the XTCAV camera. The standard analysis clusters together these lasing off images into an ( automatically or manually chosen ) number of groups. Each cluster of lasing off images is averaged to reduce noise, and then compared to "lasing on" bunches (next step, below). Clustering and subsequent matching to lasing on bunches is done by comparing the electron current as a function of time (projection of the XTCAV image onto the x-axis, the time axis), which is physically expected to be invariant, whether lasing occurred or not.
A script to generate lasing off references is provided with an interface very similar to the one for dark runs. It has quite a few options and should handle nearly all use cases.
Code Block | ||
---|---|---|
| ||
xtcavLasingOff amox23616 131 --max_shots 200 |
NOTE: in this example only 200 shots are used, but generally larger numbers of shots are important here (typically >1000) so that the lasing-off analysis has enough statistics for the various "clusters" of lasing-off behavior.
NOTE: see this page for guidance on how to set lasing-off analysis parameters: Old XTCAV Documentation#Lasing-offAnalysisParametersOutput
Code Block | ||
---|---|---|
| ||
LasingxtcavLasingOff amox23616 131 --max_shots 200 |
Output
Code Block | ||
---|---|---|
| ||
Lasing off off reference Experiment: amox23616 Runs: 131 Number of bunches: 1 Valid shots to process: 200 Dark reference run: None 100.0 % done, 200 / 200 Averaging lasing off profiles into 12 groups. |
...
Lasing On Runs
Runs with data of interest are commonly referred to as "lasing on". For each shot, we choose an aggregated lasing off profile with the most similar electron current. That reference is used to compute the energy that was lost to x-rays. There are two methods by which one can perform that calculation (physical details below): one can either compare the "energy loss" directly – ie subtract the mean electron energy in the lasing off run from the lasing on run -- or look at the increased energy spread in lasing on run vs. lasing off. Both are motivated by FEL theory.
...
Code Block | ||
---|---|---|
| ||
Using file 104-end.data for dark reference Using file 131-end.data for lasing off reference Agreement: 91.4276%; Maximum power: 6.64857; GW Pulse Delay: 18.512 Agreement: 91.1052%; Maximum power: 5.82139; GW Pulse Delay: 20.6758 Agreement: 93.136%; Maximum power: 4.88641; GW Pulse Delay: 18.0389 Agreement: 94.1173%; Maximum power: 6.76523; GW Pulse Delay: 16.2321 Agreement: 92.9344%; Maximum power: 7.14828; GW Pulse Delay: 11.2065 |
...
For real analysis, however, you likely want to incorporate some python code into whatever analysis masterpiece you are composing.
Code Block | ||||
---|---|---|---|---|
| ||||
from xtcav2.LasingOnCharacterization import LasingOnCharacterization XTCAVRetrieval = LasingOnCharacterization() for evt in ds.events(): XTCAVRetrieval.processEvent(evt) t, power = XTCAVRetrieval.xRayPower() |
Theory
The LCLS X-band Transverse deflecting mode CAVity (XTCAV) diagnostic system is shown in Fig. 1. There are two key elements to the system: the XTCAV and the final dipole magnet.
As shown in Fig. 1, immediately after the undulator is the XTCAV. This generates an effective RF (11.424 GHz) electromag- netic force pointing in the horizontal direction with an amplitude varying sinusoidally in time. Then, as illustrated in Fig. 2a, the electron bunch travels through the cavity with this RF wave at roughly the speed of light. The bunch time of arrival is set so its center coincides with when the amplitude of the RF field is zero. Then the net deflection of the bunch will also be zero.
However, the bunch has a finite length, some-thousand times shorter than the wavelength of the RF. As shown in Fig. 2a, the head sees a small, positive electric field while trailing electrons see a small, negative transverse electric field. The effect is a ∼linearly-varying force for different slices of the bunch. The head (tail) gets a transverse momentum kick to the left (right), shearing the bunch into a horizontal streak.
Analysis Setup
Two things must be done before XTCAV analysis will function: a "dark run" must be analyzed to get the pedestal values for cameras, and a "no lasing" run must be analyzed to generate sets of "no lasing" images (the latter is quite a complex process). Note that for demonstration these first two scripts write constants to a "local" calibration directory called "calib". For a real-experiment you won't need these lines because you will have permission to write to your official experiment calibration-constants directory.
Sample of dark run analysis:
Code Block | ||
---|---|---|
| ||
# these first two lines for example purposes only, to allow user to write
# calibration information to local directory called "calib"
# should be deleted for real analysis.
import psana
psana.setOption('psana.calib-dir','calib')
from xtcav.DarkBackgroundReference import *
DarkBackgroundReference(experiment='xpptut15',
run_number='300', # run number within experiment
max_shots=1000, # maximum number of shots to process
validity_range=(300,302)) # range of runs for which this dark run should be used |
Sample of "no-lasing" reference generating script:
Code Block | ||
---|---|---|
| ||
# these two lines for example purposes only, to allow user to write
# calibration information to local directory called "calib"
# should be deleted for real analysis.
import psana
psana.setOption('psana.calib-dir','calib')
from xtcav.LasingOffReference import *
LasingOffReference(
experiment='xpptut15',
run_number='301',
max_shots=200,
num_bunches=1,
validity_range=(301,)) #only give first run number to have the validity range be open-ended ("end") |
This script can be easily run in parallel by submitting a parallel MPI job to the batch system as described here. Once the dark/lasing-off analysis has been completed, users can analyze the lasing-on events using a standard psana-python script similar to the one below.
Example Analysis Script
This script assumes that dark/lasing-off data has been analyzed (see above). Unlike the previous two scripts it reads dark/lasing-off constants from the official calibration-directory.
Code Block | ||
---|---|---|
| ||
from psana import *
import matplotlib.pyplot as plt
from xtcav.LasingOnCharacterization import *
experiment = 'xpptut15'
run = '302'
mode = 'smd'
ds = psana.DataSource("exp=%s:run=%s:%s" % (experiment, run, mode))
ngood = 0
XTCAVRetrieval=LasingOnCharacterization()
for evt in ds.events():
if not XTCAVRetrieval.processEvent(evt):
continue # continue to next image if analysis fails for some reason
# time and power are lists, with each entry corresponding to
# a bunch number. The number of bunches is set by the GLOC.nb
# parameter in the lasing-off analysis. In general, one should
# also cut on the "agreement" value, which measures the agreement
# between the first and second moment analysis (larger is better).
time, power = XTCAVRetrieval.xRayPower(method='RMS')
agreement = XTCAVRetrieval.reconstructionAgreement()
ngood += 1
print 'Agreement: %g%% ' % (agreement*100)
plt.plot(time[0],power[0])
plt.xlabel('Time (fs)')
plt.ylabel('Lasing Power (GW)')
plt.title('Agreement %4.2f'%agreement)
plt.show()
if ngood > 1:
break |
Notes: The LasingOnCharacerization module uses the detector interface to find the datasource being used. The program will fail if you try to process events without first setting a datasource. If you are analyzing an older experiment, you may find that psana does not support the 'smd' mode. Instead, use the 'idx' mode.
One caveat: this data shows "horns" at the beginning and the end of the bunch which confuse the algorithm (the accelerator often behaves in this manner). Only the middle peak in the plotted spectrum is the relevant lasing peak.
Some tips for lasing-on analysis:
- The second-moment analysis (energy spread method) of the XTCAV image generally provides a better estimate of the true power profile of the XTCAV image. It is recommended that you use method='RMS' when calling xRayPower(), as demonstrated above.
- Ignore shots where the X-ray intensity is low, but cutting on the FEEGasDetector value to select stronger shots
Examining The "Ingredients" of the XTCAV Analysis
This is a utility that can help users understand why XTCAV results look the way they do. Run it like this:
Code Block |
---|
xtcavDisplay exp=xpptut15:run=302 |
It produces plots that look like this:
These plots show the following quantities for both lasing-on shot that is being analyzed and the lasing-off shot that it is being compared to (which is selected as the one having the closest current-profile):
- Current
- Energy computed using the first-moment ("Delta") method
- Energy computed using the second-moment ("Sigma") method
- Power
Close the existing plot window to show the plots for the next event.
XTCAV Analysis Parameters
-experiment: Name of XTCAV experiment (i.e. 'xpptut15')
-run_number: Run number within experiment. Can be in string or integer format.
-num_bunches: Number of bunches: typically 1, but can be 2+ for some LCLS experiments. For analysis of a lasing on run, this must be the same as the number of bunches in the reference lasing off run that you're using.
-max_shots: number of images to process for dark background or lasing off reference generation. In principle the bigger the better, but around 1000 should work fine. If you select a bigger number for dark background generation, then you should get a more representative average of the background noise. If you select a bigger number for lasing off reference generation, you will get more references. This will take longer to process but you'll have a better chance of finding a well matched profile when processing lasing on shots.
-num_groups: The number of groups into which lasing off profiles are clustered. Setting this number to 1 would average all of the lasing off profiles in the run together while setting this number to 'maxshots' would not average your profiles at all. Averaging profiles helps to remove some of the noise you might find in an individual profile but may also make comparisons less accurate. Our experiments have found that setting this parameter to maxshots/3 (which should theoretically lead to groups of ~3) works best.
-dark_reference_path/lasingoff_reference_path: File paths for the generated dark and lasing off references you want to use. If none given, the most recent generated reference file (with the run being processed included in the 'validity_range') will be chosen. You can set the `dark_reference_path` when initializing both the LasingOnCharacterization and the LasingOffReference. You can set the `lasingoff_reference_path` when initializing the LasingOnCharacterization.
-roi_expand (default 1): Ratio by which to expand the image from the max and min of the region of interest. This should not affect any analysis but will affect the returned image. Do not set to <1! Call the method 'processedXTCAVImage'. If the image does not look clipped, the parameters are fine.
-roi_fraction (default 0.001): fraction of pixels that must be non-zero in roi(s) of image for analysis. If you find that all of your images are registering as 'empty', try decreasing this parameter.
-snr_filter (default 10): Number of sigmas for the noise threshold. This is an important parameter. If you find that all of your images are registering as 'empty', try decreasing this parameter. If you find that your images contain a lot of noise, try increasing this parameter. This parameter is particularly important for 2+ bunch data. Since it sets a threshold based on the noise in an image, zeroing out pixels below that threshold, it is the factor that can separate the two island in case of two bunches. For two bunch data, you may need to tweak this parameter until the two bunches are separated for most of the shots. Bottom line:
- For single bunch: As low as possible while still removing the noise (i.e.) not getting noise current profiles.
- For double bunch: As low as possible while separating the two bunches. Call the 'processedXTCAVImage' method again, and look at the size of axis 0: this will be the number of bunches detected. You can probably write a program to automatize the determination of this parameter, until you get lets say a 95% of two bunch detection.
-validity_range (default `(current_run_number - end)`): The range of run numbers for which the reference run you are creating is valid. Parameter can be set in initialization of both the DarkBackground and the LasingOffReference. See example above. Note: Hopefully this is obvious, but if you set the validity range to (574,578) then the reference run will not be used for run 579 unless the reference path is explicitly given.
-start_image (default 0): Image within the run to start building the reference from. Can be set for `DarkBackgroundReference` and `LasingOffReference`.
-island_split_method: (default value: 'scipylabel') Only necessary when analyzing data with more than 1 bunch. Several image processing algorithms have been created to separate bunches that appear on the same image. See here for some algorithm details.
- 'scipyLabel' calls the scipy label function to label contiguous regions. this is fast, but requires a region of no signal (after de-noising) between the regions.
- 'autothreshold' tries to separate the islands by automatically finding thresholds that separate them. this is the recommended mode for multi-bunch operation.
- 'contourLabel' tries to adjust a threshold until two large groups are found, grouping together pixels using an opencv contour method. there are two parameters ('ratio1/ratio2') that are settable for the contour method that determine
Differences from Previous XTCAV code
- Changes to the user interface:
- All parameters should be set during the initialization of `DarkBackgroundReference`, `LasingOffReference`, and `LasingOnCharacterization`.
- There is no longer a `.Generate()` call. A reference will be created upon initialization of the `DarkBackgroundReference` and `LasingOffReference` classes. The reference will be saved to the psana data directory (default location unless the `calibration_path` parameter is set). If you would like to access the generated reference profiles, you can either load them from the saved file or access the member variable `averaged_profiles` within the `LasingOffReference` class.
- Some parameters, such as roi_waist_threshold, have been deprecated. The current code automatically chooses this value for you based on the `snr_filter` (see documentation above). Other parameter names have changed so that naming convention is consistent across the entire xtcav package (again, see documentation above).
- Changes to analysis:
- Improved profile clustering algorithm
- Image is no longer median filtered before analysis
- Power profile for lasing off image is not averaged between RMS and COM methods
- Other changes include: improved code readability, integration with psana detector interface, and speed up in processing time.
Detector Resolution
(From Tim Maxwell)
Time resolution is around 1.1 fs RMS for soft x-rays, 2.5 fs fwhm (in quadrature, of course). So actually pulse length is probably 4.3 - 9.7 fs FWHM.
...
An example script which generates some plots:
Code Block |
---|
import psana
from xtcav2.LasingOnCharacterization import LasingOnCharacterization
import matplotlib.pyplot as plt
import sys
ds = psana.DataSource('exp=xppc00120:run='+sys.argv[1]+':smd')
XTCAVRetrieval = LasingOnCharacterization()
for evt in ds.events():
XTCAVRetrieval.processEvent(evt)
# method 1: center-of-mass
t, powerCOM = XTCAVRetrieval.xRayPower(method='COM')
if t is None: continue
# method 2: RMS
t, powerRMS = XTCAVRetrieval.xRayPower(method='RMS')
agreement = XTCAVRetrieval.reconstructionAgreement()
plt.title('agreement: %4.2f'%agreement)
plt.xlabel('Time (fs)')
plt.ylabel('Power (GW)')
# a useful named-tuple with details of the shot analysis
#res = XTCAVRetrieval.fullResults()
# [0] index needed for potential multi-bunch analysis
plt.plot(t[0],powerCOM[0,:],label='COM')
plt.plot(t[0],powerRMS[0,:],label='RMS')
plt.legend()
plt.show() |
Theory & Implementation
The current XTCAV code implements two reconstruction methods – that is, ways to compare the lasing on and lasing off XTCAV images to compute the final power-vs-time profile. These methods are:
- The "energy loss" method, also known as "COM" for center of mass, which is the way the method is actually implemented. This method computes the FEL power directly from the energy lost by the electrons, using a simple conservation of energy argument:
P(t) = (<Eoff>(t) - <Eon>(t)) x I(t)
Here, <Eoff>(t) is the average energy of the electrons in the lasing off shot, <Eon>(t) is the average energy in the on shot, and I(t) is the beam current. Practically, the current is just the projection of the XTCAV camera intensity on the time axis. The average energies are determined by taking the center of mass of the XTCAV image (hence, COM) in time – ie, just the average electron energy vs. time. The difference between the off and on shots (energy loss) gives the final power. - The "energy spread" method, also known as "RMS" for root-mean-square method. This method computes the increased spread of the electron energies in the lasing on shot, which is predicted by FEL theory:
P(t) = [ σ2on (t) - σ2off (t) ] x I2/3(t)
You will notice that in the lasing on shots (e.g. in the early figures in this article), the electrons are more spread out after lasing. This lets us compute the power they lost to lasing through the associated FEL physics model. The RMS, denoted σ here, is simply the RMS deviation of the electron energies at each point in time. The 2/3 exponent on the beam current is a small correction due to these formulas scaling with the FEL efficiency (Pierce) parameter ρ which has a 1/3-power dependence on several slice properties of the beam. Beam current is one of them and is known by this measurement.
Differences from Previous XTCAV code
In June 2018, we pushed an update to the XTCAV code that is referred to at LCLS as "xtcav2". Yes, I know what you are thinking: Chris O'Grady and TJ Lane truly excel in the creative naming department and deserve a raise as a result. In many ways, the user interface should look the same, but there were a lot of improvements under the hood. Many thanks to our talented masters student Clara Meister for tackling that beast!
- Changes to the user interface:
- All parameters should be set during the initialization of `DarkBackgroundReference`, `LasingOffReference`, and `LasingOnCharacterization`.
- There is no longer a `.Generate()` call. A reference will be created upon initialization of the `DarkBackgroundReference` and `LasingOffReference` classes. The reference will be saved to the psana data directory (default location unless the `calibration_path` parameter is set). If you would like to access the generated reference profiles, you can either load them from the saved file or access the member variable `averaged_profiles` within the `LasingOffReference` class.
- Some parameters, such as roi_waist_threshold, have been deprecated. The current code automatically chooses this value for you based on the `snr_filter` (see documentation above). Other parameter names have changed so that naming convention is consistent across the entire xtcav package (again, see documentation above).
- Changes to analysis:
- Improved profile clustering algorithm
- Image is no longer median filtered before analysis
- Power profile for lasing off image is not averaged between RMS and COM methods
- Other changes include: improved code readability, integration with psana detector interface, and speed up in processing time.
Important Questions (Both Frequently and Infrequently Asked!)
What is the resolution of the XTCAV reconstruction?
From Tim Maxwell:
Time resolution is around 1.1 fs RMS for soft x-rays, 2.5 fs fwhm for hard x-rays. This doesn't include the "slippage resolution." That is, when using the full undulator, then by the end the x-rays can have slipped out of the electron slice by ~3 fs for soft x-rays. Obviously not a small number if trying to make (eg) 5 fs pulses. They've (ACR) been advised to not use the full undulator when shorter pulses are more important than number of photons.
How Do I Know When I Can Trust My Results?
GREAT QUESTION. The first, most obvious thing to do is compare the RMS ("energy spread") and COM ("energy loss") reconstruction methods. If these agree (90+% correlation), then you are on a good track... though other validation is desirable. If they disagree, however, you know you are hosed. For now, given what we know, we recommend throwing away any shot for which the two methods do not agree, unless you have some good reason to do otherwise.
If you want to know gross values, such as pulse duration, the timing between two pulses, etc, then if you have reconstruction agreement, you can be quite confident your analysis will work with ~3 fs precision in time and ~10 eV precision in energy, though exact values depend strongly on the XTCAV settings – discuss with ACR if you care! If you are studying the power structure of the pulse, then the honest answer is that we do not know exactly how good the reconstruction is – this is an open research question. We are currently working on ways to experimentally validate existing reconstruction methods and improve them. This is seen as a high priority by LCLS management and is funded through the FEL R&D program (as of time of writing, Oct 2018).
I see a "phase error". What do I do?
This warning can cause events to be thrown away, impacting analysis:
Code Block |
---|
The phase of the bunch with the RF field is far from 0 or 180 degrees |
To disable this throwing away of events, set an environment variable:
Code Block |
---|
export XTCAV_IGNORE_PHASE_WARNING=1 |
When this happens ACR should also be notified so they can adjust the phase on their end.
Tim Maxwell wrote about this issue on Dec. 11, 2022:
"I also see that the operating phase is indeed getting close to 0. I suspect the wording of the error message itself just had it the wrong way around. Seems to be the phase shifted after changing to low charge. We can also walk back the offset so that it no longer throws the error. (The physical phase is still actually +90 currently, so this is okay.)
You could make it only a warning if the phase is approaching 0 or 180. In this case, the risk is the time axis may be reversed. At some point it could be noted to ACR to please adjust the phase offset for the cavity.
In the case of lasing off references that may mean contaminating reconstructions if it's ambiguous though. Not noticing the warning, potentially getting it wrong, then matching all later events to a backwards image.
(Or the risk that the cavity would become unstable. But ACR would certainly notice this and fix it.)
Thinking of how we can just do this better on our side as well to not be an issue."
Who Do I Contact for More Help?
On XTCAV physics, parameters, limitations, improvements, etc: Tim Maxwell <tmaxwell@slac.stanford.edu>
On the workings of the XTCAV code in psana: Chris O'Grady <cpo@slac.stanford.edu>
Additional Resources
There is some good reading out there – including in peer-reviewed journals! – that can probably answer any of the detailed questions you have about the workings of the XTCAV, a truly sweet x-ray FEL diagnostic.
- The code: https://github.com/slaclab/xtcav2
- The "users document" from the accelerator directorate: xtcav-users-v0p4.pdf
- Slides from Tim Maxwell on the reconstruction procedure: 140613 - Analysis basics.pptx
- Paper published in Nat Comm (2014): Behrens-NatPhot-XTCAV.pdf
- Paper published in PRST (2011): Ding XTCAV.pdf