Page History
Mar. 24, 2017: "Reasons For Re-Processing Data"
Link to slides:
* Cheetah has peak finder that is used a lot in crystallography
* Question of how to validate beam center finding algorithm and detector geometry
* bad pixels: after strong bragg peak the pixel might be temporarily damanged and should be masked out for consecutive shots until it recovers
## Reasons for rerunning data
* Anton Barty, Tom, Alexander (CFEL)
* poor detector calibration
* if results are not obvious to confirm if algorithm is working
* spectroscopy if detector corrections are required
* jet streaks for angular integration
* wrong event code triggereing, confirming that pump laser was running, backup diagnostic
* parameter tuning for example for peak finding
* hitfinding mostly works on first pass -> 10 times data reduction, indexing multiple times
* wrong hitfinding might bias physics
* if there is not enough data so hitfinding is important to really get all hits
* reprocessing data with new algorithms
* reliable gain maps for detectors (escpecially for high intensity), nonlinear detector response
* diffuse scattering requires whole image
* Peter Schwander
* veto signal for SPI for non hits, real time hitfinding
* convert data to single photons
* Filipe Maia
* detector artifacts
* signal influences gain for pnccd and cspad
* hitfinding can be diffcult for single protein SPI, because signal is very weak
* use ion tof or other hardware tools for hitfinding
* Mariano Trigo
* time tool calibration important for cube
* Ti-Yen
* halo close to beamcenter makes hitfinding diffcult for SPI
* converting ADU to photons is sufficient for SPI
* Aaron Brewster
* reprocessing because of unkown crystal unit cell
* unit cell can drift during the experiment, depending on sample preperation
* Peter Zwart
* hitrate can be quite high for imaging can be up to 80%
* clustering algorithm for hitfinding
* stable beam and sample delivery required
* difficulties in converting ADU to photons for pnccd, rounding errors
* check photon conversion on simulated data, understand errors in conversion
Extra detail from Anton Barty:
Thoughts from Tim van Driel:
If we are careful with measurement, and the diagnostic tools perform adequately we can instead rely on littledata and cube. To fully rely on littledata and cube, we would require both for future experiments as they are sensitive/insensitive to different types of errors.
When going from Full data analysis to cube/littledata the same corrections are needed all in all, but the necessity differs from littledata to cube processing.
If new detectors behave less ideally than the CSPAD does now, we are back to needing the full datasets to develop the necessary filtering and corrections.
A quick note regarding radial integration: All pump-probe diffuse scattering experiments have anisotropy at early times (usually <10ps, but can be up to ns) it may be negligible if the solute signal is relatively large as for protein crystallography. The anisotropy can be separated using legendre filters of different order but is probably easiest to do on the fly by integrating the data along phi and theta. I would use at least 17 bins which makes the assumption of 1e3 reduction for diffuse scattering 1e2 instead.
Reasons for reprocessing the data:
Full data analysis (used on experiments before 2016)
- Detector calibration
- Detector geometry
- Common mode subtraction
- Sample-detector distance
- Correlated behavior, outliers non-linear corrections
- Time-tool calibration
- Masking
- Binning
- Experimental detector corrections (solid angle coverage, polarization, jet geometry, sample composition)
Littledata (used from 2016)
- Detector calibration
- Detector geometry
- Common mode subtraction
- Sample-detector distance
- Correlated behavior, outliers non-linear corrections
- Masking
- Experimental detector corrections (solid angle coverage, polarization, jet geometry, sample composition)
Cube (used from 2016)
- Detector calibration
- Common mode subtraction
- Correlated behavior, outliers non-linear corrections
- Time-tool calibration
- Binning
- Outlier rejection (usually based on littledata analysis)
XES (dispersed spectral signal on small area detector)
CSPAD 140k (before 2016)
- Detector calibration
- Common mode subtraction
- filtering based on XDS
- pixel-by-pixel analysis to separate 1-photon peak from noise, the choice of algorithm depends on the signal strength on the detector
- masking
EPIX
- Detector calibration
- Dropletizing parameters
- Droplet output
XAS (0d signal on a diode or on a small area detector)
- Detector calibration
- Common mode subtraction
- masking
Feb. 22, 2017: "Introduction To Data Reduction"
Link to slides:
...