Dionisio, JJ, Matt, Ric, Wilko, Valerio
Discussing on-detector data reduction and (a little) about off-detector software data reduction.
Dec. 8, 2023
- epixUHR on-detector FPGA data reduction is not well-defined
- will descramble data to put in natural order
- could do dark subtraction
- can we also do gain correction?
- two types: auto-ranging, per-pixel
- how do we represent corrected data?
- do we want to have extra bits so all pixels are directly comparable, or keep separate high/low gain bits to keep data small?
- pseudo-floating point?
- do we want something "invertible"? (so can recover raw data)
- meeting with XPP scientists: they feel strongly that the calibrations can be "good enough" to trust in real time, based on their experience. they can deal with somewhat imperfect data.
- guess for on-detector to-do list:
- descramble
- correct the data
- threshold
- speckle-finding (like a more complex thresholding). like sparkpix S except with higher dynamic range?
- sparkpix data reduction is better defined
- multiple variants
- original: "sparkpix ED":
- send low-resolution coarse data (sum of 9 pixels)
- this coarse is for a level-0 "trigger decision"
- trigger decision made by FPGA on-detector or external FPGA?
- in small 1 ASIC prototype the trigger decision was done with the FPGA on-detector. how would this look for a large detector? would we aggregate the low-resolution across the detector pieces? unclear.
- if something says "event is interesting" then send all the data (whole camera)
- trigger decision made by FPGA on-detector or external FPGA?
- "sparkpix S"
- XPP will get this one
- low dynamic range (5 or 6 photons per pixel). no auto-gain-ranging.
- 12 bit ADC
- unclear what encoding of the pixel "address" is
- design of ASIC starts in January 2024
- checks threshold per-pixel in ASIC, if crosses threshold then send it out
- expected occupancy 3-5%
- "sparkpix T"
- a.k.a. tixel
- thresholded pixels
- get time and pixel-address, but no charge information
- "sparkpix RT"
- collaboration with ANL
- thresholding and compression ("poisson encoding"?). A variation of sqrt(N) logarithmic-style encoding? Reduces number of bits sent.
- not clear if a beam line will want this
- like an epixUHR?
- need to be able to swap out reduction algorithms "easily" (different FPGA image: doable with rogue)
- the data formats are non-trivial
- someone will need to decode
- cpo thinks we have a good system for this
- we label all detector data with a major-minor-micro version number
- two types:
- label individual arrays with a version number (data encoding change)
- label the aggregate of all arrays with a version number (software change)
- that determines which python class gets instantiated
- this does the decoding
- cpo thinks this version number covers both data changes and software (see above)
- could we do on-detector lossy compression (e.g. jpeg-style)
- do compression with corrected data or raw data?
- raw data is messy (gain, pixel-to-pixel pedestals)
- with SZ compression we use corrected data because SZ depends on standard formats (float32, uint16...)
- get a factor of 9
- why can't we do this with the raw data? could we do this with enough work? should we?
- what do we do if we can't send the data out? e.g. a high-occupancy image?
- feels like we need to dump the data and set damage?
- the sparsifying detectors will do the equivalent of this (e.g. if occupancy exceeds the 3-5% number)
- data reduction is done to save on disk space
- a "soft limit"
- exception: an epixUHR at 100kHz can't send out all its data
- could we find other bottlenecks (e.g. weka) that would introduce similar requirements. but buffering will help with this.
- for the data-reductions that algorithms that depend on the precise physics data values we have existing experiments/software (from LCLS1)
- have it for XPP speckle finding and TXI SAXS/WAXS.
Overview
Content Tools