Requirements

• has to be joined to the "shutter closing" event, so it needs that timestamp (Dan says natural idea in IOC is timestamping with shutter-opening trigger).  Possible hack: use two different event-codes: one for opening the shutter (IOC ignores this one) and one for closing the shutter (IOC listens to this one for the timestamp).  
• data would contain the "shutter opening" timestamp: the makes the recorded data a more complicated structure, but maybe not: Matt points out that the timing system already has this information, so no need to complicate the data structure.

All of this requires event-code sequencing (complex).

(uncommon) Maybe not all experiments require precise timestamping: could use imprecise software triggering (python script, human doing something on keyboard?).  Dan thinks they always need to know the precise shots.

These two requirements suggest we might want to consider moving away from IOC/PV/AreaDetector reuse and use a standalone program, but maybe the hacks above would work?

RIX says they want this "in the future" (see the sub-page Integrating Detectors in RIX)

psana Design Options

(discussion on March 16, 2022)

simple timestamp example:

andor (1hz): s  4s  8 (s=start of integration)
hsd   (4hz): 12345-78 (minus=missing shot)

batch size now has to keep three things happy:
- performance/efficiency
- integrating detectors
- batching memory, which is only for small data (hopefully we can deal with this)
  o big data isn't batched, handled on event-by-event basis, and
    memory is handled naturally, so not a problem

requirement: only one integrating detector? more is OK but they have to
be synced.
- can't support conflicting requirements from 2 detectors where an
  an event has to go to two places
- verify with experimenters

timescale is Sept. 2022

option 1:
- put batching on an andor boundary (4, 8)
  o shots 1 thru 4 go to 1 EB core and 1 BD core
- user needs to specify a "special" stream file (andor, above)
- batch size becomes dynamic.
  o current batch size is fixed: default 1000, can be set by user
  o currently last batch is dynamic
- two ideas currently: chunk-size between SMD0 and EB, and batch
size between EB and BD. both are currently fixed numbers
- could use many EB cores (unlike destination callback)
- could put multiple andor images in 1 batch for efficiency

option 2:
workaround? maybe we could reuse the destination callback?
- has a limitation that we can only use 1 EB core
- Mona thinks that if "we" control destination callback then
  we can perhaps have more than 1 EB core.  SMD0 has to be in-charge

Question emailed to hutch scientists on mar 16, 2022:

As we move to higher rates we are thinking of ways we can support “integrating detectors”, e.g. a slow 1Hz camera running with a 1MHz high-speed detector (e.g. hsd) where you need to analyze precisely all 1MHz shots that contributed to the slow image (assuming that the slow device’s “shutter" is precise, which is likely not true for all devices … that’s a separate issue).

With psana parallelization (necessary for high-rate analysis) it will be difficult for us to support arbitrary integration periods for multiple detectors, e.g. detector1 integrating over shots 1,2,3 (and 4,5,6) while detector2 integrates over 1,2,3,4 (and 5,6,7,8).  We think we can support multiple detectors with one integrating at a “highest harmonic” (e.g. 1,2,3 then 4,5,6) and another simultaneously integrating in the same way, or at an integer “lower harmonic" (e.g., 1,2,3,4,5,6 then 7,8,9,10,11,12)

My question: can you let us know if you foresee that the above “highest harmonic” requirement would not be sufficient for your experiments?

EPIX Detector Integration Discussion (HR and M)

March 23, 2022 (cpo, lorenzo rota, phil hart, dan damiani, kaz nakahara, larry ruckman, mark McKelvey, matt weaver, jana thayer, valerio mariani)

• pedestal can change depending on integration time
  • could have continuous pedestal calculation (dropped shots get strange when integrating over shots: could drop many shots to get a dark image)
• CDS: correlated double-sampling (removed baseline)
  • Phil asks: do we need to turn this off?
  • Lorenzo says noise will take a big hit if CDS disabled (maybe not a big issue at low-rate readout?)
• Lorenzo isn't sure detector response is fast enough to reject MHz shots (10us OK, 1us tricky).  Harder to reject strong signal than weak signal.
  • different fpga clock-domains make it more difficult since they add effective jitter?
• Dionisio says this MHz response is a new requirement.  Should be discussed with Angelo.
• can the epix receive beam while it's being read out? ("gating")  answer: Maybe for epixHR (although Lorenzo/Dionisio will look to see if there is a sample-and-hold that may allow this), Yes for epixM (but need to investigate a bit more: could be issues)
  • fallback is to use "burst mode" to inhibit beam during readout (less efficient)
  • matt thinks that we'll have to inhibit the beam anyhow because response is different in different parts of the window
• maybe we send a RUN trigger at the beginning and a DAQ trigger at the end which has the "last shot" timestamp which gets attached to the image?
  • matt suggests setting acqWidth (currently 32 bits), and send down a DAQ trigger with the last-shot timestamp, and a RUN trigger with start-shot timestamp.
    • since acqWidth isn't precise Matt suggests dropping the beam to control the exposure (acqWidth clock will be tied to accelerator clock for UHR and beyond)
  • Dionisio says that dropped DAQ triggers could be problematic.  Matt's suggestion sounds better to him.
    • not a big deal to attach the second DAQ trigger timestamp to the image data
  • Lorenzo says that electronics settling time is 20-30us, so controlling the camera on timescales below that is problematic (related to Matt's comment about dropping the beam to be "conservative")
  • frame include both timestamps but the "real" one (from DAQ perspective) is the last-shot timestamp
  • want to start again as quickly as possible

easiest approach for camera, may make experiments more complicated (burst-mode, acqWidth precision):

• run the beam in burst-mode
• use acqWidth/RunTrigger/DaqTrigger (Phil worries about stability of acqwidth)

to do:

• need a future mtg with gabriel, georgi, kenney, angelo, aquila (use-case from georgi).  charge codes?
• Lorenzo/Dionisio think about "gating"

Second Meeting

April 13, 2022.  Main goal: to connect hutch scientists with detector builders to determine what can reasonably be achieved with the existing hardware.

Georgi requests:

Need I0 normalization.  Ideally 1MHz continuous beam with slower epixM (5kHz).

Andy requests:

TXI Need I0 normalization.  Burst mode is OK.  SPI with 10kHz beam but 5kHz imaging (most are blank).  How long does it take the detector to clear if another signal come in?  Same question as beam coming in while detector is reading out.

daq deadtime is an issue (can lose data from other detectors that contribute to image)

epixHR: 3 parts: presampling, reset front end (24us), integrate (minimum 24us?) then readout (sum of all these is 200us → 5kHz).  can't have any pulse come during reset because of the CDS (correlated double-sampling).  can reduce the R0 time (reset time) but can't be zero.  increasing the integration time, but adds to the total time (reducing rate).

dionisio/lorenzo say there is a sample/hold that can take a snapshot of the charge after integration.  Angelo says need to be tested if there are effects from outside the integration window (should not be a problem)

Can reduce the frame rate of the camera by increasing integration time (lowering frame rate) but leakage rate starts to matter which is more noise.  Can lower temp from +10 to -30 to reduce leakage current.

Angelo says 100us should be fine for epixHR.  epixM is more sensitive.  need to see how big the integration time can be for epixM.

Chris K. says that have to run beam at about <50kHz to avoid hitting the integration window.

Angelo says burst mode is key.

Andy says spectroscopy can really use the higher rate pulses.

Executive summary:

• need burst mode (no beam outside integration window)
• daq needs to think about timestamping requirements in burst mode
• longer integration requires better cooling (Chris K. suggests limit of 400us integration? maybe 800us? Angelo thinks above >100us for epixM could be problematic, but needs testing)
• different running modes require different calibrations (pedestals, gains, offsets)
  • need to have set of standardized conditions to choose from (requires psana support)
• Action item: test longer integration times for epixHR and epixM (400us or 800us). (2 weeks?)
• Action item: Andy/Georgi provide a few use-cases
• Action item: Detector group produces a list of examples that "make sense" e.g. frame rate vs. deadtime and noise issues
• Needed May 2023
• Need to know if epixM needs better cooling (there are limits to what can be done)