Requirements

  • has to be joined to the "shutter closing" event, so it needs that timestamp (Dan says natural idea in IOC is timestamping with shutter-opening trigger).  Possible hack: use two different event-codes: one for opening the shutter (IOC ignores this one) and one for closing the shutter (IOC listens to this one for the timestamp).  
  • data would contain the "shutter opening" timestamp: the makes the recorded data a more complicated structure, but maybe not: Matt points out that the timing system already has this information, so no need to complicate the data structure.

All of this requires event-code sequencing (complex).

(uncommon) Maybe not all experiments require precise timestamping: could use imprecise software triggering (python script, human doing something on keyboard?).  Dan thinks they always need to know the precise shots.

These two requirements suggest we might want to consider moving away from IOC/PV/AreaDetector reuse and use a standalone program, but maybe the hacks above would work?

RIX says they want this "in the future" (see the sub-page Integrating Detectors in RIX)

psana Design Options

(discussion on March 16, 2022)

simple timestamp example:

andor (1hz): s  4s  8 (s=start of integration)
hsd   (4hz): 12345-78 (minus=missing shot)

batch size now has to keep three things happy:
- performance/efficiency
- integrating detectors
- batching memory, which is only for small data (hopefully we can deal with this)
  o big data isn't batched, handled on event-by-event basis, and
    memory is handled naturally, so not a problem

requirement: only one integrating detector? more is OK but they have to
be synced.
- can't support conflicting requirements from 2 detectors where an
  an event has to go to two places
- verify with experimenters

timescale is Sept. 2022

option 1:
- put batching on an andor boundary (4, 8)
  o shots 1 thru 4 go to 1 EB core and 1 BD core
- user needs to specify a "special" stream file (andor, above)
- batch size becomes dynamic.
  o current batch size is fixed: default 1000, can be set by user
  o currently last batch is dynamic
- two ideas currently: chunk-size between SMD0 and EB, and batch
size between EB and BD. both are currently fixed numbers
- could use many EB cores (unlike destination callback)
- could put multiple andor images in 1 batch for efficiency

option 2:
workaround? maybe we could reuse the destination callback?
- has a limitation that we can only use 1 EB core
- Mona thinks that if "we" control destination callback then
  we can perhaps have more than 1 EB core.  SMD0 has to be in-charge

Question emailed to hutch scientists on mar 16, 2022:

As we move to higher rates we are thinking of ways we can support “integrating detectors”, e.g. a slow 1Hz camera running with a 1MHz high-speed detector (e.g. hsd) where you need to analyze precisely all 1MHz shots that contributed to the slow image (assuming that the slow device’s “shutter" is precise, which is likely not true for all devices … that’s a separate issue).

With psana parallelization (necessary for high-rate analysis) it will be difficult for us to support arbitrary integration periods for multiple detectors, e.g. detector1 integrating over shots 1,2,3 (and 4,5,6) while detector2 integrates over 1,2,3,4 (and 5,6,7,8).  We think we can support multiple detectors with one integrating at a “highest harmonic” (e.g. 1,2,3 then 4,5,6) and another simultaneously integrating in the same way, or at an integer “lower harmonic" (e.g., 1,2,3,4,5,6 then 7,8,9,10,11,12)

My question: can you let us know if you foresee that the above “highest harmonic” requirement would not be sufficient for your experiments?

EPIX Detector Integration Discussion (HR and M)

March 23, 2022 (cpo, lorenzo rota, phil hart, dan damiani, kaz nakahara, larry ruckman, mark McKelvey, matt weaver, jana thayer, valerio mariani)

  • pedestal can change depending on integration time
    • could have continuous pedestal calculation (dropped shots get strange when integrating over shots: could drop many shots to get a dark image)
  • CDS: correlated double-sampling (removed baseline)
    • Phil asks: do we need to turn this off?
    • Lorenzo says noise will take a big hit if CDS disabled (maybe not a big issue at low-rate readout?)
  • Lorenzo isn't sure detector response is fast enough to reject MHz shots (10us OK, 1us tricky).  Harder to reject strong signal than weak signal.
    • different fpga clock-domains make it more difficult since they add effective jitter?
  • Dionisio says this MHz response is a new requirement.  Should be discussed with Angelo.
  • can the epix receive beam while it's being read out? ("gating")  answer: Maybe for epixHR (although Lorenzo/Dionisio will look to see if there is a sample-and-hold that may allow this), Yes for epixM (but need to investigate a bit more: could be issues)
    • fallback is to use "burst mode" to inhibit beam during readout (less efficient)
    • matt thinks that we'll have to inhibit the beam anyhow because response is different in different parts of the window
  • maybe we send a RUN trigger at the beginning and a DAQ trigger at the end which has the "last shot" timestamp which gets attached to the image?
    • matt suggests setting acqWidth (currently 32 bits), and send down a DAQ trigger with the last-shot timestamp, and a RUN trigger with start-shot timestamp.
      • since acqWidth isn't precise Matt suggests dropping the beam to control the exposure (acqWidth clock will be tied to accelerator clock for UHR and beyond)
    • Dionisio says that dropped DAQ triggers could be problematic.  Matt's suggestion sounds better to him.
      • not a big deal to attach the second DAQ trigger timestamp to the image data
    • Lorenzo says that electronics settling time is 20-30us, so controlling the camera on timescales below that is problematic (related to Matt's comment about dropping the beam to be "conservative")
    • frame include both timestamps but the "real" one (from DAQ perspective) is the last-shot timestamp
    • want to start again as quickly as possible

easiest approach for camera, may make experiments more complicated (burst-mode, acqWidth precision):

  • run the beam in burst-mode
  • use acqWidth/RunTrigger/DaqTrigger (Phil worries about stability of acqwidth)

to do:

  • need a future mtg with gabriel, georgi, kenney, angelo, aquila (use-case from georgi).  charge codes?
  • Lorenzo/Dionisio think about "gating"

Second Meeting

April 13, 2022.  Main goal: to connect hutch scientists with detector builders to determine what can reasonably be achieved with the existing hardware.

Georgi requests:

Need I0 normalization.  Ideally 1MHz continuous beam with slower epixM (5kHz).

Andy requests:

TXI Need I0 normalization.  Burst mode is OK.  SPI with 10kHz beam but 5kHz imaging (most are blank).  How long does it take the detector to clear if another signal come in?  Same question as beam coming in while detector is reading out.

daq deadtime is an issue (can lose data from other detectors that contribute to image)

epixHR: 3 parts: presampling, reset front end (24us), integrate (minimum 24us?) then readout (sum of all these is 200us → 5kHz).  can't have any pulse come during reset because of the CDS (correlated double-sampling).  can reduce the R0 time (reset time) but can't be zero.  increasing the integration time, but adds to the total time (reducing rate).

dionisio/lorenzo say there is a sample/hold that can take a snapshot of the charge after integration.  Angelo says need to be tested if there are effects from outside the integration window (should not be a problem)

Can reduce the frame rate of the camera by increasing integration time (lowering frame rate) but leakage rate starts to matter which is more noise.  Can lower temp from +10 to -30 to reduce leakage current.

Angelo says 100us should be fine for epixHR.  epixM is more sensitive.  need to see how big the integration time can be for epixM.

Chris K. says that have to run beam at about <50kHz to avoid hitting the integration window.

Angelo says burst mode is key.

Andy says spectroscopy can really use the higher rate pulses.

Executive summary:

  • need burst mode (no beam outside integration window)
  • daq needs to think about timestamping requirements in burst mode
  • longer integration requires better cooling (Chris K. suggests limit of 400us integration? maybe 800us? Angelo thinks above >100us for epixM could be problematic, but needs testing)
  • different running modes require different calibrations (pedestals, gains, offsets)
    • need to have set of standardized conditions to choose from (requires psana support)
  • Needed May 2023
  • Need to know if epixM needs better cooling (there are limits to what can be done)
  • Action item: test longer integration times for epixHR and epixM (400us or 800us). (2 weeks?)
  • Action item: Andy/Georgi provide a few use-cases
  • Action item: Detector group produces a list of examples that "make sense" e.g. frame rate vs. deadtime and noise issues
  • Action item: DAQ/analysis group figure out timestamping requirements in burst mode

Third Meeting

April 28, 2022

  • DAQ wants: a run trigger (always running), acq-width, DAQ trigger.  NEW: Image needs to be timestamped with the DAQ trigger timestamp.
    • action item for DAQ: if we're running "normally" (no integration) what needs to change on the DAQ end?
    • need to have run trigger timestamp in the data passed back
    • acq_width has a precision of 10ns (detector clock).  small compared to 1us time between shots, so should be precise enough to identify shots, even with clock-domain-crossing jitter
  • Dionisio looked at HR longer integration times: boards have just arrived back.  perhaps done in next 2 weeks.
  • Lorenzo looked at M longer integration times: could only study up to 20us because firmware hard to change.  measured noise and pedestal shift and extrapolate.
    • when can he go to longer times? needs prototype board. few months.
    • andy wants to distinguish at the 1 photon level, ideally half photon level.  800eV photon is 200electrons.  Georgi was 285eV, so 85 electrons so 2-sigma above the 35 electrons lorenzo predicts at 400us integration time.  Georgi is OK with it since they're not interested in single photon information when integrating.
    • may have better noise performance with next-gen boards
  • Andy Aquila use-case: metric: number of photons per pulse times rep rate (number of photons per second)
    • spectrometer is only use case
    • expects shorter integration times
    • low fluence (1 photon per pixel per shot) (~100 photons per 1000 pixels).  integrating over 100-300 shots would be fine.  maximum occupancy is one photon per pixel per shot.
    • probably run at 300kHz max
    • dionisio: in high gain 400us (high gain), less in medium gain
  • Georgi use-case:
    • is it beneficial to integrate?  maybe not.  Chris K. argues yes: since the beam rate helps so much, vs. the noise penalty
    • not important to see single photons
    • important to know which pulses contributed to image
  • For deadtime vs. framerate: always have readout deadtime
    • livefraction=(integrationtime-20us)/(150us(readout time)+24us(presampletime)+integrationtime)
  • there may be limitations on what the accelerator will allow the experiments to do vis-a-vis beam-kicking (e.g. the dump can't take the full beam-rate)
  • could timing system put sequence in the config object?  can't trivially put the sequence-generating-code in the config object, but can see the result: the event codes in the timing-system data.

Slides from Lorenzo showing behavior of epixM noise vs. integration time (up to the 20us he could currently reach):

20220428_ePixM_integ.pptx

To Do List

  • epixHR noise measurements vs. integration time (Dionisio will email out)
  • Longer term (months) epixM noise measurements vs. integration time
  • make sure not too much has to change for DAQ between integrating-mode and "normal" mode
    • camera shouldn't change
    • just have to switch between running a sequence and normal running
  • next meeting will not be held until we have noise measurements from both epixM and epixHR (waiting some months for new epixM hardware).

DAQ Timestamping Requirements

Discussion with Matt, Valerio, Chris on April 14, 2022

Ascii-art timing diagram:

------------BBBBB-----------BBBBB-------
epixtrig  R       D       R       D
hsdtrig     DDDDDDD         DDDDDDD
acqwidth    /---\           /---\

time ---->

Legend:
R=readout trigger (start acqwidth integration)
D=daq trigger (do readout)
B=beam

How flexible are we?
- The D for epixtrig has to be the same as the last D for hsdtrig because
  of Ric's EB requirement.
- B's have to contained within acqwidth
- R must come before acqwidth
- can have extra triggers with no beam after acqwidth, but doesn't help anything

when we're not running always send R to keep camera temperature stable

original proposal:
- we use the epixtrig D timestamp to label the event, and also include R timestamp in the data
- mona would use the epixtrig D timestamp to mark the end of a psana batch that gets sent to
  1 core (allows the core to sum up the appropriate events to get I0)

question: now that we're using burst mode, can we have a simpler timestamping scheme?
- matt suggested that we could look at the timing system sequence data to see when the end
  of a burst
  o minus: we already have to timestamp epix with D (no matter what). original proposal
    feels "self contained" with the epix detector. only adds complexity.

conclusion: feels like we should stick with original proposal


Normalization for Integrating Detectors

We typically accumulate shot-by-shot measurements of intensity from BLD, for instance, to use as normalization for the exposure of the integrating detectors.  Deadtime will sometimes result in the loss of some of the events that contribute to the integrated normalization.  How do we estimate this loss?  The XPM measures deadtime, but it is not added to the science data in any way.  One possibility is to capture the total deadtime (lost events/total events) at EndRun or EndStep.  Another possibility is to capture that number for a given trigger.  That could involve adding the deadtime numbers into the timing segment level, for instance, by modifying the XPM to broadcast those numbers on the timing stream.


  • No labels