Presentation

A presentation from Matt:

...

Fundamental Numbers

• The accelerator provides exactly 910,000 buckets in exactly 0.98 seconds
• The accelerator patterns repeat on this 0.98s interval
• The ratio of these two numbers is what gives rise to the 929kHz (~1MHz) machine rate.  Equivalently: accelerator rate is 13/14MHz, bucket period is 14/13us.
• timing eventcodes arrive ~100us prior to beam
• The 71428Hz rate is special: common between LCLS1 and LCLS2

Presentation

Recording of Matt's presentation on the timing system.

Link to Matt's presentation.

...

It's a common question to ask what are the rates that the accelerator can generate.  The table below lists the rates that are natural for the accelerator.  These are the periodic rates that repeat exactly at the ~0.98 second period of the timing/MPS system.  Of course, we can generate other rates as well that aren't exactly periodic by doing the sort of thing you described - dropping shots or irregular sequences.  This has been useful for the OPCPA laser development in deciding the exact frequencies they will support.

If a base accelerator rate is chosen (e.g. 33kHz) then other rates having the same prime factors as the accelerator rate are allowed, according to this table:

(python3_env) [weaver@lcls-dev3 lcls2-timing-patterns]$ python lcls/fixed_rate_table.py

Matt provides this example of how you can determine if various trigger rates overlap: 16kHz (16581 in the table) has factors (2,2,2,7).  102Hz has factors (2,2,5,5,7,13).  It's missing one factor of 2, so it only overlaps 1/2 of the time.  51Hz has (2,2,2,5,5,7,13), so it contains all the factors from 16kHz and thus overlaps 100%.

If a base accelerator rate is chosen (e.g. 33kHz) then other rates having the same prime factors as the accelerator rate are allowed, according to this table:

(python3_env) [weaver@lcls-dev3 lcls2-timing-patterns]$ python lcls/fixed_rate_table.py

 rate, Hz  | factor | factors
 928571 rate, Hz  | factor | factors
 928571          1   1
 464285          2   (2,)
 232142          4   (2, 2)
 185714          5   (5,)
 132653          7   (7,)
 116071          8   (2, 2, 2)
  92857 1   1
 464285         10 2   (2, 5)
  71428 232142         13 4   (132, 2)
  185714 66326         14 5   (25, 7)
  132653 58035         16 7   (2, 2, 2, 2)
  464287,)
 116071         20 8   (2, 2, 52)
  3714292857         2510   (52, 5)
  3571471428         2613   (213, 13)
  3316366326         2814   (2, 2, 7)
  2653058035         3516   (52, 2, 2, 72)
  2321446428         4020   (2, 2, 2, 5)
  1857137142         5025   (25, 5, 5)
  1785735714         5226   (2, 2, 13)
  1658133163         5628   (2, 2, 2, 7)
  1428526530         6535   (5, 137)
  1326523214         7040   (2, 52, 2, 75)
  1160718571         8050   (2, 2, 25, 2, 5)
  1020417857         9152   (72, 2, 13)
  16581  9285         10056   (2, 2, 52, 57)
   892814285          10465   (25, 2, 2, 13)
   8290 13)
  13265          11270   (2, 2, 2, 25, 7)
   742811607          12580   (52, 2, 2, 52, 5)
  10204  7142         13091   (27, 5, 13)
   6632 9285        140 100   (2, 2, 5, 75)
   5306 8928        175 104   (52, 2, 52, 713)
   5102 8290        182 112   (2, 72, 132, 2, 7)
   4642 7428        200 125   (25, 2, 5, 5)
   7142        130   (2, 5, 513)
   4464 6632        208 140   (2, 2, 25, 2, 137)
   3714 5306        250 175   (2, 5, 5, 57)
   3571 5102        260 182   (2, 2, 57, 13)
   3316 4642        280 200   (2, 2, 2, 5, 75)
   2857 4464        325 208   (52, 52, 2, 2, 13)
   2653 3714        350 250   (2, 5, 5, 75)
   2551 3571        364 260   (2, 2, 75, 13)
   2321 3316        400 280   (2, 2, 2, 2, 5, 57)
   2040 2857        455 325   (5, 75, 13)
   1857 2653        500 350   (2, 2, 5, 5, 57)
   1785 2551        520 364   (2, 2, 27, 5, 13)
   1658 2321        560 400   (2, 2, 2, 2, 5, 75)
   1485 2040        625 455   (5, 57, 5, 513)
   1428 1857        650 500   (2, 2, 5, 5, 135)
   1326 1785        700 520   (2, 2, 52, 5, 713)
   1275 1658        728 560   (2, 2, 2, 72, 5, 137)
   1061 1485        875 625   (5, 5, 5, 75)
   1020 1428        910 650   (2, 5, 75, 13)
    1326 928       1000 700   (2, 2, 2, 5, 5, 57)
    1275 892       1040 728   (2, 2, 2, 27, 5, 13)
    742 1061       1250 875   (2, 5, 5, 5, 57)
    1020 714       1300 910   (2, 2, 5, 57, 13)
    663928       14001000   (2, 2, 2, 5, 5, 75)
    637892       14561040   (2, 2, 2, 2, 75, 13)
    571742       16251250   (2, 5, 5, 5, 135)
    530714       17501300   (2, 52, 5, 5, 713)
    510663       18201400   (2, 2, 2, 5, 75, 137)
    464637       20001456   (2, 2, 2, 2, 57, 5, 513)
    408571       22751625   (5, 5, 75, 13)
    371530       25001750   (2, 2, 5, 5, 5, 57)
    357510       26001820   (2, 2, 2, 5, 57, 13)
    331464       28002000   (2, 2, 2, 2, 5, 5, 75)
    285408       32502275   (2, 5, 5, 57, 13)
    265371       35002500   (2, 2, 5, 5, 5, 75)
    255357       36402600   (2, 2, 2, 5, 75, 13)
    212331       43752800   (52, 2, 2, 52, 5, 5, 7)
    204285       45503250   (2, 5, 5, 75, 13)
    185265       50003500   (2, 2, 2, 5, 5, 5, 57)
    178255       52003640   (2, 2, 2, 2, 5, 57, 13)
    142212       65004375   (25, 2, 5, 5, 5, 137)
    132204       70004550   (2, 2, 2, 5, 5, 57, 713)
    127185       72805000   (2, 2, 2, 25, 5, 75, 135)
    114178       81255200   (52, 2, 2, 52, 5, 5, 13)
    106142       87506500   (2, 52, 5, 5, 5, 713)
    102132       91007000   (2, 2, 2, 5, 5, 75, 137)
    127  92       100007280   (2, 2, 2, 2, 5, 57, 5, 513)
     81114        113758125   (5, 5, 5, 75, 13)
     71106        130008750   (2, 25, 2, 5, 5, 5, 137)
     66102        140009100   (2, 2, 2, 2, 5, 5, 57, 713)
     57 92      16250 10000   (2, 2, 2, 52, 5, 5, 5, 135)
     53 81      17500 11375   (2, 2, 5, 5, 5, 57, 713)
     51 71      18200 13000   (2, 2, 2, 5, 5, 75, 13)
     40 66      22750 14000   (2, 52, 2, 2, 5, 5, 75, 137)
     35 57      26000 16250   (2, 25, 2, 2, 5, 5, 5, 13)
     28 53      32500 17500   (2, 2, 5, 5, 5, 5, 137)
     26 51      35000 18200   (2, 2, 2, 5, 5, 57, 5, 713)
     25 40      36400 22750   (2, 2, 2, 25, 5, 5, 7, 13)
     20 35      45500 26000   (2, 2, 52, 2, 5, 5, 75, 13)
     16 28      56875 32500   (52, 2, 5, 5, 5, 75, 13)
     14 26      65000 35000   (2, 2, 2, 5, 5, 5, 5, 137)
     13 25      70000 36400   (2, 2, 2, 2, 5, 5, 57, 5, 713)
     10 20      91000 45500   (2, 2, 2, 5, 5, 5, 7, 13)
      8 16     113750 56875   (2, 5, 5, 5, 5, 7, 13)
      14 7     130000 65000   (2, 2, 2, 2, 5, 5, 5, 5, 13)
      5 13     182000 70000   (2, 2, 2, 2, 5, 5, 5, 75, 137)
      10 4     227500 91000   (2, 2, 52, 5, 5, 5, 7, 13)
      28     455000113750   (2, 2, 2, 5, 5, 5, 5, 7, 13)
      17     910000130000   (2, 2, 2, 2, 5, 5, 5, 5, 7, 13)

Event Codes

...

• some have well-defined meanings (in progress)
• 16 highest bits are hutch specific for sequences (272-287)
• DAQ readout groups are "extra bits" included at end of timing frame

...

13)
      5     182000   (2, 2, 2, 2, 5, 5, 5, 7, 13)
      4     227500   (2, 2, 5, 5, 5, 5, 7, 13)
      2     455000   (2, 2, 2, 5, 5, 5, 5, 7, 13)
      1     910000   (2, 2, 2, 2, 5, 5, 5, 5, 7, 13)

Triggering Devices With Beam

To trigger on "every shot" with beam (the equivalent of LCLS1's eventcode 140) use "Mode=FixedRate:1MHz, Destination=Include:DumpSXR" (without the "destination" field I believe you will just get a pure 1MHz trigger)

...

.  See table below where DumpSXR destination corresponds to destination bit 4.   It is also important to set the "select" field to "Inclusive" (see below for daq configuration object GUI).

Another important method (used especially for low rate 1,10,100Hz triggers that are in-synch with the beam) is to use an event-code generated by ACR (see Control Sequence Bit for currently supported event codes).  cpo thinks event codes like "SC_SXR BSA" (e.g. 100Hz is eventcode 30) on that page would be commonly used by the hutches. NOTE: We have seen empirically that these event codes do not include a destination setting, so it is necessary to also select destination 4 with the "select" field set to "inclusive" with them.  Matt writes: "Event code 30 will follow the offset used by SXR, but it isn't aware of MPS mitigating the beam rate.  We use the destination 4 filter to also pickup when MPS has inhibited the beam."

Image Added

Some details:

• there are 288 bits of "event codes" available
  • some have well-defined meanings, like the low-rate ones described above (in progress)
  • 16 highest bits are hutch specific for sequences (272-287)
  • DAQ readout groups are "extra bits" included at end of timing frame
• timing frames have "destinations": e.g. bykiks, and bykikh both go to "bsy" dump
• unlike LCLS1 we will not be using event-codes to understand when bykiks has fired: use destinations instead, in particular DumpBSY for "dropped" or "background" shots
• some devices reference to "sequencer engineer number" and "sequence bit number" instead of "eventcode".  The formula to convert between the two: eventcode=(sequenceEngine#<<16 | sequencerBit#).

Trigger Destinations and Types

Timing receivers share a common logic module for generating triggers.  They consist of the logical AND of two components, a rate component and a beam destination component.

• Rate component.
  • sequences / event code
  • ac-line rate (1,5,10,30Hz...) + timeslot mask
  • fixed-rate patterns (1H, 10H, 100H, 1kH, 10kH, 71kH, 1MH)
• Beam destination component
  • "Dont Care" : beam or no beam, we don't care
  • "Inclusive" : specify a set/mask of destinations to which the beam must be destined; see the table below
  • "Exclusive" : specify a set/mask of destinations to which the beam must not be destined; see the table below.

Note: destination "DumpBSY" corresponds to dropped-shots.

Need to coordinate hutch sequence-pattern scripts with accelerator sequence-pattern (patterns repeat on 1H marker).

Sequencing Example

See https://github.com/slac-lcls/lcls2/blob/master/psdaq/psdaq/seq/rixexample.sh (high level example for bursts and period patterns) and Integrating Detectors in RIX.  See also Matt's documentation.

In this example the XPM is configured to insert event codes 272-287 into the timing stream.  Four event codes are generated by each sequence engine within the XPM.  The XPM has 4 sequence engines allowing 16 event codes total.

A current example for doing this is done by executing:

(first use Matt's high-level utility to generate two event codes that run at 10Hz and 100Hz (periods 91000 and 9100 respectively)

(ps-4.5.26) drp-srcf-mon001:psana$ periodicgenerator -p 91000 9100 -s 0 0 -d '10Hz' '100Hz' >& 10hz_100hz.py

(now program sequence engine 0 and start it using the python code generated above. since we're programming engine 0, the first two event codes will be 272,273 according to the above table)

(this command must be executed on a node that can access the XPM PV's, e.g. drp-srcf-*)

(ps-4.5.26) drp-srcf-mon001:psana$ seqprogram --seq 4:10hz_100hz.py --pv DAQ:NEH:XPM:3 --start
** engine 0 fname 10hz_100hz.py **
descset  None
seqcodes {0: '10Hz', 1: '100Hz'}
Remove -1
idx 2
Removing seq 2
[10, 1, 5, 3, 0, 0, 0, 0, 2, 0, 6, 2048, 0, 0, 0, 3, 2, 1, 3, 3, 0, 0, 2, 0, 6, 908, 0, 0, 0, 1, 5, 2, 0, 0, 0, 0, 3, 2, 1, 0, 8, 0, 0, 2, 0, 6, 2048, 0, 0, 0, 3, 2, 6, 3, 3, 0, 0, 2, 0, 6, 908, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0]
Confirmed ninstr 10
Sequence  found at index 2
desc ['10Hz', '100Hz', '', '', '', '', '', '', '', '', '', '', 'burst', '', '', '']
seqcodes_pv epics:nt/NTTable:1.0 
    string[] labels [EventCode, Description, Rate, Enabled]
    structure value
        int[] EventCode [272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287]
        string[] Description [10Hz, 100Hz, , , , , , , , , , , burst, , , ]
        int[] Rate [11,102,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
        byte[] Enabled [1,1,1,1,0,0,0,0,0,0,0,0,1,1,1,1]
    string descriptor 
    alarm_t alarm
        int severity 0
        int status 0
        string message 
    time_t timeStamp
        long secondsPastEpoch 1683163040
        int nanoseconds 339389440
        int userTag 0

(now query the xpm to see what is running in the 4 sequence engines, each having 4 event codes)

(ps-4.5.26) drp-srcf-mon001:psana$ pvget DAQ:NEH:XPM:3:SEQCODES
DAQ:NEH:XPM:3:SEQCODES 2023-05-03 18:17:31.339    
EventCode Description Rate Enabled
      272        10Hz   10       1
      273       100Hz  102       1
      274                0       1
      275                0       1
      276                0       0
      277                0       0
      278                0       0
      279                0       0
      280                0       0
      281                0       0
      282                0       0
      283                0       0
      284       burst    0       1
      285                0       1
      286                0       1
      287                0       1
(ps-4.5.26) drp-srcf-mon001:psana$ 

Note: the periodicgenerator "repeat" argument specifies the number of repeats of the slowest event-code.

There is a similar "traingenerator" script which you can read about in Matt's documentation.

Visualizing Sequences

This can be done with the same environment used to generate the sequences above.

cd ~tmoopr/daq
source setup_env.sh
seqplot --seq 0:codes.py 3:beam.py --time 2.0

codes.py and beam.py were created with these scripts in the psdaq package:

traingenerator -s 14000 -b 28 -n 32001 -r 2 -d "burst" -t 910000 > beam.py
periodicgenerator -p 91 91000 -s 0 0 -d '10kHz' '10Hz' --repeat 3 --notify > codes.py

The seqplot command brings up this window which shows the two periodic event codes (programmed into the first simulated sequence engine) and the one "beam burst" event code (programmed into the fourth simulated sequence engine):

Image Added

Sequence Start/Stop

Some technical details from a discussion on March 23, 2023

Since a sequence in a lower-level XPM can "overwrite" sequence eventcodes from a higher-level XPM it is important to be able to stop lower-level sequences.  Matt says one does this by setting DAQ:NEH:XPM:2:SEQENG:3:ENABLE (for example) to 0.  The current example where we have to careful of this is for XPM0 which the laser group uses for eventcodes 272,273,274.

Matt has added a Groups/EventCodes tab to the xpmpva tool.  It shows you the effect of overwriting and which xpm is the current source of eventcodes.

currently the XPM starts sequences on 1Hz boundary (e.g. every 910000 buckets). this slows down the daq scan steps, so support for a second mode is being considered:

(1) current seq mode (only program time-in-seconds or numL1Accepts)
    have to coordinate the start of an accelerator burst with the daq sequence
    - are there two sequences?  one for accelerator and one for daq?
      not clear how much control the accelerator will want (e.g. burst
      all the time for stability, but silke says can damage samples)
    - each step we start one sequence:
      o may also include control of accelerator, but if not
      o have to coordinate the start of the daq seq with accelerator seq
        o acc seq synced to 1Hz marker (everything repeats at 1Hz on
      accelerator side)
    o trickier to sync to rates faster than 1Hz
(2) future-firmware seq mode
    - make the daq enable/disable on a user-defined event code
      o would be faster on the enable since don't have to wait for 1Hz
        marker.  still have to understand the periodicity of the accelerator
    so the user-defined event codes get sent down at the right time
    but it does mean we don't need the slow 1Hz marker.  some things like
    laser electronic delay scanning may be faster than 1Hz so could benefit
    from not using the 1Hz marker.

andor integrates over 10 shots
6 andor images in a step

mode (1):
- program 6 andor-readout-group "L1Accepts".  might be set to the parent
  group now in control.py

the readout group that is counted for the sequence is currently managed
by these lines in control.py.  Currently defaults to the "platform" readout
group which is the primary readout group: need to make this more flexible
so we can change it to the integrating detector readout group:

self.pv_xpm_part_base = pv_base + ':XPM:%d:PART:%d' % (xpm_master, platform)
self.pva.pv_put(self.pva.pvStepEnd, self.readoutCumulative)

Generating a Low Rate Sequence

From Matt.

periodicgenerator --period 1820000 --start_bucket 91000 --description 'Example 0.5Hz sequence' > 0.5Hseq.py
(ps-4.6.1) bash-4.2$ seqplot --seq 0:0.5Hseq.py --time 4.0
eng 0 fn 0.5Hseq.py
0: FixedRateSync(929kHz) # occ(2048)
1: Branch to line 0 until ctr3=43
2: FixedRateSync(929kHz) # occ(888)
3: ControlRequest word 0x1 [0]
4: FixedRateSync(929kHz) # occ(2048)
5: Branch to line 4 until ctr3=843
6: FixedRateSync(929kHz) # occ(488)
7: Branch unconditional to line 0
start, stop: 0,3640000
engine exited request 0  instr 1  returnaddr None  frame 3642047  ccnt [0, 0, 0, 0]

Should show eventcode 272 at bucket 91,000 and every 1,820,000 (2 seconds) after that.

Image Added

Rates

• timing arrives 100us prior to beam
• accelerator max rate is 13/14MHz=929kHz
• The 71428Hz rate is special: common between LCLS1 and LCLS2
• 928kHz is max rate, but accelerator pattern repeats every 910k buckets

Trigger Types

• timing trigger types:
  • sequences
  • ac-line rate (1,5,10,30Hz...)
  • fixed-rate patterns (1, 10, 100, 1000, 10000, 71kHz, 929kHz)
• need to coordinate hutch sequence-pattern scripts with accelerator sequence-pattern

Sequencing Example

See https://github.com/slac-lcls/lcls2/blob/master/psdaq/psdaq/seq/rixexample.sh (high level example for bursts and period patterns) and Integrating Detectors in RIX.  See also Matt's documentation.

In this example the XPM is configured to insert event codes 272-287 into the timing stream.  Four event codes are generated by each sequence engine within the XPM.  The XPM has 4 sequence engines allowing 16 event codes total.

...

Engine

...

Event Codes

...

A current example for doing this is done by executing:

(first use Matt's high-level utility to generate two event codes that run at 10Hz and 100Hz (periods 91000 and 9100 respectively)

(ps-4.5.26) drp-srcf-mon001:psana$ periodicgenerator -p 91000 9100 -s 0 0 -d '10Hz' '100Hz' >& 10hz_100hz.py

(now program sequence engine 0 and start it using the python code generated above. since we're programming engine 0, the first two event codes will be 272,273 according to the above table)

(this command must be executed on a node that can access the XPM PV's, e.g. drp-srcf-*)

(ps-4.5.26) drp-srcf-mon001:psana$ seqprogram --seq 0:10hz_100hz.py --pv DAQ:NEH:XPM:3 --start
** engine 0 fname 10hz_100hz.py **
descset  None
seqcodes {0: '10Hz', 1: '100Hz'}
Remove -1
idx 2
Removing seq 2
[10, 1, 5, 3, 0, 0, 0, 0, 2, 0, 6, 2048, 0, 0, 0, 3, 2, 1, 3, 3, 0, 0, 2, 0, 6, 908, 0, 0, 0, 1, 5, 2, 0, 0, 0, 0, 3, 2, 1, 0, 8, 0, 0, 2, 0, 6, 2048, 0, 0, 0, 3, 2, 6, 3, 3, 0, 0, 2, 0, 6, 908, 0, 0, 0, 1, 2, 0, 0, 0, 0, 0]
Confirmed ninstr 10
Sequence  found at index 2
desc ['10Hz', '100Hz', '', '', '', '', '', '', '', '', '', '', 'burst', '', '', '']
seqcodes_pv epics:nt/NTTable:1.0 
    string[] labels [EventCode, Description, Rate, Enabled]
    structure value
        int[] EventCode [272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287]
        string[] Description [10Hz, 100Hz, , , , , , , , , , , burst, , , ]
        int[] Rate [11,102,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
        byte[] Enabled [1,1,1,1,0,0,0,0,0,0,0,0,1,1,1,1]
    string descriptor 
    alarm_t alarm
        int severity 0
        int status 0
        string message 
    time_t timeStamp
        long secondsPastEpoch 1683163040
        int nanoseconds 339389440
        int userTag 0

(now query the xpm to see what is running in the 4 sequence engines, each having 4 event codes)

(ps-4.5.26) drp-srcf-mon001:psana$ pvget DAQ:NEH:XPM:3:SEQCODES
DAQ:NEH:XPM:3:SEQCODES 2023-05-03 18:17:31.339    
EventCode Description Rate Enabled
      272        10Hz   10       1
      273       100Hz  102       1
      274                0       1
      275                0       1
      276                0       0
      277                0       0
      278                0       0
      279                0       0
      280                0       0
      281                0       0
      282                0       0
      283                0       0
      284       burst    0       1
      285                0       1
      286                0       1
      287                0       1
(ps-4.5.26) drp-srcf-mon001:psana$ 

There is a similar "traingenerator" script which you can read about in Matt's documentation.

Sequence Start/Stop

Some technical details from a discussion on March 23, 2023

(1) current seq mode (only program time-in-seconds or numL1Accepts)
    have to coordinate the start of an accelerator burst with the daq sequence
    - are there two sequences?  one for accelerator and one for daq?
      not clear how much control the accelerator will want (e.g. burst
      all the time for stability, but silke says can damage samples)
    - each step we start one sequence:
      o may also include control of accelerator, but if not
      o have to coordinate the start of the daq seq with accelerator seq
        o acc seq synced to 1Hz marker (everything repeats at 1Hz on
      accelerator side)
    o trickier to sync to rates faster than 1Hz
(2) future-firmware seq mode
    - make the daq enable/disable on a user-defined event code
      o would be faster on the enable since don't have to wait for 1Hz
        marker.  still have to understand the periodicity of the accelerator
    so the user-defined event codes get sent down at the right time
    but it does mean we don't need the slow 1Hz marker.  some things like
    laser electronic delay scanning may be faster than 1Hz so could benefit
    from not using the 1Hz marker.

caveat: currently start sequences on 1Hz boundary

andor integrates over 10 shots
6 andor images in a step

mode (1):
- program 6 andor-readout-group "L1Accepts".  might be set to the parent
  group now in control.py

the readout group that is counted for the sequence is currently managed
by these lines in control.py.  Currently defaults to the "platform" readout
group which is the primary readout group: need to make this more flexible
so we can change it to the integrating detector readout group:

self.pv_xpm_part_base = pv_base + ':XPM:%d:PART:%d' % (xpm_master, platform)
self.pva.pv_put(self.pva.pvStepEnd, self.readoutCumulative)