The ePixHR readout is connected to the RIX DAQ for its October 4, 2021, beam test. It will run one ASIC at 120 Hz, 4 kHz, and 5 kHz readout rates. A new detector definition "epixhr2x2" has been added to psana, and the daq instance of this detector is "epixhr_0".
DAQ Setup
The trigger rates have been implemented with a sequencer running in DAQ:NEH:XPM:3. The script for this is found in ~rixopr/scripts/epixhr/seq_epixhr.py. The default arguments are sufficient. This script need only be executed once after any reset of the DAQ:NEH:XPM:3 board. The result of the sequencer is to create 3 event codes (272, 273, 274). Useful eventcodes are:
| event code | rate | description |
|---|---|---|
| 9 | 360 Hz | LCLS fiducials |
| 10 | 120 Hz | LCLS fiducials used for beam |
| 272 | 120 Hz | duplicate of event code 10 |
| 273 | 3960 Hz | for each event code 272, 32 events with this code are generated with spacing equal to 233*14/13 (~250.9) microseconds |
| 274 | 4920 Hz | for each event code 272, 40 events with this code are generated with spacing equal to 186*14/13 (~200.3) microseconds |
During the > 120 Hz running, the epixhr_0 detector should be run in a separate readout group along with timing_0, so that the other detectors are not triggered at the higher rate. This should be the lowest numbered readout group (1). This is entered in the "Select" screen from the DAQ GUI shown below. Note that the event code used for the readout groups is selected in the "timing_0" detector in the configuration GUI under "user.Cu.groupX_eventcode.
Detector Configuration
The complete detector configuration is found in the configuration database accessible via python script and the DAQ GUI. The DAQ GUI view is shown below.
The "user" section choices override any associated settings in the "expert" section. This section is meant as a shortcut to common configuration changes. The variables shown in the GUI above are described in the following table. All variables under the expert.EpixHR section should be familiar to detector experts.
| Variable | Units | Description |
|---|---|---|
| user.pixel_map | none | A full 288*392 ndarray (np.uint8) of settings. This should be accessed via script for convenience. |
| user.start_ns | ns | The delay on the external trigger sent to the detector over PGP. |
| user.gain_mode | H,M,L,AHL,AML,user | A global setting for the pixel map, unless "user" is selected. |
| user.asic_enable | bitmask | A bit mask of ASICs to configure and expect for readout. (1=ASIC0, 2=ASIC1, 4=ASIC2, 8=ASIC3, sum for multiple ASICs) |
| expert.DevPcie | - | These settings are automatically managed. |
| expert.EpixHR | - | The full set of variables familiar to detector experts. These can be loaded via yml files for convenience (epixhr_config_from_yaml_set.py) |
The expert.EpixHR variables may be loaded into the database from .yml files. The script for doing this is executed from the rix-daq machine, rixopr account, ~rixopr/scripts directory with
python epixhr/epixhr_config_from_yaml_set.py
Edit the script to change the particular set of yml files to upload. An example pixel_map loading script is not yet provided.
DAQ & Detector Health Monitoring
The following is a link to the grafana page that reports the readout rates and damage for the RIX DAQ.
The temperatures, voltages, and currents from the detector can be monitored (via an EDM screen) from the rix-daq using the following command:
epixHR Monitoring EDM screen
/cds/group/pcds/epics-dev/ddamiani/epixHRtests/edm-det-epixHR.cmd
Dan writes: the IOC will restart automatically on reboot of the machine (cmp014 at the moment) or if the process dies. The ioc can be disabled/restarted using the ioc manager
https://confluence.slac.stanford.edu/display/PCDS/IOC+Manager+for+Users. you can caput 1 to DET:EPIXHR:01:SET_MONITOR to rerun that code that is run at boot time manually.
Online Analysis/Monitoring
The detector shows up in the AMI GUI as "epixhr". You can view its raw data as shown below. The "epixhr.raw.image" will also be viewable shortly as the calibration support progresses, in which case, the Take box will not be necessary.
Scanning
A few scripted scans have been prepared in the ~rixopr/scripts/epixhr directory. They are described in the following table. They may be launched from the command-line on rix-daq (rixopr account), ~rix/scripts directory.
| Scan Type | Command | Description |
|---|---|---|
| Pedestal | python epixhr/epixhr_pedestal_scan.py | Scan through the five global gain settings and record a fixed number of events (default 2000) |
| Delay | python epixhr/epixhr_timing_scan.py | Scan linearly across the external trigger delay range and record a fixed number of events (default 2000). Minimum delay is ~2000 ns. Beam arrives at ~93000 ns. |
| Charge Injection | python epixhr/epixhr_chargeinj_scan.py | Scan across a square grid of pixels for charge injection and record a fixed number of events as the charge ramps (default 2000). |
| Configuration | python epixhr/epixhr_config_scan.py | This type of scan is not currently working. Scan a generic expert.EpixHR parameter
|
Expert Notes
The release used is ~rixopr/git/lcls2_100421. Recent updates include to pyxpm and ts_config_store to support event codes > #255 (from the xpm sequencer).
It was necessary to lower the L0Delay for the readout group that epixhr_0 uses, so that the detector can scan delays well before the beam.
(ps-4.5.5) rix-daq:scripts> pvput DAQ:NEH:XPM:3:PART:1:L0Delay 0
Old : 2021-10-03 09:05:24.704 90
New : 2021-10-03 10:58:04.361 0
(ps-4.5.5) rix-daq:scripts> pvput DAQ:NEH:XPM:3:PART:2:L0Delay 0
Old : 2021-10-03 09:05:24.733 80
New : 2021-10-03 12:10:26.841 0
(ps-4.5.5) rix-daq:scripts> pvput DAQ:NEH:XPM:3:PART:5:L0Delay 0
Old : 2021-10-03 09:05:24.821 90
New : 2021-10-03 15:08:16.554 0
DAQ Issues
Lowest numbered readout group appears to be linked to platform number in the meb/teb. Use of lower readout groups fails to deliver events to the meb.- epixHR internal triggering for monitoring appears to make configuration/recovery unreliable.
- rogue.GeneralError: Block::checkTransaction: General Error: Transaction error for block ePixHr10kT.EpixHR.Hr10kTAsic0.PrepareMultiConfig with address 0x88020000. Error Non zero status message returned on fpga register bus in hardware: 0x2; when this happens repeatedly, I find that disabling the Analog and Digital power for several minutes allows it to become reliable again.
- ami ScalarPlot shows the structure of the parallel event handling node events clumped together (node 0 events followed by node 1 events followed by ...). Comment from cpo: I think one is supposed to use a TimePlot for this.
ami crashes on infinite recursion. This is new from recent psana commits.- saving ami flowchart for a scan and then loading causes the following crash on ami-node: pyarrow.lib.SerializationCallbackError: pyarrow does not know how to serialize objects of type <class 'int'>. Recorded run 5 to debug offline.
- drp-neh-cmp015 processes received many broken/corrupt Configure phase2 transitions; seems to only appear if bld or fims are in the partition. Not happening today; I don't know why.
- Lcls2EpixHrXilinxKcu1500Pgp4_6Gbps has strange timing feedback link behavior - no link errors, but wrong TxId appears in xpmpva and message rate is low (~40kHz instead of ~20MHz). Deadtime is disabled.
- High rate running (>4 kHz) causes epixhr segment level to crash with jump in event counter after about 20-30 seconds. Independent of presence of other (120Hz) detectors in the partition.
sometimes the asic readout fails, and the axistreambatchereventbuilder times out its contribution. The segment level should flag this as damage, but the damage does not propagate to grafana. So, watching ami is the only way to see this. This occurs reproducibly during a configuration scan. It could be the config_update write pixel map process.- Found that the presence of the environment IOC makes configuration fail much more frequently. Temperature readout seems wrong anyways. I stopped the IOC (telnet drp-neh-cmp014 30100; ^T ^X). Configuration can be more reliable by disabling the monitoring stream, but there are still asic data drops.
- Found that the pedestal calibration fails sometimes at 360 Hz. Reduce to 120 Hz.
Running Rogue
I run rogue via the following:
ssh drp-neh-cmp014
bash
cd ~weaver/epixhr/pcie
source setup_l2si.sh
python scripts/devGui.py &
cd ../epix-hr-single-10k
python software/scripts/ePixHr10kTDaqLCLSII.py --dev /dev/datadev_0 &
Your mileage may vary.
Detector calibration
Set LCLS2 psana production environment
ssh -Y pslogin
ssh -Y psana
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
Dark data processing
epix10ka_pedestals_calibration -k exp=rixc00221,run=42 -d epixhr
-processes dark run and saves per-panel per-gain range constants in the intermediate repository.
-x is optional parameter which by default points to /reg/d/psdm/asc/ascdaq18/xtc/. It needs to be specified for non-default xtc2 file location.
Constants merging and deployment
NOTE: For deployment of constants it is important to have a Kerberos ticket (this is how the calibration database authenticates the write-permission, see Configuration/Calibration Database#Permissions).
epix10ka_deploy_constants -k exp=rixc00221,run=42 -d epixhr -D
-grabs available constants from intermediate repository, merges them for gain ranges and panels and deploys constants in the calibration data base.
Other commands to check data and calibration constants
datinfo -e ascdaq18 -r 46 -d epixhr
-prints information about data in xtc2 file for specified experiment run and detector.
calibman
-launches GUI to browse/manage calibration constants.
References
Overview
Content Tools