Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Troubleshooting common issues with DAQ

IssueSolution/Workaround
DAQ not saving scalar data (BSA and non BSA) and getting F2_EventClass errorThis is caused by a bad PV in one of the lists
DAQs returning no matches and UniqueId_RBV PV not updating + fiducial rate stuck and/or redReboot the server

DAQ (1D or 2D) fails and shows the following error:

Error using repelem
Replication factors must be nonnegative integer-valued scalars or vectors.

http://physics-elog.slac.stanford.edu/facetelog/show.jsp?dir=/2024/24/15.06&pos=2024-06-15T21:36:00

This happens when a camera trigger is not set up correctly

A workaround is to find a camera that is on the same trigger and add it to your DAQ

Ask Sharon for help if this isn't clear

Bugs and Robustness

Add bugs here, newest entries on top:

Live planning notes

Goal: Update DAQ to HDF5

  • What has already been done?
    • We have a working version of the HDF5 DAQ, and now we are working on making it robust
    • Sharon has shown that HDF5 file writing in "Streaming" mode is very fast.
    • Now we want to extend capability for scans so that we have 1 file per camera.
  • How to save scan image data into HDF5 file?
    • Use PVs:
      • extraDimSizeN = Number of shots per step
      • nExtraDims = Number of scan dimensions (0 = simple DAQ, 1 = 1D scan, 2 = 2D scan . . .)
      • extraDimSizeX = Number of steps for Dim 1
      • extraDimSizeY = Number of steps for Dim 2
      • extraDimSize3 = Number of steps for Dim 3
      • . . .
    • These PVs implicitly control the NumCapture PV
      • NumCapture_RBV = extraDimSizeN * extraDimSizeX * extraDimSizeY * . . . (depending on number of nExtraDims)
  • In this scheme, we have to carefully control the triggers so that each scan step gets exactly the right number
    • We will use a new feature to do this:
      • EVNT:SYS1:1:PMAQCTRL.H = extraDimSizeN = Number of shots per step
      • EVNT:SYS1:1:PMAQCTRL.E = 0/1 to stop/start triggers to cameras
      • Need to toggle back to 0 after every step
      • Set EVNT:SYS1:1:PMAQCTRL.H back to zero at end of scan
  • What happens if the camera does not receive all triggers?
    • Then scan steps are kind of messed up . . . think about how best to handle this with pulse ID
  • Prefer "NDArrayUniqueId" to be "NDArrayPulseId"
  • Make caput/lcaPut/lcaPutSmart consistent
  • "BSA buffer" for UV spectrometer IOC?
  • Find a reliable fix to shot counting issue
    • checkShots is not counting the correct number of shots at some steps. we think the DAQ is just not waiting long enough for the shot number to go up, because the shots do get saved eventually
    • for E332 shift on 5/30, we just added a while/pause loop to force DAQ to wait for shot numbers to go up
    • if we decide to keep this functionality we should at least add a break for the while loop but for now it is working again

DAQ tests E320 shift 4/30/24

Testing TIFF/HDF5 switch
- take a DAQ with TIFF and try to load it in DAN
- results look normal

Testing HDF5 DAQ
- take a DAQ with HDF5, simple DAQ, 20 shots, 1 camera (LBG_LFOV)
    - all 20 shots saved
- test with 1000 shots
    - saved 999 shots
    - took a little over 2 minutes
- add more cameras (LBG_LFOV, EOS2, GAMMA1)
    - saved 993 shots for each camera
    - ~2 min 10 seconds in total
- try streaming
    - 1000 shots
    - failed TEST 7480
- try streaming but with 10 shots
    - success with all matches
- try streaming with 200 shots
    - success with 197 matches
- try streaming with 700 shots
    - says DAQ failed on step 1 but 641 matches
- try streaming with TIFF, 700 shots
    - saved 155 matches
    - took a little more than 3 minutes
- try a 1D scan, 10 shots, 3 steps
    - did not restore to initial value
    - Spec_Quad_E
    - 28 matches
- trying a 2D scan, 10 x 2 x 2
    - saved 40 shots
- trying 2D scan with streaming
    - saved 40 shots
- try to open HDF5 images in DAN
    - error when "Plot Correlation" button pushed, expecting a TIFF image
    - Error in hlpGetImage with “imgType”

Bugs and Robustness

IssuePersonPriorityStatusNotes
IssuePersonPriorityStatusNotes

Load/clear config buttons don't work


Low

Warning: Unable to save App Designer app object. Save not supported for matlab.apps.AppBase objects. 
Warning: org.epics.ca.Context@42373389 is not serializable 
Warning: org.epics.ca.impl.ChannelImpl@3a027fd6 is not serializable 

SharonLow
Seems to be harmless
DAQ crashes and takes down the whole SIOC when total number of shots exceeds 2200 (no streaming, try with streaming on)


Reproduced this with PM03

Has to be more than 5 cameras

Background not being savedSharonHighDONE
Not saving non BSA data and getting F2_EventClass errorSharonHighThis is caused by a PV in a list being down

Implement HDF5 file saving for imagesSharonHigh
Testing
DONE
Modify DAQ to write image data to "RAM disk"
HighN/A
Calculate how much space on disk is required
High

Install 10G switch and NAS in S10ArjunHighPlanning
Check what happens if new DAQ is started before old one is finished (failed abort)
High

Fix binning and ROI for CMOS cameras
Medium

Can't get magnet values on first DAQ?
Medium

Check that the beam rate is not equal zero if rate set to "BEAM"SpencerMedium

Fix "Use_PV" functionality.SharonMediumDONE
Fix camera triggers if DAQ fails.SharonHighDONE
"Fix cameras" button disabled if DAQ hasnt run yetSharonMediumDONE
"Fix cameras" button should use default values? - yesSharonMediumDONE
Timestamping fails on certain ROI settings?SpencerHighDONE
Upgrade all machines in S20 to 10G connectionSpencerMediumDONE
Camera binning messes up pulse ID findingSpencerMediumDONE

Feature requests

IssuePersonPriorityStatus
Add check/remove bad PVs and notify user in an obvious waySharon

Add nonBSA arrays so we can get UV spectrometer (Claudio request)SharonHighDONE
DAQ Monitor GUISharonHighDONE
Find a way to globally trigger a modifier bit (e.g. "DUMP_2_9").David

Scan function that pauses until you click 'ok'


Generate "camera config" file including trigger info and add a save/restore button. Monitor if this finished ok or not.Sharon

Load/Save Config files. Default config files.Claire

Robustness issues. . .SpencerHighIn progress
DAN featuresSharon

Documentation/commentingSpencerHighIn progress
Fix "Use_PV" functionality.SharonMediumDONE

...