Debugging

REMEMBER:
- for hsd: need to do "configure" IN HSD WINDOW, also "clearreadout" and "clear"
- need to reset the RxLink for XPM front-panel 
- to fix hsd configure problem have to tweak params in hsdcas.py and start like this:
/reg/neh/home5/weaver/lcls2/psdaq/psdaq/scripts/launch_pcaspy /reg/neh/home/cpo/git/lcls2/psdaq/psdaq/scripts/hsdcas.py -P DAQ:LAB2:HSD:DEV02 &

debugging no events:

- look for deadtime
- check that the "partition" window (with the trigger-enable checkbox) is talking to the right XPM (currently "2" ... selects which XPM to use (across crates))
- HSD is not configured to do anything (Check the HSD config tab for no channels enabled)
- XPM is not configured to forward triggers to HSD ("LinkEnable" for that link on the XPM GUI)
- if hsd timing frames are not being received at 926kHz (status here), click TxLink Reset in XPM window
- L0Delay set to 99
- The HSD configuration partition number does not match platform number in .cnf file (Check the HSD "base" tab)
- DST Select (in PART window) set to "DontCare" (could be Dontcare/Internal)
- also check that HEADERCNTL0 is incrementing in "status" tab of HSD cfg window.
- check processes in lab3-base.cnf are running
- run psdaq/build/psdaq/pgp/kcu1500/app/kcuStatus and kcuDmaStatus in kcuDmaStatus "blockspause" and "blocksfree" determine whether or not deadtime is set. if blocksfree drops below blockspause then it will assert deadtime. in hsd window "pgp last rx opcode" 0 means no backpressure, 1 means backpressure
- in hsd window timepausecnt is number of clocks we are dead
- in hsd expert window "full threshold(events)" sets threshold for hsd deadtime
- in hsd status window "fex free events" and "raw free events" are the current free events.
- in hsd status window "write fifo count" is number of timing headers waiting for HSD data to associate.

- "pgp tx frame count" on hsd status goes up when we send a transition
- check for multiple drp executables
- check kcuStatus for "locPause" non-zero (a low level pgp FIFO being full).  If this happens then:  configure hsd, clear readout, reboot drp node with KCU
- clearReadout broadcasts a message to receiving hsd's (or kcu's) telling them to reset timing-header FIFOs.
- xpm tx link reset: resets a tx PLL, rx link reset: retrains the link on K characters
- if running "drp" executable, check that lane mask is correct (e.g. 1 for "ganged" mode)