This page wants to collect all known procedures to restore functionalities of all parts of the DAQ (2024-06-03)

Stop any other running instance of DAQ using same platform number.

HSD

ProblemSolution
HSD is not connecting/configuringopen log and search for errors
if logs shows a too general failurecheck if IOCs are running
> procmgr status hsd.cnf
IOC not started> procmgr stopall hsd.cnf
> procmgr start hsd.cnf
IOC still not starting> ping daq-tmo-hsd-01
if no response:
reboot node via web ipmi or hard reboot on the node
still not configuringcheck ns_start and ns_gate values, modify them to:
 'user': {'fex': {'gate_ns': 200,
                  'prescale': 0,
                  'start_ns': 108000,
                'raw': {'gate_ns': 3000,
                  'prescale': 1,
                  'start_ns': 108000}}}
These are valid values, if HSD works it means that Operator needs to find correct values for its own project
Remember to set the parameters back to previous values (look in logs to find previous values )
check linkupconnect to drp node:
    find node:
    > grep hsd tmo.cnf
> ssh X drp-srcf-cmpXXX
use the command:
> kcuStatus
and check for linkup and frequency
if no linkup, check XPMs, or check fibers and transceivers.
when running an hsd as a single

Remember to disable the associated hsd in the hsdpva
TMO:
1A:B    3D:B    DA:B    B2:B    89:B
1A:A    3D:A    DA:A    B1:B    89:A
1B:B    3E:A    5E:A    B2:A    88:B
1B:A    3E:B    5E:B    B1:A    88:A

RIX:
1A:B
1A:A
1B:B
1B:A

> hsdpva DAQ:TMO(RIX):HSD:1_XX:A DAQ:TMO(RIX):HSD:1_XX:B

Piranha/Opal

ProblemSolution
Camera is not connecting/configuring

Select Open console in ProcStat and click on the camera
Press cntrl-X and observe the log

Another possible cause is the camera link box (black box with two LEDs) is not powered. Both LEDs should be green. The bottom one can be red upon turning on but once connected to the daq and received configuration, it will turn green.

still not connecting/ configuring check camera directly from the nodefind the node running the camera
    > grep #piranha# #TMO.cnf# (replace detector and cnf file with appropriate values)
    note node name and lane
    ex: "{ host: 'drp-srcf-cmp007', id:'tmo_atmpiranha_0', flags:'spu', env:epics_env, cmd:drp_cmd0+' -l 0x1 -D piranha4'},"

    > ssh X drp-srcf-cmpXXX 
check file system

> ls -al (...)

if command reports nothing, then file system is currupted and computer needs restarting:
> /cds/group/pcds/admin/ipmicreds/psipmi power reset

check kcu linkup

> cat /proc/datadev_1
Check link up
If not locked in then check XPMs, fibers and transceivers

(Opal and Piranha do not use kcuSim or kcuStatus, only HSD and timing (To Be Verified))

check devGui

Stop any other running instance of DAQ using same platform number

Open devGui directly on the node:
    
> cd cameralink-gateway

check cameralink version:
must be v8.2.3 (as for 2024-06-03)
If not, 
    > git pull --rebase
    > git checkout v8.2.3
    > git submodules update 

> cd cameralink-gateway/software
> python scripts/devGui --pgp4 0 --laneConfig 0=Piranha4 --pcieBoardType Kcu1500 --enLclsII 1 --enableConfig 0 --startupMode 1 --enVcMask 0

Replace detector name with: Piranha4 or Opal1000
(remove --enVcMask 0 if you want to talk to the FEB, this allows to run the DAQ at the same time)
Also remember that:

laneexponentlane mask (hex of exponent)
02^0=10x01
12^1=20x02
22^2=40x04
32^3=80x08
42^4=160x10
52^5=320x20
62^6=640x40


XPM

ProblemSolution
no linkup

for RIX start 
> xpmpva DAQ:NEH:XPM:0 DAQ:NEH:XPM:3 DAQ:NEH:XPM:5

for TMO start
> xpmpva DAQ:NEH:XPM:0 DAQ:NEH:XPM:2 DAQ:NEH:XPM:4 DAQ:NEH:X
PM:6

RxLinkReset

try by pressing button for reset

need to restart XPM

on psdev
> source /cds/sw/package/IPMC/env.sh (used to be "source /afs/slac/g/reseng/IPMC/env.sh")

> fru_deactivate (xpm crate name)
> fru_activate (xpm crate name)

xpm0

shm-neh-daq01:2

xpm2

shm-tmo-daq01:3

xpm3

shm-rix-daq01:3

xpm4

shm-tmo-daq01:5

xpm5

shm-neh-daq01:4


  • No labels