General Information

This page is to help expand off of the TID instructions: Notes re ioc/common/pgpCamlink deployment to PROD

Since versions, file names, and cpus are subject to change over time we will be substituting the versions and files with <insert fiber channel>, <insert server pgp slot>, <version>, name of file-<version>, <cpu>

Some things are subject to change from system to system, such as if there is multiple Gen4 pgp cards installed and how many FEBs are attached to the card.

  • If only one Gen4 pgp card is present the python program run as seen below. If more than one is present you will need to add "--dev /dev/datadev_<PV PGP_BOARD from st.cmd>" otherwise only /dev/datadev_0 slot will run.
  • If multiple FEB's are attached to the pgp card you will need to specify which device you want to communicate with. The numbering scheme will match what is used in the iocboot script.

The Python scripts will not end automatically. You will need to press CTR-C to end the programs.

Any IOCs running on the sever must be killed before attempting to program FEB.

Dev3 Help

This example will be using information for the LCLS server since at the time of creation LCLS is the only one utilizing this system. Use the appropriate usernames for cpu logins.

It is recommended that these scripts are one after the other in order to ensure the same version is running on both the FEB and FPGA.

Programing FEB on Dev
ssh laci@<cpu>
cd $PACKAGE_SITE_TOP/cameralink_gateway/cameralink-gateway-git/software
source ./setup_env_slac.sh
python scripts/updateFebFpga.py --lane <insert fiber channel> --pgp4 1 --mcs ../../images/<version>/ClinkFeb-<version>

 If there are more than one card installed you will need specify the card you want to update by adding "–dev /dev/datadev_<card number>" by default it is "–dev /dev/datadev_0"

Programing PCIe FPGA on Dev
python scripts/updatePcieFpga.py --path ../../images/<version>

During this python program it will ask you which firmware should be loaded on the card. Current recommednation is the "ClinkSlacPgpCardG4Pgp4" option which should be option "0".

PROD Help

This example will be using information for the LCLS server since at the time of creation LCLS is the only one utilizing this system. Use the appropriate usernames for cpu logins.

It is recommended that these scripts are one after the other in order to ensure the same version is running on both the FEB and FPGA.

Programing FEB on PROD
ssh laci@<cpu>
cd $PACKAGE_SITE_TOP/cameralink_gateway/software
source ./setup_env_slac.sh
python scripts/updateFebFpga.py --lane <insert fiber channel> --pgp4 1 --mcs ../current/ClinkFeb-<version>

 If there are more than one card installed you will need specify the card you want to update by adding "–dev /dev/datadev_<card number>" by default it is "–dev /dev/datadev_0"

Programing PCIe FPGA on PROD
python scripts/updatePcieFpga.py --path ../current/

During this python program it will ask you which firmware should be loaded on the card. Current recommendation is the "ClinkSlacPgpCardG4Pgp4" option which should be option "0".

Examples of What to Expect to See

Below are examples of what you can expect to see if the programs were completed successfully with on dev3. The same results would be visible in PROD.

Example: Programing FEB
[curtisco@lcls-dev3 ~]$ ssh laci@cpu-b024-pm01
sh: /usr/bin/xauth: not found
Entering commonSetup.sh
Pacific timezone is PST8PDT
EPICS_IOCS=/afs/slac/g/lcls/epics/iocCommon
EPICS_CPUS=/afs/slac/g/lcls/epics/cpuCommon
CPU_ARCH=linuxRT-x86_64
Adding to PATH EPICS_CPU_BIN=/afs/slac/g/lcls/epics/iocTop/ProfileMonitorAD/ProfileMonitorAD-git/bin/linuxRT-x86_64
caRepeater=/afs/slac/g/lcls/epics/iocTop/ProfileMonitorAD/ProfileMonitorAD-git/bin/linuxRT-x86_64/caRepeater
EPICS_CPU_LIB=/afs/slac/g/lcls/epics/iocTop/ProfileMonitorAD/ProfileMonitorAD-git/lib/linuxRT-x86_64
Exiting commonSetup.sh
[ laci@cpu-b024-pm01]$ cd $PACKAGE_SITE_TOP/cameralink_gateway/cameralink-gateway-git/software
[ laci@cpu-b024-pm01]$ source ./setup_env_slac.sh
(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updateFebFpga.py --lane 1 --pgp4 1 --mcs ../../images/v7.12.0/ClinkFeb-0x07120000-20220322145837-ruckman-0a8891d.mcs
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts
Start: Started zmqServer on ports 9099-9101
###################################################
#                 Old Firmware                    #
###################################################
Path         = MyRoot.ClinkFeb[1].AxiVersion
FwVersion    = 0x7150000
UpTime       = 1:57:02
GitHash      = 0x84ba82e136a3bada0e03ce22702e328192f7d819
XilinxDnaId  = 0x38348069204854
FwTarget     = ClinkFeb
BuildEnv     = Vivado v2021.2
BuildServer  = rdsrv318 (Ubuntu 20.04.5 LTS)
BuildDate    = Thu 01 Sep 2022 09:38:40 AM PDT
Builder      = ruckman
LoadMcsFile: ../../images/v7.12.0/ClinkFeb-0x07120000-20220322145837-ruckman-0a8891d.mcs
CypressS25Fl Manufacturer ID Code  = 0x1
CypressS25Fl Manufacturer Type     = 0x20
CypressS25Fl Manufacturer Capacity = 0x18
CypressS25Fl Status Register       = 0x2
Reading .MCS:    [####################################]  100%
Erasing PROM:    [####################################]  100%
Writing PROM:    [####################################]  100%
Verifying PROM:  [####################################]  100%
LoadMcsFile() took 0:00:42 to program the PROM


            ***************************************************
            ***************************************************
            The MCS data has been written into the PROM.
            To reprogram the FPGA with the new PROM data,
            a IPROG CMD or power cycle is be required.
            ***************************************************
            ***************************************************



Reloading FPGA firmware from PROM ....

Reloading FPGA done
###################################################
#                 New Firmware                    #
###################################################
Path         = MyRoot.ClinkFeb[1].AxiVersion
FwVersion    = 0x7120000
UpTime       = 0:00:03
GitHash      = 0xa8891d07cf12bfebe9b90f79ea4c03abed1daa5
XilinxDnaId  = 0x38348069204854
FwTarget     = ClinkFeb
BuildEnv     = Vivado v2021.2
BuildServer  = rdsrv303 (Ubuntu 20.04.4 LTS)
BuildDate    = Tue 22 Mar 2022 02:58:37 PM PDT
Builder      = ruckman
^CException ignored in: <module 'threading' from '/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt
Example: Programing PCIe FPGA
(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updatePcieFpga.py --path ../../images/v7.12.0/
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/axi-pcie-core/scripts
Start: Started zmqServer on ports 9099-9101
WARNING: PcieTop.AxiPcieCore.numDmaLanes = 1 != PcieTop.AxiPcieCore.AxiVersion.DMA_SIZE_G = 4
#########################################
Current Firmware Loaded on the PCIe card:
#########################################
Path         = PcieTop.AxiPcieCore.AxiVersion
FwVersion    = 0x7150000
UpTime       = 2:14:30
GitHash      = 0xb47b5f8dfc343609631b335f82544d4224cdb840
XilinxDnaId  = 0x40020001814bdfa615306245
FwTarget     = ClinkSlacPgpCardG4Pgp4
BuildEnv     = Vivado v2021.2
BuildServer  = rdsrv318 (Ubuntu 20.04.5 LTS)
BuildDate    = Thu 01 Sep 2022 08:16:41 AM PDT
Builder      = ruckman
#########################################
0 : ../../images/v7.12.0/ClinkSlacPgpCardG4Pgp4-0x07120000-20220322144331-ruckman-0a8891d
1 : ../../images/v7.12.0/ClinkFeb-0x07120000-20220322145837-ruckman-0a8891d
2 : ../../images/v7.12.0/ClinkKcu1500Pgp2b-0x07120000-20220322144240-ruckman-0a8891d
3 : ../../images/v7.12.0/ClinkKcu1500Pgp4-0x07120000-20220322144253-ruckman-0a8891d
4 : ../../images/v7.12.0/ClinkSlacPgpCardG4Pgp2b-0x07120000-20220322144315-ruckman-0a8891d
5 : Exit
Enter image to program into the PCIe card's PROM: 0
PcieTop.AxiPcieCore.AxiMicronN25Q[0].LoadMcsFile: ../../images/v7.12.0/ClinkSlacPgpCardG4Pgp4-0x07120000-20220322144331-ruckman-0a8891d_primary.mcs
PROM Manufacturer ID Code  = 0x20
PROM Manufacturer Type     = 0xbb
PROM Manufacturer Capacity = 0x19
PROM Status Register       = 0x2
PROM Volatile Config Reg   = 0xfb
Reading .MCS:    [####################################]  100%
Erasing PROM:    [####################################]  100%
Writing PROM:    [####################################]  100%
Verifying PROM:  [####################################]  100%
LoadMcsFile() took 0:01:04 to program the PROM


            ***************************************************
            ***************************************************
            The MCS data has been written into the PROM.
            To reprogram the FPGA with the new PROM data,
            a IPROG CMD or power cycle is be required.
            ***************************************************
            ***************************************************


PcieTop.AxiPcieCore.AxiMicronN25Q[1].LoadMcsFile: ../../images/v7.12.0/ClinkSlacPgpCardG4Pgp4-0x07120000-20220322144331-ruckman-0a8891d_secondary.mcs
PROM Manufacturer ID Code  = 0x20
PROM Manufacturer Type     = 0xbb
PROM Manufacturer Capacity = 0x19
PROM Status Register       = 0x2
PROM Volatile Config Reg   = 0xfb
Reading .MCS:    [####################################]  100%
Erasing PROM:    [####################################]  100%
Writing PROM:    [####################################]  100%
Verifying PROM:  [####################################]  100%
LoadMcsFile() took 0:01:03 to program the PROM


            ***************************************************
            ***************************************************
            The MCS data has been written into the PROM.
            To reprogram the FPGA with the new PROM data,
            a IPROG CMD or power cycle is be required.
            ***************************************************
            ***************************************************



Reloading FPGA firmware from PROM ....

Please reboot the computer
^CException ignored in: <module 'threading' from '/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt

Common Errors

There are two common errors that are fixable; trying to program the FEB with an IOC still running and the program can't find the FEB. Other errors should be brought up to TID.

Example: Programing FEB - IOC running
[ laci@cpu-b024-pm01]$ source ./setup_env_slac.sh
(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updateFebFpga.py --lane 1 --pgp4 1 --mcs ../../images/v7.12.0/ClinkFeb-0x07120000-20220322145837-ruckman-0a8891d.mcs
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts
Traceback (most recent call last):
  File "scripts/updateFebFpga.py", line 183, in <module>
    base = MyRoot(dev=args.dev,pgp4=args.pgp4)
  File "scripts/updateFebFpga.py", line 105, in __init__
    self.dmaStreams = axipcie.createAxiPcieDmaStreams(dev, {lane:{dest for dest in range(4)} for lane in range(4)}, 'localhost', 8000)
  File "/afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts/../../axi-pcie-core/python/axipcie/_AxiPcieDma.py", line 26, in createAxiPcieDmaStreams
    d[lane][dest] = rogue.hardware.axi.AxiStreamDma(driverPath, (0x100*lane)|dest, True)
rogue.GeneralError: AxiStreamDma::AxiStreamDma: General Error: Failed to open device file /dev/datadev_0 with dest 0x0! Another process may already have it open!
Example: Programing FEB - Nothing Connected to Fiber or Incorrect PGP Slot
(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updateFebFpga.py --lane 1 --pgp4 0 --mcs ../../images/v7.13.0/ClinkFeb-0x07130000-20220325175734-ruckman-c962153.mcs
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts
Start: Started zmqServer on ports 9099-9101
Traceback (most recent call last):
  File "scripts/updateFebFpga.py", line 202, in <module>
    raise ValueError(f'Pgp[lane={args.lane}] is down')
ValueError: Pgp[lane=1] is down
^CException ignored in: <module 'threading' from '/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt


(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updateFebFpga.py --lane 0 --pgp4 0 --mcs ../../images/v7.13.0/ClinkFeb-0x07130000-20220325175734-ruckman-c962153.mcs
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts
Start: Started zmqServer on ports 9099-9101
Traceback (most recent call last):
  File "scripts/updateFebFpga.py", line 202, in <module>
    raise ValueError(f'Pgp[lane={args.lane}] is down')
ValueError: Pgp[lane=0] is down
^CException ignored in: <module 'threading' from '/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt


(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updateFebFpga.py --lane 0 --pgp4 0 --mcs ../../images/v7.12.0/ClinkFeb-0x07120000-20220322145837-ruckman-0a8891d.mcs
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts
Start: Started zmqServer on ports 9099-9101
Traceback (most recent call last):
  File "scripts/updateFebFpga.py", line 202, in <module>
    raise ValueError(f'Pgp[lane={args.lane}] is down')
ValueError: Pgp[lane=0] is down
^CException ignored in: <module 'threading' from '/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt


(rogue_v5.10.0) [ laci@cpu-b024-pm01]$ python scripts/updateFebFpga.py --lane 0 --pgp4 1 --mcs ../../images/v7.12.0/ClinkFeb-0x07120000-20220322145837-ruckman-0a8891d.mcs
Rogue/pyrogue version v5.10.0. https://github.com/slaclab/rogue
Basedir = /afs/slac.stanford.edu/g/lcls/vol9/package/cameralink_gateway/cameralink-gateway-git/firmware/submodules/clink-gateway-fw-lib/scripts
Start: Started zmqServer on ports 9099-9101
Traceback (most recent call last):
  File "scripts/updateFebFpga.py", line 202, in <module>
    raise ValueError(f'Pgp[lane={args.lane}] is down')
ValueError: Pgp[lane=0] is down
^CException ignored in: <module 'threading' from '/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py'>
Traceback (most recent call last):
  File "/afs/slac.stanford.edu/g/reseng/vol31/anaconda/anaconda3/envs/rogue_v5.10.0/lib/python3.7/threading.py", line 1307, in _shutdown
    lock.acquire()
KeyboardInterrupt
  • No labels