...
Term | Description |
---|---|
ATCA | Advanced Telecommunications Computing Architecture |
BSA | Beam Synchronous Acquisition |
CPU | Central Processing Unit |
EPICS | Experimental Physics and Industrial Control System |
EVR | Event Receiver |
FPGA | Field Programmable Gate Array |
IOC | Input Output Controller |
RF | Radio Frequency |
SHM | Shelf Manager |
TPR | Timing Pattern Receiver |
VCO | Voltage Controlled Oscillator |
...
Linac Locking - A System Level Overview
In this section, IOC deployment details are provided pertaining to the App hosting the phase locker IOC, as well as the IOC name and the assigned CPUs in both Dev and Production.
The IOC is housed by the masterSource App. The intended released tag in Production is R2.2.3.
...
IOC App Name
...
App Version
...
The configuration used in Production is seen below.
...
IOC Name
...
CPU Name
...
The configuration used in Dev is seen below. As of now, there is no designated test stand in Dev.
...
IOC Name
...
CPU Name (Intended)
...
List of required packages and associated versions.
...
Package Name
...
Package Version
...
cpsw/framework
...
R4.4.1
...
yaml-cpp
...
yaml-cpp-0.5.3_boost-1.64.0
...
boost
...
1.64.0
...
pcre
...
mrllrf/llrfLib
...
R1.1.0
...
timing/hpsTpr
...
R2.3.0
...
atca/commonATCA
...
R1.3.1
List of required modules and associated versions.
...
Module Name
...
Module Version
...
Module Name
...
Module Version
...
ATCACommon
...
R7.0.3.1-1.0
...
agilent53220A
...
R1.0.6
...
asyn
...
R4.39-1.0.1
...
autosave
...
R5.10-1.1.0
...
bkhAsyn
...
R0.4.6
...
caPutLog
...
R4.0-1.0.0
...
iocAdmin
...
R3.1.16-1.3.2
...
miscUtils
...
R2.2.5
...
modbus
...
R3.2-1.0.1
...
seq
...
R2.2.4-1.2
...
std
...
R3.2-1.0.5
...
streamdevice
...
R2.8.9-1.2.1
...
timingApi
...
R0.9
...
tprTrigger
...
R2.5.3-4.0-2
...
yamlLoader
...
R2.3.5
...
ycpswasyn
...
R3.3.6-1.0
The Timing Pattern Receiver (TPR) is an application that replaced the Event Receiver (EVR) for LCLS-II and its purpose is to generate triggers for control, data acquisition and other purposes. There are two main flavors in which this application is deployed: (i) as a PCIe board and (ii) as an ATCA board. The IOC described here is for a PCIe card. The TPR operates at a faster rate than the EVR (i.e. higher bandwidth), has a more compact frame and also has an LCLS-I timing fiber input, thus allowing it to switch to an LCLS-I operating mode if needed.
(PCIe TPR - Borrowed from PCIe TPR peer review slide deck, see Reference section below)
The PCIe TPR can provide up to 12 hardware triggers and/or channels. A channel is a source for a trigger and can be AND-ed or OR-ed with its adjacent channel. The user is also able to control --besides rate-- polarity, delay and width. Below, one can find a comprehensive view of the functionality of the TPR that spans firmware, middleware and software. For more details, please refer to the attached slide deck at the bottom of this page.
(PCIe TPR Firmware and Software - Borrowed from PCIe TPR peer review slide deck, see Reference section below)
The IOC is used in conjunction with the TPR displays to not only read diagnostics from the firmware, but also set up triggers, assign channels to triggers and timestamp Beam Synchronous Acquisition (BSA) data.
In this section, IOC deployment details are provided pertaining to the App hosting the phase locker IOC, as well as the IOC name and the assigned CPUs in both Dev and Production.
The IOC is housed by the masterSource App. The intended released tag in Production is R2.2.3.
IOC App Name | App Version |
---|---|
masterSource | R2.2.3 |
The configuration used in Production is seen below.
IOC Name | CPU Name |
---|---|
sioc-sys0-ms09 | cpu-sys0-sp01 |
The configuration used in Dev is seen below. As of now, there is no designated test stand in Dev.
IOC Name | CPU Name (Intended) |
---|---|
sioc-b084-ms09 | cpu-b084-sp12 |
List of required packages and associated versions.
Package Name | Package Version |
---|---|
cpsw/framework | R4.4.1 |
yaml-cpp | yaml-cpp-0.5.3_boost-1.64.0 |
boost | 1.64.0 |
pcre | 8.37 |
mrllrf/llrfLib | R1.1.0 |
timing/hpsTpr | R2.3.0 |
atca/commonATCA | R1.3.1 |
List of required modules and associated versions.
Module Name | Module Version |
---|---|
Module Name | Module Version |
ATCACommon | R7.0.3.1-1.0 |
agilent53220A | R1.0.6 |
asyn | R4.39-1.0.1 |
autosave | R5.10-1.1.0 |
bkhAsyn | R0.4.6 |
caPutLog | R4.0-1.0.0 |
iocAdmin | R3.1.16-1.3.2 |
miscUtils | R2.2.5 |
modbus | R3.2-1.0.1 |
seq | R2.2.4-1.2 |
std | R3.2-1.0.5 |
streamdevice | R2.8.9-1.2.1 |
timingApi | R0.9 |
tprTrigger | R2.5.3-4.0-2 |
yamlLoader | R2.3.5 |
ycpswasyn | R3.3.6-1.0 |
In this section, a detailed description of the IOC bootup process is laid out.
First, define the usual environment variables in the startup script.
Code Block | ||
---|---|---|
| ||
# ===========================================
# ENVIRONMENT VARIABLES
# ===========================================
epicsEnvSet("AREA", "SYS0" )
epicsEnvSet("UNIT", "MS09" )
epicsEnvSet("LOCATION", "" )
epicsEnvSet("IOC_PV", "SIOC:$(AREA):$(UNIT)" )
epicsEnvSet("TPR_PV", "TPR:$(AREA):$(UNIT)" )
epicsEnvSet("IOC", "sioc-sys0-ms09" )
epicsEnvSet("HASH", "58070ee" )
epicsEnvSet("YAML_DIR", "${TOP}/yaml/EvrCardG2-$(HASH).yaml")
epicsEnvSet("YAML_TOP_FILE", "${YAML_DIR}/000TopLevel.yaml" )
# Support Large Arrays/Waveforms; Number in Bytes
epicsEnvSet("EPICS_CA_MAX_ARRAY_BYTES", "2000000")
|
Load dbd and IOC health/monitoring records. Also, set up autosave and load the LCLS access configuration file. Finally, copy the archive file in $IOC_DATA to enable record archiving.
Code Block | ||
---|---|---|
| ||
## Load dbd
dbLoadDatabase("dbd/masterSource.dbd", 0, 0)
masterSource_registerRecordDeviceDriver(pdbbase)
## Environment variables
epicsEnvSet("ENGINEER", "")
epicsEnvSet("STREAM_PROTOCOL_PATH", "${TOP}/db")
## Tag log messages with IOC name
epicsEnvSet("EPICS_IOC_LOG_CLIENT_INET", "${IOC}")
## Load records
# **** Load iocAdmin databases to support IOC Health and monitoring ****
dbLoadRecords("db/iocAdminSoft.db", "IOC=${IOC_NAME}")
dbLoadRecords("db/iocAdminScanMon.db", "IOC=${IOC_NAME}")
dbLoadRecords("db/iocRelease.db", "IOC=${IOC_NAME}")
## Autosave initialization
< $(TOP)/iocBoot/common/autosave_init.cmd
## Channel Access Security:
# This is required if you use caPutLog.
# Set access security file
# Load common LCLS Access Configuration File
< ${ACF_INIT}
## Update archive pv list in ${IOC_DATA}
system("cp $(TOP)/archive/${IOC}.archive ${IOC_DATA}/${IOC}/archive/")
|
Tell the IOC where to find the list of record names and where they will be autosaved. Also, restore previously autosaved values.
Code Block | ||
---|---|---|
| ||
# =====================================================================
# Load database for autosave status
# =====================================================================
dbLoadRecords("db/save_restoreStatus.db", "P=${IOC_NAME}:")
# ============================================================
# If all PVs don't connect continue anyway
# ============================================================
save_restoreSet_IncompleteSetsOk(1)
# ============================================================
# created save/restore backup files with date string
# useful for recovery.
# ============================================================
save_restoreSet_DatedBackupFiles(1)
# ============================================================
# Where to find the list of PVs to save
# ======================= |
In this section, a detailed description of the IOC bootup process is laid out.
First, define the usual environment variables in the startup script.
Code Block | ||
---|---|---|
| ||
# =========================================== # ENVIRONMENT VARIABLES # = set_requestfile_path("${IOC_DATA}/${IOC}/autosave-req") # ============================================================ epicsEnvSet("AREA", "SYS0" ) epicsEnvSet("UNIT", "MS09" ) epicsEnvSet("LOCATION", "" ) epicsEnvSet("IOC_PV", "SIOC:$(AREA):$(UNIT)" ) epicsEnvSet("TPR_PV", "TPR:$(AREA):$(UNIT)" ) epicsEnvSet("IOC", "sioc-sys0-ms09" ) epicsEnvSet("HASH", "58070ee" ) epicsEnvSet("YAML_DIR", "${TOP}/yaml/EvrCardG2-$(HASH).yaml") epicsEnvSet("YAML_TOP_FILE", "${YAML_DIR}/000TopLevel.yaml" ) # Support Large Arrays/Waveforms; Number in Bytes epicsEnvSet("EPICS_CA_MAX_ARRAY_BYTES", "2000000") |
Load dbd and IOC health/monitoring records. Also, set up autosave and load the LCLS access configuration file. Finally, copy the archive file in $IOC_DATA to enable record archiving.
Code Block | ||
---|---|---|
| ||
## Load dbd
dbLoadDatabase("dbd/masterSource.dbd", 0, 0)
masterSource_registerRecordDeviceDriver(pdbbase)
## Environment variables
epicsEnvSet("ENGINEER", "")
epicsEnvSet("STREAM_PROTOCOL_PATH", "${TOP}/db")
## Tag log messages with IOC name
epicsEnvSet("EPICS_IOC_LOG_CLIENT_INET", "${IOC}")
## Load records
# **** Load iocAdmin databases to support IOC Health and monitoring ****
dbLoadRecords("db/iocAdminSoft.db", "IOC=${IOC_NAME}")
dbLoadRecords("db/iocAdminScanMon.db", "IOC=${IOC_NAME}")
dbLoadRecords("db/iocRelease.db", "IOC=${IOC_NAME}")
## Autosave initialization
< $(TOP)/iocBoot/common/autosave_init.cmd
## Channel Access Security:
# This is required if you use caPutLog.
# Set access security file
# Load common LCLS Access Configuration File
< ${ACF_INIT}
## Update archive pv list in ${IOC_DATA}
system("cp $(TOP)/archive/${IOC}.archive ${IOC_DATA}/${IOC}/archive/")
|
Tell the IOC where to find the list of record names and where they will be autosaved. Also, restore previously autosaved values.
# Where to write the save files that will be used to restore
# ============================================================
set_savefile_path("${IOC_DATA}/${IOC}/autosave")
# ============================================================
# Prefix that is use to update save/restore status database
# records
# ============================================================
save_restoreSet_status_prefix("${IOC_NAME}:")
## Restore datasets
set_pass0_restoreFile("info_positions.sav")
set_pass1_restoreFile("info_positions.sav")
set_pass0_restoreFile("info_settings.sav")
set_pass1_restoreFile("info_settings.sav")
|
Besides the auto-generated record names for autosave, copy the manually generated list of records to be autosaved and restore their previously autosaved values.
Code Block | ||
---|---|---|
| ||
# Copy over the manual settings request file
system("cp $(TOP)/autosave/${IOC}.req ${IOC_DATA}/${IOC}/autosave-req/")
# Load manual settings from autosave
set_pass0_restoreFile("${IOC}.sav")
set_pass1_restoreFile("${IOC}.sav")
|
Load the YAML hierarchy and register configuration.
Code Block | ||
---|---|---|
| ||
# Use slot A pcie tpr for root_0, override to use slot_a
cpswLoadYamlFile("${YAML_TOP_FILE}", "MemDev", "", "/dev/tpra", "root_0")
|
Load the phase locker records.
Code Block | ||
---|---|---|
| ||
# ===========================================
# DB LOADING
# ===========================================
# TPR driver DB
dbLoadRecords("db/pcie_tprTrig.db", "DEV=${IOC_PV}:0,LOCA=$(AREA),IOC_UNIT=$(UNIT),INST=0,PORT=trig0")
# Load database for autosave status
dbLoadRecords("db/save_restoreStatus.db", "P=${IOC_PV}:")
|
Create the ycpswAsynDriver with a dictionary option, i.e. use a limited number of interfaces to the FPGA registers.
Code Block | ||
---|---|---|
| ||
# ===========================================
# DRIVER SETUP
# ===========================================
# Configure crossbarControlAsynDriver
crossbarControlAsynDriverConfigure("crossbar0", "PCIe:/mmio/SfpXbar", "root_0")
# Setup tprTrigger Driver
tprTriggerAsynDriverConfigure("trig0", "PCIe:/mmio", "root_0")
|
Code Block | ||
---|---|---|
| ||
A complete list of the records to archive is provided below.
(base) skoufis@aird-b50-srv01 (master) $ cat archive/sioc-sys0-ms09.archive
# archive_iocAdmin Archive template
# Format:
# <PVNAME> <PERIOD> [ monitor ]
# Archive these values only when they change
SIOC:SYS0:MS09:APP_DIR 30 monitor
SIOC:SYS0:MS09:APP_DIR1 30 monitor
SIOC:SYS0:MS09:APP_DIR2 30 monitor
SIOC:SYS0:MS09:CA_CLNT_CNT 1 monitor
SIOC:SYS0:MS09:CA_CONN_CNT 1 monitor
SIOC:SYS0:MS09:EPICS_VERS 30 monitor
SIOC:SYS0:MS09:FD_CNT 1 monitor
SIOC:SYS0:MS09:FD_FREE 1 monitor
SIOC:SYS0:MS09:FD_MAX 1 monitor
SIOC:SYS0:MS09:HOSTNAME 30 monitor
SIOC:SYS0:MS09:KERNEL_VERS 30 monitor
SIOC:SYS0:MS09:RECORD_CNT 10 monitor
SIOC:SYS0:MS09:START_CNT 30 monitor
SIOC:SYS0:MS09:STARTTOD 30 monitor
SIOC:SYS0:MS09:ST_SCRIPT 30 monitor
SIOC:SYS0:MS09:ST_SCRIPT1 30 monitor
SIOC:SYS0:MS09:ST_SCRIPT2 30 monitor
SIOC:SYS0:MS09:SUSP_TASK_CNT 30 monitor
SIOC:SYS0:MS09:SYSRESET 30 monitor
# Archive these values at the specified rate
SIOC:SYS0:MS09:SYS_CPU_LOAD 10
SIOC:SYS0:MS09:LOAD 10
SIOC:SYS0:MS09:MEM_BLK_FREE 10
SIOC:SYS0:MS09:MEM_FREE 10
SIOC:SYS0:MS09:MEM_USED 10
SIOC:SYS0:MS09:UPTIME 3600
|
Initialize the IOC and logging capability.
Code Block | ||
---|---|---|
| ||
# | ||
Code Block | ||
| ||
# ===================================================================== # Load database for autosave status # ===================================================================== dbLoadRecords("db/save_restoreStatus.db", "P=${IOC_NAME}:") # ============================================================ # If all PVs don't connect continue anyway # ============================================================ save_restoreSet_IncompleteSetsOk(1) # ============================================================ # created save/restore backup files with date string # useful for recovery. # ============================================================ save_restoreSet_DatedBackupFiles(1) # ============================================================ # Where to find the list of PVs to save # ============================================================ set_requestfile_path("${IOC_DATA}/${IOC}/autosave-req") # ============================================================ # Where to write the save files that will be used to restoreIOC INIT # =========================================== iocInit() # =================== set_savefile_path("${IOC_DATA}/${IOC}/autosave") # ======================== # CAPUTLOG # =========================================== # Prefix that is use Turn on caPutLogging: # Log values only on change to update save/restore status database # records # ==the iocLogServer: caPutLogInit("${EPICS_CA_PUT_LOG_ADDR}") caPutLogShow(2) |
Start the autosave process.
Code Block | ||
---|---|---|
| ||
# ========================================================== save_restoreSet_status_prefix("${IOC_NAME}:") ## Restore datasets set_pass0_restoreFile("info_positions.sav") set_pass1_restoreFile("info_positions.sav") set_pass0_restoreFile("info_settings.sav") set_pass1_restoreFile("info_settings.sav") |
Besides the auto-generated record names for autosave, copy the manually generated list of records to be autosaved and restore their previously autosaved values.
Code Block | ||
---|---|---|
| ||
# Copy over the manual settings request file system("cp $(TOP)/autosave/====== # AUTOSAVE START # =========================================== # Autosave start < $(TOP)/iocBoot/common/autosave_start.cmd create_monitor_set("${IOC}.req ${IOC_DATA}/${IOC}/autosave-req/") # Load manual settings from autosave set_pass0_restoreFile("${IOC}.sav") set_pass1_restoreFile("${IOC}.sav") |
", 5, "") |
For a complete list of the Resynch Trigger IOC Process Variables (PVs), please check out the corresponding database file (i.e. pcie_tprTrig.db) in directory db/ of the masterSource projectLoad the YAML hierarchy and register configuration.
Code Block | ||
---|---|---|
| ||
# Use slot A pcie tpr for root_0, override to use slot_a
cpswLoadYamlFile("${YAML_TOP_FILE}", "MemDev", "", "/dev/tpra", "root_0")
|
Load the phase locker records.
Code Block | ||
---|---|---|
| ||
# ===========================================
# DB LOADING
# ===========================================
# TPR driver DB
dbLoadRecords("db/pcie_tprTrig.db", "DEV=${IOC_PV}:0,LOCA=$(AREA),IOC_UNIT=$(UNIT),INST=0,PORT=trig0")
# Load database for autosave status
dbLoadRecords("db/save_restoreStatus.db", "P=${IOC_PV}:")
|
Create the ycpswAsynDriver with a dictionary option, i.e. use a limited number of interfaces to the FPGA registers.
Code Block | ||
---|---|---|
| ||
# ===========================================
# DRIVER SETUP
# ===========================================
# Configure crossbarControlAsynDriver
crossbarControlAsynDriverConfigure("crossbar0", "PCIe:/mmio/SfpXbar", "root_0")
# Setup tprTrigger Driver
tprTriggerAsynDriverConfigure("trig0", "PCIe:/mmio", "root_0")
|
A complete list of the records to archive is provided below.
Initialize the IOC and logging capability.
Code Block | ||
---|---|---|
| ||
# ===========================================
# IOC INIT
# ===========================================
iocInit()
# ===========================================
# CAPUTLOG
# ===========================================
# Turn on caPutLogging:
# Log values only on change to the iocLogServer:
caPutLogInit("${EPICS_CA_PUT_LOG_ADDR}")
caPutLogShow(2)
|
Start the autosave process.
Code Block | ||
---|---|---|
| ||
# ===========================================
# AUTOSAVE START
# ===========================================
# Autosave start
< $(TOP)/iocBoot/common/autosave_start.cmd
create_monitor_set("${IOC}.req", 5, "") |
To access and launch the display in Production, follow the steps outlined below.
The released software will be made available in the main IOC release area post-deployment (Dev: /afs/slac/g/lcls/epics/iocTop/masterSource/, Production:/usr/local/lcls/epics/iocTop/masterSource/). See below for the specific repositories and tags.
See software release details below.
...
Repository
...
Version
...
IOC Name
...
(base) skoufis@aird-b50-srv01 (master) $ cat db/pcie_tprTrig.db
record(longin, "TPR:$(LOCA):$(IOC_UNIT):$(INST):FPGAVER")
{
field(DESC, "TimingCore FPGA version")
field(DTYP, "asynInt32")
field(INP, "@asyn($(PORT),0) fpgaVersion")
field(SCAN, "I/O Intr")
}
record(longin, "TPR:$(LOCA):$(IOC_UNIT):$(INST):UPTIME")
{
field(DESC, "TimingCore Uptime")
field(DTYP, "asynInt32")
field(INP, "@asyn($(PORT),0) uptime")
field(SCAN, "I/O Intr")
}
...
record(bo, "TPR:$(LOCA):$(IOC_UNIT):$(INST):EVSOFT07_TCTL")
{
field(DESC, "Soft Event 07 Enable/Disable")
field(DTYP, "asynInt32")
field(OUT, "@asyn($(PORT),0) softEvEnable_07")
field(OMSL, "supervisory")
field(ZNAM, "Disabled")
field(ONAM, "Enabled")
field(PINI, "YES")
info(autosaveFields, "VAL")
}
record(longout, "TPR:$(LOCA):$(IOC_UNIT):$(INST):EVSOFT07") {
field(DESC, "Soft Event 07")
field(DTYP, "asynInt32")
field(OUT, "@asyn($(PORT),0) softEv_07")
field(OMSL, "supervisory")
field(PINI, "YES")
info(autosaveFields, "VAL")
}
|
Displays in both Dev and Production exist for the control of the triggers.
To set the triggers and corresponding channels in Production, follow the instructions below.
Code Block | ||
---|---|---|
| ||
[softegr@lcls-srv01 skoufis]$ cd $EPICS_IOCS/sioc-sys0-ms09/iocSpecificRelease/masterSourceApp/srcDisplay/
[softegr@lcls-srv01 srcDisplay]$ ./sioc-sys0-ms09_gui.sh
|
This should launch the main display, from which we can invoke the appropriate channel and trigger screens for setup. See an example of those scenes below.
To set the triggers and corresponding channels in Dev, see the steps below.
Code Block | ||
---|---|---|
| ||
[softegr@lcls-srv01 skoufis]$ cd $EPICS_IOCS/sioc-b084-ms09/iocSpecificRelease/masterSourceApp/srcDisplay/
[softegr@lcls-srv01 srcDisplay]$ ./sioc-b084-ms09_gui.sh |
The released software will be made available in the main IOC release area post-deployment (Dev: /afs/slac/g/lcls/epics/iocTop/masterSource/, Production:/usr/local/lcls/epics/iocTop/masterSource/). See below for the specific repositories and tags.
See software release details below.
Repository | Version | IOC Name |
---|---|---|
masterSource | R2.2.3 | sioc-sys0-ms09 |
See firmware release details below.
Repository | URL | Version |
---|---|---|
slaclab / EvrCardG2 | https://github.com/slaclab/evr-card-g2/releases/tag/v2.2.0 | v2.2.0 |
TPR Documentation
View file | ||||
---|---|---|---|---|
|
PCIe TPR Peer Review
View file | ||||
---|---|---|---|---|
|
See firmware release details below.
...
Repository
...
Version
...