Overview

See below for a brief introduction to the active timing fanout concept, implementation and current deployment.

Theory of operation

The signal delay of an optical signal traversing a fiber can be measured by a phase delay measurement of the optical signal traversing down the fiber, looped back in the reverse direction, and received back at the source, as shown below.  The one-way traversal delay D is thus half the total measured phase delay.



The drift in the traversal delay is compensated by a programmable delay element inserted in the transmission path at the signal generated as shown in the Figure below.  Drift compensation is achieved when the tuned delay change ∆C is equal and opposite to the measured change of phase delay on the return signal.


Implementation

The Timing Delay Module is implemented on the LCLS-II HPS common platform hardware.  Its physical elements are illustrated below.



The front panel consists of 14 SFP+ transceiver sockets - 1 for the timing input and 13 for the timing output distribution.  The transceiver TX/RX signals are routed through a high-speed repeater block between the transceiver and the carrier board FPGA.  It is within the carrier board FPGA that the input signal is routed to the 13 output channels with configurable delay for each output.  

Deployment

Three boards were installed into the LINAC at locations Sector 2 (“GUNB”), Sector 20 (“IN20”), and Bldg 5 (“LTU0”) as pictured below.



The ports with no fiber connection were put into loopback mode.  The port assignments are tabulated below.

Module

Channel

Remote End

Length [m]

TPF:GUNB:000

1 “Out0-1”

S5 L2KG05-6133

403

 

2 “Out0-2”

S8 L2KG08-3C33

686

 

3 “Out0-3”

S11 L2KF11-4033

1026

 

4 “Out0-4”

S17 L2KF17-3D33

1661

 

5 “Out0-5”

S14 L2KF14-7D33

1390

 

6 “Out0-6”

S20 TPF:IN20:000

1996

TPF:IN20:000

8 “Out1-1”

S23 KF23-7D37

440

 

9 “Out1-2”

S26 KF26-7D33

761

 

10 “Out1-3”

S29 L2KG29-7D33

1050

 

12 “Out1-5”

B005 TPF:LTU0:000

-9*

TPF:LTU0:000

1 “Out0-1”

 

542

 

2 “Out0-2”

 

98

 

3 “Out0-3”

 

248

 

4 “Out0-4”

 

422

 

5 “Out0-5”

 

602

IOC packages

See below the third-party and internal non-EPICS libraries (i.e. packages) used in this IOC software.

Package NamePackage Version
CPSWR4.4.1
yaml-cppyaml-cpp-0.5.3_boost-1.64.0
boost1.64.0
yamlReaderR1.2.0
deviceLibraryR1.2.0
hpsTpr R1.1.0
commonATCAR1.3.0

IOC modules

See below the EPICS libraries (i.e. modules) --besides EPICS core-- used in this IOC software

EPICS Module NameEPICS Module Version
ATCACommonR1.7.0
asynR4.39-1.0.1
autosaveR5.8-2.1.0
caPutLogR3.5-1.0.0
iocAdminR3.1.15-1.9.0
yamlDownloaderR1.2.2
yamlLoaderR2.2.0
ycpswasynR3.3.5

Algorithms

The algorithms below pertain to: (I) the conversion of the raw phase originating in the (link status) phase registers and (II) the calculation of power in the transceiver links. 

Link status phase calculation

In the code snippet below (af_asubRPhase.c), an EPICS array subroutine record is used to convert phase to nanoseconds.  Input a represents the raw values in an array format and output vala, also in array format, is the converted phase.

static long ConvPhaseEGU(aSubRecord *prec)
{
  long i;
  epicsUInt32 *a    = (epicsUInt32 *) prec -> a;
  float       *vala = (float       *) prec -> vala;

  for (i = 0; i < prec -> noa; ++i) 
  {
    // Convert raw integer to float 
    vala[i] = ((float)((a[i] & 0xBFFFFFF) << 6) / (float)(1ULL << 32) - 0.5) * (700 / 13);
  }
  return 0; /* process output links */
}

Transmitter/Receiver power calculation

In the code that follows (af_asubRPower.c), receiver and transmitter power is calculated for each SFP link.  Input a is the diagnostics block array, input b is the external calibration block data and input c is the diagnostics monitoring type.  In addition, outputs vala and valb are the calculated receiver and transmitter power respectively.

static long ComputePower(aSubRecord *prec)
{
  long  i, j;
  float rxp_c[5];

  // a: DiagBlock, b: ExtCalBlock, c: DiagMonType
  epicsUInt32 *a = (epicsUInt32 *) prec -> a;
  epicsUInt32 *b = (epicsUInt32 *) prec -> b;
  epicsUInt32 *c = (epicsUInt32 *) prec -> c;

  // vala: rxPower, valb: txPower
  float *vala = (float *) prec -> vala;
  float *valb = (float *) prec -> valb;

  unsigned short int txp_slo;
  signed   short int txp_off;

  // Initialize rxp_c
  for (i = 0; i < 5; ++i)
    rxp_c[i] = 0.0;

  unsigned short int txp = (unsigned short int)( ( ( a[7] & 0xFF ) << 8 ) | ( a[6] & 0xFF ) );
  unsigned short int rxp = (unsigned short int)( ( ( a[9] & 0xFF ) << 8 ) | ( a[8] & 0xFF ) );

  if ( c[0] & (1 << 4) )
  {
    txp_slo = (unsigned short int)( ( ( b[25] & 0xFF ) << 8 ) | ( b[24] & 0xFF ) ) / 256;
    txp_off = (signed short int  )( ( ( b[27] & 0xFF ) << 8 ) | ( b[26] & 0xFF ) );

    rxp_c[4] = (float)( ( ( b[3 ] & 0xFF ) << 24 ) | ( ( b[2 ] & 0xFF ) << 16 ) | ( ( b[1 ] & 0xFF ) << 8 ) | ( ( b[0 ] & 0xFF ) ) ); 
    rxp_c[3] = (float)( ( ( b[7 ] & 0xFF ) << 24 ) | ( ( b[6 ] & 0xFF ) << 16 ) | ( ( b[5 ] & 0xFF ) << 8 ) | ( ( b[4 ] & 0xFF ) ) ); 
    rxp_c[2] = (float)( ( ( b[11] & 0xFF ) << 24 ) | ( ( b[10] & 0xFF ) << 16 ) | ( ( b[9 ] & 0xFF ) << 8 ) | ( ( b[8 ] & 0xFF ) ) ); 
    rxp_c[1] = (float)( ( ( b[15] & 0xFF ) << 24 ) | ( ( b[14] & 0xFF ) << 16 ) | ( ( b[13] & 0xFF ) << 8 ) | ( ( b[12] & 0xFF ) ) ); 
    rxp_c[0] = (float)( ( ( b[19] & 0xFF ) << 24 ) | ( ( b[18] & 0xFF ) << 16 ) | ( ( b[17] & 0xFF ) << 8 ) | ( ( b[16] & 0xFF ) ) ); 
  }
  else
  {
    txp_slo  = 1;
    txp_off  = 0;
    rxp_c[1] = 1.0;
  }

  float txpwr = (float)( txp * txp_slo + txp_off ) * 0.0001; // mW
  float rxpwr = (float)( rxp_c[0] );

  unsigned short int rxpn = rxp;

  for (j = 0; j < 5; ++j)
  {
    rxpwr += (float)rxp_c[j] * (float)rxpn;
    rxpn  *= rxp;
  }

  rxpwr *= 0.0001; // mW
  vala[0] = rxpwr;
  valb[0] = txpwr;

  return 0; /* process output links */
}

EPICS record types used

Find below the description of the EPICS record types used to stand up this IOC.

Records attached to registers

The EPICS records attached to the registers of the delay element FPGA are mainly three: longin, longout and waveform.  See some examples below.  An EPICS macros substitutions file is used with record templates to automate the creation of these records.

record(longin,  "$(P):RXREADYSUMY") {
    field(DTYP, "asynInt32")
    field(DESC, "RX Ready")
    field(PINI, "YES")
    field(SCAN, "1 second")
    field(INP,  "@asyn($(PORT),0)RX_READY")
}
record(longout, "$(P):TXRSTSUMY") {
    field(DTYP, "asynInt32")
    field(DESC, "TX reset")
    field(PINI, "NO")
    field(SCAN, "Passive")
    field(OUT,  "@asyn($(PORT),1)TX_RESET")
}
record(waveform,    "$(P):LOOPPHAS") {
    field(DTYP,     "asynInt32ArrayIn")
    field(DESC,     "Loop phase measurement [14:0]")
    field(PINI,     "YES")
    field(SCAN,     "1 second")
    field(NELM,     "15")
    field(FTVL,     "LONG")
    field(INP,      "@asyn($(PORT),0)LOOP_PHASE")
}

Bit-referencing records

To isolate individual bits from read integer values, we use two main records: mbbiDirect to read the bits of a PV value connected to a read-only register and mbboDirect to write bits to a PV value tied to an FPGA read/write register.  See examples below.

record(mbbiDirect, "$(P):TXREADY") {
    field(DESC, "Tx Ready bits")
    field(PINI, "NO")
    field(SCAN, "Passive")
    field(INP,  "$(P):TXREADYSUMY CP")
}
record(mbboDirect, "$(P):LOOPBACKCTRL") {
    field(DESC, "Loopback write")
    field(PINI, "YES")
    field(SCAN, "Passive")
    field(OMSL, "supervisory")
    field(OUT,  "$(P):LOOPBACKENB PP")
    field(FLNK, "$(P):LOOPBACKRBV.PROC")
}

Array-splitting records

When a waveform PV is given, which essentially implies an array of registers, then we use the subArray record to isolate each element of the array in its own separate PV.  For instance:

record(subArray,    "$(P):CH0_TXBUF") {
    field(DESC,     "Transmitter fifo depth [0]")
    field(PINI,     "NO")
    field(NELM,     "1")
    field(MALM,     "15")
    field(INDX,     "0")
    field(FTVL,     "LONG")
    field(INP,      "$(P):TXBUFDEPTH.VAL CP")
}
record(subArray,    "$(P):CH3_LOOPPHAS") {
    field(DESC,     "Loop phase [3]")
    field(EGU,      "ns")
    field(PINI,     "NO")
    field(NELM,     "1")
    field(MALM,     "15")
    field(INDX,     "3")
    field(FTVL,     "DOUBLE")
    field(INP,      "$(P):LOOPPHASEGU.VALA CP")
}
record(subArray,    "$(P):CH10_LOOPCLK") {
    field(DESC,     "Loop clocks measurement [10]")
    field(PINI,     "NO")
    field(NELM,     "1")
    field(MALM,     "15")
    field(INDX,     "10")
    field(FTVL,     "LONG")
    field(INP,      "$(P):LOOPCLKS.VAL CP")
}

Power calculation records

To compute the power in the transmitter/receiver pair, we use the following EPICS record types:

seq (i.e. <DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERPWR),

ao (i.e. <DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_MUXCTRL) and

aSub (i.e. <DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_PWRLNK)


The following PVs are required to compute the power on each SFP.

PV Name
<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_MUXCTRL
<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_PWRLNK
<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERPWR
<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERDIAGBLK
<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBEREXTCAL
<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERTYPE


The <AMC_ID>:FIBERPWR record is the main engine of this mechanism. 

This PV triggers <DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_MUXCTRL to write to the I2C MUX control register in order to select which module from the 7 SFPs to read data from.  Upon selecting the module, we read in the following PVs for that SFP:

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERDIAGBLK,

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBEREXTCAL and 

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERTYPE.

The power computation takes place in the <DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_PWRLNK.  We do this for a maximum of 7 times for each AMC card, for a total of 14 times.   Please find the algorithm for this calculation in the Algorithms section above. 

See the block diagram below for a graphical representation of the power calculation process and the interaction between the involved PVs.



Note that if an SFP module is absent, it is skipped.  This is accomplished by populating a mask field in the seq record informing it about the present SFPs.  The PV populating the mask field is  <DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERADPINVINT.

PV tables

The PV names have been distributed to various categories depending on the intended functionality.  Overall the IOC uses 272 PVs.  Below only the PVs directly attached to the FPGA registers or those appearing on the GUI are provided.  The intermediate PVs have been omitted to avoid confusion.

Timing status PVs

PV NameEPICS Record TYPEAssociated PV (Derived From)PV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:FIDCNT

longinPrimaryFiducial countRO

<DEVICE>:<SECTOR>:<STATION>:FIDCNTDIFF

calc<DEVICE>:<SECTOR>:<STATION>:FIDCNTFiducial count differenceRO

Clock PVs

PV NameEPICS Record TYPEAssociated PV (Derived From)PV DescriptionMode
<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOKC_DOMAIN_ID>:RATElonginPrimaryRateRO
<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOKC_DOMAIN_ID>:RATEMHZcalc<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOKC_DOMAIN_ID>:RATERate in MHzRO

Link control PVs

PV NameEPICS Record TYPEAssociated PV (Derived From)PV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:TXRST

mbbiDirectTXRSTRBVTX reset (R)RO

<DEVICE>:<SECTOR>:<STATION>:TXRSTCTRL

mbboDirectTXRSTSUMYTX reset (W)RW
<DEVICE>:<SECTOR>:<STATION>:RXRSTmbbiDirectRXRSTRBVRX reset (R)RO
<DEVICE>:<SECTOR>:<STATION>:RXRSTCTRLmbboDirectRXRSTSUMYRX reset (W)RW

<DEVICE>:<SECTOR>:<STATION>:RXRSTCNT

mbbiDirect

RXRSTCNTRBV

RX reset count (R)RO

<DEVICE>:<SECTOR>:<STATION>:RXRSTCNTCTRL

mbboDirect

RXRSTCNTSUMY

RX reset count (W)RW

<DEVICE>:<SECTOR>:<STATION>:LOOPBACK

mbbiDirect

LOOPBACKRBV

Loopback (R)RO

<DEVICE>:<SECTOR>:<STATION>:LOOPBACKCTRL

mbboDirect

LOOPBACKENB

Loopback (W)RW

<DEVICE>:<SECTOR>:<STATION>:TXBUFRST

mbbiDirect

TXBUFRSTRBV

TX FIFO reset (R)RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFRSTCTRL

mbboDirect

TXBUFRSTSUMY

TX FIFO reset (W)RW

<DEVICE>:<SECTOR>:<STATION>:TXBUFHOLD

mbbiDirect

TXBUFHOLDRBV

TX FIFO hold (R)RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFHOLDCTRL

mbboDirect

TXBUFHOLDSUMY

TX FIFO hold (W)RW

<DEVICE>:<SECTOR>:<STATION>:TXBUFADV

mbbiDirect

TXBUFADVRBV

TX FIFO advance (R)RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFADVCTRL

mbboDirect

TXBUFADVSUMY

TX FIFO advance (W)RW

<DEVICE>:<SECTOR>:<STATION>:TXDLYFAST

mbbiDirect

TXDLYFASTRBV

TX delay fast (R)RO

<DEVICE>:<SECTOR>:<STATION>:TXDLYFASTCTRL

mbboDirect

TXDLYFASTSUMY

TX delay fast (W)RW

<DEVICE>:<SECTOR>:<STATION>:TXPIRST

mbbiDirect

TXPIRSTRBV

TX PI reset (R)RO

<DEVICE>:<SECTOR>:<STATION>:TXPIRSTCTRL

mbboDirect

TXPIRSTSUMY

TX PI reset (W)RW

Link status PVs

PV NameEPICS Record TYPEAssociated PV (Derived From)PV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_RXERRCNTS

subArray

<DEVICE>:<SECTOR>:<STATION>:RXERRCNTS

RX error countsRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_RXLATENCY

subArray

<DEVICE>:<SECTOR>:<STATION>:_RXLATENCYRX latencyRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_RXREADY

bi<DEVICE>:<SECTOR>:<STATION>:RXREADY.B<BIT_ID>RX ReadyRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_TXREADY

bi<DEVICE>:<SECTOR>:<STATION>:TXREADY.B<BIT_ID>TX ReadyRO

<DEVICE>:<SECTOR>:<STATION>:TXRSTDONE

mbbiDirect

TXRSTDONESUMY

TX reset doneRO
<DEVICE>:<SECTOR>:<STATION>:RXRSTDONEmbbiDirect

RXRSTDONESUMY

RX reset doneRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_TXBUF

subArray<DEVICE>:<SECTOR>:<STATION>:TXBUFDEPTHTX FIFO depthRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_LOOPPHAS

subArray<DEVICE>:<SECTOR>:<STATION>:LOOPPHASEGULoop phaseRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_LOOPPHASCOMP

subArray

<DEVICE>:<SECTOR>:<STATION>:LOOPCOMPEGU

Comp loop phaseRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_LOOPCLK

subArray<DEVICE>:<SECTOR>:<STATION>:LOOPCLKSLoop clocksRO

<DEVICE>:<SECTOR>:<STATION>:RXALIGNMASK

waveformPrimaryRX align maskRW

<DEVICE>:<SECTOR>:<STATION>:RXALIGNMASKRBV

waveformPrimaryRX align mask RBVRO

<DEVICE>:<SECTOR>:<STATION>:RXALIGNTGT

waveformPrimaryRX align targetRW

<DEVICE>:<SECTOR>:<STATION>:RXALIGNTGTRBV

waveformPrimaryRX align target RBVRO

<DEVICE>:<SECTOR>:<STATION>:TXDLY

waveformPrimaryTX delayRW

<DEVICE>:<SECTOR>:<STATION>:TXDLYRBV

waveformPrimaryTX delay RBVRO

PLL PVs

PV NameEPICS Record TYPEAssociated PV (Derived From)PV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:LOL

longinPrimaryLoss of lockRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:LOLCNT

longinPrimaryLoss of lock countRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:LOS

longinPrimaryLoss of signalRO

<DEVICE>:<SECTOR>:<STATION>:PLL0:BWSEL

longoutPrimaryBandwidth selectRW

<DEVICE>:<SECTOR>:<STATION>:PLL0:CLKSEL

longoutPrimaryClock selectRW

<DEVICE>:<SECTOR>:<STATION>:PLL0:FREQSEL

longoutPrimaryFrequency selectRW

<DEVICE>:<SECTOR>:<STATION>:PLL0:FREQTBLSEL

longoutPrimaryFrequency table selectRW

<DEVICE>:<SECTOR>:<STATION>:PLL0:RATESEL

longoutPrimaryRate selectRW

<DEVICE>:<SECTOR>:<STATION>:PLL0:ACTIVELOWRST

longoutPrimaryResetRW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:BWRBV

longinPrimaryBandwidth RBVRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:CLKRBV

longinPrimaryClock RBVRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:FREQRBV

longinPrimaryFrequency RBVRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:FREQTBLRBV

longinPrimaryFrequency table RBVRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:RATERBV

longinPrimaryRate RBVRO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:ACTIVELOWRSTRBV

longinPrimaryReset RBVRO

<DEVICE>:<SECTOR>:<STATION>:RXLNKPOLRBV

longinPrimaryRX polarityRO

SFP status PVs

PV NameEPICS Record TYPEAssociated PV (Derived From)PV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_FIBERSTATUS

bi

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERSTATUS.B<BIT_ID>

Loss of signalRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_FIBERADAPTOR

bi<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERNOADAPTOR.B<BIT_ID>Module absentRO

<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_RXPOWER

ai<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_PWRLNK.VALARX powerRO
<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_TXPOWERai<DEVICE>:<SECTOR>:<STATION>:CH<CHANNEL_ID>_PWRLNK.VALBTX powerRO

Primary PVs

The primary PVs are PVs that are directly attached to the FPGA registers and are defined with an EPICS macros substitutions file.

Read-Only PVs

PV NameEPICS Record TYPEAssociated FPGA RegisterPV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:RXREADYSUMY

longinrxReady

Receive ready

RO

<DEVICE>:<SECTOR>:<STATION>:RXRSTDONESUMY

longinrxResetDone

Receive reset done

RO

<DEVICE>:<SECTOR>:<STATION>:RXLNKUP

longin

RxLinkUp

Receive link status

RO

<DEVICE>:<SECTOR>:<STATION>:TXREADYSUMY

longintxReady

Transmit ready

RO

<DEVICE>:<SECTOR>:<STATION>:TXRSTDONESUMY

longintxResetDone

Transmit reset done

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:LOL

longinlol

Loss of lock status

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:LOLCNT

longinlolCnt

Loss of lock counts

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:LOS

longinlos (PLL)

Loss of signal status

RO
<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERSTATSUMYlonginlos (SFP)

Loss of signal

RO
<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERADAPTORSUMYlonginmodabsModule absentRO
<DEVICE>:<SECTOR>:<STATION>:FIDCNTlonginFidCount

Valid frame count

RO

<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOCK_DOMAIN_ID>:FAST

longin

SyncClockFreq[<0,1,2,3,4>]/fast

Overflow

RO

<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOCK_DOMAIN_ID>:LOCK

longinSyncClockFreq[<0,1,2,3,4>]/lock

Measurement locked

RO

<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOCK_DOMAIN_ID>:RATE

longinSyncClockFreq[<0,1,2,3,4>]/rate

Measured rate

RO

<DEVICE>:<SECTOR>:<STATION>:SYNCLK<CLOCK_DOMAIN_ID>:SLOW

longinSyncClockFreq[<0,1,2,3,4>]/slow

Underflow

RO

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERTYPE

longinDiagMonType

Diagnostic monitoring type

RO

<DEVICE>:<SECTOR>:<STATION>:LOOPCLKS

waveform

loopClocks[0-14]

Loop clocks measurement

RO

<DEVICE>:<SECTOR>:<STATION>:LOOPPHAS

waveform

loopPhase[0-14]

Loop phase measurement

RO

<DEVICE>:<SECTOR>:<STATION>:LOOPCOMP

waveform

loopPhaseCompl[0-14]

Loop phase complementary measurement

RO

<DEVICE>:<SECTOR>:<STATION>:RXERRCNTS

waveform

rxErrCnts[0-14]

Receive error counts

RO

<DEVICE>:<SECTOR>:<STATION>:RXLATENCY

waveform

rxLatency[0-14]

Receive comma latency

RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFDEPTH

waveform

txFifoDepth[0-14]

Transmitter FIFO depth

RO

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERDIAGBLK

waveform

DiagBlock[0-9]

Diagnostics data block

RO

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBEREXTCAL

waveform

ExtCalBlock[0-35]

External calibration constants blockRO

Read-Write PVs

PV NameEPICS Record TYPEAssociated FPGA RegisterPV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:LOOPBACKENB

longout

loopback

Transceiver loopback control

RW

<DEVICE>:<SECTOR>:<STATION>:RXRSTCNTSUMY

longout

rxCntReset

Receiver error count reset

RW

<DEVICE>:<SECTOR>:<STATION>:RXRSTSUMY

longoutrxReset

Reset receiver

RW

<DEVICE>:<SECTOR>:<STATION>:TXDLYFASTSUMY

longout

txDelayFast

Transmitter delay quick step

RW

<DEVICE>:<SECTOR>:<STATION>:TXBUFADVSUMY

longout

txFifoAdvance

Advance transmit fifo

RW

<DEVICE>:<SECTOR>:<STATION>:TXBUFHOLDSUMY

longout

txFifoHold

Hold transmit fifo

RW

<DEVICE>:<SECTOR>:<STATION>:TXBUFRSTSUMY

longout

txFifoReset

Transmitter FIFO reset

RW

<DEVICE>:<SECTOR>:<STATION>:TXPIRSTSUMY

longout

txPiReset

Transmitter delay reset

RW

<DEVICE>:<SECTOR>:<STATION>:TXRSTSUMY

longout

txReset

Reset transmitter

RW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:BWSEL

longout

TimingDelayPll[<0,1>]/bwSel

Bandwidth select

RW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:CLKSEL

longout

TimingDelayPll[<0,1>]/clkSel

Clock select

RW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:FREQSEL

longout

TimingDelayPll[<0,1>]/frqSel

Frequency select

RW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:FREQTBLSEL

longout

TimingDelayPll[<0,1>]/frqTbl

Frequency table

RW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:RATESEL

longout

TimingDelayPll[<0,1>]/rate

Rate selection

RW

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:ACTIVELOWRST

longout

TimingDelayPll[<0,1>]/rstn

Active low reset

RW

<DEVICE>:<SECTOR>:<STATION>:RXLNKPOLCTRL

longout

RxPolarity

Invert receive link polarity

RW

<DEVICE>:<SECTOR>:<STATION>:RXLNKRST

longout

RxReset

Reset receive link

RW

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERMUXCTRL

longout

MpsSfpAmc[<0,1>]/I2cMux/control

I2c Multiplexer bus selector

RW

<DEVICE>:<SECTOR>:<STATION>:RXALIGNMASK

waveform

rxAlignMask[0-14]

Receive alignment mask

RW

<DEVICE>:<SECTOR>:<STATION>:RXALIGNTGT

waveform

rxAlignTarget[0-14]

Receive alignment target

RW

<DEVICE>:<SECTOR>:<STATION>:TXDLY

waveform

txDelay[0-14]

Transmitter delay target

RW

<DEVICE>:<SECTOR>:<STATION>:TGTCLKS

waveform

tgtClks[0-14]

Link FSM target clks

RW

<DEVICE>:<SECTOR>:<STATION>:TGTPHAS

waveform

tgtPhase[0-14]

Link FSM target phase

RW

RBVs for Read-Write PVs

PV NameEPICS Record TYPEAssociated FPGA RegisterPV DescriptionMode

<DEVICE>:<SECTOR>:<STATION>:LOOPBACKRBV

longin

loopback

Transceiver loopback control

RO

<DEVICE>:<SECTOR>:<STATION>:RXRSTCNTRBV

longin

rxCntReset

Receiver error count reset

RO

<DEVICE>:<SECTOR>:<STATION>:RXRSTRBV

longinrxReset

Reset receiver

RO

<DEVICE>:<SECTOR>:<STATION>:TXDLYFASTRBV

longin

txDelayFast

Transmitter delay quick step

RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFADVRBV

longin

txFifoAdvance

Advance transmit fifo

RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFHOLDRBV

longin

txFifoHold

Hold transmit fifo

RO

<DEVICE>:<SECTOR>:<STATION>:TXBUFRSTRBV

longin

txFifoReset

Transmitter FIFO reset

RO

<DEVICE>:<SECTOR>:<STATION>:TXPIRSTRBV

longin

txPiReset

Transmitter delay reset

RO

<DEVICE>:<SECTOR>:<STATION>:TXRSTRBV

longin

txReset

Reset transmitter

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:BWRBV

longin

TimingDelayPll[<0,1>]/bwSel

Bandwidth select

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:CLKRBV

longin

TimingDelayPll[<0,1>]/clkSel

Clock select

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:FREQRBV

longin

TimingDelayPll[<0,1>]/frqSel

Frequency select

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:FREQTBLRBV

longin

TimingDelayPll[<0,1>]/frqTbl

Frequency table

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:RATERBV

longin

TimingDelayPll[<0,1>]/rate

Rate selection

RO

<DEVICE>:<SECTOR>:<STATION>:PLL<AMC_ID>:ACTIVELOWRSTRBV

longin

TimingDelayPll[<0,1>]/rstn

Active low reset

RO

<DEVICE>:<SECTOR>:<STATION>:RXLNKPOLRBV

longin

RxPolarity

Invert receive link polarity

RO

<DEVICE>:<SECTOR>:<STATION>:RXLNKRSTRBV

longin

RxReset

Reset receive link

RO

<DEVICE>:<SECTOR>:<STATION>:<AMC_ID>:FIBERMUXRBV

longin

MpsSfpAmc[<0,1>]/I2cMux/control

I2c Multiplexer bus selector

RO

<DEVICE>:<SECTOR>:<STATION>:RXALIGNMASKRBV

waveform

rxAlignMask[0-14]

Receive alignment mask

RO

<DEVICE>:<SECTOR>:<STATION>:RXALIGNTGTRBV

waveform

rxAlignTarget[0-14]

Receive alignment target

RO

<DEVICE>:<SECTOR>:<STATION>:TXDLYRBV

waveform

txDelay[0-14]

Transmitter delay target

RO

<DEVICE>:<SECTOR>:<STATION>:TGTCLKSRBV

waveform

tgtClks[0-14]

Link FSM target clks

RO

<DEVICE>:<SECTOR>:<STATION>:TGTPHASRBV

waveform

tgtPhase[0-14]

Link FSM target phase

RO

Engineering GUI

See below the engineering GUI tabs.  To start the GUI we do the following (assuming that DEVICE="TPF", SECTOR="B084", STATION="000"):


$ cd tpgSupportApp/screens/pydm/
$ pydm -m "P=TPF:B084:000" activeFanout.ui

Timing status tab

Clocks tab

Link status tab

Link control tabs

PLLs tab

SFP status tab

IOC Deployment

This IOC is deployed as a separate executable in an existing group of timing-supporting binaries.  Startup scripts have been put together to boot the IOC using the active timing fanout executable.  The scripts are configured as needed depending on the case.  For more information on the startup scripts see the section on IOC booting below.

Housing module (TpgSupport)

The active fanout IOC is housed under the TpgSupport umbrella of IOCs.  To check out the master branch of TpgSupport, you would need to do the following:

$ eco -m TpgSupport


To get eco in your path, do the following:

$ source /afs/slac/g/lcls/tools/script/ENVS64.bash

Binary 

Binary NameLinux Distribution
activeFanout

rhel6-x86_64

linuxRT-x86_64

Databases

Database NameDatabase Description

af_asubRPhase.db

Phase calculation records

af_asubRPower.db

Power calculation records

af_cpsw.db

Records for FPGA register R/W Operations

af_records_proc.db

Records to process primary PVs

af_support_records.db

Other various support records

af_summary.db

Landing PVs for RX/TX ready status and power 

af_summary_fiber.db

Landing PVs for fiber and SFP module status

Booting the IOC

File structure

In its current form, the booting mechanism involves three main scripts: af_pre_init.cmd, af_post_init.cmd and st.cmd.  See below for more details on each one of these.  The pre-init and post-init scripts are common for all IOCs and are therefore placed in iocBoot/common/.  On the other hand, the st.cmd script is specific to the application at hand each time and hence is placed into its own IOC directory as usual.

Startup scripts

af_pre_init.cmd

The af_pre_init.cmd startup script is responsible for setting the necessary environment variables, loading the required databases, loading the yaml files from the FPGA PROM or locally, configuring CPSW and enabling autosave/restore mode.

## Set environment variables
epicsEnvSet("EPICS_CA_MAX_ARRAY_BYTES", "1000000")                                                                                                            
epicsEnvSet("DEVICE","TPF")
epicsEnvSet("SUBSYS","${DEVICE}")
epicsEnvSet("P","${DEVICE}:${SECTOR}:${STATION}")
#epicsEnvSet("YAML_DIR", "${TOP}/yaml-AmcCarrierTimingDelay")
epicsEnvSet("YAML_DIR", "${IOC_DATA}/${IOC}/yaml")
epicsEnvSet("YAML_DICT_DIR", "${TOP}/yaml")
epicsEnvSet("TOP_YAML","${YAML_DIR}/000TopLevel.yaml")

# Setting YAML_PATH so yaml files that are missing from the FPGA PROM can be added locally.
# YAML_PATH is a hardcoded macro used by cpswLoadYamlFile.
# The path follows a standard seen in other IOC apps:
# firmware/<firmware name>/<hash with 7 digits>/yaml_local
epicsEnvSet("YAML_PATH", "${TOP}/firmware/AmcCarrierTimingDelay/hash-12c3011/yaml_local") 

# =====================================================
# Source Channel Access Security
# =====================================================

< ${ACF_INIT}
< ${LOG_INIT}

## Register all support components
dbLoadDatabase("${TOP}/dbd/activeFanout.dbd",0,0)
activeFanout_registerRecordDeviceDriver(pdbbase) 

cd ${TOP}

## ======================================================================================
# Uncomment two lines below to read the yaml files from the FPGA PROM.
## Call the Yaml Downloader to download the YAML files from the FPGA.
# DownloadYamlFile(
#   FPGA_IP,    # Target FPGA IP address.
#   DEST_PATH)  # The destination folder where the YAML file will be written.
DownloadYamlFile("${FPGA_IP}", "${YAML_DIR}")
## ======================================================================================

## ======================================================================================
## Load FPGA hierarchy and register map from YAML files.
## cpswLoadYamlFile ( YAML hierarchy description and register map file,
##                    Root Device Name (optional; default = 'root'),
##                    directory where YAML includes can be found (optional),
##                    IP address for carrier board (override the IP address in YAML) )
##
## ======================================================================================
cpswLoadYamlFile("${TOP_YAML}", "NetIODev", "", "${FPGA_IP}")

## ================================================================================================
## Load FPGA configuration/initialization from a YAML file
## cpswLoadConfigFile ( CONFIG YAML file (NOT hierarchy YAML!),
##                      Path prefix (if config YAML is not based of the root device; optional),
##                      directory where YAML includes can be found (optional) )
## ================================================================================================
#cpswLoadConfigFile("${YAML_DIR}/config/defaultsGenRTMV2.yaml", "mmio", "")

## Don't use Automatic generation of records from the YAML definition
## 0 = No, 1 = Yes
epicsEnvSet("AUTO_GEN", 0)

## Dictionary file for manual (empty string if none)
epicsEnvSet("DICT_FILE", "activeFanout.dict")

## ==========================================================================================================================
## YCPSWASYN Configuration and Initialzation: EPICS Records will be born.
## ==========================================================================================================================
## Configure asyn port driver
## YCPSWASYNConfig(
##    Port Name,                 # the name given to this port driver
##    Root Path                  # OPTIONAL: Root path to start the generation. If empty, the Yaml root will be used
##    Record name Prefix,        # Record name prefix (ignored if not using auto-generation of records)
##    Use DB Autogeneration,     # Set to 1 for autogeneration of records from the YAML definition. Set to 0 to disable it
##    Load dictionary,           # Dictionary file path with registers to load. An empty string will disable this function
##    Named Root                 # OPTIONAL: Name of the root to be used, only necessary to support multiple ATCA blades on a single IOC.
## ==========================================================================================================================
YCPSWASYNConfig("ycpswPORT", "", "${P}", "${AUTO_GEN}", "${YAML_DICT_DIR}/${DICT_FILE}", "")

# =====================================================
# =====================================================
## Setup autosave/restore
# =====================================================
save_restoreSet_IncompleteSetsOk( 1 )
save_restoreSet_DatedBackupFiles( 1 )

set_requestfile_path("${TOP}/autosave")
set_requestfile_path("${IOC_DATA}/${IOC}/autosave-req")
set_savefile_path("${IOC_DATA}/${IOC}/autosave")

set_pass0_restoreFile("info_positions.sav")
set_pass0_restoreFile("${IOC}.sav")
set_pass1_restoreFile("info_settings.sav")

save_restoreSet_status_prefix("${P}:")
dbLoadRecords("db/save_restoreStatus.db", "P=${IOCPV}:")

# =====================================================
# =====================================================
## iocAdmin
# =====================================================
dbLoadRecords("db/iocAdminSoft.db"  , "IOC=${IOCPV}")
dbLoadRecords("db/iocRelease.db"  , "IOC=${IOCPV}")

# =====================================================
# =====================================================
## Load Active Timing Fanout databases
# =====================================================
## load records from the atf cpsw substitutions database
dbLoadRecords("db/af_cpsw.db", "P=${P}, PORT=ycpswPORT")

## load asub records
dbLoadRecords("db/af_asubRPhase.db", "P=${P}")

## load records that process the records from ycpswAsyn
dbLoadRecords("db/af_records_proc.db", "P=${P}")

## load records for general support
dbLoadRecords("db/af_support_records.db", "P=${P}")

## load records to support user displays
dbLoadRecords("db/af_summary.db", "P=${P}")
dbLoadRecords("db/af_summary_fiber.db", "P=${P}")

af_post_init.cmd

The af_post_init.cmd startup script enables the archiving of data, turns on logging, starts saving datasets and ensures proper alarm configuration.

system("cp archive/${IOC}.archive ${IOC_DATA}/${IOC}/archive/") 

# =====================================================
# Turn on caPutLogging:
# Log values only on change to the iocLogServer:
# =====================================================
caPutLogInit("${EPICS_CA_PUT_LOG_ADDR}")
caPutLogShow(2)

# =====================================================
## Start saving datasets
# =====================================================
cd("${IOC_DATA}/${IOC}/autosave-req")

makeAutosaveFiles()

create_monitor_set("info_positions.req", 60, "")
create_monitor_set("info_settings.req", 60, "")
create_monitor_set("${IOC}.req", 60, "")

# Make sure that alarm configuration is correctly set for ycpswAsyn records
dbpf ${P}:RXLNKUP.LOW 0
dbpf ${P}:RXLNKUP.LSV MAJOR

# Bits 1 to 13 (starting from 0) must be set. Otherwise the alarm is MINOR.
# Bits 1 to 13 = 0x3FFE = 16382
dbpf ${P}:RXREADYSUMY.HIGH 16383
dbpf ${P}:RXREADYSUMY.LOW 16381
dbpf ${P}:RXREADYSUMY.HSV MINOR
dbpf ${P}:RXREADYSUMY.LSV MINOR

st.cmd

See below an example of a startup script utilizing the pre-init and post-init scripts from above.  The variables SECTOR, STATION, IOCPV and FPGA_IP are specific to our test station case only.  These variables are meant to reflect the particular facility and hardware configuration.

#!../../bin/linuxRT-x86_64/activeFanout

< envPaths

## Set environment variables
epicsEnvSet("SECTOR","B084")
epicsEnvSet("STATION","000")
epicsEnvSet("IOCPV","SIOC:B084:TS08")
epicsEnvSet("FPGA_IP", "10.0.1.106")

cd ${TOP}

< iocBoot/common/af_pre_init.cmd

iocInit()

< iocBoot/common/af_post_init.cmd

Booting options

<ToDo>

There are different ways to boot an IOC. 


  • By directly running the st.cmd startup script
cd <ioc-boot-directory>
./st.cmd


  • Using iocconsole
iocconsole <ioc-name>

References

  1. Timing Delay Module
  2. TPR Active Fanout PV Name Worksheet


Timing Delay Module-2.docx

  • No labels