Introduction

The Bunch Length system is responsible for measuring the length of the bunch of electrons after the bunch compressors, using indirect measurement techniques. For a detailed and hopefully entertaining introduction on the physics and mathematics behind the system, please read the document How the Length of the Bunch is Measured.

The system is based on the High-Performance System (HPS) over ATCA built at SLAC. For an introduction, please read ATCA 101.

Creating a new st.cmd

Use cases

Actors

The figures with white heads represent a person and the ones with grey head represent a system.

• Operator: controls the accelerator, driving the beam with the parameters requested by the user (the users are not represented on this diagram). The Operator is responsible for identifying issues and solve them or call IOC maintainers or Physicists in order to get the best quality of a beam.
• Physicist: understands and design the connection between the electron and photon beam parameters, how sub-systems are interconnected and how the changing in parameters affects the overall beam quality.
• IOC maintainer: engineer responsible for diagnosing problems in IOCs, fixing bugs, and improving the software. For ATCA based platform, the IOC maintainer actor can also mean a firmware maintainer, that deals with the FPGA on carrier boards.
• BPM IOC: this represents an IOC responsible for measuring the electron beam position on a cross-section of the accelerator.
• Fast feedback: this system receives parameters from key accelerator systems and acts to correct the electron beam.
• MPS: this is the system responsible for the safety of some of the equipment of the accelerator.
• BSA: system responsible to hold important data aligned with a specific beam pulse in a way that a user can compare the state of many parameters of the accelerator within the same pulse.

Calculate the maximum current and signal sum

The goal is to receive inputs from sensors and other systems and calculate the maximum current and charge (called signal sum) of the electron beam. Those are indirect measurements representing the amplitude and area of a Gaussian, which are easily translated to a bunch length by the Physicists. Details on this process and the calculation itself are described in How the Length of the Bunch is Measured. 

The sequence diagram below breaks the bunch length system into software and firmware to be more precise. In this context, the software is considered the code made in C/C++ and running in a CPU with Linux RT as the operating system. Firmware is the code synthesized in the FPGA from a VHDL source code running on the carrier board in the ATCA crate. The maximum current is called IMAX and the charge is called SUM.

The BPM IOC and the Bunch Length firmware are triggered by the timing system. The firmware reads and digitizes the sensors upon receiving the trigger. Once the software receives TMIT from the BPM through the FCOM network, it resends the info to the firmware that uses it to calculate IMAX and SUM. The firmware sends the results to the software, that directs the info to two different places: the Fast Feedback receives it through the FCOM network and the BSA has its records filled with them.

Introduction to FCOM can be found here: Using FCOM in the IOC App.

The Bunch Length system uses two separate threads to deal with the data:

The FCOM thread is responsible for sending the TMIT information to the firmware as soon as it arrives from the BPM IOC. The BSA thread is responsible to receive the result from the firmware, send it to the Fast Feedback system, and store it in the BSA records.

The sequence of actions of each thread is described in details below:

Thread task in a continuous loop to deal with FCOM receiving. It is in the file blenFcom.cpp.

processBlob: a function called by the FCOM task. It is in the file blenFcom.cpp.

Thread task in a continuous loop to deal with BSA storing and data being sent to the fast feedback system. It is in the file cpswStreamBSA.cpp.

Control filters and check status of them

The system controls some optical filters and reads the position of the filters by using digital I/O located on the RTM attached to the blade dedicated to the Bunch Length system.

The IOC application reads and writes to the FPGA registers by using CPSW and the control is done entirely in the EPICS database. For reading the status of the filters the register provides an integer where each bit represents a status of a filter. For commanding the filter, an integer is written to the register, using this same bit code. The bit mapping is the following:

The EPICS database files are blenFilterDecoders.db, blenFilters.substitutions, blenFilters.template.

Check length parameters and measurement

The system provides data related to two different bunch length measurements for the Operators, one for each pyrodetector. This is the PyDM screen that can be used to check the system:

The streaming of the waveforms can be turned on or off to reduce the data from the ATCA to the CPU. The Operator can choose 4 different status for the stream: Running, Maintenance, Test, and OFFLINE. The first 3 don't change the behavior of the streaming, they are used only for alerting other users. "OFFLINE" is the only status the effectively shut down the streaming. The database that controls the stream is streamControl.db.

The Sum and IMAX are calculated by the firmware. TMIT is provided by the BMP IOC and sent to the firmware as described earlier. The TMIT value is replied back by the firmware and the software publishes it as the PV we can see on the screen. IMAX can show "nan" in case an error flag is sent by the firmware. In the example above, the pyrodetector is disconnected and, so, Sum is calculated on noise, resulting in a negative number. IMAX was also negative and the firmware alerted us about this, what we can see as a "nan" inside a pink box.

The dashed blue lines indicate where the integration window is placed. The left side tells the system where to integrate the background noise while the right side configures where the real signal is. The chart below the raw waveform shows the result when multiplying the background by +1 and the signal by -1. The system will integrate this second waveform to get the non-calibrated length measurement. The calibration parameters and windows adjustments are described in the next topic. 

Adjust calibration parameters

TMIT and the integral of the raw waveform are used to calculate the bunch length. The parameters that need to be input in the system are the integration windows, plus a linear adjustment function [f(x)=a.x+b] for the sum of the signal, plus the coefficients of this equation:

Each group of elements will be broken into sections for better comprehension:

Adjustment of the integration windows

The system provides a way to adjust 3 regions for the signal. The leftmost region, shown as "Duration pre edge (ns)" on the picture below, integrates the background noise to be subtracted from the signal region. The rightmost region, shown as "Duration pos edge (ns)" on the picture below, integrates the signal. An optional "Duration inter edges (ns)" defines a region between the background noise and the signal where the integration is ignored. It is useful for removing the slope from the calculation. This middle region is not shown in the chart below.

Both pre-edge and inter-edge regions have its zero nanosecond reference in the same place: the starting point of a trigger (corresponds to the beginning of the chart). That's why selecting an inter-edge with a value less than the pre-edge makes the middle region disappear. Check the diagram below to have things more clear. The post-edge zero reference starts at the end of the pre-edge or the inter-edge, which one is greater.

The firmware doesn't work in terms of nanoseconds, but it uses clock ticks. A transformation must be done from clock ticks to nanoseconds and vice-versa. Also, the firmware doesn't accept odd values for tick counts. The first 28 bytes of the raw waveform sent by the firmware contains diagnostics data and not sampled data and have to be removed. All of these operations are described in the section "Waveform and integration windows".

Linear adjustment function for the sum of the signal

This step ends with the result for ARAW for the formula described before. This step is performed by the firmware, not the software.

In the first step, we selected integration windows for the background (the left region on the pictures above) and for the signal (the right region). What the firmware does is to integrate over the points in one window, integrate over the points in the other window and subtract them. Essentially, this corresponds to subtracting the background from the signal. I will call these two integrations as BKGND and SIGNAL. In this first and simple approach, we are just doing:

[1] ARAW = SIGNAL - BKGND

The system provides a better level of flexibility where you can define separate weights for each integration. Suppose that the Physicist selected a small region for the background and a big region for the signal. We would need to give more weight for the background in order to compensate for the smallest number of samples to integrate. So, we add the weight factors A0 and A1 to the formula:

[2] ARAW = SIGNAL*A1 - BKGND *A0

The last level of flexibility is an offset. This is for adjusting the non-normalized sum for, as an example, avoiding negative values for the sum. The final formula for the sum is:

[3] ARAW = SIGNAL*A1 - BKGND *A0 + OFFSET

This is the screen that can be used to set the parameters:

The weight parameters A0 and A1, and Offset are the first 3 boxes on the snapshot.

The table below connects registers, Asyn parameters, and PVs. Offset is called as the Coefficient I by the firmware.

The last section of this document showed how the integration windows are adjusted. An EPICS database takes care of changing A0 and A1 automatically when the windows' width is changed. So, this guarantees that if the background and signal are approximately horizontal lines, no change in the window's width will change the final result of ARAW. A deep detail about this database is described in Auto adjustment of weight factors A0 and A1 below.

Coefficients of the equation

Revisiting the equation defined some sections above, we already have described the origin of two components:

• TMIT: as described in Calculate the maximum current and signal sum, this component comes from the BPM IOC, through the FCOM network.
• ARAW: the calculation of this component was explained in the previous section.

The remaining coefficients are given directly by the Physicist and are PVs that write to and read directly from the FPGA registers. The screen that exposes these PVs is the same one shown in the previous section. This is the table connecting registers, Asyn parameters, and PVs:

Access a list with bunch length parameters aligned with the same timestamp

The Bunch Length IOC application provides 4 parameters for the BSA (Beam Synchronous Acquisition) system. A brief introduction for BSA can be found here: Beam Synchronous Acquisition (BSA).

These are the parameters based on the two pyrodetector sensors:

• AIMAX: the result of the equation for sensor A, the peak current measured at the sensor and normalized, that can be transformed into the bunch length.
• BIMAX: the result of the equation for sensor B, the peak current measured at the sensor and normalized, that can be transformed into the bunch length.
• ARAW: the integral of the signal of the sensor A, subtracted by the integral of the background noise.
• BRAW: the integral of the signal of the sensor B, subtracted by the integral of the background noise.
  

The first thing that the IOC needs to do is to create one BSA channel for each parameter. What triggers this action is the IOC shell function blenConfigure, on st.cmd. The first parameter of this function is the "Station Name" that is the beginning of the PV names of the IOC. For instance, in sector 21, st.cmd has:

# Area, position, and instrument names to be used in record names
epicsEnvSet("AREA", "LI24")
epicsEnvSet("POS", "886")
epicsEnvSet("INST", "BL21")


(...)
 
# blenConfigure parameters:
# 1 - Station name
# 2 - BSA stream name must be identical to definition in yaml file
# 3 - PV used to get TMIT from FCOM
# 4 - IP address and port to send TMIT information to ATCA
blenConfigure "BLEN:${AREA}:${POS}" "${BSA_STREAM_YAML_NAME}" "${TMIT_PV}" "${FPGA_IP}:${IP_PORT_TMIT}"

blenConfigure will, among other things, call the function BSA_CreateChannel, from BsaApi.h. Here are the relevant parts from blenBSA.h and blenBSA.cpp:

#define NUM_CHANNELS 4
static const char* const channelNames[NUM_CHANNELS] = {"AIMAX", "BIMAX", "ARAW", "BRAW"};
 
class BlenBSA {
private:
    BsaChannel *bsaChannels;
(...)
public:
(...)
    int createChannels(const char *stationName);
(...)
}

int BlenBSA::createChannels(const char *stationName) {
    char param_name[64];
    bsaChannels = new BsaChannel[NUM_CHANNELS];

    /* Create NUM_CHANNELS BSA channels with the name formed by the 
     * stationName that comes from the IOC shell, plus the channel
     * acronym defined by the constant array channelNames. */
    using namespace std;
    for (int i = 0; i < NUM_CHANNELS; i++) {
        sprintf(param_name, "%s:%s", stationName, channelNames[i]);
        bsaChannels[i] = BSA_CreateChannel(param_name);
    }
    return 0;
}

As explained in Calculate the maximum current and signal sum, there is a thread responsible to store the data to the BSA Core as soon as the data is ready. The firmware sends the data to the software using a CPSW stream channel. The name of the stream channel is defined in the YAML file and must be informed to the BSA thread by the second parameter of the st.cmd function blenConfigure (explained before). Every time the data is ready, the firmware sends it asynchronously and the software must be prepared to read and process it. The complete data structure of this stream can be checked here. For what is in use by the current IOC application, this is the struct that is mapped to the stream packet:

typedef struct {
    epicsTimeStamp  time;
    epicsUInt32     mod[6];
    epicsUInt32     edefAvgDoneMask;
    epicsUInt32     edefMinorMask;
    epicsUInt32     edefMajorMask;
    epicsUInt32     edefInitMask;
} timing_header_t;

typedef struct {
    epicsFloat32    sum;
    epicsFloat32    iMax;
    epicsFloat32    scaledTMIT;
    epicsUInt32     status0;
    epicsUInt32     status1;
    // Seven 32-bit words reserved for future use
    epicsUInt32     reservedWords[6];
} payload_t;

/* This structure represents the entire content of the BSA stream message that
 * comes from the firmware. If the message content is changed, this structure
 * must be updated */
typedef struct {
    // Stream header is not described at the specification.
    epicsUInt64     streamHeader;
    epicsUInt32     headerWord;
    timing_header_t timingHeader;
    payload_t       AMC0_Data;
    payload_t       AMC1_Data;
} stream_t;

If you retake the BSA thread task activity diagram, the data receive some processing and is mapped to another data structure to make it easier to extract data to the BSA core. This is the structure:

// Data to be sent to BSA channels
typedef struct bsaData_t {
    epicsTimeStamp timeStamp;
    double         aImax;
    double         aSum;
    BsaStat        aStat;
    BsaSevr        aSevr;
    double         bImax;
    double         bSum;
    BsaStat        bStat;
    BsaSevr        bSevr;
} bsaData_t;

Once this structure is filled, the last step is to send the data to the BSA channels:

// bsaData is of type bsaData_t
void BlenBSA::sendData(bsaData_t* bsaData) {
    BSA_StoreData(bsaChannels[0], bsaData->timeStamp, bsaData->aImax,
                  bsaData->aStat, bsaData->aSevr);
    BSA_StoreData(bsaChannels[1], bsaData->timeStamp, bsaData->bImax,
                  bsaData->bStat, bsaData->bSevr);
    BSA_StoreData(bsaChannels[2], bsaData->timeStamp, bsaData->aSum,
                  bsaData->aStat, bsaData->aSevr);
    BSA_StoreData(bsaChannels[3], bsaData->timeStamp, bsaData->bSum,
                  bsaData->bStat, bsaData->bSevr);
}

After this step, the internal arrays with timestamp aligned data are filled and ready to be accessed as PVs by the interested users. It was not mentioned before, but the BSA database should be loaded for the PVs to be made available. This loading is done in the file iocBoot/common/blenCommon.cmd, with a call to dbLoadRecords for each BSA channel:

# BSA records
dbLoadRecords("db/Bsa.db", "DEVICE=BLEN:${AREA}:${POS},ATRB=AIMAX, SINK_SIZE=1")
dbLoadRecords("db/Bsa.db", "DEVICE=BLEN:${AREA}:${POS},ATRB=BIMAX, SINK_SIZE=1")
dbLoadRecords("db/Bsa.db", "DEVICE=BLEN:${AREA}:${POS},ATRB=ARAW, SINK_SIZE=1")
dbLoadRecords("db/Bsa.db", "DEVICE=BLEN:${AREA}:${POS},ATRB=BRAW, SINK_SIZE=1")

Check diagnostic reports

The IOC application provides an IOC shell function called blenReport to help the IOC Maintainer to diagnose the system. Once in the IOC shell, blenReport will output:

The first half of the output deals with statistics related to FCOM. It is broken into two parts: Rx for received packets and Tx for transmitted packets. The numbers are cumulative since when the IOC was booted.

The second half deals with general data received and transmitted. The table below explains each of the shown parameters.

Read data arriving from the firmware

Sometimes the IOC Maintainer would like to check the content of the raw BSA message that the firmware is sending to the IOC. For this, the function blen_dumpBSAStream can be used. It accepts one parameter: the number of packets in the sequence that is going to be printed on the IOC shell. The content of the bytes for this function is described in this document. This is a partial output after this function finishes printing 2 packets.

Algorithm description

Waveform and integration windows

All the algorithm is processed in the EPICS database layer.

First thing, the waveform is delivered from the firmware to the software with the time axis measured in tick counts. We need to convert it to nanoseconds. The EPICS record responsible for this is in the weightFunctionXAxis.db database. The firmware informs half of its clock frequency in Hz for each of the AMC cards by using the registers /mmio/AppTop/AppCore/AmcGenericAdcDacCore[n]/AmcGenericAdcDacCtrl/AmcClkFreq, with n varying from 0 to 1 according to the AMC card. The registers are translated to Asyn parameters AMC0:AmcClkFreq and AMC1:AmcClkFreq by the dictionary blenMR.dict. The CALC field converts from Hz to nanoseconds, remembering that the firmware gives the number related to half of the frequency.

Next step is to calculate the time related to each section of the integration window: pre-edge, inter-edges, and post-edge. Basically, we multiply the size of the window section in ticks by the period of the tick in nanoseconds, as calculated before. The inter-edges is different because the firmware delivers half of the size. That's why we have B*2 in the calculation. The register names and Asyn parameter names for each of the AMC cards are in the table below.

The next step is to provide PVs for the user to change the pre, inter, and post edges windows in nanoseconds and convert them back to tick counts. The user changes the PVs TIME_PRE, TIME_MID, and TIME_POS and calc records convert it to tick counts, writing the value to the firmware through the records NumSamp0, NumSamp1, and TrigDelay1.

To convert from nanoseconds to tick counts we need the AMC clock frequency, as before, multiply by the number of nanoseconds and divide by 1 billion, as the clock frequency is taken as Hz. Also, the clock frequency informed by the firmware is half of what is real, so we need to multiply everything by 2. This is what gives us the equation 2*A*B/1000000000. This number is rounded with the FLOOR operator, that can generate an odd number. The firmware expects only even numbers, so we need to guarantee that the LSB is always zero by doing an AND with the result and ((~0)-1).  ~0 gives all "1" for all the bits. Subtracting 1 gives us all bits "1" besides the last one.

The inter-edges window must be divided by 2 per firmware requirement. Also, we are guaranteeing that the size of the inter-edge window is never lower than the pre-edge window for the reason explained before in Adjustment of the integration windows. That's the function of the record TIME_MIDC.

The firmware registers for these records are the same as described before, as you can see by the OUT field with the Asyn parameter name.

The last feature provided by the weightFunctionXAxis.db is to generate an array for the X-axis of the chart, containing the translation to nanosecond for each tick count. This is held by the aSub record BLEN:$(AREA):$(POS):$(INST)_WF_TIME, using the function calcTimeArray() available in the file asubWeightXAxis.c. The input for the record is the PV AmcClkFreq already described before.

Auto adjustment of weight factors A0 and A1

As seen before, moving the sliders to change the width of the integration windows changes the PVs NumSamp0 and NumSamp1, that, in sequence, send the value to the FPGA registers. This section is the continuation of this story.

Once NumSamp0 and NumSamp 1 changes, the coefficients A0 and A1 must change proportionally. The database diagram below shows that the records A0_NS0 and A1_NS1 will process whenever NumSamp0 and NumSamp1 are changed. CALC_Const_A0 and CALC_Const_A1 will not process because their INPA fields are not set with the CP directive. Once A0_NS0 process, it will divide A0_CONST by NumSamp0 and write the result to CoefA0, changing the value of the coefficient in the FPGA register. The calculation record has protection against division by zero using "A=0? B : B/A". The same process happens for CoefA1.

So, keeping it simple, whenever the user changes the width of the integration window, the A coefficient will change proportionally.

Now it comes to the reason for the existence of Reset_A0 and Reset_A1. These are tools that a user can use to force the values of CoefA0 and CoefA1. Once a number is written to Reset_A0, CALC_Const_A0 will make the inverted calculation compared to A0_NS0 and store it at A0_CONST. All these record chain processes and force A0_NS0 to process, undoing what CALC_Const_A0 did before. The final result is that the exact number that the user wrote to Reset_A0 will end up in Coef_A0. The same rules apply to the CoefA1 record chain.

BLEN MR Cheat Sheets

Database files

Files in blenApp/Db

Source code files

iocBoot files

IOC shell functions

Starting PyDM screens