Overview

We have an IOC application that controls the Scandinova modulator. We inherited the IOC application from Berkeley. At that time, the app was entirely Python scripts and EPICS databases (i.e., no C or C++ code). After inheriting it, the Python was removed in favor of C++.

Scandinova is a company in Sweden. They make modulators. UED at SLAC uses a Scandinova modulator. UED uses Berkeley's IOC application as a read-only IOC. It is not capable of changing any values on the modulator. This is something we have changed when porting to C++. Our ported IOC application can change values on the modulator. Both UED and Berkeley have shown interest in using the C++ IOC application.

Links

Application

Modbus

Scandinova exports control of the modulator over various interfaces. SLAC selected the modbus interface. All status and values are read via modbus. All controls are written via modbus. There are many values that can be read, and very few that can be written. For the values that can be written, there are read-back registers to confirm the value was taken by the modulator.

Modbus is not a well-typed interface. All values are 16-bit, but the endianness is not defined, and if values larger than 16-bits are used, the order of the words (16-bit values) is also not defined. Further, there are structures defined by Scandinova. These structures contain variables of various types including timestamps.

Debug

I found the modbus interface difficult to understand. Using only the EPICS modbus module, I wasn't sure if the issues I was encountering were with the modulator, the modbus interface, or the EPICS modbus module. In the end all of them had issues or at least things that were confusing to me. To gain an initial foothold, I had to write code outside EPICS to pull the raw data from modbus. Only then could I understand the data, and what I needed EPICS to do with it. The scripts are checked in to the IOC app under the "test" directory.

modbusrd.sh - read a value over modbus
modbuswr.sh - write a value to modbus
modbuswrrd.sh - read a value from a modbus write address

The scripts aren't perfect. They assume the modbus interface is at a specific IP address. In other words, you can't enter the modbus IP address as a parameter to these scripts. No need to do so, yet.

XML strings

Scandinova exports modulator status by integers, and the integers represent strings. The integers are used for a look-up table that is generated from an XML document that Scandinova provides. The XML document represents the features of the modulator and is constructed by Scandinova per customer. Scandinova also provides a modulator simulator and the simulator has its own XML document.

The XML document is parsed at run-time by the IOC application. The IOC application exports a function called from within an IOC start-up script:

parse_resources_xml(char *file)

The file is the XML file provided by Scandinova. The XML file that came with the simulator contained a couple XML syntax errors. Rather than work around the syntax errors in code, I manually fixed the errors in the XML document. I don't know if future XML documents will contain XML syntax errors. I raised the issue with Scandinova.

The XML document must have these sections for the IOC app to run properly:

State
Warning
Interlock
Error
Param
Message

Value types

Some of the warnings and errors have associated values. The values are all encoded the same in modbus and must be transformed in code. These are the value types as defined by Scandinova and their C type as defined by me:

Real - float (single precision floating point; Scandinova calls this a "single")
Bool - int32_t
Int - int32_t
Uint - uint32_t
Word - int32_t
Dint - int32_t
Udint - uint32_t
Dword - uint32_t

Events

Scandinova defines an event to have this structure in modbus:

typedef struct __attribute__ (packed) {
    uint16_t increment;
    uint16_t type;
    uint64_t epoch_raw;
    uint32_t trigger;
    uint16_t matrix_index;
    uint16_t text_number;
    uint16_t data_type;
    uint32_t data;
} event_struct_t;

Scandinova defines two structures of the above type in modbus. It calls them "first interlock event since previous reset" and "actual state of the modulator". They would be better called "current interlock" and "current event". An interlock prevents state progression but does not prevent all operations, therefore it is necessary to have an interlock structure so that we don't lose information about why the state progression was halted.

Note that the current event structure may be updated quickly in succession, within milliseconds. Don't try to capture events here. You will not see all of them as they pass too quickly. Use this structure only for human eyeball display purposes. Scandinova uses separate modbus registers that in turn we use to capture all events.

To capture all events, Scandinova uses arrays of modbus registers. In general, each field of an event (as seen above in event_struct_t) has an associated array of size 50. In them, we see information about the previous 50 events, but we must piece the events together from each modbus array. The arrays update in order, but do not update atomically. In other words, event 10 will post in its entirety after 9 and before 11, but you must wait for 10 to post completely (not assume all fields are populated). The arrays are circular - event 200 is found at index 0, event 201 is found at index 1, etc. So if we are not careful, we will read part of event 200 and part of event 150 and patch them together to construct a bogus event. There is no deterministic way to know when an event has posted completely, other than the "next" event begins to post. We know when an event begins to post by a modbus register that exports the count of events. Because the next event may not post for a long time (or ever), we have a 1-second timer that waits for event fields to stop changing.

The EPICS application uses the EPICS data type aSub to be notified when a modbus register has been modified by the modulator. We are notified when any the event fields are modified. In the IOC application, there is a thread that waits for the notifications to arrive. In general, the notifications for a single event all arrive within one second. So we have a 1-second timer that begins a countdown each time an update arrives. When the timer expires, all events are new since the last expiration are read over modbus. There can be any number of events unread (up to 50, and the IOC application is ok with this possibility), but in general it is 1 but sometimes up to 4.

The events are saved in memory and also to a text file. The IOC application exports a function called from within an IOC start-up script:

set_event_output_file(char *file)

The file is the path and name of the file to which events are saved. This is a text file. The text file can grow infinitely long. In memory, the IOC application follows the modbus data, and there are only 50 events.

Operation

The modulator has four states:

Off
Standby
HV (high voltage)
Trig (triggered)

The user selects the desired state via modbus. In practice, the operator will select a state from the EPICS GUI. Either the modulator enters the desired state or it enters an interlock. Interlocks can happen for many reasons. Scandinova calls the states either a master state or a slave state. The master states (listed above) are catch-all states. The various subsystems of the modulator have their own states and these are what Scandinova calls the slave states. Each subsystem is asked to enter a particular state and it either says yes or no and marks its status as such. Only when all the subsystems are happy does the master state change too, and this is what is reflected on the GUI.

The modulator comes with a GUI. The GUI runs on a Windows machine in the cabinet provided by Scandinova. Our intention is to not use this GUI after commissioning of the modulator. The modulator exposes a writeable modbus variable that puts the modulator into a remote control mode. When the modulator is in remote control, the local GUI will not allow writing values. When the modulator is in local control, the writeable modbus values are not writeable. While in remote control, the modulator requires that a watchdog value be incremented each second. If the watchdog is not incremented, the modulator defaults back to local control, and someone would have to walk back to the cabinet to put the modulator back in remote control.

Inputs to the modulator

These are the inputs to the modulator from the IOC app via modbus. They are listed as "OutputRegisters" in the Scandinova modbus spec.

watchdog - must increment every second to keep the remote connection operational
modulator target state - off, standby, high voltage, trigger
command - reset, hard reset
voltage set - capacitor charge of power supplies
pulse width set - pulse width of switch units
waveform sequence - define a waveform (? - need clarification from Scandinova)

Outputs from the modulator

These are the outputs from the modulator to the IOC app via modbus. They are listed as "InputRegisters" in the Scandinova modbus spec.

modbus protocol identification
modbus protocol revision number
watchdog - reflects the value given as an input to the modulator
modulator state - reflects the current state of the modulator (i.e., not necessarily the target state)
status bits - the reason(s) for the current interlock, if any
access level - remote, operator, admin, manufacturer (Scandinova)
warm-up timer (s) - remaining time for the filament to warm up sufficiently
pulse repetition frequency
checksum sequence - if we want to verify the software that the modulator is executing; we don't plan to implement this for the first release of the IOC
checksum versions - if we want to verify the software versions that the modulator is executing; we don't plan to implement this for the first release of the IOC
target state - reflects the value given as an input to the modulator (i.e., not necessarily the current state)
voltage (V) - reflects the value given as an input to the modulator
pulse width (us) - reflects the value given as an input to the modulator
voltage of capacitor charging power supplies (V)
current of filament power supply (A)
voltage of filament power supply (V)
current of ion pump controller (A)
voltage of ion pump controller (V)
pressure of ion pump controller (Pa)
current of solenoid power supply (A)
voltage of solenoid power supply (V)
pulse current amplitude (A)
pulse voltage amplitude (kV)
pulse width at 50% amplitude (us)
average power (kW)
oil temperature (degrees C)
oil level (mm)
forward temperature (degrees C)
return temperature (degrees C)
ambient temperature (degrees C)
solenoid temperature (degrees C)
water flow rate (6) (l/m)
collector power loss (kW)
RF out forward power (dBm)
RF out reflected power (dBm)
RF out VSWR
RF out pulse width (us)
events (50)
interlock event
current event
status bits - subsystem statuses (ok, warning, interlock)
waveform sequence - status of waveform upload
waveform information - timestamp, id, samples, waveform (1000)

Process Variables

For the most part, modbus registers are represented by process variables. We have a few extra process variables for:

tracking when a modbus register changes values
incrementing the watchdog
converting modbus integers to the strings they represent.

Event Change Tracking

This is a process variable for tracking the change of the interlock event:

record (aSub,"$(P):InterlockEventStructBridge") {
    field(SNAM,"handle_interlock_event_struct_modify")

    field(INPA,"$(P):InterlockEventStruct CP")
    field(FTA,"LONG")
    field(NOA,"13")

    field(OUTA,"$(P):InterlockEventTimeStr PP")
    field(FTVA,"STRING")
    field(NOVA,"64")

    field(OUTB,"$(P):InterlockEventTypeStr PP")
    field(FTVB,"STRING")
    field(NOVB,"64")

    field(OUTC,"$(P):InterlockEventTrigger PP")
    field(FTVC,"LONG")
    field(NOVC,"1")

    field(OUTD,"$(P):InterlockEventSubsystemStr PP")
    field(FTVD,"STRING")
    field(NOVD,"64")

    field(OUTE,"$(P):InterlockEventTextStr PP")
    field(FTVE,"STRING")
    field(NOVE,"64")

    field(OUTF,"$(P):InterlockEventTextStr2 PP")
    field(FTVF,"STRING")
    field(NOVF,"64")

    field(OUTG,"$(P):InterlockEventUnitStr PP")
    field(FTVG,"STRING")
    field(NOVG,"64")

    field(OUTH,"$(P):InterlockEventDataStr PP")
    field(FTVH,"STRING")
    field(NOVH,"64")
}

Here's the code that handles the change notification:

typedef struct {
    int increment;
    time_t epoch;
    char timestamp[64];
    int trigger;
    int type;               // index for Strings and event types
    char type_str[32];
    int text_number;        // index for Strings[type]
    char text_str[64];      // from Strings[type][text_number]
    int matrix_index;       // index for MatrixItems
    char subsystem_str[32]; // from MatrixItems[matrix_index].Name
    char units_str[8];      // from MatrixItems[matrix_index].Unit
    int data_type;          // index for data types
    char data_type_str[16];
    uint32_t data;
    char data_str[32];
} event_info_t;

static long
handle_interlock_event_struct_modify(aSubRecord *prec)
{
    event_info_t event_info;

    modbus_to_event_info((uint32_t *) prec->a, &event_info);

    memset(prec->vala, 0, prec->nova);
    memset(prec->valb, 0, prec->novb);
    memset(prec->valc, 0, sizeof(long) * prec->novc);
    memset(prec->vald, 0, prec->novd);
    memset(prec->vale, 0, prec->nove);
    memset(prec->valf, 0, prec->novf);
    memset(prec->valg, 0, prec->novg);
    memset(prec->valh, 0, prec->novh);

    strncpy((char *) prec->vala, event_info.timestamp, prec->nova - 1); 
    strncpy((char *) prec->valb, event_info.type_str, prec->novb - 1); 
    *(long *) prec->valc = event_info.trigger;
    strncpy((char *) prec->vald, event_info.subsystem_str, prec->novd - 1); 
    strncpy((char *) prec->vale, event_info.text_str, prec->nove - 1); 
    strncpy((char *) prec->valf, event_info.subsystem_str, prec->novf - 1); 
    strncpy((char *) prec->valg, event_info.units_str, prec->novg - 1); 
    strncpy((char *) prec->valh, event_info.data_str, prec->novh - 1); 

#if 0
    printf("interlock ------------------------------------\n");
    printf("increment:  %d\n",                     event_info.increment);
    printf("trigger id: %d\n",                     event_info.trigger);
    printf("timestamp:  %s\n",                     event_info.timestamp);
    printf("type:       \"%s\" (%d)\n",            event_info.type_str, event_info.type);
    printf("subsystem:  \"%s\" (%d)\n",            event_info.subsystem_str, event_info.matrix_index);
    printf("log text:   \"%s\" (%d/%d)\n",         event_info.text_str, event_info.type, event_info.text_number
    printf("data type:  \"%s\" (%d)\n",            event_info.data_type_str, event_info.data_type);
    printf("data:       \"%s\" \"%s\" (0x%08x)\n", event_info.data_str, event_info.units_str, event_info.data);
    printf("\n");
#endif

    return 0;
}

Mostly we are converting integers to strings. The strings are exported as process variables.

Some interlock events may be skipped. In other words, the IOC application is not notified of the change until the "next" change has occurred. This is ok because we use this mechanism for displaying to human eyeballs only.

The current event has the same mechanism as the interlock event. The only difference is the change is notified via a different function (handle_current_event_struct_modify).

Event Change Tracking For Logging

Logging (to a file) is more involved. We don't want to miss any events.

record (aSub,"$(P):EventBridge") {
    field(SNAM,"handle_event_log_modify")

    field(INPA,"$(P):EventIndex CP")
    field(FTA,"LONG")
    field(NOA,"1")

    field(INPB,"$(P):EventIncrement CP")
    field(FTB,"LONG")
    field(NOB,"50")

    field(INPC,"$(P):EventTime1 CP")
    field(FTC,"LONG")
    field(NOC,"50")

    field(INPD,"$(P):EventTime2 CP")
    field(FTD,"LONG")
    field(NOD,"50")

    field(INPE,"$(P):EventType CP")
    field(FTE,"LONG")
    field(NOE,"50")

    field(INPF,"$(P):EventTrigger CP")
    field(FTF,"LONG")
    field(NOF,"50")

    field(INPG,"$(P):EventCause CP")
    field(FTG,"LONG")
    field(NOG,"50")

    field(INPH,"$(P):EventTextNum CP")
    field(FTH,"LONG")
    field(NOH,"50")

    field(INPI,"$(P):EventDataType CP")
    field(FTI,"LONG")
    field(NOI,"50")

    field(INPJ,"$(P):EventData CP")
    field(FTJ,"LONG")
    field(NOJ,"50")
}

Here's the code that handles the notification:

static long
handle_event_log_modify(aSubRecord *prec)
{
    int event_index = 0;
    uint32_t upper_32 = 0;
    uint32_t lower_32 = 0;
    event_info_t *event_info = NULL;
    event_struct_t event_struct;

    memset(&event_struct, 0, sizeof(event_struct));

    pthread_mutex_lock(&log_mutex);

    current_event_index = *(int *) prec->a;
    assert(current_event_index >= 0); 
    assert(current_event_index < 50);

    for (event_index = 0; event_index < 50; event_index++) {
        event_info = &event_infos[event_index];

        // event timestamp

        if (event_index < 25) {
            upper_32 = ((uint32_t *) prec->c)[event_index * 2 + 1]; 
            lower_32 = ((uint32_t *) prec->c)[event_index * 2]; 
        } else {
            upper_32 = ((uint32_t *) prec->d)[(event_index - 25) * 2 + 1]; 
            lower_32 = ((uint32_t *) prec->d)[(event_index - 25) * 2]; 
        }

        event_struct.increment = (uint16_t) ((uint32_t *) prec->b)[event_index];
        event_struct.epoch_raw = ((uint64_t) upper_32 << 32) | lower_32;
        event_struct.type = (uint16_t) ((uint32_t *) prec->e)[event_index];
        event_struct.trigger = ((uint32_t *) prec->f)[event_index];
        event_struct.matrix_index = (uint16_t) ((uint32_t *) prec->g)[event_index];
        event_struct.text_number = (uint16_t) ((uint32_t *) prec->h)[event_index];
        event_struct.data_type = (uint16_t) ((uint32_t *) prec->i)[event_index];
        event_struct.data = ((uint32_t *) prec->j)[event_index];

        event_struct_to_event_info(&event_struct, event_info);
#if 0
        printf("increment:  %d\n",                     event_info→increment);
        printf("trigger id: %d\n",                     event_info->trigger);
        printf("timestamp:  %s\n",                     event_info->timestamp);
        printf("type:       \"%s\" (%d)\n",            event_info->type_str, event_info->type);
        printf("subsystem:  \"%s\" (%d)\n",            event_info->subsystem_str, event_info->matrix_index);
        printf("log text:   \"%s\" (%d/%d)\n",         event_info->text_str, event_info->type, event_info->text
        printf("data type:  \"%s\" (%d)\n",            event_info->data_type_str, event_info->data_type);
        printf("data:       \"%s\" \"%s\" (0x%08x)\n", event_info->data_str, event_info->units_str, event_info-
        printf("\n");
#endif
    }

    // the time at which the event data is done being updated (clean)
    clock_gettime(CLOCK_REALTIME, &next_clean_time);
    next_clean_time.tv_sec++;

    pthread_mutex_unlock(&log_mutex);

    return 0;
}

In that function, each time we are notified, we emplace the changed data into our in-memory representation of the 50 events. We also restart our 1-second timer before it expires. In a separate thread, we write the new events to a file when the 1-second timer has expired. This way, we are sure all events have fully posted before writing them to the log file.

Watchdog

We must write to a modbus register every second, incrementing the value by 1. We are able to accomplish this with only record definitions.

record (calcout,"$(P):CalcWatchdog") {
    field(FLNK,"$(P):CommWatchdog.PROC")
    field(SCAN,"1 second")
    field(INPA,"$(P):CalcWatchdog.VAL NPP NMS")
    field(CALC,"(A+1)%0x10000")
    field(OUT,"$(P):CommWatchdog")
    field(OOPT,"Every Time")
    field(DOPT,"Use CALC")
}

file ao.template
{
    pattern
    { P,      R,                 PORT,        OFFSET, BITS, EGUL, EGUF,  PREC }
    { "${P}", ":CommWatchdog",   "write0",    0,      16,   0,    65535, 0    }   
}

Digitized Waveform

The EPICS modbus module has a restriction on the number of bytes it transfers in any single transaction. The limit is 200 bytes. The digitized waveform register in the modulator is 512 words = 1024 bytes. That includes:

8 bytes for the timestamp
8 bytes for the pulse id
2 bytes for the sample count
1006 bytes for the waveform

We split the waveform into six PVs (Waveform1-6).

file intarray_in.template
{
    pattern
    { P,      R,                       PORT,        OFFSET, NELM, SCAN       }
    { "${P}", ":WaveformTime",         "read3001",  0,         4, "I/O Intr" }
    { "${P}", ":Waveform1",            "read3010",  0,       100, "I/O Intr" }
    { "${P}", ":Waveform2",            "read3110",  0,       100, "I/O Intr" }
    { "${P}", ":Waveform3",            "read3210",  0,       100, "I/O Intr" }
    { "${P}", ":Waveform4",            "read3310",  0,       100, "I/O Intr" }
    { "${P}", ":Waveform5",            "read3410",  0,       100, "I/O Intr" }
    { "${P}", ":Waveform6",            "read3510",  0,         3, "I/O Intr" }
}

file ai.template
{
    pattern
    { P,      R,              PORT,       OFFSET, BITS, EGUL, EGUF,  PREC, SCAN       }
    { "${P}", ":SampleCount", "read3009", 0,      16,   0,    65535, 0,    "I/O Intr" }
}

file int64in.template
{
    pattern
    { P,      R,          PORT,       OFFSET, DATA_TYPE, SCAN       }
    { "${P}", ":PulseId", "read3005", 0,      UINT64_LE, "I/O Intr" }
}

The timestamp (waveform time) requires processing the same as the log entries. We use an EPICS aSub record to be notified when the value changes, and when it does we process the change in code and from that populate the WaveformTimeStr PV with a human-readable time string.

record (stringin,"$(P):WaveformTimeStr") {
    field(DESC, "Timestamp of the selected waveform")
}

record (aSub,"$(P):WaveformTimeBridge") {
    field(SNAM, "handle_waveform_timestamp_modify")

    field(INPA, "$(P):WaveformTime CP")
    field(FTA,  "LONG")
    field(NOA,  "4")

    field(OUTA, "$(P):WaveformTimeStr PP")
    field(FTVA, "STRING")
    field(NOVA, "64")
}

Space shortcuts

Confluence Content

Child pages

Overview

Links

Application

Modbus

Debug

XML strings

Value types

Events

Operation

Inputs to the modulator

Outputs from the modulator

Process Variables

Event Change Tracking

Event Change Tracking For Logging

Watchdog

Digitized Waveform

Space shortcuts

Confluence Content

Child pages

Scandinova Modulator

Overview

Links

Application

Modbus

Debug

XML strings

Value types

Events

Operation

Inputs to the modulator

Outputs from the modulator

Process Variables

Event Change Tracking

Event Change Tracking For Logging

Watchdog

Digitized Waveform