We have an IOC application that controls the Scandinova modulator. We inherited the IOC application from Berkeley. At that time, the app was entirely Python scripts and EPICS databases (i.e., no C or C++ code). After inheriting it, the Python was removed in favor of C++.
Scandinova is a company in Sweden. They make modulators. UED at SLAC uses a Scandinova modulator. UED uses Berkeley's IOC application as a read-only IOC. It is not capable of changing any values on the modulator. This is something we have changed when porting to C++. Our ported IOC application can change values on the modulator. Both UED and Berkeley have shown interest in using the C++ IOC application.
Scandinova exports control of the modulator over various interfaces. SLAC selected the modbus interface. All status and values are read via modbus. All controls are written via modbus. There are many values that can be read, and very few that can be written. For the values that can be written, there are read-back registers to confirm the value was taken by the modulator.
Modbus is not a well-typed interface. All values are 16-bit, but the endianness is not defined, and if values larger than 16-bits are used, the order of the words (16-bit values) is also not defined. Further, there are structures defined by Scandinova. These structures contain variables of various types including timestamps.
I found the modbus interface difficult to understand. Using only the EPICS modbus module, I wasn't sure if the issues I was encountering were with the modulator, the modbus interface, or the EPICS modbus module. In the end all of them had issues or at least things that were confusing to me. To gain an initial foothold, I had to write code outside EPICS to pull the raw data from modbus. Only then could I understand the data, and what I needed EPICS to do with it. The scripts are checked in to the IOC app under the "test" directory.
The scripts aren't perfect. They assume the modbus interface is at a specific IP address. In other words, you can't enter the modbus IP address as a parameter to these scripts. No need to do so, yet.
Scandinova exports modulator status by integers, and the integers represent strings. The integers are used for a look-up table that is generated from an XML document that Scandinova provides. The XML document represents the features of the modulator and is constructed by Scandinova per customer. Scandinova also provides a modulator simulator and the simulator has its own XML document.
The XML document is parsed at run-time by the IOC application. The IOC application exports a function called from within an IOC start-up script:
The file is the XML file provided by Scandinova. The XML file that came with the simulator contained a couple XML syntax errors. Rather than work around the syntax errors in code, I manually fixed the errors in the XML document. I don't know if future XML documents will contain XML syntax errors. I raised the issue with Scandinova.
The XML document must have these sections for the IOC app to run properly:
Some of the warnings and errors have associated values. The values are all encoded the same in modbus and must be transformed in code. These are the value types as defined by Scandinova and their C type as defined by me:
Scandinova defines an event to have this structure in modbus:
typedef struct __attribute__ (packed) { uint16_t increment; uint16_t type; uint64_t epoch_raw; uint32_t trigger; uint16_t matrix_index; uint16_t text_number; uint16_t data_type; uint32_t data; } event_struct_t;
Scandinova defines two structures of the above type in modbus. It calls them "first interlock event since previous reset" and "actual state of the modulator". They would be better called "current interlock" and "current event". An interlock prevents state progression but does not prevent all operations, therefore it is necessary to have an interlock structure so that we don't lose information about why the state progression was halted.
Note that the current event structure may be updated quickly in succession, within milliseconds. Don't try to capture events here. You will not see all of them as they pass too quickly. Use this structure only for human eyeball display purposes. Scandinova uses separate modbus registers that in turn we use to capture all events.
To capture all events, Scandinova uses arrays of modbus registers. In general, each field of an event (as seen above in event_struct_t) has an associated array of size 50. In them, we see information about the previous 50 events, but we must piece the events together from each modbus array. The arrays update in order, but do not update atomically. In other words, event 10 will post in its entirety after 9 and before 11, but you must wait for 10 to post completely (not assume all fields are populated). The arrays are circular - event 200 is found at index 0, event 201 is found at index 1, etc. So if we are not careful, we will read part of event 200 and part of event 150 and patch them together to construct a bogus event. There is no deterministic way to know when an event has posted completely, other than the "next" event begins to post. We know when an event begins to post by a modbus register that exports the count of events. Because the next event may not post for a long time (or ever), we have a 1-second timer that waits for event fields to stop changing.
The EPICS application uses the EPICS data type aSub to be notified when a modbus register has been modified by the modulator. We are notified when any the event fields are modified. In the IOC application, there is a thread that waits for the notifications to arrive. In general, the notifications for a single event all arrive within one second. So we have a 1-second timer that begins a countdown each time an update arrives. When the timer expires, all events are new since the last expiration are read over modbus. There can be any number of events unread (up to 50, and the IOC application is ok with this possibility), but in general it is 1 but sometimes up to 4.
The events are saved in memory and also to a text file. The IOC application exports a function called from within an IOC start-up script:
The file is the path and name of the file to which events are saved. This is a text file. The text file can grow infinitely long. In memory, the IOC application follows the modbus data, and there are only 50 events.
The modulator has four states:
The user selects the desired state via modbus. In practice, the operator will select a state from the EPICS GUI. Either the modulator enters the desired state or it enters an interlock. Interlocks can happen for many reasons. Scandinova calls the states either a master state or a slave state. The master states (listed above) are catch-all states. The various subsystems of the modulator have their own states and these are what Scandinova calls the slave states. Each subsystem is asked to enter a particular state and it either says yes or no and marks its status as such. Only when all the subsystems are happy does the master state change too, and this is what is reflected on the GUI.
The modulator comes with a GUI. The GUI runs on a Windows machine in the cabinet provided by Scandinova. Our intention is to not use this GUI after commissioning of the modulator. The modulator exposes a writeable modbus variable that puts the modulator into a remote control mode. When the modulator is in remote control, the local GUI will not allow writing values. When the modulator is in local control, the writeable modbus values are not writeable. While in remote control, the modulator requires that a watchdog value be incremented each second. If the watchdog is not incremented, the modulator defaults back to local control, and someone would have to walk back to the cabinet to put the modulator back in remote control.
These are the inputs to the modulator from the IOC app via modbus. They are listed as "OutputRegisters" in the Scandinova modbus spec.
These are the outputs from the modulator to the IOC app via modbus. They are listed as "InputRegisters" in the Scandinova modbus spec.
For the most part, modbus registers are represented by process variables. We have a few extra process variables for:
This is a process variable for tracking the change of the interlock event:
record (aSub,"$(P):InterlockEventStructBridge") { field(SNAM,"handle_interlock_event_struct_modify") field(INPA,"$(P):InterlockEventStruct CP") field(FTA,"LONG") field(NOA,"13") field(OUTA,"$(P):InterlockEventTimeStr PP") field(FTVA,"STRING") field(NOVA,"64") field(OUTB,"$(P):InterlockEventTypeStr PP") field(FTVB,"STRING") field(NOVB,"64") field(OUTC,"$(P):InterlockEventTrigger PP") field(FTVC,"LONG") field(NOVC,"1") field(OUTD,"$(P):InterlockEventSubsystemStr PP") field(FTVD,"STRING") field(NOVD,"64") field(OUTE,"$(P):InterlockEventTextStr PP") field(FTVE,"STRING") field(NOVE,"64") field(OUTF,"$(P):InterlockEventTextStr2 PP") field(FTVF,"STRING") field(NOVF,"64") field(OUTG,"$(P):InterlockEventUnitStr PP") field(FTVG,"STRING") field(NOVG,"64") field(OUTH,"$(P):InterlockEventDataStr PP") field(FTVH,"STRING") field(NOVH,"64") }
Here's the code that handles the change notification:
typedef struct { int increment; time_t epoch; char timestamp[64]; int trigger; int type; // index for Strings and event types char type_str[32]; int text_number; // index for Strings[type] char text_str[64]; // from Strings[type][text_number] int matrix_index; // index for MatrixItems char subsystem_str[32]; // from MatrixItems[matrix_index].Name char units_str[8]; // from MatrixItems[matrix_index].Unit int data_type; // index for data types char data_type_str[16]; uint32_t data; char data_str[32]; } event_info_t;
static long handle_interlock_event_struct_modify(aSubRecord *prec) { event_info_t event_info; modbus_to_event_info((uint32_t *) prec->a, &event_info); memset(prec->vala, 0, prec->nova); memset(prec->valb, 0, prec->novb); memset(prec->valc, 0, sizeof(long) * prec->novc); memset(prec->vald, 0, prec->novd); memset(prec->vale, 0, prec->nove); memset(prec->valf, 0, prec->novf); memset(prec->valg, 0, prec->novg); memset(prec->valh, 0, prec->novh); strncpy((char *) prec->vala, event_info.timestamp, prec->nova - 1); strncpy((char *) prec->valb, event_info.type_str, prec->novb - 1); *(long *) prec->valc = event_info.trigger; strncpy((char *) prec->vald, event_info.subsystem_str, prec->novd - 1); strncpy((char *) prec->vale, event_info.text_str, prec->nove - 1); strncpy((char *) prec->valf, event_info.subsystem_str, prec->novf - 1); strncpy((char *) prec->valg, event_info.units_str, prec->novg - 1); strncpy((char *) prec->valh, event_info.data_str, prec->novh - 1); #if 0 printf("interlock ------------------------------------\n"); printf("increment: %d\n", event_info.increment); printf("trigger id: %d\n", event_info.trigger); printf("timestamp: %s\n", event_info.timestamp); printf("type: \"%s\" (%d)\n", event_info.type_str, event_info.type); printf("subsystem: \"%s\" (%d)\n", event_info.subsystem_str, event_info.matrix_index); printf("log text: \"%s\" (%d/%d)\n", event_info.text_str, event_info.type, event_info.text_number printf("data type: \"%s\" (%d)\n", event_info.data_type_str, event_info.data_type); printf("data: \"%s\" \"%s\" (0x%08x)\n", event_info.data_str, event_info.units_str, event_info.data); printf("\n"); #endif return 0; }
Mostly we are converting integers to strings. The strings are exported as process variables.
Some interlock events may be skipped. In other words, the IOC application is not notified of the change until the "next" change has occurred. This is ok because we use this mechanism for displaying to human eyeballs only.
The current event has the same mechanism as the interlock event. The only difference is the change is notified via a different function (handle_current_event_struct_modify).
Logging (to a file) is more involved. We don't want to miss any events.
record (aSub,"$(P):EventBridge") { field(SNAM,"handle_event_log_modify") field(INPA,"$(P):EventIndex CP") field(FTA,"LONG") field(NOA,"1") field(INPB,"$(P):EventIncrement CP") field(FTB,"LONG") field(NOB,"50") field(INPC,"$(P):EventTime1 CP") field(FTC,"LONG") field(NOC,"50") field(INPD,"$(P):EventTime2 CP") field(FTD,"LONG") field(NOD,"50") field(INPE,"$(P):EventType CP") field(FTE,"LONG") field(NOE,"50") field(INPF,"$(P):EventTrigger CP") field(FTF,"LONG") field(NOF,"50") field(INPG,"$(P):EventCause CP") field(FTG,"LONG") field(NOG,"50") field(INPH,"$(P):EventTextNum CP") field(FTH,"LONG") field(NOH,"50") field(INPI,"$(P):EventDataType CP") field(FTI,"LONG") field(NOI,"50") field(INPJ,"$(P):EventData CP") field(FTJ,"LONG") field(NOJ,"50") }
Here's the code that handles the notification:
static long handle_event_log_modify(aSubRecord *prec) { int event_index = 0; uint32_t upper_32 = 0; uint32_t lower_32 = 0; event_info_t *event_info = NULL; event_struct_t event_struct; memset(&event_struct, 0, sizeof(event_struct)); pthread_mutex_lock(&log_mutex); current_event_index = *(int *) prec->a; assert(current_event_index >= 0); assert(current_event_index < 50); for (event_index = 0; event_index < 50; event_index++) { event_info = &event_infos[event_index]; // event timestamp if (event_index < 25) { upper_32 = ((uint32_t *) prec->c)[event_index * 2 + 1]; lower_32 = ((uint32_t *) prec->c)[event_index * 2]; } else { upper_32 = ((uint32_t *) prec->d)[(event_index - 25) * 2 + 1]; lower_32 = ((uint32_t *) prec->d)[(event_index - 25) * 2]; } event_struct.increment = (uint16_t) ((uint32_t *) prec->b)[event_index]; event_struct.epoch_raw = ((uint64_t) upper_32 << 32) | lower_32; event_struct.type = (uint16_t) ((uint32_t *) prec->e)[event_index]; event_struct.trigger = ((uint32_t *) prec->f)[event_index]; event_struct.matrix_index = (uint16_t) ((uint32_t *) prec->g)[event_index]; event_struct.text_number = (uint16_t) ((uint32_t *) prec->h)[event_index]; event_struct.data_type = (uint16_t) ((uint32_t *) prec->i)[event_index]; event_struct.data = ((uint32_t *) prec->j)[event_index]; event_struct_to_event_info(&event_struct, event_info); #if 0 printf("increment: %d\n", event_info→increment); printf("trigger id: %d\n", event_info->trigger); printf("timestamp: %s\n", event_info->timestamp); printf("type: \"%s\" (%d)\n", event_info->type_str, event_info->type); printf("subsystem: \"%s\" (%d)\n", event_info->subsystem_str, event_info->matrix_index); printf("log text: \"%s\" (%d/%d)\n", event_info->text_str, event_info->type, event_info->text printf("data type: \"%s\" (%d)\n", event_info->data_type_str, event_info->data_type); printf("data: \"%s\" \"%s\" (0x%08x)\n", event_info->data_str, event_info->units_str, event_info- printf("\n"); #endif } // the time at which the event data is done being updated (clean) clock_gettime(CLOCK_REALTIME, &next_clean_time); next_clean_time.tv_sec++; pthread_mutex_unlock(&log_mutex); return 0; }
In that function, each time we are notified, we emplace the changed data into our in-memory representation of the 50 events. We also restart our 1-second timer before it expires. In a separate thread, we write the new events to a file when the 1-second timer has expired. This way, we are sure all events have fully posted before writing them to the log file.
We must write to a modbus register every second, incrementing the value by 1. We are able to accomplish this with only record definitions.
record (calcout,"$(P):CalcWatchdog") { field(FLNK,"$(P):CommWatchdog.PROC") field(SCAN,"1 second") field(INPA,"$(P):CalcWatchdog.VAL NPP NMS") field(CALC,"(A+1)%0x10000") field(OUT,"$(P):CommWatchdog") field(OOPT,"Every Time") field(DOPT,"Use CALC") }
file ao.template { pattern { P, R, PORT, OFFSET, BITS, EGUL, EGUF, PREC } { "${P}", ":CommWatchdog", "write0", 0, 16, 0, 65535, 0 } }
The EPICS modbus module has a restriction on the number of bytes it transfers in any single transaction. The limit is 200 bytes. The digitized waveform register in the modulator is 512 words = 1024 bytes. That includes:
We split the waveform into six PVs (Waveform1-6).
file intarray_in.template { pattern { P, R, PORT, OFFSET, NELM, SCAN } { "${P}", ":WaveformTime", "read3001", 0, 4, "I/O Intr" } { "${P}", ":Waveform1", "read3010", 0, 100, "I/O Intr" } { "${P}", ":Waveform2", "read3110", 0, 100, "I/O Intr" } { "${P}", ":Waveform3", "read3210", 0, 100, "I/O Intr" } { "${P}", ":Waveform4", "read3310", 0, 100, "I/O Intr" } { "${P}", ":Waveform5", "read3410", 0, 100, "I/O Intr" } { "${P}", ":Waveform6", "read3510", 0, 3, "I/O Intr" } } file ai.template { pattern { P, R, PORT, OFFSET, BITS, EGUL, EGUF, PREC, SCAN } { "${P}", ":SampleCount", "read3009", 0, 16, 0, 65535, 0, "I/O Intr" } } file int64in.template { pattern { P, R, PORT, OFFSET, DATA_TYPE, SCAN } { "${P}", ":PulseId", "read3005", 0, UINT64_LE, "I/O Intr" } }
The timestamp (waveform time) requires processing the same as the log entries. We use an EPICS aSub record to be notified when the value changes, and when it does we process the change in code and from that populate the WaveformTimeStr PV with a human-readable time string.
record (stringin,"$(P):WaveformTimeStr") { field(DESC, "Timestamp of the selected waveform") } record (aSub,"$(P):WaveformTimeBridge") { field(SNAM, "handle_waveform_timestamp_modify") field(INPA, "$(P):WaveformTime CP") field(FTA, "LONG") field(NOA, "4") field(OUTA, "$(P):WaveformTimeStr PP") field(FTVA, "STRING") field(NOVA, "64") }