Recent cybersecurity testing of SLAC systems and the impact on Experiment Control Systems
SLAC is undergoing a DOE cybersecurity assessment throughout this year. SLAC IT carried out some preparations for this assessment beginning in March of this year. These preparations included hiring a security testing company, Shorebreak Security, to conduct various cybersecurity-related tests of the SLAC IT systems.
These tests included attempts to gain access to computing devices connected to the SLAC intranet, first from the public Internet, then from starting within the SLAC office network. Our system security and robustness is generally focused around the network being completely closed off except for a small number of explicitly identified ports and systems. Network configuration is complex and the test enabled us to identify areas where the intended configuration and understanding did not match the reality,
The Control System (Software) Working Group (CSWG) engaged with SLAC Cybersecurity to identify especially sensitive networks which would be off-limits during these and future tests, as well as networks which would require special handling in any security testing. Special handling may include coordination with Operations.
Greg White helped to ensure the SLAC Cybersecurity and relevant control system experts met to coordinate and raised awareness of these activities. McCullough, Mark became an excellent point of contact from SLAC Cybersecurity, patiently working with us to balance preparation for the DOE assessment and preventing excessive disruption. Future improvements in communication are planned to further enable all key parties to be aware of planned activities, provide notice of emerging threats as well as a feedback loop to discuss proposed architectures and activities.
ECS had experienced control system disruptions in the past due to port scanning. This disruption resulted in outages as some network-connected components could not handle the unexpected traffic gracefully. Effects can include COTS devices becoming non-communicative until power-cycled, but there are also more dramatic possibilities which may affect physical systems, causing equipment and human hazards.
To address these concerns, additional safety and security measures are being planned and implemented. These measures include improved overall security of interface devices, review of network segmentation to ensure isolation of systems as appropriate, as well as implementing detection methods to identify undesired network traffic before it causes a problem. Plans are also under way, initiated by CSWG and ECS, to add strong cybersecurity to EPICS as a part of SLAC's contribution to the Collaboration. We don’t intend to stop there. As the Cybersecurity team likes to say, security is a journey, not a destination.
Note to all other teams: Purchasing new hardware
We want to remind everyone that anything ordered for integration or inclusion in the control system needs to be listed in the Supported Device List Supported Devices: Long Term Support. There is a process for adding new components to this list but there is no guarantee a component you order will be accepted and integrated. Furthermore, ordering an unlisted component before preliminary evaluation is strongly discouraged. If a nearly equivalent part already exists in the SDL, and there is not a very strong case and specific reason to use the new part number you ordered, then ECS will insist that you return the unlisted component and use a standard, already supported component. If a component is rejected or receives an unfavorable evaluation the same will apply.
The process for new component evaluation can take at least a month, depending on competing priorities, so please plan accordingly. You can begin the process of component evaluation here: Submit a Proposal for New Supported Device.
SC Readiness
We are closing out the effort to move from EVRs to TPRs for all devices expected to run triggered in the controls system. In addition, we added support to receive the LCLS1 timing from the LCLS2 fiber which sits behind the DAQs XPM as well as support for the DAQ partition bit.
PLC Continuous Integration Testing Pipeline
This project kicked off with Jakob Sagatowski, the author of the world-renowned All TwinCAT Blog, and leading expert in all things Beckhoff/TwinCAT, in late March/early April. We're very excited to be working with Jakob to advance our PLC workflows, making it easier to achieve a high degree of quality.
New PLC operating system: TwinCAT BSD
We started testing the new TwinCAT BSD operating system for the Beckhoff PLCs. This OS will replace our Windows Compact Embedded 7 PLC image which is standard on all PLCs. Eventually all of our PLCs will migrate to this new OS, and it will be glorious.
TCBSD is based on FreeBSD, a derivative of Unix, which makes it very similar to our typical linux operating environment. We get many enhancements using TCBSD including:
- bash
- python
- Ansible management (remote management and provisioning of the PLC image)
- Package management for PLC and OS libraries
- Improved potential security
- TCBSD can be virtualized, leading to possibilities for CI pipelines
- Remote recovery of the realtime task; ie., the realtime task can crash terribly and we can still recover the PLC remotely
- etc.
Given the current focus on cybersecurity we expect to upgrade systems on an as-needed basis to TCBSD. All new projects using PLCs are directed to use the new CX5240 with TCBSD as the OS.
LCLSPC-440 - Getting issue details... STATUS
LCLS-II HE
The LCLS-II-HE controls team continues to advance designs for the various instrument areas. CXI instrument PDR review was conducted April 19th. FXT, FEE, and XRT Transport Engineering Peer Review was conducted April 28th. The team is awaiting the committee's report from those two reviews in order to address any recommendation and move forward into the next activity. Common Components EPR is targeted for mid- to late May. The team also finalized the BCR for Lasers with the XES lasers scope changes.
MEC-U
MEC-U Controls has slowly begun ramping up in March and April and will begin to shift focus to laser beam transport controls and infrastructure design. Some notable MEC-U topics from the past couple months can be found below.
Rack Estimates and Tours:
In March and April we revisited SLAC Controls rack and rack power estimates in preparation for Facility's 90% DGPS. We estimated 21 full-sized racks and 14 shallow-depth racks at MEC-U's base scope. With NNSA's additional scope, we estimated an extra 19 full-sized racks and 18 shallow-depth racks. These are still preliminary quantities, but are needed for Facilities to estimate the approximate footprint needed for base scope and for future expansion.
The MEC-U team is also looking forward to meeting with various rack vendors such as Rittal and Steven Engineering in June as well as future suppliers (tbd). The rack visits with Rittal and Steven Engineering will consist of an initial visit at SLAC where we will provide them with a tour to see our current rack implementations and discuss the best path forward to improving this for future projects. The second visit will happen onsite at Steven Engineering in South San Francisco where we will be able to play/tinker with their Rittal rack hardware to obtain hands-on experience with their various solutions. They are encouraging SLAC visitors to bring any hardware we wish to test fit or use to spec out their solutions.
NNSA and DMPL:
The National Nuclear Security Administration (NNSA) sponsors an additional project, the Dynamic Material Properties Laser (DMPL) Project, to upgrade MEC-U's long pulse laser capacity. DMPL would add additional long pulse beamlines to the existing HE-LP laser in MEC-U scope, summing to four beamlines at 1.25kJ per beamline (5kJ total). In April, Controls participated in a cost estimate for this additional scope which will be presented to NNSA in the coming months.
FAC Review:
The Facility Advisory Committee Review is not a critical path review, but it is an opportunity for each subsystem to present their progress, designs, challenges, and risks to experienced personnel who have backgrounds in similar laser facilities and projects. The review is set for mid-July, but our team has begun planning to present several items in order to answer their controls-specific charge question: "Are the deliverables for the Control Systems unified across the three laboratories to enable value engineering and efficient operations and maintenance?"
In order to prepare for this, we are collaborating with LLNL and LLE to compile a deliverables list to present to the FAC to sufficiently answer their question. Tangential to this list, Alex is actively working on an overall MEC-U controls architecture diagram so that we can present this to the FAC and to ensure each lab has a clear picture of the path forward. He will also compile a Software Quality Assurance Plan for MEC-U that will be useful in regular ECS development and operations.
Jira:
We are currently organizing ECS and SLAC CosyLab work through Jira using the MEC-U Kanban board. So far, the board has proven to be effective at increasing visibility of the tasks to be completed at the SLAC Controls Level. Each tasks is organized under an epic, and (most) epics correspond to project activities within P6. The team continues to compile tasks under each active epic and we meet weekly to track progress and identify bottlenecks.
pcds_conda
pcds_conda version pcds-5.7.1 has been released! We released pcds-5.7.0, found a few issues, and then released pcds-5.7.1 a week later.
The intention of this release is to fulfill some long-standing package requests and get us ready for the new run with a stable python foundation.
Full release notes here:
We're preparing for a migration to python 3.10 and pyqt 5.15 within the next few months. This is for both performance and security reasons, as fewer and fewer new package versions support old builds of pyqt and of python. In particular, we're going to quickly run into a problem with openssl incompatibilities as openssl 1.1.1 reaches end of life and no longer receives security updates.
Motion and DC system architecture updates
Purpose
The architecture pages discussed below serve as one-stop references to aid in making design decisions as well as maintaining consistencies within the system.
Page Links
Motion Architecture
The motion control architecture page serves as a guide to using the standard connectors from the motor all the way to the Beckhoff components. The page covers all the standard architecture for stepper motors and encoders. It also covers architecture for specialized motion and legacy devices. The page is still a work in progress as there are new motors added for different applications. Any new motor/encoder is first approved by the Motion SME and the ECS-QA team before being added to the Motion Architecture page.
DC architecture
The DC system architecture page consists of architecture drawings depicting the connections from the supply to the end device which can be a component or a DIN-rail. The page covers which cables to use with which components and links to the released cable drawing page. A example of the architecture is shown in the image below.
Stepper Motor Torque Calculations and Serial Impedance Matching
Two new whitepapers were prepared to help resolve some commonly asked questions and points of confusion.
- Stepper Motor Sizing: This page is intended to help determine the pullout torque required for a particular application, given some mechanical parameters. The page also covers the effect that microstepping has on the motor torque, and why it should not be used for increasing resolution.
- Cable Impedance Mismatches: This page is intended to help evaluate the effects of an impedance mismatch between a signal's source, cable, and destination. Some mismatches are tolerable, while others are not. This page provides resources and a built in calculator to help with these evaluations.
Record of Decision regarding Micronix systems
After much discussion and evaluation, a Record of Decision has been written regarding the use of Micronix piezo systems in future designs. The RoD has yet to be finalized, but is discussed in more detail here and outlined below.
- The Micronix MMC-100 has a history of communication issues, requiring specialized controller knowledge and leading to a burden on Operations.
- The MMC-100 control has therefore been deemed EOL (end-of-life) in the Supported Devices List, and is considered to be legacy equipment. This means:
- The MMC-100 may only be purchased to replace non-functional units that have already been put into operations.
- It is strongly suggested that any units that are already in operation be replaced with LTS (long term support) equipment.
- The MMC-100, even if the hardware is already in use in LCLS-I, may not be re-used in upcoming projects such as LCLS-II HE.
Please direct any questions regarding this Record of Decision or the Supported Device List to ECS.
NALMS
The NALMS project has made significant strides in recent months, but the team faced some technical difficulties during the final phase of development. While the NALMS deployment in S3DF Kubernetes is almost complete, there are still some communication issues that need to be resolved. To track these issues, the team has created an epic ticket on Jira that outlines the remaining tasks that need to be completed in the coming weeks to achieve a full deployment.
Despite these challenges, the project has reached a prototype stage where it is ready for use, and the team is eager to receive feedback from users.
A recent demonstration with major users Bill and Stefan for Vacuum and GMD and XGMD fields was successfully completed.
If you're interested in trying out NALMS, see this dedicated web page for installation.
You can find more information about NALMS on Confluence, including a description and an explanation of the NALMS workflow. The team is committed to delivering NALMS in the next weeks.
ATEF
ATEF continues to make steady progress toward becoming a useful tool for guiding and documenting checkouts.
Since the last time it was featured in the newsletter (Aug/Sept2022), ATEF has received a reworked GUI, added report generation, and begun implementing active checkout support.
What does this mean?
- GUI rework: ATEF now sports a GUI that not only allows users to edit and compose their checkouts, but also run them. The GUI is being expanded to support active checkout steps as they are added.
The passive checkout GUI also now supports more complex groupings of passive checks, allowing checks to be grouped by device, PV, or tool type. - Report auto-generation: Reports can now be generated from completed checkouts. These reports summarize the checkout settings, results, and collected data if applicabale. As more checkout steps are added, their corresponding report output styles will also be updated.
- Active checkouts: In contrast to passive checkouts, active checkouts involve making changes to the controls system (setting PVs, moving motors, etc). This is the current focus of our development, so please be patient with us!
For more information, see the atef summary page.
ATEF is still in pre-release, but if you are interested in testing it out and providing feedback on its functionality, let us know!
Standards, Guides, and Quality Assurance Plans
Ken Lauer Alex Wallace Federica Murgia
MEC-U and HE are driving ECS to develop and release a set of quality assurance plans that will document our engineering, build, and test processes. This effort will be tracked in this epic:
LCLSPC-705 - Getting issue details... STATUS
We may aim to validate these plans at the lab-wide level if possible.
Github Enterprise
Github Enterprise has been approved by the SLAC Engineering Council and the Controls Software Working Group to become the lab solution for source code management. We are in the process of receiving cybersecurity approval of the platform and will proceed with a trial and full licensing as soon as possible after that.
More details about the trial and how to use GHE will emerge in the coming months. Very exciting!
EPLAN and Teamcenter
We're planning to integrate Teamcenter PLM and EPLAN. This integration will bring several advantages, such as improved efficiency through automatically reflecting changes made in EPLAN to Teamcenter, better part tracking through Teamcenter's BOM, improved lifecycle management with a complete view of history provided by Teamcenter, and enhanced document control through specific repositories and tags.
Furthermore, Teamcenter offers a document control that will check all the blocks (check, revision, approval) and then release the drawing in order to have always a complete ad update version. Moreover, after the release, only an official revision can modify the drawing, ensuring consistency of updates.
The Teamcenter admins will ensure that the integration module is updated with EPLAN to maintain compatibility and optimize system performance.
To properly manage Teamcenter content, several tutorials are accessible with a Teamcenter license. We'll start familiarizing ourselves with the software soon.
Several meetings are scheduled to present the last version of Teamcenter to the ECS group and to start integrating Teamcenter and EPLAN.
UI/UX development process and systems engineering
Some new considerations are being made in the area of UI/UX development. In particular the team is considering new software that allows for quick GUI wireframing and mock ups. These methods originated in the form of pencil and paper and enabled a designer to quickly deploy and demonstrate the flow and appearance of an application. In this way stakeholders could quickly identify areas of the GUI that were confusing and failed to adhere to common GUI heuristics such as the ones listed here: https://www.nngroup.com/articles/ten-usability-heuristics/.
This pen-and-pencil wireframing method has made the jump to modern-day web applications. One can find a multitude of easy-to-use applications with a quick Google search.
These tools not only enable a designer to create a wireframe, but also lets the stakeholder join in. What's more, multiple users can edit a wireframe at the same time!
Here's an example of a Figma session with a quick mock up for the Compound Refractive Lens.
We’ll be evaluating a number of prototyping tools and processes, if you have thoughts about this topic let us know!
Hannover Messe
A detachment of ECS engineers headed to Germany April 17-21 to attend the Hannover Messe, an industry automation fair. In addition of getting to know the latest developments made by some of our usual suppliers (Beckhoff, Phoenix Contact), the team also investigated solutions for wireless sensors, power supplies, and new motions and position measurement systems.
Some highlights, in no particular order:
- Wireless I/O link by IFM
- Beckhoff EJ terminals solution for motion systems
- Ubuntu Core real-time engine
- JVL intelligent motors
Deployed EPICS IOCs and module usage statistics
To view information on all currently deployed EPICS IOCs, see the following document:
EPICS IOCs Deployed in IOC Manager
To see statistics regarding EPICS modules and versions, see the following document:
SLAC IT's Newsletter
We'd like to highlight the SLAC IT newsletter and site and give props. It looks great and has a lot of useful information. Check it out!
https://it.slac.stanford.edu/news
Hello, Goodbye
We had to say goodbye to Ortiz, Jose on May 1st and wish him and his family all the best for their lives (back) on the east coast. He will remain with us on a casual basis to ensure continuity for DXS/XCS.
Hill, Bruce officially retired on March 24th. He still works casually, but please go through ECS to request work that he used to take on!
On the plus side, we hired Josue Zamudio as an SEA. He started March 17th.
Jira Results
to
Overview
Content Tools