Locking for edits
Publishing
Work done to is fair game
Recent cybersecurity testing of SLAC systems and the impact on Experiment Control Systems
SLAC will undergo a DOE cybersecurity assessment in June/July of this year. SLAC IT carried out some preparations for this assessment in late March and early April. These preparations included hiring a security testing company, Shorebreak Security, to conduct various cybersecurity-related tests of the SLAC IT systems.
These tests included attempts to gain access to and the ability to execute processes on computing devices connected to the SLAC intranet, first from the internet, then from within the SLAC network (e.g., a computer connected to a SLAC IT-managed switch or router). This process includes a variety of attack methods including port scanning and attempts to exploit known vulnerabilities in IT equipment and software.
It was somewhat unsettling to learn about the upcoming assessment incidentally and the ongoing testing after it had begun. ECS has experienced control system disruptions due to SLAC Cybersecurity's standard and regular port scanning activities in the past. This disruption resulted in outages as some network-connected components could not handle port-scanning traffic gracefully. On multiple occasions ECS has requested to be notified any time these kinds of probing activities are taking place so we can prepare to recover affected systems and coordinate with operations. Effects can include COTS devices becoming non-communicative until power-cycled, but there are also more dramatic possibilities which may affect physical systems, causing equipment and human hazards.
Our system security and robustness is generally addressed by designing the network to be completely closed off except for a small number of explicitly identified ports. Network configuration is complex and occasionally due to misconfiguration and lack of coordination we have experienced issues as a result of the security testing. To be perfectly clear, ECS appreciates the importance and necessity of what SLAC Cybersecurity does and we understand their methodology. Together we have experienced a learning curve to get to a place where the testing can be performed and the impact to operations can be mitigated, but there is still room for improvement.
In the case of the recent assessment by Shorebreak, ECS and AD EED became aware of testing after it began. After becoming aware of the ongoing assessment, the Control System (Software) Working Group (CSWG) engaged with SLAC Cybersecurity to identify especially sensitive networks which would be off-limits during these tests, as well as networks which would require advance notice from Shorebreak before they began their tests, so we could inform Operations and prepare to recover systems. Greg White helped to ensure the SLAC Cybersecurity and relevant control system experts met to coordinate and raised awareness of these activities. McCullough, Mark became an excellent point of contact from SLAC Cybersecurity, patiently working with us to balance preparation for the DOE assessment and preventing excessive disruption.
Given the present environment at SLAC with regards to work planning and control, it is somewhat surprising that this activity would proceed without a wider broadcast and approval process. Our control systems are designed and built to operate nominally within a network environment consisting of known types of traffic. Testing during installation and commissioning confirms—to the extent possible—that our systems are robust in the ways we designed them to be. The introduction of cybersecurity testing, which is an aspect we don't strictly consider in our designs, is risky. That risk is compounded with poor communication. These are lessons we must learn from. On the positive side, we now have a stronger relationship with SLAC Cybersecurity than we have had before. Also, while system security was a growing concern for the CSWG, these assessments have increased our attention and perhaps motivated us to more seriously consider the topic.
Note to all other teams: Purchasing new hardware
We want to remind everyone that anything ordered for integration or inclusion in the control system needs to be listed in the Supported Device List Supported Devices: Long Term Support. There is a process for adding new components to this list but there is no guarantee a component you order will be accepted and integrated. Furthermore, ordering an unlisted component before preliminary evaluation is strongly discouraged. If a nearly equivalent part already exists in the SDL, and there is not a very strong case and specific reason to use the new part number you ordered, then ECS will insist that you return the unlisted component and use a standard, already supported component. If a component is rejected or receives an unfavorable evaluation the same will apply.
The process for new component evaluation can take at least a month, depending on competing priorities, so please plan accordingly. You can begin the process of component evaluation here: Submit a Proposal for New Supported Device.
SC Readiness
We are closing out to effort to move from EVRs to TPRs for all devices expected to run triggered in the controls system. In addition, we added support to receive the LCLS1 timing from the LCLS2 fiber which sits behind the DAQs XPM as well as support for the DAQ partition bit.
PMPS Readiness
Margaret Ghaly Zachary L Lentz
PLC Continuous Integration Testing Pipeline
This project kicked off with Jakob Sagatowski, the author of the world renowned All TwinCAT Blog, and leading expert in all things Beckhoff/TwinCAT in late March early April. We're very excited to be working with Jakob to advance our PLC workflows, making it easier to achieve a high degree of quality.
New PLC operating system: TwinCAT BSD
We started testing the new TwinCAT BSD operating system for the Beckhoff PLCs. This OS will replace our Windows Compact Embedded 7 PLC image which is standard on all PLCs. Eventually all of our PLCs will migrate to this new OS, and it will be glorious.
TC BSD is based on FreeBSD, a derivative of Unix, which makes it very similar to our typical linux operating environment. There are many enhancements we get from using TCBSD including:
- bash
- python
- Ansible management (remote management and provisioning of the PLC image)
- Package management for PLC and OS libraries
- Improved potential security
- TCBSD can be virtualized, leading to possibilities for CI pipelines
- Remote recovery of the realtime task, ie. the realtime task can crash terribly and we can still recover the PLC remotely
- etc.
Given the current focus on cybersecurity we expect to upgrade systems on an as needed basis to TCBSD. All new projects using PLCs are directed to use the new CX5240 with TCBSD as the OS.
LCLSPC-440 - Getting issue details... STATUS
LCLS-II HE
The LCLS-II-HE controls team continue to advance designs for the various instrument areas. CXI instrument PDR review was conducted April 19th. FXT, FEE and XRT Transport, Engineering Peer Review has also been conducted April 28th. The team is awaiting the committee's report from those two reviews in order to address any recommendation and move forward into the next activity. Common Components EPR targeted for mid to late May. The team also finalized the BCR for Lasers with the XES lasers scope changes.
MEC-U
MEC-U Controls has slowly begun ramping up in March and April and will begin to shift focus to laser beam transport controls and infrastructure design. Some notable MEC-U topics from the past couple months can be found below.
Rack Estimates and Tours:
In March and April we revisited SLAC Controls rack and rack power estimates in preparation for Facility's 90% DGPS. We estimated 21 full-sized racks and 14 shallow-depth racks at MEC-U's base scope. With NNSA's additional scope, we estimated an extra 19 full-sized racks and 18 shallow-depth racks. These are still preliminary quantities, but are needed for Facilities to estimate the approximate footprint needed for base scope and for future expansion.
The MEC-U team is also looking forward to meeting with various rack vendors such as Rittal and Steven Engineering in May as well as future suppliers tbd. The rack visits with Rittal and Steven Engineering will consist of an initial visit at SLAC where we will provide them a tour to see our current rack implementations and discuss the best path forward to improving this for future projects. The second visit will happen onsite at Steven Engineering in South San Francisco where we will be able to play/tinker with their Rittal rack hardware to obtain hands-on experience with their various solutions. They are encouraging SLAC visitors to bring any hardware we wish to test fit or use to spec out their solutions.
NNSA and DMPL:
NNSA, National Nuclear Security Administration, are sponsors for an additional project, the Dynamic Material Properties Laser (DMPL) Project, to upgrade MEC-U's long pulse laser capacity. DMPL would add additional long pulse beamlines to the existing HE-LP laser in MEC-U scope, summing to four beamlines at 1.25kJ per beamline, 5kJ total. In April, Controls participated in a cost estimate for this additional scope which will be presented to NNSA in the coming months.
FAC Review:
The Facility Advisory Committee Review is not a critical path review, but it is an opportunity for each subsystem to present their progress, designs, challenges, and risks to experienced personnel who have backgrounds in similar laser facilities and projects. The review is set for mid-July, but our team has begun planning to present several items in order to answer their controls-specific charge question: "Are the deliverables for the Control Systems unified across the three laboratories to enable value engineering and efficient operations and maintenance?"
In order to prepare for this, we are collaborating with LLNL and LLE to compile a deliverables list to present to the FAC to sufficiently answer their question. Tangential to this list, Alex is actively working on an overall MEC-U controls architecture diagram so that we can present this to the FAC and to ensure each lab has a clear picture of the path forward. He will also be compiling a Software Quality Assurance Plan for MEC-U that will also prove to be useful in regular ECS development and operations.
Jira:
We are currently organizing ECS and SLAC CosyLab work through Jira using the MEC-U Kanban board. So far, the board has proven to be effective at increasing visibility of the tasks to be completed at the SLAC Controls Level. Each tasks is organized under an epic, and (most) epics correspond to project activities within P6. The team continues to compile tasks under each active epic and we meet weekly to track progress and identify bottlenecks.
pcds_conda
pcds_conda version pcds-5.7.1 has been released! We released pcds-5.7.0, found a few issues, and then released pcds-5.7.1 a week later.
The intention of this release is to fulfill some long-standing package requests and to get us ready for the new run with a stable python foundation.
Full release notes here:
We're preparing for a migration to python 3.10 and pyqt 5.15 within the next few months. This is for both performance and security reasons, as fewer and fewer new package versions support old builds of pyqt and of python. In particular, we're going to quickly run into a problem with openssl incompatibilities as openssl 1.1.1 reaches end of life and no longer receives security updates.
Motion and DC system architecture updates
Purpose
The architecture pages serves as a one stop reference to make design decisions and also help maintain consistencies within the system.
Page Links
Motion Architecture
The motion control architecture page serves as a guide to use the standard connectors from the motor all the way to the beckhoff components. The Page covers all the standard architecture to follow for Stepper motors & encoders. It also consists of architecture for specialized motion and legacy devices. The page is still work in progress as there are new motors added for different application. Any new motor/encoder is first approved by the Motion SME and the ECS-QA team before getting added to the Motion Architecture page.
DC architecture
The DC system architecture page consists of architecture drawings which depicts the connection from the supply to the end device which can be a component or a Din-Rail. The page goes over the cables to be used with which component with the links to the released cable drawing page, A example of the architecture can be shown as below
Stepper Motor Torque Calculations and Serial Impedance Matching
Two new whitepapers were prepared to help resolve some commonly asked questions and points of confusion.
- Stepper Motor Sizing: This page is intended to help determine the pullout torque required for a particular application, given some mechanical parameters. The page also covers the effect that microstepping has on the motor torque, and why it should not be used for increasing resolution.
- Cable Impedance Mismatches: This page is intended to help evaluate the effects of an impedance mismatch between a signal's source, cable, and destination. Some mismatches are tolerable, while others are not. This page provides resources and a built in calculator to help with these evaluations.
Record of decision regarding Micronix systems
After much discussion and evaluation, a Record of Decision has been written regarding the use of Micronix piezo systems in future designs. The RoD has yet to be finalized but is discussed in more detail here, but is outlined below:
- The Micronix MMC-100 has a history of communication issues, requiring specialized controller knowledge, leading to a burden on operations.
- The MMC-100 control has therefore been deemed EOL (end of life) in the Supported Devices List, and is considered to be legacy equipment. This means:
- The MMC-100 may only be purchased to replace non-functional units that have already been put into operations.
- Any units that are already in operations are strongly suggested to be replaced with LTS (long term support) equipment.
- The MMC-100, even if the hardware is already in use in LCLS-I, may not be re-used in upcoming projects such as LCLS-II HE.
Please direct any questions regarding this Record of Decision or the Supported Device List to ECS.
NALMS
The NALMS project has made significant strides in recent months, but the team faced some technical difficulties during the final phase of development. While the NALMS deployment in S3DF Kubernetes is almost complete, there are still some communication issues that need to be resolved. To track these issues, the team has created an epic ticket on Jira that outlines the remaining tasks that need to be completed in the coming weeks to achieve a full deployment.
Despite these challenges, the project has reached a prototype stage where it is ready for use, and the team is eager to receive feedback from users.
A recent demonstration with major users Bill and Stefan, for Vacuum and GMD and XGMD fields was successfully completed.
If you're interested in trying out NALMS, there is a dedicated web page for installation.
You can find more information about NALMS on Confluence, including a description of what NALMS is and how NALMS workflow works. The team is committed to delivering NALMS in the next weeks.
ATEF
ATEF continues to make steady progress toward becoming a useful tool guiding and documenting checkouts.
Since the last time ATEF was featured in the newsletter (Aug/Sept2022), ATEF has received a reworked GUI,
added report generation, and begun implementing active checkout support. What does this mean?
- GUI rework: ATEF now sports a GUI that not only allows users to edit and compose their checkouts, but also run said checkouts. The GUI is also being expanded to support active checkout steps as they are added.
The passive checkout GUI also now supports more complex groupings of passive checks, allowing checks to be grouped by device, PV, or tool type. - Report auto-generation: Reports can now be generated from completed checkouts. These reports summarize the checkout settings, results, and collected data if applicabale. As more checkout steps are added, their corresponding report output styles will also be updated.
- Active Checkouts: In contrast to passive checkouts, active checkouts involve making changes to the controls system (setting PV's, moving motors, etc). This is the current focus of our development, so please be patient with us!
For more information, see the atef summary page.
ATEF is still in pre-release, but if you are interested in testing it out and providing feedback on its functionality, let us know!
- Robert Tang-Kong
Standards, Guides, and Quality Assurance Plans
Ken Lauer Alex Wallace Federica Murgia
MEC-U and HE are driving ECS to develop and release a set of quality assurance plans that will document our engineering, build, and test processes. This effort will be tracked in this epic:
LCLSPC-705 - Getting issue details... STATUS
We think we may aim to validate these plans at the lab-wide level if possible.
Github Enterprise
Github Enterprise has been approved by the SLAC Engineering Council and the Controls Software Working Group to become the lab solution for source code management. We are in the process of receiving cybersecurity approval of the platform and will proceed with a trial and full licensing as soon as possible after that.
More details about the trial and how to use GHE will emerge in the coming months. Very exciting!
EPLAN and Teamcenter
We're planning to integrate Teamcenter PLM and EPLAN. This integration will bring several advantages, such as improved efficiency by automatically reflecting changes made in EPLAN to Teamcenter, better part tracking through Teamcenter's BOM, improved lifecycle management with a complete view of history provided by Teamcenter, and enhanced document control through specific repositories and tags.
Furthermore, Teamcenter offers a document control that will check all the blocks (check, revision, approval) and then release the drawing in order to have always a complete ad update version. Moreover, after the release, only an official revision can modify the drawing, ensuring consistency of updates.
The Teamcenter admins will ensure that the integration module is updated with EPLAN to maintain compatibility and optimize system performance.
To properly manage Teamcenter content, several tutorials accessible with a Teamcenter license. We'll start getting familiar with the software soon.
Several meetings are scheduled to present the last version of Teamcenter to the ECS group and to start integrating Teamcenter and EPLAN.
UI/UX development process and systems engineering
Some new considerations are being made in the area of UI/UX development. In particular the team is considering new software that allows for quick GUI wire framing and mock ups. These methods originated in the form of pencil and paper and allow a designer to quickly deploy and demonstrate the flow and appearance of an application. In this way stake holders can quickly identify areas of the GUI that need improvement. That is, what areas of the GUI do they get stuck in and don't know what to do? More broadly, where does the GUI fail to adhere to the common GUI heuristics: https://www.nngroup.com/articles/ten-usability-heuristics/.
This pen and pencil wire framing method has made the jump to modern day web application. In recent years, the number of available applications has ballooned. One can find a multitude of available applications with a quick google search.
These tools allow not only the designer to create a wire frame, but also allow the stake holder to join in. Whats more, multiple users can edit the GUI wire frame at the same time! Here's an example of a Figma session with a quick mock up for the Compound Refractive Lens.
Hannover Messe
A detachment of ECS engineers headed to Germany April 17-21 to attend the Hannover Messe, an industry automation fair. In addition of getting to know the latest developments made by some of our usual suppliers (Beckhoff, Phoenix Contact), the team also investigated solutions for wireless sensors, power supplies or new motions and position measurement systems.
Some highlights, in no particular order:
- Wireless I/O link by IFM
- Beckhoff EJ terminals solution for motion systems
- Ubuntu Core real-time engine
- JVL intelligent motors
Deployed EPICS IOCs and module usage statistics
To view information on all currently-deployed EPICS IOCs, see the following document:
EPICS IOCs Deployed in IOC Manager
To see statistics regarding EPICS modules and versions, see the following document:
SLAC IT's Newsletter
We wanted to highlight the SLAC IT newsletter and site and give props, it looks great has a lot of useful information. Check it out!
https://it.slac.stanford.edu/news
Hello, Goodbye
We had to say goodbye to Ortiz, Jose on May 1st and wish him and his family all the best for their lives (back on) the east coast. He will remain with us on a casual basis to ensure continuity for DXS/XCS.
Hill, Bruce officially retired on March 24th. He still works casually, but please go through ECS to request work that he used to take on!
On the plus-side, we hired Josue Zamudio as an SEA starting in that role on March 17th.
Jira Results
to
Overview
Content Tools