Locking for edits
Publishing
Work done to is fair game
Recent cybersecurity testing of SLAC systems and the impact on Experiment Control Systems
SLAC will undergo a DOE cybersecurity assessment in June/July of this year. SLAC IT began preparations for this assessment starting in late March and early April. These preparations included hiring an external security testing company, Shorebreak, to conduct various cybersecurity related tests of the SLAC IT systems.
These tests included attempts to gain access to and the ability to execute processes on computing devices connected to the SLAC intranet first from the internet, then from within the SLAC network (e.g. a computer connected to a SLAC IT managed switch or router). This process includes a variety of attack methods including port scanning and attempts to exploit known vulnerabilities in IT equipment and software.
It was somewhat unsettling to learn about the upcoming assessment incidentally and the ongoing testing after it had begun. ECS has experienced control system disruption due to SLAC Cybersecurity's standard and regular port scanning activities. This disruption has manifested in the form of outages as network connected components may not handle port scanning traffic gracefully. ECS has requested on multiple occasions to be notified anytime these kinds of probing activities are taking place so we can prepare to recover affected systems and coordinate with operations. Effects can include COTS devices becoming non-communicative until power-cycled, but there are also more dramatic possibilities which are not far from the realm of possibility and may affect physical systems causing equipment and human hazards.
Our network system security and robustness is generally addressed by designing our network to be completely closed off except for a small number of explicitly identified ports. Network configuration is a complex activity and occasionally due to misconfiguration and lack of coordination we have experienced Cybersecurity induced control system outages.
In the case of the recent assessment by Shorebreak, ECS and AD EED became aware of testing after it began. After becoming aware of the ongoing assessment the Control System (Software) Working Group (CSWG) engaged with SLAC Cybersecurity to identify especially sensitive networks which would be off-limits during these tests, as well as networks would would require advance notice from Shorebreak before they began their tests so we could inform operations and prepare to recover systems. Greg White helped to ensure the SLAC Cybersecurity and relevant control system experts were able to meet and coordinate and raised awareness of these activities. McCullough, Mark was an excellent point of contact from SLAC IT and patiently helped us find a workable path towards preparation for the DOE assessment and preventing excessive disruption.
Given the present environment at SLAC with regards to work planning and control, it is somewhat surprising that this activity would proceed without a wider broadcast and approval process. Our control systems are designed and built to operate nominally within a network environment consisting of known types of traffic. Testing during installation and commissioning confirms to the extent possible that our systems are robust in the ways we designed them to be. The introduction of cybersecurity testing, which is an aspect we don't strictly consider in our designs, is risky. That risk is compounded due to lack of communication. These are lessons we must learn from. On the positive side, we now have a stronger relationship with SLAC Cybersecurity than we have had before. Also, while system security was a growing concern for the CSWG, these assessments have increased our attention and perhaps motivated us to more seriously consider the topic.
Notice to all other teams: Purchasing new hardware
We want to gently remind everyone that anything ordered for integration or inclusion in the control system needs to be listed in the Supported Device List Supported Devices: Long Term Support. There is a process for adding new component to this list but there is no guarantee a component you order will be accepted and integrated. If a nearly equivalent part already exists in the SDL, and there is not a very strong case and specific reason to use the new part number you ordered, then ECS will insist that you return the unlisted component and use a standard, already supported component. If a component is rejected or receives an unfavorable evaluation the same will apply.
The process for new component evaluation can take at least a month, depending on competing priorities so please plan accordingly. You can begin the process of component evaluation here: Submit a Proposal for New Supported Device.
SC Readiness
PMPS Readiness
Margaret Ghaly Zachary L Lentz
PLC Continuous Integration Testing Pipeline
New PLC operating system: TwinCAT BSD
LCLS-II HE
MEC-U
MEC-U Controls has slowly begun ramping up in March and April and will begin to shift focus to laser beam transport controls and infrastructure design. Some notable MEC-U topics from the past couple months can be found below.
Rack Estimates and Tours:
In March and April we revisited SLAC Controls rack and rack power estimates in preparation for Facility's 90% DGPS. We estimated 21 full-sized racks and 14 shallow-depth racks at MEC-U's base scope. With NNSA's additional scope, we estimated an extra 19 full-sized racks and 18 shallow-depth racks. These are still preliminary quantities, but are needed for Facilities to estimate the approximate footprint needed for base scope and for future expansion.
The MEC-U team is also looking forward to meeting with various rack vendors such as Rittal and Steven Engineering in May as well as future suppliers tbd. The rack visits with Rittal and Steven Engineering will consist of an initial visit at SLAC where we will provide them a tour to see our current rack implementations and discuss the best path forward to improving this for future projects. The second visit will happen onsite at Steven Engineering in South San Francisco where we will be able to play/tinker with their Rittal rack hardware to obtain hands-on experience with their various solutions. They are encouraging SLAC visitors to bring any hardware we wish to test fit or use to spec out their solutions.
NNSA and DMPL:
NNSA, National Nuclear Security Administration, are possible sponsors for additional scope under the MEC-U umbrella. Their project, DMPL, would add additional long pulse beamlines to the existing HE-LP laser, summing to four beamlines at 1.25kJ per beamline, 5kJ total. In April, Controls participated in a cost estimate for this additional scope which will be presented to NNSA in the coming months.
FAC Review:
The Facility Advisory Committee Review is not a critical path review, but it is an opportunity for each subsystem to present their progress, designs, challenges, and risks to experienced personnel who have backgrounds in similar laser facilities and projects. The review is set for mid-July, but our team has begun planning to present several items in order to answer their controls-specific charge question: "Are the deliverables for the Control Systems unified across the three laboratories to enable value engineering and efficient operations and maintenance?"
In order to prepare for this, we are collaborating with LLNL and LLE to compile a deliverables list to present to the FAC to sufficiently answer their question. Tangential to this list, Alex is actively working on an overall MEC-U controls architecture diagram so that we can present this to the FAC and to ensure each lab has a clear picture of the path forward. He will also be compiling a Software Quality Assurance Plan for MEC-U that will also prove to be useful in regular ECS development and operations.
Jira:
We are currently organizing ECS and SLAC CosyLab work through Jira using the MEC-U Kanban board. So far, the board has proven to be effective at increasing visibility of the tasks to be completed at the SLAC Controls Level. Each tasks is organized under an epic, and (most) epics correspond to project activities within P6. The team continues to compile tasks under each active epic and we meet weekly to track progress and identify bottlenecks.
pcds_conda
pcds_conda version pcds-5.7.1 has been released! We released pcds-5.7.0, found a few issues, and then released pcds-5.7.1 a week later.
The intention of this release is to fulfill some long-standing package requests and to get us ready for the new run with a stable python foundation.
Full release notes here:
We're preparing for a migration to python 3.10 and pyqt 5.15 within the next few months. This is for both performance and security reasons, as fewer and fewer new package versions support old builds of pyqt and of python. In particular, we're going to quickly run into a problem with openssl incompatibilities as openssl 1.1.1 reaches end of life and no longer receives security updates.
Motion and DC system architecture updates
Stepper Motor Torque Calculations and Serial Impedance Matching
Two new whitepapers were prepared to help resolve some commonly asked questions and points of confusion.
- Stepper Motor Sizing: This page is intended to help determine the pullout torque required for a particular application, given some mechanical parameters. The page also covers the effect that microstepping has on the motor torque, and why it should not be used for increasing resolution.
- Cable Impedance Mismatches: This page is intended to help evaluate the effects of an impedance mismatch between a signal's source, cable, and destination. Some mismatches are tolerable, while others are not. This page provides resources and a built in calculator to help with these evaluations.
Record of decision regarding Micronix systems
After much discussion and evaluation, a Record of Decision has been written regarding the use of Micronix piezo systems in future designs. The RoD has yet to be finalized but is discussed in more detail here, but is outlined below:
- The Micronix MMC-100 has a history of communication issues, requiring specialized controller knowledge, leading to a burden on operations.
- The MMC-100 control has therefore been deemed EOL (end of life) in the Supported Devices List, and is considered to be legacy equipment. This means:
- The MMC-100 may only be purchased to replace non-functional units that have already been put into operations.
- Any units that are already in operations are strongly suggested to be replaced with LTS (long term support) equipment.
- The MMC-100, even if the hardware is already in use in LCLS-I, may not be re-used in upcoming projects such as LCLS-II HE.
Please direct any questions regarding this Record of Decision or the Supported Device List to ECS.
NALMS
The NALMS project has made significant strides in recent months, but the team faced some technical difficulties during the final phase of development. While the NALMS deployment in S3DF Kubernetes is almost complete, there are still some communication issues that need to be resolved. To track these issues, the team has created an epic ticket on Jira that outlines the remaining tasks that need to be completed in the coming weeks to achieve a full deployment.
Despite these challenges, the project has reached a prototype stage where it is ready for use, and the team is eager to receive feedback from users.
A recent demonstration with major users Bill and Stefan, for Vacuum and GMD and XGMD fields was successfully completed.
If you're interested in trying out NALMS, there is a dedicated web page for installation.
You can find more information about NALMS on Confluence, including a description of what NALMS is and how NALMS workflow works. The team is committed to delivering NALMS in the next weeks.
ATEF
ATEF continues to make steady progress toward becoming a useful tool guiding and documenting checkouts.
Since the last time ATEF was featured in the newsletter (Aug/Sept2022), ATEF has received a reworked GUI,
added report generation, and begun implementing active checkout support. What does this mean?
- GUI rework: ATEF now sports a GUI that not only allows users to edit and compose their checkouts, but also run said checkouts. The GUI is also being expanded to support active checkout steps as they are added.
The passive checkout GUI also now supports more complex groupings of passive checks, allowing checks to be grouped by device, PV, or tool type. - Report auto-generation: Reports can now be generated from completed checkouts. These reports summarize the checkout settings, results, and collected data if applicabale. As more checkout steps are added, their corresponding report output styles will also be updated.
- Active Checkouts: In contrast to passive checkouts, active checkouts involve making changes to the controls system (setting PV's, moving motors, etc). This is the current focus of our development, so please be patient with us!
For more information, see the atef summary page.
ATEF is still in pre-release, but if you are interested in testing it out and providing feedback on its functionality, let us know!
Standards, Guides, and Quality Assurance Plans
Ken Lauer Alex Wallace Federica Murgia
Github Enterprise
EPLAN and Teamcenter
We're planning to integrate Teamcenter PLM and EPLAN. This integration will bring several advantages, such as improved efficiency by automatically reflecting changes made in EPLAN to Teamcenter, better part tracking through Teamcenter's BOM, improved lifecycle management with a complete view of history provided by Teamcenter, and enhanced document control through specific repositories and tags.
Furthermore, Teamcenter offers a document control that will check all the blocks (check, revision, approval) and then release the drawing in order to have always a complete ad update version. Moreover, after the release, only an official revision can modify the drawing, ensuring consistency of updates.
The Teamcenter admins will ensure that the integration module is updated with EPLAN to maintain compatibility and optimize system performance.
To properly manage Teamcenter content, several tutorials accessible with a Teamcenter license. We'll start getting familiar with the software soon.
Several meetings are scheduled to present the last version of Teamcenter to the ECS group and to start integrating Teamcenter and EPLAN.
UI/UX development process and systems engineering
Hannover Messe
Deployed EPICS IOCs and module usage statistics
Hello, Goodbye
Overview
Content Tools