Recent cybersecurity testing of SLAC systems and the impact on Experiment Control Systems

Alex Wallace 

SLAC will undergo is undergoing a DOE cybersecurity assessment in June/July of throughout this year. SLAC IT carried out some preparations for this assessment beginning in late March and early AprilMarch of this year. These preparations included hiring a security testing company, Shorebreak Security, to conduct various cybersecurity-related tests of the SLAC IT systems.

These tests included attempts to gain access to and the ability to execute processes on computing devices connected to the SLAC intranet, first from the internetpublic Internet, then from starting within the SLAC office network (e.g., a computer connected to a SLAC IT-managed switch or router). This process includes a variety of attack methods including port scanning and attempts to exploit known vulnerabilities in IT equipment and software.

It was somewhat unsettling to learn about the upcoming assessment incidentally and the ongoing testing after it had begun. ECS has experienced control system disruptions due to SLAC Cybersecurity's standard and regular port scanning activities in the past. This disruption resulted in outages as some network-connected components could not handle port-scanning traffic gracefully. On multiple occasions ECS has requested to be notified any time these kinds of probing activities are taking place so we can prepare to recover affected systems and coordinate with operations. Effects can include COTS devices becoming non-communicative until power-cycled, but there are also more dramatic possibilities which may affect physical systems, causing equipment and human hazards.

Our system security and robustness is generally addressed by designing the network to be completely closed off except for a small number of explicitly identified ports. Network configuration is complex and occasionally due to misconfiguration and lack of coordination we have experienced issues as a result of the security testing. To be perfectly clear, ECS appreciates the importance and necessity of what SLAC Cybersecurity does and we understand their methodology. Together we have experienced a learning curve to get to a place where the testing can be performed and the impact to operations can be mitigated, but there is still room for improvement.

In the case of the recent assessment by Shorebreak, ECS and AD EED became aware of testing after it began. After becoming aware of the ongoing assessment, the Control System (Software) Working Group (CSWG) engaged with SLAC Cybersecurity to identify especially sensitive networks which would be off-limits during these tests, as well as networks which would require advance notice from Shorebreak before they began their tests, so we could inform Operations and prepare to recover systems. Greg White helped to ensure the SLAC Cybersecurity and relevant control system experts met to coordinate and raised awareness of these activities. McCullough, Mark became an excellent point of contact from SLAC Cybersecurity, patiently working with us to balance preparation for the DOE assessment and preventing excessive disruption.

. Our system security and robustness is generally focused around the network being completely closed off except for a small number of explicitly identified ports and systems. Network configuration is complex and the test enabled us to identify areas where the intended configuration and understanding did not match the reality,

The Control System (Software) Working Group (CSWG) engaged with SLAC Cybersecurity to identify especially sensitive networks which would be off-limits during these and future tests, as well as networks which would require special handling in any security testing. Special handling may include coordination with Operations. 

Greg White helped to ensure the SLAC Cybersecurity and relevant control system experts met to coordinate and raised awareness of these activities. McCullough, Mark became an excellent point of contact from SLAC Cybersecurity, patiently working with us to balance preparation for the DOE assessment and preventing excessive disruption.  Future improvements in communication are planned to further enable all key parties to be aware of planned activities, provide notice of emerging threats as well as a feedback loop to discuss proposed architectures and activities.

ECS had experienced control system disruptions in the past due to port scanning.  This disruption resulted in outages as some network-connected components could not handle the unexpected traffic gracefully. Effects can include COTS devices becoming non-communicative until power-cycled, but there are also more dramatic possibilities which may affect physical systems, causing equipment and human hazards.

To address these concerns, additional safety and security measures are being planned and implemented.  These measures include improved overall security of interface devices, review of network segmentation to ensure isolation of systems as appropriate, as well as implementing detection methods to identify undesired network traffic before it causes a problem. Plans are also under way, initiated by CSWG and ECS, to add strong cybersecurity to EPICS as a part of SLAC's contribution to the Collaboration. We don’t intend to stop there.  As the Cybersecurity team likes to say, security is a journey, not a destinationGiven the present environment at SLAC with regards to work planning and control, it is somewhat surprising that this activity would proceed without a wider broadcast and approval process. Our control systems are designed and built to operate nominally within a network environment consisting of known types of traffic. Testing during installation and commissioning confirms—to the extent possible—that our systems are robust in the ways we designed them to be. The introduction of cybersecurity testing, which is an aspect we don't strictly consider in our designs, is risky. That risk is compounded with poor communication. These are lessons we must learn from. On the positive side, we now have a stronger relationship with SLAC Cybersecurity than we have had before. Also, while system security was a growing concern for the CSWG, these assessments have increased our attention and perhaps motivated us to more seriously consider the topic.

Note to all other teams: Purchasing new hardware

...

We are closing out the effort to move from EVRs to TPRs for all devices expected to run triggered in the controls system. In addition, we added support to receive the LCLS1 timing from the LCLS2 fiber which sits behind the DAQs XPM as well as support for the DAQ partition bit. 

PMPS Readiness

Margaret Ghaly Zachary L Lentz 

PLC Continuous Integration Testing Pipeline

...

The MEC-U team is also looking forward to meeting with various rack vendors such as Rittal and Steven Engineering in May June as well as future suppliers (tbd). The rack visits with Rittal and Steven Engineering will consist of an initial visit at SLAC where we will provide them with a tour to see our current rack implementations and discuss the best path forward to improving this for future projects. The second visit will happen onsite at Steven Engineering in South San Francisco where we will be able to play/tinker with their Rittal rack hardware to obtain hands-on experience with their various solutions. They are encouraging SLAC visitors to bring any hardware we wish to test fit or use to spec out their solutions.  

...

The DC system architecture page consists of architecture drawings depicting the connections from the supply to the end device which can be a component or a DinDIN-Railrail. The page covers which cables to use with which components and links to the released cable drawing page. A example of the architecture is shown in the image below. 

...

We think we may aim to validate these plans at the lab-wide level if possible.

...

Here's an example of a Figma session with a quick mock up for the Compound Refractive Lens.

We’ll be evaluating a number of prototyping tools and processes, if you have thoughts about this topic let us know!

Hannover Messe

Vincent Esposito 

...