Introduction

The release of this newsletter was delayed and then delayed again as at first we were coming back from a winter break and then trying to wrap up our overhaul of the supported device list (SDL).

Future of Cameras

Collected here are some notes regarding ECS's direction for camera development. Please consider these in the context of your current camera needs and let Silke Nelson and Alex Wallace  know if you have any concerns or questions. Our intent is not to disable anything in production or planned, but rather to start to bring some consolidation and uniformity to our vision systems. This is just the start (or is it a continuation) of what will be likely an extended effort to shore up camera reliability and address any outstanding camera integration requests (device and feature).

• Discontinue Pulnix camera usage
  • Identified as problematic, and should be replaced at the earliest opportunity.
  • Currently classified as DO NOT USE in our SDL.
• Deprioritizing CamLink Development
  
  • Halting further development of CamLink including: new feature requests, new camera integrations, bugfixes, and integration of new libraries.
  • Still the only way to get a real "hardware" timestamp on a frame at 120Hz.
• Moving towards USB3
  • Current camera integration requests are mostly USB 3 
  • Prioritizing development of solution for proper timestamping but it seems OK for now (OBO issues).
• See SDL list.

Getting Help from ECS, Seminar

We had another control system seminar in January all about ECS scope and how to use our Jira system to request help from the team.

We think it's a highly valuable presentation for new and veteran LCLS employees to understand what ECS can do for them.

Getting Help From ECS Control System Seminar

All the other control system seminars are posted here: Control System Seminar Series

HE Controls

ECS presented XES controls design status for the February Director's review. The review focused on preparation for CD2/3 with focus on plan, design status and resource profile. The team demonstrated that as per December 2020 DOE Status review recommendation that lessons learned have been addressed and being implemented. 

The team also showed the plan and ramp up towards CD2/3 now that we have secured the resources needed to execute the project.

The review committee observed in their reports that our designs appear to be sufficient at this point. 

Currently, the team is focused on preparing for the upcoming DOE Review in March.

Controls ICD under review

We recently completed a draft of an ICD for experiment controls for HE and it turned out so well we're thinking about making it a general purpose ICD for ECS to any and all projects going forward. ICD stands for Interface Control Document, and in systems engineering it can have a varied definition. Some projects use ICDs as a way to describe the physical and functional aspects of an interface between two subsystems. In HE and MEC-U's case the ICD will be used to describe the project interface between subsystems, clarifying who has what scope, and what information is needed from ECS and the respective other aspects of the projects.

It's a good document to read and understand because it clarifies what ECS will do in the HE and MEC-U project, and also in general.

 LCLSII-HE-1.4-IC-0488.docx

ATEF Development

We are working on refocusing our ATEF development efforts, with a primary target of passive testing of control system devices.

The "passive" portion of this means that it will be fully-automated and non-intrusive (will not move your motors or otherwise cause a PV to change).

We are looking to create a configuration GUI for easy specification of these checks and a report generation system for all results.

Future development will include: active tests (guided, with humans in the loop) including integration with bluesky, a synoptic for viewing the atef-reported status of all devices, integration with Grafana, views of devices in typhos/hutch-python at a given time in the past, and many other things.

Alarm System Campaign Kickoff

Federica Murgia has kicked off the design and implementation of the alarm system at LCLS.

The NewALarMSystem, developed by the TID CDS Advanced Controls group, is designed for the availability, integrability, and extensibility of the alarm systems at SLAC. 
The installation of the NALMS will start from the FEE in the coming weeks. Future updates will be posted as comments on the Jira ticket.

The scope of the campaign is working to capture and implement alarm thresholds, and hierarchical summaries of alarm states throughout the control systems, using NALMS.

What does this mean? For example, if there is an error with an encoder readback, and it will prevent motion of a diagnostic element, that error will be flagged in EPICS as an alarm, and that alarm could be propagated to a very high-level summary display for operators to understand and trace. So at any given time an operator could see for a given area (like the FEE), if there are ANY issues with mechatronics, and then which component has an issue, and then which sub-component has an issue.

This will not only make our operations more robust as issues can be discovered earlier, but also pave the way for us to obtain real metrics on device reliability and availability.

Experiment State Tracker

One outcome of an LCLS upper-MGMT retreat in December 2022 was the determination that we need more data on how beamtime is spent. Having these metrics will begin to give us an understanding of the impact of various changes we might make to operating procedures, software, hardware, beamline configurations, etc. etc. Basically everything...

It should be noted that this system is a very high-level measurement of a very complex system. Our expectation is to see some correlation of beamtime spending changes to any other changes, but how we will determine a cause is another matter. At the very least it is a start from which we can develop a more refined system health measurement.

The first pass at the state-tracker was assembled by Ken Lauer and Zach Lentz. Further development is owned by Mike Browne with Sebastien Boutet as the stakeholder. If you have any suggestions for features or changes to the state tracker please reach out to Sebastien. If you notice any bugs or issues with the tracker please issue a Jira ticket and let your PoC know.

Grafana Public (login-less) Dashboards

If you have been following the LCLS-II superconducting linac cryomodule cooldown status, you have (perhaps unknowingly) already been using our new non-interactive public-facing Grafana dashboards.

The current list of available dashboards can be found here:
→ https://pswww.slac.stanford.edu/swdoc/ecs_dashboards/ ←
These public-facing dashboards are non-interactive rendered images of the full dashboard. They are accessible to anyone on the Internet, with no SLAC login or affiliation required.

If you have additional dashboards you would like to see listed, please let us know.

ECS Supported Device List

The supported device list (SDL) is a primary component of our Confluence documentation. The original intent of the SDL was to document what kinds of devices users and other LCLS teams could bring to LCLS that could be integrated with the control system (I think). Then it sort of morphed into a list that tried to capture details of any device we had used with the control system. Things like manuals, datasheets, self-written user guides. Our unwritten rule for device support development was to create at least a page for each new device. This resulted in some pages being perfectly sufficient, others being woefully sparse, and still others excessively verbose, nearly to the point of duplicating a manual. Furthermore the SDL became a sort of Interface Control Document (ICD) between PCDS and other teams. Sometimes we might ask, "is it on the SDL?" and if the answer was no, then we didn't support a device. Other times the answer didn't matter.  Someone perusing the SDL might occasionally find a listing had a part number, and then place an order without another check and find later that it was a bad part number. Other issues included perhaps ordering a piece of hardware they found on the list, only to find that it had been ordered with a different interface than we had seen before, or any other variety of divergent configurations. There were problems. 

So we set out to revamp this crucial piece of documentation. You can see some of our development notes here: 2021-11-12 ECS PDG Meeting Notes 2021-11-19 ECS PDG Meeting Notes.

A few months later and we're nearly done with the overhaul. Of course this is really just the start as not every device has been captured in this pass, and new devices will need integration as we go along. Most of our work focused on stuff that had been in the list before. We defined a standard way of creating new device pages (called stubs), created an easy to use template button, a page labelling scheme to organize our content, and leveraged some cool Confluence macros. We ended up with a collection of stubs, which we then worked on properly organizing (highlighting another area of Confluence that needed attention, our Subsystems area). The stubs have also given us an excellent structure for further organization of our help-guides and notes on particular devices. 

Other notable features about the new SDL include: global scope, and the ability to create arbitrary organizations of stubs. The former means anyone anywhere on confluence (in any space), can make a stub, and if they use our labeling scheme, we can include the stub in our lists, meaning our stubs can be merged with other groups' to form a lab-wide view of supported devices. The latter meaning we can organize stubs according to any category we create a label for. As an example, a temperature controller might be used for multiple categories, timing system stabilization and sample delivery instrumentation. By applying each subsystem label, plus one for "thermal controls" we can then present this device in any of those contexts. So anyone perusing our SDL could find that temperature controller multiple ways, meaning there's a better chance of them locating information they need quickly.

Finally one of the best features in the new system is the indication of support status. The previous SDL had to maintain information on all devices in production, even (and maybe even especially) the ones we might have wanted to see retired. This would lead to confusion as other people viewing the list might misunderstand that just because something was on the list didn't mean it was still "supported." Now the SDL uses a specific label for Long Term Support, End of Life and Do Not Use so it is crystal clear if the device you're looking at is really still supported or if we're just keeping a stub around to make sure we never buy another one... More explanation of these terms is available on the SDL page: Supported Devices (SDL).

PS. As the newsletter editor I want to highlight that the SDL overhaul was a herculean team effort and a bit of a grind. Thank you to the team for your dedication to making real progress improving our documentation!

QRIX

We have been busy with testing, checkouts and install of QRIX endstation Controls. The schedule has been pulled forward with tasks that needs to be completed/ready in preparation for Toyama's visit to install and checkout the spectrometer arm.

All the in-vacuum stages have been functionally tested prior to installation.

Installation and termination of 80% of vacuum control components (DRLs and cables) are complete and 50% of motion controls components.

The qRIX Motion and Vacuum PLC are up and running with their IOCs and vacuum screen live and added to RIX LUCID screen.

DC System Procedure Updates

The DC Power confluence page has been updated to include a section on the EEIP standards and processes. The EEIP certification section gives an overview of the EEIP requirements, EEIP reports and documentation required before powering up a new system. 

We have a new DC System Checkout Page which can be used as a template for completing DC Certification of new systems and modification to existing systems.

We also have a new DC List of Certification Page which will be used to update any new DC certification and EEIP reports. This will help maintain a database of all the DC certification.

The DC power confluence page has an update on the section Example Component Panels. The section has updated drawings for component PLC & motion DIN Rail. Additionally, a new document has been added for points of contact to remember while doing DC connections on DRL.

Vacuum Control System Architecture Expansions

Adding a new variable leak valve, VAT Series 590 dosing valve, to our supported device list. Check the confluence page here. This valve has integrated controller with EtherCAT interface.  One more member to our increasing EtherCAT family! This valve will be installed on TMO vacuum system soon.  We will also test two new gauges during March. Watch next month ECS newsletter for the announcement!  

MEC-U

Project is on track! This month we are busy with gathering requirements for kicking off the control system design.  Continued with LLNL and LLE collaboration meetings on MEC-U laser control systems.  We had 4 successful information exchange sessions in the past two months.  We will have LLNL & LLE onsite visiting during the next week.

New team members!

Jira

We're getting better at Jira. Check it out:

Our unresolved tickets grew by about 50% from December to the end of February. We're keeping pace, and perhaps with our new resolution workflows we'll start to cut down on the complete backlog.

One thing to note is we'll soon cross a point where it is a good idea to check Jira for prior tickets before submitting a new one. Much like Stack Overflow, Jira tickets are a kind of knowledgebase. 

Note, the list below might have some tickets on it that were resolved in an earlier period. Due to a confluence workflow change some statuses were changed to "Closed" or "Done" again in January. This issue only appears to affect this Confluence-Jira plugin.