The release of this newsletter was delayed and then delayed again as at first we were coming back from a winter break and then trying to wrap up our overhaul of the supported device list (SDL).
Collected here are some notes regarding ECS's direction for camera development. Please consider these in the context of your current camera needs and let Silke Nelson and Alex Wallace know if you have any concerns or questions. Our intent is not to disable anything in production or planned, but rather to start to bring some consolidation and uniformity to our vision systems. This is just the start (or is it a continuation) of what will be likely an extended effort to shore up camera reliability and address any outstanding camera integration requests (device and feature).
We had another control system seminar in January all about ECS scope and how to use our Jira system to request help from the team.
We think it's a highly valuable presentation for new and veteran LCLS employees to understand what ECS can do for them.
Getting Help From ECS Control System Seminar
All the other control system seminars are posted here: Control System Seminar Series
ECS presented XES controls design status for the February Director's review. The review focused on preparation for CD2/3 with focus on plan, design status and resource profile. The team demonstrated that as per December 2020 DOE Status review recommendation that lessons learned have been addressed and being implemented.
The team also showed the plan and ramp up towards CD2/3 now that we have secured the resources needed to execute the project.
The review committee observed in their reports that our designs appear to be sufficient at this point.
Currently, the team is focused on preparing for the upcoming DOE Review in March.
We recently completed a draft of an ICD for experiment controls for HE and it turned out so well we're thinking about making it a general purpose ICD for ECS to any and all projects going forward. ICD stands for Interface Control Document, and in systems engineering it can have a varied definition. Some projects use ICDs as a way to describe the physical and functional aspects of an interface between two subsystems. In HE and MEC-U's case the ICD will be used to describe the project interface between subsystems, clarifying who has what scope, and what information is needed from ECS and the respective other aspects of the projects.
It's a good document to read and understand because it clarifies what ECS will do in the HE and MEC-U project, and also in general.
We are working on refocusing our ATEF development efforts, with a primary target of passive testing of control system devices.
The "passive" portion of this means that it will be fully-automated and non-intrusive (will not move your motors or otherwise cause a PV to change).
We are looking to create a configuration GUI for easy specification of these checks and a report generation system for all results.
Future development will include: active tests (guided, with humans in the loop) including integration with bluesky, a synoptic for viewing the atef-reported status of all devices, integration with Grafana, views of devices in typhos/hutch-python at a given time in the past, and many other things.
Federica Murgia has kicked off the design and implementation of the alarm system at LCLS.
The NewALarMSystem, developed by the TID CDS Advanced Controls group, is designed for the availability, integrability, and extensibility of the alarm systems at SLAC.
The installation of the NALMS will start from the FEE in the coming weeks. Future updates will be posted as comments on the Jira ticket.
The scope of the campaign is working to capture and implement alarm thresholds, and hierarchical summaries of alarm states throughout the control systems, using NALMS.
What does this mean? For example, if there is an error with an encoder readback, and it will prevent motion of a diagnostic element, that error will be flagged in EPICS as an alarm, and that alarm could be propagated to a very high-level summary display for operators to understand and trace. So at any given time an operator could see for a given area (like the FEE), if there are ANY issues with mechatronics, and then which component has an issue, and then which sub-component has an issue.
This will not only make our operations more robust as issues can be discovered earlier, but also pave the way for us to obtain real metrics on device reliability and availability.
One outcome of an LCLS upper-MGMT retreat in December 2022 was the determination that we need more data on how beamtime is spent. Having these metrics will begin to give us an understanding of the impact of various changes we might make to operating procedures, software, hardware, beamline configurations, etc. etc. Basically everything...
It should be noted that this system is a very high-level measurement of a very complex system. Our expectation is to see some correlation of beamtime spending changes to any other changes, but how we will determine a cause is another matter. At the very least it is a start from which we can develop a more refined system health measurement.
The first pass at the state-tracker was assembled by Ken Lauer and Zach Lentz. Further development is owned by Mike Browne with Sebastien Boutet as the stakeholder. If you have any suggestions for features or changes to the state tracker please reach out to Sebastien. If you notice any bugs or issues with the tracker please issue a Jira ticket and let your PoC know.
If you have been following the LCLS-II superconducting linac cryomodule cooldown status, you have (perhaps unknowingly) already been using our new non-interactive public-facing Grafana dashboards.
The current list of available dashboards can be found here:
→ https://pswww.slac.stanford.edu/swdoc/ecs_dashboards/ ←
These public-facing dashboards are non-interactive rendered images of the full dashboard. They are accessible to anyone on the Internet, with no SLAC login or affiliation required.
If you have additional dashboards you would like to see listed, please let us know.
The supported device list (SDL) is a primary component of our Confluence documentation. The original intent of the SDL was to document what kinds of devices users and other LCLS teams could bring to LCLS that could be integrated with the control system (I think). Then it sort of morphed into a list that tried to capture details of any device we had used with the control system. Things like manuals, datasheets, self-written user guides. Our unwritten rule for device support development was to create at least a page for each new device. This resulted in some pages being perfectly sufficient, others being woefully sparse, and still others excessively verbose, nearly to the point of duplicating a manual. Furthermore the SDL became a sort of Interface Control Document (ICD) between PCDS and other teams. Sometimes we might ask, "is it on the SDL?" and if the answer was no, then we didn't support a device. Other times the answer didn't matter. Someone perusing the SDL might occasionally find a listing had a part number, and then place an order without another check and find later that it was a bad part number. Other issues included perhaps ordering a piece of hardware they found on the list, only to find that it had been ordered with a different interface than we had seen before, or any other variety of divergent configurations. There were problems.
So we set out to revamp this crucial piece of documentation. You can see some of our development notes here: 2021-11-12 ECS PDG Meeting Notes 2021-11-19 ECS PDG Meeting Notes.
A few months later and we're nearly done with the overhaul. Of course this is really just the start as not every device has been captured in this pass, and new devices will need integration as we go along. Most of our work focused on stuff that had been in the list before. We defined a standard way of creating new device pages (called stubs), created an easy to use template button, a page labelling scheme to organize our content, and leveraged some cool Confluence macros. We ended up with a collection of stubs, which we then worked on properly organizing (highlighting another area of Confluence that needed attention, our Subsystems area). The stubs have also given us an excellent structure for further organization of our help-guides and notes on particular devices.
Other notable features about the new SDL include: global scope, and the ability to create arbitrary organizations of stubs. The former means anyone anywhere on confluence (in any space), can make a stub, and if they use our labeling scheme, we can include the stub in our lists, meaning our stubs can be merged with other groups' to form a lab-wide view of supported devices. The latter meaning we can organize stubs according to any category we create a label for. As an example, a temperature controller might be used for multiple categories, timing system stabilization and sample delivery instrumentation. By applying each subsystem label, plus one for "thermal controls" we can then present this device in any of those contexts. So anyone perusing our SDL could find that temperature controller multiple ways, meaning there's a better chance of them locating information they need quickly.
Finally one of the best features in the new system is the indication of support status. The previous SDL had to maintain information on all devices in production, even (and maybe even especially) the ones we might have wanted to see retired. This would lead to confusion as other people viewing the list might misunderstand that just because something was on the list didn't mean it was still "supported." Now the SDL uses a specific label for Long Term Support, End of Life and Do Not Use so it is crystal clear if the device you're looking at is really still supported or if we're just keeping a stub around to make sure we never buy another one... More explanation of these terms is available on the SDL page: Supported Devices (SDL).
PS. As the newsletter editor I want to highlight that the SDL overhaul was a herculean team effort and a bit of a grind. Thank you to the team for your dedication to making real progress improving our documentation!
We have been busy with testing, checkouts and install of QRIX endstation Controls. The schedule has been pulled forward with tasks that needs to be completed/ready in preparation for Toyama's visit to install and checkout the spectrometer arm.
All the in-vacuum stages have been functionally tested prior to installation.
Installation and termination of 80% of vacuum control components (DRLs and cables) are complete and 50% of motion controls components.
The qRIX Motion and Vacuum PLC are up and running with their IOCs and vacuum screen live and added to RIX LUCID screen.
The DC Power confluence page has been updated to include a section on the EEIP standards and processes. The EEIP certification section gives an overview of the EEIP requirements, EEIP reports and documentation required before powering up a new system.
We have a new DC System Checkout Page which can be used as a template for completing DC Certification of new systems and modification to existing systems.
We also have a new DC List of Certification Page which will be used to update any new DC certification and EEIP reports. This will help maintain a database of all the DC certification.
The DC power confluence page has an update on the section Example Component Panels. The section has updated drawings for component PLC & motion DIN Rail. Additionally, a new document has been added for points of contact to remember while doing DC connections on DRL.
Adding a new variable leak valve, VAT Series 590 dosing valve, to our supported device list. Check the confluence page here. This valve has integrated controller with EtherCAT interface. One more member to our increasing EtherCAT family! This valve will be installed on TMO vacuum system soon. We will also test two new gauges during March. Watch next month ECS newsletter for the announcement!
Project is on track! This month we are busy with gathering requirements for kicking off the control system design. Continued with LLNL and LLE collaboration meetings on MEC-U laser control systems. We had 4 successful information exchange sessions in the past two months. We will have LLNL & LLE onsite visiting during the next week.
| Robert Tang-Kong has joined the ECS Platforms Development team, and will primarily work on the hutch python environments. Prior to joining ECS, Robert got his PhD in Materials Science at Stanford and worked at SSRL as part of their data acquisition pilot project. In his downtime, Robert enjoys reliving his glory days in competitive yo-yoing and playing video games. |
| Peregrine McGehee has joined the ECS Delivery group as the new MEC PoC. His background includes teaching physics and astronomy at Ventura College, working as a staff astronomer at NASA'S Infrared Processing and Analysis Center, and prior to that as a controls engineer at the Los Alamos Neutron Science Center. A few other things that take his time include being a co-chair of one of the Rubin Observatory science collaborations and continuing the study of Balinese gamelan. | |
| Mike Estrada Mike Estrada has joined ECS as the new PoC for the XBD group. His hobbies include weightlifting, seeing live music, cooking, trying new restaurants, and travelling. He likes to fill his spare hours playing video games, visiting with family, and going wine tasting. |
We're getting better at Jira. Check it out:
Our unresolved tickets grew by about 50% from December to the end of February. We're keeping pace, and perhaps with our new resolution workflows we'll start to cut down on the complete backlog.
One thing to note is we'll soon cross a point where it is a good idea to check Jira for prior tickets before submitting a new one. Much like Stack Overflow, Jira tickets are a kind of knowledgebase.
Note, the list below might have some tickets on it that were resolved in an earlier period. Due to a confluence workflow change some statuses were changed to "Closed" or "Done" again in January. This issue only appears to affect this Confluence-Jira plugin.
|
