Introduction
This edition is extra special as we have for the first time a section for LCLS IT recording some of the improvements and maintenance activities of Omar's team!
Safety Stand-down and EEIP
The newsletter is being released approximately 1 week later than we originally planned due to the SLAC-wide safety stand-down on Wednesday 5/4. While the circumstances prompting the stand-down are not good, the outcome of the activity was positive.
ECS took time to refamiliarize ourselves with the WPC and electrical safety aspects of our work. We also spent some time collecting data from the division on (over)work, including consecutive hours, and days worked, as well as peak number of assignments in an effort to better understand the issues Mike Dunne mentioned in his all-hands.
ECS would like to take a moment to emphasize the importance of EEIP at LCLS and SLAC in general and remind everyone reading this newsletter to look into what EEIP is, and how to make sure your equipment is compliant. You can learn more about EEIP here: https://slacspace.slac.stanford.edu/sites/pcd2/eeip/default.aspx
ES&H Ch8 (Electrical Safety) is everyone's responsibility.
LCLS-II-HE update
The ECS team continues making progress, advancing instruments’ designs, and ramping up efforts with greater engagement from instrument’s leads. The goal for the remainder of FY22 will be to refresh the overall cost estimate as recommended by the committee, while at the same time remaining committed to advancing the design maturity of all Experiment control systems to Preliminary Design Review (PDR) (60%) level. The PDR for controls is slated for August of 2022.
Roles and Team Fully Released
All major roles in LCLS-II HE Experiment Controls have been assigned to ECS team members. Check out the list below to determine who you can talk to in the ECS team for help with HE. If you have any questions please contact Margaret Ghaly
L2HE Project Roles and assignment table can be found here. (This will be the updated list at any time.)
MEC-U update
Rack allocations
We spent some time doing rough estimates of rackspace for MECU, here are some of the figures we developed for your consideration. These don't include the other partner labs in the count, their estimates are elsewhere. Other systems outside of this count include Safety (PPS/BCS/HPS/LSS), Timing and IT infrastructure.
| Full depth racks (42U) | Shallow racks (42RU, for small depth devices) | |
| Laser BTS controls | 4 | 2 |
| Xray BTS | 3 | 3 |
| TCX | 6 | 3 |
| TCO | 4 | 2 |
| Total | 17 | 10 |
Other controls requirements
More controls requirements were developed in March and April. These requirements were specifically focused on EPICS network size, DAQ bandwidth, EPICS archiver size and reliability, as well as the logging systems. High-level requirements are being developed as ConOps are updated/ generated. If there is any particularly important requirement of the control system, or high-level functionality please let Alex Wallace or Jing Yin know.
LLE+LLNL Onsite Visit
The Rochester and Livermore team came to SLAC on March 22nd and 23rd to learn more about MEC operations and collaborate in person on the project. This was the first time all three partner labs met in person to work on MEC-U. The controls breakout session reviewed the operations and the high-level supervisory control software of the Omega laser facility. We also learned more details about the mechatronics, industrial and timing system controls of the LLE system.
Hutch Python Update
Hutch python's conda environment was updated to pcds-5.3.1 near the start of April.
You can read the full release notes here: PCDS Conda Release Notes: pcds-5.3.1
Robert S. Tang-Kong Zachary L Lentz
New Dual-Acting Valve Widget
Dual acting valves are now integrated into pcdswidgets with a nice looking icon to represent the functionality of the valve.
Extensive documentation on the process of making a new pcdswidget was provided as well as scripts for testing widgets in development.
Issues with mechatronics in production
A streak of issues with motor driven actuators in March and April ate a significant amount of beamtime. Many of these issues could be classified as stalls (spurious and real), where a motor is commanded to move and its actuator readback indicates no movement has occurred within some time. Stalls can occur for a variety of reasons, real physical obstruction, or improperly set motor or protection parameters.
In an effort to track and reduce such issues (and motion system issues in general) we built a new Grafana dashboard using data from our logging system (L2SI systems) to organize and track all of these motion system issues in one place. The dashboard can filter between different kinds of motor issues, and specific devices as well as give us a clear view of the most problematic parts our system.
We also met with our Mechanical Engineering counterparts and began a long-term collaboration on motion systems in general together. We plan to use this collaboration to create better design recipes for LCLS mechatronics, and study and correct pervasive issues in deployed systems. We will supply updates from this work as it develops.
If you have any questions about this section please contact:
Lightpath Campaign
Lightpath is a tool that aims to show users and scientists which components are blocking the beam path, greatly expediting troubleshooting and recovery of operation.
Unfortunately the tool has been plagued with bugs and efficiency issues for quite some time. We are planning a substantial rework of the tool, with the goal of making the tool available and useful for all hutches.
A JIRA Epic has been created to gather existing lightpath issues/bug reports. If you have ideas for features or bugs to report, feel free to add an issue to the epic. We will be basing the re-design on requested features, so submitting an issue now is the best way to see those issues solved. We will continue to reach out for input throughout development, so be ready to hear from us!
New ECS PD Campaign: Code Review
ECS currently has nearly all of our hutch-python and related Python source code in the pcdshub organization on GitHub.
We utilize a standardized workflow that improves understanding among our team, provides automated quality and style fixes, and in the end results in a higher quality set of code.
We want to apply the same standards we have for Python development to all PLC, EPICS module and EPICS IOC development.
This will be an ongoing effort over the course of months.
See more details on the initiative page here - ECS GitHub and Code Review Campaign - feedback is welcome.
ATEF Status Update
We are continuing our ATEF development effort, targeting passive testing of control system devices. The "passive" portion of this means that it will be fully-automated and non-intrusive (that is, it will not move your motors or otherwise execute a caput).
We are making progress on the user-facing components of atef for passive checks. This includes a work-in-progress set of GUI elements, allowing for a more straightforward specification of a passive checkout configuration, with a happi device search tool and ophyd device inspector. A preliminary command-line based passive check running tool has also been developed.
Future development will include: active tests (guided, with humans in the loop) including integration with bluesky, a synoptic for viewing the atef-reported status of all devices, integration with Grafana, views of devices in typhos/hutch-python at a given time in the past, and many other things.
shared-dotfiles
What are dotfiles?
- Dotfiles are configuration files that sit in your home directory, like
.bashrc. - Dotfiles set per-user configuration settings for different applications, like the text editor vim or your shell.
- If you have a unix account, you likely have dotfiles in your home directory on pslogin, whether you have customized them or not.
In a survey of users' configurations, we found that many were similar, likely copied and tweaked from one user's home directory to another when they joined ECS.
In some strange form of the telephone game, a bit of meaning was lost between each successive copy, with configuration lines remaining with not a clue as to why they were there.
To make things better we created a shared repository as good starting point for new engineers, or existing ones that want to improve (or just better understand) their settings.
Almost every line contains an explanation as to why it is there, with some offering suggestions as to how to adjust it to your preference.
Here is a link to that repository:
→ https://github.com/pcdshub/shared-dotfiles
It includes a variety of suggested configuration files for various basic tools like ssh, bash, vim, and so on that ECS staff and others who use the same computing infrastructure may find useful.
It also explains a bit about what scripts are available and how to better navigate your environment in its documentation.
If you have ever felt like your command line interface was missing commands, or you wish it had more colors to help with readability, or you just want to make sure you're not missing out on some nice features, check out the repo.
Vacuum System
Added a new supported gauge! Instrument hot cathode gauge IGM401 will be used in TMO. Please check here.
We also gave a seminar on the LCLS II vacuum control system. If you missed the seminar, you can find the slides and meeting video here .
QRIX Controls
The Spectrometer Arm has been installed and checked out. The spectrometer arm has a total of 27 axes installed with coupled and coordinated motion with strict EPS requirements. The acceptance test went very well with Toyama and even were complete ahead of schedule.
The QRIX vacuum PLC code has been merged with that of the spectrometer arm. The vacuum controls integration is complete for all the installed devices.
TXI
Cables were successfully pulled in the FEE for patch panels for both K and L lines.
TMO
First deployment of VAT590 variable valve in LAMP. Jing Yin Govednik, Janez Maarten Thomas-Bosum
This was used with the Magnetic Bottle (MBES) iteration of the LAMP endstation.
TMO requested this very-low flowrate leak valve for experiment gas delivery and vacuum level control in the IP1 chamber.
The valve plate acts as a throttling element and varies the conductance of the valve opening.
Actuation is performed with a stepper motor and controller. The stepper motor/controller version ensures accurate pressure control due to exact gate positioning. The onboard controller and built-in PID loop were integrated into the control network of the LAMP vacuum PLC and IOC, and a custom PyDM GUI was developed as a user interface for this device.
In the future, this device will be integrated into the vacuum control systems of other endstations in TMO. Furthermore, work will be done to allow users to set and tune the PID loop parameters themselves, which are currently locked into the lower level CoE settings. This is because the control loop needs to be adjusted for chambers of different volume, gas flow rate etc.
Documentation page: LAMP MBES Variable Leak Valve Commissioning 3/22
Radiation Monitoring in the FEE
We installed a live radiation measurement system in the FEE! We have been busy integrating this equipment for the past several months.
From four different locations we gained 2 sensor readouts each. Here we see Hard Xray beam energy mapped against dose rates.
A shift was made to bring the rest of the sensors online. Looking forward to collecting more data this month!
The plan is to collect data on radiation in the FEE and EBD to prepare a baseline for comparison with SC beam when it arrives. Ultimately we want to be able to discern between device failure due to normal use and radiation damage.
X-ray Optics Controls Updates
MR1L3, MR2L3, and MR1L4 XRT HOMS Mirrors
- Temperature sensors added to XRT HOMS Temperature
- State Control fully implemented! We can now reach mirror coating positions and move the mirrors in and out of the beam with one click!
MR1K4 FEE TMO Mirror
- Newly upgraded with coating state control.
- PMPS states implemented.
MR3K4, MR4K3, MR2L1, and MR3L1 TXI Mirror
- Site Acceptance complete: TXI SAT
MR4K4, MR5K4 TMO DREAM Mirror
- Granite installed/Mirror moved into the hutch.
- Controls fully checked out: DREAM Install
- Coming: Internal RTD, Pressure, and Flow Sensors.
MR2K4, MR3K4 TMO LAMP Mirror
- Look for some higher performance in the axis control
- Control loops changed and deadbands tightened
Experiment State Tracker and Eloggrubber
Based on popular demand, the Experiment State Tracker has been integrated with the eloggrubber and the possible state list has been extended (with much more detail for 'instrument down' in particular.)
The eloggrubbers JIRA implementation has changed to allow automatic assignment to the instruments as well as posting of the attachments Murali Shankar
Grafana has also been extended to allow the creation of piechart data from the archived Experiment State Tracker output.
Jungfrau [0.5/1]M IOC
The smaller jungfrau detectors which use a Rohde-Schwartz power supply now have an IOC (ioc/common/roving_jf) to make the process of turning them on/off consistent with the various epix-detectors. This IOC will trip the detector should a problem with the chiller flow or power supply is detected.
Jungfrau4M Tripper
Thanks in large part to Daniel Scott Damiani, a tripper has been created for the Jungfrau 4M detector in CXI. This is similar to the CSPAD tripper and will help to protect the detector from intense X-ray by blocking the beam when too many hot pixels are detected. The tripper runs through a combination of an IOC and AMI plugin. The IOC host PVs that for input and output to the tripper and closes the shutter when necessary. The window below can be easily seen on the CXI Home Screen, and can be used to adjust the threshold or clear a trip. From now on, the tripper should always be enabled to prevent burning out any more pixels.
The "ADU Threshold" is the number of ADU below the saturation in low gain allowed before a pixel is considered too high. The "Pixel Count" is the number of high pixels allowed before tripping the beam. Please do not adjust these values without discussing with Mengning Liang or Tyler NT Pennebaker beforehand.
As another form of feedback, the AMI plugin adds an option in the AMI window called "JungfrauTrip" that allows the user to watch the number of pixels above the threshold over time. Below, we can see a constant 20 pixels above the threshold, using the current settings without beam, due to the misbehaving already-burnt pixels. If you have any questions about the tripper, please feel free to reach out to me!
CSU Chico Engineering Capstone Collaboration
This year is a prototype system for determining interaction point component alignment using machine vision sponsored by Diling Zhu. The CSU Chico Engineering Capstone Collaboration project team visited XPP in early March to do some final fit measurements around the interaction point for installation of their project later this year.
The team won Best Project this year at the CSUC design expo (May 13th), competing with ~20 other project teams from the program. The team will also present their project this summer in Dallas TX at a national competition for engineering capstone projects.
L-R Anthony Aliotti Mitchell Cabral Clarice Rucklos Jasmine Nguyen |
The team in front of their winning project and display showing off the robot, their SLAC affiliate badges, and the numerous iterations on the 3D printed camera mount. |
The team hosted a local Girl Scout group and used the collaboration project to demo robotics. |
For the past five years LCLS departments (SED and ME) have sponsored projects with the Mechanical/Mechatronic Engineering departments of CSU Chico for their engineering degree capstone. Each year the collaboration project offers a chance for LCLS to develop prototype concepts, or even to offload a manageable portion of work to a team of 4-5 students and a faculty advisor (for a nominal fee + M&S). These projects offer us a chance to discover talented candidates and provide the students with an incredible experience doing a project at a national lab. If you have an idea for a project, please contact Alex Wallace for details (or if you want help setting up a collaboration with your school). On a final note, this year we are excited to announce that we hired a student from this year's team, Mitchell Cabral. He will join ECS as a control system integrator in June.
HXR Single Shot Spectrometer Updates
Restoration and recommissioning of the HXR spectrometer mechatronics has been completed. The spectrometer has already been utilized in experiments including some in MFX. Below are some images taken from the Orca camera. Some minor updates to the gui have been made including power cycle control as well as quick access to the webcam used for remotely viewing how the spectrometer is behaving.
Full design and motion system details can be found here
Hutch Rack Cleanup Efforts
Progress in hutch cleanup has began with network switch rack organization in both XCS and XPP, below are before and after images taken from each:
XCS:
| Before | After |
|---|---|
|
|
XPP:
| Before | After |
|---|---|
|
|
IT Updates
Spencer, Aalayah I Omar Quijano Otero, Julieth
Network Reorganization
Over the past few months, the CDS-IT (hereafter IT) team has worked to upgrade and simplify the network set up across the NEH and FEH Hutches, XRT, and FEE. Currently, CXI is the only Hutch remaining to undergo the new network restructuring.
The purpose is to provide 100GbE stacking and/or uplinks for maximum reliability and multigigabit access. The first 24x (1 - 24) ports provide 1000 Mbps connection and the next 24x (25 - 48) ports provide 10,000 Mbps connectivity. All ports provide POE+/802.3bt (90W per port) with up to 1500W power budget with 2 power supplies.
Ruckus ICX 7850-48ZP
WEKA Home/FFB Cluster Upgrade
LCLS has 2 WEKA storage clusters: i) provides the home directories and software for all users and staff, and ii) the FFB for the user's experiments. There was a bug with the previous release of WEKA affecting quota. The latest version was installed to fix the quota bug and improve performance. The team is still working with the WEKA team to deliver a feature for users to check their current quota usage.
DSS and MON Nodes
There have been many incidents where the DSS and MON nodes fell into disrepair during an experiment and communication about their state as well as a checklist for the required setup was missing. To provide transparency across the multiple teams, the IT team has created a confluence page that explains the validation methodology and provides a log of the current functional DSS and MON servers. This will prevent the teams from guessing which nodes are available and allow the IT team to fix those that are not available. Coming soon, the IT team will deploy a new image with monitoring that will provide alerts so that actions can be taken in a timely manner.
SRCF Datamover Bandwidth Upgrade
There has been a request by the Data Management team to provide more bandwidth to the SRCF datamovers to keep up with the current data rates. A 2 x 10Gb bond was created to support a bandwidth of 20Gbps. There are 5 servers used by the Data Management team as SRCF datamovers to support the various experiments.
Workstations and Servers Matrix
To organize, clean, and validate the current host database, the IT team decided to check each Hutch for the currently existing online and offline devices. After inspecting in detail the host database, information was gathered with the objective of removing the devices that were no longer required. This process involved the participation of the POCs, SEAs, instrument leads, and the IT team to ensure that the information was coherent. Currently, the team is still working with the various staff to finalize this extensive activity.
Remote Operations Hardware Maintenance Schedule
Although LCLS has devices for making remote operations more efficient, each hutch counts with a Realwear HMT-1 headset and the Double3 robot. The Realwear HMT-1 headset can be used for an interactive hands-free video conferencing call (expert troubleshooting) and documenting by taking pictures/recording of such call. The Double3 is a telepresence robot with video conferencing and remote driving capabilities. There has been a lack of support for these devices, so the IT team is working on providing a weekly and monthly maintenance guide. In addition, each device will be transferred to its respective custodian, and training will be provided to more IT members so that they can provide assistance to the various staff personnel. A weekly status report will be logged for the continued maintenance and support of these devices: ROH Maintenance Logs.
Integrate UED to LCLS Standards
There is an effort in progress to support UED's revamp to the LCLS standard. Now, UED has a main system named ued-daq with multiple screens, which support both the LCLS and ASTA controls. The idea is to integrate the Accelerator Network into the LCLS managed devices to provide the current LCLS IOCs and DAQ systems with the capability of accessing the Accelerator PVs. This step will provide two key functions: i) removal of the UED local gateway and leverage the LCLS gateways, and ii) allow UED to slowly transition their control system to the LCLS environment.
New Ciara Purchase
A new purchase for workstations and 1U AMD Rome servers has been made. There will be an additional 10 control room workstations and 65 servers. The servers will be deployed across LCLS for new DAQ and Control Systems, or to replace faulty ones. The workstations will be deployed across the different control rooms and hutches.
Pit Cleanup
Over the past few weeks, members of TID have been supporting the Pit Cleanup effort. Old infrastructure and non-working devices have been set aside to be sent to salvage. At this time, the team is trying to coordinate a time with the RPFO office to do the radiological survey for all unwanted hardware. Once the survey has been completed with the appropriate paperwork, TID staff will move the hardware to salvage.
Upcoming IT Tasks
The following are short-term tasks the IT team is working to deliver:
- CXI network upgrade.
- Switch tool for user management.
- Diskless server upgrade to provide resilience.
- Diskless monitoring to make sure all DSS and MON nodes are operational
- Extra 20 batch nodes for the FEH queue.
- Finalize the ioc-xrt-rec07 image to validate 10G network.
- Finalize and validate GNOME key binding shortcuts process.
- Deploy NoMachine extended capabilities to support remote operations.
Jira
Jira showed an interesting trend in the past two months.
We saw the highest number of resolved issues in March and in general a leveling out of delivery related issues.
Overview
Content Tools