Introduction

April brought the end of run 18 at LCLS, the first run since the long shutdown.

ECS

Subject Matter Expert Changeup

Subsystem architectures changed hands, the major changes are highlighted here:

Vacuum : Margaret Ghaly → Jing Yin

PMPS : Alex Wallace → Margaret Ghaly

Motion : Zachary L Lentz → Tyler Johnson

A complete SME list from platform development is available here:

ECS Subject Matter Expertise List (SME)

Temporary Point of Contact Arrangements

Experiment areas have been assigned temporary point of contacts while we work to hire more people into the operations team:

TMO → Alex Wallace and Silke Nelson

MEC → Tyler Johnson

UED → Zachary L Lentz and Ken Lauer

Hutch Python

Hutch python received small updates this month, with a big push to get a GUI set up for UED.

  • Deployed four updates of the conda environment. The latest is pcds-4.1.3.
  • Integrated the LCLS-II DAQ into the UED hutch python, and deployed the beta of the standard bluesky GUI.
  • Fix a long-standing set_current_position bug for IMS dumb motors.
  • Revise the standard means of setting up LXT_TTC to be more user-friendly.
  • Add functions in pcdscalc for working with pmps eV bitmasks.

We also did the hutch-python seminar, part 1! See Hutch Python Seminar

PMPS upgrades

PMPS v1.0.0 was finally released and deployed to the L-line (HXR) in the FEE thanks to the efforts of Maggie and Zach. 

  • Overhauled the internal beam parameter structure
    • Updated to a 32-bit photon energy register (was previously 16bits). This greatly enhances operations by enabling more absorption edges to be guarded, and provides better transitions between photon energies.
    • Changed transmission from a fixed-point representation to a floating point representation, and dropped the % EGU, for a unitless 1.0 factor (representation). This is a major step in really finishing attenuator automation.
    • Renamed stopper list to veto device list.
    • Dropped bunch charge from the structure, since there are no plans to govern this parameter via the PMPS at this time. It may reappear for SC, when beam classes go into effect.

For May we hope to see these changes deployed to the K-line, as well as introduction of the date-time widget for the bypass system (and wave good riddance to the ms bypass units). Once these changes are in effect, the KFE gas attenuator should remain in PMPS mode most of the time.

The PMPS GUI also received an overhaul, which will hopefully improve visibility among other things.

RIX

RIX Optics PLC Stability Issues

The RIX optics PLC experienced some instability after some of the upgrade work in April. This caused the PLC to stop on a number of occasions. After some investigation it was discovered to have encountered a floating-point exception, a result of using NaNs in some of the new PLC logic. Once we understood this was the fault, we were able to simply tell the PLC to handle NaNs without exception and the PLC was stable again. Initially we thought this might be an issue with task prioritization, starvation, or PLC resource scarcity, or something we had never seen before, but it turned out to be much more mundane. It has been stable since before the week-downtime, so roughly 4 weeks to today (before it was halting after ~3-4 days).

Monochromator PMPS (whole enchilada)

The RIX monochromator PMPS protection scheme was implemented and deployed thanks to the efforts of Margaret Ghaly at the end of the April PAMM. This is easily the most complex application of the PMPS architecture to date and will facilitate easier operation of the mono.

AT2K2-SOLID Attenuator

AT2K2 was installed, checked out, and integrated with EPICS and the new-style Python-based calculator (as in AT1K4 and AT2L0). The next step will be determining filter positions during beamtime to fully enable the calculator's functionality.

PLC absolute encoder with DAQ timestamping

Zachary L Lentz  completed and deployed a feature to the RIX monochromator PLC system for timestamping and streaming the readback of the grating pitch position from an absolute (BiSS-C) encoder into the DAQ. 

K-line lightpath

The K-line now has a lightpath!

M3K2

RIX M3K2 is now a contributing member of the LCLS society, its young brethren M2K2 and M4K2 will be following shortly to start spewing X-rays down the beamline.

Gateway Maintenance

The gateways were reworked a bit to improve performance for the gas detector PVs:

  • LFE gateway was split into two, now the LFE and LFE-XRT gateways
  • LFE-CAM gateway was split into two, now the LFE-CAM and LFE-XRT-CAM gateways
  • LFE-GDET gateway was split into three, now the LFE-GDET, LFE-GDET-ARRAYS1 and LFE-GDET-ARRAYS2

In addition, we now have six gateway machines (pscag01, pscag02, ... pscag06) replacing the old four gateway machine system (pscag1, pscag2, ... pscag4).

Check out the updated gateway grafana page for a cleaned-up look at gateway statuses: https://pswww.slac.stanford.edu/ctl/grafana/d/PrlcqvJMk/gateway-status?orgId=1

Remote collaboration headsets

The Realwear HMT-1 headsets have been deployed to the field for about a month now, and used on a few occasions. Up to this point we have called them AR headsets, but until we actually use them for AR, we'll call them remote collaboration headsets. Check in with your local SEA to give them a try. They can be very handy for remote troubleshooting.

Vacuum

We are happy to announce that we add InstruTech CVM201 Pirani gauge  and CCM501/502 Cold Cathode gauge to our supported device list. Check out the confluence page for vacuum devices we supported: Vacuum Devices.


MR2L3

XRTM3(MR2L3) was upgraded to a Beckhoff controls! Another Elmo Controller was switched off this month and the drive burden shifted to Beckhoff drive slices. The installation wasn't entirely smooth, with some terminations needing to be re-routed, a limit switch not working, and an encoder cable modification. In the end however, we were able to get the mirror up and moving by the weeks end.

MEC Laser Environment Sensors

In other news, with some help from Tyler Johnson, some laser environment sensors were deployed inside the MEC long pulse laser. Cunningham, Eric Flint was suspecting some volatile humidities affecting the laser, these sensors will help him keep the LPL stable.

IT

Filesystems

The Weka cluster that supports all home, group, and software was upgraded to version 3.11.2 to provide the following functionality:

  • Better NFS overall performance.
  • Number of POSIX groups per user is no longer limited by 16. Now users can be a member of a nearly limitless number of groups. Since POSIX groups are used for experiment data access among other things this is an important improvement.

Any filesystem mounted via Weka has now snapshots enabled (only EPICS IOCs are not mounted via Weka) . This will allow users to recover any mistakenly moved or deleted files.  More information can be found in Folder Backups.  The team is in the process of redefining the retention policy.

LAS Subnet Migration

With the new LCLS-II instruments, the Laser team requirement for networked devices has significantly increased.  Thus, a new subnet was provided to increase from 254 addresses to 1024.

ANA01

The new Lustre Filesystem ANA01 is now in production.  This offline storage has the latest Lustre release 2.12.6 and has 2.5P capacity.  This has allowed the team to retire old infrastructure, such as ANA14 and ANA04, and upgrade 2 more filesystems. This work improved flexibility and robustness for experiment operations.

Backup Server

Newly-deployed backup server psbackup02 is accepting backups from core services like pswww, pspuppet, pspxe, databases and other systems.  Backups are retained in daily, weekly, and monthly increments up to one year in total. This enhances our capability to recover from primary system failures.

New Export Nodes

The 4 new psexport nodes are now in production (data transfer hosts).  These servers have a 100G direct connection to the routers, allowing for faster throughput, and better overall performance.

Hello, Goodbye

ECS has been on a hiring spree lately, new faces are imminent. Sadly we also said goodbye to an excellent teammate, Adam Egger. Adam was the controls point of contact for CXI and TMO. He supported the TMO commissioning and CXI operations, and still found a bit of time to work on jet-tracking. He will be sorely missed.

Another coworker, Slepicka, Hugo Henrique left SLAC late April as well. He was the primary contributor to the pydm project, worked on typhos+lucid, and was a valuable expert on all things EPICS, and software development in general.

We did get to say welcome to another member of the ECS team, Gabriel Castillo, joined us in late March and has been keeping busy learning the control system, putting out fires, and gathering kyber crystals for the next stage of his training.