Blog from March, 2021

ECS Platform Development Newsletter March 2021
Alex Wallace posted on Mar 31, 2021

Introduction

This is the first ECS PD newsletter. Hopefully we start putting these out on a monthly basis to take stock of the improvements and work we're doing, and let the rest of LCLS know what new things are available from ECS! Note, even more was done than is listed here. We gave ECS 2 days to fill in highlights, but everyone is super busy and so this is just what we managed to write down. 

Imaging Systems

Fixed server configuration issues that caused the network interface on the camera servers to periodically reset, disrupting data collection and often worse.

PMPS

Squashed a bug related to how the photon energy bitmask is generated to make things more failsafe.

An initialization bug for the preemptive PMPS persists, PAMMs should continue to include a cycle of the Arbiter PLC, although not strictly necessary.

Work began on a batch of PMPS UI improvements including date and time overrides, a more streamlined UI arrangement, bigger windows for faults and requests, among many other things.

After some apparent inconsistencies with veto logic, a deep review of a majority of PLC veto logic was conducted to ensure the PMPS would produce faults only when necessary, and not when an obvious veto device should be keeping things safe. 

Improperly latching "BPTM timeout" faults have been resolved and will no longer prevent beam seemingly at random.

Hutch Python

Fixed bugs:

  • Fixed lxt_ttc oddities
  • Fixed synchronized axis support in general
  • Fixed issue where DAQ would disconnect itself after a scan in some scenarios
  • Fixed various malformed status printouts
  • Fixed malformed metadata from Newport motors and from derived unit conversion signals

Added new tools:

  • InterfaceDevice, InterfaceComponent, and ObjectComponent can be used to use fully instantiated ophyd devices as components in other ophyd devices, rather than needing to duplicate their constructors.
  • FuncPositioner can be used to similar effect as the legacy python 2 VirtualMotor support
  • SyncAxis class as a replacement to the deprecated SyncAxesBase class with a more solid foundation and some extra features

And also added several many new devices for LCLS-II!

UI/UX

  • Added missing soft limit configuration to typhos expert screens

X-ray Optics

March began with some PMPS hacks to protect mirror coatings as the undulators and operators continue to impress us all with increasing beam power. These hacks were formalized in a subsequent upgrade that added state-based control to the mirror vertical axes, and the full PMPS integration (on the L line only). 

XRT Mirrors M1, M2 and  M3 are now connected to the PMPS arbiter and coating protection is now active.

XRT M1 (MR1L3) Upgrade

The goal was to get the old Elmo drives out and install a new Beckhoff din rail drive panel. These panels drive 6 axis, 6 absolute encoders, and interface with an E-621 Piezo Controller. Also, the rail was supplied with a new 24/48 volt din rail power supply that had to be racked and wired, so even the wire job was considerable; shout out to Du, Danh  who did some fine electrical work. On top of these tasks we had to run the software for MR1L2 on a PLC running software for 2 other flat mirrors in the XRT running older Elmo drive software. Other tasks included getting a new IOC spun up, a new control screen, and re pointing older screens to use the updated MR1L2 screens.

The wiring job took two days, so we stayed late on the second day to get the rail powered up. The PDU was reading negative voltage, shout out to Sameen Yunus  for not letting me blow the panel up. We flipped the PDU wires, got positive voltage; got the rail powered on. The Beckhoff 5041s were throwing some error lights and complaining about absolute positions being off.

As Day 3 started, we had a full controls day ahead of us. As I walked in that morning I discussed the encoder errors with Jackson R Sheppard and it was decided to swap the cables. The encoders were a bit funny, we swapped 1 encoder cable out with another that was properly shielded/grounded, and 5 of the encoder errors went away! The emergency STO was checked, and small mechatronics movements began. All axes were moving! except the pitch =(  Alex Wallace Margaret Ghaly , led the charge and began digging relentlessly into the heart of the E-621 serial commands. Ultimately, it was decided that the E-621 controller had left us for a better place. 

Final day! We moved on from the E-621 and implemented the pitch motion directly with the new actuator Hardin, Corey L. had installed. He was excited to tell us all about the new specs the actuator had, and was hopeful that the actuator was such a high performer that we wouldn't need the E-621. With the IOC spun up by Zachary L Lentz in mere 15 min and Margaret Ghaly checking out M2, M3 I was able to focus on the final full movements of every axis on MR1L2. The Y axis didn't like moving at 200 um/s and would error occasionally, but ran without complaints at 100 um/s. Soft limit positions were set close enough to touch each limit switch without having to activate. 

We had about an hour to spare before the XRT was getting scheduled for search. With that Gabriel Castilloand I went and tidied some cables up. Success! 

SP1K1-g_pi Support in the DAQ

LCLS-II DAQ support has been added for SP1K1's grating pitch! This works up to 120Hz and involves triggering the motion PLC directly to get encoder counts out and sent directly to the DAQ. This can feasibly be re-used to get any values we'd like from the PLC to the DAQ directly. The values are timed into the DAQ using an EVR.

Grafana, Logging, Alerting

  • EPICS Archiver data source now allows for alerting, thanks to an ongoing fruitful collaboration with KEK, the Japanese High Energy Accelerator Research Organization. This has been a promised functionality for years, so we're very excited to finally roll it out.
  • A new PLC logging daemon ("ads-log-daemon") was installed to automatically stream logs from any PLC configured on the network.
    • Due to limitations of Beckhoff hardware, the previous configuration did not catch all messages coming from PLCs.
    • This new daemon hooks into the same system that TwinCAT XAE does, and therefore should be able to catch all messages from PLCs - even without a valid TF6310 TCP/IP license installed.
  • A beta-level data source was added to log full IOC output in Grafana.
    • The current (stable) data source only accepts EPICS errLog messages, which not all IOC authors may be using.
    • Due to the extreme amount of log messages generated (>100 million per week), some pre-filtering is required, which will take time to perfect.
  • EPICS Archiver Appliance statistics are now being tracked and monitored.


Solid Attenuators

The attenuator AT1K4 was installed and checked out. Coordinating with TMO, individual filter positions were determined and a calculator IOC was deployed to support easy normalized transmission-based operation.

Double3 Robot

Kaneda, our trusty ears and eyes in the NEH, now roams the halls freely (with supervision). Let the robot revolution commence. Login and access guidance can be found here. More robots following soon in other LCLS areas. Drive safe, have fun!

AR Headsets

 

The RealWear HMT-1 AR headset was at last deployed to the field at LCLS. Headsets have been handed out to SEAs in nearly every area of LCLS for initial deployment and training. The headsets are assigned to each hutch, as well as ECS, SED and Technical support. Check in with your local SEA to get instructions on usage!

You can find more information here.

FIM Commissioning

The first FIM commissioning shift was conducted successfully this month.     Once timed in, we started getting good beam signals on all the wave8 channels.     Tests were done w/ both LCLS1 and LCLS2 timing and some lessons were learned re which registers contribute to the timing delay for LCLS1 vs LCLS2.    There were also some issues w/ firmware or timing signals freezing where we needed to reset the timing.    Will continue to look into this.    We'll also be looking to do some improvements to integrate the FIM screens into TMO home and to provide some FIM specific aliases for Wave8 PVs which are of special significance for the FIM.

makepeds

The newest version of makepeds lives in R1.8.5 of engineering_tools. Changes to prior versions:

  • support for the FFB: specify with -F or by specifying a queue in the new FFB (ffb[h/l][1/2/3]q). Please ask your ECS POC which queue is appropriate - this will typically be the same as used for data processing.
  • data processing is 'local/no queue' by default. To use the queue, you need to specify '-q <queue>'. We recommend the queue only for epix10k pedestals as this allow for parallel processing, speeding up processing.
  • geometry option: '-g <experiment_to_copy_from>' will copy the geometry file from that experiment to the specified one with correct permissions and updating the 'history' as well. You will need to pass a run still so that detectors that need a geometry can be identified. Please note that the Jungfrau detectors also need the offset-files which are not yet included here.

Jira and Issue Reporting

ECS Platform Dev began exploring the use of Jira as our future ticketing system. We developed an embedded form for submitting issues to our Jira backlog, which can be found in various confluence locations. The goal is to make a very simple and easy process for people to request help from ECS PD. Ultimately we are looking to embed this process directly into our GUI so control system users can effortlessly report an issue, and provide all the helpful context we need to troubleshoot issues. An additional goal is visualizing the ECS PD work queue. Using Jira we can easily embed our backlog in confluence so anyone can see where their ticket is in the queue. So far, our backlog is filling up with tasks from the XBD team as well as our own work. The next step will be to really apply Agile processes to organizing and prioritizing our work items.

MEC-U CD-1

MEC-U successfully passed CDR review on March. It is a critical milestone achievement for the MEC-U project.  Next goal for ECS is to get controls WBS, WBS dictionary, and EOB ready for Director's review and Independent cost review. 

L2SI project 

For March we closed these activities. Congratulations to task owners! 

EPICS: Engineering and Design for FEE lockup

EASE: Engineering and Design for Component Assembly

Exit Slit: Engineering and Design for Assembly

We will get back to TMO DREAM starting April.  Near-term goal is to complete final design of 80% of the DREAM control systems. Long-term goal is to get ready for installation by April 2022.


PAMM 3/23/2021
Alex Wallace posted on Mar 23, 2021

Veto Logic Review

Recently it has become apparent that veto logic through all the PLC systems needs to be reviewed. I think the initial documentation for the PMPS did not really focus on veto logic and so it wasn't consistently implemented. Additionally, many of the veto devices have only recently been added.

PLC Project List

  • xrt-optics
  • lfe-optics
  • lfe-vacuum
  • tmo-optics
  • tmo-vac
  • tmo-motion
  • rix-optics
  • rix-vacuum
  • rix-motion

Notes

  • lfe-optics has no components downstream of a stopper
  • tmo optics needed m2k4 and m3k4 vetoed, and an mk1k1 veto for the entire line. the preemptive systems was not in use. Redeployed the code with an online change.
  • reviewed tmo vac with Allen D Pai, did a find all on ffo1, 2, 3, 4 and verified everything using ffo2 is supposed to be vetoed by m1k1 and st3k4 and arbiter2 is also vetoed by st3k4 and m1k1 and is only being used by stuff downstream of st3k4.
  • Allen D Pai confirmed all devices in tmo-motion are downstream of st3k4 and st3k4 is being used a veto for the reactive a preemptive systems.
  • Jyoti, Joshi Margaret Ghaly and I checked optics and motion and confirmed preemptive and reactive systems are being vetoed properly.
  • kfe vacuum received a thorough review of veto logic well before this. Another quick check by Jyoti, Joshi and Margaret Ghaly confirmed things looked OK.


DG2 STP 1 and XRT-M2 Out Fault

This is an old one. What seems to have happened is the code for the XRT-M2 In/Out signal to the AMPS was dropped from the XRT HOMS PLC. The result is this fault cropping up because the A signal as shown above was always false. Before concluding this code was dropped unintentionally we went through the process of tracing this signal from the XRT HOMS PLC out to the link node in B999, in the mezzanine. At first I (Alex Wallace) tried checking the link node input screen, but did not see the signal changing when I toggled it from the PLC. This lead me to believe something had happened to the wiring and after asking Waters, Nickand Gabriel Castillo to trace the cable and finding nothing wrong I realized that I was confused about which signal I was watching in the link node display. It turns out there is a vestigial HXX:HXM:MMS 01:INBEAM signal that wasn't toggling but I was watching and lead me to believe something was broken. After realizing this and checking HXX:HXM:MMS 01:B instead I found everything was OK, the signal was propagating. Then following up with the engineers working on the HOMS PLC to confirm the IN/OUT code was back in the project I concluded that things would be fine at the end of the PAMM and moved on with life.


eV Bitmask Failsafe Did Not

Bill Schlotter reported that during some beamtime on the K line the undulators were set to an unusually high K value. This put them out of our current setting for active range (PMPS Photon Energy) so no seed undulator could be found, resulting in a calculation for eV of 0. The actual effective eV was around 600eV. The photon energy bitmask was then set to 2#0000_0000_0000_0001 which was inaccurate and problematic as it circumvented some faults. To fix this issue I (Alex Wallace ) changed the bitmask logic so a value of zero will result in the current photon energy bitmask being set to 2#1111_1111_1111_1111.


2.2 Mono DAQ Encoder (SP1K1)

We have been planning for a while to have the RIXS optics PLC send its mono grating pitch values directly to the LCLS2 DAQ. This was deployed this PAMM, through a joint effort between Zachary L Lentz, Christopher Ford, Margaret Ghaly, and Jyoti, Joshi. The encoder can be selected in the DAQ, at which point it will send triggers directly to the PLC, which responds appropriately up to 60Hz. The plan was to respond appropriately up to 120Hz, but this issue was not caught until nearly the end of the PAMM. There will be follow-up work to get this to work properly at 120Hz.

2.2 ZOS to Rix Optics PLC

Through Joint effort between Zachary L Lentz, Jyoti, Joshiand Margaret Ghaly the ZOS is now controlled by the rix-optics plc instead of rix-motion plc. Zachary L Lentzremoved all the ZOS code from the motion PLC and rebuilt the IOC. Margaret GhalyAdded the ZOS code to the rix-optics and rebuilt the IOC. Jyoti, Joshitested and checked out that the ZOS is working, no interplay between ZOS motion and gauges were found. 

2.2 Single Point Operation PMPS

Margaret Ghaly Implemented SPO PMPS Pitch faults when the MR1K1 bender is out of safe range as requested by the scientists. The bender soft limits were also changed but not the PMPS Limits. The changes were tested and verified by Jyoti, Joshi and Margaret Ghaly and the Scientists who approved the changes.

MR1K2 Overcurrent Investigation

Margaret Ghalynoticed that the x-down terminal on MR1K2 had an overcurrent warning. Zachary L Lentz joined in and noticed that the terminal had a "short circuit" error. Sameen Yunus investigated more closely with a multimeter and confirmed the short-circuit, and that it was on the terminal side (the motor seemed fine). The next day, after some thought, Margaret Ghaly went into the FEE ready to replace the terminal, but we found that everything worked again with no errors once the phase pins from the motor cable were removed and re-inserted. We tested a lot in this state to make sure things were really OK because we didn't quite believe it, but we ended up running with no swaps. We are continuing to watch this but it seems fine.


Motion States Fixes (BPTM Timeouts)

Zachary L Lentz fixed a bug on all the common components where under specific race conditions the states function block could get stuck in the "finishing" state, causing a permanent "BPTM timeout" fast fault. This has been resolved, deployed, and tested.


MR1L0, MR2L0 Coating States

Margaret Ghaly implemented a states selector block for the FEE HOMS mirrors Y-motors, complete with PMPS support and eV request states. Zachary L Lentz assisted in the debug. Now we can tell at a glance in the screen which coating the Y motor is at. Further work was requested to also automatically set the pitch on a coating state move, but this has not yet been implemented.


RIX Gateway Connections/Config

Jyoti, Joshi noticed some problems with getting PVs into RIX. Zachary L Lentz fixed some configuration issues preventing the optics PVs from reaching RIX and also a separate issue resulting in duplicated PVs, and Omar Quijano fixed a switch configuration issue preventing many of the RIX operator machines from communicating with the gateways at all.

Spinning Rotor Gauge IOC issues

Nolan Brown fixed a bug where the Spinning rotor gauge IOC being used for the GMD and XGMD would not reliable follow given configurations. This appears to be resolved but we are keeping an eye on the both gauges to ensure that they continue to behave properly. A second issue in which the ball would reset to zero Hz when the configuration is changed is still being investigated. MKS has been contacted for support but this behavior may not be changeable.


EM1K0, EM2K0, switchable power supplies

Nolan Brown is working on adding switchable power supplies for the local controllers on EM1K0 and EM2K0. Task is stalled due to a need for additional parts. Parts have been ordered and are in transit.


XRT HOMS MPS Fault

Margaret Ghalyterminated the fault connection from the XRT HOMS PMPS. The Fault was tested with Jeremy Mock, who confirmed that he can see the signal when it is toggled.


TMO Vacuum

Allen D Pai updated the PLC code for the LAMP vacuum plc with etherCAT bridge communication. The EtherCAT bridge logic was updated so more items are checked so confirm that the etherCAT bridge is indeed sending valid data. The FIM HV enable Digital output was properly configure from the PLC side to send a +24V signal.