Make sure the timeline is not too rushed when the alignment team comes to align motor stages to specifications on hard limit switches.
I spent a week ahead of time inspecting Test absorber and getting pictures of how to connect and disconnect the cables. Metal housing has a 12 pin Burndy connector attached.
Pictures of test absorber.
The day before the meeting, I went through the pinout of all my motors and stages to figure out which wire went to which pin. I would double check what type of limit switch your stage has and adjust accordingly. I had Baumer limit switches ( please read about the Bane of Baumer). They do not use the standard 24V supply because the specs say it has a 15VDC limit. After talking with Zach, I learned that the two input terminals on the EL7041 have a digital input of '1' at anything higher than 2.5V. Therefore, I used a 5V supply for the Baumer switches. I tested them out with circuit breakers to make sure the LEDs would light up on the EL7041, thereby registering that the 5V switches worked correctly.
I spent time thinking about how to disconnect the cable within the metal housing for future repairs. Directly pinning out onto the housing would require the technician to unscrew 6 screws and rotate a metal panel through the metal housing and rest it somewhere below. I emailed and called Lupe to get some technician help with terminating the test absorber. I had a drawing of my wire and what color wires would go with what pin on the housing. We walked over to inspect the test absorber. Our collective decision with the Tyson, the cable tech, and Peter, the vacuum tech, was to create a straight through dongle that would allow easier connecting and disconnecting of the motor cable. Tyson took the metal housing to his workshop and started working. I made the mistake of waiting for him to come back. I should have scheduled with him to call me back when he was done. When I went back to go eat lunch, he was waiting for me. We went through the termination process together. I initially explained it and he said that I could leave and he could do it on his own. I wasn't sure and stuck around. After a while, I asked him which pair of wires was for the positive and negative limit switches. He asked me, "There are two?" I was glad to have stayed and made sure the cable termination was done properly. I think this step help me establish rapport with Tyson. When we terminated the cable, and stuck on the metal housing, the connector heads did not match so Tyson went back and re-terminated the metal housing. It was the wrong cable to re-terminate.
I had ordered a din rail with parts from Jing's development Beckhoff components for this XTES project. They did not turn it around in half a day. I only ordered the parts and did not have any wiring on it. I also ordered the +24V and +48V Phoenix Contact power supply plugs. When I received the plugs, they were in the reverse order and the +48V connector did not have all 4 pins. I later made the mistake of turning plugging into the wrong two pins on the +48V supply. I went back and got it remade soon after with all 4 pins even though two of them were not connected to anything.
Using the supplies outside of the office bullpen, I was able to cable up the power supplies and some extra circuit breakers to make the test absorber motor controller with Beckhoff parts. I asked to borrow one of Ernesto's Laptop's and installed Twincat on it. I asked Zach, Maggie, Rajan, and Jackson on how to create a motor PLC to move a motor. They graciously took the time to go over every step and help me debug some issues. In two days, I had learned enough to move a motor with the PLC and connected all the virtual wires to make the limit switches and brakes work correctly. One thing I am proud of is creating a switch (by using the circuit breaker) to turn on the motor axis in Twincat and disable the brake. It proved useful in teaching Peter and also was a good safety check to disable the motor when not in use.
I called Peter and let him know when I was coming in so he could help me with finding
I brought all of the equipment
When I needed
Observe for mechanical clues on where to strain relief and run wires that don't collide with moving parts or get wires pinched. <- most work was here!
Establish rapport with your techs. They will make your life much easier! The more they can do the less you have to do.
Make sure to double check their work. It saves time in the long run.
Don't take shortcuts and let Lupe's shop know when the cables are not working. The quick and dirty method can have consequences.
Using the wires and ferrules in the office is much faster and easier than asking lupe's shop if the drawing has not been made.
Having 48V circuit breakers would be much safer when turning things on and off when redoing EL terminals connections on the fly.
Can never have too may rail mountable relays. I like to click buttons instead of clicking keys. I like having master power switches for both 24V and 48V.
Bringing some zip ties and EL terminal drivers would help. I should get a fanny pack for this purpose.
I would like Tyson to reterminate the dongle connector and the Motor cable on the test absorber so each direction has the same connector. I had to order a special cable for testing the test absorber motor with the metal housing off. the metal housing needs to be off in order for people to access the limit switches and measure the travel distance.
Coming back from the summer shutdown, the XRT HOMS had a couple of issues.
- Pitch piezos weren't activating properly.
- Gantry differences appeared on M3H and M1H.
- Some EPICS PVs were still missing
Fixes and Notes
Piezos weren't working
This is an outstanding HOMS issue. Deadbands govern when the pitch state machine transitions from stepper to piezo. These deadbands are adjustable from EPICS. Their values are reset to zero following a power-cycle of the PLC.
The values have to be restored manually at this point. I wrote a little script and squirreled it away in the iocBoot directory for the xrt-homs IOC.
M1H Stepper Tuning
M1H still wasn't behaving. Really strange because it looked good earlier this summer. I modified the pitch control block to be more specific when logging an error in the coarse move step. Uncovered that SmoothMover was throwing an error, 0x4B07, or move timeout. It seemed the drive deadband was too large, so it was giving up before NC was happy. I can't change that deadband easily (need to connect directly to the drive), so I opted to widen the Target and Position values in the NC to avoid the error for now. Now the absolute move blocks in SmoothMover don't throw errors so the transition to the piezo succeeds.
I am still quite puzzled by why this cropped up now...
We'll have to tighten the drive deadband at some point. They can get to about a urad accuracy.
After the power-outage, the system came back, but the drives had errors. When the drives have errors they are not allowed to have any power. Until the errors are clear the system may relax mechanically, leading to small gantry differences.
I could automate this recovery a bit and try a few times to clear errors and recouple. Maybe in another life. M1H had acquired a nice 0.3mm of gantry difference with the power off. Fixing this is simple, I use our DECOUPLE PV backdoor, reduce the gantry difference by hand, and recouple. dusts off hands
I decided it would be worth 10 minutes to accumulate some archiver links, and put them into the HOMS Engineering and Ops Notes page. This makes it easy for us to assess the history of the system.
Missing EPICS PVs
I noticed that M3H gantry difference wasn't present in the archiver and purple on the screen. No good. Checking the st.cmd the culprit turned out to be a failing modbus port setup. Float ports can only handle 125 words, so the PVs trying to access the M3H gantry values were denied because those values were available at addresses > 125...
The fix is to just open another Float port (float port 2), change the PV address offsets, and go on living your life.
M.C. Browne alerted me to this really interested document produced earlier this year at LCLS. Give it a read!
The XRT had several disabled axes and nearly broken chassis that were causing all kinds of hackery and finagling for ops as a result of rushed commissioning. To make things better we decided to go through each chassis 1 by 1 and fix outstanding issues, enabling all axes, including bending. This post goes over what was fixed and changed, and why.
- Enable all axes
- Fix all chassis
- Fix all cables
- Verify motion
- Verify all drive settings and capture on google drive
- Improve gantry tuning and control
- Misc improvements
Enable all the axes
I made a benchtesting procedure, while verifying the first chassis myself. I then trained Ivan on how to verify chassis on the bench and field. He then worked independently with Danh Du to complete the checkout for M3. Danh then completed the verification for M2 installation with May Ling.
With each verified chassis we had a solid reference for finding mistakes in cables. A loose wire on the bender motor cable for M1 was discovered. Other errors in the chassis were discovered during benchtesting. Sorting out the problems after chassis installation would have been painful (that's what we tried to do originally).
HOMS Chassis Bench testing Procedure
XRT HOMS M1 work log:
XRT HOMS M3 work log:
XRT M3 HOMS ELMO CHASSIS work
Bench test Chassis #1805527 (originally installed in XRT M3)
A dead piezo!
While recommissioning all the XRT HOMS we encountered a piezo failure on M2H. For no apparent reason the piezo died, and we had to scramble to replace it. Actually Corey and Daniele replaced it, while Teddy and I completed verification of all other axes. We don't understand why it failed. Here's the voltage from when it failed:
Improve gantry tuning and control
When moving the axes in gantry a gantry difference would accumulate over time. This would ultimately lead to a parasitic pitch. Each axis would move in-sync pretty well, but it might give up moving before it actually reached a final target, and each drive would give up on its own. This lead to a gantry error that would vary between +/- 10um. Not very good for pointing to 30nrad!
Each gantry axis had a large settling window, and extremely short settling time. By adjusting in the direction of perfection the axes were able to settle a bit more, leading to a more accurate final position. Additionally, by setting the holding current to something other than zero (0.5 A) the axes would also compensate for any other drift and maintain a sharp encoder position. Now the only concern is any heat making its way into the rest of the system. We'll see how that turns out!
After some testing we found that the gantry error was being maintained over several moves to <1um. A significant improvement.
Oops, looks like there is some weird drift there... Turns out when we added some guards for position lag monitoring they tripped the axes off due to some rough handling of the mirror enclosure. Resetting the axis did the trick:
But this means we need to implement the EPICS interface for resetting such faults easily, and diagnosing the problems.
EtherCAT Sync Groups
What are they?
EtherCAT frames have a working counter (notice WC anywhere?) variable that is decremented by 1 for each slave in a sync-group. The idea being, if any of your interfaces on the ethercat network aren't reachable, or there is some other problem, the working counter will be > 0 when it reaches the master. The master then invalidates the entire bus, moving all of the ethercat devices to a non-operational state. Usually if there is a problem with the bus, the slaves downstream of the problem will notice, but any slaves upstream of the problematic one will not know, so the master must tell them something is wrong. That's why we have the working counter. Also the PLC code pays attention to the working counter (when it is programmed well), and state machines will go to a safe state if the WC > 0.
You can configure your ethercat bus to include different slaves in different sync groups. This makes your design robust against functionally separate faults. We can essentially create firewalls in our ethercat bus, so if one branch of our topology has troubles, others remain functional. For the HOMS this means we can assign branch 1 to M1, branch 2 to M2, and branch 3 to M3. If any of the branches have an issue, only that branch will be affected. The other branches will continue to run.
Application to the HOMS
To set this up you have to have a star network. I designed the HOMS PLC to use this topology. Each EK1122 gives two branches apart from the default line. Two EK1122 in the XRT config is overkill but whatever.
Open up the IO tree and look for the SyncUnits item:
Expanding the item shows the sync groups I established:
One for each mirror. Selecting M* shows the devices in the sync group in the left hand pane:
Only devices in <default> can be added to a sync group.
Once this is done the XRT HOMS will be more robust, it would be inexcusable if M1 or M3 took down M2 operations and visa versa.
this is a test news item posted 30 March 2008