Group C interlock board

Procedures

Normal startup

Check dry air is flowing, and all interlock inputs are plugged in. Both chillers and MPOD should be powered but not running. Interlock board should be off.

Test MPOD interlock: With the interlock board still off, start EPICS controls and verify that MPOD channels cannot be turned on.

Setup SVT chiller: Set SVT chiller setpoint using its touchscreen. When done, switch to remote RS-232 control (Menu->Connect Unit->Remote control). Verify watchdog is enabled (Menu->Connect Unit->Digital Interfaces).

Setup FEB chiller: Start FEB chiller and adjust setpoint, then stop the chiller.

Turn on the interlock board: Green LED should turn on immediately; MPOD trip LED (third from top) should clear as chillers start. The other LEDs should stay off.

Turn on MPOD channels through EPICS.

In a screen session, start the Arduino monitor script on hpstracker ("~/interlock/arduino_monitor.py /dev/ttyS0 9600"). It should send an initial status tweet.

Special startup (interlock disabled, no unattended running allowed)

Open the interlock board's enclosure and flip the switch on the interlock board to the "BYP" position to bypass the MPOD interlock (the switch is reachable through the side of the enclosure, but you need to feel around for it) - this disconnects the interlock board's trip signal.

In this mode you can unplug anything you want from the interlock board (RS-232 cables, flow and dry air sensors) without tripping the MPOD.

Turn on the interlock board. The top two LEDs (green and yellow) on the interlock board should be on; the bottom two may also be on.

Turn on MPOD channels through EPICS.

Disabling interlocks on the MPOD

Plug the MPOD controller into the crappy Windows laptop and run MUSEcontrol. Go to System->Config and turn on "ignore hardware interlocks."

There is no way to tell the interlocks have been disabled without testing the trip, or running MUSEcontrol. Do not leave the MPOD in this state.

Reset interlocks on the MPOD

ppa-pc91245:mpod_scripts/clear_interlock.sh

#!/bin/bash
for j in `seq 5 8`;do for i in `seq  -w 00 15` ; do snmpset -v2c -m +WIENER-CRATE-MIB -c guru ppa-mpod1 outputSwitch.u$j$i i 10 ; done;done

Shutdown

Stop the Arduino monitor script on hpstracker (this is not necessary if you don't mind it sending alarm tweets).

Take the SVT chiller off of remote control (Menu->Connect unit->Remote control). Check the setpoint and make sure it's what you want.

Turn off the interlock board and chillers, in any order. This should trip the MPOD.

Remote control

Stop the Arduino monitor script on hpstracker. Start minicom (or other terminal, configured to /dev/ttyS0 9600 8N1).

Commands starting with "F" or "S" will be forwarded to the FEB or SVT chiller, respectively. If the chiller responds with a message, the response will be printed.

Useful commands:

Quit Minicom and start the monitor script.

Programming the interlock board

The crappy Windows laptop has the Arduino IDE installed.

The interlock board is normally powered from a 24 V supply. When the USB cable is plugged in, the board is powered by USB. Do not have both cables connected at the same time. Unplug 24 V  before connecting USB, and vice versa.

Monitoring

Monitor script runs on hpstracker in a screen: run "screen -rd" to attach the screen, and repeat Ctrl-A Space until you see the monitor script.

Start the monitor script:

~/interlock/arduino_monitor.py /dev/ttyS0 9600

Status messages are sent to twitter.com/hpsbot; alarms and alerts are sent to twitter.com/heavyphoton.

The full output of the Arduino is saved to ~/interlock/log, with timestamps. When this file gets too big, gzip it and name it with the current date.

Status format

Example: "Status: T=24.48,RH=-2.18,dew=-273.10,fl=0,fS=1,sS=1,fT=20.0,fSP=20.0,fFl=0.3,fF=0,sC=1,sT=19.94,sSP=20.00,sPw=26,sP=2.05,io1=0.84"

Alarms

Failure analysis

FEB chiller shutdown (overtemp, power loss)

Flow switch trips -> interlock trips MPOD. This interlock is independent of the Arduino, and the MPOD will trip if the interlock board loses power or the flow switch is unplugged.

Arduino lockup

Arduino monitor script will tweet a "no status received" error after a timeout (10 sec).

SVT chiller will go to a noncondensing temperature (20 C) after 30 seconds, protecting the SVT from dry air failure.

FEB will still be protected: FEB flow interlock is independent of Arduino, so FEB chiller shutdown will still trip the MPOD.

SVT will not be protected from SVT chiller trip. Unlikely coincidence.

Interlock board failure/power loss

Power loss on the interlock board will remove the "MPOD enable" signal, shutting down the MPOD.

Arduino is powered from the interlock board, so Arduino monitor script will tweet a "no status received" error after a timeout (10 sec).

SVT will not be protected from a dry air failure. Not likely.

SVT chiller shutdown

Chiller status will indicate a trip, or chiller will not respond to RS-232. In either case, the MPOD will trip.

Coolant leak

Both chillers have low level protection; a leak in either loop will stop that chiller, and the interlock will stop the MPOD.

Flow switch failure

Unplugging the flow switch, or a break in any line of the flow switch cable, will trip the MPOD.

Interlock board description

Functions

FEB protection

FEBs must not be powered without cooling: MPOD trips and does not start if FEB flow switch is below setpoint.

SVT protection

SVT must not be cooled without dry air: SVT chiller setpoint is reset at 20 C unless dry air dewpoint is below setpoint.

SVT must not be powered without cooling: MPOD trips and does not start if SVT chiller doesn't respond to RS-232 or is not running.

Monitoring

Over Twitter: Report system status (every 10 minutes) to twitter.com/hpsbot, and failures (as they happen) to twitter.com/heavyphoton. People can subscribe to text messages from either Twitter feed.

Logging: all system variables, to ~/interlock/log on hpstracker at 1-second intervals.

Remote control

RS-232 commands sent by the PC are relayed to one of the chillers depending on first character: "Sstatus" sends "status" to the SVT chiller, "FRG" sends "RG" to the FEB chiller, etc.

Inputs:

• HTM2500LF temperature+humidity sensor inside SVT box
• Flow switch in FEB chiller return line: relay switch closes if flow is above setpoint
• RS-232 connection to FEB chiller: status, set T, measured pressure/flow/T (for monitoring only)
• RS-232 connection to SVT chiller: status, set T, measured pressure/T (status is used for interlocks)
• RS-232 to PC: commands to relay to chillers

Thresholds:

• Ambient dewpoint inside SVT box < -10 C (set in Arduino script)
• Ambient RH inside SVT box < 10 % (set in Arduino script)
• Cooling system flow > 20 GPH (adjustment screw on flow switch)

Logic:

• Check inputs every 1 s; if any input is abnormal, trip and stay tripped (must reset Arduino board to clear)

Outputs:

• RS-232 to FEB chiller: start/stop, setpoint
• RS-232 to SVT chiller: start/stop, setpoint, watchdog
• Enable to MPOD crate (5V) - always on if interlock board is powered
• Interlock to MPOD crate (0V normal, 5V tripped) - trip turns off all outputs immediately
• RS-232 to PC: print all variables and status

LED indicators

Top (green): power on/MPOD enable - should always be on

#2 (yellow): MPOD interlock bypassed (controlled by switch on interlock board)

#3 (red): MPOD trip signal (OR of flow switch trip and Arduino output) - this trips the MPOD unless the interlock is bypassed

#4 (red): dewpoint not OK (this is an Arduino output)

interlock.ino