Table of Contents
Table of Contents |
---|
Summary
In the case of pressure rise on gauges 3259 and 3291 due to a failure of one or more pumps in the DS-DPS system, this procedure should be used to isolate the beamline and stop the pressure from rising further. See the attached document for the full procedure.
In most cases this will isolate the beamline and allow the other remaining operating stage to pump down the beamline and allow operations to commence.
Link to full procedure: FACET S20 DPS Troubleshooting - 03-04-2022.pdf
How to silence the roughing pump alarms in the tunnel
The roughing pumps will make an audible alarm if they are displaying a warning or alarm state. This is how to
Ideal method:
To check the cause of the alarm:
- Open the "DPS full schematic" from the LI20 Vacuum Epics page.
- For each roughing pump - click on the pump icon, then on "More..."
- You should see a list of all the warning and alarm states. If any of these has light beside it, then that is warning/alarm that is causing the problem.
- Make an Elog entry with the warning and alarm states that you find
If you understand the cause of the alarm and the implication of clearing it, then you can go back to the roughing pump control window (with the speedometer) and click on "Reset" to clear the warning/alarm state. You may have to hit reset a few times if it doesn't stop beeping on the the first try.
Alternative method:
If you do not understand why the particular error message is being displayed or if you are not comfortable clearing the error, but you need to stop the incessant noise from driving you crazy - then you can simply unplug the handheld controller panel that sits beside the roughing pumps under the light pipe. If you chose this path, then make all reasonable efforts to ensure that the controller panel is eventually reconnected.
In all cases - please notify the appropriate authorities that you have done something!
Troubleshooting if pumps keep turning themselves off during startup
At the moment you cannot restart a turbopump with the DPSvacWatcher running. It will detect that things aren't in the nominal operating state and will stop the pump (or insert the foreline valve). This will be signaled by an entry in the elog.
Stop the watcher, restart the pumps, then restart the watcher once the valves are in a good state. Alternatively, once a turbo is tripped off then you have ~10 minutes while the watcher ignores that stage. So if the turbo is shut off by the DPS when restarting, just try to restart again a second time within the 10 minute window and you should be fine.
Step-by-step troubleshooting guide for vacuum issues
...
If you get to the green “Everything looks good” box, then this stage appears to be in a good state.
The most common problems with the DPS is the roughing pumps tripping off and causing a cascade of other faults. So the first thing to look for is what stage or stages are faulted, and what roughing pump is likely to blame.
There is some redundancy in the system, so depending on the type of fault, the system should be able to continue to operate well enough to keep the beamline valves open if a single stage fails, but you may not be able to operate any gas sources without the full DPS operational. See the "Pump redundancy" section for more info.
Overview of Operation and Troubleshooting of the the DPS
dstorey_2022-11-3 - DPS Operations and Troubleshooting Summary.pdf
Troubleshooting guide:
Check for your problem in the table below, then see full instructions below that for how to fix the issue.
Symptom | Potential Issue | What to do |
---|---|---|
Roughing pump tripped off | Radiation related fault | See - Resetting a roughing pump fault |
Roughing pump won't start | Error message is present | Check "More..." status to see what error is listed. |
Roughing pump won't reset an error | There is a real problem, or pump has been damaged | See - Resetting a roughing pump fault |
Turbopump tripped off | Most likely related to loss of backing pressure | Fix the problem with the roughing pump/foreline configuration then restart the turbo as described in Restarting DPS pumps. |
Radiation related fault | See - Resetting a turbopump fault | |
Roughing pump or turbo not responsive | Pump is in local mode, not remote | Put pump into remote mode. See Setting pumps to remote mode |
US4 stage cannot be restarted | Probably radiation damage | See - Pump redundancy |
Foreline valve won't open | Foreline interlocks need to be cleared | Reset foreline interlocks from "Foreline Vlv Intlks..." panel |
Resetting a roughing pump fault
This may be indicated by one or more of the following:
- The roughing pump is stopped (red) or disconnected (purple), or
- The foreline vacuum gauge is reading a value >1 Tor and the foreline valve is closed, or
- A warning or alarm is present in the "More..." panel of the roughing pump panel, or
- The beamline vacuum performance on a stage is not as expected, or
- Something else that I have not though of.
Note that in rare cases the roughing pump controller can be scrambled by radiation and it may report inaccurate statuses, so check the pressure readings to get the most reliable state of affairs.
Steps to attempt to recover a faulted roughing pump:
Step 1: Try to diagnose the problem. Determine what pump is stopped by looking at the "DPS full schematic..." panel for the symptoms listed above. The screenshot below shows the nominal state shown in the DPS full schematic:
Step 2: Open the "More..." panel in the roughing pump panel to see more status info. The screenshot below shows the roughing pump and "More..." panels in their nominal state. If there is an alarm or warning listed in the "More..." panel then report this error to the FACET elog and interpret as follows:
- Water flow low - check the water flow rate listed at the top panel. If <1 L/min then inform Doug
- Comm error, MP driver prot active, BP driver pro active - these are common radiation related faults. Attempt to reset and restart the pump.
- Any other alarm or warning - see if it clears and the pump restarts. If it doesn't then Doug can investigate more
Step 3: Reset the roughing pump error (if required) by pressing the reset button. If the error does not clear immediately then press reset a few more times over the span of 1-2 minutes.
Step 4: If the pump is not responsive, or if resetting did not clear the error, then power cycling the pump may clear the error. Power cycling the pump requires the cycling of the breaker that powers that specific pump. See the instructions below regarding this procedure:
Step 5: If the error will still not clear, then try powering off the breaker for 30 minutes, then power back on and retry the pump. If this does not work, then leave the power off and inform Doug.
Step 6: In the meantime - check the status of the handheld controller in the FKG20-23 rack - see the photo below.
- Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
- If it is, then press the PRG... button for 3 seconds
- The screen should now say "SET CONTROL MODE". If it says something else then try cycling through the menu using the up/down buttons.
- Press enter when the screen displays "SET CONTROL MODE"
- Use the up/down buttons to scroll through the options until you see "NOW:LOCAL MODE, SET:COM MODE?"
- Press enter to select this option. The "LOCAL" indicator light should turn off
- NOTE: do NOT select "REMOTE" mode. The correct mode is "COM MODE".
- Check that there are no error messages on the screen and that the "ALARM" and "ERROR" lights are not illuminated
- If they are then you need to try to clear the error.
- If power cycling the pump does not clear the error then it will likely take more in depth investigation to inform Doug.
- When the controller is at the main screen (displaying BP: xxx kW, MP: xx kW), you can cycle through the menu using the up/down buttons to get more information, i.e temperatures, water flow rate, any alarm messages. You can look through the alarm history by pressing ENTER when the screen displays "ALARM/WARNING HISTORY?"
- To get back the main menu you can press the PRG button to go back.
- Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
And the controller panel looks like this:
Step 7: If you were able to clear the error and the foreline pressure has pumped down to under 1 Torr, then you can restart the turbopump and open the foreline valve as described in Restarting DPS pumps.
If the pump will not restart, then it likely has a radiation damaged internal component. Repair instructions are located here: Instructions for fixing radiation damaged Ebara EV-S100P roughing pumps
Resetting a turbopump fault
Turbopump failures are much less common than roughing pump failures. Most of the time when a turbopump trips off it will be because of vacuum related issues such as the loss of backing pressure due to a failed roughing pump, or too high of gas load on the system for some other reason. If you correct the vacuum issues then you should be able to simply restart the turbopump as described in Restarting DPS pumps.
But in the case that a turbopump goes into an alarm or failure mode, then follow these steps to recover:
Step 1: Check if the turbopump is reporting a failure.
- Open the "More..." panel from the turbopump panel
- Check if the "Alarm" or "Failure" statuses are indicated
Step 2: If no error is present, then you can proceed to attempt the restart the turbo as described in Restarting DPS pumps. The screenshot below shows the turbopump and "More..." panels for a properly operating turbopump:
Step 3: If there is an error, then try pressing the "Reset" button on the turbopump panel to see if that clears the error.
Step 4: If the turbopump remains or returns to failure mode - then you will have to reset the controller in S20. See the "Controller locations" section to find the controller for the pump that has failed in S20. Remember that US-DPS turbos are in a different location than DS-DPS turbos.
Step 5: Check the status reported on the controller screen and report this error to the FACET elog. See the photo below for an example of a possible error message that could be displayed, and the "Failure" status light illuminated. Press the enter button once to illuminate the panel if the backlight is off.
Step 6: Find the power switch on the back of the controller and power off for a few seconds, then turn back on. Ensure that it powers off completely by checking the front panel
Step 7: Ensure that the controller is not reporting a failure or alarm indicator after it turns back on and finishes the initialization, which takes about 20 seconds.
Step 8: Ensure the controller is in remote mode. The indicator light beside "Remote" on the column of status lights should illuminated. If it is not, then press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds).
- Note: You will almost certainly not have to do this
Step 9: At this point, if the failure has been cleared, the pump should be able to be restarted as described in Restarting DPS pumps.
Setting a pump to remote mode
This must be down from the controllers in the S20 gallery.
To set a roughing pump to "COM MODE:
From the controller in the FKG20-23 racks:
- Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
- From the main screen, press the PRG... button for 3 seconds.
- The screen should now say "SET CONTROL MODE". If it says something else then try cycling through the menu using the up/down buttons, or pressing PRG to get back to main screen to restart the process at step 2.
- Press enter when the screen displays "SET CONTROL MODE"
- Use the up/down buttons to scroll through the options until you see "NOW:LOCAL MODE, SET:COM MODE?"
- Press enter to select this option. The "LOCAL" indicator light should turn off
NOTE: do NOT select "REMOTE" mode on the roughing pumps. The correct mode is "COM MODE".
To set a turbopump controller to "Remote" mode:
From the controllers in S20, if the indicator light beside "Remote" on the column of status lights is NOT illuminated, then do the following:
- Press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds).
- Check that the "Remote" light is on
Controller locations
The roughing pumps have handheld controllers in the S20 FKG20-24 rack, in the back of the rack, on the lower shelf:
And they look like this:
The turbopump controllers are in two places in the S20 gallery. The DS-DPS are behind the gas bottles beside the 20-12 penetration, and the US-DPS controllers are in the black standalone rack beside the 20-10 penetration.
DS-DPS controllers at penetration 20-12:
US-DPS controllers at penetration 20-10:
Power cycling roughing pumps using a breaker
Note - This procedure will be updated to include the required WPC processes required to undertake this procedure.
The roughing pumps and turbopumps are powered through the breaker panel 2PK102E2 on south wall of S20 gallery
Circuits are tied together in sets of 3:
- 13,15,17 = 3151:US2 roughing pump
- 19,21,23 = 3171:US1 roughing pump
- 31,33,35 = 3259:DS1 roughing pump
- 30,32,34 = 3291:DS2 roughing pump
The arc-flash hazard category has been updated to category 3 on 7/20/2023.
If a roughing pump will not reset an error message, power cycling the pump may clear the error. This work must be performed by authorized worker.
- Acquire authorizations and proper PPE for category 3 breaker
- Go to breaker panel 2PK102E2
- Open the breaker for the faulted pump
- Wait 10 minutes
- Close the breaker
- Wait for the pump to restart and check if the error has cleared (pump → More...)
- Click Run, and wait for the pump to reach full speed.
Power cycling roughing pumps using the relays
The DS1 and DS2 roughing pumps can be power cycled by energizing a relay using the Beckoff rather than cycling the breaker.
Do not power cycle from the Beckoff when the pumps are running!
The channels are G3 and G4 on the Beckoff for DS1 and DS2 respectively. ON means the relay is energized and the pump is OFF.
Pump redundancy
The forelines are reconfigurable to overcome the failure of a pump, in most cases. But this should only be attempted by an expert that knows what they are doing because doing this in the wrong order can lead to problems
...
.
If you reach this state and cannot continue or experience any difficulties – please call Doug Storey at SLAC extension # 3020, or cell # 650-656-5940.
...