...
There is some redundancy in the system, so depending on the type of fault, the system should be able to continue to operate well enough to keep the beamline valves open if a single stage fails, but you may not be able to operate any gas sources without the full DPS operational. See the "Pump redundancy" section for more info.
Overview of Operation and Troubleshooting of the the DPS
dstorey_2022-11-3 - DPS Operations and Troubleshooting Summary.pdf
Troubleshooting guide:
Check for your problem in the table below, then see full instructions below that for how to fix the issue.
...
Symptom | Potential Issue | What to do |
---|---|---|
Roughing pump tripped off | Radiation related fault | See - Resetting a roughing pump fault |
Roughing pump won't start | Error message is present | Check "More..." status to see what error is listed. |
Roughing pump won't reset an error | There is a real problem, or pump has been damaged | See - Resetting a roughing pump fault |
Turbopump tripped off | Most likely related to loss of backing pressure | Fix the problem with the roughing pump/foreline configuration then restart the turbo as described in Restarting DPS pumps. |
Radiation related fault | See - Resetting a turbopump fault | |
Roughing pump or turbo not responsive | Pump is in local mode, not remote | Put pump into remote mode. See Setting pumps to remote mode |
US4 stage cannot be restarted | Probably radiation damage | See - Pump redundancy |
Foreline valve won't open | Foreline interlocks need to be cleared | Reset foreline inerlocks interlocks from "Foreline Vlv Intlks..." panel |
...
Step 3: Reset the roughing pump error (if required) by pressing the reset button. If the error does not clear immediately then press reset a few more times over the span of 1-2 minutes.
Step 4: If the pump is not responsive, or if resetting did not clear the error, then try power cycling the pump . The breaker panel is shown below.
- See "Controller locations" to find the circuit breaker for the roughing pumps, appropriate PPE, and the correct set of breakers
- Go out to S20 and turn off the breaker for at least 30 seconds
- Turn the breaker back on
- Check if the error is cleared and the pump will restart.
may clear the error. Power cycling the pump requires the cycling of the breaker that powers that specific pump. See the instructions below regarding this procedure:
Step 5: If the error will still not clear, then try powering off the breaker for 30 minutes, then power back on and retry the pump. If this does not work, then leave the power off and inform Doug.
...
Step 7: If you were able to clear the error and the foreline pressure has pumped down to under 1 Torr, then you can restart the turbopump and open the foreline valve as described in Restarting DPS pumps.
If the pump will not restart, then it likely has a radiation damaged internal component. Repair instructions are located here: Instructions for fixing radiation damaged Ebara EV-S100P roughing pumps
Resetting a turbopump fault
...
Step 6: Find the power switch on the back of the controller and power off for a few seconds, then turn back on. Ensure that it powers off completely by checking the front panel
Step 7: Ensure that the controller is not reporting a failure or alarm indicator after it turns back on and finishes the initialization, which takes about 20 seconds.
Step 8: Ensure the controller is in remote mode. The indicator light beside "Remote" on the column of status lights should illuminated. If it is not, then press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds).
- Note: You will almost certainly not have to do this
Step 9: At this point, if the failure has been cleared, the pump should be able to be restarted as described in Restarting DPS pumps.
...
- Press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds).
- Check that the "Remote" light is on
Controller locations
Pump redundancy
In the case of pressure rise on gauges 3259 and 3291 due to a failure of one or more pumps in the DS-DPS system, this procedure should be used to isolate the beamline and stop the pressure from rising further. See the attached document for the full procedure.
In most cases this will isolate the beamline and allow the other remaining operating stage to pump down the beamline and allow operations to commence.
Link to full procedure: FACET S20 DPS Troubleshooting - 03-04-2022.pdf
How to silence the roughing pump alarms in the tunnel
The roughing pumps will make an audible alarm if they are displaying a warning or alarm state. This is how to
Ideal method:
To check the cause of the alarm:
- Open the "DPS full schematic" from the LI20 Vacuum Epics page.
- For each roughing pump - click on the pump icon, then on "More..."
- You should see a list of all the warning and alarm states. If any of these has light beside it, then that is warning/alarm that is causing the problem.
- Make an Elog entry with the warning and alarm states that you find
If you understand the cause of the alarm and the implication of clearing it, then you can go back to the roughing pump control window (with the speedometer) and click on "Reset" to clear the warning/alarm state. You may have to hit reset a few times if it doesn't stop beeping on the the first try.
Alternative method:
If you do not understand why the particular error message is being displayed or if you are not comfortable clearing the error, but you need to stop the incessant noise from driving you crazy - then you can simply unplug the handheld controller panel that sits beside the roughing pumps under the light pipe. If you chose this path, then make all reasonable efforts to ensure that the controller panel is eventually reconnected.
In all cases - please notify the appropriate authorities that you have done something!
Troubleshooting if pumps keep turning themselves off during startup
At the moment you cannot restart a turbopump with the DPSvacWatcher running. It will detect that things aren't in the nominal operating state and will stop the pump (or insert the foreline valve). This will be signaled by an entry in the elog.
Stop the watcher, restart the pumps, then restart the watcher once the valves are in a good state. Alternatively, once a turbo is tripped off then you have ~10 minutes while the watcher ignores that stage. So if the turbo is shut off by the DPS when restarting, just try to restart again a second time within the 10 minute window and you should be fine.
Power cycling using a breaker
Breaker panel for the DPS roughing pumps is:
Panel 2PK102E2 on south wall of S20 gallery
208V 3 phase
Flash hazard Category 0
PPE: Long sleeve shirt and pants (natural fiber), safety glasses
The roughing pumps have handheld controllers in the S20 FKG20-24 rack, in the back of the rack, on the lower shelf:
And they look like this:
The turbopump controllers are in two places in the S20 gallery. The DS-DPS are behind the gas bottles beside the 20-12 penetration, and the US-DPS controllers are in the black standalone rack beside the 20-10 penetration.
DS-DPS controllers at penetration 20-12:
US-DPS controllers at penetration 20-10:
Power cycling roughing pumps using a breaker
Note - This procedure will be updated to include the required WPC processes required to undertake this procedure.
The roughing pumps and turbopumps are powered through the breaker panel 2PK102E2 on south wall of S20 gallery
Circuits are tied together in sets of 3:
- 13,15,17 = 3151:
...
- US2 roughing pump
- 19,21,23 = 3171:
...
- US1 roughing pump
- 31,33,35 = 3259:
...
- DS1 roughing pump
- 30,32,34 = 3291:DS2 roughing pump
The arc-flash hazard category has been updated to category 3 on 7/20/2023.
DS2
If a roughing pump will not reset an error message, turn power off on breaker
panel for 30 seconds, then power back on. More status info is available on the hand held
controllers in the FKG racks.
Turbopumps also receive power from this panel. Turbo controllers are at penetration 20-10. There
is a small brown half rack on the floor.
Step-by-step troubleshooting guide for vacuum issues
...
If you get to the green “Everything looks good” box, then this stage appears to be in a good state.
power cycling the pump may clear the error. This work must be performed by authorized worker.
- Acquire authorizations and proper PPE for category 3 breaker
- Go to breaker panel 2PK102E2
- Open the breaker for the faulted pump
- Wait 10 minutes
- Close the breaker
- Wait for the pump to restart and check if the error has cleared (pump → More...)
- Click Run, and wait for the pump to reach full speed.
Power cycling roughing pumps using the relays
The DS1 and DS2 roughing pumps can be power cycled by energizing a relay using the Beckoff rather than cycling the breaker.
Do not power cycle from the Beckoff when the pumps are running!
The channels are G3 and G4 on the Beckoff for DS1 and DS2 respectively. ON means the relay is energized and the pump is OFF.
Pump redundancy
The forelines are reconfigurable to overcome the failure of a pump, in most cases. But this should only be attempted by an expert that knows what they are doing because doing this in the wrong order can lead to problems
...
.
If you reach this state and cannot continue or experience any difficulties – please call Doug Storey at SLAC extension # 3020, or cell # 650-656-5940.
...