...
There is some redundancy in the system, so depending on the type of fault, the system should be able to continue to operate well enough to keep the beamline valves open if a single stage fails, but you may not be able to operate any gas sources without the full DPS operational. See the "Pump redundancy" section for more info.
Overview of Operation and Troubleshooting of the the DPS
dstorey_2022-11-3 - DPS Operations and Troubleshooting Summary.pdf
Troubleshooting guide:
Check for your problem in the table below, then see full instructions below that for how to fix the issue.
Troubleshooting guide:
| Symptom | Potential Issue | What to do |
|---|---|---|
| Roughing pump tripped off | Radiation related fault | See - Resetting a roughing pump fault |
| Roughing pump won't start | Error message is present | Check "More..." status to see what error is listed. |
| Roughing pump won't reset an error | There is a real problem, or pump has been damaged | See - Resetting a roughing pump fault |
Turbopump tripped off | Most likely related to loss of backing pressure | Fix the problem with the roughing pump/foreline configuration then restart the turbo as described in Restarting DPS pumps. |
| Radiation related fault | See - Resetting a turbopump fault | |
| Roughing pump or turbo not responsive | Pump is in local mode, not remote | Put pump into remote mode. See Setting pumps to remote mode |
| US4 stage cannot be restarted | Probably radiation damage | See - Pump redundancy |
| Foreline valve won't open | Foreline interlocks need to be cleared | Reset foreline inerlocks interlocks from "Foreline Vlv Intlks..." panel |
...
Step 3: Reset the roughing pump error (if required) by pressing the reset button. If the error does not clear immediately then press reset a few more times over the span of 1-2 minutes.
Step 4: If the pump is not responsive, or if resetting did not clear the error, then try power cycling the pump . The breaker panel is shown below.
- See "Controller locations" to find the circuit breaker for the roughing pumps, appropriate PPE, and the correct set of breakers
- Go out to S20 and turn off the breaker for at least 30 seconds
- Turn the breaker back on
- Check if the error is cleared and the pump will restart.
may clear the error. Power cycling the pump requires the cycling of the breaker that powers that specific pump. See the instructions below regarding this procedure:
Step 5: If the error will still not clear, then try powering off the breaker for 30 minutes, then power back on and retry the pump. If this does not work, then leave the power off and inform Doug.
...
Step 7: If you were able to clear the error and the foreline pressure has pumped down to under 1 Torr, then you can restart the turbopump and open the foreline valve as described in Restarting DPS pumps.
If the pump will not restart, then it likely has a radiation damaged internal component. Repair instructions are located here: Instructions for fixing radiation damaged Ebara EV-S100P roughing pumps
Resetting a turbopump fault
...
Step 6: Find the power switch on the back of the controller and power off for a few seconds, then turn back on. Ensure that it powers off completely by checking the front panel
Step 7: Ensure that the controller is not reporting a failure or alarm indicator after it turns back on and finishes the initialization, which takes about 20 seconds.
Step 8: Ensure the controller is in remote mode. The indicator light beside "Remote" on the column of status lights should illuminated. If it is not, then press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds).
- Note: You will almost certainly not have to do this
Step 9: At this point, if the failure has been cleared, the pump should be able to be restarted as described in Restarting DPS pumps.
...
The roughing pumps have handheld controllers in the S20 FKG20-23 24 rack, second from the top shelf, sitting on the "CORNING" box:in the back of the rack, on the lower shelf:
And they look like this:
...
DS-DPS controllers at penetration 20-12:
US-DPS controllers at penetration 20-10:
Power cycling roughing pumps using a breaker
Note - This procedure will be updated to include the required WPC processes required to undertake this procedure.
The roughing pumps and turbopumps are powered through the breaker panel The breaker panel for the DPS roughing pumps (and turbopumps) is:
Panel 2PK102E2 on south wall of S20 gallery
- 208V 3 phase
- Flash hazard Category 0
- PPE: Long sleeve shirt and pants (natural fiber), safety glasses
Circuits are tied together in sets of 3:
- 13,15,17 = 3151:US2 roughing pump
- 19,21,23 = 3171:US1 roughing pump
- 31,33,35 = 3259:DS1 roughing pump
- 30,32,34 = 3291:DS2 roughing pump
The arc-flash hazard category has been updated to category 3 on 7/20/2023.
If a roughing pump will not reset an error message, turn power off on breaker
panel for 30 seconds, then power back on. More status info is available on the hand held
controllers in the FKG racks.
Pump redundancy
OLD STUFF BELOW
In the case of pressure rise on gauges 3259 and 3291 due to a failure of one or more pumps in the DS-DPS system, this procedure should be used to isolate the beamline and stop the pressure from rising further. See the attached document for the full procedure.
In most cases this will isolate the beamline and allow the other remaining operating stage to pump down the beamline and allow operations to commence.
Link to full procedure: FACET S20 DPS Troubleshooting - 03-04-2022.pdf
Troubleshooting if pumps keep turning themselves off during startup
At the moment you cannot restart a turbopump with the DPSvacWatcher running. It will detect that things aren't in the nominal operating state and will stop the pump (or insert the foreline valve). This will be signaled by an entry in the elog.
Stop the watcher, restart the pumps, then restart the watcher once the valves are in a good state. Alternatively, once a turbo is tripped off then you have ~10 minutes while the watcher ignores that stage. So if the turbo is shut off by the DPS when restarting, just try to restart again a second time within the 10 minute window and you should be fine.
Step-by-step troubleshooting guide for vacuum issues
...
If you get to the green “Everything looks good” box, then this stage appears to be in a good state.
...
power cycling the pump may clear the error. This work must be performed by authorized worker.
- Acquire authorizations and proper PPE for category 3 breaker
- Go to breaker panel 2PK102E2
- Open the breaker for the faulted pump
- Wait 10 minutes
- Close the breaker
- Wait for the pump to restart and check if the error has cleared (pump → More...)
- Click Run, and wait for the pump to reach full speed.
Power cycling roughing pumps using the relays
The DS1 and DS2 roughing pumps can be power cycled by energizing a relay using the Beckoff rather than cycling the breaker.
Do not power cycle from the Beckoff when the pumps are running!
The channels are G3 and G4 on the Beckoff for DS1 and DS2 respectively. ON means the relay is energized and the pump is OFF.
Pump redundancy
The forelines are reconfigurable to overcome the failure of a pump, in most cases. But this should only be attempted by an expert that knows what they are doing because doing this in the wrong order can lead to problems
...
.
If you reach this state and cannot continue or experience any difficulties – please call Doug Storey at SLAC extension # 3020, or cell # 650-656-5940.
...