DPS Pump Troubleshooting

Table of Contents

Summary

The most common problems with the DPS is the roughing pumps tripping off and causing a cascade of other faults. So the first thing to look for is what stage or stages are faulted, and what roughing pump is likely to blame.

There is some redundancy in the system, so depending on the type of fault, the system should be able to continue to operate well enough to keep the beamline valves open if a single stage fails, but you may not be able to operate any gas sources without the full DPS operational.  See the "Pump redundancy" section for more info.

Overview of Operation and Troubleshooting of the the DPS

dstorey_2022-11-3 - DPS Operations and Troubleshooting Summary.pdf

Troubleshooting guide:

Check for your problem in the table below, then see full instructions below that for how to fix the issue.

Resetting a roughing pump fault

This may be indicated by one or more of the following:

1. The roughing pump is stopped (red) or disconnected (purple), or
2. The foreline vacuum gauge is reading a value >1 Tor and the foreline valve is closed, or
3. A warning or alarm is present in the "More..." panel of the roughing pump panel, or
4. The beamline vacuum performance on a stage is not as expected, or
5. Something else that I have not though of.

Note that in rare cases the roughing pump controller can be scrambled by radiation and it may report inaccurate statuses, so check the pressure readings to get the most reliable state of affairs.

Steps to attempt to recover a faulted roughing pump:

Step 1: Try to diagnose the problem. Determine what pump is stopped by looking at the "DPS full schematic..." panel for the symptoms listed above. The screenshot below shows the nominal state shown in the DPS full schematic:

Step 2: Open the "More..." panel in the roughing pump panel to see more status info.  The screenshot below shows the roughing pump and "More..." panels in their nominal state.  If there is an alarm or warning listed in the "More..." panel then report this error to the FACET elog and interpret as follows:

1. 1. Water flow low - check the water flow rate listed at the top panel. If <1 L/min then inform Doug
   2. Comm error, MP driver prot active, BP driver pro active - these are common radiation related faults. Attempt to reset and restart the pump.
   3. Any other alarm or warning - see if it clears and the pump restarts. If it doesn't then Doug can investigate more

Step 3: Reset the roughing pump error (if required) by pressing the reset button. If the error does not clear immediately then press reset a few more times over the span of 1-2 minutes.

Step 4: If the pump is not responsive, or if resetting did not clear the error, then power cycling the pump may clear the error. Power cycling the pump requires the cycling of the breaker that powers that specific pump. See the instructions below regarding this procedure:

• Power cycling roughing pumps using the relays
• Power cycling roughing pumps using a breaker

Step 5:  If the error will still not clear, then try powering off the breaker for 30 minutes, then power back on and retry the pump. If this does not work, then leave the power off and inform Doug.

Step 6: In the meantime - check the status of the handheld controller in the FKG20-23 rack - see the photo below.

1. 1. Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
      1. If it is, then press the PRG... button for 3 seconds
      2. The screen should now say "SET CONTROL MODE". If it says something else then try cycling through the menu using the up/down buttons.
      3. Press enter when the screen displays "SET CONTROL MODE"
      4. Use the up/down buttons to scroll through the options until you see "NOW:LOCAL MODE, SET:COM MODE?"
      5. Press enter to select this option. The "LOCAL" indicator light should turn off
      6. NOTE: do NOT select "REMOTE" mode. The correct mode is "COM MODE".
   2. Check that there are no error messages on the screen and that the "ALARM" and "ERROR" lights are not illuminated
      1. If they are then you need to try to clear the error.
      2. If power cycling the pump does not clear the error then it will likely take more in depth investigation to inform Doug.
   3. When the controller is at the main screen (displaying BP: xxx kW, MP: xx kW), you can cycle through the menu using the up/down buttons to get more information, i.e temperatures, water flow rate, any alarm messages. You can look through the alarm history by pressing ENTER when the screen displays "ALARM/WARNING HISTORY?"
   4. To get back the main menu you can press the PRG button to go back.

And the controller panel looks like this:

Step 7: If you were able to clear the error and the foreline pressure has pumped down to under 1 Torr, then you can restart the turbopump and open the foreline valve as described in Restarting DPS pumps.

If the pump will not restart, then it likely has a radiation damaged internal component. Repair instructions are located here: Instructions for fixing radiation damaged Ebara EV-S100P roughing pumps

Resetting a turbopump fault

Turbopump failures are much less common than roughing pump failures. Most of the time when a turbopump trips off it will be because of vacuum related issues such as the loss of backing pressure due to a failed roughing pump, or too high of gas load on the system for some other reason. If you correct the vacuum issues then you should be able to simply restart the turbopump as described in Restarting DPS pumps.

But in the case that a turbopump goes into an alarm or failure mode, then follow these steps to recover:

Step 1: Check if the turbopump is reporting a failure.

1. 1. Open the "More..." panel from the turbopump panel
   2. Check if the "Alarm" or "Failure" statuses are indicated

Step 2: If no error is present, then you can proceed to attempt the restart the turbo as described in Restarting DPS pumps. The screenshot below shows the turbopump and "More..." panels for a properly operating turbopump:

Step 3: If there is an error, then try pressing the "Reset" button on the turbopump panel to see if that clears the error.

Step 4: If the turbopump remains or returns to failure mode - then you will have to reset the controller in S20.  See the "Controller locations" section to find the controller for the pump that has failed in S20.  Remember that US-DPS turbos are in a different location than DS-DPS turbos.

Step 5: Check the status reported on the controller screen and report this error to the FACET elog. See the photo below for an example of a possible error message that could be displayed, and the "Failure" status light illuminated. Press the enter button once to illuminate the panel if the backlight is off.

Step 6: Find the power switch on the back of the controller and power off for a few seconds, then turn back on.

Step 7: Ensure that the controller is not reporting a failure or alarm indicator after it turns back on and finishes the initialization, which takes about 20 seconds.

Step 8: Ensure the controller is in remote mode. The indicator light beside "Remote" on the column of status lights should illuminated. If it is not, then press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds). 

Step 9: At this point, if the failure has been cleared, the pump should be able to be restarted as described in Restarting DPS pumps.

Setting a pump to remote mode

This must be down from the controllers in the S20 gallery.

To set a roughing pump to "COM MODE:

From the controller in the FKG20-23 racks:

1. Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
2. From the main screen, press the PRG... button for 3 seconds.
3. The screen should now say "SET CONTROL MODE". If it says something else then try cycling through the menu using the up/down buttons, or pressing PRG to get back to main screen to restart the process at step 2.
4. Press enter when the screen displays "SET CONTROL MODE"
5. Use the up/down buttons to scroll through the options until you see "NOW:LOCAL MODE, SET:COM MODE?"
6. Press enter to select this option. The "LOCAL" indicator light should turn off

NOTE: do NOT select "REMOTE" mode on the roughing pumps. The correct mode is "COM MODE".

To set a turbopump controller to "Remote" mode:

From the controllers in S20, if the indicator light beside "Remote" on the column of status lights is NOT illuminated, then do the following:

1. Press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds). 
2. Check that the "Remote" light is on

Controller locations

The roughing pumps have handheld controllers in the S20 FKG20-24 rack, in the back of the rack, on the lower shelf:

And they look like this:

The turbopump controllers are in two places in the S20 gallery. The DS-DPS are behind the gas bottles beside the 20-12 penetration, and the US-DPS controllers are in the black standalone rack beside the 20-10 penetration.

DS-DPS controllers at penetration 20-12:

US-DPS controllers at penetration 20-10:

Power cycling roughing pumps using a breaker

Note - This procedure will be updated to include the required WPC processes required to undertake this procedure.

The roughing pumps and turbopumps are powered through the breaker panel 2PK102E2 on south wall of S20 gallery 

Circuits are tied together in sets of 3:

• 13,15,17 = 3151:US2 roughing pump
• 19,21,23 = 3171:US1 roughing pump
• 31,33,35 = 3259:DS1 roughing pump
• 30,32,34 = 3291:DS2 roughing pump

The arc-flash hazard category has been updated to category 3 on 7/20/2023.

If a roughing pump will not reset an error message, power cycling the pump may clear the error. This work must be performed by authorized worker.

1. Acquire authorizations and proper PPE for category 3 breaker
2. Go to breaker panel 2PK102E2
3. Open the breaker for the faulted pump
4. Wait 10 minutes
5. Close the breaker
6. Wait for the pump to restart and check if the error has cleared (pump → More...)
7. Click Run, and wait for the pump to reach full speed.

Power cycling roughing pumps using the relays

The DS1 and DS2 roughing pumps can be power cycled by energizing a relay using the Beckoff rather than cycling the breaker.

Do not power cycle from the Beckoff when the pumps are running!

The channels are G3 and G4 on the Beckoff for DS1 and DS2 respectively. ON means the relay is energized and the pump is OFF.

Pump redundancy

The forelines are reconfigurable to overcome the failure of a pump, in most cases. But this should only be attempted by an expert that knows what they are doing because doing this in the wrong order can lead to problems.

If you reach this state and cannot continue or experience any difficulties – please call Doug Storey at SLAC extension # 3020, or cell # 650-656-5940.