DPS Pump Troubleshooting

Table of Contents

Summary

The most common problems with the DPS is the roughing pumps tripping off and causing a cascade of other faults. So the first thing to look for is what stage or stages are faulted, and what roughing pump is likely to blame.

There is some redundancy in the system, so depending on the type of fault, the system should be able to continue to operate well enough to keep the beamline valves open if a single stage fails, but you may not be able to operate any gas sources without the full DPS operational.  See the "Pump redundancy" section for more info.

Overview of Operation and Troubleshooting of the the DPS

dstorey_2022-11-3 - DPS Operations and Troubleshooting Summary.pdf

Troubleshooting guide:

Check for your problem in the table below, then see full instructions below that for how to fix the issue.

Resetting a roughing pump fault

This may be indicated by one or more of the following:

1. The roughing pump is stopped (red) or disconnected (purple), or
2. The foreline vacuum gauge is reading a value >1 Tor and the foreline valve is closed, or
3. A warning or alarm is present in the "More..." panel of the roughing pump panel, or
4. The beamline vacuum performance on a stage is not as expected, or
5. Something else that I have not though of.

Note that in rare cases the roughing pump controller can be scrambled by radiation and it may report inaccurate statuses, so check the pressure readings to get the most reliable state of affairs.

Steps to attempt to recover a faulted roughing pump:

Step 1: Try to diagnose the problem. Determine what pump is stopped by looking at the "DPS full schematic..." panel for the symptoms listed above. The screenshot below shows the nominal state shown in the DPS full schematic:

Step 2: Open the "More..." panel in the roughing pump panel to see more status info.  The screenshot below shows the roughing pump and "More..." panels in their nominal state.  If there is an alarm or warning listed in the "More..." panel then report this error to the FACET elog and interpret as follows:

1. 1. Water flow low - check the water flow rate listed at the top panel. If <1 L/min then inform Doug
   2. Comm error, MP driver prot active, BP driver pro active - these are common radiation related faults. Attempt to reset and restart the pump.
   3. Any other alarm or warning - see if it clears and the pump restarts. If it doesn't then Doug can investigate more

Step 3: Reset the roughing pump error (if required) by pressing the reset button. If the error does not clear immediately then press reset a few more times over the span of 1-2 minutes.

Step 4: If pump is not responsive, or if resetting did not clear the error, then try power cycling the pump. The breaker panel is shown below.

1. 1. See "Controller locations" to find the circuit breaker for the roughing pumps, appropriate PPE, and the correct set of breakers
   2. Go out to S20 and turn off the breaker for at least 30 seconds
   3. Turn the breaker back on
   4. Check if the error is cleared and the pump will restart. 

Step 5:  If the error will still not clear, then try powering off the breaker for 30 minutes, then power back on and retry the pump. If this does not work, then leave the power off and inform Doug.

Step 6: In the meantime - check the status of the handheld controller in the FKG20-23 rack - see the photo below.

1. 1. Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
      1. If it is, then press the PRG... button for 3 seconds
      2. The screen should now say "SET CONTROL MODE". If it says something else then try cycling through the menu using the up/down buttons.
      3. Press enter when the screen displays "SET CONTROL MODE"
      4. Use the up/down buttons to scroll through the options until you see "NOW:LOCAL MODE, SET:COM MODE?"
      5. Press enter to select this option. The "LOCAL" indicator light should turn off
      6. NOTE: do NOT select "REMOTE" mode. The correct mode is "COM MODE".
   2. Check that there are no error messages on the screen and that the "ALARM" and "ERROR" lights are not illuminated
      1. If they are then you need to try to clear the error.
      2. If power cycling the pump does not clear the error then it will likely take more in depth investigation to inform Doug.
   3. When the controller is at the main screen (displaying BP: xxx kW, MP: xx kW), you can cycle through the menu using the up/down buttons to get more information, i.e temperatures, water flow rate, any alarm messages. You can look through the alarm history by pressing ENTER when the screen displays "ALARM/WARNING HISTORY?"
   4. To get back the main menu you can press the PRG button to go back.

And the controller panel looks like this:

Step 7: If you were able to clear the error and the foreline pressure has pumped down to under 1 Torr, then you can restart the turbopump and open the foreline valve as described in Restarting DPS pumps.

Resetting a turbopump fault

Turbopump failures are much less common than roughing pump failures. Most of the time when a turbopump trips off it will be because of vacuum related issues such as the loss of backing pressure due to a failed roughing pump, or too high of gas load on the system for some other reason. If you correct the vacuum issues then you should be able to simply restart the turbopump as described in Restarting DPS pumps.

But in the case that a turbopump goes into an alarm or failure mode, then follow these steps to recover:

Step 1: Check if the turbopump is reporting a failure.

1. 1. Open the "More..." panel from the turbopump panel
   2. Check if the "Alarm" or "Failure" statuses are indicated

Step 2: If no error is present, then you can proceed to attempt the restart the turbo as described in Restarting DPS pumps. The screenshot below shows the turbopump and "More..." panels for a properly operating turbopump:

Step 3: If there is an error, then try pressing the "Reset" button on the turbopump panel to see if that clears the error.

Step 4: If the turbopump remains or returns to failure mode - then you will have to reset the controller in S20.  See the "Controller locations" section to find the controller for the pump that has failed in S20.  Remember that US-DPS turbos are in a different location than DS-DPS turbos.

Step 5: Check the status reported on the controller screen and report this error to the FACET elog. See the photo below for an example of a possible error message that could be displayed, and the "Failure" status light illuminated. Press the enter button once to illuminate the panel if the backlight is off.

Step 6: Find the power switch on the back of the controller and power off for a few seconds, then turn back on.

Step 7: Ensure that the controller is not reporting a failure or alarm indicator after it turns back on and finishes the initialization, which takes about 20 seconds.

Step 8: Ensure the controller is in remote mode. The indicator light beside "Remote" on the column of status lights should illuminated. If it is not, then press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds). 

Step 9: At this point, if the failure has been cleared, the pump should be able to be restarted as described in Restarting DPS pumps.

Setting a pump to remote mode

This must be down from the controllers in the S20 gallery.

To set a roughing pump to "COM MODE:

From the controller in the FKG20-23 racks:

1. Check that the pump is NOT in local mode - the "LOCAL" indicator light should NOT be illuminated.
2. From the main screen, press the PRG... button for 3 seconds.
3. The screen should now say "SET CONTROL MODE". If it says something else then try cycling through the menu using the up/down buttons, or pressing PRG to get back to main screen to restart the process at step 2.
4. Press enter when the screen displays "SET CONTROL MODE"
5. Use the up/down buttons to scroll through the options until you see "NOW:LOCAL MODE, SET:COM MODE?"
6. Press enter to select this option. The "LOCAL" indicator light should turn off

NOTE: do NOT select "REMOTE" mode on the roughing pumps. The correct mode is "COM MODE".

To set a turbopump controller to "Remote" mode:

From the controllers in S20, if the indicator light beside "Remote" on the column of status lights is NOT illuminated, then do the following:

1. Press and hold the "Local/Remote" button on the front panel until the "Remote" status indicator comes on (this should take only a few seconds). 
2. Check that the "Remote" light is on

Controller locations

The roughing pumps have handheld controllers in the S20 FKG20-24 rack, in the back of the rack, on the lower shelf:

And they look like this:

The turbopump controllers are in two places in the S20 gallery. The DS-DPS are behind the gas bottles beside the 20-12 penetration, and the US-DPS controllers are in the black standalone rack beside the 20-10 penetration.

DS-DPS controllers at penetration 20-12:

US-DPS controllers at penetration 20-10:

Power cycling roughing pumps using a breaker

The breaker panel for the DPS roughing pumps (and turbopumps) is: 
 
Panel 2PK102E2 on south wall of S20 gallery 

• 208V 3 phase
• Flash hazard Category 0
• PPE: Long sleeve shirt and pants (natural fiber), safety glasses 

Circuits are tied together in sets of 3:

• 13,15,17 = 3151:US2 roughing pump
• 19,21,23 = 3171:US1 roughing pump
• 31,33,35 = 3259:DS1 roughing pump
• 30,32,34 = 3291:DS2 roughing pump
   

If a roughing pump will not reset an error message, turn power off on breaker
 panel for 30 seconds, then power back on. More status info is available on the hand held
 controllers in the FKG racks. 

Pump redundancy

OLD STUFF BELOW

In the case of pressure rise on gauges 3259 and 3291 due to a failure of one or more pumps in the DS-DPS system, this procedure should be used to isolate the beamline and stop the pressure from rising further.  See the attached document for the full procedure.

In most cases this will isolate the beamline and allow the other remaining operating stage to pump down the beamline and allow operations to commence.

Link to full procedure:   FACET S20 DPS Troubleshooting - 03-04-2022.pdf

Troubleshooting if pumps keep turning themselves off during startup

At the moment you cannot restart a turbopump with the DPSvacWatcher running.  It will detect that things aren't in the nominal operating state and will stop the pump (or insert the foreline valve). This will be signaled by an entry in the elog.

Stop the watcher, restart the pumps, then restart the watcher once the valves are in a good state. Alternatively, once a turbo is tripped off then you have ~10 minutes while the watcher ignores that stage.  So if the turbo is shut off by the DPS when restarting, just try to restart again a second time within the 10 minute window and you should be fine.

Step-by-step troubleshooting guide for vacuum issues

1. Open the full schematic: Select “DPS full schematic…” from the LI20 Vacuum page on Epics.
   
   
2. If the 3259:DS1 FV-DS1 foreline valve is closed AND the 3259:DS1 turbo is off, then this stage is in the normal “off” state – go to step 4.
   
   
3. If the 3259:DS1 FV-DS1 foreline valve is open, and either the 3259 turbo or roughing pump (or both) are off, then this indicates a problem with this stage. The flow chart indicates the actions to be taken to put the stage into an “off” state:
   
   DS1 troubleshooting flow chart:
   
   
   

   If you get to the green “Everything looks good” box, then this stage appears to be in a good state.

   
   

4. Now – assess the DS2 stage in the same way: If the 3291:DS2 FV-DS2 foreline valve is closed AND the 3291:DS2 turbo is off, then this stage is in the normal “off” state – go to step 6.
   
   
5. If the 3291:DS2 FV-DS1 foreline valve is open, and either the 3291 turbo or roughing pump (or both) are off, then this indicates a problem with this stage. The flow chart indicates the actions to be taken to put the stage into an “off” state:
   
   DS2 troubleshooting flow chart:
   
   
6. If at this point either of the 3259 or 3291 turbos are running, the beamline should be pumping down to <1e-6 Torr in a very short period of time. If neither is running, but the 3250 valve is open – then the beamline should be slowly being pumped down through the dump line turbo.
   
   
7. If both turbos are off and the 3250 valve is closed, then this section of the beamline will not pump down until one of the pumps is restarted.

If you reach this state and cannot continue or experience any difficulties – please call Doug Storey at SLAC extension # 3020, or cell # 650-656-5940.