Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • G4 propagator error
  • Overlapping chunks

If recon dies complaining about g4propagator, we can't fix it.
In either of the above cases, email Heather and Anders. Include a link to the log file, which will tell them where the core file is.

If trending merges complain about overlapping chunks, tell Bryson and datamonlist@glast2.Stanford.EDU. Rollback won't help.

Everything to know about the Rollback

How to rollback

When to rollback

The dontCleanUp file

I just caught a merge not finding all of its input files because of a read failure for the first time. Remember to check the log watcher for merge errors. In a future L1 version, I'll add the run ID to the message target so it can be filtered on.

From the glast-ground page, click "Logging" (about 1/3 of the way down left side)
Click "Selection" (upper right)
Click "More Selections" (lower right)
In the "Select Program" field, enter "mergeStuff.py"
Click "Table"

Now you should have a big list of complaining about not being able to find input files. Can't filter on run number yet, sorry. If any of the complaining merges are downstream of a failed process, you can ignore them. This is usually the case. But if they are not downstream from a failure, it means there was a disk read error while setting up the merge. Look at the logfile for that merge, it will say what files are missing. If they are really there, the merge can be rolled back and will probably work the second time. If they are not there, there's a bigger problem, I don't even know what.

Any time one of these messages is generated, cleanup for the run is disabled by a file called dontCleanUp in the run directory on u52/L1. All cleanup jobs will fail if that file is present. If everything is OK, that file can be removed and the jobs rolled back.