Page History

...

Only one delivery can process a run at a time. This is enforced by a lock file in the run directory on u52/L1. If findChunks fails or there are permanent failures in the run and another part of the run is waiting, it has to be removed by hand. It should never be removed unless the only failures in the run are findChunks or permanent ones, or there's a deadlock. Even then you have to wear a helmet and sign a waiver.

Too Much Data

Hopefully this whole section is now obsolete, see the next one.

Sometimes we get no data from the MOC for half a day and then it all arrives at once. This will overload the AFS buffers (see next section) unless L1 processing is throttled by hand (we are working on implementing an automatic throttle, and ditching AFS buffering in favor of xrootd). You do this by hand-creating run locks for runs that haven't arrived yet and suspending batch jobs.

...

There's now a throttle that limits the number of RDLs that can be in process (or in the hard part, at least) at once. It works by making files with names like /nfs/farm/g/glast/u52/L1/throttle/1.lock at the same time as it makes the run lock. Usually set at 2, 3 is probably safe but we're still not quite convinced. It's still under development and a bit fragile, so it's probably better not to mess with it for now.

AFS buffers & idle threads

...