Task of the Data Processing on-call expert
L1Proc and halfPipe. Every time a red cross appears on the Data Processing page, next to the L1Proc or halfPipe processing status bar, the Level 1 on-call expert will be paged. We are not on-call for ASP/GRB search (Jim Chiang is on call for these failures) and we are definitely NOT on-call for infrastructure problems (can't see monitoring plots, etc.).
Different types of failures
Transient failures
- afs hiccup (can't read from/write to a resource located on an afs disk)
- nfs hiccup (can't read from/write to a resource located on an afs disk)
- LSF hiccup (an lsf batch machine crashes or is rebooted)
- lost Data Base connection (can happen on MOOT or more rarely on Oracle)
How to recognize transient failures: they usually affect only one single job (disk or DB failures), or a small number of jobs all on the same machine (LFS failure).
What to do in case of transient failures: rollback the affected process(es) when possible (see below). Look for the dontCleanUp file and check the Log Watcher (see below).
Infrastructure failures
- single batch host is killing several jobs ('rogue' LSF host)
- group of batch hosts crashed or went offline
- afs crashed on one of the host machines
- scratch disk is full on one or more of the host machines
- staging disk is full
- u52 or u15 are full
How to recognize infrastructure failures: they usually affect a large number of jobs, either on the same LSF host or on different LSF hosts.
What to do in case of infrastructure failures: these failures involve a large number of people to be taken care of (the infrastructure expert on-call and often also the SCCS), so for the time being still page Warren and/or Maria Elena (see L1 shift schedule) if you think that one of those failures might be happening during the night.