L1Proc 1.61, which was put into production at 5pm PDT today, has the feature that it crashes in digitization if it detectes a mismatch between the LatC key in the data and the LatC key from the expected Moot key. The first delivery we got all crashed with the message:
MootSvc FATAL Hw key 6680 in data does not match hw key belonging to Moot Config #2225
MootSvc FATAL MootSvc exiting on fatal error..
Going back in the logfiles I can see that this has happened since we started to take nomSciOps_noSkirtCno_noCno runs after the calibOps runs, starting with
Jul/15/2008 03:02:20.965 nomSciOps_noSkirtCno_noCno, run 237783738
Since we did not crash until today this was not detected i.e. we introduced a very useful feature in time! This also means that currently the L1 processing is down.
Talking to the Shift Coordinator it appears that the runs are not nomSciOps_noSkirtCno_noCno runs but plain old nomSciOps. The only differences are
1/ CNO is allowed to open the trigger window.
2/ The skirt tiles are in the CNO.
Looking at the monitoring for a few runs I see indications(!) for this. The deadtime seems to be slightly higher than it should be and the CNO arrival time has a peak at zero.
Note that the data products are completely correct. The only problem this causes (beyond the fact that we seem to be taking nomSciOps when we really wanted to take nomSciOps_noSkirtCno_noCno) is that the runs are currently mislabeled in the web tables and in the Data Catalog. This can easily be fixed afterwards (we have already done this several times).
Tomorrow there will be a PROC request to change the run type to what we really want. In the meantime we should continue processing the data as all the data products are correct. Richard, as head of the CCB allowed, us to hotpatch the L1 pipeline to remove the job option that is causing us to crash:
MootSvc.ExitOnFatal = true;
The job options in the pipeline are dynamically generated so the change is just to comment out this line in a pipeline script.
anders