L1Proc 1.61, which was put into production at 5pm PDT today, has the feature that it crashes in digitization if it detectes a mismatch between the LatC key in the data and the LatC key from the expected Moot key. The first delivery we got (80717001) all crashed with the message:
...
Note that the data products are completely correct. The only problem this causes (beyond the fact that we seem to be taking nomSciOps when we really wanted to take nomSciOps_noSkirtCno_noCno!) is that the runs are currently mislabeled in the web tables and in the Data Catalog. This can easily be fixed afterwards (we have already done this several times).
...
The job options in the pipeline are dynamically generated so the change is just to comment out this line in a pipeline script.
This is now done! Two of the four crashed jobs are doing fine after having been rolled back. The two other ones were on a batch host that died on us. So we have to wait for the reaper before we can roll them back. But the L1 processing is up again!
anders