...

The solution is to simply roll back the upstream process so it gets a clean execution.

Failure in checkRun

Check run failed and I thought at first it was a case of bad data as the process ran simultaneously on two different hosts.  However a simple rollback did not solve the problem as there was a lock left from running the mergeCalChunks upstream process.  This lock was left because of an infrastructure glitch preventing proper access to the xroot data disks.  Rerunning that process removed the lock file but checkRun still didn't work, complaining about the existence of a dontCleanUp file.  This file was actually left from the bad mergeCalChunks run.  Before deleting it we need to check that the files listed there actually exist.  This command:

for ff in $(egrep '^root' <path to>/dontCleanUp) ; do xrd.pl --where stat $ff ; done

will poll the xroot server for information on the files listed in the dontCleanUp file.

If they are all there, we are good to go and can remove the dontCleanUp file and rollback the checkRun process.  If not, we need to look further upstream in the data processing chain.