Location of the scripts and of the reprocessing directory:
/nfs/farm/g/glast/u38/Reprocess-tasks/P310-FT2 |
The reprocessing is usually split into three steps (expand for details):
I have a file called "preparelist.py" that helps querying the data catalog and saves the list of runs in an appropriate text file, which will be used in the next step. The file (/nfs/farm/g/glast/u38/Reprocess-tasks/P310-FT2/preparelist.py) looks like this:
It basically does two calls to the data catalog. The first retrieves the list of runs to reprocess, and the second retrieves the list of runs already reprocessed. Files are created to keep track of these files, and the names of the files contain the minimum and the maximum run number. |
As the last print statement suggests, I split the list of files into files containing 25 runs. First, I create two directories, and I then cd in the todo one. For example:
Then, the command I use is simply:
This will create a series of files containing 25 run each. |
There is a simple file (submitter-prod-2020-11) containing the sequence of bash commands I submit:
Note that this has to be modified every time I create a backfill (todo-2020-11/ and done-2020-11/). What this does is read one file in the todo-2020-11 directory, and submit N streams of the P310-FT2 task. Each task has the input run (RUNID) as an argument. In our case, N=25. Then it will move the input file in the done-2020-11 directory. Then it sleeps for 5 minutes. |
This is very similar to reprocessing runs. Check the BAD time periods page which lists runs with bad time intervals (BTIs). Then, create a list (called BTI-list.txt here) with the runs in the reprocessing that contain BTIs. As before, I split the file into subfiles with 25 runs:
and then I run the following script (submitter-prod-bti)
which submits the flagFT2-P310 task. Note that here the script submits 50 files and sleeps for 10 minutes. |