...
Expand | |||||
---|---|---|---|---|---|
| |||||
I have a file called "preparelist.py" that helps querying the data catalog and save the list of runs in a appropriate text file, which will be used in the next step. The file (/nfs/farm/g/glast/u38/Reprocess-tasks/P310-FT2/preparelist.py) looks like this:
It basically does two calls to the datacatalog. The first to retrieve the list of run to reprocess, the second to retrieve the list of run already reprocessed. Files are created to keep track of this files, and the names of the files contain the minimum and the maximum run number. |
Expand | ||||
---|---|---|---|---|
| ||||
As the last print statement suggests, I split the list of files in files containing 25 runs. First I create 2 directories, and I cd in the todo one. For example:
Then, the command I use is simply, for example:
This will create a series of files containing 25 run each. |
Expand | ||
---|---|---|
| ||
There is a simple file (submitter-prod-2020-11) containing the sequence of bash command I submit:
Note that this has to be modified every time I create a backfill (todo-2020-11/ and done-2020-11/). What this does read one file in the todo-2020-11 directory, and submit N streams of the P310-FT2 task, each task has the input run (RUNID) as argument. In our case, N=25. Then it will move the input file in the done-2020-11 directory. Then it sleeps for 5 minutes. |
...