...
P120-MERIT
Status chronology
- 7/21/2010 - block 1 reprocessing complete
- 7/20/2010 - agree upon 'pilot block' of runs (239557417 - 243220241), 637 runs. Begin...
- 7/19/2010 - submit first test run. success. await feedback
...
- The processClump step is taking ~40 hequ-minutes (or ~65 fell-minutes). With >500 simultaneous jobs running, there is little noticeable strain on xroot. There are five servers in the yellow-orange load range and they are claiming ~110-130 MB/s I/O rate.
- The mergeClumps step is taking ~5 hequ-minutes
- It was observed that submitting 70 runs at once put a strain on /u30, home of GlastRelease. Some 93 of ~540 jobs failed with I/O error, but succeeded upon rollback.
Load balancing
Introduce new trickleStreams.py script to (partially) assess pipeline activity and only the number of jobs considered appropriate based on available data.
| Code Block |
|---|
maxProcessClumps = 600 ## prevent overload of xroot
maxMergeClumps = 20 ## prevent overload of xroot (inactive)
maxStreamsPerCycle = 20 ## prevent overload of /u30 on startup
timePerCycle = 900 ## 15 minutes: allow time for dust to settle
|
With these parameters, it took ~ 5 hours to reach a point where fewer than 20 jobs per cycle were regularly submitted. Another 4.5 hours for the task to complete. On average, one run generated 7.5 processClump batch jobs.
P120-FT1
Status chronology
...