While digitization and reconstruction are happening, essentially no I/O occurs. This is because the chunks (for digi) and crumbs (for recon) are all pretty much the same size.
If they were not all the same size, results fron the smaller ones could be written out while the larger ones were being processed. Ideally, just as the smallest piece finished writing out its results, the second-smallest would start, then the third start as the second finished, etc. This requires that the sizes of the peices follow a geometric progression, which results in a lot of small pieces. For chunks, that should be fine, but making more crumbs out of a chunk is kind of a bad thing because all of the recon jobs have to read the whole chunk.
Now (2007-11-09) I'm staging output files, but not inputs. Everything is at least as fast as it was when staging both inputs & outputs.
Recon chunk merging and SVAC tuple (the 2 longest-running steps) are a lot faster. Runs are finished in about 2 hours now
(3-3.5 when staging inputs).
Also implemented variable sized crumbs. Before, there were usually 6 crumbs per chunk, all (about) the same size. Now
there are always 7, sizes are a geometric sequence with multiplier 0.95, the smallest crumb is ~3/4 the size of the largest
(which is the same size that all of them were before). When the crumbs were all the same size, recon files were transferred
out at 2-4 MB/s; now it's 17-37 MB/s. Beacuse there are more crumbs/chunk than before, I/O is a bit heavier when recon
starts up. Having a constant number of crumbs/chunk is necessary to reap the full benefits of variable sized chunks (which
is not implemented yet, but 1 chunk in each run (the last) is smaller than the others).
Between those 2, the periods with no I/O have shrunk, <~10 min now.
Below are ganglia I/O graphs for both AFS staging servers for orbits 1 and 2. I'm not sure which orbit the previous graph was from.
