Testing xrootd with re-processing jobs

Reprocessing jobs were used to load the xrootd Fermi cluster. Most of the load is on wain017 to wain021.

Two test were run:

1. 20090821_1249:
   This was an repeat of the processing end of May 2009. This test used the old xrootd client that comes with the Fermi -gl3 root release.
2. 20090824_1323
   The intent was to repeat the previous test with a new root release (-gl5) that uses the latest xrootd client (20090721-0636). However the test started out with jobs that used the old release and only the last jobs used the new xrootd client.

The plots below show the network and disk activity during the tests. The network IO was obtained using kstat and the disk IO rates are from iostat and summing all disks that belong to the glast zfs zpool. Note the disk IO-rates reported by Ganglia during the tests are smaller then the one from iostat. The reason is that the zpool tool used by ganglia shows a lower rate compared to iostat. For wain020 which runs a later update of Solaris 10 (and therfore zfs) this difference seems to be gone.

Test 20090821_1249: old xrootd client

Test 20090824_1323: new xrootd client

This test was intended to use the new xrootd client (-gl5 root release), but it started out using the old xrootd client and only later jobs used the new xrootd client.

Comparing iostat and zpool-iostat

The plots below show the io rates and transaction rates for wain017 and
wain020 obtained with the <em>zpool iostat</em> or iostat itself (for iostat one
has to sum over all the drives that belong to the zfs zpool). wain017 run an older version
of Solaris10, update 3, compared to wain020 which runs Solaris 10 update 7.
On wain020 the io-rate from zpool and iostat match whereas on wain017 zpool shows a much lower
rate. The transaction rates is lower with zpool becuase it counts the transactions against
a vdev which might result into reads to multiple disks.