The Fermi xrootd gpfs pool has two storage pools that are locally attached to fermigpfs01/02 or fermi-gpfs05/06:
| host server | gpfs pool name |
|---|---|
| fermi-gpfs01/02 | system |
| fermi-gpfs05/06 | dpool1 |
The gpfs servers have access all pools but files that are on a non-local pool have to be read over the network (Ethernet). For example on fermi-gpfs01 accessing files that are stored on dpool1 will be read from fermi-gpfs05/06. In this case monitoring will show outgoing traffic for fermi-gpfs05/06 and incoming traffic for fermi-gpfs01. In order to find the pool for a file run the command:
% /usr/lpp/mmfs/bin/mmlsattr -L <file-name>
The throughput for reading from a remote pool was tested by reading 5 large files (~30GB per file) in parallel on a gpfs server and the files are on a remote pool:
| fermi-gpfsN | pool read from | rate GB/s |
|---|---|---|
| 01 | dpool1 | ~1.3 |
| 02 | dpool1 | ~2.0 |
| 05 | system | ~2.0 |
| 06 | system | ~2.0 |
All but fermi-gpfs01 showed the expected rate (2x10Gbits/s). It is not clear why fermi-gpfs01 shows a lower rate. There might be some interference with other nodes (but I don't now the network configuration). The following plot shows the interface activities on these nodes during the tests:
