...
- As of 1 Aug 2014, Fermi has 184 TB of 'free space' in the xroot cluster, of which about 21 TB resides on READ-ONLY servers, leaving about 163 TB of accessible free space. At the current rate of consumption and ignoring future reprocessings, this storage could last as long as early January 2015.
- As of 1 Aug 2014 Fermi NFS cluster consists of four 32-GB TB wain-class servers (128 TB), of which 98 TB (77%) have been allocated. There are 30 TB of unallocated space and 34 TB of unused allocated space. Space, per se, is not an obvious issue with the NFS cluster, and groups of clients have been segregated to independent machines to prevent unwanted interactions between, say, the ISOC data acquisition and LAT user/group disk users. The four NFS user groups currently on their own servers are:
- ISOC
- MonteCarlo/RSP/ASP/Reprocessing
- ReleaseManager
- Users & Groups
- In the NFS domain, there is a considerable amount of "inactive" historical data which is seldom accessed. Some of this data is labeled as "Fill" on the spreadsheet describing the migration from sulky->wain machines two years ago (link).
...
- The four existing NFS servers are operating on equipment that is over 5 years old. Each of these servers hosts 32 GB TB of storage.
- wain025, the user/group disk, is routinely heavily loaded and not infrequently overloaded, sometimes leading to hangs and crashes.
- within the xroot cluster, the oldest four 32-GB TB servers are over 6 years old (wain017, 019, 020, 021)
- the 11 next oldest 32-GB TB xroot servers were purchased in 2009 (wain033, 034, 035, 036, 037, 038, 039, 053, 054, 055, and 056)
- six xroot servers have been made READ-ONLY by Wilko due to frequent problems (wain053, 054, 055, 056, 069, 071) This may be due to the extensive use of Seagate drives, which fail in a way that can hang the I/O bus. Note that Maintech has begun to use Hitachi drives for replacement.
...