As reported by email from the PingER cronjob we ran out of space on /nfs/slac/g/net/pinger/ (example). This was confirmed as seen below, however there was space in the parent directory in pingerdata.unite.
207cottrell@pinger:~$df /nfs/slac/g/net/pinger
Filesystem 1K-blocks Used Available Use% Mounted on
netfs03:/u2/g.net.pinger
845881344 837422080 0 100% /nfs/slac/g/net/pinger
206cottrell@pinger:~$df /nfs/slac/g/net/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
17001308 5586492 10551380 35% /
~$ls -l /nfs/slac/g/net/
total 1
drwxr-xr-x 2 root root 0 Dec 2 10:40 iepm-bw/
drwxrwsr-x 28 1049 iepm 1024 Jan 23 2014 pinger/
drwxr-xr-x 2 root root 0 Dec 2 10:40 pingerdata.unite/
Looking in more detail at:
$ls -l /nfs/slac/g/net/pinger/* > dir ls: cannot open directory /nfs/slac/g/net/pinger/lost+found: Permission denied Exit 2
Looking in dir, it is apparent that apart from the directories:
pingerlod/, pingerreports/, pingerdata/, pinger_mon_data/, pinger2/, tulip/
most of this data is old and from earlier projects. Rather than lose it we decided to move this other data from
/nfs/slac/g/net/pinger to /nfs/slac/g/net/pingerdata.unite/hence preserving a copy if we run into problems.
We copied and deleted:
/u2/g.net.pinger/hep#
/u2/g.net.pinger/hep--size
/u2/g.net.pinger/iepm-bw
/u2/g.net.pinger/shahryar
/u2/g.net.pinger/slac-gateways
/u2/g.net.pinger/traceroute
This saved about 15GBytes
then we deleted
bandwidth-tests
data
iepm-bw.slac.stanford.edu
monalisa
nan-backup
nettest2scratch
node2.slac.stanford.edu
pinger_mysql
sc2002
sc2003
sc2004
223cottrell@pinger:~$df /nfs/slac/g/net/pinger
Filesystem 1K-blocks Used Available Use% Mounted on
netfs03:/u2/g.net.pinger
845881344 816465920 20956160 98% /nfs/slac/g/net/pinger
i.e. we saved ~ 44Gbytes total