Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
/u2/g.net.pinger/hep#
/u2/g.net.pinger/hep--size
/u2/g.net.pinger/iepm-bw
/u2/g.net.pinger/shahryar
/u2/g.net.pinger/slac-gateways
/u2/g.net.pinger/traceroute
This saved about 15GBytes
then we deleted
bandwidth-tests
data
iepm-bw.slac.stanford.edu
monalisa
nan-backup
nettest2scratch
node2.slac.stanford.edu
pinger_mysql
sc2002
sc2003
sc2004

223cottrell@pinger:~$df /nfs/slac/g/net/pinger
Filesystem           1K-blocks      Used Available Use% Mounted on
netfs03:/u2/g.net.pinger
                     845881344 816465920  20956160  98% /nfs/slac/g/net/pinger
i.e. we saved ~ 44Gbytes total

More saving 3/11/2016

Once again we are receiving:

...

Use rm -r -v to remove files that have been copied from /nfs/slac/g/net/pinger/pingerreports/new/

More saving 8/31/2016

We are receiving

Code Block
Your "cron" job

/afs/slac/package/pinger/tulip/vtrace0chk.pl

produced the following output:

can't close tmp
file=/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt: No space left on
device at /afs/slac/package/pinger/tulip/vtrace0chk.pl line 168.

TRSrun@pinger: Command exited with value 28

...

Note (see PingER data flow at SLAC) pingerdata holds the rawdata gathered from the MAs. pingerreports holds the analyzed data in particular the hourly data.

Next steps

The big elephant is pingerdata. It consists of 

...

Code Block
54cottrell@pinger:~$df -h /nfs/slac/g/net/pingerdata.unite
Filesystem            Size  Used Avail Use% Mounted on
netfs03:/pingerdata.unite
                      610G  188G  417G  32% /nfs/slac/g/net/pingerdata.unite

Problem 10/19/2016

Getting message:

Code Block
/afs/slac/package/pinger/tulip/vtracefromchk.pl

produced the following output:

Can't close list file: No space left on device at
/afs/slac/package/pinger/tulip/vtracefromchk.pl line 150.
 
AND 
/afs/slac/package/pinger/unite-monthly.pl

produced the following output:

28: write error
 
AND
/afs/slac/package/pinger/tulip/vtrace0chk.pl

produced the following output:


can't close tmp
file=/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt: No space left on
device at /afs/slac/package/pinger/tulip/vtrace0chk.pl line 213.
Code Block
323cottrell@pinger:~$cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/oldftp /nfs/slac/g/net/pingerdata.unite/oldftp
`/nfs/slac/g/net/pinger/pingerdata/oldftp' -> `/nfs/slac/g/net/pingerdata.unite/oldftp'
`/nfs/slac/g/net/pinger/pingerdata/oldftp/ping' -> `/nfs/slac/g/net/pingerdata.unite/oldftp/ping'
`/nfs/slac/g/net/pinger/pingerdata/oldftp/ping/2001' -> `/nfs/slac/g/net/pingerdata.unite/oldftp/ping/2001'
`/nfs/slac/g/net/pinger/pingerdata/oldftp/ping/2001/data-2001-01.tar.gz' -> `/nfs/slac/g/net/pingerdata.unite/oldftp/ping/2001/data-2001-01.tar.gz'
...
326cottrell@pinger:~$ls -l /nfs/slac/g/net/pingerdata.unite/oldftp/*/*
-rw-r--r-- 1 cottrell iepm      51 Jan 25  2005 /nfs/slac/g/net/pingerdata.unite/oldftp/ping/README
-rwxr-xr-x 1 cottrell iepm     603 Jan 25  2005 /nfs/slac/g/net/pingerdata.unite/oldftp/ping/ral.pl*
-rw-r--r-- 1 cottrell iepm 8127136 Jan 25  2005 /nfs/slac/g/net/pingerdata.unite/oldftp/ping/ral.txt
/nfs/slac/g/net/pingerdata.unite/oldftp/ping/2001:
total 3324088
-rw-r--r-- 1 cottrell iepm 211453443 Jan 25  2005 data-2001-01.tar.gz
-rw-r--r-- 1 cottrell iepm 191452967 Jan 25  2005 data-2001-02.tar.gz
...

Via ServiceNow ticket INC0117383 I requested that the files in /nfs/slac/g/net/pinger/pingerdata have their ownership group changed to pinger iepm.  I tried using sudo with chown but that does not work.  Once this is done I can delete /nfs/slac/g/net/pinger/pingerdata/oldftp hence freeing up space for pinger.

Code Block
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/oldftp
total 2
-rw-r--r-- 1 pinger iepm   0 Oct 20 13:11 junk
drwxr-sr-x 4 pinger iepm 512 Jan 25  2005 ping/
drwxr-sr-x 5 pinger iepm 512 Jan 25  2005 traceroute/
pinger@pinger $ ls -l /nfs/slac/g/net/pingerdata.unite/oldftp
total 2
-rw-r--r-- 1 cottrell iepm   0 Oct 20 13:11 junk
drwxr-xr-x 4 cottrell iepm 512 Jan 25  2005 ping/
drwxr-xr-x 5 cottrell iepm 512 Jan 25  2005 traceroute/
pinger@pinger $ rm -r -v /nfs/slac/g/net/pinger/pingerdata/oldftp
removed `df -0ping/2001/data-2001-01.tar.gz'
removed `/nfs/slac/g/net/pinger/pingerdata/oldftp/ping/2001/data-2001-02.tar.gz'
...


The copies and deletes were repeated for:

Code Block
351cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata$du -sh *
22M     1997
406M    1998
815M    1999
2.3G    2000
3.0G    2001
3.7G    2002
3.9G    2003
e.g.
309cottrell@pinger:~$cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/1997 /nfs/slac                                                                           `/nfs/slac/g/net/pinger/pingerdata/1997' -> `/nfs/slac/g/net/pingerdata.unite/1997'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-05.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-05.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-06.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-06.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-07.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-07.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-08.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-08.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-09.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-09.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-10.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-10.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-11.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-11.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-12.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-12.txt.gz'

Since all this did not save much space and the next thing to copy hep/ requires 0.5TB I requested 1TB archive space via INC0117666 as /nfs/slac/staas/fs1/g/pinger. It has the same permissions as /nfs/slac/g/net/pinger drwxrwsr-x 2 pinger iepm. We looked at where the space was going.

Code Block
318cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerdata/hep/
601G    /nfs/slac/g/net/pinger/pingerdata/hep/
319cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerdata/hep/*
2.5G    /nfs/slac/g/net/pinger/pingerdata/hep/RRDCache
2.2G    /nfs/slac/g/net/pinger/pingerdata/hep/backup
237G    /nfs/slac/g/net/pinger/pingerdata/hep/data
1.0K    /nfs/slac/g/net/pinger/pingerdata/hep/data.unite
359G    /nfs/slac/g/net/pinger/pingerdata/hep/failures

I am unsure what failures/ are. I contacted Joao who also does not know what they are so we then made copy of /failures

Code Block
 cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/hep/failures /nfs/slac/staas/fs1/g/pinger/pingerdata/hep &
We watched progress with:

317cottrell@pinger:~$ls -l /nfs/slac/staas/fs1/g/pinger/pingerdata/hep/failures | tail
-rw-rw-r-- 1 cottrell iepm 1254300337 Oct 17 08:40 log-09-24
-rw-rw-r-- 1 cottrell iepm 1258901031 Oct 17 08:40 log-09-25
-rw-rw-r-- 1 cottrell iepm 1151960612 Oct 17 08:40 log-09-26
-rw-rw-r-- 1 cottrell iepm 1198576167 Oct 17 08:41 log-09-27
-rw-rw-r-- 1 cottrell iepm 1198256583 Oct 17 08:41 log-09-28
-rw-rw-r-- 1 cottrell iepm 1120964327 Oct 17 08:41 log-09-29
-rw-rw-r-- 1 cottrell iepm 1118152578 Oct 17 08:41 log-09-30
-rw-rw-r-- 1 cottrell iepm 1158262461 Oct 17 08:41 log-10-01
-rw-rw-r-- 1 cottrell iepm 1042220531 Oct 17 08:42 log-10-02
-rw------- 1 cottrell iepm  309641216 Oct 26 16:47 log-10-03
...
[cottrell@pinger ~]$ rm -r -v /nfs/slac/g/net/pinger/pingerdata/hep/failures 
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-06-17['
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-01-01'
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-01-02'
...
[cottrell@pinger ~]$ du -sh /nfs/slac/g/net/pinger/pingerdata/hep/*
2.5G	/nfs/slac/g/net/pinger/pingerdata/hep/RRDCache
2.2G	/nfs/slac/g/net/pinger/pingerdata/hep/backup
237G	/nfs/slac/g/net/pinger/pingerdata/hep/data
1.0K	/nfs/slac/g/net/pinger/pingerdata/hep/data.unite

[cottrell@pinger ~]$ df /nfs/slac/g/net/pinger/
Filesystem           1K-blocks      Used Available Use% Mounted on
netfs03:/u2/g.net.pinger
                     845881344 461184000 376238080  56% /nfs/slac/g/net/pinger

 

Problem 11/2/2016

Code Block
Your "cron" job

/afs/slac/package/pinger/getdata.pl >
/afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err

produced the following output:

gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/comsatsswl.seecs.edu.pk/ping-2016-11-01.txt.gz:
No space left on device

ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/comsatsswl.seecs.edu.pk/ping-2016-11-01.txt.*:
No such file or directory

gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/perfsonar-unimas.myren.net.my/ping-2016-11-01.txt.gz:
No space left on device

ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/perfsonar-unimas.myren.net.my/ping-2016-11-01.txt.*:
No such file or directory

gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.isra.edu.pk/ping-2016-11-01.txt.gz:
No space left on device

ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.isra.edu.pk/ping-2016-11-01.txt.*:
No such file or directory

gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.uum.edu.my/ping-2016-11-01.txt.gz:
No space left on device

ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.uum.edu.my/ping-2016-11-01.txt.*:
No such file or directory

gzip: /nfs/slac/g/net/pinger/pingerdata/hep/data/pingeramity.in/ping-2016-11-01.txt.gz:
No space left on device

ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pingeramity.in/ping-2016-11-01.txt.*:
No such file or directory
Code Block
321cottrell@pinger:~$cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/2003 /nfs/slac/staas/fs1/g/pinger/pingerdata/2003
`/nfs/slac/g/net/pinger/pingerdata/2003' -> `/nfs/slac/staas/fs1/g/pinger/pingerdata/2003'
`/nfs/slac/g/net/pinger/pingerdata/2003/cache01.ansp.br' -> `/nfs/slac/staas/fs1/g/pinger/pingerdata/2003/cache01.ansp.br'
...
pinger@pinger:~$rm -r  -v /nfs/slac/staas/fs1/g/pinger/pingerdata/2003
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/2003
ls: cannot access /nfs/slac/g/net/pinger/pingerdata/2003: No such file or directory

pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/
total 2
drwxrwsr-x 6 pinger iepm 512 Oct 27 00:32 hep/
-rw-r--r-- 1 pinger iepm   0 Oct 20 11:18 log
drwxrwsr-x 3 pinger iepm 512 Mar  8  2012 new/
Code Block
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/pingerdata/hep
244G    /nfs/slac/g/net/pinger/pingerdata/hep
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/pingerdata/
244G    /nfs/slac/g/net/pinger/pingerdata/
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/
du: cannot read directory `/nfs/slac/g/net/pinger/lost+found': Permission denied
691G    /nfs/slac/g/net/pinger/
338cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/
total 8
drwx-w---- 2 root   root  512 Jan  7  2005 lost+found/
drwxrwsr-x 3 pinger iepm  512 May  7 13:11 pinger2/ #3.3G
drwxrwsr-x 2 iepm   iepm 2560 Jul 11 11:13 pinger_mon_data/ #3G
drwxrwsr-x 4 pinger iepm  512 Nov  2 10:29 pingerdata/ #244G
drwxrwsr-x 3 iepm   iepm  512 Mar 13  2016 pingerreports/ #54G
drwxrwsrwx 4 pinger iepm  512 Nov  1 16:02 tulip/ #391G
#Total 695.3GB
339cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerreports/
54G     /nfs/slac/g/net/pinger/pingerreports/
340cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pinger2
3.3G    /nfs/slac/g/net/pinger/pinger2
341cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pinger_mon_data
3.0K    /nfs/slac/g/net/pinger/pinger_mon_data
342cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip
391G    /nfs/slac/g/net/pinger/tulip
pinger@pinger $ df -h /nfs/slac/g/net/pinger
Filesystem            Size  Used Avail Use% Mounted on
netfs03:/u2/g.net.pinger
                      807G  799G     0 100% /nfs/slac/g/net/pinger

It appears tulip is a big user (391G).

Edited $vim /afs/slac/package/pinger/tulip/vtrace0chk.pl to change the directory from /nfs/slac/g/net/pinger/tulip/cachetr to /nfs/slac/staas/fs1/g/pinger/tulip/cachetr

 

Code Block
366cottrell@pinger:~$ls -lt /nfs/slac/g/net/pinger/tulip
total 2173
-rw-r--r-- 1 pinger   iepm       0 Nov  2 15:50 fromlist.txt
drwxrwxrwx 2 pinger   iepm  835584 Nov  2 06:15 cache/
drwxrwsrwx 2 pinger   iepm    3584 Nov  2 03:26 cachetr/
-rw-r--r-- 1 pinger   sf      4098 Nov  1 16:02 from.txt
-rwxr--r-- 1 pinger   iepm    6755 Oct  3  2014 perfdb.pl*
-rw-r--r-- 1 rajaasad iepm    5508 Oct  3  2014 perfsonar-10-4-14.txt
-rw-r--r-- 1 rajaasad iepm 1116103 May 31  2014 cities15000.csv
-rw-r--r-- 1 rajaasad iepm    7498 May 31  2014 undns.pl
-rwxr-xr-x 1 pinger   iepm    6542 Feb 18  2014 vtracefromchk.pl*
-rw-rw-r-- 1 cottrell iepm    1762 Feb 18  2014 from.bak.txt
-rw-r--r-- 1 pinger   iepm  175702 Feb 10  2014 citidb.txt
-rw-r--r-- 1 pinger   iepm   14637 Feb 10  2014 psonar_vtrace.txt
-rw-r--r-- 1 pinger   iepm    6394 Feb  8  2014 pinger_vtrace.txt
370cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
391G    /nfs/slac/g/net/pinger/tulip/cachetr
399cottrell@pinger:~$`/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache_tmp.txt'
`/nfs/slac/g/net/pinger/tulip/cachetr/cache-tmp.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache-tmp.txt'
`/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_19357.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache_tmp_19357.txt'
...
406cottrell@pinger:~$rm -f -v /nfs/slac/g/net/pinger/tulip/cachetr/*
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache-tmp.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache.old'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_10503.txt'
...
417cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
7.0K    /nfs/slac/g/net/pinger/tulip/cachetr
Nb


A potentially useful command is:

 

Code Block
343cottrell@pinger:~$find /nfs/slac/g/net/pinger -printf '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141

 


Code Block
366cottrell@pinger:~$ls -lt /nfs/slac/g/net/pinger/tulip
total 2173
-rw-r--r-- 1 pinger   iepm       0 Nov  2 15:50 fromlist.txt
drwxrwxrwx 2 pinger   iepm  835584 Nov  2 06:15 cache/
drwxrwsrwx 2 pinger   iepm    3584 Nov  2 03:26 cachetr/
-rw-r--r-- 1 pinger   sf      4098 Nov  1 16:02 from.txt
-rwxr--r-- 1 pinger   iepm    6755 Oct  3  2014 perfdb.pl*
-rw-r--r-- 1 rajaasad iepm    5508 Oct  3  2014 perfsonar-10-4-14.txt
-rw-r--r-- 1 rajaasad iepm 1116103 May 31  2014 cities15000.csv
-rw-r--r-- 1 rajaasad iepm    7498 May 31  2014 undns.pl
-rwxr-xr-x 1 pinger   iepm    6542 Feb 18  2014 vtracefromchk.pl*
-rw-rw-r-- 1 cottrell iepm    1762 Feb 18  2014 from.bak.txt
-rw-r--r-- 1 pinger   iepm  175702 Feb 10  2014 citidb.txt
-rw-r--r-- 1 pinger   iepm   14637 Feb 10  2014 psonar_vtrace.txt
-rw-r--r-- 1 pinger   iepm    6394 Feb  8  2014 pinger_vtrace.txt
370cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
391G    /nfs/slac/g/net/pinger/tulip/cachetr


A potentially useful command is:

 

Code Block
343cottrell@pinger:~$find /nfs/slac/g/net/pinger -printf '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141