...
Code Block |
---|
/u2/g.net.pinger/hep# /u2/g.net.pinger/hep--size /u2/g.net.pinger/iepm-bw /u2/g.net.pinger/shahryar /u2/g.net.pinger/slac-gateways /u2/g.net.pinger/traceroute This saved about 15GBytes then we deleted bandwidth-tests data iepm-bw.slac.stanford.edu monalisa nan-backup nettest2scratch node2.slac.stanford.edu pinger_mysql sc2002 sc2003 sc2004 223cottrell@pinger:~$df /nfs/slac/g/net/pinger Filesystem 1K-blocks Used Available Use% Mounted on netfs03:/u2/g.net.pinger 845881344 816465920 20956160 98% /nfs/slac/g/net/pinger i.e. we saved ~ 44Gbytes total |
More saving 3/11/2016
Once again we are receiving:
...
Use rm -r -v to remove files that have been copied from /nfs/slac/g/net/pinger/pingerreports/new/
More saving 8/31/2016
We are receiving
Code Block |
---|
Your "cron" job /afs/slac/package/pinger/tulip/vtrace0chk.pl produced the following output: can't close tmp file=/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt: No space left on device at /afs/slac/package/pinger/tulip/vtrace0chk.pl line 168. TRSrun@pinger: Command exited with value 28 |
...
Note (see PingER data flow at SLAC) pingerdata holds the rawdata gathered from the MAs. pingerreports holds the analyzed data in particular the hourly data.
Next steps
The big elephant is pingerdata. It consists of
...
Code Block |
---|
54cottrell@pinger:~$df -h /nfs/slac/g/net/pingerdata.unite Filesystem Size Used Avail Use% Mounted on netfs03:/pingerdata.unite 610G 188G 417G 32% /nfs/slac/g/net/pingerdata.unite |
Problem 10/19/2016
Getting message:
Code Block |
---|
/afs/slac/package/pinger/tulip/vtracefromchk.pl produced the following output: Can't close list file: No space left on device at /afs/slac/package/pinger/tulip/vtracefromchk.pl line 150. AND /afs/slac/package/pinger/unite-monthly.pl produced the following output: 28: write error AND /afs/slac/package/pinger/tulip/vtrace0chk.pl produced the following output: can't close tmp file=/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt: No space left on device at /afs/slac/package/pinger/tulip/vtrace0chk.pl line 213. |
...
Code Block |
---|
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/oldftp total 2 -rw-r--r-- 1 pinger iepm 0 Oct 20 13:11 junk drwxr-sr-x 4 pinger iepm 512 Jan 25 2005 ping/ drwxr-sr-x 5 pinger iepm 512 Jan 25 2005 traceroute/ pinger@pinger $ ls -l /nfs/slac/g/net/pingerdata.unite/oldftp total 2 -rw-r--r-- 1 cottrell iepm 0 Oct 20 13:11 junk drwxr-xr-x 4 cottrell iepm 512 Jan 25 2005 ping/ drwxr-xr-x 5 cottrell iepm 512 Jan 25 2005 traceroute/ pinger@pinger $ rm -r -v /nfs/slac/g/net/pinger/pingerdata/oldftp removed `/nfs/slac/g/net/pinger/pingerdata/oldftp/ping/`df -0ping/2001/data-2001-01.tar.gz' removed `/nfs/slac/g/net/pinger/pingerdata/oldftp/ping/2001/data-2001-02.tar.gz' ... |
The copies and deletes were repeated for:
Code Block |
---|
351cottrell@pinger:/nfs/slac/g/net/pinger/pingerdata$du -sh *
22M 1997
406M 1998
815M 1999
2.3G 2000
3.0G 2001
3.7G 2002
3.9G 2003
e.g.
309cottrell@pinger:~$cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/1997 /nfs/slac `/nfs/slac/g/net/pinger/pingerdata/1997' -> `/nfs/slac/g/net/pingerdata.unite/1997'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-05.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-05.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-06.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-06.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-07.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-07.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-08.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-08.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-09.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-09.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-10.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-10.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-11.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-11.txt.gz'
`/nfs/slac/g/net/pinger/pingerdata/1997/ping-1997-12.txt.gz' -> `/nfs/slac/g/net/pingerdata.unite/1997/ping-1997-12.txt.gz'
|
Since all this did not save much space and the next thing to copy hep/ requires 0.5TB I requested 1TB archive space via INC0117666 as /nfs/slac/staas/fs1/g/pinger. It has the same permissions as /nfs/slac/g/net/pinger drwxrwsr-x 2 pinger iepm. We looked at where the space was going.
Code Block |
---|
318cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerdata/hep/
601G /nfs/slac/g/net/pinger/pingerdata/hep/
319cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerdata/hep/*
2.5G /nfs/slac/g/net/pinger/pingerdata/hep/RRDCache
2.2G /nfs/slac/g/net/pinger/pingerdata/hep/backup
237G /nfs/slac/g/net/pinger/pingerdata/hep/data
1.0K /nfs/slac/g/net/pinger/pingerdata/hep/data.unite
359G /nfs/slac/g/net/pinger/pingerdata/hep/failures |
I am unsure what failures/ are. I contacted Joao who also does not know what they are so we then made copy of /failures
Code Block |
---|
cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/hep/failures /nfs/slac/staas/fs1/g/pinger/pingerdata/hep &
We watched progress with:
317cottrell@pinger:~$ls -l /nfs/slac/staas/fs1/g/pinger/pingerdata/hep/failures | tail
-rw-rw-r-- 1 cottrell iepm 1254300337 Oct 17 08:40 log-09-24
-rw-rw-r-- 1 cottrell iepm 1258901031 Oct 17 08:40 log-09-25
-rw-rw-r-- 1 cottrell iepm 1151960612 Oct 17 08:40 log-09-26
-rw-rw-r-- 1 cottrell iepm 1198576167 Oct 17 08:41 log-09-27
-rw-rw-r-- 1 cottrell iepm 1198256583 Oct 17 08:41 log-09-28
-rw-rw-r-- 1 cottrell iepm 1120964327 Oct 17 08:41 log-09-29
-rw-rw-r-- 1 cottrell iepm 1118152578 Oct 17 08:41 log-09-30
-rw-rw-r-- 1 cottrell iepm 1158262461 Oct 17 08:41 log-10-01
-rw-rw-r-- 1 cottrell iepm 1042220531 Oct 17 08:42 log-10-02
-rw------- 1 cottrell iepm 309641216 Oct 26 16:47 log-10-03
...
[cottrell@pinger ~]$ rm -r -v /nfs/slac/g/net/pinger/pingerdata/hep/failures
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-06-17['
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-01-01'
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-01-02'
...
[cottrell@pinger ~]$ du -sh /nfs/slac/g/net/pinger/pingerdata/hep/*
2.5G /nfs/slac/g/net/pinger/pingerdata/hep/RRDCache
2.2G /nfs/slac/g/net/pinger/pingerdata/hep/backup
237G /nfs/slac/g/net/pinger/pingerdata/hep/data
1.0K /nfs/slac/g/net/pinger/pingerdata/hep/data.unite
[cottrell@pinger ~]$ df /nfs/slac/g/net/pinger/
Filesystem 1K-blocks Used Available Use% Mounted on
netfs03:/u2/g.net.pinger
845881344 461184000 376238080 56% /nfs/slac/g/net/pinger |
Problem 11/2/2016
Code Block |
---|
Your "cron" job
/afs/slac/package/pinger/getdata.pl >
/afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err
produced the following output:
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/comsatsswl.seecs.edu.pk/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/comsatsswl.seecs.edu.pk/ping-2016-11-01.txt.*:
No such file or directory
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/perfsonar-unimas.myren.net.my/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/perfsonar-unimas.myren.net.my/ping-2016-11-01.txt.*:
No such file or directory
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.isra.edu.pk/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.isra.edu.pk/ping-2016-11-01.txt.*:
No such file or directory
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.uum.edu.my/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.uum.edu.my/ping-2016-11-01.txt.*:
No such file or directory
gzip: /nfs/slac/g/net/pinger/pingerdata/hep/data/pingeramity.in/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pingeramity.in/ping-2016-11-01.txt.*:
No such file or directory |
Code Block |
---|
321cottrell@pinger:~$cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/2003 /nfs/slac/staas/fs1/g/pinger/pingerdata/2003
`/nfs/slac/g/net/pinger/pingerdata/2003' -> `/nfs/slac/staas/fs1/g/pinger/pingerdata/2003'
`/nfs/slac/g/net/pinger/pingerdata/2003/cache01.ansp.br' -> `/nfs/slac/staas/fs1/g/pinger/pingerdata/2003/cache01.ansp.br'
...
pinger@pinger:~$rm -r -v /nfs/slac/staas/fs1/g/pinger/pingerdata/2003
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/2003
ls: cannot access /nfs/slac/g/net/pinger/pingerdata/2003: No such file or directory
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/
total 2
drwxrwsr-x 6 pinger iepm 512 Oct 27 00:32 hep/
-rw-r--r-- 1 pinger iepm 0 Oct 20 11:18 log
drwxrwsr-x 3 pinger iepm 512 Mar 8 2012 new/
|
Code Block |
---|
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/pingerdata/hep
244G /nfs/slac/g/net/pinger/pingerdata/hep
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/pingerdata/
244G /nfs/slac/g/net/pinger/pingerdata/
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/
du: cannot read directory `/nfs/slac/g/net/pinger/lost+found': Permission denied
691G /nfs/slac/g/net/pinger/
338cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/
total 8
drwx-w---- 2 root root 512 Jan 7 2005 lost+found/
drwxrwsr-x 3 pinger iepm 512 May 7 13:11 pinger2/ #3.3G
drwxrwsr-x 2 iepm iepm 2560 Jul 11 11:13 pinger_mon_data/ #3G
drwxrwsr-x 4 pinger iepm 512 Nov 2 10:29 pingerdata/ #244G
drwxrwsr-x 3 iepm iepm 512 Mar 13 2016 pingerreports/ #54G
drwxrwsrwx 4 pinger iepm 512 Nov 1 16:02 tulip/ #391G
#Total 695.3GB
339cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerreports/
54G /nfs/slac/g/net/pinger/pingerreports/
340cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pinger2
3.3G /nfs/slac/g/net/pinger/pinger2
341cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pinger_mon_data
3.0K /nfs/slac/g/net/pinger/pinger_mon_data
342cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip
391G /nfs/slac/g/net/pinger/tulip
pinger@pinger $ df -h /nfs/slac/g/net/pinger
Filesystem Size Used Avail Use% Mounted on
netfs03:/u2/g.net.pinger
807G 799G 0 100% /nfs/slac/g/net/pinger |
It appears tulip is a big user (391G).
Edited $vim /afs/slac/package/pinger/tulip/vtrace0chk.pl to change the directory from /nfs/slac/g/net/pinger/tulip/cachetr to /nfs/slac/staas/fs1/g/pinger/tulip/cachetr
Code Block |
---|
366cottrell@pinger:~$ls -lt /nfs/slac/g/net/pinger/tulip
total 2173
-rw-r--r-- 1 pinger iepm 0 Nov 2 15:50 fromlist.txt
drwxrwxrwx 2 pinger iepm 835584 Nov 2 06:15 cache/
drwxrwsrwx 2 pinger iepm 3584 Nov 2 03:26 cachetr/
-rw-r--r-- 1 pinger sf 4098 Nov 1 16:02 from.txt
-rwxr--r-- 1 pinger iepm 6755 Oct 3 2014 perfdb.pl*
-rw-r--r-- 1 rajaasad iepm 5508 Oct 3 2014 perfsonar-10-4-14.txt
-rw-r--r-- 1 rajaasad iepm 1116103 May 31 2014 cities15000.csv
-rw-r--r-- 1 rajaasad iepm 7498 May 31 2014 undns.pl
-rwxr-xr-x 1 pinger iepm 6542 Feb 18 2014 vtracefromchk.pl*
-rw-rw-r-- 1 cottrell iepm 1762 Feb 18 2014 from.bak.txt
-rw-r--r-- 1 pinger iepm 175702 Feb 10 2014 citidb.txt
-rw-r--r-- 1 pinger iepm 14637 Feb 10 2014 psonar_vtrace.txt
-rw-r--r-- 1 pinger iepm 6394 Feb 8 2014 pinger_vtrace.txt
370cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
391G /nfs/slac/g/net/pinger/tulip/cachetr
399cottrell@pinger:~$`/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache_tmp.txt'
`/nfs/slac/g/net/pinger/tulip/cachetr/cache-tmp.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache-tmp.txt'
`/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_19357.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache_tmp_19357.txt'
...
406cottrell@pinger:~$rm -f -v /nfs/slac/g/net/pinger/tulip/cachetr/*
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache-tmp.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache.old'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_10503.txt'
...
417cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
7.0K /nfs/slac/g/net/pinger/tulip/cachetr
Nb |
A potentially useful command is:
Code Block |
---|
343cottrell@pinger:~$find /nfs/slac/g/net/pinger -printf '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 |
Code Block |
---|
366cottrell@pinger:~$ls -lt /nfs/slac/g/net/pinger/tulip
total 2173
-rw-r--r-- 1 pinger iepm 0 Nov 2 15:50 fromlist.txt
drwxrwxrwx 2 pinger iepm 835584 Nov 2 06:15 cache/
drwxrwsrwx 2 pinger iepm 3584 Nov 2 03:26 cachetr/
-rw-r--r-- 1 pinger sf 4098 Nov 1 16:02 from.txt
-rwxr--r-- 1 pinger iepm 6755 Oct 3 2014 perfdb.pl*
-rw-r--r-- 1 rajaasad iepm 5508 Oct 3 2014 perfsonar-10-4-14.txt
-rw-r--r-- 1 rajaasad iepm 1116103 May 31 2014 cities15000.csv
-rw-r--r-- 1 rajaasad iepm 7498 May 31 2014 undns.pl
-rwxr-xr-x 1 pinger iepm 6542 Feb 18 2014 vtracefromchk.pl*
-rw-rw-r-- 1 cottrell iepm 1762 Feb 18 2014 from.bak.txt
-rw-r--r-- 1 pinger iepm 175702 Feb 10 2014 citidb.txt
-rw-r--r-- 1 pinger iepm 14637 Feb 10 2014 psonar_vtrace.txt
-rw-r--r-- 1 pinger iepm 6394 Feb 8 2014 pinger_vtrace.txt
370cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
391G /nfs/slac/g/net/pinger/tulip/cachetr
|
A potentially useful command is:
Code Block |
---|
343cottrell@pinger:~$find /nfs/slac/g/net/pinger -printf '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 |