...
Since all this did not save much space and the next thing to copy hep/ requires 0.5TB I requested 1TB archive space via INC0117666 as /nfs/slac/staas/fs1/g/pinger. It has the same permissions as /nfs/slac/g/net/pinger drwxrwsr-x 2 pinger iepm. We looked at where the space was going.
| Code Block |
|---|
318cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerdata/hep/
601G /nfs/slac/g/net/pinger/pingerdata/hep/
319cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerdata/hep/*
2.5G /nfs/slac/g/net/pinger/pingerdata/hep/RRDCache
2.2G /nfs/slac/g/net/pinger/pingerdata/hep/backup
237G /nfs/slac/g/net/pinger/pingerdata/hep/data
1.0K /nfs/slac/g/net/pinger/pingerdata/hep/data.unite
359G /nfs/slac/g/net/pinger/pingerdata/hep/failures |
I am unsure what failures/ are. I contacted Joao who also does not know what they are so we then made copy of /failures
| Code Block |
|---|
cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/hep/failures /nfs/slac/staas/fs1/g/pinger/pingerdata/hep &
We watched progress with:
317cottrell@pinger:~$ls -l /nfs/slac/staas/fs1/g/pinger/pingerdata/hep/failures | tail
-rw-rw-r-- 1 cottrell iepm 1254300337 Oct 17 08:40 log-09-24
-rw-rw-r-- 1 cottrell iepm 1258901031 Oct 17 08:40 log-09-25
-rw-rw-r-- 1 cottrell iepm 1151960612 Oct 17 08:40 log-09-26
-rw-rw-r-- 1 cottrell iepm 1198576167 Oct 17 08:41 log-09-27
-rw-rw-r-- 1 cottrell iepm 1198256583 Oct 17 08:41 log-09-28
-rw-rw-r-- 1 cottrell iepm 1120964327 Oct 17 08:41 log-09-29
-rw-rw-r-- 1 cottrell iepm 1118152578 Oct 17 08:41 log-09-30
-rw-rw-r-- 1 cottrell iepm 1158262461 Oct 17 08:41 log-10-01
-rw-rw-r-- 1 cottrell iepm 1042220531 Oct 17 08:42 log-10-02
-rw------- 1 cottrell iepm 309641216 Oct 26 16:47 log-10-03
...
[cottrell@pinger ~]$ rm -r -v /nfs/slac/g/net/pinger/pingerdata/hep/failures
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-06-17['
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-01-01'
removed `/nfs/slac/g/net/pinger/pingerdata/hep/failures/log-01-02'
...
[cottrell@pinger ~]$ du -sh /nfs/slac/g/net/pinger/pingerdata/hep/*
2.5G /nfs/slac/g/net/pinger/pingerdata/hep/RRDCache
2.2G /nfs/slac/g/net/pinger/pingerdata/hep/backup
237G /nfs/slac/g/net/pinger/pingerdata/hep/data
1.0K /nfs/slac/g/net/pinger/pingerdata/hep/data.unite
[cottrell@pinger ~]$ df /nfs/slac/g/net/pinger/
Filesystem 1K-blocks Used Available Use% Mounted on
netfs03:/u2/g.net.pinger
845881344 461184000 376238080 56% /nfs/slac/g/net/pinger |
Problem 11/2/2016
| Code Block |
|---|
Your "cron" job
/afs/slac/package/pinger/getdata.pl >
/afs/slac/g/www/www-iepm/pinger/slaconly/getdata.err
produced the following output:
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/comsatsswl.seecs.edu.pk/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/comsatsswl.seecs.edu.pk/ping-2016-11-01.txt.*:
No such file or directory
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/perfsonar-unimas.myren.net.my/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/perfsonar-unimas.myren.net.my/ping-2016-11-01.txt.*:
No such file or directory
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.isra.edu.pk/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.isra.edu.pk/ping-2016-11-01.txt.*:
No such file or directory
gzip:
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.uum.edu.my/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.uum.edu.my/ping-2016-11-01.txt.*:
No such file or directory
gzip: /nfs/slac/g/net/pinger/pingerdata/hep/data/pingeramity.in/ping-2016-11-01.txt.gz:
No space left on device
ls: cannot access
/nfs/slac/g/net/pinger/pingerdata/hep/data/pingeramity.in/ping-2016-11-01.txt.*:
No such file or directory |
| Code Block |
|---|
321cottrell@pinger:~$cp -r -p -v /nfs/slac/g/net/pinger/pingerdata/2003 /nfs/slac/staas/fs1/g/pinger/pingerdata/2003
`/nfs/slac/g/net/pinger/pingerdata/2003' -> `/nfs/slac/staas/fs1/g/pinger/pingerdata/2003'
`/nfs/slac/g/net/pinger/pingerdata/2003/cache01.ansp.br' -> `/nfs/slac/staas/fs1/g/pinger/pingerdata/2003/cache01.ansp.br'
...
pinger@pinger:~$rm -r -v /nfs/slac/staas/fs1/g/pinger/pingerdata/2003
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/2003
ls: cannot access /nfs/slac/g/net/pinger/pingerdata/2003: No such file or directory
pinger@pinger $ ls -l /nfs/slac/g/net/pinger/pingerdata/
total 2
drwxrwsr-x 6 pinger iepm 512 Oct 27 00:32 hep/
-rw-r--r-- 1 pinger iepm 0 Oct 20 11:18 log
drwxrwsr-x 3 pinger iepm 512 Mar 8 2012 new/
|
| Code Block |
|---|
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/pingerdata/hep
244G /nfs/slac/g/net/pinger/pingerdata/hep
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/pingerdata/
244G /nfs/slac/g/net/pinger/pingerdata/
pinger@pinger $ du -sh /nfs/slac/g/net/pinger/
du: cannot read directory `/nfs/slac/g/net/pinger/lost+found': Permission denied
691G /nfs/slac/g/net/pinger/
338cottrell@pinger:~$ls -l /nfs/slac/g/net/pinger/
total 8
drwx-w---- 2 root root 512 Jan 7 2005 lost+found/
drwxrwsr-x 3 pinger iepm 512 May 7 13:11 pinger2/ #3.3G
drwxrwsr-x 2 iepm iepm 2560 Jul 11 11:13 pinger_mon_data/ #3G
drwxrwsr-x 4 pinger iepm 512 Nov 2 10:29 pingerdata/ #244G
drwxrwsr-x 3 iepm iepm 512 Mar 13 2016 pingerreports/ #54G
drwxrwsrwx 4 pinger iepm 512 Nov 1 16:02 tulip/ #391G
#Total 695.3GB
339cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pingerreports/
54G /nfs/slac/g/net/pinger/pingerreports/
340cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pinger2
3.3G /nfs/slac/g/net/pinger/pinger2
341cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/pinger_mon_data
3.0K /nfs/slac/g/net/pinger/pinger_mon_data
342cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip
391G /nfs/slac/g/net/pinger/tulip
pinger@pinger $ df -h /nfs/slac/g/net/pinger
Filesystem Size Used Avail Use% Mounted on
netfs03:/u2/g.net.pinger
807G 799G 0 100% /nfs/slac/g/net/pinger |
It appears tulip is a big user (391G).
Edited $vim /afs/slac/package/pinger/tulip/vtrace0chk.pl to change the directory from /nfs/slac/g/net/pinger/tulip/cachetr to /nfs/slac/staas/fs1/g/pinger/tulip/cachetr
| Code Block |
|---|
366cottrell@pinger:~$ls -lt /nfs/slac/g/net/pinger/tulip
total 2173
-rw-r--r-- 1 pinger iepm 0 Nov 2 15:50 fromlist.txt
drwxrwxrwx 2 pinger iepm 835584 Nov 2 06:15 cache/
drwxrwsrwx 2 pinger iepm 3584 Nov 2 03:26 cachetr/
-rw-r--r-- 1 pinger sf 4098 Nov 1 16:02 from.txt
-rwxr--r-- 1 pinger iepm 6755 Oct 3 2014 perfdb.pl*
-rw-r--r-- 1 rajaasad iepm 5508 Oct 3 2014 perfsonar-10-4-14.txt
-rw-r--r-- 1 rajaasad iepm 1116103 May 31 2014 cities15000.csv
-rw-r--r-- 1 rajaasad iepm 7498 May 31 2014 undns.pl
-rwxr-xr-x 1 pinger iepm 6542 Feb 18 2014 vtracefromchk.pl*
-rw-rw-r-- 1 cottrell iepm 1762 Feb 18 2014 from.bak.txt
-rw-r--r-- 1 pinger iepm 175702 Feb 10 2014 citidb.txt
-rw-r--r-- 1 pinger iepm 14637 Feb 10 2014 psonar_vtrace.txt
-rw-r--r-- 1 pinger iepm 6394 Feb 8 2014 pinger_vtrace.txt
370cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
391G /nfs/slac/g/net/pinger/tulip/cachetr
399cottrell@pinger:~$`/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache_tmp.txt'
`/nfs/slac/g/net/pinger/tulip/cachetr/cache-tmp.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache-tmp.txt'
`/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_19357.txt' -> `/nfs/slac/staas/fs1/g/pinger/tulip/cachetr/cache_tmp_19357.txt'
...
406cottrell@pinger:~$rm -f -v /nfs/slac/g/net/pinger/tulip/cachetr/*
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache-tmp.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache.old'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp.txt'
removed `/nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_10503.txt'
...
417cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
7.0K /nfs/slac/g/net/pinger/tulip/cachetr
Nb |
A potentially useful command is:
| Code Block |
|---|
343cottrell@pinger:~$find /nfs/slac/g/net/pinger -printf '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 |
| Code Block |
|---|
366cottrell@pinger:~$ls -lt /nfs/slac/g/net/pinger/tulip
total 2173
-rw-r--r-- 1 pinger iepm 0 Nov 2 15:50 fromlist.txt
drwxrwxrwx 2 pinger iepm 835584 Nov 2 06:15 cache/
drwxrwsrwx 2 pinger iepm 3584 Nov 2 03:26 cachetr/
-rw-r--r-- 1 pinger sf 4098 Nov 1 16:02 from.txt
-rwxr--r-- 1 pinger iepm 6755 Oct 3 2014 perfdb.pl*
-rw-r--r-- 1 rajaasad iepm 5508 Oct 3 2014 perfsonar-10-4-14.txt
-rw-r--r-- 1 rajaasad iepm 1116103 May 31 2014 cities15000.csv
-rw-r--r-- 1 rajaasad iepm 7498 May 31 2014 undns.pl
-rwxr-xr-x 1 pinger iepm 6542 Feb 18 2014 vtracefromchk.pl*
-rw-rw-r-- 1 cottrell iepm 1762 Feb 18 2014 from.bak.txt
-rw-r--r-- 1 pinger iepm 175702 Feb 10 2014 citidb.txt
-rw-r--r-- 1 pinger iepm 14637 Feb 10 2014 psonar_vtrace.txt
-rw-r--r-- 1 pinger iepm 6394 Feb 8 2014 pinger_vtrace.txt
370cottrell@pinger:~$du -sh /nfs/slac/g/net/pinger/tulip/cachetr
391G /nfs/slac/g/net/pinger/tulip/cachetr
|
A potentially useful command is:
| Code Block |
|---|
343cottrell@pinger:~$find /nfs/slac/g/net/pinger -printf '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 '%s %p\n'| sort -nr | head -10
find: `/nfs/slac/g/net/pinger/lost+found': Permission denied
236243945489 /nfs/slac/g/net/pinger/tulip/cachetr/cache.txt
191324110848 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1509.txt
125348712448 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_1522.txt
14292140032 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_14177.txt
9648889856 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_26404.txt
4080336896 /nfs/slac/g/net/pinger/tulip/cachetr/cache_tmp_8874.txt
719821346 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-10.txt
693718734 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-11.txt
688237297 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-12.txt
684539559 /nfs/slac/g/net/pinger/pingerdata/hep/data/pinger.slac.stanford.edu/ping-2015-08.txt
Exit 141 |