On u41 log files of the HalfPipe, L1Proc, and ASP are being stored. From time to time u41 runs full bringing data processing to a halt. Options are to either ask computing to give us more disk space, or to free disk space on u41. The latter was done till 2017 (Warren?). "runs" files were moved to /nfs/farm/g/glast/g/isoc. For each million of mission elapsed time a hash directory exists. E.g. today (29 Sep 2022 09:48 UTC, MET 686137660) run files are stored below the 686 directory. Each hash directory contains about 7 GB of data, which is slightly more than the 6 GB we store every day.
Computing alerts Richard when the disk space usage on u41 reaches 98% (df rounds up, 97.01% is rounded to 98%). With our u41 size right now this means it would trigger at about 475 GB. Anyway I start moving stuff when available disk space falls below 600 GB. We also have to watch glast/g. I will reconsider moving when the available space on glast/g falls below 1 TB, regardless if this was caused by moving runs files or by activity of another user.
Step-by-step guide
check the disk space:
$ df -Ph /nfs/farm/g/glast/u41 /nfs/farm/g/glast/g/isoc Filesystem Size Used Avail Use% Mounted on staas-cnfs50lb:/gpfs/slac/ha/fs1/g/fermi/u41 16T 15T 599G 97% /nfs/farm/g/glast/u41 fermi-cnfslb1:/gpfs/slac/fermi/fs2/g 6.0T 3.6T 2.5T 59% /nfs/farm/g/glast/g
ssh as glastraw to fermi-lnx08. Execute the following commands:
$ ssh glastraw@fermilnx-v08 glastraw@fermilnx-v08 $ bash bash-4.1$ cd /nfs/farm/g/glast/u41/L1/runs/ bash-4.1$ # do an ls to check which are the oldest dirs which can be moved bash-4.1$ for i in 54?; do echo $i; time mv $i /nfs/farm/g/glast/g/isoc/L1/runs/ && ln -s /nfs/farm/g/glast/g/isoc/L1/runs/$i; done
recheck the disk space:
$ df -Ph /nfs/farm/g/glast/u41 /nfs/farm/g/glast/g/isoc Filesystem Size Used Avail Use% Mounted on staas-cnfs50lb:/gpfs/slac/ha/fs1/g/fermi/u41 16T 15T 668G 96% /nfs/farm/g/glast/u41 fermi-cnfslb1:/gpfs/slac/fermi/fs2/g 6.0T 3.7T 2.4T 61% /nfs/farm/g/glast/g