Versions Compared

Old Version 1

changes.mady.by.user Michael Kuss

Saved on

compared with

New Version Current

changes.mady.by.user Michael Kuss

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

On u41 log files of the HalfPipe, L1Proc, and ASP are being stored.  From time to time u41 runs full bringing data processing to a halt. Options are to either ask computing to give us more disk space, or to free disk space on u41. The latter was done till 2017 (Warren?).  "runs" files were moved to /nfs/farm/g/glast/g/isoc.  For each million of mission elapsed time a hash directory exists.  E.g. today (29 Sep 2022 09:48 UTC, MET 686137660) run files are stored below the 686 directory.  Each hash directory contains about 7 GB of data, which is slightly more than the 6 GB we store every day.

Computing alerts Richard when the disk space usage on u41 reaches 98% (df rounds up, 97.01% is rounded to 98%).  With our u41 size right now this means it would trigger at about 475 GB.  Anyway I start moving stuff when available disk space falls below 600 GB.  We also have to watch glast/g.  I will reconsider moving when the available space on glast/g falls below 1 TB, regardless if this was caused by moving runs files or by activity of another user.to be filled later

Step-by-step guide

  1. check the disk space:

    Code Block
    $ df -Ph /nfs/farm/g/glast/u41 /nfs/farm/g/glast/g/isoc         
Filesystem                                    Size  Used Avail Use% Mounted on
staas-cnfs50lb:/gpfs/slac/ha/fs1/g/fermi/u41   16T   15T  599G  97% /nfs/farm/g/glast/u41
fermi-cnfslb1:/gpfs/slac/fermi/fs2/g          6.0T  3.6T  2.5T  59% /nfs/farm/g/glast/g

  2. ssh as glastraw to fermi-lnx08.  Execute the following commands:

    Code Block
    $ ssh glastraw@fermilnx-v08
glastraw@fermilnx-v08 $ bash
bash-4.1$ cd /nfs/farm/g/glast/u41/L1/runs/
bash-4.1$ # do an ls to check which are the oldest dirs which can be moved
bash-4.1$ for i in 54?; do echo $i; time mv $i /nfs/farm/g/glast/g/isoc/L1/runs/ && ln -s /nfs/farm/g/glast/g/isoc/L1/runs/$i; done

  3. recheck the disk space:

...

Content by Label
showLabelsfalse
max5
spacesds
showSpacefalse
sortmodified
reversetrue
typepage
cqllabel = "disk" and type = "page" and space = "ds"
labelsdisk

  1. Code Block
    $ df -Ph /nfs/farm/g/glast/u41 /nfs/farm/g/glast/g/isoc
Filesystem                                    Size  Used Avail Use% Mounted on
staas-cnfs50lb:/gpfs/slac/ha/fs1/g/fermi/u41   16T   15T  668G  96% /nfs/farm/g/glast/u41
fermi-cnfslb1:/gpfs/slac/fermi/fs2/g          6.0T  3.7T  2.4T  61% /nfs/farm/g/glast/g

...

hiddentrue

...