Versions Compared

Old Version 72

changes.mady.by.user Maria Elena Monzani

Saved on

compared with

New Version 73

changes.mady.by.user Maria Elena Monzani

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Essential dependencies:


Migration milestones:

  •  Build L1 locally on containers
  •  Run single task in S3DF pipeline
  •  Update the L1Proc xml template
  •  Run full L1Proc in S3DF with LPA

Infrastructure needs:

Non-L1  milestones:

  •  Test full L1Proc in S3Df with LCI data
  •  Update non-L1 xml (flagFT2, merge)
  •  Test non-L1 tasks (flag FT2, merge)
  •  Migrate DQM/trending reports
  •  Migrate cronjobs for DQM reports

Sustainability and optimization:

  •  Get rid of SCons builds for L1 code
  •  Upgrade python to v3 and above
  •  Can we avoid crumbling altogether?
  •  At a minimum, reduce # of crumbs
  •  Use flash for staging instead of xrootd?


Open questions:

  •  do we still need a wrapper for cleanup? 
  •  do we need isocRun for every executable?
  •  what are the SLURM queue options?
  •  non-preemptable queue: --account fermi:L1
  •  how does the scratch space work on S3DF?


S3DF Cheatsheet:

  • Install ssh config as described here
  • to login from terminal: ssh slacd
  • to ssh as glastraw, run kinit first
  • NOTE: use terminal not XQuartz(?)
  • afs was migrated to: /sdf/group/fermi/a/
  • nfs was migrated to: /sdf/group/fermi/g/

Infrastructure needs:

Where to find/install stuff:

  • working area: /sdf/group/fermi/ground/PipelineConfig/L1Proc
  • pipeline cmd: /sdf/home/g/glast/a/pipeline-II/prod/pipeline
  • singularity example (05/02/23): pipeline task, xml file
  • additional slurm parameters (10/10/23): --account fermi:L1
  • python used by ST:/afs/slac/g/glast/ground/GLAST_EXT/redhat6-x86_64-64bit-gcc44/python/2.7.2-gl4/bin/python

S3DF Cheatsheet:

  • Install ssh config as described here
  • to login from terminal: ssh slacd
  • to ssh as glastraw, run kinit first
  • NOTE: use terminal not XQuartz(?)
  • afs was migrated to: /sdf/group/fermi/a/
    • nfs was migrated to: /sdf/group/fermi/g/
    Info
    titleSingularity cheatsheet

    source /sdf/group/fermi/ground/PipelineConfig/L1Proc/6.0/start_rhel6.sh → updated to 09/22 container version!

    singularity shell --bind /sdf/group/fermi/a:/afs/slac/g/glast --bind /sdf:/sdf /sdf/group/fermi/sw/containers/fermi-rhel6.20230922.sif 

    Info
    titlePipeline cheatsheet

    singularity exec --bind /sdf/group/fermi/a:/afs/slac/g/glast --bind /sdf:/sdf /sdf/group/fermi/sw/containers/fermi-rhel6.20230922.sif ./install.py

    /sdf/home/g/glast/a/pipeline-II/prod/pipeline load -m DEV /sdf/group/fermi/ground/PipelineConfig/L1Proc/6.0/xml/L1ProcS3df-6.0.xml


    Historical information (obsolete):

    • CVS, to passively download old stuff:  CVSROOT=/sdf/group/fermi/g/glast_ground/cvs
    • CVS, when live version is needed: CVSROOT=:ext:monzani@centaurusa.slac.stanford.edu:/nfs/slac/g/glast/ground/cvs
    • the CVS viewer was moved a new location