Page History
Essential dependencies:
- Move away from afs disks
- Run executables using container
- Use SLURM commands in xml
- Migrate all L1 packages to github
Migration milestones:
- Build L1 locally on containers
- Run single task in S3DF pipeline
- Update the L1Proc xml template
- Run full L1Proc in S3DF with LPA
Infrastructure needs:
- CVS is going away
- Singularity containers
- Pipeline updates
- Restore pipeline_wrapper
- Fix the log viewer
- Data migration (u41)
Non-L1 milestones:
- Test full L1Proc in S3Df with LCI data
- Update non-L1 xml (flagFT2, merge, setQual)
- Test non-L1 tasks (flag FT2, merge, setQual)
- Migrate DQM/trending reports
- Migrate cronjobs for DQM reports
Sustainability and optimization:
-
Get rid of SCons builds for L1 code -
Upgrade python to v3 and above - Can we avoid crumbling altogether?
- At a minimum, reduce # of crumbs
- Use flash for staging instead of xrootd?
Open questions:
- do we still need a wrapper for cleanup?
- do we need isocRun for every executable?
- what are the SLURM queue options?
- non-preemptable queue: --account fermi:L1
- how does the scratch space work on S3DF?
S3DF Cheatsheet:
- Install ssh config as described here
- to login from terminal:
ssh slacd - to ssh as glastraw, run
kinitfirst - NOTE: use terminal not XQuartz(?)
- afs was migrated to:
/sdf/group/fermi/a/ - nfs was migrated to:
/sdf/group/fermi/g/
Where to find/install stuff:
- working area:
/sdf/group/fermi/ground/PipelineConfig/L1Proc - pipeline cmd:
/sdf/home/g/glast/a/pipeline-II/prod/pipeline - singularity example (05/02/23): pipeline task, xml file
- additional slurm parameters (10/10/23):
--account fermi:L1 - python used by ST: /afs/slac/g/glast/ground/GLAST_EXT/redhat6-x86_64-64bit-gcc44/python/2.7.2-gl4/bin/python
| Info | ||
|---|---|---|
| ||
source /sdf/group/fermi/ground/PipelineConfig/L1Proc/6.0/start_rhel6.sh → updated to 09/22 container version! singularity shell --bind /sdf/group/fermi/a:/afs/slac/g/glast --bind /sdf:/sdf /sdf/group/fermi/sw/containers/fermi-rhel6.20230922.sif |
| Info | ||
|---|---|---|
| ||
singularity exec --bind /sdf/group/fermi/a:/afs/slac/g/glast --bind /sdf:/sdf /sdf/group/fermi/sw/containers/fermi-rhel6.20230922.sif ./install.py /sdf/home/g/glast/a/pipeline-II/prod/pipeline load -m DEV /sdf/group/fermi/ground/PipelineConfig/L1Proc/6.0/xml/L1ProcS3df-6.0.xml |
Historical information (obsolete):
- CVS, to passively download old stuff:
CVSROOT=/sdf/group/fermi/g/glast_ground/cvs - CVS, when live version is needed:
CVSROOT=:ext:monzani@centaurusa.slac.stanford.edu:/nfs/slac/g/glast/ground/cvs - the CVS viewer was moved a new location
Overview
Content Tools