You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 17
Next »
- Add S3DF/S3DFDATA sites to DEV Pipeline, for L1 testing
- The PROD pipeline interface hangs when loading a task (example)
- Ability to delete tasks with runs from the pipeline admin (DEV and PROD)
SLURM questions:
- Document all SLURM options supported by the pipeline interface Brian Van Klaveren
- What are all the available SLURM configurations (queue, memory, etc.) for S3DF? Yang, Wei
- Do we need the analogous of a preempt queue in S3DF? What would that be? Richard Dubois
- TBD: currently only roma and milano partitions available during the test period. Worst case is we reserve 1k cores for L1 when the time comes.
Trying to create a branch in svac/L1Pipeline with cvs tag -b, however:
- stag complains that my tag is not standard. But I’m not calling stag (L1Pipeline was never ported to SCons)
/afs/slac.stanford.edu/g/glast/applications/install/@sys/usr/bin/stag
disappeared from path (see comment below)- stagging with -custom and -branch does not work (basically, stag won’t let me create a branch either)
Here are some options (porting the whole svac to git might be best for long-term support)
- disentangle cvs from stag (no idea what that would look like, and why cvs is still attached to stag on s3df)
- port svac to git, which honestly at this point would be my preferred option (no need to make it public tho)
- SCons symlinks were not migrated correctly:
/afs/slac/g/glast/applications/install/@sys/usr/bin/scons
was not copied to /sdf- RD: not sure the correct fix for the @sys, but it did get copied to /sdf/data/fermi/a/applications/SCons/2.1.0/bin/scons and can be used from there
- MEM: this is an acceptable workaround for now. But really we should get rid of this whole SCons nightmare
- stag is also broken:
/afs/slac.stanford.edu/g/glast/applications/install/@sys/usr/bin/stag
was not installed correctly on /sdf- RD: stag is was linked to nfs and is here: /sdf/data/fermi/n/u05/stag/0.2.5/redhat6-x86_64-64bit/stag
- MEM: this is also broken (just try calling it). The only working path is the afs one, but only works from rhel6-64
- xrootd stuff is also broken: PROD, DEV folders in here:
/afs/slac.stanford.edu/g/glast/applications/xrootd
- Need a container where the architecture is set explicitly to rhel6 instead of centos6 (architecture name confuses SCons)
- While we are at it, can we add CVS and vim to the new container?
- Singularity gets confused by symlinks (example:
/sdf/home/m/monzani/fermi-user
vs /sdf/group/fermi/user/monzani
). Is that the expected behavior? For now, workaround is --bind /sdf:/sdf
(ugly but workable)
- Surreal behavior with folder names: only difference is in the last line
#!/sdf/group/fermi/a/isoc/flightOps/rhel6_gcc44/ISOC_PROD/bin/shisoc python2.6
import os
import sys
L1Name = os.environ.get('L1_TASK_NAME') or "L1Proc"
L1Version = os.environ.get('PIPELINE_TASKVERSION') or os.environ.get('L1_TASK_VERSION') or "5.9"
BuildVolume = '/sdf/group/fermi/a/ground/releases/volume11'
L1BuildBase = os.environ.get('L1_BUILD_DIR') or os.path.join(BuildVolume, 'L1Proc')
L1Build = os.path.join(L1BuildBase, '5.9')
#!/sdf/group/fermi/a/isoc/flightOps/rhel6_gcc44/ISOC_PROD/bin/shisoc python2.6
import os
import sys
L1Name = os.environ.get('L1_TASK_NAME') or "L1Proc"
L1Version = os.environ.get('PIPELINE_TASKVERSION') or os.environ.get('L1_TASK_VERSION') or "5.9"
BuildVolume = '/sdf/group/fermi/a/ground/releases/volume11'
L1BuildBase = os.environ.get('L1_BUILD_DIR') or os.path.join(BuildVolume, 'L1Proc')
L1Build = os.path.join(L1BuildBase, L1Version)