You are viewing an old version of this page. View the current version.
Compare with Current
View Page History
« Previous
Version 21
Next »
- Add S3DF/S3DFDATA sites to DEV Pipeline, for L1 testing
- The PROD pipeline interface hangs when loading a task (example)
- Ability to delete tasks with runs from the pipeline admin (DEV and PROD)
SLURM questions:
- Document all SLURM options supported by the pipeline interface Brian Van Klaveren
- What are all the available SLURM configurations (queue, memory, etc.) for S3DF? Yang, Wei
- Do we need the analogous of a preempt queue in S3DF? What would that be? Richard Dubois
- TBD: currently only roma and milano partitions available during the test period. Worst case is we reserve 1k cores for L1 when the time comes.
Trying to create a branch in svac/L1Pipeline with cvs tag -b, however:
- stag complains that my tag is not standard. But I’m not calling stag (L1Pipeline was never ported to SCons)
/afs/slac.stanford.edu/g/glast/applications/install/@sys/usr/bin/stag
disappeared from path (see comment below)- stagging with -custom and -branch does not work (basically, stag won’t let me create a branch either)
Here are some options (porting the whole svac to git might be best for long-term support)
- disentangle cvs from stag (no idea what that would look like, and why cvs is still attached to stag on s3df)
- port svac to git, which honestly at this point would be my preferred option (no need to make it public tho)
Documentation from Joanne on making branch tags with cvs (super restrictive but potentially workable for now).
- SCons symlinks were not migrated correctly:
/afs/slac/g/glast/applications/install/@sys/usr/bin/scons
was not copied to /sdf- RD: not sure the correct fix for the @sys, but it did get copied to /sdf/data/fermi/a/applications/SCons/2.1.0/bin/scons and can be used from there
- MEM: this is an acceptable workaround for now. But really we should get rid of this whole SCons nightmare
- stag is also broken:
/afs/slac.stanford.edu/g/glast/applications/install/@sys/usr/bin/stag
was not installed correctly on /sdf- RD: stag is was linked to nfs and is here: /sdf/data/fermi/n/u05/stag/0.2.5/redhat6-x86_64-64bit/stag
- MEM: this is also broken (just try calling it). The only working path is the afs one, but only works from rhel6-64
- xrootd stuff is also broken. See for example the PROD, DEV folders in here:
/sdf/group/fermi/a/applications/xrootd(Wilko Kroeger )
- MEM: this /sdf/group/fermi/a/applications/xrootd/dist/v3.1.1/i386_rhel60/bin/glast_wrapper.sh does not work (can't find fs)
- Need a container where the architecture is set explicitly to rhel6 instead of centos6 (architecture name confuses SCons)
- While we are at it, can we add CVS and vim to the new container?
- Singularity gets confused by symlinks (example:
/sdf/home/m/monzani/fermi-user
vs /sdf/group/fermi/user/monzani
). Is that the expected behavior? For now, workaround is --bind /sdf:/sdf
(ugly but workable)
- Surreal behavior with folder names: only difference is in the last line
#!/sdf/group/fermi/a/isoc/flightOps/rhel6_gcc44/ISOC_PROD/bin/shisoc python2.6
import os
import sys
L1Name = os.environ.get('L1_TASK_NAME') or "L1Proc"
L1Version = os.environ.get('PIPELINE_TASKVERSION') or os.environ.get('L1_TASK_VERSION') or "5.9"
BuildVolume = '/sdf/group/fermi/a/ground/releases/volume11'
L1BuildBase = os.environ.get('L1_BUILD_DIR') or os.path.join(BuildVolume, 'L1Proc')
L1Build = os.path.join(L1BuildBase, '5.9')
#!/sdf/group/fermi/a/isoc/flightOps/rhel6_gcc44/ISOC_PROD/bin/shisoc python2.6
import os
import sys
L1Name = os.environ.get('L1_TASK_NAME') or "L1Proc"
L1Version = os.environ.get('PIPELINE_TASKVERSION') or os.environ.get('L1_TASK_VERSION') or "5.9"
BuildVolume = '/sdf/group/fermi/a/ground/releases/volume11'
L1BuildBase = os.environ.get('L1_BUILD_DIR') or os.path.join(BuildVolume, 'L1Proc')
L1Build = os.path.join(L1BuildBase, L1Version)