Page History

...

• Update ASP CVS repository with most recent modifications.
• Convert the ASP and ASP-scons CVS areas to git repos.
  • Move these repos to the fermi-lat github org.
• Write up current installation instructionson Confluence.
• S3DF installation of GPLtools:  /sdf/data/fermi/a/ground/PipelineConfig/GPLtools/prod.   Modifications are needed.  See below.
• Singularity container (generated by Wei) to use with /sdf/data/fermi/a/ground/releases/volume02/ScienceTools-09-32-05/:   /sdf/group/fermi/sw/containers/fermi-rhel6.20230922.sif .  This contains the 64-bit BLAS and LAPACK libraries needed by numpy.
• Example ASP P-II task on S3DF using legacy Fermi software releases, including Oracle db queries:  ASP_db_query_test
• Example ASP P-II task that uses the dataCatalog and GPLtools staging code: ASP_datacat_test
  • Modify GPLtools to use paths to xrootd locations on S3DF
• Prototype P-II task to demo registering outputs to datacatalog
• Prototype P-II task that creates substreams
  • NB:  Creating substreams seem to need to be done from a python3 script included as a <job> via a CDATA section replacing an explicit executable attribute.    Would be good to be able to import custom python code in that CDATA section.
  • Supposedly, using the pipeline  object within scriptlets works now (We'll see...)
• Code to replace scriptlets  in order to run datacat and pipeline commands in batch jobs
  • This can be done via code like this.

• Ensure schema locations in XML pipeline defs use "https": 

  Code Block
  <pipeline xmlns="http://glast-ground.slac.stanford.edu/pipeline" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:schemaLocation="http://glast-ground.slac.stanford.edu/pipeline https://glast-ground.slac.stanford.edu/Pipeline-II/schemas/2.2/pipeline.xsd">

• File paths used by ASP that need S3DF equivalents:
  • /afs/slac/g/glast/ground/links/data/ASP/GCN_Archive (GCN_Notice_processor.py)
  • /afs/slac/g/glast/ground/links/data/ASP/scratch (makeDrpLcTables.py)
  • /afs/slac/g/glast/isoc/flightOps/rhel5_gcc41/ISOC_PROD/bin/isoc (makeDrpLcTables_part2.py, fastCopy.py for FastCopy)
  • /afs/slac/g/glast/ground/links/data/ASP/Results (aspLauncher.py)
  • /afs/slac/g/glast/ground/ASP/catalogs (launchStreams.py)
  • /afs/slac/g/glast/ground (DbEntry.py)
• Building ASP against ST-09-32-05 in container.  ST-10-01-01 is available at /sdf/data/fermi/a/ground/releases/volume09/ScienceTools-10-01-01/
  • It would be good to be able to build against ST-10-01-01 since the bash wrapper scripts generated by our SConstruct files unfortunately write explicit paths rather than using env vars so that all of the wrapper scripts need to be updated to use the SDF3 paths.   However, the SCons build code for ST is extremely obscure, and it's not at all clear how the  value provided via the  --with-GLAST-EXT  command line option is passed (or if it is in fact passed at all) to the generated bash wrappers.
• Sort out batch options to use in P-II task xml definitions, e.g., --account fermi.
• S3DF installation of Oracle drivers.  On rhel6-64/centos7, the required distribution is in /afs/slac.stanford.edu/package/oracle/f/11.1.0/amd64_linux26/11.1.0.   An rsync'd copy on /sdf seems to work.
• Replace trscron job for sending Monitored Source light curve data to GSFC.
• Update ASP P-II tasks to run on s3df, using dev db tables.
  • Update paths to pipeline commands, etc.
• Remove all GRB-related executables from P-II tasks.

...