Page History

...

• Update ASP CVS repository with most recent modifications.
• Convert the ASP and ASP-scons CVS areas to git repos.
  • Move these repos to the fermi-lat github org.
• Write up current installation instructionson Confluence.
• S3DF installation of GPLtools:  /sdf/data/fermi/a/ground/PipelineConfig/GPLtools/prod.   Modifications are needed.  See below.
• Singularity container (generated by Wei) to use with /sdf/data/fermi/a/ground/releases/volume02/ScienceTools-09-32-05/:   /sdf/group/fermi/sw/containers/fermi-rhel6.20230922.sif .  This contains the 64-bit BLAS and LAPACK libraries needed by numpy.
• Example ASP P-II task on S3DF using legacy Fermi software releases, including Oracle db queries:  ASP_db_query_test
• Example ASP P-II task that uses the dataCatalog and GPLtools staging code: ASP_datacat_test
  • Modify GPLtools to use paths to xrootd locations on S3DF
• Prototype P-II task to demo registering outputs to datacatalog
• Prototype P-II task that creates substreams
  • NB:  Creating substreams seem to need to be done from a python3 script included as a <job> via a CDATA section replacing an explicit executable attribute.    Would be good to be able to import custom python code in that CDATA section.
  • Supposedly, using the pipeline  object within scriptlets works now (We'll see...)
• Code to replace scriptlets  in order to run datacat and pipeline commands in batch jobs
  • This can be done via code like this.
• Ensure schema locations in XML pipeline defs use "https": 

  Code Block
  <pipeline xmlns="http://glast-ground.slac.stanford.edu/pipeline" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:schemaLocation="http://glast-ground.slac.stanford.edu/pipeline https://glast-ground.slac.stanford.edu/Pipeline-II/schemas/2.2/pipeline.xsd">

• File paths used by ASP that need S3DF equivalents:
  • /afs/slac/g/glast/ground/links/data/ASP/GCN_Archive (GCN_Notice_processor.py)
  • /afs/slac/g/glast/ground/links/data/ASP/scratch (makeDrpLcTables.py)
  • /afs/slac/g/glast/isoc/flightOps/rhel5_gcc41/ISOC_PROD/bin/isoc (makeDrpLcTables_part2.py, fastCopy.py for FastCopy)
  • /afs/slac/g/glast/ground/links/data/ASP/Results (aspLauncher.py)
  • /afs/slac/g/glast/ground/ASP/catalogs (launchStreams.py)
  • /afs/slac/g/glast/ground (DbEntry.py)
• Building ASP against ST-09-32-05 in container.  ST-10-01-01 is available at /sdf/data/fermi/a/ground/releases/volume09/ScienceTools-10-01-01/
• Sort out batch options to use in P-II task xml definitions, e.g., --account fermi.
• S3DF installation of Oracle drivers.  On rhel6-64/centos7, the required distribution is in /afs/slac.stanford.edu/package/oracle/f/11.1.0/amd64_linux26/11.1.0.   An rsync'd copy on /sdf seems to work.
• Replace trscron job for sending Monitored Source light curve data to GSFC.
• Update ASP P-II tasks to run on s3df, using dev db tables.
  • Update paths to pipeline commands, etc.
• Remove all GRB-related executables from P-II tasks.

...