Partial list of activities completed for Pass8.
  • Update Reprocessing Summary page
  • Create P300 category in History Processing tables
  • Create new P300 Confluence page
  • Clone P202-ROOT/config to P300-ROOT/config
  • Select GR (20-08-12) to start off, but will change
    • GR now built for rhel6-64, but not validated - revert to rhel5-32
  • (arbitrarily) select ST (09-31-02) to start off, but will likely change
    • ST now built for rhel5-64 and rhel6-64 - not certain which should/must be used...
  • Discuss w/Yemi about generalized scratch space in batch farm, and short of that, the best way to determine whether to use /scratch or /lustre/ki/pfs/fermi_scratch
  • Get working jobOpts file
    • combine initial templates from Tracy/Johan (Pass8) and P202-ROOT
    • correction for need to produce GCR files (remove Cal3.Members={};)
    • correction for need to produce MERIT (not PRIME) ntuple
    • corrections for new ALG naming (e.g., PtValsAlg1 and AcdReconAlgV2)
  • gitify P300-ROOT and commonTools working directories for better backup, tagging, etc.
  • Create new version of commonTools, 00-02-00 for Pass8
  • Update commonTools/setupGR.sh
    • replace rhel4/rhel5 with rhel5/rhel6
  • Update commonTools/setupSkimmer.sh
    • Select rhel5 only (must be compatible with TSkim, ROOT, and GR rootData.so files)
    • TSkim v08-02-05 (rhel5-32 and rhel6-64 available...no rhel5-64)
    • ROOT version v5.34.03.gr01 (use rhel5-32 build)
    • fix problem with $ROOTSYS directory structure (for releases beyond 5.26)
    • fix problem with specification of file paths to search for rootData.so files
  • Update .xrayrc to use heasoft 6.13
  • Update commonTools/setupFtools.sh
    • remove force to 32-bit tools (no longer needed)
  • update repTools.py
    • clean up code
  • Update genRunList.sh and generate new list of DIGI files to reprocess (only first handful)
  • Improve/debug ability of Pipeline task to limit the number of events reprocessed per run and per clump
  • Add feature to config.py to call python's resource.setrlimit() to limit core dumps to 512 kB
  • New repro configuration: like P202 but without CAL and with RELATION files being produced (and including TSkim file prefix and merge dict)
  • Remove various ROOT-related hacks in P202 which worked around now-obsolete problems in v5.24
  • Modify GPLTools to use proper scratch space, i.e. /lustre via infiniband on the bullets, if available
Still to do.
  • Tune clump size for xlong queue
    • trade off more shorter jobs vs (increased LSF and DB transactions, and horrible response of Pipeline web interface)
  • Tune trickleStream for max number of parallel jobs, watching NFS and xroot behavior
  • Clone/update scratch cleanup mechanism
  • Clone/update removal of old (non-Pass8) RECON/CAL files
  • Validate sIntent query criteria, then generate full DIGI list
  • Add new section to Official LAT Datasets page
  • Create P300-FITS for generating FITS files
Other issues
  • bullet0095 and bullet0151 do not have proper installation of libXm.so.4
  • No labels