...

•  Prioritize this task list
•  Make a detailed status for each of the main software package (Summary table below)
•  Make a very simple list of use cases: L1Proc, End user, Developer...
  
•  Summarize pros and cons of VMs vs Containers (considering use cases)
   Summarize pros and cons of Docker vs Singularity vs Shifter (considering use cases, and what computer centers like most...)
  
•  Summarize Virtualization and Containerization status at SLAC and CC-IN2P3
•  Can the Release Manager create and track the VM/Containers?

...

• Here is summary table with the main software packages
  • see data flow section of the Software Week Meeting notes
    

    • created by Johan on Tuesday 6^th 2017, limiting myself to RHEL5, RHEL6 and RHEL7/CentOS7
      

      Name	Build platform	Running platform	Special dependencies	Upgradable?	Existing VM	Existing container	Links	Comments	Date
      FastCopy	RHEL5	RHEL5 or RHEL6 ?	?				FASTCopy processing chain	to be reviewed by experts	06 Jun 2017
      Halfpipe	RHEL6	RHEL6	Commercial Qt !?	unlikely			Halfpipe	to be reviewed by experts	06 Jun 2017
      Isoc Monitoring	RHEL5	RHEL5, works on RHEL6						to be reviewed by Steve T.	06 Jun 2017
      GlastRelease	RHEL6	RHEL6	ROOT v5.34	may be with a lot of work and clean up				huge work, including science verification	06 Jun 2017
      DataMonitoring	RHEL6	RHEL6	python2, numpy, ROOT	probably if GR is upgraded				svac/monitor is similar to GR, then mostly python code in dataMonitoring	06 Jun 2017
      ScienceTools	RHEL7	RHEL7	RHEL7	NA	Giacomo	Sam's / Matt's		there was also one VM for the Fermi Summer school	06 Jun 2017
      ASP	RHEL7	RHEL7	RHEL7	NA				should be similar to ScienceTools, ask Jim	06 Jun 2017

...

...

Anchor
Container Techs Container Techs

Virtualization vs Containers? which containers?

• Many questions to be tackled under this item:
  • building vs running
  • our software vs 3rd party libraries vs system libraries
    
  • LAT calibrations, connection to mysql db, file transfer via xroot
    
    
• Following June 6th 2017 meeting, we'll initially focused on Containers as it's very unlikely that the SLAC farm will move to Virtual Machines (Brian's comment)
  • For SLAC farm - docker containers for GlastRelease. Need docker registry

  • Use their system run to RHEL6 container, but batch host is RHEL7.
    

  • Johan: VMs might still be useful for code development and debugging, for GR in particular.

...

• a few technologies to consider Docker, Shifter (Nersk for HPC) and Singularity
• advantages over a full VM usually are:
  • light weight
  • better performance
  • container has to be carefully built
• Docker: the container
• Shifter: NERSC container optimized for HPC
• Singularity: containers for scientist, compatible with Docker
  

...

Details for main packages

...

•  GlastRelease needs virtualizations
  • RHEL 6 is last release that we have the personnel to support
  • A few people running GlastRelease (Developers) - nice use case for Docker. Getting GlastRelease to run on your laptop is painful. 
  • GlastRelease carries around Geant4
• Note that in principle RHEL6 and RHEL7 are supposed to be binary compatible, so it might be possible to just run a GR built on RHEL6 on a RHEL7 host...
  • but there might be some issue with some dependencies from the GLAST_EXT
  • doing that would require some validation of the output of GR (whereas it may not be necessary if GR is fully containerized)
• Is there a distinction between Users and Developers for GlastRelease?
  • developer clearly need to build the code, where users just need a working run time environment
    
• Use Cases
  • L1 processing, reprocessing in SLAC batch farm
    • A few numbers on the main steps done in separate jobs in L1: digi, recon/merit, FT1
      
      • digitization is at the "chunk" level and is just converting LDF binary data into ROOT files : requires <50 parallel jobs per run
      • recon/merit is at the "crumb" and is the heaviest part, involves calibrations and query to mysql db, then reconstruction eats most of the CPU, while merit is straightforward: up to more that 500 parallel jobs per run
      • makeFT1 is at the "chunk" level and is (almost) just converting the merit ROOT file into a FITS file, requires <50 parallel jobs per run
      • taking all this, if L1 ends up processing 3 deliveries with 3 pieces of a run each, we might in principle get up to 3000 cores running in parallel, in practice I'd say we never go over 1500.
    • RHEL6 container on a RHEL7 host
    • do FT1, FT2 files go to xrootd? (Warren)yes, as far as I know (Johan)
      • Johan: I kind of remember that some step of L1Proc, makeFT1 or makeFT2 actually need both GR and ST to be setup to run... wonder if that's still true?
    • separate containers for L1?
      • Maybe not an issue if we can preload batch nodes. We're guessing ~5 GB image.
      • distinct simplified containers for digi, recon/merit and makeFT1 may actually be a good idea
        • containers for L1 could get rid of the Geant4 and all the MC stuff
        • but it means then to cache 3 different containers but the "recon" container could be potentially not too large
        • data reprocessing would need only "recon" and "FT1" containers
    • Other solution: a simple RHEL6 base container, used just as a runtime environment, while the software is accessed as usual from AFS or NFS (or whatever network file system is available)
  • Simulations at Lyon, SLAC, GRID
    • maybe the same as for SLAC - check with Samuel for details
    • simulations require a full GR in the container, as all the steps MC, DIGI, RECON, MERIT, are done within the same job
  • Developers & Users
    • maybe separate versions for debug symbols and source for developers. Could be on-demand production of this version.
     

...

• GPL_TOOLS (staging and logging)
• REPRO common tools
• REPRO task scripts
• GlastRelease
• ScienceTools
• GLAST_EXT software (e.g., python, root)
• Ftools (KIPAC installation)
• ROOT skimmer
• FITS skimmer (possibly unnecessary?)
• evtClassDefs
• calibration and alignment files
• diffuse models
• xroot tools
• xroot /glast/Scratch space
• /scratch on local batch machines
• data catalog query (FT2 file and current version of FITS files)
• mySQL DB (calibration and alignment)
• Fermi astrotools (could probably eliminate)

Anchor
Farms Status Farms Status

Farms

...

Status

...