Versions Compared

Old Version 231

changes.mady.by.user Kelly (Arrighi), Heather

Saved on

compared with

New Version 232

changes.mady.by.user Kelly (Arrighi), Heather

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Week of Feb 26 2018

  •  Fermi
    •  get Brian to fully clean up between GR Jenkins builds
    •  get L1 GR to build 
  •  nEXO
    •  Work on HistogramSvc for ROOT files
  •  DM-DC2
    •  Run Monday Meeting and write minutes
    •  Update GitHub issues
    •  Contact Chris W about flats
    •  Talk to Tom about setting up directory for transfer to IN2P3
    •  test ingestDriver.py at NERSC
  •  NERSC
  •  SLAC
    •  Watch JupyterLab demo install and keep Steve informed
    •  Try out eT DB updates to prep for Joanne's departure
  •  ATS At Harvard

Week of Feb 19 2018

  •  Fermi
    •  Finish Anaconda 3 install
    •  Fix up L1 builds on Jenkins
    •  Fermi Meeting
  •  nEXO
    •  Start new HistogramSvc for ROOT files
    •  Simulation Meeting Thursday
  •  DM-DC2
    •  First Task Force Meeting
    •  Migrate DC2 Cookbook issues to IPP (ImageProcessingPipeline) repo
    •  Create new issues in IPP repo
  •  NERSC
    •  sort out HPSS allocation
  •  SLAC
    •  Help David L access his account
  •  ATS at Harvard
    •  Meeting with Tony & Owen on Tuesday
    •  Go to Harvard on Friday

Week of Feb 12 2018

  •  Fermi
    •  attend offline meetings!
    •  Finish Anaconda3 installation 
    •  Work out distribution of externals in the new regime
    •  Install swig 3.0.12 RH6 and RH7 on GLAST_EXT
    •  work out L1 build in Jenkins
  •  nEXO
    •  Meet with sims people
  •  NERSC Support
    •  re-re-install cosmosis to help out the WL folks who wanted the latest along the develop branch 
    •  Reach out to Joe to encourage tagging of cosmosis and provide some feedback on their setup script for NERSC
    •  Reach out to wl pipeline group to ask about cosmosis installations.
    •  Start up automated weekly stack installations
    •  Document weekly stack installations
    •  Send Joe Z updates to the setup-cosmosis-nersc script
    •  Help Erfan get going with cosmosis at NERSC
  •  SLAC support
    •  Read over Joanne's docs
    •  Pow-wow with Joanne over the DP and eT and the eT DB
    •  set up docker for Jim on IR2
    •  Update DESC and Fermi documentation on getting a SLAC account
  •  DRP/L2 Processing
    •  Task Force
      •  Phil mentioned that "they" had to write up a charge - figure out what that means
        •  Modify charge after receiving suggestions
      •  At Nicolas' suggestion reach out to Fabio H concerning weekly stack installations
  •   ATS at Harvard
    •  Attend CCS meeting to find out how best to support Harvard
    •  Contact Harvard to sort out list of devices & plans for their use for testing
    •  Meet with Kirk
    •  write up minutes from Kirk/Nick meeting and prepare to iterate with Kirk

Week of Feb 5 2018

Week of Jan 29nd 2018

  •  Data Portal
    •  support Joanne as she learns about the plotting
  •  DRP/L2 Processing
    •  Check on James P
    •  Get a copy of the DC1 WF going
    •  Update WF to run Jim's latest imSim Data 
    •  Learn how to deal with pipeline issue in JMX dealing with large number of jobs stuck in submitted state
    •  Run Jim's imSim data through SLAC WF (first attempt in progress)
    •  Prep to run using Phosim 1.1p
    •  Document using the Slac WF and add scripts to a GH repo in LSSTDESC area
    •  attend Friday dc2 meeting
  •  SLAC support
    •  review, release, and install the latest version of datacat-utilities
    •  Read over Joanne's documentation
    •  -automate weekly install of DMstack binaries for camera- 
    •  install datacat-utilities on lsst-dc10
  •  NERSC support
    •  Upload latest SQRE DMstack (w_2018_03) Docker image into Shifter
    •  Install w_2018_03 plus sims from source
    •  Create w_2018_03+sims Shifter Image (debugging issue with afw - issue identified.. fixing it is something else)
      •  create sims Shifter image the long way, without using lsstsqre's base image
    •  Install WeakLensingDeblending for Javier.
  •  Random LSST Camera and DESC items
    •  SSim meeting
    •  Meet with Wil on Friday
    •  Write up notes from meeting
    •  Look over DESC architects confluence page
  •  nEXO
    •  reinstall SNiPER in the new disk space
    •  try out examples
    •  sign up for the collaboration mailing list
  •  Fermi
    •  Investigate trouble with AcdDigi in L1 branch
    •  Find a way to upgrade Anaconda external for ST
    •  Install miniconda py3
    •  Contact Helen

Week of Jan 22nd 2018

  •  Data Portal
    •  Consider rerunning all the ITL sensors in need of updated MET ingest and MET 05 by hand
    •  plots - commit latest code and write up intro for Joanne
    •  CCB request and deploy MET updates for the DP
    •  Deploy DP 0.5.11
  •  SLAC support
    •  automate weekly install of DMstack binaries for camera (Jim doesn't seem to feel this is a priority)
    •  Read over Joanne's documentation
  •  NERSC support
    •  update DESCQA install to include tensor flow
    •  write up instructions for w_2018_01 shifter install
    •  automate installation of shifter images
    •  Create docker image of new sims release based on w_2018_04
    •  Reintall cosmosis for WL group due to updated FFTW module, trying cray-fftw this time
    •  Send instructions to Josh to set up his own test galsim using the extlibs we already have set up for Cori KNL gcc
    •  install Phosim 3.7.7 and 3.7.8 for knl-intel and haswell-intel
    •  meet with Mustafa about HPSS
    •  Fix Cosmosis installation
  •  Workflows/L2 DC2 Pipeline
    •  send plea to Tony
    •  Meeting with Tom about using the SLAC pipeline + pilots
    •  work out setting up test Pipeline2 job reusing DC1 L2 task and a handful of visits
    •  Prep for CI meeting
  •  Fermi
    •  Work through errors with AcdDigi on L1 branch
    •  Set up new Anaconda external
    •  Work out with Brian the requirements for setting up externals these days
  •  nEXO
    •  get Tony an disk space estimate for SLAC installs
    •  get some disk space from Tony at SLAC
    •  set up SNiPER at SLAC in a temp location (in progress)
    •   attend Tuesday sims meeting

Week of Jan 15th 2018

  •  Data Portal
    •  revisit plots
    •  Holding off MET updates until new traveler to reingest ITL sensors is available
  •  datacat-utilities
    •  Install new release 0.0.9 at SLAC
  •  SLAC support
    •  automate weekly installs of DMstack binaries for camera
  •  NERSC Support
    •  update DESCQA install to include tensorflow
    •  Write up instructions for w_2018_01 shifter install
    •  automate installation of shifter images
    •  Add WeakLensingDeblending 
    •  Check on CCL installation
  •  Workflows/L2 DC2 Pipeline
  •  Fermi
    •  Set up draft f2f meeting page Fermi Software Week Winter 2018
    •  Continue working on GR L1 branch - completed checkout of L1 code successfully
    •  Work through errors with AcdDigi on L1 branch
  •  nEXO

Week of Jan 8th 2018

  •  DataPortal
    •  finish off ITL-3800C-368 ingest after sorting out http authentication errors
    •  revisit plots
    •  Hold off on MET updates until new traveler to reingest ITL sensors is available.
  •  datacat-utilities
    •  Install new release
  •  SLAC suport
    •  Install w_2018_01 binaries lsst_distrib for camera
    •  Work out automated weekly installs via cron (u1 is 97% full, ~990 GB left)
    •  Install Merlin's eotest branch
  •  NERSC Support
  •  Workflows/L2 DC2 pipeline
    •  Set up lsst_sims docker image for NERSC & uploaded to Dockerhub
    •  Try another lsst_sims docker image using lsstsqre image as the base rather than centos7 (initially failed, have new plans to retry)
  •  Fermi
  •  nEXO & SNiPER

Week of Dec 18th 2017

  •  DataPortal
    •  MET updates
    •  
      Jira
      serverSLAC National Accelerator Laboratory
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
      keyLSSTTD-1221

    •  
      Jira
      serverSLAC National Accelerator Laboratory
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
      keyLSSTTD-1220
  •  Workflows/L2 DC2 pipeline
    •  Reproduce error James P sees
    •  Repeat experiment with multiple visits
    •  Repeat experiment using Head of ImageProcessingPipeline repo
    •  Potentially meet with Jim & James this week
    •  Catch up on Parsl progress
  •  CI
    •  Set up new Docker image to use with imSIm's travis builds
    •  Create a smaller Docker images for use with travis
  •  NERSC Support
    •  Shared Stack
    •  JupyterHub documentation and simplification
    •  clean old project space by removing MBII-snapshots
    •  Request reduction in old project space quota to 40 TB
  •  imSim
  •  Fermi
    •  Set up GR L1 branch in GitHub
    •  Test build of L1 branch
  •  SNiPER
    •  Read up on SNiPER
    •  Meet with Lisa K on Tuesday 5 PM Eastern
    •  send feedback on documentation
    •  set up user accounts on Wiki, LLNL, OSG
    •  talk to Tony about nExo

 

Week of Dec 11th 2017

  •  Data Portal
    •  Deploy ETCCB-429
    •  MET updates - almost complete
  •  CCS ATS
    •  Install mvn and associated libraries and get Nick going with Lakeshore driver
  •  Workflows/L2 DC2 pipeline
    •  Send James P a link to the recordings from last week
    •  Repeat test using Jim's parsl wf at NERSC
  •  CI
    •  Docker image for use in travis builds
  •  imSIM
  •  NERSC Support
    •  Fresh Shared Stack installation for KNL - stuck working out disk space in the common software area  – started
    •  dmtcp installation
    •  Update jupiter-dev documentation
    •  quota checker
  •  Fermi
    •  Try GR with repoman
    •  Documentation

Week of Nov 27th 2017

  •  L2 pipeline DC1 rerun
    •  Finish off write up of chat with Tony
    •  Ask James P about remote participation during Spring Week
      M & W 8 - 12 central time 
    •  Try a run using Simon's cookbook
    •  Meet with Jim & James P
      •  write up notes
    •  Try Parsl pipeline at NERSC
  •  NERSC software installations
    •  Shared Stack installation - verify that the DESCQA and py3 install is working
      •  Add CCL
    •  Install new sims 2.4.2 using DMstack w.2017.46 Haswell-gcc6 using py3
    •  Ask about cosmosis needs
      •  Install cosmosis, TreeCorr, and MLZ in projecta area under the WL WG directory
    •  Install phosim 3.7.3
    •  dmtcp
    •  quota checker
    •  Consider requesting more common software space
      obtained 50 GB more 
  •  NERSC project space
    •  Verify MBII-snapshots is in HPSS and remove from project area
      Found the content in the desc hsi area, and viewed the contents of some snap.tar via htar -tf CS/MBII-DMO-snapshots/snapdir_074.tar   Also noticed that the group_xxx.tar files are duplicated in the Archive home directory.  Contacted Mustafa to sort that out and remove duplicates.
  •  Data Portal
    •  MET updates
      •  Meet with Anders and Seth
  •  Harvard ATS
    •  Check that ccs reboot restarts the MySQL, LocalRestServices, and Trending
    •  Check in with Nick about the Lakeshore
  •  Fermi
    •  Try GR using repoman
    •  Documentation
  •  Discuss Gaudi for nEXO
    https://gaudi.web.cern.ch/gaudi/releases/v30r0/
    http://lcgsoft.web.cern.ch/lcgsoft/release/
    https://gitlab.cern.ch/gaudi/Gaudi
  •  Attend Stanford IT Unconference
    https://itunconference.stanford.edu/sessions/remote

Week of Nov 20th 2017

Week of Nov 13th 2017

  •  Harvard ATS
    •  Create user accounts and ccs sudo account which grants sudo access to all users
    •  Set up directory structure following IR2 model
    •  Install DMstack v14
    •  Install eotest compatible with v14
    •  document the above on the Confluence page ATS Test Stands
    •  Set up Trending Database
      •  MariaDB installation
  •  L2 pipeline DC1 rerun https://github.com/LSSTDESC/SSim_DC1/issues/47
    •  Meet with Jim on Monday
      •  Write up Notes
    •  Contact James P
    •  Try to run Simon's Cookbook
    •  Chat with Tony
  •  NERSC Software Installations
    •  Shared Stack
      •  first try failed due to conflicts with libiconv installed presumably for 
    •  Phosim 3.7.2 bug fix for haswell gcc & intel, KNL gcc & intel 
    •  DMTCP intel build
    •  Help out with cosmosis installation https://github.com/LSSTDESC/WLPipe/issues/31
  •  Data Portal
    •  Chat with Tony about plots
    •  Plots
      •  Bad Channels
      •  read noise
  •  NERSC Project Space
    •  Test MBII-snapshot HPSS copy and remove from project area

Week of Nov 6th 2017

  •  QA Testing
    •  Investigate cause of differences between Lauren's plots from NCSA and those generated at NERSC
      •  look at 'conda list' on both NCSA and NERSC (no differences discovered)
      •  generate cal_refs as was done at NCSA
      •  try gcc 6 build as is being done at NCSA
    •  Determine the cause of differences between ref catalogs at NCSA and NERSC
  •  L2 Pipeline
  •  Data Portal
    •  Finish off first round of Richard's plots
      •  bad channels
      •  read_noise
  •  NERSC Project Space
    •  test MBII-snapshot HPSS copy
  •  ATS
    •  prep for Tony & Owen's visit
  •  Installations
    •  SLAC
      •  Py3 on SLAC RH6-64 for Joanne to test out
    •  NERSC
      •  finish testing NERSC's cfitsio installation
      •  install phosim 3.7.1 for Cori haswell gcc & intel, KNL gcc & intel
        •  ask Glenn to install module files
      •  install phosim 3.7.2 for Cori haswell gcc & intel, KNL gcc & intel
      •  Shared Stack Installation using w.2017.41
        •  Query DESCQA and CCL groups
        •  py2 w.2017.41 installation
        •  py3 w.2017.41 installation

Week of Oct 30th 2017

  •  Data Portal
    •  Plots
  •  DC2 prep
    •  QA testing
      •  w.2017.41 installation plus branches of obs_lsstSim and pipe_analysis
      •  Run visitAnalysis, coaddAnalysis, etc
  •  NERSC Workflow work
  •  NERSC Project Space
    •  Work with Mustafa to get MBII-snapshot backed up
  •  NERSC Installations
    •  w.2017.? stack installation plus sims - waiting on new sims release using the camera geometry updates
    •  Phosim 3.7
      •  Testing NERSC cfitsio installation
  •  SLAC Installations
    •  descsoft w.2017.? stack plus sims installation
  •  CCS
    •  See if there is any prep work necessary for Tony's visit later in Nov
      •  Request Nick set up new box with CentOS7
      •  set up user accounts and groups
    •  Determine if Mike is still coming out ~Nov 1st
  •  Attend SuperCDMS meeting concerning their potential move to NERSC.

Week of Oct 23rd 2017

  •  Data Portal
    •  Richard's Plots - bad channels (in progress)
    •  Look at new Report Labels and determine what the DP still requires from the Reports Table
      Jira
      serverSLAC National Accelerator Laboratory
      serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
      keyLSSTTD-1190
       
    •  Make sure we request a restart of Tomcat
  •  NERSC Workflow work
    •  Try out HTCondor and Pegasus on cori21
    •  Take a look at Jim's imsim script
  •  NERSC Project Space
    •  Set the old project directory to read-only (in progress)
      •  claim desc ownership of the whole directory space
    •  Clear out all directories where the owners have indicated it is fine to do so
    •  Determine the end state of merging old project and projecta areas and handling of www
    •  Set up disk quota monitor
    •  Coordinate HPSS storage of MBII-snapshots
  •  NERSC installations
    •  Phosim v3.7
      •  Coordinate with NERSC to get cfitsio v3410 installed (in progress)
      •  Rebuild using NERSC supplied cfitsio
      •  Coordinate with Glenn to set up module files
    •  Set up software for DC2 testing
    •  v14_0 lsst_distrib + sims 2.4.0
  •  SLAC installations
    •  Set up same imsim at SLAC as used at NERSC for KNL testing (in progress)
    •  v14_0 on RH6-64 public (on hold pending resolution to obs_base testCameraMapper.py unit test failure)
      •  camera area
      •  descsoft
      •  test eotest branch
    •  v14_0 on IR-2
  •  Document KNL vs Haskell mkl imsim testing at NERSC
    •  Initial mkl tests indicate no huge improvement - putting testing on back burner for now
  •  CCS - still waiting to hear from Peter, I cannot access their TS3 machine to install and test a new version of the test stand subsystem

Week of Oct 16th 2017

  •  Richard's plots in DP
    •  Complete Sensor Arrival Time plot
    •  Create JSON 
  •  NERSC installations
    •  Phosim v3.7
      •  Try out NERSC supplied FFTW 
      •  Request new cfitsio v3410 installation required by Phosim
      •  Complete installation of all flavors:  haswell-gcc, haswell-intel, knl-gcc, knl-intel
        •  Waiting on cfitsio v3410 before knl-intel will be available
  •  NERSC Project Space
    •  Notify all users of the need to migrate
    •  Handle all help requests - migrating some directories to the new projecta area
    •  Install disk quota monitor
    •  Check on old project status and prepare to set it to read-only
    •  Coordinate backup of nkhandai data to tape with Mustafa
  •  Install latest version of teststand subsystem at Harvard there is a machine made available - waiting on response from Peter 
  •  Keep an eye out for v14 DMstack

Week of October 9th 2017

  •  Continue work on Richard's plots in DP
  •  
    Jira
    serverSLAC National Accelerator Laboratory
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
    keyLSSTTD-1165
    •  Released Data Portal 0.5.7
  •  Testing imSim built on KNL
    •  Continue MKL tests at NERSC once the system recovers from the weekend outage
    •  Install the same version of the stack at SLAC to allow similar testing with and without MKL
    •  Attempt build of DMstack using the intel compiler at NERSC
  •  NERSC Project space
    •  Install disk quota monitor
    •  Coordinate backup of nkhandai data to tape with Mustafa
    •  Prepare to set the old project area as read-only
    •  Send mail to all owners of files on the old project space
  •  NERSC HPSS SRU overallocation
  •  Install latest version of datacat-utilities 0.0.8 at SLAC
  •  Install the latest teststand snapshot at Harvard to support use of the monochromator when Peter returns.
    •  Contacted Peter - I can't connect to the archon machine to install the test stand snapshot
    •  Investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly create a harvard main routine.
  •  Keep an eye out for the v14 DMstack release and prepare to install on IR-2 and NERSC
    •  This release will include binary installations.. I'm thinking we may still want to install from source, perhaps that's most useful on IR-2, but we'll see.

Week of October 2nd 2017

  •  Release DataPortal 0.5.6 which will include
    •  Addition of Grade and Procurement labels for the Sensor Acceptance pages
    •  RTM assignment
      Jira
      serverSLAC National Accelerator Laboratory
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
      keyLSSTTD-1121

    •  NCR "Ready to close"
      Jira
      serverSLAC National Accelerator Laboratory
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
      keyLSSTTD-1144

  •  Implement Richard's plots in the DP starting from some of Tony's examples for trending and Richard's python code.
  •  Continue testing KNL performance using galsim and imsim
    •  Attempt to compile galsim and possibly all of DMstack with the intel compiler
    •  Run longer tests using the MKL libraries on Haswell and KNL
    •  Rebuild imsim with MKL, using an updated meas_base
  •  Install an initial shared stack in the new software area at NERSC for testing as discussed at last week's "Shared Stack" meeting
  •  NERSC Project Space
    •  Set up an initial quota monitor at NERSC
    •  Re-connect with Mustafa to get the mkhandai 42 TB archived to tape.
  •  Investigate an upgrade of the monochromator driver at Harvard and confirm that to achieve what Peter plans to do - to run his ATS work in the lab alongside most of the devices from his TS3 setup.. is possible by running both the teststand and rafts subsystem simultaneously.
    •  Install latest SNAPSHOT of test stand at Harvard and investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly creating a harvard main routine.
  •  Keep an eye out for the v14 release of the DMstack and prepare to install at SLAC/IR-2 and NERSC.
    Though this is assigned to Jim, so perhaps I have no need to worry about SLAC:
    Jira
    serverSLAC National Accelerator Laboratory
    serverId1b8dc293-975d-3f2d-b988-18fd9aec1546
    keyLSSTTD-1157

...