You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 187 Next »

Week of Jan 29nd 2018

  • Data Portal
    • support Joanne as she learns about the plotting
  • DRP/L2 Processing
    • Check on James P
    • Get a copy of the DC1 WF going
    • Update WF to run Jim's latest imSim Data 
    • Learn how to deal with pipeline issue in JMX dealing with large number of jobs stuck in submitted state
    • Run Jim's imSim data through SLAC WF (first attempt in progress)
    • Prep to run using Phosim 1.1p
    • Document using the Slac WF and add scripts to a GH repo in LSSTDESC area
    • attend Friday dc2 meeting
  • SLAC support
    • review, release, and install the latest version of datacat-utilities
    • Read over Joanne's documentation
    • automate weekly install of DMstack binaries for camera 
    • install datacat-utilities on lsst-dc10
  • NERSC support
    • Upload latest SQRE DMstack (w_2018_03) Docker image into Shifter
    • Install w_2018_03 plus sims from source
    • Create w_2018_03+sims Shifter Image (debugging issue with aft - issue identified.. fixing it is something else)
      • create sims Shifter image the long way, without using lsstsqre's base image
    • Install WeakLensingDeblending for Javier.
  • Random LSST Camera and DESC items
    • SSim meeting
    • Help out setting up I&T access to modern stack + eotest at SLAC via ?
  • nEXO
    • reinstall SNiPER in the new disk space
    • try out examples
    • sign up for the collaboration mailing list
  • Fermi
    • Investigate trouble with AcdDigi in L1 branch
    • Find a way to upgrade Anaconda external for ST

Week of Jan 22nd 2018

  • Data Portal
    • Consider rerunning all the ITL sensors in need of updated MET ingest and MET 05 by hand
    • plots - commit latest code and write up intro for Joanne
    • CCB request and deploy MET updates for the DP
    • Deploy DP 0.5.11
  • SLAC support
    • automate weekly install of DMstack binaries for camera (Jim doesn't seem to feel this is a priority)
    • Read over Joanne's documentation
  • NERSC support
    • update DESCQA install to include tensor flow
    • write up instructions for w_2018_01 shifter install
    • automate installation of shifter images
    • Create docker image of new sims release based on w_2018_04
    • Reintall cosmosis for WL group due to updated FFTW module, trying cray-fftw this time
    • Send instructions to Josh to set up his own test galsim using the extlibs we already have set up for Cori KNL gcc
    • install Phosim 3.7.7 and 3.7.8 for knl-intel and haswell-intel
    • meet with Mustafa about HPSS
    • Fix Cosmosis installation
  • Workflows/L2 DC2 Pipeline
    • send plea to Tony
    • Meeting with Tom about using the SLAC pipeline + pilots
    • work out setting up test Pipeline2 job reusing DC1 L2 task and a handful of visits
    • Prep for CI meeting
  • Fermi
    • Work through errors with AcdDigi on L1 branch
    • Set up new Anaconda external
    • Work out with Brian the requirements for setting up externals these days
  • nEXO
    • get Tony an disk space estimate for SLAC installs
    • get some disk space from Tony at SLAC
    • set up SNiPER at SLAC in a temp location (in progress)
    •  attend Tuesday sims meeting

Week of Jan 15th 2018

  • Data Portal
    • revisit plots
    • Holding off MET updates until new traveler to reingest ITL sensors is available
  • datacat-utilities
    • Install new release 0.0.9 at SLAC
  • SLAC support
    • automate weekly installs of DMstack binaries for camera
  • NERSC Support
    • update DESCQA install to include tensorflow
    • Write up instructions for w_2018_01 shifter install
    • automate installation of shifter images
    • Add WeakLensingDeblending 
    • Check on CCL installation
  • Workflows/L2 DC2 Pipeline
  • Fermi
    • Set up draft f2f meeting page Fermi Software Week Winter 2018
    • Continue working on GR L1 branch - completed checkout of L1 code successfully
    • Work through errors with AcdDigi on L1 branch
  • nEXO

Week of Jan 8th 2018

  • DataPortal
    • finish off ITL-3800C-368 ingest after sorting out http authentication errors
    • revisit plots
    • Hold off on MET updates until new traveler to reingest ITL sensors is available.
  • datacat-utilities
    • Install new release
  • SLAC suport
    • Install w_2018_01 binaries lsst_distrib for camera
    • Work out automated weekly installs via cron (u1 is 97% full, ~990 GB left)
    • Install Merlin's eotest branch
  • NERSC Support
  • Workflows/L2 DC2 pipeline
    • Set up lsst_sims docker image for NERSC & uploaded to Dockerhub
    • Try another lsst_sims docker image using lsstsqre image as the base rather than centos7 (initially failed, have new plans to retry)
  • Fermi
  • nEXO & SNiPER

Week of Dec 18th 2017

  • DataPortal
  • Workflows/L2 DC2 pipeline
    • Reproduce error James P sees
    • Repeat experiment with multiple visits
    • Repeat experiment using Head of ImageProcessingPipeline repo
    • Potentially meet with Jim & James this week
    • Catch up on Parsl progress
  • CI
    • Set up new Docker image to use with imSIm's travis builds
    • Create a smaller Docker images for use with travis
  • NERSC Support
    • Shared Stack
    • JupyterHub documentation and simplification
    • clean old project space by removing MBII-snapshots
    • Request reduction in old project space quota to 40 TB
  • imSim
  • Fermi
    • Set up GR L1 branch in GitHub
    • Test build of L1 branch
  • SNiPER
    • Read up on SNiPER
    • Meet with Lisa K on Tuesday 5 PM Eastern
    • send feedback on documentation
    • set up user accounts on Wiki, LLNL, OSG
    • talk to Tony about nExo

 

Week of Dec 11th 2017

  • Data Portal
    • Deploy ETCCB-429
    • MET updates - almost complete
  • CCS ATS
    • Install mvn and associated libraries and get Nick going with Lakeshore driver
  • Workflows/L2 DC2 pipeline
    • Send James P a link to the recordings from last week
    • Repeat test using Jim's parsl wf at NERSC
  • CI
    • Docker image for use in travis builds
  • imSIM
  • NERSC Support
    • Fresh Shared Stack installation for KNL - stuck working out disk space in the common software area  – started
    • dmtcp installation
    • Update jupiter-dev documentation
    • quota checker
  • Fermi
    • Try GR with repoman
    • Documentation

Week of Nov 27th 2017

  • L2 pipeline DC1 rerun
    • Finish off write up of chat with Tony
    • Ask James P about remote participation during Spring Week
      M & W 8 - 12 central time 
    • Try a run using Simon's cookbook
    • Meet with Jim & James P
      • write up notes
    • Try Parsl pipeline at NERSC
  • NERSC software installations
    • Shared Stack installation - verify that the DESCQA and py3 install is working
      • Add CCL
    • Install new sims 2.4.2 using DMstack w.2017.46 Haswell-gcc6 using py3
    • Ask about cosmosis needs
      • Install cosmosis, TreeCorr, and MLZ in projecta area under the WL WG directory
    • Install phosim 3.7.3
    • dmtcp
    • quota checker
    • Consider requesting more common software space
      obtained 50 GB more 
  • NERSC project space
    • Verify MBII-snapshots is in HPSS and remove from project area
      Found the content in the desc hsi area, and viewed the contents of some snap.tar via htar -tf CS/MBII-DMO-snapshots/snapdir_074.tar   Also noticed that the group_xxx.tar files are duplicated in the Archive home directory.  Contacted Mustafa to sort that out and remove duplicates.
  • Data Portal
    • MET updates
      • Meet with Anders and Seth
  • Harvard ATS
    • Check that ccs reboot restarts the MySQL, LocalRestServices, and Trending
    • Check in with Nick about the Lakeshore
  • Fermi
    • Try GR using repoman
    • Documentation
  • Discuss Gaudi for nEXO
    https://gaudi.web.cern.ch/gaudi/releases/v30r0/
    http://lcgsoft.web.cern.ch/lcgsoft/release/
    https://gitlab.cern.ch/gaudi/Gaudi
  • Attend Stanford IT Unconference
    https://itunconference.stanford.edu/sessions/remote

Week of Nov 20th 2017

Week of Nov 13th 2017

  • Harvard ATS
    • Create user accounts and ccs sudo account which grants sudo access to all users
    • Set up directory structure following IR2 model
    • Install DMstack v14
    • Install eotest compatible with v14
    • document the above on the Confluence page ATS Test Stands
    • Set up Trending Database
      • MariaDB installation
  • L2 pipeline DC1 rerun https://github.com/LSSTDESC/SSim_DC1/issues/47
    • Meet with Jim on Monday
      • Write up Notes
    • Contact James P
    • Try to run Simon's Cookbook
    • Chat with Tony
  • NERSC Software Installations
    • Shared Stack
      • first try failed due to conflicts with libiconv installed presumably for 
    • Phosim 3.7.2 bug fix for haswell gcc & intel, KNL gcc & intel 
    • DMTCP intel build
    • Help out with cosmosis installation https://github.com/LSSTDESC/WLPipe/issues/31
  • Data Portal
    • Chat with Tony about plots
    • Plots
      • Bad Channels
      • read noise
  • NERSC Project Space
    • Test MBII-snapshot HPSS copy and remove from project area

Week of Nov 6th 2017

  • QA Testing
    • Investigate cause of differences between Lauren's plots from NCSA and those generated at NERSC
      • look at 'conda list' on both NCSA and NERSC (no differences discovered)
      • generate cal_refs as was done at NCSA
      • try gcc 6 build as is being done at NCSA
    • Determine the cause of differences between ref catalogs at NCSA and NERSC
  • L2 Pipeline
  • Data Portal
    • Finish off first round of Richard's plots
      • bad channels
      • read_noise
  • NERSC Project Space
    • test MBII-snapshot HPSS copy
  • ATS
    • prep for Tony & Owen's visit
  • Installations
    • SLAC
      • Py3 on SLAC RH6-64 for Joanne to test out
    • NERSC
      • finish testing NERSC's cfitsio installation
      • install phosim 3.7.1 for Cori haswell gcc & intel, KNL gcc & intel
        • ask Glenn to install module files
      • install phosim 3.7.2 for Cori haswell gcc & intel, KNL gcc & intel
      • Shared Stack Installation using w.2017.41
        • Query DESCQA and CCL groups
        • py2 w.2017.41 installation
        • py3 w.2017.41 installation

Week of Oct 30th 2017

  • Data Portal
    • Plots
  • DC2 prep
    • QA testing
      • w.2017.41 installation plus branches of obs_lsstSim and pipe_analysis
      • Run visitAnalysis, coaddAnalysis, etc
  • NERSC Workflow work
  • NERSC Project Space
    • Work with Mustafa to get MBII-snapshot backed up
  • NERSC Installations
    • w.2017.? stack installation plus sims - waiting on new sims release using the camera geometry updates
    • Phosim 3.7
      • Testing NERSC cfitsio installation
  • SLAC Installations
    • descsoft w.2017.? stack plus sims installation
  • CCS
    • See if there is any prep work necessary for Tony's visit later in Nov
      • Request Nick set up new box with CentOS7
      • set up user accounts and groups
    • Determine if Mike is still coming out ~Nov 1st
  • Attend SuperCDMS meeting concerning their potential move to NERSC.

Week of Oct 23rd 2017

  • Data Portal
    • Richard's Plots - bad channels (in progress)
    • Look at new Report Labels and determine what the DP still requires from the Reports Table
      LSSTTD-1190 - Getting issue details... STATUS  
    • Make sure we request a restart of Tomcat
  • NERSC Workflow work
    • Try out HTCondor and Pegasus on cori21
    • Take a look at Jim's imsim script
  • NERSC Project Space
    • Set the old project directory to read-only (in progress)
      • claim desc ownership of the whole directory space
    • Clear out all directories where the owners have indicated it is fine to do so
    • Determine the end state of merging old project and projecta areas and handling of www
    • Set up disk quota monitor
    • Coordinate HPSS storage of MBII-snapshots
  • NERSC installations
    • Phosim v3.7
      • Coordinate with NERSC to get cfitsio v3410 installed (in progress)
      • Rebuild using NERSC supplied cfitsio
      • Coordinate with Glenn to set up module files
    • Set up software for DC2 testing
    • v14_0 lsst_distrib + sims 2.4.0
  • SLAC installations
    • Set up same imsim at SLAC as used at NERSC for KNL testing (in progress)
    • v14_0 on RH6-64 public (on hold pending resolution to obs_base testCameraMapper.py unit test failure)
      • camera area
      • descsoft
      • test eotest branch
    • v14_0 on IR-2
  • Document KNL vs Haskell mkl imsim testing at NERSC
    • Initial mkl tests indicate no huge improvement - putting testing on back burner for now
  • CCS - still waiting to hear from Peter, I cannot access their TS3 machine to install and test a new version of the test stand subsystem

Week of Oct 16th 2017

  • Richard's plots in DP
    • Complete Sensor Arrival Time plot
    • Create JSON 
  • NERSC installations
    • Phosim v3.7
      • Try out NERSC supplied FFTW 
      • Request new cfitsio v3410 installation required by Phosim
      • Complete installation of all flavors:  haswell-gcc, haswell-intel, knl-gcc, knl-intel
        • Waiting on cfitsio v3410 before knl-intel will be available
  • NERSC Project Space
    • Notify all users of the need to migrate
    • Handle all help requests - migrating some directories to the new projecta area
    • Install disk quota monitor
    • Check on old project status and prepare to set it to read-only
    • Coordinate backup of nkhandai data to tape with Mustafa
  • Install latest version of teststand subsystem at Harvard there is a machine made available - waiting on response from Peter 
  • Keep an eye out for v14 DMstack

Week of October 9th 2017

  • Continue work on Richard's plots in DP
  • LSSTTD-1165 - Getting issue details... STATUS
    • Released Data Portal 0.5.7
  • Testing imSim built on KNL
    • Continue MKL tests at NERSC once the system recovers from the weekend outage
    • Install the same version of the stack at SLAC to allow similar testing with and without MKL
    • Attempt build of DMstack using the intel compiler at NERSC
  • NERSC Project space
    • Install disk quota monitor
    • Coordinate backup of nkhandai data to tape with Mustafa
    • Prepare to set the old project area as read-only
    • Send mail to all owners of files on the old project space
  • NERSC HPSS SRU overallocation
  • Install latest version of datacat-utilities 0.0.8 at SLAC
  • Install the latest teststand snapshot at Harvard to support use of the monochromator when Peter returns.
    • Contacted Peter - I can't connect to the archon machine to install the test stand snapshot
    • Investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly create a harvard main routine.
  • Keep an eye out for the v14 DMstack release and prepare to install on IR-2 and NERSC
    • This release will include binary installations.. I'm thinking we may still want to install from source, perhaps that's most useful on IR-2, but we'll see.

Week of October 2nd 2017

  • Release DataPortal 0.5.6 which will include
    • Addition of Grade and Procurement labels for the Sensor Acceptance pages
    • RTM assignment
      LSSTTD-1121 - Getting issue details... STATUS
    • NCR "Ready to close"
      LSSTTD-1144 - Getting issue details... STATUS
  • Implement Richard's plots in the DP starting from some of Tony's examples for trending and Richard's python code.
  • Continue testing KNL performance using galsim and imsim
    • Attempt to compile galsim and possibly all of DMstack with the intel compiler
    • Run longer tests using the MKL libraries on Haswell and KNL
    • Rebuild imsim with MKL, using an updated meas_base
  • Install an initial shared stack in the new software area at NERSC for testing as discussed at last week's "Shared Stack" meeting
  • NERSC Project Space
    • Set up an initial quota monitor at NERSC
    • Re-connect with Mustafa to get the mkhandai 42 TB archived to tape.
  • Investigate an upgrade of the monochromator driver at Harvard and confirm that to achieve what Peter plans to do - to run his ATS work in the lab alongside most of the devices from his TS3 setup.. is possible by running both the teststand and rafts subsystem simultaneously.
    • Install latest SNAPSHOT of test stand at Harvard and investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly creating a harvard main routine.
  • Keep an eye out for the v14 release of the DMstack and prepare to install at SLAC/IR-2 and NERSC.
    Though this is assigned to Jim, so perhaps I have no need to worry about SLAC:
    LSSTTD-1157 - Getting issue details... STATUS

 

  • No labels