You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 376 Next »

Week of June 25

  • DM-DC2
  • NERSC
    • Monitor NERSC ticket concerning GPFS AFM
    • Inquire about monitoring SCRATCH space for upcoming purges
    • Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
    • Fix weekly builds of DMstack to use limited number of cores and skip w_2018_24, since it will not build due at NERSC
  • SLAC
    • Check on set up of
  • nEXO
    • Respond to CMT=>CMake project for SNiPER
    • Ping Marco (Gaudi) about their CMT=>CMake experience
  • Fermi
    • Verify that we can checkout L1 tags using --force and update doc

Week of June 18

  • DM-DC2
  • NERSC
    • Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
    • Investigate load use of weekly installs and modify the cron job to limit the number of cores utilized.
    • update to GCRCatalogs 0.7.8 in the shared python environment as requested by Yao
  • SLAC
    • Install DMstack v15 for IR2, py3
    • Install w_2018_23 for summer grad student
  • nEXO
    • Write up slides for Docker intro for collaboration meeting and present
    • Write up slides for SNiPER update and present
    • Listen in to Simulation session 9PM-1:30AM
  • Fermi
    • Weekly software meeting

Week of Jun 11 2018

  • DM-DC2
    • Write up agenda, run, and take notes for weekly DM-DC2 meeting
  • NERSC
    • update fast3tree installation for Yao to allow writing to the install directory
  • SLAC
  • nEXO
    • Complete docker image for collaboration meeting
    • Inform collaborators and help get them set up before the meeting
  • Fermi
    • New L1 Release tag for GR See JIRA FDM-5
    • Build new L1 Release tag at SLAC for Philippe to use

Week of Jun 4 2018

  • DM-DC2
    • CI Meeting update
    • Prepare agenda, run DM-DC2 meeting, and take notes
    • Try constructFlat at NERSC
      • ingest subset of Run1-2p data
    • Doodle Poll to find new meeting time
      • chat with Dominique & Johann first
    • Ingest all Run1-2p data at NERSC
      • Merge latest IN2P3 workflow into NERSC workflow
      • Start up the jobs
  • NERSC
    • investigate installing new python module Yao requested
    • Talk to Joe Z about his miniconda env at NERSC to see about pulling it into the shared python env
    • Reinstall stack weeklies when the next python update for DM is released
    • deal with inode use on /global/projecta/projectdirs/lsst/
      • Issues is associated with the phosim SEDs for every visit, these files must be tarred.
  • SLAC
    •  
  • nEXO
    • Data Task Force Meeting Wednesday
    • Prepare proper docker image with tutorials
  • Fermi
    • merge slac doc in github
    • Offline Software meeting
    • Help Philippe make L1 updates to TkrDigi
    • get a new GR L1 release tag made

Week of May 28 2018

  • DM-DC2
  • NERSC
    • Finish slides for seminar and present
    • Test Javier's notebook
    • Fix py3-4.3.21 installation with numpy and numexpr
    • install phosim v3.7.15 for Haskell-Intel & KNL-intel
    • Install CCL for Yao in the shared python area
  • SLAC
    • Handle vendor ingests while Tom is away
  • nEXO
    • Attend sims meeting and report on docker status
    • Update docker image with tutorials
  • Fermi

Week of May 21 2018

Week of May 14 2018

  • DM-DC2
    • set up agenda and write up notes for Monady's DM-DC2 meeting
  • NERSC
    • Check getting started at NERSC documentation
    • Contact Michael & Javier about the NERSC seminar May 30th
    • Run phosim v3.7.9 and v3.7.14 test data through miniDRP
    • Check on kubernetes availability at NERSC jupyterhub and set up meeting
  • nEXO
    • update docker image to remove sudo and add in tutorials
    • coordinate archiving nexo-sniper
  • SLAC
    • set up order for new laptop
  • Fermi

Week of May 9 2018

  • DM-DC2
    • Setup agenda and write up notes for Monday's DM-DC2 meeting
    • Create darks using phosim 3.7.9
    •  
  • NERSC
    • Install new versions of phosim as they appear
  • nEXO
    • Attend sims meeting and make presentation concerning SNiPER visit
    • Update docker image with tokens
    • contact Raymond about timing correlation
  • SLAC
    • Set up order for new laptop
    • Requested and tested git lfs install at SLAC for nexo
  • Fermi
    • Attend software meeting and write up notes

Week of April 30 2018

  • DM-DC2
    • Set up agenda and write up notes for Monday's DM-DC2 meeting
    • Rerun one processEimage job on one core on KNL using a source build of the stack
    • Send Salman and Katrin directions about the steps to run "mini-DRP" so they can try it at ANL
    • Set up full pipeline at NERSC to run on KNL
    • Finish creating calibration products and document
    • Prepare Monday's CI update
  • NERSC
    • Update python environment to include potables
    • Create "current" symlink to point to the current python installation
    • Test and document setting up jupyter-dev using a shared stack install
    • Install phosim v3.7.10 for both haswell-intel and knl-intel
    • install phosim v3.7.11 for both haswell-intel and knl-intel
  • nEXO
    • Set up Docker image for nexo-offline + SNiPER
    • Provide daily summaries on the Slack meeting channel
    • Test new SNiPER using the last collaboration meeting examples
    • Understand how to implement Time Correlation
    • Attend Tuesday Update meeting with nexo collaborators
  • SLAC
    • Help Tom M set up python on his laptop
    • Update elog Slack app to add a new channel as requested by Aaron

Week of Apr 23 2018

  • DM-DC2
    • Set up agenda and write up notes for Monday's DM-DC2 meeting
    • Rerun one processEimage job on one core on KNL as requested by Salman
    • Send Salman directions about the steps to setup the DMstack so they can try it at ANL
    • Provide NERSC update for DC2 meeting
    • Attend DC2 meeting
  • NERSC
    • Create an area of IN2P3 to transfer their data products
  • nEXO
    • Install new version of SNiPER for centos7
      • request new tbb system library be installed
    • Send some initial SLAC usage instructions to the SNiPER developers
  • SLAC
    • Help Steve ssh tunnel to new IR2 jupyterhub

Week of Apr 16th 2018

  • DM-DC2
    • In support of SkyCorrection Task
      • Generate darks using phosim v3.7.9
      • Play with constructFlat.
    • Rerun miniDRP task using KNL and produce some timing information
    • Update LSSTDESC/obs_lsstSIm for Phosim v3.7.9
      • Tagged Run1.1p obs_lsstSim fork
      • Successfully updated the obs_lsstSIm files using the bias output
      • Work through unit test failures
      • PR
  • NERSC
    • Clear out out Stack builds as advertised on Slack
    • Install parts in shared python env + weeklies + v15
      • Consider setting up a common shared lib area
    • Install a modern dmtcp on knl-intel and haswell-intel
      • multilib install failed, went back to 64 bit only for now - which should be fine for phosim
  • nEXO
    • Prepare for travel to SLAC
    • Help SNiPER Developers get set up with SLAC accounts
    • Install new version of SNiPER at SLAC

 

Week of Apr 9 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO 
    • HistogramSvc
    • Write up SLAC SNiPER installation instructions on wiki
    • Send Lisa some journal references for Gaudi
  • DM-DC2
    • Prepare for Task Force Meeting and write up Live Notes
    • Complete initial run of first Run 1.2p visits through minidrp
    • Create new dc2 Shifter image with room for a NERSC volume 
    • Create dc2 Shifter image using a dc2 tag to try to avoid version mismatch errors
    • update workflow to use Shifter and submit PR
    • Create w_2018_14 + LSSTDESC/obs_lsstSim Shifter image
    • Test pipeline with w_2018_14+LSSTDESC/obs_lsstSim Shifter
    • update NERSC workflow to try out processEimage on rafts
  • NERSC
    • Prepare updates for CI meeting
    • Attend Ci meeting
    • Send Katrin some information about how the set up NERSC for DESC
    • Ask NERSC about anticipated need to recompile software after the scheduled Cori OS update on Apr 25-26
    • Install DMstack v15 in the common software area
    • Request to remove some older stack installations at NERSC
  • SLAC
    • Write up portal section in Overleaf
    • Complete Research integrity course
  • Aux Telescope

Week of Apr 2 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO 
    • HistogramSvc
    • Attend Simulations Meeting
    • Write up SLAC SNiPER installation instructions on wiki
  • DM-DC2
    • Set up new Docker image which includes patched obs_lsstSIm
    • Prep slides and attend SSim meeting
  • NERSC
    • Check on weekly builds and document their new location
    • Create new docker image for new sims 2_7_0 release
    • Install sims 2_7_0 from source in the common software area
  • SLAC
    • Complete CyberSecurity Training
    • Write up portal section in Overleaf
    • get antivirus going on home centos7 to allow VPN connection required to deploy portal
  • Aux Telescope

Week of Mar 26 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO 
    • HistogramSvc
  • DM-DC2
    • Prep for Task Force Meeting
    • Write up Live Notes
    • Contact Dominque B concerning processEimage failure
      • Now fixed with new config file
      • add this config file to github
    • Write update for Friday's DC2 meeting
    • Write update for CI meeting
    • Start up new mini-DRP task in SRS pipeline 
      • Run ingestData for all 66 visits from Run 1.1p
      • Run processEimage (obtain time and memory consumption stats)
  • SLAC
    • Fix LSSTTD-1257
    • Deploy data portal 0.5.14
  • NERSC
    • move weekly builds to common software area by updating cron job
  • Aux Telescope
    • On Hold until we hear from Kirk

Week of Mar 19 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO
    • Fix build of ROOT on centos7, missing libEve and libRGL
    • Finish building nexo SNiPER on centos7
    • Work on HistogramSvc
    • Attend simulations meeting
  • DM-DC2
    • DM-DC2 Meeting prep
    • Attend SSims meeting
    • Write up Live notes
    • Extend DRP Workflow to run processEimage
    • Update DRP Workflow to run processEimage for multiple visits
    • Meet with Nicolas to discuss DRP
  • SLAC
    • Implement and deploy new run search box
    • Update my own centos7 desktop to use SLAC VPN, NetBeans, etc so I can continue supporting the data portal after the VPN transition at SLAC.
    • Run vendor data ingests
  • NERSC
    • Install TreeCorr 
    • Install new GCRCatalogs in all conda environments
    • Move weekly stack installs
      • Clean up old contrib area
      • request another 100 GB of space
      • update cron job
  • Aux Telescope
    • Wait to hear from Kirk

Week of Mar 12 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
  • nEXO
    • Fix build of ROOT on centos7, missing libEve and libRGL
    • Finish building nexo SNiPER on centos7
    • Work on HistogramSvc
  • DM-DC2
    • Update Nicolas' ImageProcessingPipelines branch for NERSC
    • Demonstrate running SRS workflow for ingest 
  • NERSC
    • Work out production installation of lsst_sims + w_2018_09 using the shared python env
  • SLAC
    • Create some documentation to use jupyterlab at SLAC
    • Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
    • LSSTTD-1256 - Getting issue details... STATUS
  • Aux Telescope

Week of Mar 5 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
  • nEXO
    • Attend sims meeting
    • Work with SLAC IT to get all necessary system libraries installed on centos7 to build nEXO's externals
      • libXpm-devel
      • bzip2-devel
      • Motif
      • Motif-devel
      • mesa-libGLU mesa-libGLU-devel
    • Work through build problems with Coin3d 3.1.3 (an external lib)
      https://github.com/nEXO-collaboration/nexo-env/issues/1 
    • Finish building nexo SNiPER on centos7
    • Work on HistogramSvc
    • Create some initial dev doc in their wiki
  • DM-DC2
    • Attend and take notes for task force meeting
    • Attend SSims meeting Thursday
  • DESC
    • Create docker image using new lsst_sims + w_2018_09 
  • NERSC
  • SLAC
    • Continue watching jupyterlab progress 
    • Create some documentation to use jupyterlab at SLAC
    • Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
    • LSSTTD-1256 - Getting issue details... STATUS
  • ATS At Harvard
    • Attend test stand tutorial 9 AM - 10 AM PT Wednesday

Week of Feb 26 2018

  • Fermi
    • get Brian to fully clean up between GR Jenkins builds - created JIRA
    • get L1 GR to build 
    •  figure out how to clean up workspaces in Jenkins on my own.
    • Tag first GlastRelease release in github GlastRelease-20-10-04-gr03  only change is to the xxxRootData packages
    • Figure out the release process
  • nEXO
    • send an email update to Lisa
    • update Slack channel about Raymond's discussions with SNiPER developers
    • Work on HistogramSvc for ROOT files
  • DM-DC2
  • NERSC
    • Poke NERSC about jupyter-dev outage - now fixed
  • SLAC
    • Watch JupyterLab demo install and keep Steve informed
    • Try out eT DB updates to prep for Joanne's departure
    • Update DESC and Fermi documentation on getting a SLAC account
    • Check on Tom M's reported problem with the status of some CCD-011 bits in the DP
  • ATS At Harvard

Week of Feb 19 2018

  • Fermi
    • Finish Anaconda 3 install
    • Fix up L1 builds on Jenkins
    • Fermi Meeting
  • nEXO
    • Start new HistogramSvc for ROOT files
    • Simulation Meeting Thursday
  • DM-DC2
    • First Task Force Meeting
    • Migrate DC2 Cookbook issues to IPP (ImageProcessingPipeline) repo
    • Create new issues in IPP repo
  • NERSC
    • sort out HPSS allocation
  • SLAC
    • Help David L access his account
  • ATS at Harvard
    • Meeting with Tony & Owen on Tuesday
    • Go to Harvard on Friday

Week of Feb 12 2018

  • Fermi
    • attend offline meetings!
    • Finish Anaconda3 installation 
    • Work out distribution of externals in the new regime
    • Install swig 3.0.12 RH6 and RH7 on GLAST_EXT
    • work out L1 build in Jenkins
  • nEXO
    • Meet with sims people
  • NERSC Support
    • re-re-install cosmosis to help out the WL folks who wanted the latest along the develop branch 
    • Reach out to Joe to encourage tagging of cosmosis and provide some feedback on their setup script for NERSC
    • Reach out to wl pipeline group to ask about cosmosis installations.
    • Start up automated weekly stack installations
    • Document weekly stack installations
    • Send Joe Z updates to the setup-cosmosis-nersc script
    • Help Erfan get going with cosmosis at NERSC
  • SLAC support
    • Read over Joanne's docs
    • Pow-wow with Joanne over the DP and eT and the eT DB
    • set up docker for Jim on IR2
    • Update DESC and Fermi documentation on getting a SLAC account
  • DRP/L2 Processing
    • Task Force
      • Phil mentioned that "they" had to write up a charge - figure out what that means
        • Modify charge after receiving suggestions
      • At Nicolas' suggestion reach out to Fabio H concerning weekly stack installations
  •  ATS at Harvard
    • Attend CCS meeting to find out how best to support Harvard
    • Contact Harvard to sort out list of devices & plans for their use for testing
    • Meet with Kirk
    • write up minutes from Kirk/Nick meeting and prepare to iterate with Kirk

Week of Feb 5 2018

Week of Jan 29nd 2018

  • Data Portal
    • support Joanne as she learns about the plotting
  • DRP/L2 Processing
    • Check on James P
    • Get a copy of the DC1 WF going
    • Update WF to run Jim's latest imSim Data 
    • Learn how to deal with pipeline issue in JMX dealing with large number of jobs stuck in submitted state
    • Run Jim's imSim data through SLAC WF (first attempt in progress)
    • Prep to run using Phosim 1.1p
    • Document using the Slac WF and add scripts to a GH repo in LSSTDESC area
    • attend Friday dc2 meeting
  • SLAC support
    • review, release, and install the latest version of datacat-utilities
    • Read over Joanne's documentation
    • -automate weekly install of DMstack binaries for camera- 
    • install datacat-utilities on lsst-dc10
  • NERSC support
    • Upload latest SQRE DMstack (w_2018_03) Docker image into Shifter
    • Install w_2018_03 plus sims from source
    • Create w_2018_03+sims Shifter Image (debugging issue with afw - issue identified.. fixing it is something else)
      • create sims Shifter image the long way, without using lsstsqre's base image
    • Install WeakLensingDeblending for Javier.
  • Random LSST Camera and DESC items
    • SSim meeting
    • Meet with Wil on Friday
    • Write up notes from meeting
    • Look over DESC architects confluence page
  • nEXO
    • reinstall SNiPER in the new disk space
    • try out examples
    • sign up for the collaboration mailing list
  • Fermi
    • Investigate trouble with AcdDigi in L1 branch
    • Find a way to upgrade Anaconda external for ST
    • Install miniconda py3
    • Contact Helen

Week of Jan 22nd 2018

  • Data Portal
    • Consider rerunning all the ITL sensors in need of updated MET ingest and MET 05 by hand
    • plots - commit latest code and write up intro for Joanne
    • CCB request and deploy MET updates for the DP
    • Deploy DP 0.5.11
  • SLAC support
    • automate weekly install of DMstack binaries for camera (Jim doesn't seem to feel this is a priority)
    • Read over Joanne's documentation
  • NERSC support
    • update DESCQA install to include tensor flow
    • write up instructions for w_2018_01 shifter install
    • automate installation of shifter images
    • Create docker image of new sims release based on w_2018_04
    • Reintall cosmosis for WL group due to updated FFTW module, trying cray-fftw this time
    • Send instructions to Josh to set up his own test galsim using the extlibs we already have set up for Cori KNL gcc
    • install Phosim 3.7.7 and 3.7.8 for knl-intel and haswell-intel
    • meet with Mustafa about HPSS
    • Fix Cosmosis installation
  • Workflows/L2 DC2 Pipeline
    • send plea to Tony
    • Meeting with Tom about using the SLAC pipeline + pilots
    • work out setting up test Pipeline2 job reusing DC1 L2 task and a handful of visits
    • Prep for CI meeting
  • Fermi
    • Work through errors with AcdDigi on L1 branch
    • Set up new Anaconda external
    • Work out with Brian the requirements for setting up externals these days
  • nEXO
    • get Tony an disk space estimate for SLAC installs
    • get some disk space from Tony at SLAC
    • set up SNiPER at SLAC in a temp location (in progress)
    •  attend Tuesday sims meeting

Week of Jan 15th 2018

  • Data Portal
    • revisit plots
    • Holding off MET updates until new traveler to reingest ITL sensors is available
  • datacat-utilities
    • Install new release 0.0.9 at SLAC
  • SLAC support
    • automate weekly installs of DMstack binaries for camera
  • NERSC Support
    • update DESCQA install to include tensorflow
    • Write up instructions for w_2018_01 shifter install
    • automate installation of shifter images
    • Add WeakLensingDeblending 
    • Check on CCL installation
  • Workflows/L2 DC2 Pipeline
  • Fermi
    • Set up draft f2f meeting page Fermi Software Week Winter 2018
    • Continue working on GR L1 branch - completed checkout of L1 code successfully
    • Work through errors with AcdDigi on L1 branch
  • nEXO

Week of Jan 8th 2018

  • DataPortal
    • finish off ITL-3800C-368 ingest after sorting out http authentication errors
    • revisit plots
    • Hold off on MET updates until new traveler to reingest ITL sensors is available.
  • datacat-utilities
    • Install new release
  • SLAC suport
    • Install w_2018_01 binaries lsst_distrib for camera
    • Work out automated weekly installs via cron (u1 is 97% full, ~990 GB left)
    • Install Merlin's eotest branch
  • NERSC Support
  • Workflows/L2 DC2 pipeline
    • Set up lsst_sims docker image for NERSC & uploaded to Dockerhub
    • Try another lsst_sims docker image using lsstsqre image as the base rather than centos7 (initially failed, have new plans to retry)
  • Fermi
  • nEXO & SNiPER

Week of Dec 18th 2017

  • DataPortal
  • Workflows/L2 DC2 pipeline
    • Reproduce error James P sees
    • Repeat experiment with multiple visits
    • Repeat experiment using Head of ImageProcessingPipeline repo
    • Potentially meet with Jim & James this week
    • Catch up on Parsl progress
  • CI
    • Set up new Docker image to use with imSIm's travis builds
    • Create a smaller Docker images for use with travis
  • NERSC Support
    • Shared Stack
    • JupyterHub documentation and simplification
    • clean old project space by removing MBII-snapshots
    • Request reduction in old project space quota to 40 TB
  • imSim
  • Fermi
    • Set up GR L1 branch in GitHub
    • Test build of L1 branch
  • SNiPER
    • Read up on SNiPER
    • Meet with Lisa K on Tuesday 5 PM Eastern
    • send feedback on documentation
    • set up user accounts on Wiki, LLNL, OSG
    • talk to Tony about nExo

 

Week of Dec 11th 2017

  • Data Portal
    • Deploy ETCCB-429
    • MET updates - almost complete
  • CCS ATS
    • Install mvn and associated libraries and get Nick going with Lakeshore driver
  • Workflows/L2 DC2 pipeline
    • Send James P a link to the recordings from last week
    • Repeat test using Jim's parsl wf at NERSC
  • CI
    • Docker image for use in travis builds
  • imSIM
  • NERSC Support
    • Fresh Shared Stack installation for KNL - stuck working out disk space in the common software area  – started
    • dmtcp installation
    • Update jupiter-dev documentation
    • quota checker
  • Fermi
    • Try GR with repoman
    • Documentation

Week of Nov 27th 2017

  • L2 pipeline DC1 rerun
    • Finish off write up of chat with Tony
    • Ask James P about remote participation during Spring Week
      M & W 8 - 12 central time 
    • Try a run using Simon's cookbook
    • Meet with Jim & James P
      • write up notes
    • Try Parsl pipeline at NERSC
  • NERSC software installations
    • Shared Stack installation - verify that the DESCQA and py3 install is working
      • Add CCL
    • Install new sims 2.4.2 using DMstack w.2017.46 Haswell-gcc6 using py3
    • Ask about cosmosis needs
      • Install cosmosis, TreeCorr, and MLZ in projecta area under the WL WG directory
    • Install phosim 3.7.3
    • dmtcp
    • quota checker
    • Consider requesting more common software space
      obtained 50 GB more 
  • NERSC project space
    • Verify MBII-snapshots is in HPSS and remove from project area
      Found the content in the desc hsi area, and viewed the contents of some snap.tar via htar -tf CS/MBII-DMO-snapshots/snapdir_074.tar   Also noticed that the group_xxx.tar files are duplicated in the Archive home directory.  Contacted Mustafa to sort that out and remove duplicates.
  • Data Portal
    • MET updates
      • Meet with Anders and Seth
  • Harvard ATS
    • Check that ccs reboot restarts the MySQL, LocalRestServices, and Trending
    • Check in with Nick about the Lakeshore
  • Fermi
    • Try GR using repoman
    • Documentation
  • Discuss Gaudi for nEXO
    https://gaudi.web.cern.ch/gaudi/releases/v30r0/
    http://lcgsoft.web.cern.ch/lcgsoft/release/
    https://gitlab.cern.ch/gaudi/Gaudi
  • Attend Stanford IT Unconference
    https://itunconference.stanford.edu/sessions/remote

Week of Nov 20th 2017

Week of Nov 13th 2017

  • Harvard ATS
    • Create user accounts and ccs sudo account which grants sudo access to all users
    • Set up directory structure following IR2 model
    • Install DMstack v14
    • Install eotest compatible with v14
    • document the above on the Confluence page ATS Test Stands
    • Set up Trending Database
      • MariaDB installation
  • L2 pipeline DC1 rerun https://github.com/LSSTDESC/SSim_DC1/issues/47
    • Meet with Jim on Monday
      • Write up Notes
    • Contact James P
    • Try to run Simon's Cookbook
    • Chat with Tony
  • NERSC Software Installations
    • Shared Stack
      • first try failed due to conflicts with libiconv installed presumably for 
    • Phosim 3.7.2 bug fix for haswell gcc & intel, KNL gcc & intel 
    • DMTCP intel build
    • Help out with cosmosis installation https://github.com/LSSTDESC/WLPipe/issues/31
  • Data Portal
    • Chat with Tony about plots
    • Plots
      • Bad Channels
      • read noise
  • NERSC Project Space
    • Test MBII-snapshot HPSS copy and remove from project area

Week of Nov 6th 2017

  • QA Testing
    • Investigate cause of differences between Lauren's plots from NCSA and those generated at NERSC
      • look at 'conda list' on both NCSA and NERSC (no differences discovered)
      • generate cal_refs as was done at NCSA
      • try gcc 6 build as is being done at NCSA
    • Determine the cause of differences between ref catalogs at NCSA and NERSC
  • L2 Pipeline
  • Data Portal
    • Finish off first round of Richard's plots
      • bad channels
      • read_noise
  • NERSC Project Space
    • test MBII-snapshot HPSS copy
  • ATS
    • prep for Tony & Owen's visit
  • Installations
    • SLAC
      • Py3 on SLAC RH6-64 for Joanne to test out
    • NERSC
      • finish testing NERSC's cfitsio installation
      • install phosim 3.7.1 for Cori haswell gcc & intel, KNL gcc & intel
        • ask Glenn to install module files
      • install phosim 3.7.2 for Cori haswell gcc & intel, KNL gcc & intel
      • Shared Stack Installation using w.2017.41
        • Query DESCQA and CCL groups
        • py2 w.2017.41 installation
        • py3 w.2017.41 installation

Week of Oct 30th 2017

  • Data Portal
    • Plots
  • DC2 prep
    • QA testing
      • w.2017.41 installation plus branches of obs_lsstSim and pipe_analysis
      • Run visitAnalysis, coaddAnalysis, etc
  • NERSC Workflow work
  • NERSC Project Space
    • Work with Mustafa to get MBII-snapshot backed up
  • NERSC Installations
    • w.2017.? stack installation plus sims - waiting on new sims release using the camera geometry updates
    • Phosim 3.7
      • Testing NERSC cfitsio installation
  • SLAC Installations
    • descsoft w.2017.? stack plus sims installation
  • CCS
    • See if there is any prep work necessary for Tony's visit later in Nov
      • Request Nick set up new box with CentOS7
      • set up user accounts and groups
    • Determine if Mike is still coming out ~Nov 1st
  • Attend SuperCDMS meeting concerning their potential move to NERSC.

Week of Oct 23rd 2017

  • Data Portal
    • Richard's Plots - bad channels (in progress)
    • Look at new Report Labels and determine what the DP still requires from the Reports Table
      LSSTTD-1190 - Getting issue details... STATUS  
    • Make sure we request a restart of Tomcat
  • NERSC Workflow work
    • Try out HTCondor and Pegasus on cori21
    • Take a look at Jim's imsim script
  • NERSC Project Space
    • Set the old project directory to read-only (in progress)
      • claim desc ownership of the whole directory space
    • Clear out all directories where the owners have indicated it is fine to do so
    • Determine the end state of merging old project and projecta areas and handling of www
    • Set up disk quota monitor
    • Coordinate HPSS storage of MBII-snapshots
  • NERSC installations
    • Phosim v3.7
      • Coordinate with NERSC to get cfitsio v3410 installed (in progress)
      • Rebuild using NERSC supplied cfitsio
      • Coordinate with Glenn to set up module files
    • Set up software for DC2 testing
    • v14_0 lsst_distrib + sims 2.4.0
  • SLAC installations
    • Set up same imsim at SLAC as used at NERSC for KNL testing (in progress)
    • v14_0 on RH6-64 public (on hold pending resolution to obs_base testCameraMapper.py unit test failure)
      • camera area
      • descsoft
      • test eotest branch
    • v14_0 on IR-2
  • Document KNL vs Haskell mkl imsim testing at NERSC
    • Initial mkl tests indicate no huge improvement - putting testing on back burner for now
  • CCS - still waiting to hear from Peter, I cannot access their TS3 machine to install and test a new version of the test stand subsystem

Week of Oct 16th 2017

  • Richard's plots in DP
    • Complete Sensor Arrival Time plot
    • Create JSON 
  • NERSC installations
    • Phosim v3.7
      • Try out NERSC supplied FFTW 
      • Request new cfitsio v3410 installation required by Phosim
      • Complete installation of all flavors:  haswell-gcc, haswell-intel, knl-gcc, knl-intel
        • Waiting on cfitsio v3410 before knl-intel will be available
  • NERSC Project Space
    • Notify all users of the need to migrate
    • Handle all help requests - migrating some directories to the new projecta area
    • Install disk quota monitor
    • Check on old project status and prepare to set it to read-only
    • Coordinate backup of nkhandai data to tape with Mustafa
  • Install latest version of teststand subsystem at Harvard there is a machine made available - waiting on response from Peter 
  • Keep an eye out for v14 DMstack

Week of October 9th 2017

  • Continue work on Richard's plots in DP
  • LSSTTD-1165 - Getting issue details... STATUS
    • Released Data Portal 0.5.7
  • Testing imSim built on KNL
    • Continue MKL tests at NERSC once the system recovers from the weekend outage
    • Install the same version of the stack at SLAC to allow similar testing with and without MKL
    • Attempt build of DMstack using the intel compiler at NERSC
  • NERSC Project space
    • Install disk quota monitor
    • Coordinate backup of nkhandai data to tape with Mustafa
    • Prepare to set the old project area as read-only
    • Send mail to all owners of files on the old project space
  • NERSC HPSS SRU overallocation
  • Install latest version of datacat-utilities 0.0.8 at SLAC
  • Install the latest teststand snapshot at Harvard to support use of the monochromator when Peter returns.
    • Contacted Peter - I can't connect to the archon machine to install the test stand snapshot
    • Investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly create a harvard main routine.
  • Keep an eye out for the v14 DMstack release and prepare to install on IR-2 and NERSC
    • This release will include binary installations.. I'm thinking we may still want to install from source, perhaps that's most useful on IR-2, but we'll see.

Week of October 2nd 2017

  • Release DataPortal 0.5.6 which will include
    • Addition of Grade and Procurement labels for the Sensor Acceptance pages
    • RTM assignment
      LSSTTD-1121 - Getting issue details... STATUS
    • NCR "Ready to close"
      LSSTTD-1144 - Getting issue details... STATUS
  • Implement Richard's plots in the DP starting from some of Tony's examples for trending and Richard's python code.
  • Continue testing KNL performance using galsim and imsim
    • Attempt to compile galsim and possibly all of DMstack with the intel compiler
    • Run longer tests using the MKL libraries on Haswell and KNL
    • Rebuild imsim with MKL, using an updated meas_base
  • Install an initial shared stack in the new software area at NERSC for testing as discussed at last week's "Shared Stack" meeting
  • NERSC Project Space
    • Set up an initial quota monitor at NERSC
    • Re-connect with Mustafa to get the mkhandai 42 TB archived to tape.
  • Investigate an upgrade of the monochromator driver at Harvard and confirm that to achieve what Peter plans to do - to run his ATS work in the lab alongside most of the devices from his TS3 setup.. is possible by running both the teststand and rafts subsystem simultaneously.
    • Install latest SNAPSHOT of test stand at Harvard and investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly creating a harvard main routine.
  • Keep an eye out for the v14 release of the DMstack and prepare to install at SLAC/IR-2 and NERSC.
    Though this is assigned to Jim, so perhaps I have no need to worry about SLAC:
    LSSTTD-1157 - Getting issue details... STATUS

 

  • No labels