You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 704 Next »

Week of 16 Sept 2019

  • DESC DM
    • instance catalogs for Run2.1i do not need to be regeneratd?  how does that affect what is stored to HPSS?
    • take a look at throughputs tags in docker image
    • Generate new imsim Docker image with new Galsim
    • Set up agenda and run DESC DM portion of Wednesday meeting
    • set up docker image for use in SRS pipeline
    • validate DDF data from Run2.1i y5

  • CO
    • Make Confluence pages consistent and update content
    • Update DESC Calendar
    • Data Transfer Discussion
  • NERSC
    • set up instance catalog generation software
    • NaMaster in desc-python
    • update for GCR-catalogs
    • Help Bruno & Bob access Run2.1i data

Week of 3 Sept 2019

  • DM-DC2
    • finish processing 4 tracts of Run2.1.1i through multiband
    • run merge_tract_cat to produce HDF files on all the 14 tracts of Run2.1.1i
    • rerun processing with skyCorrection
    • finish documenting running at NERSC
    • Converse with Ben as questions arise as he works thorugh implementing DRP in parsl
  • DESC
    • Check on imsim Docker image and the throughputs tag 
    • Install NaMaster in docker image
  • NERSC
    • Check on CSCRATCH before outage and transfer to tape
      • Back up Run2.1.1i 4 tract nersc processing
      • recopy RUn1.2i and reapply symlink fix 
      • Create back up of Run2.1.1i, Run2.1i on projecta for users to use during outage
    • Inquire about Jupyter inaccessiability
  • SLAC
    • install DMstack v18.1.0 for IR2 LSSTTD-1433 - Getting issue details... STATUS
  • Fermi
    • Announce, run, and note take Fermi mtg Sept 4th
    • Tag L1 fitsGen after rebasing with master and create a new L1 GR tag
  • nEXO
    • write up tutorial for python access
    • retrieve all requirements for OSG (onhold)
    • meet with Lisa
    • merge Emma's WT branch into issue_31 after resolving conflicts
    • Present "How to create a Data Object and write it out in SNiPER"
    • attend nEXO software/sim meeting

PCW

  • Catch up with Scott and Chris about the throughputs tags

Week of Aug 5 2019

  • DM-DC2
  • DESC
    • Retrieve Y2 DR1B instance catalogs from HPSS 
    • Attend DC2 validation/next steps meeting
    • Document
  • NERSC
    • Poke NERSC about file system status
    • Demonstrate running SRS workflow at NERSC
    • Continue clearing projecta space
  • SLAC
  • Fermi
    • announce, attend and note take for Fermi Offline mtg
  • nexo
    • attend sims/software meetign
    • present update on nexo-offline status
    • Help update doc on Docker
    • contact Tony about setting up Jenkins builds
    • Work through Save All Events problem in nexo-offline
    • finish testing cmake in docker
    • Look at pySniper algorithm

Week of July 29 2019

  • DM-DC2
    • announce, write agenda and note take for weekly meeting
  • DESC
    • attend DC2 meeting
  • NERSC
    • Recover from Cori upgrade
      • Update desc-python to use gcc 7.3.0
      • Adjust setup_current_stack and setup_current_sims to use new gcc 7.3.0 build of w_2019_19 on CVMFS
        • test new gcc 7.3.0 build
  • SLAC
  • Fermi
  • nEXO
    • attend sims/software meeting
    • Run nexo-offline and nEXO_MC using TPCVesselU.mac
    • attend LLBL cmke training remotely

Week of July 22 2019

week of DESC Meeting

  • DM-DC2
    • Help set up Tom to parsl sfd
    • New confluence page for CCB DESC requests
  • Python
    • NaMaster
    • sacc
    • DIA kernel
  • NERSC
    • Install gcr-catalogs 0.13.1 and 0.14.0 into desc-python and desc-nersc for Hack Day
  •  
    • Check on ability to use later gcc version with CVMFS installation
    • Doc use of jupyter compute nodes
    • Fix up DESCPYTHONPATH
    • doc use of batch for desc-stack
    • instance catalog from tape Y6

Week of July 1 - 12 2019

  • DM-DC2
    • Write agenda, run and note take DM-DC2 meeting for both July 1 & 8th
  • DESC
  • NERSC
    • update lsstdesc/nersc to remove PYTHONSTARTUP for desc-stack
    • Document the use of DESCPYTHONPATH
  • SLAC
  • Fermi
  • nEXO

Week of June 24 2019

  • DM-DC2
    • Listen to recording of weekly meeting and transcribe notes
    • Plan any DM-DC2 sessions for collaboration meeting
    • Document new DR naming convention
  • DESC
    • Deal with docker removal from lsst-dc10, find another place to build
  • NERSC
    • Continue clearing out the instance catalogs to HPSS
    • Quality check Y5 simulated data and update the tracking DB
  • SLAC
  • Fermi
    • announce, attend, and note take Fermi Offline Software meeting
  • nEXO
    • attend collaboration meeting at Stony Brook
    • add Cards to nexo-offline master and u/heather999/issue_31
    • Testing running docker image with nEXO_MC
    • Run nexo-offline and nexo_MC TPC jobs for regression testing
    • Take a look at historgrams generated
    • Keep track of Soud's progress with Clustering and Reconstruction port

Week of June 17 2019

  • DM-DC2
  • DESC
    • Prepare new w_2019_23 stack sims docker image for imsim group
  • NERSC
    • Continue moving instance catalogs to HPSS
    • Alert IN2P3 when there is enough space for y1-y2 calexp and coadds
  • SLAC
    • attend stand-up meeting
  • Fermi
    • announce, take notes, attend offline software meeting
  • nEXO
    • Complete new nexo-base docker image include NEST2.0
    • write up slides for collaboration meeting

Week of June 10 2019

  • DM-DC2
    • Write up agenda and run DM-DC2 meeting
    • quality check y3-wfd sim data and update tracking db
    • Help coordinate transfer for new calexp-v1 to NERSC from IN2P3
    • Create branch and PR for forcedPhotCcd_metadata template https://github.com/lsst/obs_lsst/pull/116
  • DESC
  • NERSC
    • Consider increasing desc CSCRATCH quota
    • Copy and remove DC1 deepCoadd to tape from projecta
    • Check over y1-y3 instance catalogs on HPSS
      • remove y1-y3 from projecta
    • move y7-10 to HPSS
    • Help demonstrate cvmfs problems at NERSC
  • SLAC
    • attend group meeting
  • Fermi
    • announce, note take and attend software meeting
  • nEXO
    • attend software meeting
    • report on nexo-offline status at software meeting
    • Update nexo-env to handle NEST2.0
    • Create new docker base image with NEST2.0
    • send Samuele/Lisa slide for collab meeting
    • check with Tony about status of Jenkins builds

Week of May 28 2019

  • DM-DC2
  • DESC
    • Create stack-sims:w_2019_19-sims_w_2019_19 image for use at NERSC
  • NERSC
  •  
    • create new docker image based on w_2019_19 plus all desc python packages
      • include stile
      • install at NERSC as desc-stack-dev
      • Update desc-stack-prod
       
    • Update BF Kernel cp_pipe installation for w_2019_19 after Tom's reports that the older installation using w_2019_17 no longer worked
    • continue transferring data into NERSC
      • https://github.com/LSSTDESC/DC2-production/issues/343
      • DC2 Run2.1i instance catalogs
      • Remove DC1/DC1-imsim/dithered and undithered
      • Remove DC1/astrometry
      • Start cleaning DC1/DM/DC1-imsim-dithered
        • icExp contains just temp files
        • compare random calexp and deepCoadd fits files using fdiff with the copy in HPSS
  • SLAC
    • attend group stand up meeting
  • Fermi
  • nEXO
    • attend software meeting
    • Get some more info about Simulation/DetSim/Nest
    • Add

Week of May 20 2019

  • DM-DC2
    • Write agenda, take notes, and run DM-DC2 meeting
    • Welcome Dino and send along some getting started links
  • DESC
    • Open up a new ticket for Darren White
    • Attend DC2 meeting and lead discussion concerning what data needs to be available for the analysis groups
  • NERSC
    • Continue transferring data into HPSS
    • Open GitHub issue to discuss what can be removed from projecta
  • SLAC
  • Fermi
    • Announce and note take for offline meeting
  • nEXO
    • attend sims meeting
    • setup Soud with u/heather999/issue_31 branch at SLAC
    • contact Tony about getting some Jenkins builds going before collab meeting
    • Contact Pere and Gabriele C about NEST2 linking errors against G4
    • Poke Jason and Samuele about nEXO_MC PR for NEST

Week of May 13 2019

Week of May 6 2019

  • DM-DC2
    • Write agenda, take notes, and run DM-DC2 meeting
    • Update visit DB to mark files and visits tables as quality_controlled and quality_completed, respectively
    • Create updated CALIB products tarball including BF kernels
    • Update Antonio's visit DB to note that all Y2-WFD visits and files have been quality controlled.
  • DESC
    • attend CI meeting
    • attend DC2 meeting
      • review Dominique's slides for DC2 mtg
    • attend 3x2 mtg
      • help Francois access Run1.2i force photometry data - only to learn that newer DMstacks cannot deal with the older repos.
    • Deal with user questions on #desc-nersc
    • Start uploading DC2 data into HPSS to free projecta space
  • NERSC
    • install stile
    • Install new python environment compatible with recent DMstacks
    • Create new docker image for sims w_2019_17 which will be the base of new jupyter kernels
  • Fermi
    • announce, take notes and attend Fermi offline s/w mtg
  • nEXO
    • attend sims meeting
    • Clean up merge of nEXO_MC and nexo-offline
      • compile on SLAC with gcc 7.3 and LCG 94a
      • compile with gcc 4.8.5
      • run some tests
      • submit PR

Week of Apr 29 2019

  • DM-DC2
    • Write agenda, take notes and run DM-DC2 mtg
    • Check y2 Run2.1i data for incomplete files
    • Ferry sky correction configs into master of lsst/obs_lsst
      • rebase, update comments, squash commits
    • Update dc2/run2.1 lsst/obs_lsst  branch for use at IN2P3
    • Rerun skyCorrection for eyeballer visits
      • Due to decision to skip skyCorrection, this attempt was abandoned
    • Generate new set of BF kernels
      • first attempt failed
  • DESC
  • NERSC
    • Update setup scripts at NERSC to use CVMFS
    • Update confluence documentation for NERSC software installations
    • Install updated cp_pipe for BF kernel generation
    • Investigate inode use on /global/project/projectdirs/m1727
      • Poke owners of files
  • SLAC/Camera
  • Fermi
    • Inquire about software meeting
  • nEXO
    • Test CMake build of SNiPER
    • Test CMake build of nexo-offline
    • Test CMT build of nexo-offline against CMake build of SNiPER
    • Set up LCG 94a at SLAC

 

Week of Apr 15 2019

  • DM-DC2
    • Write agenda, take notes and run DM-DC2 mtg
    • Deliver CALIB to IN2P3
    • Check 2y Run2.1i data for incomplete files 
    • test updated sky configs and running makeFpSummary
  • DESC
  • NERSC
    • Clear out Javier's copy of DC1 undithered data
    • Start transfer of Run1.1 and Run1.2 data into HPSS
    • Update gcr-catalog in jupyter kernels and underlying shifter image
  • SLAC/Camera
  • Fermi
    • announce, attend, and note take for Fermi software meeting
    • Help Bill get set up with GR at SLAC for debugging
  • nEXO
    • attend software meeting
    • Review Git and Code review docs

Week of Apr 8 2019

  • DM-DC2
    • Write agenda, take notes and run DM-DC2 mtg
    • Regnerate BF flats with w_2019_14
    • update config and ingest new Reference catalog
    • Rerun constructDark with doCrosstalkBeforeAssemble=False
    • Rerun constructFlat with doCrosstalkBeforeAsseble=False
    • Set up updated CALIB products using w_2019_14 and setting isr.doCrosstalkBeforeAssemble=False
    • Update obs_lsst with isr.doCrosstalkBeforeAssemble=False https://jira.lsstcorp.org/browse/DM-19218
    • Try out sky construction using Jim's sky frames
    • Try sky construction again with flats available
    • Get sky configs updates into obs_lsst
    • Go through outstanding LSSTDESC/ImageProcessingPipelines issues and close where possible
    • Work with Johann to clean up the IPP/workflows/srs/dm_configs area and ultimately deleted it
    • Reviewed the reworked SFD pipeline SRS workflow with Johann
  • DESC
    • Attend and present at CI meeting
    • Attend Rachel's Computing WG meeting
  • NERSC
  • SLAC/Camera
  • Fermi
    • announce, attend, and note take for Fermi software meeting
  • nEXO
    • attend software meeting and present update
    • Try out LCG view installation at SLAC and update nEXO wiki doc
    • Complete LLNL security training
    • Contact LLNL to get my account activated

Week of Apr 1 2019

  • DM-DC2
    • Write agenda, take notes and run DM-DC2 mtg
    • Script to find all potential incomplete Run2.1i visits in the first production run
    • Write up todo list of DRP processing of Run2.1i-y2wfd
    • Schedule mtg with Katrin, Jim, Rachel, Dominique, Fabio, Johann concerning Run2.1i processing
    • meeting with Katrin,Jim, Rachel, Dominique, Fabio about Run2.1i processing
  • DESC
    • Q2 report
    • Attend Data access meeting
  • NERSC
    • Arrange Run1.1p, 1.2p, 1.2i, 2.0i data and tabulate disk use
    • Prepare HPSS disk request
    • Put in HPSS project area request
    • Set up new conda area based on updated DMstack dependencies
  • SLAC/Camera
  • Fermi
    • announce, attend, and note take for Fermi Software Meeting
  • nEXO  

Week of Mar 25 2019

  • DM-DC2
    • write agenda, take notes, and run DM-DC2 mtg
    • Generate flats calibration products for Run2.1i
      • regenerate R22 S11 using updated configs
      • test run1.2i R22 S11 flat raw using same stack w_2019_09
      • test run2.1i 80k flats 
    • generate subset of bf flats 
    • Ask Johann for metacal summary
    • Create RFC or Confluence page to request DM support for DC2: https://lsstc.slack.com/archives/C978LTJGN/p1553708211293400
  • DESC
    • send out announcement concerning NERSC jupyter
  • NERSC
    • Set up Yadu with a copy of ci_hsc at NERSC: https://github.com/lsst/ci_hsc
    • check on mcnanna usage
    • Continue arranging the /global/projecta/projectdirs/lsst/production area for Run1.2p and Run1.2i
    • Pull out DC1 undithered calexp from HPSS for Javier 
      • deal with desc account being over quota in CSCRATCH
  • SLAC/Camera
    • inquire about BNL computing status
    • SISA Information Security I
  • Fermi
    • announce, attend, and note take for Fermi Software Meeting
  • nEXO

 

Week of Mar 18 2019

 

Week of Mar 11 2019

  • DM-DC2

    • Write agenda, take notes, and attend DM-DC2 mtg
    • Generate BF flats
      • tarball and store on projecta
    • set up CSCRATCH area with CALIB, ref_cats and bf flats
  • DESC
    • Attend and present at CI meeting
    • Attend DC2 meeting
    • update jupyter doc for jupyter-dev changes
      • test after maintenance is completed on 3/15
    • Build new w.2019.10 sims docker image
  • NERSC
  • SLAC/Camera
    • shut down eLog slackbot
    • Install DMstack v17 + a few additional python packages and setup constraint file
  • Fermi
    • Find SLAC anaconda 3 installation
      • send note to SLAC unix-admin about public installation
  • nEXO
    • attend and apparently run software meeting

Week of Mar 4 2019

Week of Feb 18, 2019

  • DM-DC2
    • Create workflow for imsim biases and darks
    • run test through patch of Run1.2i
      • ingest
      • sfd
      • coadd
      • multiband
  • DESC
    • CI quarterly report
    • set up Parsl discussion with Jim/Katrin/Tom/Yadu/Ben/Antonio
  • NERSC
    • check into mweiser problems accessing lsst resources
    • Inquire about globus and collab use
    • Inquire about HPSS project directory
    • add new ccl and postgressql to docker and jupyter kernels
    • Finish copying DC2 data to projecta
    • request increase in inode quota of descdm $CSCRATCH
    • Clean up $CSCRATCH descdm
    • update desc-python and desc-stack with new gcr-catalogs release in time for collaboration mtg
    • Inform NERSC of Hack Day and planned Tuesday tutorials using jupyter-dev
  • SLAC/Camera
    • Data portal updates for NCRs
  • Fermi
  • nEXO
    • Test cmake
    • attend sims meeting
    • release nexo-ofline and install at SLAC and create docker image
    • meet with Lisa about workshop agenda
    • Demo Doxygen for Lisa
    • meet with Lisa to go over git branches on nexo_MC
    • PR updates to Guofu's optical photon changes

Week of Feb 11, 2019

  • DM-DC2
    • Prepare agenda, attend and note take for the DM-DC2 mtg
    • Set Tom up with Run1.2i calibration products
    • Create workflow for imsim biases and darks
    • Contact Scott about adding obs_lsst to lsst_sims
    • Coordinate merge of https://jira.lsstcorp.org/browse/DM-17875
    • Meeting with Jim & Katrin
      • Discuss disk space needs for Run2.1i (plan to add 1 Pb per year for now)
        • How much data will come back from IN2P3?
      • Where will Run2.1i processing occur?
  • DESC
    • CI quarterly report
  • NERSC
    • Install new CCL release
    • add new ccl and postgressql lib to docker & jupyter kernels
    • Finish copying the DC2 data into projecta and prepare count of all contents
    • Plan HPSS use
  • SLAC/Camera
    • Data Portal Update for NCRs
    • install pyds9 on lsst-dc10 and maybe also afs
      • sent inquiry to Jim and Richard
  • Fermi
    • Attend and take notes for offline meeting
  • nEXO
    • attend software meeting
    • create list of open branches in nexo_MC
    • Ask SNiPER developers about the ability to submit jobs via the python config scripts
    • Prepare discussion on documentation for next software meeting:  Doxygen & ROOT's THtml
    • Respond to Samuele's Gaudi questions
    • test cmake build of sniper against nexo-offline
    • install py3 at SLAC 
    • RElease new nexo-offline and install at SLAC & create docker/singularity containers
      • OSG?
    • Document a tutorial for nexo-offline for workshop
    • Lessons learned?

 

Week of Feb 4 2019

  • DM-DC2
    • Prepare agenda, attend and note take for DM-DC2 mtg
    • Help Tom set up a suitable stack for BF kernel generation testing
    • Test BF kernel generation by running the data produced through a modified version of Michael's notebook
    • Set Tom up with Ru1.2i calibration data
    • Test obs_lsst master using Run2.1 data
    • Create workflow for imsim biases and darks generation
  • DESC
    • Continue moving DC2 data to production area at NERSC
    • Prepare a space request for HPSS based on DC2 needs 
    • Attend Data-Access meeting
    • CI quarterly report
    • Finish looking over Joanne's postgres notebooks
    • Send in Travel request
  • NERSC
    • Ask NERSC about setting up a shared HPSS area accessible via globus
    • Install latest pyCCL on desc-python and in shifter images
    • Document and promote use of CVMFS at NERSC
  • SLAC/Camera
    • contact Arash about setting up a DB for elog storage
  • Fermi
    • announce, set up agenda, attend and take notes for s/w mtg
  • nEXO
    • attend sims meeting
    • Meet with Lisa to plan Software Workshop

Week of Jan 28 2019

  • DM-DC2
  • DESC
    • Prepare NERSC space discussion and attend CI mtg
    • Attend DC2 mtg
    • Prepare DM-DC2 XWG Confluence page
  • NERSC
    • Prepare space request for HPSS based on DC2 needs
    • Request IN2P3 transfer Run1.2p raw repackaged data back to NERSC
    • set up psycog2 at NERSC
  • SLAC/Camera
    • eLog - check new FNAL account and contact Margherita about transferring data to external DB
    • contact Arash about a DB?
    • alert camera community about impending elog shut down
  • Fermi
    • Announce, attend, and take notes for offline mtg
  • nEXO
    • Help SLAC users set up nEXO and EXO env
    • check out Gaudi

 

Week of Jan 21 2019

  • DM-DC2
    • Reprocess requests 28 visits through minidrp
    • Prepare for new Run2.1i branch of obs_lsst
  • DESC
    • attend ssims meeting
    • Review Joanne's postgres notebook
  • NERSC
    • Write up proposal for DC2 disk space use
    • Try Exposure Checker
    • Update jupyter kernels set up to add current working directory to PYTHONPATH
    • update desc-dc2-dm-data for Run1.2p_v4
    • Copy Run1.2p_v4 to CSCRATCH 
    • Build and release new docker image that supports Run1.2p_v4
  • SLAC
  • Fermi
    • Attend and take notes for Offline mtg
  • nEXO
    • Attend Sims meeting
    • Write up Communications tools page
    • Help SLAC users set up nEXO and EXO env

 

Week of Jan 14 2019

  • DM-DC2
    • Prepare agenda, run and take minutes for DM-DC2 mtg
    • Merge minidrp Run2.1i into IPP repo
    • Chat with Tom about pipeline help
  • DESC
    • Prepare NERSC disk space use slides and attend CI mtg
    • Write up NERSC DC2 disk space use
  • NERSC
    • Set up collaboration accounts
  • SLAC
  • Fermi
    • S/W mtg
  • nEXO
    • Look into Gaudi
    • Install Anaconda Python
    • Provide installation instructions
    • Getting started at SLAC doc
    • Install nEXO_MC and nexo_offline at SLAC
    • Fast Charge

 

Week of Dec 17

  • DM-DC2
    • Prepare agenda, run and take minutes for DM-DC2 mtg
    • Test Run2,0i minidrp workflow
    • Transfer IN2P3 Run1.2i configs to lsst-dev
    • Get copies of CALIB and ref_cats
    • Fix up handling of config.isr.doCrosstalk in obs_lsst
  • DESC
    • Prepare Run1.2i/p update and attend CI meeting
  • NERSC
  • SLAC
  • Fermi
    • Attend software meeting
  • nEXO
    • Attend sims meeting
    • Attend analysis discussion
    • Contact Sara about docker
    • Take another look at migrating FastCharge to nexo-offline
    • Contact Vladimir about CMake progress
    • Test singularity image at OSG

 

Week of Dec 10

  • DM-DC2
    • Prepare agenda, run and note take for DM-DC2 mtg
    • Run1.2i processing
    • Prepare Run2.1i minidrp workflow
  • DESC
    • nersc user mailing list?
    • docker image for sims+mkl
    • New docker image sims 2_13_1
  • NERSC
  • SLAC
  • Fermi
    • write agenda and notetake for offline meeting
  • nEXO
    • complete travel request
    • help Sara with Docker
    • Poke G4 guy about speed of light G4Exceptions

 

Week of Dec 3

  • DM-DC2
    • Prepare agenda, run and note take for DM-DC2 mtg
    • Test new biases and darks
    • Tarball and send Johann new biases and darks, request CALIB directory be transferred to NERSC
    • Run1.2i processing on KNL
    • rebase and merge PR#41 obs_lsst
  • DESC
    • nersc user mailing list?
    • new sims image 2_13_0
      • tag obs_lsst for Run2.1i
    • docker image for sims+mkl
    • attend and present 1.2i/2.0i processing update CI mtg
  • NERSC
    • install new jupyter-kernels
    • update desc-python with GCRCatalogs 0.9.3
    • update desc-stack with GCRCatalogs 0.9.3
    • update desc-python and desc-stack with GCR 0.8.7
  • SLAC
  • Fermi
    • Help with recon root reading
  • nEXO
    • sims meeting
    • meet with Lisa
    • setup nexo-env so it can work with base nexo external image
    • Help Sara with Docker
    • Test for G4Exception using Guofu's mac file on v0r0p1
    • Merge zepeng's chargesim updates
      • new release tag and docker image

 

Week of Nov 26

  • DM-DC2
    • prepare agenda, run, and take notes for DM-DC2 mtg
    • Test newly generated biases and darks 
    • look at obs_lsst PRs
    • Run1.2i processing
  • DESC
    • nersc user mailing list?
    • docker image for sims+mkl
  • NERSC
    • update desc-python to include emcee
  • SLAC
  • Fermi
    • s/w meeting
    • Help with Recon root files
  • nEXO
    • Meeting with Lisa
    • Merge Guofu's updates and tag new release
    • Create new v0r1p0 Docker images
    • create external only base for docker images

 

Week of Nov 19

  • DM-DC2
    • prepare agenda and run, take notes for DM-DC2 mtg
    • Regenerate darks and biases for Run1.2p using phosim v3.7.9
    • Test generating bias file and sky flat using haswell build vs knl build at NERSC
    • continue Run1.2i processing
    • make sure all outstanding obs_lsstCam PRs are merged 
      • figure out default setting for xtalk for Ru1.2p versus Run1.2i
  • DESC
    • ask Charlotte about setting up a nersc user mailing list
    • Update stack-sims images to keep test files and retain eigen includes
  • NERSC
  • SLAC
  • Fermi
    • weekly s/w mtg
  • nEXO
    • singularity at SLAC
    • update wiki doc
    • generate test data for Guofu's update
    • attend sims mtg

 

Week of Nov 12

  • DM-DC2
    • continue processing Run1.2i
    • Maintain copy of IN2P3 on NERSC $CSCRATCH
  • DESC
    • ask Charlotte about setting up a nersc user mailing list
    • Field ANL questions about docker and stack+sims building
  • NERSC
    • create new jupyter-kernels repo
    • Update desc-python at NERSC
    • Get my.nersc.gov login fixed after enabling MFA
  • SLAC
  • Fermi
    • weekly s/w meeting
  • nEXO
    • meet with Lisa
    • singluarity at SLAC
    • ask Raymond about OSG
      • still not fixed
    • Test Guofu's updates

 

Week of Nov 5

  • DM-DC2
    • Prepare and run DM-DC2 mtg
    • Repackage ALL of Run1.2i data
  • DESC
    • Attend CI meeting
    • Document DESCPYTHONPATH for jupyter
  • NERSC
    • Transfer NCSA Run1.2i data to NERSC?
    • Update desc-python at NERSC
    • Add GCR==0.8.6
  • SLAC
    • Attend weekly stand up meeting
  • nEXO
    • sims meeting
    • Check on Guofu's testing of his updates in nexo-offline
    • singularity at SLAC
    • doc use of SLAC in nexo WIki

 

Week of Oct 8

  • DM-DC2
    • Prepare and run DM-DC2 Meeting
    • Chat with Leanne and attend meeting with Leanne, Johann, Bob, Dominique
    • Process Run1.2i using w_2018_39 + obs_lsstCam dc2/run1.2-v2
  • DESC
    • Create w_2018_30-run1.2 docker image + DESC python
  • NERSC
    • take a look at dask-jobqueue
    • Help write up problems/request for upcoming ERCAP application
  • SLAC
  • nEXO
    • attend Sims meeting
    • Release new Docker and singularity image for testing
    • Port optical updates in nEXO_MC to nexo-offline
      • Ask Guofu to test
      • Make a new Docker image
  • Fermi
    • run and attend s/w meeting

 

Week of Oct 1

  • DM-DC2
    • Prepare and run DM-DC2 meeting
    • test obs_lsstCam master + w_2018_39 ingest
    • Run calexps and checkCcdAstrometry on two visits shared between Run1.2i and Run1.2p
  • DESC
    • attend data access meeting on Monday
  • NERSC
    • Check on availability of /usr/bin/time on Cori compute nodes
  • SLAC
    • Help Johann get started with jupyter on IR2
    • Install Anaconda 5.2.0 for python 3.6
  • nEXO
    • meet with Lisa on Wednesday
    • Create updated docker image of nexo-offline
      • Create Singularity image
  • Fermi
    • s/w meeting
    • Respond to W's questions about fitsGen & astro updates

 

Week of Sept 24

  • DM-DC2
    • Prepare and run DM-DC2 Meeting
    • Update SRS workflow
    • Regenerate calibration data for Run1.2p
      • repackage
      • ingest
    • test creation of multiple ingest repo
      • merge separate repos
    • Revisit darks/bias/flats generation from a couple of months ago
    • Stay on top of obs_lsstCam updates
      • merge xtalk PR
      • Contact Robert about merging outstanding PRs
      • Contact Robert about obs_lsstCam=>obs_lsst migration
      • Update run1.2 branch
      • Contact Robert about run_visit updates for jointcal
  • DESC
    • attend CI meeting
    • Provide CV
    • Attend DC2 meeting and take minutes
  • NERSC
    • Create new w_2018_38 docker image for MWV
    • Work out problems with movable stack builds
    • Check on /usr/bin/time on Cori and Edison and contact NERSC
  • SLAC
    • Deal with IR2 stack installs and python questions
      • Document on Confluence
  • Fermi
    • attend s/w meeting
  • nEXO
    • attend sims meeting
    • install latest sniper-offline at SLAC
    • Check on OSG installation
    • Meet with Lisa on Wednesdays
    • Check in with Raymond about OSG

 

Week of Sept 17

  • DM-DC2
    • Prepare and run DM-DC2 Meeting
    • Test and update new SRS workflow 
      • move to Edison
        • cannot run on Edison - chatted with Tom, sent a note to Tony, Brian, and W
          • Tried Tony's suggested fix and it seems to be working
      • move back to Cori
      • write up for Warren
    • Stay on top of obs_lsstCam updates
      • Create run2.0 branch
      • Update run1.2 branch
    • Create scripts for merging ingested repos
  • DESC
  • NERSC
    • Handle permissions on $CSCRATCH/../desc/DC2/Run2.0i
    • Work out kinks setting up movable stack builds
  • SLAC
    • Reinstall v16 binaries and test updating python install
  • Fermi
    • Tag evtClassDefs
    • attend s/w meeting
  • nEXO

 

Week of Sept 10

 

Week of Sept 3

  • DM-DC2
    • Prepare and run DM-DC2 meeting on Tuesday
    • Validation processing of Run1.2i
      • generate amplifier files from original eimages
      • run ingest on Cori Haswell and Cori KNL
      • run singleFrameDriver and report errors
      • run makeFpSummary
      • Try visualizeVisit
        • update obs_lsstCam to include necessary template
      • Keep documentation updated
      • Answer Tom U's questions about ingestDriver
    • Coordinate obs_lsstCam branch and create Run1.2 tag
  • DESC
    • Create new lsst_sims docker image for Jim and upload to NERSC
  • NERSC
    • Inquire about desc access to globus
    • Clean up inode use on /projecta
  • SLAC
    • install w_2018_26 for Emily
    • Handle vendor ingests (5 so far)
  • nEXO
  • Fermi
    • software meeting
    • check celestialSources used in GR L1 versus master
    • contact Luis about updates to eblAtten

 

Week of Aug 20

  • DM-DC2
    • Prepare and run DM-DC2 meeting
  • NERSC
    • install phosim 3.7.16
  • SLAC
  • nEXO
  • Fermi
    • Software Meeting

Week of August 13

  • DM-DC2
    • Prepare and run DM-DC2 meeting
    • Attend CI meeting
  • NERSC
    • finish coadd processing Run1-2iTest
    • Test use of DMstack Shifter image in jupyter-dev
    • update jupyter-dev confluence pages
    • fix inode use on projecta
    • finish testing relocatable Stack weekly build
  • SLAC
    • release and install latest datacat-utilities on SLAC public and IR2
  • nEXO
    • attend Sims meeting
    • Install nEXO_MC and test running FastCharge
  • Fermi
    • respond to Katie Chapman's Fermipy helpsoft issue
    • Update workbook pages to remove POI form

Week of July 16th

  • DM-DC2
    • Prepare and run DM-DC2 meeting
  • NERSC
    • Update shared python installation
      • GCRCatalogs
      • new CCL
        • track down reported notebook failure
        • track down reported notebook failure
        • reinstall at NERSC pyccl 0.3.5
  • SLAC
    • Follow up on laptop purchase
    • Release and install datacat-utiltiies-0.1.1
    • Release and install datacat-utiltiies-0.1.1
    • Install sims 2_9_0 at SLAC
  • nEXO
    • Sims Meeting
    • Test Docker image in Singularity
  • Fermi
    • Offline S/W meeting

 

Week of July 9th

  • DM-DC2
    • Prepare and run weekly DM-DC2 meeting
    • Prepare for XWG DM-DC2 Meeting
  • NERSC
  • SLAC
    • Follow up on laptop purchase
  • nEXO
    • Meet with Lisa
    • contact Raymond about fast charge
    • contact Alex about fast charge (email sent)
    • Gain access to LLNL account
    • Pursue Singularity container
      • Install Singularity on CentOS7
      • Try using our docker image within Singularity
    • Document Docker container
    • Set up version control for SNiPER framework
    • Check OSG external library installation and send email to Benedikt
  • Fermi
    • Offline Software meeting

 

Week of June 25

  • DM-DC2
  • NERSC
    • Monitor NERSC ticket concerning GPFS AFM
    • Inquire about monitoring SCRATCH space for upcoming purges
    • Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
    • Fix weekly builds of DMstack to use limited number of cores and skip w_2018_24, since it will not build due at NERSC
    • NERSC Ticket to work out details of weekly DMstack builds
    • Update CCL doc for collab meeting
    • Clean up disk space in /global/common/software
    • Fix inode use on projecta space
    • New sims 2_9_0 release, docker image
    • Install sims 2_9_0 at NERSC
    • Contact owner of galsim files installed under our w.2018.13-sims_2.7.0 installation in /global/common/software/lsst
  • SLAC
    • Check on set up of datacat-utilities
  • nEXO
    • Respond to CMT=>CMake project for SNiPER
    • Ping Marco (Gaudi) about their CMT=>CMake experience
  • Fermi
    • Verify that we can checkout L1 tags using --force and update doc
    • Create new GR FDM-10 build:  FDM-10 - Getting issue details... STATUS

Week of June 18

  • DM-DC2
  • NERSC
    • Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
    • Investigate load use of weekly installs and modify the cron job to limit the number of cores utilized.
    • update to GCRCatalogs 0.7.8 in the shared python environment as requested by Yao
  • SLAC
    • Install DMstack v15 for IR2, py3
    • Install w_2018_23 for summer grad student
  • nEXO
    • Write up slides for Docker intro for collaboration meeting and present
    • Write up slides for SNiPER update and present
    • Listen in to Simulation session 9PM-1:30AM
  • Fermi
    • Weekly software meeting

Week of Jun 11 2018

  • DM-DC2
    • Write up agenda, run, and take notes for weekly DM-DC2 meeting
  • NERSC
    • update fast3tree installation for Yao to allow writing to the install directory
  • SLAC
  • nEXO
    • Complete docker image for collaboration meeting
    • Inform collaborators and help get them set up before the meeting
  • Fermi
    • New L1 Release tag for GR See JIRA FDM-5
    • Build new L1 Release tag at SLAC for Philippe to use

Week of Jun 4 2018

  • DM-DC2
    • CI Meeting update
    • Prepare agenda, run DM-DC2 meeting, and take notes
    • Try constructFlat at NERSC
      • ingest subset of Run1-2p data
    • Doodle Poll to find new meeting time
      • chat with Dominique & Johann first
    • Ingest all Run1-2p data at NERSC
      • Merge latest IN2P3 workflow into NERSC workflow
      • Start up the jobs
  • NERSC
    • investigate installing new python module Yao requested
    • Talk to Joe Z about his miniconda env at NERSC to see about pulling it into the shared python env
    • Reinstall stack weeklies when the next python update for DM is released
    • deal with inode use on /global/projecta/projectdirs/lsst/
      • Issues is associated with the phosim SEDs for every visit, these files must be tarred.
  • SLAC
    •  
  • nEXO
    • Data Task Force Meeting Wednesday
    • Prepare proper docker image with tutorials
  • Fermi
    • merge slac doc in github
    • Offline Software meeting
    • Help Philippe make L1 updates to TkrDigi
    • get a new GR L1 release tag made

Week of May 28 2018

  • DM-DC2
  • NERSC
    • Finish slides for seminar and present
    • Test Javier's notebook
    • Fix py3-4.3.21 installation with numpy and numexpr
    • install phosim v3.7.15 for Haskell-Intel & KNL-intel
    • Install CCL for Yao in the shared python area
  • SLAC
    • Handle vendor ingests while Tom is away
  • nEXO
    • Attend sims meeting and report on docker status
    • Update docker image with tutorials
  • Fermi

Week of May 21 2018

Week of May 14 2018

  • DM-DC2
    • set up agenda and write up notes for Monady's DM-DC2 meeting
  • NERSC
    • Check getting started at NERSC documentation
    • Contact Michael & Javier about the NERSC seminar May 30th
    • Run phosim v3.7.9 and v3.7.14 test data through miniDRP
    • Check on kubernetes availability at NERSC jupyterhub and set up meeting
  • nEXO
    • update docker image to remove sudo and add in tutorials
    • coordinate archiving nexo-sniper
  • SLAC
    • set up order for new laptop
  • Fermi

Week of May 9 2018

  • DM-DC2
    • Setup agenda and write up notes for Monday's DM-DC2 meeting
    • Create darks using phosim 3.7.9
    •  
  • NERSC
    • Install new versions of phosim as they appear
  • nEXO
    • Attend sims meeting and make presentation concerning SNiPER visit
    • Update docker image with tokens
    • contact Raymond about timing correlation
  • SLAC
    • Set up order for new laptop
    • Requested and tested git lfs install at SLAC for nexo
  • Fermi
    • Attend software meeting and write up notes

Week of April 30 2018

  • DM-DC2
    • Set up agenda and write up notes for Monday's DM-DC2 meeting
    • Rerun one processEimage job on one core on KNL using a source build of the stack
    • Send Salman and Katrin directions about the steps to run "mini-DRP" so they can try it at ANL
    • Set up full pipeline at NERSC to run on KNL
    • Finish creating calibration products and document
    • Prepare Monday's CI update
  • NERSC
    • Update python environment to include potables
    • Create "current" symlink to point to the current python installation
    • Test and document setting up jupyter-dev using a shared stack install
    • Install phosim v3.7.10 for both haswell-intel and knl-intel
    • install phosim v3.7.11 for both haswell-intel and knl-intel
  • nEXO
    • Set up Docker image for nexo-offline + SNiPER
    • Provide daily summaries on the Slack meeting channel
    • Test new SNiPER using the last collaboration meeting examples
    • Understand how to implement Time Correlation
    • Attend Tuesday Update meeting with nexo collaborators
  • SLAC
    • Help Tom M set up python on his laptop
    • Update elog Slack app to add a new channel as requested by Aaron

Week of Apr 23 2018

  • DM-DC2
    • Set up agenda and write up notes for Monday's DM-DC2 meeting
    • Rerun one processEimage job on one core on KNL as requested by Salman
    • Send Salman directions about the steps to setup the DMstack so they can try it at ANL
    • Provide NERSC update for DC2 meeting
    • Attend DC2 meeting
  • NERSC
    • Create an area of IN2P3 to transfer their data products
  • nEXO
    • Install new version of SNiPER for centos7
      • request new tbb system library be installed
    • Send some initial SLAC usage instructions to the SNiPER developers
  • SLAC
    • Help Steve ssh tunnel to new IR2 jupyterhub

Week of Apr 16th 2018

  • DM-DC2
    • In support of SkyCorrection Task
      • Generate darks using phosim v3.7.9
      • Play with constructFlat.
    • Rerun miniDRP task using KNL and produce some timing information
    • Update LSSTDESC/obs_lsstSIm for Phosim v3.7.9
      • Tagged Run1.1p obs_lsstSim fork
      • Successfully updated the obs_lsstSIm files using the bias output
      • Work through unit test failures
      • PR
  • NERSC
    • Clear out out Stack builds as advertised on Slack
    • Install parts in shared python env + weeklies + v15
      • Consider setting up a common shared lib area
    • Install a modern dmtcp on knl-intel and haswell-intel
      • multilib install failed, went back to 64 bit only for now - which should be fine for phosim
  • nEXO
    • Prepare for travel to SLAC
    • Help SNiPER Developers get set up with SLAC accounts
    • Install new version of SNiPER at SLAC

 

Week of Apr 9 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO 
    • HistogramSvc
    • Write up SLAC SNiPER installation instructions on wiki
    • Send Lisa some journal references for Gaudi
  • DM-DC2
    • Prepare for Task Force Meeting and write up Live Notes
    • Complete initial run of first Run 1.2p visits through minidrp
    • Create new dc2 Shifter image with room for a NERSC volume 
    • Create dc2 Shifter image using a dc2 tag to try to avoid version mismatch errors
    • update workflow to use Shifter and submit PR
    • Create w_2018_14 + LSSTDESC/obs_lsstSim Shifter image
    • Test pipeline with w_2018_14+LSSTDESC/obs_lsstSim Shifter
    • update NERSC workflow to try out processEimage on rafts
  • NERSC
    • Prepare updates for CI meeting
    • Attend Ci meeting
    • Send Katrin some information about how the set up NERSC for DESC
    • Ask NERSC about anticipated need to recompile software after the scheduled Cori OS update on Apr 25-26
    • Install DMstack v15 in the common software area
    • Request to remove some older stack installations at NERSC
  • SLAC
    • Write up portal section in Overleaf
    • Complete Research integrity course
  • Aux Telescope

Week of Apr 2 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO 
    • HistogramSvc
    • Attend Simulations Meeting
    • Write up SLAC SNiPER installation instructions on wiki
  • DM-DC2
    • Set up new Docker image which includes patched obs_lsstSIm
    • Prep slides and attend SSim meeting
  • NERSC
    • Check on weekly builds and document their new location
    • Create new docker image for new sims 2_7_0 release
    • Install sims 2_7_0 from source in the common software area
  • SLAC
    • Complete CyberSecurity Training
    • Write up portal section in Overleaf
    • get antivirus going on home centos7 to allow VPN connection required to deploy portal
  • Aux Telescope

Week of Mar 26 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO 
    • HistogramSvc
  • DM-DC2
    • Prep for Task Force Meeting
    • Write up Live Notes
    • Contact Dominque B concerning processEimage failure
      • Now fixed with new config file
      • add this config file to github
    • Write update for Friday's DC2 meeting
    • Write update for CI meeting
    • Start up new mini-DRP task in SRS pipeline 
      • Run ingestData for all 66 visits from Run 1.1p
      • Run processEimage (obtain time and memory consumption stats)
  • SLAC
    • Fix LSSTTD-1257
    • Deploy data portal 0.5.14
  • NERSC
    • move weekly builds to common software area by updating cron job
  • Aux Telescope
    • On Hold until we hear from Kirk

Week of Mar 19 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
    • Attend Fermi Meeting
  • nEXO
    • Fix build of ROOT on centos7, missing libEve and libRGL
    • Finish building nexo SNiPER on centos7
    • Work on HistogramSvc
    • Attend simulations meeting
  • DM-DC2
    • DM-DC2 Meeting prep
    • Attend SSims meeting
    • Write up Live notes
    • Extend DRP Workflow to run processEimage
    • Update DRP Workflow to run processEimage for multiple visits
    • Meet with Nicolas to discuss DRP
  • SLAC
    • Implement and deploy new run search box
    • Update my own centos7 desktop to use SLAC VPN, NetBeans, etc so I can continue supporting the data portal after the VPN transition at SLAC.
    • Run vendor data ingests
  • NERSC
    • Install TreeCorr 
    • Install new GCRCatalogs in all conda environments
    • Move weekly stack installs
      • Clean up old contrib area
      • request another 100 GB of space
      • update cron job
  • Aux Telescope
    • Wait to hear from Kirk

Week of Mar 12 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
  • nEXO
    • Fix build of ROOT on centos7, missing libEve and libRGL
    • Finish building nexo SNiPER on centos7
    • Work on HistogramSvc
  • DM-DC2
    • Update Nicolas' ImageProcessingPipelines branch for NERSC
    • Demonstrate running SRS workflow for ingest 
  • NERSC
    • Work out production installation of lsst_sims + w_2018_09 using the shared python env
  • SLAC
    • Create some documentation to use jupyterlab at SLAC
    • Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
    • LSSTTD-1256 - Getting issue details... STATUS
  • Aux Telescope

Week of Mar 5 2018

  • Fermi
    • Finish off creating L1 branches for all GR packages
    • Get Brian to fully clean up between GR jenkins builds - see JIRA
    • Figure out release process for L1
    • Look at documentation:  old Workbook, Github, Confluence..
  • nEXO
    • Attend sims meeting
    • Work with SLAC IT to get all necessary system libraries installed on centos7 to build nEXO's externals
      • libXpm-devel
      • bzip2-devel
      • Motif
      • Motif-devel
      • mesa-libGLU mesa-libGLU-devel
    • Work through build problems with Coin3d 3.1.3 (an external lib)
      https://github.com/nEXO-collaboration/nexo-env/issues/1 
    • Finish building nexo SNiPER on centos7
    • Work on HistogramSvc
    • Create some initial dev doc in their wiki
  • DM-DC2
    • Attend and take notes for task force meeting
    • Attend SSims meeting Thursday
  • DESC
    • Create docker image using new lsst_sims + w_2018_09 
  • NERSC
  • SLAC
    • Continue watching jupyterlab progress 
    • Create some documentation to use jupyterlab at SLAC
    • Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
    • LSSTTD-1256 - Getting issue details... STATUS
  • ATS At Harvard
    • Attend test stand tutorial 9 AM - 10 AM PT Wednesday

Week of Feb 26 2018

  • Fermi
    • get Brian to fully clean up between GR Jenkins builds - created JIRA
    • get L1 GR to build 
    •  figure out how to clean up workspaces in Jenkins on my own.
    • Tag first GlastRelease release in github GlastRelease-20-10-04-gr03  only change is to the xxxRootData packages
    • Figure out the release process
  • nEXO
    • send an email update to Lisa
    • update Slack channel about Raymond's discussions with SNiPER developers
    • Work on HistogramSvc for ROOT files
  • DM-DC2
  • NERSC
    • Poke NERSC about jupyter-dev outage - now fixed
  • SLAC
    • Watch JupyterLab demo install and keep Steve informed
    • Try out eT DB updates to prep for Joanne's departure
    • Update DESC and Fermi documentation on getting a SLAC account
    • Check on Tom M's reported problem with the status of some CCD-011 bits in the DP
  • ATS At Harvard

Week of Feb 19 2018

  • Fermi
    • Finish Anaconda 3 install
    • Fix up L1 builds on Jenkins
    • Fermi Meeting
  • nEXO
    • Start new HistogramSvc for ROOT files
    • Simulation Meeting Thursday
  • DM-DC2
    • First Task Force Meeting
    • Migrate DC2 Cookbook issues to IPP (ImageProcessingPipeline) repo
    • Create new issues in IPP repo
  • NERSC
    • sort out HPSS allocation
  • SLAC
    • Help David L access his account
  • ATS at Harvard
    • Meeting with Tony & Owen on Tuesday
    • Go to Harvard on Friday

Week of Feb 12 2018

  • Fermi
    • attend offline meetings!
    • Finish Anaconda3 installation 
    • Work out distribution of externals in the new regime
    • Install swig 3.0.12 RH6 and RH7 on GLAST_EXT
    • work out L1 build in Jenkins
  • nEXO
    • Meet with sims people
  • NERSC Support
    • re-re-install cosmosis to help out the WL folks who wanted the latest along the develop branch 
    • Reach out to Joe to encourage tagging of cosmosis and provide some feedback on their setup script for NERSC
    • Reach out to wl pipeline group to ask about cosmosis installations.
    • Start up automated weekly stack installations
    • Document weekly stack installations
    • Send Joe Z updates to the setup-cosmosis-nersc script
    • Help Erfan get going with cosmosis at NERSC
  • SLAC support
    • Read over Joanne's docs
    • Pow-wow with Joanne over the DP and eT and the eT DB
    • set up docker for Jim on IR2
    • Update DESC and Fermi documentation on getting a SLAC account
  • DRP/L2 Processing
    • Task Force
      • Phil mentioned that "they" had to write up a charge - figure out what that means
        • Modify charge after receiving suggestions
      • At Nicolas' suggestion reach out to Fabio H concerning weekly stack installations
  •  ATS at Harvard
    • Attend CCS meeting to find out how best to support Harvard
    • Contact Harvard to sort out list of devices & plans for their use for testing
    • Meet with Kirk
    • write up minutes from Kirk/Nick meeting and prepare to iterate with Kirk

Week of Feb 5 2018

Week of Jan 29nd 2018

  • Data Portal
    • support Joanne as she learns about the plotting
  • DRP/L2 Processing
    • Check on James P
    • Get a copy of the DC1 WF going
    • Update WF to run Jim's latest imSim Data 
    • Learn how to deal with pipeline issue in JMX dealing with large number of jobs stuck in submitted state
    • Run Jim's imSim data through SLAC WF (first attempt in progress)
    • Prep to run using Phosim 1.1p
    • Document using the Slac WF and add scripts to a GH repo in LSSTDESC area
    • attend Friday dc2 meeting
  • SLAC support
    • review, release, and install the latest version of datacat-utilities
    • Read over Joanne's documentation
    • -automate weekly install of DMstack binaries for camera- 
    • install datacat-utilities on lsst-dc10
  • NERSC support
    • Upload latest SQRE DMstack (w_2018_03) Docker image into Shifter
    • Install w_2018_03 plus sims from source
    • Create w_2018_03+sims Shifter Image (debugging issue with afw - issue identified.. fixing it is something else)
      • create sims Shifter image the long way, without using lsstsqre's base image
    • Install WeakLensingDeblending for Javier.
  • Random LSST Camera and DESC items
    • SSim meeting
    • Meet with Wil on Friday
    • Write up notes from meeting
    • Look over DESC architects confluence page
  • nEXO
    • reinstall SNiPER in the new disk space
    • try out examples
    • sign up for the collaboration mailing list
  • Fermi
    • Investigate trouble with AcdDigi in L1 branch
    • Find a way to upgrade Anaconda external for ST
    • Install miniconda py3
    • Contact Helen

Week of Jan 22nd 2018

  • Data Portal
    • Consider rerunning all the ITL sensors in need of updated MET ingest and MET 05 by hand
    • plots - commit latest code and write up intro for Joanne
    • CCB request and deploy MET updates for the DP
    • Deploy DP 0.5.11
  • SLAC support
    • automate weekly install of DMstack binaries for camera (Jim doesn't seem to feel this is a priority)
    • Read over Joanne's documentation
  • NERSC support
    • update DESCQA install to include tensor flow
    • write up instructions for w_2018_01 shifter install
    • automate installation of shifter images
    • Create docker image of new sims release based on w_2018_04
    • Reintall cosmosis for WL group due to updated FFTW module, trying cray-fftw this time
    • Send instructions to Josh to set up his own test galsim using the extlibs we already have set up for Cori KNL gcc
    • install Phosim 3.7.7 and 3.7.8 for knl-intel and haswell-intel
    • meet with Mustafa about HPSS
    • Fix Cosmosis installation
  • Workflows/L2 DC2 Pipeline
    • send plea to Tony
    • Meeting with Tom about using the SLAC pipeline + pilots
    • work out setting up test Pipeline2 job reusing DC1 L2 task and a handful of visits
    • Prep for CI meeting
  • Fermi
    • Work through errors with AcdDigi on L1 branch
    • Set up new Anaconda external
    • Work out with Brian the requirements for setting up externals these days
  • nEXO
    • get Tony an disk space estimate for SLAC installs
    • get some disk space from Tony at SLAC
    • set up SNiPER at SLAC in a temp location (in progress)
    •  attend Tuesday sims meeting

Week of Jan 15th 2018

  • Data Portal
    • revisit plots
    • Holding off MET updates until new traveler to reingest ITL sensors is available
  • datacat-utilities
    • Install new release 0.0.9 at SLAC
  • SLAC support
    • automate weekly installs of DMstack binaries for camera
  • NERSC Support
    • update DESCQA install to include tensorflow
    • Write up instructions for w_2018_01 shifter install
    • automate installation of shifter images
    • Add WeakLensingDeblending 
    • Check on CCL installation
  • Workflows/L2 DC2 Pipeline
  • Fermi
    • Set up draft f2f meeting page Fermi Software Week Winter 2018
    • Continue working on GR L1 branch - completed checkout of L1 code successfully
    • Work through errors with AcdDigi on L1 branch
  • nEXO

Week of Jan 8th 2018

  • DataPortal
    • finish off ITL-3800C-368 ingest after sorting out http authentication errors
    • revisit plots
    • Hold off on MET updates until new traveler to reingest ITL sensors is available.
  • datacat-utilities
    • Install new release
  • SLAC suport
    • Install w_2018_01 binaries lsst_distrib for camera
    • Work out automated weekly installs via cron (u1 is 97% full, ~990 GB left)
    • Install Merlin's eotest branch
  • NERSC Support
  • Workflows/L2 DC2 pipeline
    • Set up lsst_sims docker image for NERSC & uploaded to Dockerhub
    • Try another lsst_sims docker image using lsstsqre image as the base rather than centos7 (initially failed, have new plans to retry)
  • Fermi
  • nEXO & SNiPER

Week of Dec 18th 2017

  • DataPortal
  • Workflows/L2 DC2 pipeline
    • Reproduce error James P sees
    • Repeat experiment with multiple visits
    • Repeat experiment using Head of ImageProcessingPipeline repo
    • Potentially meet with Jim & James this week
    • Catch up on Parsl progress
  • CI
    • Set up new Docker image to use with imSIm's travis builds
    • Create a smaller Docker images for use with travis
  • NERSC Support
    • Shared Stack
    • JupyterHub documentation and simplification
    • clean old project space by removing MBII-snapshots
    • Request reduction in old project space quota to 40 TB
  • imSim
  • Fermi
    • Set up GR L1 branch in GitHub
    • Test build of L1 branch
  • SNiPER
    • Read up on SNiPER
    • Meet with Lisa K on Tuesday 5 PM Eastern
    • send feedback on documentation
    • set up user accounts on Wiki, LLNL, OSG
    • talk to Tony about nExo

 

Week of Dec 11th 2017

  • Data Portal
    • Deploy ETCCB-429
    • MET updates - almost complete
  • CCS ATS
    • Install mvn and associated libraries and get Nick going with Lakeshore driver
  • Workflows/L2 DC2 pipeline
    • Send James P a link to the recordings from last week
    • Repeat test using Jim's parsl wf at NERSC
  • CI
    • Docker image for use in travis builds
  • imSIM
  • NERSC Support
    • Fresh Shared Stack installation for KNL - stuck working out disk space in the common software area  – started
    • dmtcp installation
    • Update jupiter-dev documentation
    • quota checker
  • Fermi
    • Try GR with repoman
    • Documentation

Week of Nov 27th 2017

  • L2 pipeline DC1 rerun
    • Finish off write up of chat with Tony
    • Ask James P about remote participation during Spring Week
      M & W 8 - 12 central time 
    • Try a run using Simon's cookbook
    • Meet with Jim & James P
      • write up notes
    • Try Parsl pipeline at NERSC
  • NERSC software installations
    • Shared Stack installation - verify that the DESCQA and py3 install is working
      • Add CCL
    • Install new sims 2.4.2 using DMstack w.2017.46 Haswell-gcc6 using py3
    • Ask about cosmosis needs
      • Install cosmosis, TreeCorr, and MLZ in projecta area under the WL WG directory
    • Install phosim 3.7.3
    • dmtcp
    • quota checker
    • Consider requesting more common software space
      obtained 50 GB more 
  • NERSC project space
    • Verify MBII-snapshots is in HPSS and remove from project area
      Found the content in the desc hsi area, and viewed the contents of some snap.tar via htar -tf CS/MBII-DMO-snapshots/snapdir_074.tar   Also noticed that the group_xxx.tar files are duplicated in the Archive home directory.  Contacted Mustafa to sort that out and remove duplicates.
  • Data Portal
    • MET updates
      • Meet with Anders and Seth
  • Harvard ATS
    • Check that ccs reboot restarts the MySQL, LocalRestServices, and Trending
    • Check in with Nick about the Lakeshore
  • Fermi
    • Try GR using repoman
    • Documentation
  • Discuss Gaudi for nEXO
    https://gaudi.web.cern.ch/gaudi/releases/v30r0/
    http://lcgsoft.web.cern.ch/lcgsoft/release/
    https://gitlab.cern.ch/gaudi/Gaudi
  • Attend Stanford IT Unconference
    https://itunconference.stanford.edu/sessions/remote

Week of Nov 20th 2017

Week of Nov 13th 2017

  • Harvard ATS
    • Create user accounts and ccs sudo account which grants sudo access to all users
    • Set up directory structure following IR2 model
    • Install DMstack v14
    • Install eotest compatible with v14
    • document the above on the Confluence page ATS Test Stands
    • Set up Trending Database
      • MariaDB installation
  • L2 pipeline DC1 rerun https://github.com/LSSTDESC/SSim_DC1/issues/47
    • Meet with Jim on Monday
      • Write up Notes
    • Contact James P
    • Try to run Simon's Cookbook
    • Chat with Tony
  • NERSC Software Installations
    • Shared Stack
      • first try failed due to conflicts with libiconv installed presumably for 
    • Phosim 3.7.2 bug fix for haswell gcc & intel, KNL gcc & intel 
    • DMTCP intel build
    • Help out with cosmosis installation https://github.com/LSSTDESC/WLPipe/issues/31
  • Data Portal
    • Chat with Tony about plots
    • Plots
      • Bad Channels
      • read noise
  • NERSC Project Space
    • Test MBII-snapshot HPSS copy and remove from project area

Week of Nov 6th 2017

  • QA Testing
    • Investigate cause of differences between Lauren's plots from NCSA and those generated at NERSC
      • look at 'conda list' on both NCSA and NERSC (no differences discovered)
      • generate cal_refs as was done at NCSA
      • try gcc 6 build as is being done at NCSA
    • Determine the cause of differences between ref catalogs at NCSA and NERSC
  • L2 Pipeline
  • Data Portal
    • Finish off first round of Richard's plots
      • bad channels
      • read_noise
  • NERSC Project Space
    • test MBII-snapshot HPSS copy
  • ATS
    • prep for Tony & Owen's visit
  • Installations
    • SLAC
      • Py3 on SLAC RH6-64 for Joanne to test out
    • NERSC
      • finish testing NERSC's cfitsio installation
      • install phosim 3.7.1 for Cori haswell gcc & intel, KNL gcc & intel
        • ask Glenn to install module files
      • install phosim 3.7.2 for Cori haswell gcc & intel, KNL gcc & intel
      • Shared Stack Installation using w.2017.41
        • Query DESCQA and CCL groups
        • py2 w.2017.41 installation
        • py3 w.2017.41 installation

Week of Oct 30th 2017

  • Data Portal
    • Plots
  • DC2 prep
    • QA testing
      • w.2017.41 installation plus branches of obs_lsstSim and pipe_analysis
      • Run visitAnalysis, coaddAnalysis, etc
  • NERSC Workflow work
  • NERSC Project Space
    • Work with Mustafa to get MBII-snapshot backed up
  • NERSC Installations
    • w.2017.? stack installation plus sims - waiting on new sims release using the camera geometry updates
    • Phosim 3.7
      • Testing NERSC cfitsio installation
  • SLAC Installations
    • descsoft w.2017.? stack plus sims installation
  • CCS
    • See if there is any prep work necessary for Tony's visit later in Nov
      • Request Nick set up new box with CentOS7
      • set up user accounts and groups
    • Determine if Mike is still coming out ~Nov 1st
  • Attend SuperCDMS meeting concerning their potential move to NERSC.

Week of Oct 23rd 2017

  • Data Portal
    • Richard's Plots - bad channels (in progress)
    • Look at new Report Labels and determine what the DP still requires from the Reports Table
      LSSTTD-1190 - Getting issue details... STATUS  
    • Make sure we request a restart of Tomcat
  • NERSC Workflow work
    • Try out HTCondor and Pegasus on cori21
    • Take a look at Jim's imsim script
  • NERSC Project Space
    • Set the old project directory to read-only (in progress)
      • claim desc ownership of the whole directory space
    • Clear out all directories where the owners have indicated it is fine to do so
    • Determine the end state of merging old project and projecta areas and handling of www
    • Set up disk quota monitor
    • Coordinate HPSS storage of MBII-snapshots
  • NERSC installations
    • Phosim v3.7
      • Coordinate with NERSC to get cfitsio v3410 installed (in progress)
      • Rebuild using NERSC supplied cfitsio
      • Coordinate with Glenn to set up module files
    • Set up software for DC2 testing
    • v14_0 lsst_distrib + sims 2.4.0
  • SLAC installations
    • Set up same imsim at SLAC as used at NERSC for KNL testing (in progress)
    • v14_0 on RH6-64 public (on hold pending resolution to obs_base testCameraMapper.py unit test failure)
      • camera area
      • descsoft
      • test eotest branch
    • v14_0 on IR-2
  • Document KNL vs Haskell mkl imsim testing at NERSC
    • Initial mkl tests indicate no huge improvement - putting testing on back burner for now
  • CCS - still waiting to hear from Peter, I cannot access their TS3 machine to install and test a new version of the test stand subsystem

Week of Oct 16th 2017

  • Richard's plots in DP
    • Complete Sensor Arrival Time plot
    • Create JSON 
  • NERSC installations
    • Phosim v3.7
      • Try out NERSC supplied FFTW 
      • Request new cfitsio v3410 installation required by Phosim
      • Complete installation of all flavors:  haswell-gcc, haswell-intel, knl-gcc, knl-intel
        • Waiting on cfitsio v3410 before knl-intel will be available
  • NERSC Project Space
    • Notify all users of the need to migrate
    • Handle all help requests - migrating some directories to the new projecta area
    • Install disk quota monitor
    • Check on old project status and prepare to set it to read-only
    • Coordinate backup of nkhandai data to tape with Mustafa
  • Install latest version of teststand subsystem at Harvard there is a machine made available - waiting on response from Peter 
  • Keep an eye out for v14 DMstack

Week of October 9th 2017

  • Continue work on Richard's plots in DP
  • LSSTTD-1165 - Getting issue details... STATUS
    • Released Data Portal 0.5.7
  • Testing imSim built on KNL
    • Continue MKL tests at NERSC once the system recovers from the weekend outage
    • Install the same version of the stack at SLAC to allow similar testing with and without MKL
    • Attempt build of DMstack using the intel compiler at NERSC
  • NERSC Project space
    • Install disk quota monitor
    • Coordinate backup of nkhandai data to tape with Mustafa
    • Prepare to set the old project area as read-only
    • Send mail to all owners of files on the old project space
  • NERSC HPSS SRU overallocation
  • Install latest version of datacat-utilities 0.0.8 at SLAC
  • Install the latest teststand snapshot at Harvard to support use of the monochromator when Peter returns.
    • Contacted Peter - I can't connect to the archon machine to install the test stand snapshot
    • Investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly create a harvard main routine.
  • Keep an eye out for the v14 DMstack release and prepare to install on IR-2 and NERSC
    • This release will include binary installations.. I'm thinking we may still want to install from source, perhaps that's most useful on IR-2, but we'll see.

Week of October 2nd 2017

  • Release DataPortal 0.5.6 which will include
    • Addition of Grade and Procurement labels for the Sensor Acceptance pages
    • RTM assignment
      LSSTTD-1121 - Getting issue details... STATUS
    • NCR "Ready to close"
      LSSTTD-1144 - Getting issue details... STATUS
  • Implement Richard's plots in the DP starting from some of Tony's examples for trending and Richard's python code.
  • Continue testing KNL performance using galsim and imsim
    • Attempt to compile galsim and possibly all of DMstack with the intel compiler
    • Run longer tests using the MKL libraries on Haswell and KNL
    • Rebuild imsim with MKL, using an updated meas_base
  • Install an initial shared stack in the new software area at NERSC for testing as discussed at last week's "Shared Stack" meeting
  • NERSC Project Space
    • Set up an initial quota monitor at NERSC
    • Re-connect with Mustafa to get the mkhandai 42 TB archived to tape.
  • Investigate an upgrade of the monochromator driver at Harvard and confirm that to achieve what Peter plans to do - to run his ATS work in the lab alongside most of the devices from his TS3 setup.. is possible by running both the teststand and rafts subsystem simultaneously.
    • Install latest SNAPSHOT of test stand at Harvard and investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly creating a harvard main routine.
  • Keep an eye out for the v14 release of the DMstack and prepare to install at SLAC/IR-2 and NERSC.
    Though this is assigned to Jim, so perhaps I have no need to worry about SLAC:
    LSSTTD-1157 - Getting issue details... STATUS

 

  • No labels