Page History

Snowmass https://snowmass21.org/computational/start

Week of 12 April 2021

pgpool
deploy redmapper in desc-python:bleed
sn-py-dev
Point Johann at Scott's data and contact nersc to update ownership
Arrange all Work Plan mtgs
Coding Guidelines Reboot
descqa script update
snana performance test
CI/PS discussion
Portal update DR6-v2 and follow up on coadd images
risk registry
Log of all past WG convenres and committee members
write up NERSC work
OC mtg
mtg with OM & DOMs
updated desc-drp-stack with new parsl branch, ndcctools

Week of 29 March 2021

NERSC
- continue disk cleanup
- deploy pyspark
- test desc-python with notebooks
- Update jupyter Confluence doc adding restarting Hub and fixing anchors
CO
- check on SNANA runtimes
- reach out to Debbie concerning NERSC onboarding
DESC DM
- spin up postgres 13 plus pgpool
- spin up gen3 workflow with web front end
- Upgrade public portal to include DR6 v2
- meet with James Perry on Thursday
- DESC DM/DA mtg
DOM
- meet with Manu
- Poke to set up June NERSC onboarding session
- OC
- Mgmt meeting
- Leadership mtg
- light census query
- newsletter query for new members
- add new South African institution
- update dates for a couple of members
- risk registry
- learn more google sheets
- follow up on Slack workspaces
- collect quarterlies
- Update DB with Alex's instittion fixes and expand with latex name
CC
nEXO

Week of 15 March 2021

NERSC
- Check on Spin Postgres Performance
- Deploy Postgres v13
- Continue cleaning up CSCRATCH for descdm
- Clean up CSCRATCH for desc account
- Prepare 2 coadd tracts for public release via portal
- Test https://github.com/scanon/container_proxy for use with Shifter
- Test https://github.com/scanon/container_proxy with our Shifter & gen3 stuff
- Try certbot ranger
- Set up new desc-python:pyspark
  - Coordinate with Stephane to get this set up in a maintainable way
CO
DESC DM
- DESC DM/DA meeting
SLAC
DOM
- DM/DIA meeting on Monday
- Set up computing onboarding session
CC
- PubBoard Rotation
  - announce to CC
  - arrange chat with PubManagers
- MC rotation

Week of 8 March 2021

NERSC
- shifterimg w_2021_08 - w_2021_10 use similar .eups areas and seem to cause some confusion. Moving my .eups cache helped.. Not sure what to do more generally. MIght want to do "shifter -E" more generally
CO
DESC DM
DOM
CC
Fermi
nEXO

Week of 1 March 2021

Week of 22 Feb 2021

NERSC
- Update all py envs for GCRCatalogs 1.3.0
- Update py envs for pyccl v2.2.0
- Handle new user requests in iris
- Get Postgres Spin DB going
- Confirm with NERSC when the next quarterly reduction might take place
- Meet with Rick Kessler concerning SNANA disk and CSCRATCH needs
- set up Rick's script for disk use
CO
DESC DM
- Write agenda, attend, run, note take DESC DM/DA meeting
- invite PZ folks to next DESC DM/DA meeting
- Run 3.1i DR3 through validation notebook
CC
- Attend monthly CC meeting
- Membership and PubBoard rotation work
  - present status to CC
  - contact potential nominees
DOM
- attend mgmt mtg
- Get Onboarding email script going
- set up empty Trello board
- new desc-ops-technical channel
Fermi
nEXO
- address questions running singularity on SDF

Week of 15 February 2021

NERSC
- Help Bruno migrate DIA data to CSCRATCH and use shifter jobs
- Start up initial Postgres Spin app
- Announce Cori Outage
CO
- Agenda, note take and run DESC DM/DA
- Invite Emily to present woes running at NERSC at upcoming DESC DM/DA
- Invite Ben & Tom to present at next DM/DA
- Attend CSS meeting
- Attend SNWG PS meeting where they discuss Gen3 plans
- Look at WL Shear Projects in prep for Monday Meeting
- dustmaps package
- Update desc-stack-weekly for w_2021_07 and w_2021_08, where the new rubin-env is in use.
CC
- Send emails to potential MC and PubBoard candidates
- Deal with rejections and find additional candidates
DOM
- Attend Leadership Meeting
- Attend Mgmt Meeting
- Attend Ops Committee Mtg

Week of 1 February 2021

Attend DESC Collab Meeting
NERSC
- Address desc-python issue from user who was initially unable to set up user package alongside desc-python
- Update all NERSC Confluence Docs
- Bring desc-python repo up to date with desc-python at NERSC to allow IN2P3 to update their install
- Handle new user requests
- Ferry Bruno's job allocation increase request
- Start process of migrating from lsstdesc-portal.nersc.gov to another site and SSL certificate by working with Brian & Tony & NERSC
CO
- Upgrade CCL for new release in all desc python environments
DESC DM
- Test gen3 bps at NERSC using CVMFS
- Update gen3_workflow to use shifter
- Investigate how to use singularity & slurm in gen3_workflow

Week of 25 January 2021

NERSC
- Postgres Spin deployment
- set up cron job for desc-python:bleed
DESC DM
- DM/DA mtg
- gcr-catalogs update desc-python:bleed workflow to allow automated updates
CC
- respond to various elections
- Membership Committe & PubBoard rotations - create nomination list
- Prepare Jan DESC newsletter entry
CO
- Run2.2i DR2 and Run2.2i DR6-v2 releases
  - reorg butler data to reside under /global/cfs/cdirs/lsst/shared/DC2-prod
  - reingest raw data for new location of sim
- dust off desc-drp-stack image and upgrade for Gen3 current weeklies
  - Slurm support
Fermi
- software meeting
nEXO

Week of 18 January 2021

NERSC
- attend NERSC NUG concerning Perlmutter
DESC DM
- Update DC2 Data portal to handle ANL endpoint
- Prepare updated desc-drp-stack docker image for Gen3
CC
- CC monthly meeting
- MC & PubBoard rotation discussion
CO
- Attend CSS meeting
- Fix failing desc-python:bleed builds
- Update gcr-catalogs master workflow to update desc-python:bleed
Fermi
nEXO

Jan

Write up Run2.1 & Run3.1 descriptions
Deal with versioning of DR6

Week of 7 December

Ingest Run2.2i @ NERSC
Get initial web site up adn running
DESC Note
NERSC
- Recheck on user requests
- add in description of Fawad's Compute request
Fermi
- workbook update
nEXO
- event viewer

Week of 30 November

Sprint Week
- Update all DESC conda environments with new GCRCatal
nEXO data

Week of 16 November

CO
- write agenda, attend, potentially run and note take for CO meeting
- set up Zoom #s for Sprint week
NERSC
- shut down desc-python:bleed scheduled build for now
DESCDM
- write agenda, announce, and run DESC DM/DA mtg
CC
- attend SBPC update and note take
- Write up DESC newsletter entry
Fermi
SLAC
nEXO
- attend software meeting

Week of 9 November

CO
- attend CS meeting
- Help create lsstdesc-env conda-forge package
NERSC
- install intake in desc-python:bleed
- follow up on CVMFS problems with sqlite3
- Run a proper DR6 Gen2 ingest
- Create SPIN Postgres
- Test SPIN website
DESCDM
- handle DESC DM/DA meeting
- Discuss GCR future with Yao & Joanne
- Check on DR6-WFD warps
- Look at DR2 object catalogs to regenerate the ones fixed by the finished tract/patch processing
CC
- announce, attend and run CC meeting
- handle OMP amendment voting
- handle PM amendement voting
  - edit draft collaboration announcement
- Write up DESC newsletter entry
Fermi
SLAC
- attend ADASS as possible
nEXO

Week of 2 November

CO
- attend CS meeting
NERSC
DESCDM
- DESC DM/DA meeting
CC
Fermi
SLAC
nEXO

Week of 26 October

CO
- Contact James P & George B for CO meeting
- Write agenda, attend and note take for CO meeting
- Upgrade GCRCatalogs to v1.0.2 for all python environments
- Review DC2-analysis tutorials and automate beavis running
NERSC
- attend GPU webinar
- Update desc-stack-weekly for w_2020_44
  - automate those builds
- Follow up on Jupyter issues on Monday
- Deal with reported issues with MPI and nbodykit in desc-python env
- Help SN group upgrade to GCC 9.3.0
- Address question concerning permissions
DESC DM
- Announce, run, write agenda, for DESC DM/DA
- Complete initial set of DR2 object catalogs
- Copy matched catalogs to shared area at nersc https://github.com/LSSTDESC/desc-help/issues/49
CC
- Help write up slides, update agenda for DESC-wide Software Policy meeting
- Follow up on SBPC DESC-wide meeting
- Schedule November CC meeting
- ask MC to send out full membership announcement to collab
- Prepare and send email to DESC members reminding them to update DB entries
- Follow up on SNC
- Prepare OpaVote for ratification
- Prepare Collaboration-wide comment period for FM amendment
SLAC
Fermi
nEXO

Week of 19 October

CO
- attend CS meeting
- finish Gateways seminars
- Continue viewing JupyterCon
NERSC
- try out Globus Spin app
- update user on how to intall nbodykit
- Address various desc-help requests
DESC DM
- announce, write agenda, note take and run DESC DM meeting
- Produce object catalogs for all DR2 tracts where reasonable
CC
- attend, run CC Mtg
- Update SRPC page
- Announce SP DESC-wide meeting
SLAC
Fermi
nEXO
- attend software meeting

Week of 12 October

CO
- attend PS meeting
- attend Gateways
- attend JupyterCon
- attend Rubin Bootcamp
- attend SSim meeting
- Update Jupyter at NERSC Confluence page
NERSC
DESCDM
- attend TJP meeting to help present DC2 data availability
- write agenda, note take, attend DESC DM/DA
CC
- Stand up CCNC
- Prepare for Oct 19 meeting
- OMP ratfication
SLAC
Fermi
nEXO

Week of 5 October

CO
- Attend JupyterCon tutorials
- Attends parts of Snowmass
- Q4 update
NERSC
- Follow up on Humna's questions about trouble using desc-stack during Cori outage
- Post NERSC Cori updates to desc-nersc
- Check on HPSS storage use
- Work out issues connecting to jupyterhub with CSCRATCH unavailable.
- descqa website problems
DESCDM
- Attend ParslFest
- Follow up on DR2 WWT data
CC
- Announce potential CCNC members to CC and gain approval
  - send invites
- Announce CC Oct meeting, update DESC calendar, alert SP team, write agenda
SLAC
Fermi
nEXO
- Test SDF for nEXO user using singularity
- Set up new nEXO SDF space
- Start up build to set up a private docker image including nexo-offline to test with Yee

Week of 28 September

CO
- Attend CS meeting
- Prepare for CO meeting
- Attend DIA meeting
NERSC
- set up data for WWT
- check on desc-bleed
- set up re-useable Gen2/Gen3 repo at NERSC
- refresh small dataset on HPSS
DESCDM
- Write agenda, announce, note take, attend DESC DM/DA meeting
- Back up DR2 calexps just in case
CC
- Get DESC-wide meeting set up for SRPC and SBPC
- Stand up CCNC
- Work on scheduling Oct CC Meeting
SLAC
Fermi
nEXO
- set up nexo-offline on SDF with Yee

Week of 21 September

CO
- Attend SSim meeting
NERSC
- Set up Transfer for WWT
- Transfer DC2 Run2.2i to lsstdev
- check on desc-bleed
DESC DM
- Announce, write agenda, attend DESC DM meeting
- Grab updated Gen2 calibration flats for Gen3 testing
- set up Gen2/Gen3 repo at NERSC
- Move on with Gen3 testing at NERSC
CC
- Sept CC meeting
- MRPC Meeting
- Provide newsletter update
- Announce Software Policy draft
SLAC
Fermi
nEXO
- Work with Yee to set up nexo sims on SDF
- Software meeting

Week of 14 September

CO
- write agenda, take notes for CO meeting
  - announce meeting and invite DMers
NERSC
- Set up transfer for WWT
- Update desc-python:bleed and turn automatic installation back on
DESC DM
- write agenda, attend, take notes for Wednesday meeting
- Get DR6 WFD annoucement out the door
- organize warps and put them on CSCRATCH
- fix up dr2 calexps from Run2.2i
- regenerate tract2visit sqlite DB
CC
- set up Sept mtg time
- discuss SRP edits
- Push on CCNC and SNC meetings
- Contact 2019 SNC about survey
Fermi
Slac
nEXO
- attend Slac computing discussion on Thursday
- attend software meeting
- respond to issues submitting jobs on GPUs
- Work with Yee to set up some modules on SDF
  - Figure out a good production version using LLNL's install

Week of 7 September

CO
- attend CS meeting
NERSC
- add ceci to desc-python
- add ceci to desc-stack
DESC DM
- write agenda, attend, note take for DESC DM meeting
- Keep warp copying going
- Update scripts to find the warps left to copy and try globus instead
- Chat with Tom about warps
- Set up tract2visit mapper for DR2
- Find uband noPSF config discussions
CC
- Set up Sept CC MTG poll
- Push along SNC and CCNC
SLAC
Fermi
nEXO

Week of 31 August

CO
NERSC
- check on descdm quota
DESC DM
CC
- Aug CC meeting notes for collaboration
- Get back to meetings committee concerning their question
- get MRPC meeting set up
SLAC
Fermi
nEXO
- attend software meeting
- Try out GH Actions

Week of 17 August

CO
- https://github.com/LSSTDESC/desc-help/issues/35
NERSC
- help SN set up a conda environment
  - meet with Bruno and Rick
  - inquire about TxPipe
- look at running sn-pipe in tutorial notebooks
DESC DM
- agenda, attend DESC DM/DA meeting
- continue pulling over Run2.2i DR6 data from CC
- Prepare for DR6 release
  - back up pixel level data
  - remove unwanted tracts and patches
- Reach agreement on what is DR6d
- Update DC2 calibration headers to adhere to new CALIB_ID format
- Y1 warps for DR2
- add /usr/bin/time to shifter image for parsl DR2 runs
- data transfer to lsst-dev all of Run2.2 and Run3.1
- Clean up CSCRATCH
CC
- announce, write agenda, attend Aug Meeting
- write up DESC-wide minutes
- complete MRPC note and send
- Put out one more call for CCNC and SNC members
- Questionnaire Survey
SLAC
- glastsys password
- performance review
Fermi
nEXO
- attend software meeting
- test free github area
- look at open source chat tools to replace Slack

Week of 3 August

CO
- attend leadership meeting to update SRM
- export Zoom stats
- Size of DC2 files
- fverify Run3.1i sim files
NERSC
- update sn repo wth NERSC instructions
- test GHI
- setup new SN min
- rsync coadds again, consider cron
DESC DM
- Copy DR6 coadd data to NERSC
CC
- poll for August meeting
Fermi
nEXO
- attend software meeting
- add more users to nexo UNIX group

Week of 27 July

CO
- Complete Q3 report
NERSC
- Reorg Run1.2x data to reside on CFS
- Back up Run3.0i data on HPSS and remove from CFS
- Finish cleaning up my CSCRATCH
- Clean up desc's CSCRATCH
- Setup automated running of DC2Analysis tutorials at NERSC
- add selenium and python-snappy to desc-python https://github.com/LSSTDESC/desc-help/issues/31
  - need to fix up desc-python-bleed
DESC DM
- Upgrade desc-stack-weekly for next DM weekly, which hopefully fixes their conda install issues
- Reinstall Run2.2i Gen3 repo using latest DM weekly
- Update tutorial notebook for Gen3
- IPP eups
- setup rsync of DR6 WFD coadds
- Store Run2.2i WFD in HPSS
  - Y1
  - Y2
  - Y3
  - extract unneeded Y4 and Y5 files
  - Y4
  - Y5
CC
- write up June monthly meeting notes
- schedule August meeting
  - send out poll
SLAC
Fermi
nEXO
- Evaluate open source chat options

Week of 20 July

Attend collaboration meeting
Install new desc-python-dev to downgrade pyarrow for desc-pyspark
complete Gen3 Run2.2i demo repo
Set up desc-stack-weekly-latest for w_2020_27 to be used with Gen3 repo
Write up notes for others to run notebook examining Gen3 repo
nEXO
- respond to Samuele's questoins about CVMFS
- respond to Vladimir's data proposal

Week of 13 July

CO
- attend WL meeting
- Write up builder nomination letters
- Announce, attend, note take CO meeting
DESC DM
- announce, attend, note take DESC DM/DA meeting
NERSC
- set up Run2.2i Gen3 repo
- Fix entrypoint for desc-bleed
CC
SLAC
Fermi
nEXO
- attend software meeting
- create, test and install new nexo-python docker image at SLAC and announce
- set up nexo-base cpuonly pytorch container
  - reproduce success
- Update Singularity instructions to include how to submit jobs to bsub ( and slurm)

Week of 6 July

CO
- ask Debbie to speak at next CO meeting
- attend first half of DIA meeting
DESC DM
- Write agenda, note take and attend DESC DM / DA
NERSC
- Back up Run3.0 sim data to HPSS
- Back up truth catalogs to HPSS
- finish setting up dustmaps
- add v20 stack-sims image
- setup Gen3 repo in postgres (started)
- back up catalogs https://docs.google.com/presentation/d/1GErsSx1zNKtshnfWocCgUPxbmBtTkSuIHKUzAbx9Ae0/edit#slide=id.g73aaf4c5c5_0_3
- Copy out July 7th version of metacal DR6a merged tract parquet files
- Update desc-python & desc-stack for pyarrow, dask, and distributed: https://github.com/LSSTDESC/desc-help/issues/25
CC
- Write up June meeting notes for DESC
- Check on black lives matter Slack channel
- follow up on update to OMP
- Follow up on updates to Membership Policy
Fermi
SLAC
nEXO
- attend Slac computing discussion
- attend software meeting
- Update nexo-base to use py3.7
- Update nexo-base to install pytorch cpuonly
- Update nexo-base to install pytorch with cudatoolkit == 10.0 for SLAC compatibility
- Update singularity doc to include installation at SLAC instructions
- Update Singularity instructions to include how to submit jobs to bsub ( and slurm)

Week of 29 June

CO
- Finish up slides for DC2 update for WL and present
- Help plan Gen3 sessions at collab meeting
DESC DM
- Announce, write agenda, and note take DESC DM meeting
NERSC
- Count up data sizes of all Run 2.2 and 3.0 FITS
- fverify all Run3.0 files, update sqlite tracking db and notify Fabio we are ready to transfer
- Fix up desc-bleed and make sure build is scheduled and cron job set up to import into shifter
- set up rsync of object catalog area at CC (set up cron job to continue daily updates)
- set up rsync of metacal
- finish setting up dustmaps
- add v20 stack-sims image
- set up DR2 calexps for descdm
- Clean up descim CSCRATCH
- Back up truth catalogs to HPSS
CC
- Send Membership Committee Annual report to CC
- Follow up on updates to Membership Policy
- Check on SBPC draft to make sure it is not publically available.
- Follow up with SRPC to make their google doc draft as a draft
SLAC
Fermi
nEXO
- add more users to GitHub repo
- add more users to nexo UNIX group and look into ongoing inconsistent group membership on RHEL6 and centos7
- inquire about SLAC backup policies

Week of 22 June

CO
- attend CI interviews
- set up desc-stack-sims for v20 - but first perhaps sort out which sims weekly to use with it (22 or 24)
- set up desc-stack for v20 just create the tag, don't make it the productoin desc-stack
DESC DM
- scrape metacal logs and ?,?_nImage.fits for exposure info
- Setup notebook to look at data
NERSC
- clean up descim area
- back up truth catalogs
- rsync object catalogs and coadd from CC
- setup rsync cron job
- Handle desc-help #22 copying files for Sam
- set up cron job for desc-python-bleed
CC
- Write up June meeting minutes for DESC
SLAC
Fermi
nEXO
- attend software meeting

Week of 15 June

CO
- attend CS meeting
- attend DESC-wide meeting
- attend DC2 meeting
DESC DM
- setup CI for desc-drp-stack
  - add test for numba
  - add pyccl test for desc-stack
- set up desc-stack weekly images
- Look at GH actions environment differences: https://github.com/actions/virtual-environments/blob/ubuntu18/20200604.1/images/linux/Ubuntu1804-README.md
- merge PR#221 in obs_lsst
- Look into adding configs to IPP
- Join Snowmass mailing lists and slack channels
- add dustmaps packages to desc-stack (and possibly desc-python)
NERSC
- https://github.com/LSSTDESC/desc-help/issues/19
CC
- revise EDI committee member invite
- send EDI committee invites
- Set up June meeting and write agenda
- announce EDI committee
- attend CC meeting
- Update trello
- Retrieve GoogleForm response and send to CC
SLAC
- add DESC computing doc to KIPAC github
Fermi
nEXO
- Livermore workshop
- Meet with Lisa
- Update SLAC wiki page to fix broken links
- Try setting up LHC libraries again
- Update SLAC wiki page doc

note stack-jupyter:prod was pointing to v19_0_0-v3r1 moving to v19_0_0-v6r1

Week of 8 June

CO
- attend SSim meeting
DESC DM
- Set up and attend DR2 processing meeting
- FInish setting up DR2 repo
- Meet, set up agenda, note take, to discuss DR2 processing at NERSC
- Update Run3.0i Confluence page to include pointers to data and software versions
- Validate Run3.0 y1-y3 fits files
- Update desc-drp-stack image to include llvmlite 0.30.0
NERSC
- Create ticket to help user with UNIX group membership
CC
- Call Dawson vote for EDI
- Announce end of vote for PubBoard
- Create poll for June Meeting
- Help draft strike statement
SLAC
Fermi
- Cancel software meeting
- Rebuild 20-10-04-gr07 to compare lib/<variant> vesus lib structure obtained from Jenkins versus local build
nEXO
- attend sims meeting
- add another user to the nexo group
- closeout SLAC IT ticket concerning group membership across rhel6 and centos7
- Investigate NEST submodule failure - the branch with the commit was deleted
- Test Nest update PR 141 and merge and inform Jason Bane of fix

Week of 29 May

CO
- Write up agenda, attend and note take for CO meeting
- Attend Leadership meeting
- Listen in to WL meeting
- Reorg DC2 Confluence page: DC2
- DC2 Hack Day June 5th
  - Go over Confluence Doc
  - check on desc-dm-dc2 repo and update any areas to use CFS and add in Run2.2i
  - Check on DC2-Analysis
  - Transfer DR6 object catalogs to NERSC
- Provide desc-stack-weekly
DESC DM
- Announce, write agenda, note take DESC DM meeting
- set up new desc-drp-stack repo and got new v19-v2 docker image and wrote up doc for Tom
- Started setting up new DR2 repo, need to request new quota and figure out how much room DR2 might take up.
NERSC
- Attend Super facility Brown Bag
- Transfer registry.sqlite3 file from CC after DR6 calexp transfer
CC
- Post May CC meeting Confluence Page
SLAC
- Attend Virtual Residence when possible: http://www.oscer.ou.edu/virtualresidency2020.php
Fermi
- annouce software meeting
- Check on new "externals" directory for M.E.
nEXO
- Attend sims meeting
- Copy over more chargeSim files to SLAC
- Set up Doxygen for Lisa
- Provide example running 2 instance of the same service
- Test CMake ExternalData_OBJECT_STORES
- Announce SLAC FTP site
- Provide incident example
- send Tony Dockerfile
- update Dockerfile for Doxygen

Week of 25 May 2020

CO
- Attend CS meeting
- Clean up desc-python doc and yaml files
DESC DM
- Announce, write agenda, note-take DESC DM
NERSC
- Request full copy of DR6 calexps from CC
- Follow up on outstanding NERSC account requests
- Attend Superfacility Brown Bag - SPIN
- Request early access to rancher 2
CC
- Finalize MCPs registration wording
- Summarize May meeting minutes for DESC
- Provide CC summary for DESC newsletter
SLAC
Fermi
- Annouce software meeting
nEXO
- copy chargesim data files to SLAC
- Set up doxygen for Lisa
- Advertise FTP area and updated CMake files

Week of 18 May 2020

CO
DESC DM
- Announce, write agenda, write notes for DESC DM meeting
NERSC
- https://github.com/LSSTDESC/desc-help/issues/17
- Add GCRCatalgos master to desc-python-dev
- Attend SuperFacility Brown Bag
- Attend SIGEUF
- Update desc-stack and desc-python with new v0.17.0 GCRCatalogs
- Inform DESC user when over Computing allocation
CC
- attend CC meeting (note take on the side)
- WRite up DESC summary for meeting
- Work out PubBoard rotation
- Respond to Membership Committee email concerning annual report
- Update draft for note to Meetings Committee concerning upcoming registration
- update CC trello
- Write up possible update to MCP request on registation form
SLAC
Fermi
- announce s/w meeting
nEXO
- Meet with Lisa
- Poke Tony about Jenkins builds and web accessible area for doxygen output
- Set up new doxygen for Lisa and test it out

Week of 11 May 2020

CO
- Announce, write agenda, and note take for CO Meeting
DESC DM
- Announce, write agenda, and note take DESC DM meeting
- Refresh HSC Gen3 example at NERSC
- Demonstrate gen2togen3 conversion using DMstack w_2020_19 and Run2.2i data
NERSC
- Continuing move to shared area
  - copied PhotoZDC2 catalogs: https://github.com/LSSTDESC/gcr-catalogs/issues/424
  - Copy old and new Buzzard catalogs
  - descqa/v1
  - SSim data
- Transition WGs to CFS
  - CL
  - CS
  - LSS - created new area and requested current owners move content
  - PZ
  - SSim
  - TJP - removed content as agreed
  - WL - created new area and in contact with conveners to determine who will handle the data move
- Prepare for WG NERSC migration
  - Announce on desc-announce
  - Alert all WGs
  - Back up old and new descqa areas
- attend NERSC brown bag series - New Data Management
- Inquire about NERSC DB services
CC
- write up agenda for May 18th meeting, post entry on DESC Calendar
- attend SBPC DESC-wide meeting
- write up draft announcement of SRPC status
Fermi
- announce Fermi Software meeting
- Check for L1 branches
SLAC
nEXO
- attend sims meeting
- populate new SLAC FTP area
- add new user to nexo group at SLAC
- add new users to GH organization
- Update CMake to use new FTP area
  - test at SLAC
  - test at LLNL
  - test on laptop

Week of 4 May 2020

CO
- Alert WG conveners and DC2 analysis czars about upcoming CO discussion concerning validation efforts
- Complete Q2 report
- attend leadership meeting
- attend CS meeting
DESC DM
- Update GitHub workflows for DC2-imsim docker images, to work around disk space issues on GH Actions
- Announce, write agenda, attend, note take for DESC DM meeting
NERSC
- Install new gcr-catalogs 0.16.1
- Help write up draft CFS directory organization
- Test use of desc account in Globus
- Move Buzzard-4 into new CFS shared area
- attend External User Group meeting
CC
- Set up WhenIsGood poll for next CC meeting
- write agenda for next meeting
- Contact SRPC to determine next steps
SLAC
- Weekly SLAC meeting
Fermi
- announce offline s/w meeting
nEXO
- meet with Lisa
- Add nexo-base:current tag and update documentation
- Attend sims meeting
- Look at Fireworks
  - https://www.nersc.gov/assets/Uploads/JainNERSCUserDay2.pdf
  - Debbie's PDF

Week of 27 April 2020

CO
- attend SSims meeting
- inquire about missing w_2019_42 sims on CVMFS
  - https://github.com/airnandez/sw-lsst-eu/issues/3
- request v19 lsst_distrib be pinned on CVMFS
- Test reported issues with ipympl at NERSC and CC
- Upgrade desc-python to include tabulate, flake8, new sacc, healsparse
- Report Missing sims w_2019_42 on CVMFS: https://github.com/airnandez/sw-lsst-eu/issues/4
- Confirm removal of dev installs on CVMFS: https://github.com/airnandez/sw-lsst-eu/issues/5
- Write up quarterly report
DESC DM
- https://jira.lsstcorp.org/browse/DM-24517 Keep an eye on progress
- Finish reorg of Y4/Y5 checkpoint files
  - Check final counts
- Copy Y4/Y5 checkpoint files to tape
- Announce, write agenda, note take and run DESC DM meeting
NERSC
- Transition WG to CFS
  - CL
  - CS
  - LSS
  - PZ
  - SSim
  - TJP
  - WL
- desc-help
  - https://github.com/LSSTDESC/desc-help/issues/12
CC
- Write up message to Membership Committee
- Review scheduled council activities for April/May/June
Fermi
- announce, attend and note take for software meeting
- Merge astro changes from igrf_updates branch into L1 and test in Jenkins
- Release GR including new astro
SLAC
nEXO
- attend sims meeting
- set up new area for shared data files
- Update Cmake build to down data files
- Update CMake to require sniper >= 1.3
- add new nexo user to nexo group at SLAC
- add new nexo user to nexo group at SLAC

Week of 20 Apr 2020

CO
- announce, run, and note-take CO meeting
- Update Zoom infor for all Computing meetings in Calendar and some Confluence pages
DESC DM
- https://jira.lsstcorp.org/browse/DM-24517 Keep an eye on progress
- DESC DM meeting
NERSC
- Attend NUG on 4/22
- desc-help
  - https://github.com/LSSTDESC/desc-help/issues/12
- Finish reorg of Y4/Y5 checkpoint files
  - inquire how to handle Grid duplicates
CC
Fermi
- announce Fermi software meeting
- Help Nicola will additional GR/astro building questions
SLAC
nEXO
- attend sims meeting
- Add new users to GH and provide instructions to apply for accounts at SLAC & LLNL
- Locate new location for data files at SLAC?
  - ticket opened
- Address SNIper filtering to cut down output file size
- Update Cmake build to down data files
- Update CMake to require sniper >= 1.3

Week of 13 Apr 2020

CO
- attend CS meeting
- update sacc in desc-python
- update gcr-catalogs in desc-python
- fix up pip includes in desc-python
DESC DM
- announce, write up agenda, note take DESC DM mtg
NERSC
- crack all y4 y5 files and extract checkpoints
- Store y02-missing instance catalogs to HPSS
- Store cosmoDC2 to HPSS
- Store y02-missing instance catalogs to HPSS
- Store cosmoDC2 to HPSS
CC
- Look over notes for meetings committee
- release April Zoom mtg notes
Fermi
- announce, attend, note take for software meeting
- inquire about status of new astro
- help Nicola get going with developer tools
  - Tag GRcontainerSettings to capture 2018 changes on master
  - Test new Jenkins build
  - Install miniconda2 and add latest fermi-repoman and scons
  - Test command line build of GR
SLAC
nEXO
- attend sims meeting
- Finish setting up new docker/singularity image and documentation
- Update nexo-base image on GH Actions
- Continue trying to get nexo builds with LCG libraries to work

Week of 6 Apr 2020

CO
- attend SSims meeting
- Attend DC2 meeting
- Attend Leadership meeting
- test updated desc-python with fast3tree fix
  - release v0.0.24
- update sacc in desc-python
- fix up pip includes in desc-python
DESC DM
NERSC
- Confirm Y4-Y5 raw transfer was successful and inform Fabio
  - rollback 11 failed jobs and redo transfer to HPSS
  - Confirm success
- Extract Y4 Y5 checkpoint files, organize and push into HPSS
- Contact projecta www owners and tell them about projecta transition
- Contact WG conveners about NERSC projecta => CFS transition
- Start up Science Gateway page: https://confluence.slac.stanford.edu/x/X4Z2E
- Checklist projecta => CFS transition: https://github.com/LSSTDESC/desc-help/issues/11
CC
- Write agenda, send reminder and attend monthly CC meeting
- write up DESC summary
- Help Dominique pinpoint issues cloning DMstack env on CVMFS at CC
SLAC
- Complete Cyber security Training
Fermi
- Announce, attend, and take notes for Offline S/W meeting
nEXO
- attend sims meeting
- Test updated nexo-base docker image removing NEST, going to Py3 and LLNL externals
- setup SLAC with LCG 96python3

Week of 30 Mar 2020

CO
- Write up agenda, run CO meeting
  - discuss desc-python
- Announce new Confluence page for DESC envs on desc-leadership
- write up adding user packages to desc-python
  - test nbodykit
- write up adding user packages to desc-python
  - test nbodykit
- Help CC figure out how to install and maintain desc-python
DESC DM
- Validate Y2-hole FITS files and coordinate transfer to CC
- announce, write agenda and run DESC DM mtg
NERSC
- Meeting with Rollins
- Investigate nbody dependencies
- Transfer y4-y5 raw files to HPSS
- finish setting up testdata_ci_desc_gec repo
  - empty
  - Run2.2i data
CC
- Set March/April meeing, write agenda
Fermi
- Announce, attend, note take Offline S/W meeting
SLAC
nEXO
- attend sims meeting
- New nexo-base docker image with NEST and latest sniper
- Provided updated externals at SLAC
  - find fix for ROOT compilation error with gcc 4.8.5 and LZMA
    - Get SLAC to install lzma-devel
    - move to LCG externals

Week of 23 Mar 2020

CO
- ask Erin to present at next CO meeting
- attend WL meeting and present DC2 update
- Write up confluence page listing available DESC environments and POCs
- Attend CS meeting
DESC DM
- announce, write agenda and run DESC DM mtg
- set up testdata_ci_desc repo with a portion of Run2.2i data
- Help coordinate Y4/Y5 transfer from CC with Fabio
- Work out updating the SQLite tracking DB for Y4/Y5 and Y2 hole
- Help coordinate Y2 hole transfer to CC
- Write up wiki describing how to set up a DM Gen2 repo
NERSC
- Ask about jupyter widgets
- release new conda-forge based desc-python
- Test new desc-python and adding user packages
CC
- ask meetings committee about data retention
- plan next April CC meeting
SLAC
Fermi
nEXO
- attend sim meeting
- Merge updates to lightmapsvc
- Help new nexo user set up and use docker and latest nexo-offline

Week of 16 Mar 2020

CO
- Prepare slides for Monday's WL meeting
DESC DM
- Attend DM Alg Workshop remotely Tues - Thur 8:45 AM - 2:30 PM PT
- Validate Y4-WFD fits files and request transfer to CC
- Update DC2 Run2.2i Confluence doc for DR3
NERSC
- Move Y4 WFD checkpoint files to HPSS
- Copy Y2 Hole instance catalogs to HPSS
- Respond to Rollins questions about use of key/value API within jupyter notebooks
- deploy new desc-stack production bersion vased on v19
CC
- Update newsletter Feb/March summary
SLAC
- Complete Annual Security Training
Fermi
nEXO
- Attend simulation meeting
- Help Ako with Sniper Questions
- Update LightMapSvc to fix example algorithm to allow setup once during initialize
- Update LightMapSvc to retrieve data from TH2D
- Complete LLNL Security Training

Week of 9 Mar 2020

CO
- Announce, write agenda, note take and CO meeting
DESC DM
- Announce, write agenda, run, note take DESC DM meeting
NERSC
- Pull Y2 WFD instance catalogs from tape
CC
SLAC
- Attend Data Science Training
Fermi
nEXO
- Attend simulation meeting
- Chat with Lisa about Confluence, JIRA, GitHub, etc
- Open SLAC ticket to help nexo user set default shell to bash
- Test git submodules on nexo-offline

Week of 2 Mar 2020

CO
- Finish CI slides
- Present CI work at DESC seminar
- attend CS meeting
- update desc-dc2-dm-data repo with DR3 directory
- start gcr-catalogs updates for object and meta catalogs
- Update desc-help README to provide pointers to other help faciilties
DESC DM
- announce, write up agenda, run, and note take for DESC DM mtg
NERSC
- Clear off Y1 and Y2 chkpoints, freeing 146 TB for y4-wfd transfer from scratch
- Prepare subset of DR4 on CSCRATCH for general DESC release
- Request ownership transfer of DR3 drp data
- Update Confluence doc for NERSC account
- Test ability of DR3 drp data on CFS to run on batch - fails, same locking issue as projecta
CC
SLAC
Fermi
- attend software week sessions
- Test Jenkins builds of GR
  - Report problems with GR builds to Brian
  - retest Jenkins manual triggering of builds
  - Test binary tarball at SLAC - seems to work
nEXO
- attend software meeting
- Inquire about transfer speeds between IHEP and SLAC
- Provide skeleton light map service

Week of of 24 Feb 2020

CO
- Prepare slide for Computing Infrastructure Talk at DESC Seminar
- attend SSIM meeting
DESC DM
- Attend DC2 meeting
NERSC
- Set up space for DR3 transfer from CC
- update desc-stack to resolve: https://github.com/LSSTDESC/desc-help/issues/2
- Open NERSC ticket concerning users with trouble starting up jupyter
CC
- Attend and note take for the meeting
- Update Trello with CC activities
- Write up DESC Summary of Feb mtg
- Write up Feb DESC newsletter submission
- Fix up CC Confluece area containing Council Minutes - past minutes were spread between 2 separate pages.
SLAC
Fermi
nEXO
- attend Tuesday software meeting
- Inquire at SLAC to use gridFTP for data transfer
- Work with SLAC to gain CVMFS access to IHEP filesystem

Week of 17 Feb 2020

CO
- write agenda, note take for CO meeting
DESC DM
- write agenda, run meeting,note take
NERSC
- add graphviz to desc-python
- Store Y3-wfd checkpoint files on HPSS
CC
- Start up agenda page
- Poll for meeting time
- Follow up on
  - PubBoard Rotation
  - SBPC
SLAC
Fermi
- announce, note take for software meeting
nEXO
- help track down memory leak
- attend sims meeting
- install updated external libs at SLAC
- update cmake install to handle downloading of ancillary files

Week of 10 Feb 2020

CO
- attend DESC leadership mtg and take notes
- announce new desc-help github repo
- attend and note take for CS mtg
DESC DM
NERSC
- Copy Y4 instance catalogs back off tape to CSCRATCH
- Set up 1 TB area for PLASTICC
  - submit NERSC ticket to sort out how to set quota
- Refresh NERSC Getting Started Confluence page with new NERSC links
- install w_2020_06 in shifter for Michael
- address ipyml issue in jupyter
- contact NERSC about htar compression
CC
- attend SRPC - CC meeting and take notes
SLAC
nEXO
- Look over commit history on nexo-offline branches
- add outside collaborator to one nexo repo

Week of 3 Feb 2020

CO
- Generate new stack-sims-cat using w_2019_42 and adding in pyarrow for Bryce & Joanne
- Attend SSim mtg
DESC DM
- Gen3 meeting
NERSC
- transfer y02 instCat to hpss
- transfer y05 instCat to hpss
- transfer y06 instCat to hpss
- Request increase to descim CSCRATCH quota
- Inquire about new user LSST group membership
- Inquire about PI notification concerning users over job allocation
- Transfer checkpoint files to HPSS
  - Y1
  - Y2
  - Y3
CC
- set up meeting with SRPC
SLAC
- Attend group meeting
Fermi
- announce, attend, notetake Software mtg
nexo
- attend software meeting
- Write MC filter Alg
  - PR
- Coordinate SLAC adding JuNO path to CVMFS
- Meet with Lisa
- Add new SNiPER service to read in ROOT TH2 LightMap

Week of 27 Jan 2020

CO
- write up agenda, run, and note take during CO meeting
- Set up desc-help repo and explore issue templates
DESC DM
- write up agenda, note take during DESC DM & DA meeting
- validate Y3 sim data, update tracking DB and request transfer to CC
NERSC
- Consider moving projecta content to CFS
- Look over CFS content from project area
CC
- meet with Yao to discuss to do list
Fermi
nEXO
- attend software meeting

Week of 20 Jan 2020

CO
- Set up workflow for gcr-catalogs to create desc-stack:test and desc-python:test
- Set up workflow to move desc-stack:prod to desc-stack:old
DESC DM
- Set up mtg with Ben, Tom, Jim, Adrian
- Pull out y4 instance catalogs for James P to transfer to Grid
NERSC
CC
- Organize, pick up, deliver food for cc-mgmt meeting
- Set up mtg with Yao & Eric
Fermi
- announced cancellation of Software mtg this week
nEXO
- Update NEST to latest master in nexo-base:latest
  - testing failed due to missing valgrind which seems to be required

Week of 13 Jan 2020

CO
- Provide GCRCatalog workflow triggered off master to produce desc-stack and desc-python test envs
- Set up new desc-stack repo in LSSTDESC using v19.0.0
- Introduce Docker build of desc-python in LSSTDESC
- Update desc-dc2-dm-data repo with all Run2.1i and Run2.2i data locations
- Update desc-dc2-dm-data repo with all Run2.1i and Run2.2i data locations
DESC DM
- Validate straggler y1-wfd and y2-wfd visits, update tracking DB and request transfer
- Write up slides, attend WL meeting and provide DESC processing update
NERSC
- Prepare new CFS area for DESC with directories and permissions
  - Inquire about area ownership: https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=7fb44483db1600907d2a300f7c961932%26sysparm_view=ess
- Refresh Run2.1i CSCRATCH copy
- Refresh Run1.2i CSCRATCH copy
- Address inode issue for descdm CSCRATCH
- Start moving agn_ckpts to HPSS
  - inquire about stalled HPSS jobs: https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=586cd487db5600907d2a300f7c961926%26sysparm_view=ess
  - y1-wfd
  - y2-wfd
- Finish moving instance catalogs to HPSS
  - y02
  - y05
  - y06
DESC CC
- Update CC plenary slides
- Handle food for CC-Mgmt meeting
- Remind desc members to review draft SRP and provide comments
- Help plan CC Town Hall
  - zoom recording?
Fermi
- Announce, note take and attend weekly software meeting
nEXO
- attend software meeting
- Provide fresh externals installation at SLAC for latest nexo-offline
- Meet with Samuele to discuss skeleton recon SNiPER alg
- Write up draft organization for charge and light reconstruction

Week of 6 Jan 2020

CO
- Write agenda, announce, and notetake for CO meeting
- Provide Debbie some advance dates of DESC needs
- Q1 report
- survey paper validation section
- Prepare for Collab Mtg
- Follow up on DDF checklist
DESC DM
- Validate y2-wfd and update sqlite tracking db and request transfer to CC
- Announce, write agenda, attend and note take for DESC DM meeting
- Transfer tract 3828 and 3829 to lsst-dev
  - and ancillary data including CALIB, ref_cats
- Free up space in descim CSCRATCH
  - y4
  - y3
  - y1
  - y2
  - y5
- Free up space in projecta
  - agn-checkpoints for y1-wfd
NERSC
- Refresh RUn1.2p
- Refresh Run1.2i
- Cron job for CSCRATCH?
Collaboration Council
- Start preparing CC update for Collaboration Mtg
- Help plan TownHall
Fermi
- Announce, attend, and note take offline software meeting
nEXO
- Attend software meeting
- Read, respond, and provide Fermi's Gaudi experience for Vladimir's project overview

Week of 16 Dec 2019

CO
- Section 7.2 DC2 survey paper (cross-validation)
- Attend Wednesday Leadership Mtg
DESC DM
- Ingest Run2.1i reference catalog using v19.0.0 and obs_lsst dc2/run2.2 branch
NERSC
- Update desc-stack, making sure it pin python 3.7
- Finish SPIN implementation for running integration tests using CVMFS installations
- Help Anze figure out how to use emacs after loading desc-python
Fermi
- Create any necessary JIRAs for upcoming GR astro update
nEXO
- attend Sunday evening telecon with SNiPER representatives

Week of 9 Dec 2019

CO
- Attend SSim meeting
DESC DM
- write up agenda, run, and note take DESC DM meeting
- validate y1-wfd fits and centroids
  - Request transfer to CC
NERSC
- Attend SPIN Hack day
- Attend NUG: https://www.nersc.gov/users/NUG/teleconferences/nug-webinar-dec-12-2019/
- Test gcr-catalogs uninstall and clean up old catalogs files in desc-python
  - update desc-python
  - updated desc-stack
Fermi
nEXO
- attend 3 hour (until had to go) workflow meeting
- Attend software meeting

Week of 2 Dec 2019

CO
- Announce, write agenda, run, and note take CO meeting
DESC DM
- Announce and write up agenda for DESC DM meeting
- Update setup script for IPP pipelines
- Update obs_lsst dc2/run2.2 branch for w_2019_47
NERSC
- attend SPIN training Thus and Friday afternoons
Fermi
- announce software meeting
nEXO
- attend software meeting
- Read Vladimir's work plan and comment

Week of 18 Nov 2019

CO
- prepare list of current desc-python packages to circulate, investigate CC's conda env, and open GH issue
- Consult conveners concerning who should offer GCR analysis commentary at next mtg
DESC DM
- Announce, write up agenda, note take and run DESC DM meeting
- Provide Tom and Ben an updated dc2-run2.1i docker image with astropy 3.2.3
NERSC
- Deal with Michael's questions about building against CVMFS using recent stacks
  - Upload and test recent LSSTSQRE docker image into Shifter at NERSC and worked out building
  - Tested CVMFS at CC
  - Work with DM to determine how to set ABI flags for scons builds of local packages
- Dealing with private DM NERSC questions: Bruno, Humna
- Continue to work on mrau's inode use - down from 96% to 69%
- Move more files to HPSS
- Update desc-python-dev to include CLMM and cluster toolkit.
  - indicated that we need release tags on both before I add this to desc-python
  - Note the need for GSL and adding gsl-config to PATH
Fermi
- Announce, attend and note take for software meeting
nEXO
- attend software/sims meeting
- rebase Brian's PR and merge because no one knows how to use git on nexo
- add doxyfile for Tony to test Doxygen generation in Jenkins
- update nexo-base env to include new NEST master
- - Add GitHub actions workflow
  - Release new docker image

Week of 11 Nov 2019

CO
- PR, tag, and create docker image for lsstdesc/dc2-imsim:Run2.2i-production-v1
- attend data access meeting
DESC DM
- Attend DM bootcamp Tues - Thur 2-5 PM ET
NERSC
- Deal with projecta inode quota which is currently at 96%
- continue moving files to HPSS
Fermi
- announce and note take for offline s/w mtg
nEXO
- NEST PR
  - test build at SLAC
  - Open new GH issue for cmake build failure

Week of 4 Nov 2019

CO
- write agenda, announce and note take for CO mtg
- Update Nov, Dec, Jan DESC calendar for upcoming holidays
- Prepare new stack-sims-cat docker image for instance catalog generation
DESC DM
- ~~write agenda, announce and note take DESC DM meeting~~
NERSC
- Install pyCCL 2.0.1 in desc-stack and desc-python
- Meet with Debbie to go through NERSC user survey
- Continue moving data onto HPSS
Fermi
- announce mtg
nEXO
- attend software meeting and provide update
- Help Caio with SLAC install
- Deal with doc PR

Week of 25 Oct 2019

CO
- attend Data Access meeting and note take
- attend CS meeting
- write draft DDF checklist
- Transfer Grid data from CSCRATCH to projecta space
- continue transferring data to HPSS from projecta
- Generate new sims docker containers as needed
  - imSim v0.6.1
DESC DM
- process ALCF, Cori, and Grid validation test data, through makeFpSummary
NERSC
Fermi
- announce, attend, and note take for software meeting
nEXO
- Continue answering C++ and git questions
- setup external library area at SLAC using same versions as LLNL
- attend software meeting

Week of 14 Oct 2019

CO
- Write up Agenda and run and note take for CO mtg
- Attend Data Access meeting
- Attend SRM milestones meeting
DESC DM
- Edit Parsl DRP planning doc
- setup up empty Run2.1.1i repo for testing on CSCRATCH and HPSS
- Transfer NCSA Run2.1.1i processing to NERSC
NERSC
- Transfer more data to HPSS
Fermi
nEXO
- Handle C++ questions
- Start to Investigate strax

Week of 7 Oct 2019

CO
- Attend Grid meeting
- Attend CS meeting
DESC DM
- Announce, write up agenda, note take, and run DESC DM meeting
- Help Tom out with DM questions
NERSC
- Help Bela to run shifter images in batch
- Start transferring more data to HPSS
Fermi
- Announce, attend, and note take for software mtg
- Handle M.E.'s questions concerning new GR release build
- Remove Charlotte's name from the workbook where necessary, per her request
nEXO
- Attend software meeting
- Deal with C++ questions
- Test ROOT writing performance to address Soud's inquiries

Week of 30 Sept 2019

CO
DESC DM
- Oct 2nd meeting agenda and note taking
- Meet with Ben and Tom to discuss Parsl
- Discuss Data Transfer with Fabio, Antonio, Dominique, Tom
- Review Johann's PR from dc2/run2.1
- Create new obs_lsst branch for dc2/run2.2
- Finish writing Quarterly Report
NERSC
Fermi
- attend, note take for offline software
- opt build for GR 20-10-04-gr06
nEXO
- software meeting

Week of 23 Sept 2019

CO
- Prepare agenda and run CO meeting
DESC DM
- Finish no sky correction Run2.1.1i process and make it available to Jim
- Update weekly 39 docker image to remove current tag
NERSC
- Migrate more data to hpss
- Prepare to copy instance catalogs back off tape
Fermi
- Build new L1 GR
- See that Jenkins is fixed to build L1
- offline software meeting
nEXO
- attend sims meeting
- prep singluarity image for OSG
- update docker image with new externals

Week of 16 Sept 2019

DESC DM
- instance catalogs for Run2.1i do not need to be regeneratd? how does that affect what is stored to HPSS?
- take a look at throughputs tags in docker image
- Generate new imsim Docker image with new Galsim
- Set up agenda and run DESC DM portion of Wednesday meeting
- set up docker image for use in SRS pipeline
- validate DDF data from Run2.1i y5
CO
- Make Confluence pages consistent and update content
- Update DESC Calendar
- SChedule Data Transfer Discussion
  - draft agenda: Sep 30 10 AM PT
NERSC
- set up instance catalog generation software
- NaMaster in desc-python
- update for GCR-catalogs
- Help Bruno & Bob access Run2.1i data
- Coordinate with NERSC to see if we can use Globus transfers into HPSS and transfer owernship
- Continue copying instance catalogs and Run2.1i data to HPSS to free space

Week of 3 Sept 2019

DM-DC2
- finish processing 4 tracts of Run2.1.1i through multiband
- run merge_tract_cat to produce HDF files on all the 14 tracts of Run2.1.1i
- rerun processing with skyCorrection
- finish documenting running at NERSC
- Converse with Ben as questions arise as he works thorugh implementing DRP in parsl
DESC
- Check on imsim Docker image and the throughputs tag
- Install NaMaster in docker image
NERSC
- Check on CSCRATCH before outage and transfer to tape
  - Back up Run2.1.1i 4 tract nersc processing
  - recopy RUn1.2i and reapply symlink fix
  - Create back up of Run2.1.1i, Run2.1i on projecta for users to use during outage
- Inquire about Jupyter inaccessiability
SLAC
- install DMstack v18.1.0 for IR2
  Jira
  server SLAC National Accelerator Laboratory
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1433
Fermi
- Announce, run, and note take Fermi mtg Sept 4th
- Tag L1 fitsGen after rebasing with master and create a new L1 GR tag
nEXO
- write up tutorial for python access
- retrieve all requirements for OSG (onhold)
- meet with Lisa
- merge Emma's WT branch into issue_31 after resolving conflicts
- Present "How to create a Data Object and write it out in SNiPER"
- attend nEXO software/sim meeting

PCW

Catch up with Scott and Chris about the throughputs tags

Week of Aug 5 2019

DM-DC2
DESC
- Retrieve Y2 DR1B instance catalogs from HPSS
- Attend DC2 validation/next steps meeting
- Document
NERSC
- Poke NERSC about file system status
- Demonstrate running SRS workflow at NERSC
- Continue clearing projecta space
SLAC
Fermi
- announce, attend and note take for Fermi Offline mtg
nexo
- attend sims/software meetign
- present update on nexo-offline status
- Help update doc on Docker
- contact Tony about setting up Jenkins builds
- Work through Save All Events problem in nexo-offline
- finish testing cmake in docker
- Look at pySniper algorithm

Week of July 29 2019

DM-DC2
- announce, write agenda and note take for weekly meeting
DESC
- attend DC2 meeting
NERSC
- Recover from Cori upgrade
  - Update desc-python to use gcc 7.3.0
  - Adjust setup_current_stack and setup_current_sims to use new gcc 7.3.0 build of w_2019_19 on CVMFS
    - test new gcc 7.3.0 build
SLAC
Fermi
nEXO
- attend sims/software meeting
- Run nexo-offline and nEXO_MC using TPCVesselU.mac
- attend LLBL cmke training remotely

Week of July 22 2019

DM-DC2
- Open JIRA concerning throughputs and semantic tagging
  - https://jira.lsstcorp.org/browse/SIM-2620
- Ask Michael about identifying a tract to retain on projecta from instance cats through object catalogs
- Retrieve Y6 instance catalogs from HPSS for James P for use in grid processing
- Write up Draft Confluence page for DESC DM Change Requests
  - DESC Change Requests for DM
DESC
- add NaMaster to desc-stack
- consider adding sacc to desc-stack
- add cl-classmod environment to desc-stack? or some subset.. or update the classmod env
NERSC
- install new gcr-catalogs 0.14.1 in desc-python & desc-stack
- Help Humna with her ongoing desc-stack problems
  - https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=48035aeadbbe7300200f7d321f961962%26sysparm_view=ess
- Continue moving WFD instance catalogs to HPSS
SLAC
Fermi
- Announce, attend, and note-take for Fermi Offline Meeting
nEXO
- attend software/sims meeting

week of DESC Meeting

DM-DC2
- Help set up Tom to parsl sfd
- New confluence page for CCB DESC requests
Python
- NaMaster
- sacc
- DIA kernel
NERSC
- Install gcr-catalogs 0.13.1 and 0.14.0 into desc-python and desc-nersc for Hack Day
- Check on ability to use later gcc version with CVMFS installation
- Doc use of jupyter compute nodes
- Fix up DESCPYTHONPATH
- doc use of batch for desc-stack
- instance catalog from tape Y6

Week of July 1 - 12 2019

DM-DC2
- Write agenda, run and note take DM-DC2 meeting for both July 1 & 8th
DESC
NERSC
- update lsstdesc/nersc to remove PYTHONSTARTUP for desc-stack
- Document the use of DESCPYTHONPATH
SLAC
Fermi
nEXO

Week of June 24 2019

DM-DC2
- Listen to recording of weekly meeting and transcribe notes
- Plan any DM-DC2 sessions for collaboration meeting
- Document new DR naming convention
DESC
- Deal with docker removal from lsst-dc10, find another place to build
NERSC
- Continue clearing out the instance catalogs to HPSS
- Quality check Y5 simulated data and update the tracking DB
SLAC
Fermi
- announce, attend, and note take Fermi Offline Software meeting
nEXO
- attend collaboration meeting at Stony Brook
- add Cards to nexo-offline master and u/heather999/issue_31
- Testing running docker image with nEXO_MC
- Run nexo-offline and nexo_MC TPC jobs for regression testing
- Take a look at historgrams generated
- Keep track of Soud's progress with Clustering and Reconstruction port

Week of June 17 2019

DM-DC2
- https://jira.lsstcorp.org/browse/DM-19985
DESC
- Prepare new w_2019_23 stack sims docker image for imsim group
NERSC
- Continue moving instance catalogs to HPSS
- Alert IN2P3 when there is enough space for y1-y2 calexp and coadds
SLAC
- attend stand-up meeting
Fermi
- announce, take notes, attend offline software meeting
nEXO
- Complete new nexo-base docker image include NEST2.0
- write up slides for collaboration meeting

Week of June 10 2019

DM-DC2
- Write up agenda and run DM-DC2 meeting
- quality check y3-wfd sim data and update tracking db
- Help coordinate transfer for new calexp-v1 to NERSC from IN2P3
- Create branch and PR for forcedPhotCcd_metadata template https://github.com/lsst/obs_lsst/pull/116
DESC
- attend data access meeting
- Prepare updated desc-stack-dev and install at NERSC
- Test desc-stack-dev at NERSC using DC2-analysis notebooks, resulting in the following issues
  - https://github.com/LSSTDESC/gcr-catalogs/issues/327
  - https://github.com/LSSTDESC/DC2-analysis/issues/87
  - https://github.com/LSSTDESC/DC2-analysis/issues/86
  - https://github.com/LSSTDESC/DC2-analysis/issues/85
  - additionally identified the need to add seaborn and extinction ot the docker image
NERSC
- Consider increasing desc CSCRATCH quota
- Copy and remove DC1 deepCoadd to tape from projecta
- Check over y1-y3 instance catalogs on HPSS
  - remove y1-y3 from projecta
- move y7-10 to HPSS
- Help demonstrate cvmfs problems at NERSC
SLAC
- attend group meeting
Fermi
- announce, note take and attend software meeting
nEXO
- attend software meeting
- report on nexo-offline status at software meeting
- Update nexo-env to handle NEST2.0
- Create new docker base image with NEST2.0
- send Samuele/Lisa slide for collab meeting
- check with Tony about status of Jenkins builds

Week of May 28 2019

DM-DC2
- https://github.com/LSSTDESC/ImageProcessingPipelines/issues/92
- Coordinate transfer of 6 months DM processed data from IN2P3
- Set up 6 month data at NERSC under CSCRATCH with the appropriate CALIB, ref_cats
- Make Run1.2x repos readable by later versions of the DMstack
  - https://lsstc.slack.com/archives/C978LTJGN/p1557771460413100
DESC
- Create stack-sims:w_2019_19-sims_w_2019_19 image for use at NERSC
NERSC
- create new docker image based on w_2019_19 plus all desc python packages
  - include stile
  - install at NERSC as desc-stack-dev
  - Update desc-stack-prod
- Update BF Kernel cp_pipe installation for w_2019_19 after Tom's reports that the older installation using w_2019_17 no longer worked
- continue transferring data into NERSC
  - https://github.com/LSSTDESC/DC2-production/issues/343
  - DC2 Run2.1i instance catalogs
  - Remove DC1/DC1-imsim/dithered and undithered
  - Remove DC1/astrometry
  - Start cleaning DC1/DM/DC1-imsim-dithered
    - icExp contains just temp files
    - compare random calexp and deepCoadd fits files using fdiff with the copy in HPSS
SLAC
- attend group stand up meeting
Fermi
nEXO
- attend software meeting
- Get some more info about Simulation/DetSim/Nest
- Add

Week of May 20 2019

DM-DC2
- Write agenda, take notes, and run DM-DC2 meeting
- Welcome Dino and send along some getting started links
DESC
- Open up a new ticket for Darren White
- Attend DC2 meeting and lead discussion concerning what data needs to be available for the analysis groups
NERSC
- Continue transferring data into HPSS
- Open GitHub issue to discuss what can be removed from projecta
SLAC
Fermi
- Announce and note take for offline meeting
nEXO
- attend sims meeting
- setup Soud with u/heather999/issue_31 branch at SLAC
- contact Tony about getting some Jenkins builds going before collab meeting
- Contact Pere and Gabriele C about NEST2 linking errors against G4
- Poke Jason and Samuele about nEXO_MC PR for NEST

Week of May 13 2019

DM-DC2
- Write agenda, take notes,and run DM-DC2 meeting
- Ingest new ref catalog and deliver to IN2P3
- update obs_lsst configs according to Slack conversation: https://lsstc.slack.com/archives/C978LTJGN/p1557935315430600
- inquire about using --show history https://lsstc.slack.com/archives/C2K97E3L5/p1557940644050600
  - https://jira.lsstcorp.org/browse/DM-19804
DESC
- attend data access mtg
NERSC
- Move files from projecta to HPSS to reduce disk use on projecta
  - DC2_ImSim/Run2.0i
  - DC2_ImSim_UKTest
  - Twinkles1
- ask Richard to increase "desc" HPSS quota
SLAC
- install parsl on IR2 in v17_gcc7 env
Fermi
nEXO
- attend software meeting

Week of May 6 2019

DM-DC2
- Write agenda, take notes, and run DM-DC2 meeting
- Update visit DB to mark files and visits tables as quality_controlled and quality_completed, respectively
- Create updated CALIB products tarball including BF kernels
- Update Antonio's visit DB to note that all Y2-WFD visits and files have been quality controlled.
DESC
- attend CI meeting
- attend DC2 meeting
  - review Dominique's slides for DC2 mtg
- attend 3x2 mtg
  - help Francois access Run1.2i force photometry data - only to learn that newer DMstacks cannot deal with the older repos.
- Deal with user questions on #desc-nersc
- Start uploading DC2 data into HPSS to free projecta space
NERSC
- install stile
- Install new python environment compatible with recent DMstacks
- Create new docker image for sims w_2019_17 which will be the base of new jupyter kernels
Fermi
- announce, take notes and attend Fermi offline s/w mtg
nEXO
- attend sims meeting
- Clean up merge of nEXO_MC and nexo-offline
  - compile on SLAC with gcc 7.3 and LCG 94a
  - compile with gcc 4.8.5
  - run some tests
  - submit PR

Week of Apr 29 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Check y2 Run2.1i data for incomplete files
- Ferry sky correction configs into master of lsst/obs_lsst
  - rebase, update comments, squash commits
- Update dc2/run2.1 lsst/obs_lsst branch for use at IN2P3
- Rerun skyCorrection for eyeballer visits
  - Due to decision to skip skyCorrection, this attempt was abandoned
- Generate new set of BF kernels
  - first attempt failed
DESC
- Poked DM about weekly release note page which wasn't working: https://lsst-web.ncsa.illinois.edu/~swinbank/changelog_weekly.html
NERSC
- Update setup scripts at NERSC to use CVMFS
- Update confluence documentation for NERSC software installations
- Install updated cp_pipe for BF kernel generation
- Investigate inode use on /global/project/projectdirs/m1727
  - Poke owners of files
SLAC/Camera
Fermi
- Inquire about software meeting
nEXO
- Test CMake build of SNiPER
- Test CMake build of nexo-offline
- Test CMT build of nexo-offline against CMake build of SNiPER
- Set up LCG 94a at SLAC

Week of Apr 15 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Deliver CALIB to IN2P3
- Check 2y Run2.1i data for incomplete files
- test updated sky configs and running makeFpSummary
DESC
NERSC
- Clear out Javier's copy of DC1 undithered data
- Start transfer of Run1.1 and Run1.2 data into HPSS
- Update gcr-catalog in jupyter kernels and underlying shifter image
SLAC/Camera
Fermi
- announce, attend, and note take for Fermi software meeting
- Help Bill get set up with GR at SLAC for debugging
nEXO
- attend software meeting
- Review Git and Code review docs

Week of Apr 8 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Regnerate BF flats with w_2019_14
- update config and ingest new Reference catalog
- Rerun constructDark with doCrosstalkBeforeAssemble=False
- Rerun constructFlat with doCrosstalkBeforeAsseble=False
- Set up updated CALIB products using w_2019_14 and setting isr.doCrosstalkBeforeAssemble=False
- Update obs_lsst with isr.doCrosstalkBeforeAssemble=False https://jira.lsstcorp.org/browse/DM-19218
- Try out sky construction using Jim's sky frames
- Try sky construction again with flats available
- Get sky configs updates into obs_lsst
- Go through outstanding LSSTDESC/ImageProcessingPipelines issues and close where possible
- Work with Johann to clean up the IPP/workflows/srs/dm_configs area and ultimately deleted it
- Reviewed the reworked SFD pipeline SRS workflow with Johann
DESC
- Attend and present at CI meeting
- Attend Rachel's Computing WG meeting
NERSC
- Get HPSS Project area set up
- Determine HPSS quota request
- Figure out Katrin's HPSS use
- Assess DESC user HPSS useage
- set up DM repo in CSCRATCH/descdm/DC2/Run2.1i area
- Point James Perry to Intel Python: https://docs.nersc.gov/programming/high-level-environments/python/
SLAC/Camera
Fermi
- announce, attend, and note take for Fermi software meeting
nEXO
- attend software meeting and present update
- Try out LCG view installation at SLAC and update nEXO wiki doc
- Complete LLNL security training
- Contact LLNL to get my account activated

Week of Apr 1 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Script to find all potential incomplete Run2.1i visits in the first production run
- Write up todo list of DRP processing of Run2.1i-y2wfd
- Schedule mtg with Katrin, Jim, Rachel, Dominique, Fabio, Johann concerning Run2.1i processing
- meeting with Katrin,Jim, Rachel, Dominique, Fabio about Run2.1i processing
DESC
- Q2 report
- Attend Data access meeting
NERSC
- Arrange Run1.1p, 1.2p, 1.2i, 2.0i data and tabulate disk use
- Prepare HPSS disk request
- Put in HPSS project area request
- Set up new conda area based on updated DMstack dependencies
SLAC/Camera
Fermi
- announce, attend, and note take for Fermi Software Meeting
nEXO
- Attend sims meeting
- merge https://github.com/nEXO-collaboration/nEXO_MC/pull/10
- send Alex some details about SNiPER wrapping of python
- Ask SNiPER developers about XmlObjDesc
- Improve docker images
- Update SLAC instllation
- try out CMake

Week of Mar 25 2019

DM-DC2
- write agenda, take notes, and run DM-DC2 mtg
- Generate flats calibration products for Run2.1i
  - regenerate R22 S11 using updated configs
  - test run1.2i R22 S11 flat raw using same stack w_2019_09
  - test run2.1i 80k flats
- generate subset of bf flats
- Ask Johann for metacal summary
- Create RFC or Confluence page to request DM support for DC2: https://lsstc.slack.com/archives/C978LTJGN/p1553708211293400
DESC
- send out announcement concerning NERSC jupyter
NERSC
- Set up Yadu with a copy of ci_hsc at NERSC: https://github.com/lsst/ci_hsc
- check on mcnanna usage
- Continue arranging the /global/projecta/projectdirs/lsst/production area for Run1.2p and Run1.2i
- Pull out DC1 undithered calexp from HPSS for Javier
  - deal with desc account being over quota in CSCRATCH
SLAC/Camera
- inquire about BNL computing status
- SISA Information Security I
Fermi
- announce, attend, and note take for Fermi Software Meeting
nEXO
- attend Tuesday software meeting
- summarize decision concerning nexo-env https://github.com/nEXO-collaboration/nexo-env/issues/9
  - close
- merge https://github.com/nEXO-collaboration/nEXO_MC/pull/13
- merge https://github.com/nEXO-collaboration/nEXO_MC/pull/10
- send Alex some details about SNiPER wrapping of python
- Ask SNiPER developers about XmlObjDesc
- Improve docker images
- Update SLAC instllation
- try out CMake

Week of Mar 18 2019

DM-DC2
- write agenda, take notes, and attend DM-DC2 mtg
- Fix bf flat headers
- https://jira.lsstcorp.org/browse/DM-18551
- https://jira.lsstcorp.org/browse/DM-18575
- https://jira.lsstcorp.org/browse/RFC-583
- Install latest updated cp_pipe master from Craig
- Take a look at the BF kernels in Michael's notebook
  - https://lsstc.slack.com/archives/C978LTJGN/p1553090956192800?thread_ts=1553046434.190200&cid=C978LTJGN
DESC
- Attend Data access TF meeting
- Build and release new lsstdesc/stack-sims docker image w_2019_10_1 and patch w_2019_10 due to shebangtron
- Finish CI Q1 report
NERSC
- update jupyter doc https://github.com/LSSTDESC/nersc/issues/9
- Request descim account access for Ben Clifford
- Continue cleaning up data storage on projecta
SLAC/Camera
- Completely shut down elog bot on SLACK
- Update glastsys password and escrow
Fermi
- attend and note take offline software meeting
nEXO
- attend sims meeting
- attempt to set up 2009 mac with ElCapitan
  - attempts to upgrade to later MacOS failed
  - Attempt to install a current ROOT failed
- Set up VM with MacOS

Week of Mar 11 2019

DM-DC2
- Write agenda, take notes, and attend DM-DC2 mtg
- Generate BF flats
  - tarball and store on projecta
- set up CSCRATCH area with CALIB, ref_cats and bf flats
DESC
- Attend and present at CI meeting
- Attend DC2 meeting
- update jupyter doc for jupyter-dev changes
  - test after maintenance is completed on 3/15
- Build new w.2019.10 sims docker image
NERSC
- Help Anita with sncosmo and sims jupyter kernels
  https://lsstc.slack.com/archives/C2U2K05JR/p1551995963014800
- install parsl 0.7.2 in desc-python
- Deal with ongoing permission issues due to mixed ownership in /global/projecta/projectdirs/lsst/production/DC2_ImSim/Run2.1i/instCat
SLAC/Camera
- shut down eLog slackbot
- Install DMstack v17 + a few additional python packages and setup constraint file
  - Jira
    server SLAC National Accelerator Laboratory
    serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
    key LSSTTD-1387
Fermi
- Find SLAC anaconda 3 installation
  - send note to SLAC unix-admin about public installation
nEXO
- attend and apparently run software meeting

Week of Mar 4 2019

DM-DC2
- Write agenda, take notes, and attend DM-DC2 meeting
- Github issue for tracking handling of DC2 data
- Take another look at running SingleFrameDriver at NERSC
- Generate darks and biases for Run2.1i
  - first get cvmfs caching fixed at NERSC
    https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=2abc3055db8c33407cf774608c961912%26sysparm_view=ess
- BF Kernel Generation
  - test updated configs
    https://github.com/LSSTDESC/ImageProcessingPipelines/issues/82#issuecomment-469824632
  - send Tom some instructions
DESC
NERSC
- Help mishak with AUP
- add parsl 0.7 to desc-stack
- CCL installation instructions?
  - https://github.com/LSSTDESC/CCL/issues/587#issuecomment-468841051
SLAC/Camera
- Complete SLAC Computing Security Training
Fermi
- s/w mtg and note taking
nEXO
- attend sims meeting
- contact Sniper developers about how to best embed python in Sniper algs
- Write up to do list based on software workshop notes
- Check on Venkatesh and docker set up

Week of Feb 18, 2019

DM-DC2
- Create workflow for imsim biases and darks
- run test through patch of Run1.2i
  - ingest
  - sfd
  - coadd
  - multiband
DESC
- CI quarterly report
- set up Parsl discussion with Jim/Katrin/Tom/Yadu/Ben/Antonio
NERSC
- check into mweiser problems accessing lsst resources
- Inquire about globus and collab use
- Inquire about HPSS project directory
- add new ccl and postgressql to docker and jupyter kernels
- Finish copying DC2 data to projecta
- request increase in inode quota of descdm $CSCRATCH
- Clean up $CSCRATCH descdm
- update desc-python and desc-stack with new gcr-catalogs release in time for collaboration mtg
- Inform NERSC of Hack Day and planned Tuesday tutorials using jupyter-dev
SLAC/Camera
- Data portal updates for NCRs
Fermi
nEXO
- Test cmake
- attend sims meeting
- release nexo-ofline and install at SLAC and create docker image
- meet with Lisa about workshop agenda
- Demo Doxygen for Lisa
- meet with Lisa to go over git branches on nexo_MC
- PR updates to Guofu's optical photon changes

Week of Feb 11, 2019

DM-DC2
- Prepare agenda, attend and note take for the DM-DC2 mtg
- Set Tom up with Run1.2i calibration products
- Create workflow for imsim biases and darks
- Contact Scott about adding obs_lsst to lsst_sims
- Coordinate merge of https://jira.lsstcorp.org/browse/DM-17875
- Meeting with Jim & Katrin
  - Discuss disk space needs for Run2.1i (plan to add 1 Pb per year for now)
    - How much data will come back from IN2P3?
  - Where will Run2.1i processing occur?
DESC
- CI quarterly report
NERSC
- Install new CCL release
- add new ccl and postgressql lib to docker & jupyter kernels
- Finish copying the DC2 data into projecta and prepare count of all contents
- Plan HPSS use
SLAC/Camera
- Data Portal Update for NCRs
- install pyds9 on lsst-dc10 and maybe also afs
  - sent inquiry to Jim and Richard
Fermi
- Attend and take notes for offline meeting
nEXO
- attend software meeting
- create list of open branches in nexo_MC
- Ask SNiPER developers about the ability to submit jobs via the python config scripts
- Prepare discussion on documentation for next software meeting: Doxygen & ROOT's THtml
- Respond to Samuele's Gaudi questions
- test cmake build of sniper against nexo-offline
- install py3 at SLAC
- RElease new nexo-offline and install at SLAC & create docker/singularity containers
  - OSG?
- Document a tutorial for nexo-offline for workshop
- Lessons learned?

Week of Feb 4 2019

DM-DC2
- Prepare agenda, attend and note take for DM-DC2 mtg
- Help Tom set up a suitable stack for BF kernel generation testing
- Test BF kernel generation by running the data produced through a modified version of Michael's notebook
- Set Tom up with Ru1.2i calibration data
- Test obs_lsst master using Run2.1 data
- Create workflow for imsim biases and darks generation
DESC
- Continue moving DC2 data to production area at NERSC
- Prepare a space request for HPSS based on DC2 needs
- Attend Data-Access meeting
- CI quarterly report
- Finish looking over Joanne's postgres notebooks
- Send in Travel request
NERSC
- Ask NERSC about setting up a shared HPSS area accessible via globus
- Install latest pyCCL on desc-python and in shifter images
- Document and promote use of CVMFS at NERSC
SLAC/Camera
- contact Arash about setting up a DB for elog storage
Fermi
- announce, set up agenda, attend and take notes for s/w mtg
nEXO
- attend sims meeting
- Meet with Lisa to plan Software Workshop

Week of Jan 28 2019

DM-DC2
- Prepare agenda, attend, and note take for DM-DC2 mtg
- Finish off https://jira.lsstcorp.org/browse/DM-16472
- Merge https://github.com/lsst/obs_lsst/pull/51
- Help Tom set up a suitable stack to exercise MIchael WV's BF notebook
DESC
- Prepare NERSC space discussion and attend CI mtg
- Attend DC2 mtg
- Prepare DM-DC2 XWG Confluence page
NERSC
- Prepare space request for HPSS based on DC2 needs
- Request IN2P3 transfer Run1.2p raw repackaged data back to NERSC
- set up psycog2 at NERSC
SLAC/Camera
- eLog - check new FNAL account and contact Margherita about transferring data to external DB
- contact Arash about a DB?
- alert camera community about impending elog shut down
Fermi
- Announce, attend, and take notes for offline mtg
nEXO
- Help SLAC users set up nEXO and EXO env
- check out Gaudi

Week of Jan 21 2019

DM-DC2
- Reprocess requests 28 visits through minidrp
- Prepare for new Run2.1i branch of obs_lsst
DESC
- attend ssims meeting
- Review Joanne's postgres notebook
NERSC
- Write up proposal for DC2 disk space use
- Try Exposure Checker
- Update jupyter kernels set up to add current working directory to PYTHONPATH
- update desc-dc2-dm-data for Run1.2p_v4
- Copy Run1.2p_v4 to CSCRATCH
- Build and release new docker image that supports Run1.2p_v4
SLAC
Fermi
- Attend and take notes for Offline mtg
nEXO
- Attend Sims meeting
- Write up Communications tools page
- Help SLAC users set up nEXO and EXO env

Week of Jan 14 2019

DM-DC2
- Prepare agenda, run and take minutes for DM-DC2 mtg
- Merge minidrp Run2.1i into IPP repo
- Chat with Tom about pipeline help
DESC
- Prepare NERSC disk space use slides and attend CI mtg
- Write up NERSC DC2 disk space use
NERSC
- Set up collaboration accounts
SLAC
Fermi
- S/W mtg
nEXO
- Look into Gaudi
- Install Anaconda Python
- Provide installation instructions
- Getting started at SLAC doc
- Install nEXO_MC and nexo_offline at SLAC
- Fast Charge

Week of Dec 17

DM-DC2
- Prepare agenda, run and take minutes for DM-DC2 mtg
- Test Run2,0i minidrp workflow
- Transfer IN2P3 Run1.2i configs to lsst-dev
- Get copies of CALIB and ref_cats
- Fix up handling of config.isr.doCrosstalk in obs_lsst
DESC
- Prepare Run1.2i/p update and attend CI meeting
NERSC
SLAC
Fermi
- Attend software meeting
nEXO
- Attend sims meeting
- Attend analysis discussion
- Contact Sara about docker
- Take another look at migrating FastCharge to nexo-offline
- Contact Vladimir about CMake progress
- Test singularity image at OSG

Week of Dec 10

DM-DC2
- Prepare agenda, run and note take for DM-DC2 mtg
- Run1.2i processing
- Prepare Run2.1i minidrp workflow
DESC
- nersc user mailing list?
- docker image for sims+mkl
- New docker image sims 2_13_1
NERSC
SLAC
Fermi
- write agenda and notetake for offline meeting
nEXO
- complete travel request
- help Sara with Docker
- Poke G4 guy about speed of light G4Exceptions

Week of Dec 3

DM-DC2
- Prepare agenda, run and note take for DM-DC2 mtg
- ~~Test new biases and darks~~
- Tarball and send Johann new biases and darks, request CALIB directory be transferred to NERSC
- Run1.2i processing on KNL
- rebase and merge PR#41 obs_lsst
DESC
- nersc user mailing list?
- new sims image 2_13_0
  - tag obs_lsst for Run2.1i
- docker image for sims+mkl
- attend and present 1.2i/2.0i processing update CI mtg
NERSC
- install new jupyter-kernels
- update desc-python with GCRCatalogs 0.9.3
- update desc-stack with GCRCatalogs 0.9.3
- update desc-python and desc-stack with GCR 0.8.7
SLAC
Fermi
- Help with recon root reading
nEXO
- sims meeting
- meet with Lisa
- setup nexo-env so it can work with base nexo external image
- Help Sara with Docker
- Test for G4Exception using Guofu's mac file on v0r0p1
- Merge zepeng's chargesim updates
  - new release tag and docker image

Week of Nov 26

DM-DC2
- prepare agenda, run, and take notes for DM-DC2 mtg
- Test newly generated biases and darks
- look at obs_lsst PRs
- Run1.2i processing
DESC
- nersc user mailing list?
- docker image for sims+mkl
NERSC
- update desc-python to include emcee
SLAC
Fermi
- s/w meeting
- Help with Recon root files
nEXO
- ~~Meeting with Lisa~~
- Merge Guofu's updates and tag new release
- Create new v0r1p0 Docker images
- create external only base for docker images

Week of Nov 19

DM-DC2
- prepare agenda and run, take notes for DM-DC2 mtg
- Regenerate darks and biases for Run1.2p using phosim v3.7.9
- Test generating bias file and sky flat using haswell build vs knl build at NERSC
- continue Run1.2i processing
- make sure all outstanding obs_lsstCam PRs are merged
  - figure out default setting for xtalk for Ru1.2p versus Run1.2i
DESC
- ask Charlotte about setting up a nersc user mailing list
- Update stack-sims images to keep test files and retain eigen includes
NERSC
SLAC
Fermi
- weekly s/w mtg
nEXO
- singularity at SLAC
- update wiki doc
- generate test data for Guofu's update
- attend sims mtg

Week of Nov 12

DM-DC2
- continue processing Run1.2i
- Maintain copy of IN2P3 on NERSC $CSCRATCH
DESC
- ask Charlotte about setting up a nersc user mailing list
- Field ANL questions about docker and stack+sims building
NERSC
- create new jupyter-kernels repo
- Update desc-python at NERSC
- Get my.nersc.gov login fixed after enabling MFA
SLAC
Fermi
- weekly s/w meeting
nEXO
- meet with Lisa
- singluarity at SLAC
- ask Raymond about OSG
  - still not fixed
- Test Guofu's updates

Week of Nov 5

DM-DC2
- Prepare and run DM-DC2 mtg
- Repackage ALL of Run1.2i data
DESC
- Attend CI meeting
- Document DESCPYTHONPATH for jupyter
NERSC
- Transfer NCSA Run1.2i data to NERSC?
- Update desc-python at NERSC
- Add GCR==0.8.6
SLAC
- Attend weekly stand up meeting
nEXO
- sims meeting
- Check on Guofu's testing of his updates in nexo-offline
- singularity at SLAC
- doc use of SLAC in nexo WIki

Week of Oct 8

DM-DC2
- Prepare and run DM-DC2 Meeting
- Chat with Leanne and attend meeting with Leanne, Johann, Bob, Dominique
- Process Run1.2i using w_2018_39 + obs_lsstCam dc2/run1.2-v2
DESC
- Create w_2018_30-run1.2 docker image + DESC python
NERSC
- take a look at dask-jobqueue
- Help write up problems/request for upcoming ERCAP application
SLAC
nEXO
- attend Sims meeting
- Release new Docker and singularity image for testing
- Port optical updates in nEXO_MC to nexo-offline
  - Ask Guofu to test
  - Make a new Docker image
Fermi
- run and attend s/w meeting

Week of Oct 1

DM-DC2
- Prepare and run DM-DC2 meeting
- test obs_lsstCam master + w_2018_39 ingest
- Run calexps and checkCcdAstrometry on two visits shared between Run1.2i and Run1.2p
DESC
- attend data access meeting on Monday
NERSC
- Check on availability of /usr/bin/time on Cori compute nodes
SLAC
- Help Johann get started with jupyter on IR2
- Install Anaconda 5.2.0 for python 3.6
nEXO
- meet with Lisa on Wednesday
- Create updated docker image of nexo-offline
  - Create Singularity image
Fermi
- s/w meeting
- Respond to W's questions about fitsGen & astro updates

Week of Sept 24

DM-DC2
- Prepare and run DM-DC2 Meeting
- Update SRS workflow
- Regenerate calibration data for Run1.2p
  - repackage
  - ingest
- test creation of multiple ingest repo
  - merge separate repos
- Revisit darks/bias/flats generation from a couple of months ago
- Stay on top of obs_lsstCam updates
  - merge xtalk PR
  - Contact Robert about merging outstanding PRs
  - Contact Robert about obs_lsstCam=>obs_lsst migration
  - Update run1.2 branch
  - Contact Robert about run_visit updates for jointcal
DESC
- attend CI meeting
- Provide CV
- Attend DC2 meeting and take minutes
NERSC
- Create new w_2018_38 docker image for MWV
- Work out problems with movable stack builds
- Check on /usr/bin/time on Cori and Edison and contact NERSC
SLAC
- Deal with IR2 stack installs and python questions
  - Document on Confluence
Fermi
- attend s/w meeting
nEXO
- attend sims meeting
- install latest sniper-offline at SLAC
- Check on OSG installation
- Meet with Lisa on Wednesdays
- Check in with Raymond about OSG

Week of Sept 17

DM-DC2
- Prepare and run DM-DC2 Meeting
- Test and update new SRS workflow
  - move to Edison
    - cannot run on Edison - chatted with Tom, sent a note to Tony, Brian, and W
      - Tried Tony's suggested fix and it seems to be working
  - move back to Cori
  - write up for Warren
- Stay on top of obs_lsstCam updates
  - Create run2.0 branch
  - Update run1.2 branch
- Create scripts for merging ingested repos
DESC
NERSC
- Handle permissions on $CSCRATCH/../desc/DC2/Run2.0i
- Work out kinks setting up movable stack builds
SLAC
- Reinstall v16 binaries and test updating python install
Fermi
- Tag evtClassDefs
- attend s/w meeting
nEXO

Week of Sept 10

DM-DC2
- Prepare and run DM-DC2 meeting
- Help sort out using ingestDriver from within a container
  https://lsstc.slack.com/archives/C2K97E3L5/p1536670464000100
- Contact Tom G about pilot Edison jobs
- Test running ingest on Edison
- Coordinate with Johann to get an updated workflow
DESC
- Finished creating new lsst_sims docker image over the weekend and uploaded to NERSC
- Prepare and attend DC2 meeting
- copy UK Run1.2i data to projecta and test use of ACLs
NERSC
- Fix ACls on the projecta production area
- Coordinate with NERSC to sort out problems with cvmfs: https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=725125d4dbaca3407d2a300f7c9619ee%26sysparm_view=ess
- Try to sort out source builds at NERSC on SCRATCH and migrate to /global/common/software https://lsstc.slack.com/archives/C4RKBLK33/p1536682237000100?thread_ts=1534435430.000100&cid=C4RKBLK33
  - still a work in progress, need to fix up miniconda installation
SLAC
- Install DMstack v16 on IR2
- Inquire about IR2 install of devtoolset-6
nEXO
- attend sims meeting
- Help nexo user use Centos7
- inquire about data disk filling up at SLAC
- convert FastCharge sim to Sniper
Fermi
- Attend C&A meeting

Week of Sept 3

DM-DC2
- Prepare and run DM-DC2 meeting on Tuesday
- Validation processing of Run1.2i
  - generate amplifier files from original eimages
  - run ingest on Cori Haswell and Cori KNL
  - run singleFrameDriver and report errors
  - run makeFpSummary
  - Try visualizeVisit
    - update obs_lsstCam to include necessary template
  - Keep documentation updated
  - Answer Tom U's questions about ingestDriver
- Coordinate obs_lsstCam branch and create Run1.2 tag
DESC
- Create new lsst_sims docker image for Jim and upload to NERSC
NERSC
- Inquire about desc access to globus
- Clean up inode use on /projecta
SLAC
- install w_2018_26 for Emily
- Handle vendor ingests (5 so far)
nEXO
Fermi
- software meeting
- check celestialSources used in GR L1 versus master
- contact Luis about updates to eblAtten

Week of Aug 20

DM-DC2
- Prepare and run DM-DC2 meeting
NERSC
- install phosim 3.7.16
SLAC
nEXO
Fermi
- Software Meeting

Week of August 13

DM-DC2
- Prepare and run DM-DC2 meeting
- Attend CI meeting
NERSC
- finish coadd processing Run1-2iTest
- Test use of DMstack Shifter image in jupyter-dev
- update jupyter-dev confluence pages
- fix inode use on projecta
- finish testing relocatable Stack weekly build
SLAC
- release and install latest datacat-utilities on SLAC public and IR2
nEXO
- attend Sims meeting
- Install nEXO_MC and test running FastCharge
Fermi
- respond to Katie Chapman's Fermipy helpsoft issue
- Update workbook pages to remove POI form

Week of July 16th

DM-DC2
- Prepare and run DM-DC2 meeting
NERSC
- Update shared python installation
  - GCRCatalogs
  - new CCL
    - track down reported notebook failure
    - track down reported notebook failure
    - reinstall at NERSC pyccl 0.3.5
SLAC
- Follow up on laptop purchase
- Release and install datacat-utiltiies-0.1.1
- Release and install datacat-utiltiies-0.1.1
- Install sims 2_9_0 at SLAC
nEXO
- Sims Meeting
- Test Docker image in Singularity
Fermi
- Offline S/W meeting

Week of July 9th

DM-DC2
- Prepare and run weekly DM-DC2 meeting
- Prepare for XWG DM-DC2 Meeting
NERSC
- Create symlink copy of IN2P3 data in $SCRATCH
- Help update Jupyter documentation and contact NERSC about jupyterLab
- Update CCL NERSC doc https://github.com/LSSTDESC/CCL/wiki/Using-and-Installing-CCL-on-NERSC
- Contact Debbie & Wahid about GPFS AFM
- Install sims 2_9_0 from source
- Install docker image of sim 2_9_0 at NERSC
- Continue testing weekly install cron job
- Contact NERSC about missing gcc/6.3.0 module
SLAC
- Follow up on laptop purchase
nEXO
- Meet with Lisa
- contact Raymond about fast charge
- contact Alex about fast charge (email sent)
- Gain access to LLNL account
- Pursue Singularity container
  - Install Singularity on CentOS7
  - Try using our docker image within Singularity
- Document Docker container
- Set up version control for SNiPER framework
- Check OSG external library installation and send email to Benedikt
Fermi
- Offline Software meeting

Week of June 25

DM-DC2
- Write up agenda, take notes, and run DM-DC2 meeting
- Create NERSC's raw directory for Run1.2p data transferred from IN2P3 in /global/projecta/projectdirs/lsst/global/in2p3/Run1.2p/w_2018_18
- Merge and close the makeFpSummary PR on LSSTDESC/obs_lsstSim
- Work through remaining issues, merge and close config updates https://github.com/LSSTDESC/obs_lsstSim/pull/16
- Push along updates to the Cookbook and write up a similar doc with an overview à la HSC
- Prepare to run mini-DRP on Run1.2i
- Attend DC2 prep meeting
- Attend DC2 data access meeting
- Contact Dominique F about providing an update at the DC2 meeting on July 6th
NERSC
- Monitor NERSC ticket concerning GPFS AFM
- Inquire about monitoring SCRATCH space for upcoming purges
- Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
- Fix weekly builds of DMstack to use limited number of cores and skip w_2018_24, since it will not build due at NERSC
- NERSC Ticket to work out details of weekly DMstack builds
- Update CCL doc for collab meeting
- Clean up disk space in /global/common/software
- Fix inode use on projecta space
- New sims 2_9_0 release, docker image
- Install sims 2_9_0 at NERSC
- Contact owner of galsim files installed under our w.2018.13-sims_2.7.0 installation in /global/common/software/lsst
SLAC
- Check on set up of datacat-utilities
nEXO
- Respond to CMT=>CMake project for SNiPER
- Ping Marco (Gaudi) about their CMT=>CMake experience
Fermi
- Verify that we can checkout L1 tags using --force and update doc
- Create new GR FDM-10 build:
  Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key FDM-10

Week of June 18

DM-DC2
- Write up agenda, take notes, and run DM-DC2 meeting
- Keep up with Johann's modifications to the SRS workflow
- Create NERSC's raw directory for Run1.2p data transferred from IN2P3 in /global/projecta/projectdirs/lsst/global/in2p3/Run1.2p/w_2018_18
- Merge and close the makeFpSummary PR on LSSTDESC/obs_lsstSim
- Work through remaining issues, merge and close config updates https://github.com/LSSTDESC/obs_lsstSim/pull/16
- Try running ingestCalib again using the phosim v3.7.9 produced biases and darks
- Push along updates to the Cookbook and write up a similar doc with an overview à la HSC
- Prepare to run mini-DRP on Run1.1i
NERSC
- Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
- Investigate load use of weekly installs and modify the cron job to limit the number of cores utilized.
- update to GCRCatalogs 0.7.8 in the shared python environment as requested by Yao
SLAC
- Install DMstack v15 for IR2, py3
- Install w_2018_23 for summer grad student
nEXO
- Write up slides for Docker intro for collaboration meeting and present
- Write up slides for SNiPER update and present
- Listen in to Simulation session 9PM-1:30AM
Fermi
- Weekly software meeting

Week of Jun 11 2018

DM-DC2
- Write up agenda, run, and take notes for weekly DM-DC2 meeting
NERSC
- update fast3tree installation for Yao to allow writing to the install directory
SLAC
nEXO
- Complete docker image for collaboration meeting
- Inform collaborators and help get them set up before the meeting
Fermi
- New L1 Release tag for GR See JIRA FDM-5
- Build new L1 Release tag at SLAC for Philippe to use

Week of Jun 4 2018

DM-DC2
- CI Meeting update
- Prepare agenda, run DM-DC2 meeting, and take notes
- Try constructFlat at NERSC
  - ingest subset of Run1-2p data
- ~~Doodle Poll to find new meeting time~~
  - chat with Dominique & Johann first
- ~~Ingest all Run1-2p data at NERSC~~
  - Merge latest IN2P3 workflow into NERSC workflow
  - Start up the jobs
NERSC
- investigate installing new python module Yao requested
- Talk to Joe Z about his miniconda env at NERSC to see about pulling it into the shared python env
- Reinstall stack weeklies when the next python update for DM is released
- deal with inode use on /global/projecta/projectdirs/lsst/
  - Issues is associated with the phosim SEDs for every visit, these files must be tarred.
SLAC
nEXO
- Data Task Force Meeting Wednesday
- Prepare proper docker image with tutorials
Fermi
- merge slac doc in github
- Offline Software meeting
- Help Philippe make L1 updates to TkrDigi
- get a new GR L1 release tag made

Week of May 28 2018

DM-DC2
NERSC
- Finish slides for seminar and present
- Test Javier's notebook
- Fix py3-4.3.21 installation with numpy and numexpr
- install phosim v3.7.15 for Haskell-Intel & KNL-intel
- Install CCL for Yao in the shared python area
SLAC
- Handle vendor ingests while Tom is away
nEXO
- Attend sims meeting and report on docker status
- Update docker image with tutorials
Fermi
- Help Philippe get set up with github and new JIRA area
- Write up initial doc for developing at SLAC: https://github.com/fermi-lat/doc/blob/heather999-patch-1/slacdev.rst

Week of May 21 2018

DM-DC2
- prepare for weekly meeting and take notes
- Send Bob Armstrong details for getting connected to NERSC and IN2P3
- Rerun processEimage test for v3.7.14 to capture log
- Prepare update for CI meeting
- Report unit test failure in LSSTDESC/obs_lsstSim
  - Fix the failure
- Arrange meeting with IN2P3 folks
- Attend DM-Stack-Club
- Attend and report at DC2 Meeting
- Rerun bias and darks using phosim v3.7.14
NERSC
- install latest parsl
- set up new external python environment
- Prepare slides for seminar
- Set up new docker image for new sims release and upload to shifter
- Install new sims release from source
- Transfer DC1 data from HPSS to SCRATCH
  - http://www.nersc.gov/users/storage-and-file-systems/hpss/advanced-usage-examples/hsi-tape-ordering-script/
  - http://www.nersc.gov/users/computational-systems/cori/running-jobs/advanced-running-jobs-options/
  - Confirmed /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered is the full data set stored in HPSS
- Help user build CCL paper and updated the CCL wiki https://github.com/LSSTDESC/CCL/wiki/Using-and-Installing-CCL-on-NERSC
nEXO
- update docker image to add in tutorials
SLAC
- order new laptop - waiting on response from I&T
- Handle vendor ingests while Tom is away
Fermi

Week of May 14 2018

DM-DC2
- set up agenda and write up notes for Monady's DM-DC2 meeting
NERSC
- Check getting started at NERSC documentation
- Contact Michael & Javier about the NERSC seminar May 30th
- Run phosim v3.7.9 and v3.7.14 test data through miniDRP
- Check on kubernetes availability at NERSC jupyterhub and set up meeting
nEXO
- update docker image to remove sudo and add in tutorials
- coordinate archiving nexo-sniper
SLAC
- set up order for new laptop
Fermi

Week of May 9 2018

DM-DC2
- Setup agenda and write up notes for Monday's DM-DC2 meeting
- Create darks using phosim 3.7.9
NERSC
- Install new versions of phosim as they appear
nEXO
- Attend sims meeting and make presentation concerning SNiPER visit
- Update docker image with tokens
- contact Raymond about timing correlation
SLAC
- Set up order for new laptop
- Requested and tested git lfs install at SLAC for nexo
Fermi
- Attend software meeting and write up notes

Week of April 30 2018

DM-DC2
- Set up agenda and write up notes for Monday's DM-DC2 meeting
- Rerun one processEimage job on one core on KNL using a source build of the stack
- Send Salman and Katrin directions about the steps to run "mini-DRP" so they can try it at ANL
- Set up full pipeline at NERSC to run on KNL
- Finish creating calibration products and document
- Prepare Monday's CI update
NERSC
- Update python environment to include potables
- Create "current" symlink to point to the current python installation
- Test and document setting up jupyter-dev using a shared stack install
- Install phosim v3.7.10 for both haswell-intel and knl-intel
- install phosim v3.7.11 for both haswell-intel and knl-intel
nEXO
- Set up Docker image for nexo-offline + SNiPER
- Provide daily summaries on the Slack meeting channel
- Test new SNiPER using the last collaboration meeting examples
- Understand how to implement Time Correlation
- Attend Tuesday Update meeting with nexo collaborators
SLAC
- Help Tom M set up python on his laptop
- Update elog Slack app to add a new channel as requested by Aaron

Week of Apr 23 2018

DM-DC2
- Set up agenda and write up notes for Monday's DM-DC2 meeting
- Rerun one processEimage job on one core on KNL as requested by Salman
- Send Salman directions about the steps to setup the DMstack so they can try it at ANL
- Provide NERSC update for DC2 meeting
- Attend DC2 meeting
NERSC
- Create an area of IN2P3 to transfer their data products
nEXO
- Install new version of SNiPER for centos7
  - request new tbb system library be installed
- Send some initial SLAC usage instructions to the SNiPER developers
SLAC
- Help Steve ssh tunnel to new IR2 jupyterhub

Week of Apr 16th 2018

DM-DC2
- In support of SkyCorrection Task
  - Generate darks using phosim v3.7.9
  - Play with constructFlat.
- Rerun miniDRP task using KNL and produce some timing information
- Update LSSTDESC/obs_lsstSIm for Phosim v3.7.9
  - Tagged Run1.1p obs_lsstSim fork
  - Successfully updated the obs_lsstSIm files using the bias output
  - Work through unit test failures
  - PR
NERSC
- Clear out out Stack builds as advertised on Slack
- Install parts in shared python env + weeklies + v15
  - Consider setting up a common shared lib area
- Install a modern dmtcp on knl-intel and haswell-intel
  - multilib install failed, went back to 64 bit only for now - which should be fine for phosim
nEXO
- Prepare for travel to SLAC
- Help SNiPER Developers get set up with SLAC accounts
- Install new version of SNiPER at SLAC

Week of Apr 9 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- HistogramSvc
- Write up SLAC SNiPER installation instructions on wiki
- Send Lisa some journal references for Gaudi
DM-DC2
- Prepare for Task Force Meeting and write up Live Notes
- Complete initial run of first Run 1.2p visits through minidrp
- Create new dc2 Shifter image with room for a NERSC volume
- Create dc2 Shifter image using a dc2 tag to try to avoid version mismatch errors
- update workflow to use Shifter and submit PR
- Create w_2018_14 + LSSTDESC/obs_lsstSim Shifter image
- Test pipeline with w_2018_14+LSSTDESC/obs_lsstSim Shifter
- update NERSC workflow to try out processEimage on rafts
NERSC
- Prepare updates for CI meeting
- Attend Ci meeting
- Send Katrin some information about how the set up NERSC for DESC
- Ask NERSC about anticipated need to recompile software after the scheduled Cori OS update on Apr 25-26
- Install DMstack v15 in the common software area
- Request to remove some older stack installations at NERSC
SLAC
- Write up portal section in Overleaf
- Complete Research integrity course
Aux Telescope

Week of Apr 2 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- HistogramSvc
- Attend Simulations Meeting
- Write up SLAC SNiPER installation instructions on wiki
DM-DC2
- Set up new Docker image which includes patched obs_lsstSIm
- Prep slides and attend SSim meeting
NERSC
- Check on weekly builds and document their new location
- Create new docker image for new sims 2_7_0 release
- Install sims 2_7_0 from source in the common software area
SLAC
- Complete CyberSecurity Training
- Write up portal section in Overleaf
- get antivirus going on home centos7 to allow VPN connection required to deploy portal
Aux Telescope

Week of Mar 26 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- HistogramSvc
DM-DC2
- Prep for Task Force Meeting
- Write up Live Notes
- Contact Dominque B concerning processEimage failure
  - Now fixed with new config file
  - add this config file to github
- Write update for Friday's DC2 meeting
- Write update for CI meeting
- Start up new mini-DRP task in SRS pipeline
  - Run ingestData for all 66 visits from Run 1.1p
  - Run processEimage (obtain time and memory consumption stats)
SLAC
- Fix LSSTTD-1257
- Deploy data portal 0.5.14
NERSC
- move weekly builds to common software area by updating cron job
Aux Telescope
- On Hold until we hear from Kirk

Week of Mar 19 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- Fix build of ROOT on centos7, missing libEve and libRGL
- Finish building nexo SNiPER on centos7
- Work on HistogramSvc
- Attend simulations meeting
DM-DC2
- DM-DC2 Meeting prep
  - DM-DC2 Meeting Agendas
- Attend SSims meeting
- Write up Live notes
- Extend DRP Workflow to run processEimage
- Update DRP Workflow to run processEimage for multiple visits
- Meet with Nicolas to discuss DRP
SLAC
- Implement and deploy new run search box
  - Jira
    server SLAC National Accelerator Laboratory
    columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
    key ETCCB-463
- Update my own centos7 desktop to use SLAC VPN, NetBeans, etc so I can continue supporting the data portal after the VPN transition at SLAC.
- Run vendor data ingests
NERSC
- Install TreeCorr
- Install new GCRCatalogs in all conda environments
- Move weekly stack installs
  - Clean up old contrib area
  - request another 100 GB of space
  - update cron job
Aux Telescope
- Wait to hear from Kirk

Week of Mar 12 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
nEXO
- Fix build of ROOT on centos7, missing libEve and libRGL
- Finish building nexo SNiPER on centos7
- Work on HistogramSvc
DM-DC2
- Update Nicolas' ImageProcessingPipelines branch for NERSC
- Demonstrate running SRS workflow for ingest
NERSC
- Work out production installation of lsst_sims + w_2018_09 using the shared python env
SLAC
- Create some documentation to use jupyterlab at SLAC
- Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1256
Aux Telescope

Week of Mar 5 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
nEXO
- Attend sims meeting
- Work with SLAC IT to get all necessary system libraries installed on centos7 to build nEXO's externals
  - libXpm-devel
  - bzip2-devel
  - Motif
  - Motif-devel
  - mesa-libGLU mesa-libGLU-devel
- Work through build problems with Coin3d 3.1.3 (an external lib)
  https://github.com/nEXO-collaboration/nexo-env/issues/1
- Finish building nexo SNiPER on centos7
- Work on HistogramSvc
- Create some initial dev doc in their wiki
DM-DC2
- Attend and take notes for task force meeting
- Attend SSims meeting Thursday
DESC
- Create docker image using new lsst_sims + w_2018_09
NERSC
- Install new lsst_sims + w_2018_09
- Add note about lsst_distrib + lsst_sims on Confluence: NERSC Software Installations
- Install CCL At NERSC in the shared DESCQA python env
  - GH Issue https://github.com/LSSTDESC/CCL/issues/346
  - Updated CCL NERSC Wiki: https://github.com/LSSTDESC/CCL/wiki/Using-and-Installing-CCL-on-NERSC
- Work out production installation of lsst_sims + w_2018_09 using the shared python env
SLAC
- Continue watching jupyterlab progress
- Create some documentation to use jupyterlab at SLAC
- Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1256
ATS At Harvard
- Attend test stand tutorial 9 AM - 10 AM PT Wednesday

Week of Feb 26 2018

Fermi
- get Brian to fully clean up between GR Jenkins builds - created JIRA
- get L1 GR to build
- figure out how to clean up workspaces in Jenkins on my own.
- Tag first GlastRelease release in github GlastRelease-20-10-04-gr03 only change is to the xxxRootData packages
- Figure out the release process
nEXO
- send an email update to Lisa
- update Slack channel about Raymond's discussions with SNiPER developers
- Work on HistogramSvc for ROOT files
DM-DC2
- Run Monday Meeting and write minutes
- Update GitHub issues
- Contact Chris W about flats
- Talk to Tom about setting up directory for transfer to IN2P3
- test ingestDriver.py at NERSC
  https://github.com/LSSTDESC/ImageProcessingPipelines/issues/6
- Update IPP repo with new mini-DRP Epic
  https://app.zenhub.com/workspace/o/lsstdesc/imageprocessingpipelines/boards?repos=112563170
- Create new SRS pipeline task to run ingest on eImages
  http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=50591002
- Update SRS pipeline task to handle ref cat ingest
- Update SRS pipeline task to run processEimage
- attend DC2 meeting
NERSC
- Poke NERSC about jupyter-dev outage - now fixed
SLAC
- Watch JupyterLab demo install and keep Steve informed
- Try out eT DB updates to prep for Joanne's departure
- Update DESC and Fermi documentation on getting a SLAC account
- Check on Tom M's reported problem with the status of some CCD-011 bits in the DP
  - Fixed
    Jira
    server SLAC National Accelerator Laboratory
    serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
    key LSSTTD-1220
ATS At Harvard

Week of Feb 19 2018

Fermi
- Finish Anaconda 3 install
- Fix up L1 builds on Jenkins
- Fermi Meeting
nEXO
- Start new HistogramSvc for ROOT files
- Simulation Meeting Thursday
DM-DC2
- First Task Force Meeting
- Migrate DC2 Cookbook issues to IPP (ImageProcessingPipeline) repo
- Create new issues in IPP repo
NERSC
- sort out HPSS allocation
SLAC
- Help David L access his account
ATS at Harvard
- Meeting with Tony & Owen on Tuesday
- Go to Harvard on Friday

Week of Feb 12 2018

Fermi
- attend offline meetings!
- Finish Anaconda3 installation
- Work out distribution of externals in the new regime
- Install swig 3.0.12 RH6 and RH7 on GLAST_EXT
- work out L1 build in Jenkins
nEXO
- Meet with sims people
NERSC Support
- re-re-install cosmosis to help out the WL folks who wanted the latest along the develop branch
- Reach out to Joe to encourage tagging of cosmosis and provide some feedback on their setup script for NERSC
- Reach out to wl pipeline group to ask about cosmosis installations.
- Start up automated weekly stack installations
- Document weekly stack installations
- Send Joe Z updates to the setup-cosmosis-nersc script
- Help Erfan get going with cosmosis at NERSC
SLAC support
- Read over Joanne's docs
- Pow-wow with Joanne over the DP and eT and the eT DB
- set up docker for Jim on IR2
- Update DESC and Fermi documentation on getting a SLAC account
DRP/L2 Processing
- Task Force
  - Phil mentioned that "they" had to write up a charge - figure out what that means
    - Modify charge after receiving suggestions
  - At Nicolas' suggestion reach out to Fabio H concerning weekly stack installations
ATS at Harvard
- Attend CCS meeting to find out how best to support Harvard
- Contact Harvard to sort out list of devices & plans for their use for testing
- Meet with Kirk
- write up minutes from Kirk/Nick meeting and prepare to iterate with Kirk

Week of Feb 5 2018

DRP/L2 Processing
SLAC DESC support
NERSC support
- set up automated installs : https://github.com/lsst-dm/shared-stack
- Help Alex set up web data access: http://www.nersc.gov/users/data-analytics/science-gateways/
  https://github.com/pmelchior/des-exp-checker
Other
- Read over Joanne's docs: http://corner.readthedocs.io/en/latest/install.html
- Write up IN2P3 account Confluence page
- Update DMstack install on Harvard AUX Telescope machine
nexo
- Attend Sims meeting
- Schedule time to meet Lisa next week
- Try out Raymond's example using SLAC install
- finish SLAC installation
Fermi
- Finish Anaconda 3 installation at SLAC
- work out distribution of externals for end users

Week of Jan 29nd 2018

Data Portal
- support Joanne as she learns about the plotting
DRP/L2 Processing
- Check on James P
- Get a copy of the DC1 WF going
- Update WF to run Jim's latest imSim Data
- Learn how to deal with pipeline issue in JMX dealing with large number of jobs stuck in submitted state
- Run Jim's imSim data through SLAC WF (first attempt in progress)
- Prep to run using Phosim 1.1p
- Document using the Slac WF and add scripts to a GH repo in LSSTDESC area
- attend Friday dc2 meeting
SLAC support
- review, release, and install the latest version of datacat-utilities
- Read over Joanne's documentation
- -~~automate weekly install of DMstack binaries for camera-~~
- install datacat-utilities on lsst-dc10
NERSC support
- Upload latest SQRE DMstack (w_2018_03) Docker image into Shifter
- Install w_2018_03 plus sims from source
- Create w_2018_03+sims Shifter Image (debugging issue with afw - issue identified.. fixing it is something else)
  - create sims Shifter image the long way, without using lsstsqre's base image
- Install WeakLensingDeblending for Javier.
Random LSST Camera and DESC items
- SSim meeting
- Meet with Wil on Friday
- Write up notes from meeting
- Look over DESC architects confluence page
nEXO
- reinstall SNiPER in the new disk space
- try out examples
- sign up for the collaboration mailing list
Fermi
- Investigate trouble with AcdDigi in L1 branch
- Find a way to upgrade Anaconda external for ST
- Install miniconda py3
- Contact Helen

Week of Jan 22nd 2018

Data Portal
- Consider rerunning all the ITL sensors in need of updated MET ingest and MET 05 by hand
- plots - commit latest code and write up intro for Joanne
- CCB request and deploy MET updates for the DP
- Deploy DP 0.5.11
SLAC support
- automate weekly install of DMstack binaries for camera (Jim doesn't seem to feel this is a priority)
- Read over Joanne's documentation
NERSC support
- update DESCQA install to include tensor flow
- write up instructions for w_2018_01 shifter install
- automate installation of shifter images
- Create docker image of new sims release based on w_2018_04
- Reintall cosmosis for WL group due to updated FFTW module, trying cray-fftw this time
- Send instructions to Josh to set up his own test galsim using the extlibs we already have set up for Cori KNL gcc
- install Phosim 3.7.7 and 3.7.8 for knl-intel and haswell-intel
- meet with Mustafa about HPSS
- Fix Cosmosis installation
Workflows/L2 DC2 Pipeline
- send plea to Tony
- Meeting with Tom about using the SLAC pipeline + pilots
- work out setting up test Pipeline2 job reusing DC1 L2 task and a handful of visits
- Prep for CI meeting
Fermi
- Work through errors with AcdDigi on L1 branch
- Set up new Anaconda external
- Work out with Brian the requirements for setting up externals these days
nEXO
- get Tony an disk space estimate for SLAC installs
- get some disk space from Tony at SLAC
- set up SNiPER at SLAC in a temp location (in progress)
- attend Tuesday sims meeting

Week of Jan 15th 2018

Data Portal
- revisit plots
- Holding off MET updates until new traveler to reingest ITL sensors is available
datacat-utilities
- Install new release 0.0.9 at SLAC
SLAC support
- automate weekly installs of DMstack binaries for camera
NERSC Support
- update DESCQA install to include tensorflow
- Write up instructions for w_2018_01 shifter install
- automate installation of shifter images
- Add WeakLensingDeblending
- Check on CCL installation
Workflows/L2 DC2 Pipeline
- DC2 meeting Friday Jan 19th
- Slim down docker image which includes sims - now down to 3 GB on DockerHub https://hub.docker.com/r/heather999/test/tags/
- Slim down docker image which includes sims - now down to 3 GB on DockerHub https://hub.docker.com/r/heather999/test/tags/
Fermi
- Set up draft f2f meeting page Fermi Software Week Winter 2018
- Continue working on GR L1 branch - completed checkout of L1 code successfully
- Work through errors with AcdDigi on L1 branch
nEXO
- Read over nEXO at OSG simulation notes https://ntpc.ucllnl.org/nEXO/index.php/Simulations_on_the_OSG

Week of Jan 8th 2018

DataPortal
- finish off ITL-3800C-368 ingest after sorting out http authentication errors
- revisit plots
- Hold off on MET updates until new traveler to reingest ITL sensors is available.
datacat-utilities
- Install new release
SLAC suport
- Install w_2018_01 binaries lsst_distrib for camera
- Work out automated weekly installs via cron (u1 is 97% full, ~990 GB left)
- Install Merlin's eotest branch
NERSC Support
- Update DESCQA install to include tensorflow
- Try w_2018_01 shifter install (in progress)
- Testing w_2018_01 shifter install and write up instructions
- Add WeakLensingDeblending https://lsstc.slack.com/archives/G8RU8C4VD/p1515604937000489
Workflows/L2 DC2 pipeline
- Set up lsst_sims docker image for NERSC & uploaded to Dockerhub
- Try another lsst_sims docker image using lsstsqre image as the base rather than centos7 (initially failed, have new plans to retry)
Fermi
- ~~Attend Wednesday software meeting~~
- Finish setting up GR L1 branch https://github.com/fermi-lat/GlastRelease
- Test L1 build in Jenkins https://fermi-lat.github.io/doc/workflow.html
- Try interactive build of GR L1 branch
nEXO & SNiPER
- Attend Tuesday software meeting
- Contact SNiPER team about Open Source License
- Apply for OSG and LLNL accounts
- Read over nEXO at OSG simulation notes https://ntpc.ucllnl.org/nEXO/index.php/Simulations_on_the_OSG

Week of Dec 18th 2017

DataPortal
- MET updates
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1221
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1220
Workflows/L2 DC2 pipeline
- Reproduce error James P sees
- Repeat experiment with multiple visits
- Repeat experiment using Head of ImageProcessingPipeline repo
- Potentially meet with Jim & James this week
- Catch up on Parsl progress
CI
- Set up new Docker image to use with imSIm's travis builds
- Create a smaller Docker images for use with travis
NERSC Support
- Shared Stack
- JupyterHub documentation and simplification
- clean old project space by removing MBII-snapshots
- Request reduction in old project space quota to 40 TB
imSim
Fermi
- Set up GR L1 branch in GitHub
- Test build of L1 branch
SNiPER
- Read up on SNiPER
- Meet with Lisa K on Tuesday 5 PM Eastern
- send feedback on documentation
- set up user accounts on Wiki, LLNL, OSG
- talk to Tony about nExo

Week of Dec 11th 2017

Data Portal
- Deploy ETCCB-429
- MET updates - almost complete
CCS ATS
- Install mvn and associated libraries and get Nick going with Lakeshore driver
Workflows/L2 DC2 pipeline
- Send James P a link to the recordings from last week
- Repeat test using Jim's parsl wf at NERSC
CI
- Docker image for use in travis builds
imSIM
NERSC Support
- Fresh Shared Stack installation for KNL - stuck working out disk space in the common software area – started
- dmtcp installation
- Update jupiter-dev documentation
- quota checker
Fermi
- Try GR with repoman
- Documentation

Week of Nov 27th 2017

L2 pipeline DC1 rerun
- Finish off write up of chat with Tony
- Ask James P about remote participation during Spring Week
  M & W 8 - 12 central time
- Try a run using Simon's cookbook
- Meet with Jim & James P
  - write up notes
- Try Parsl pipeline at NERSC
NERSC software installations
- Shared Stack installation - verify that the DESCQA and py3 install is working
  - Add CCL
- Install new sims 2.4.2 using DMstack w.2017.46 Haswell-gcc6 using py3
- Ask about cosmosis needs
  - Install cosmosis, TreeCorr, and MLZ in projecta area under the WL WG directory
- Install phosim 3.7.3
- dmtcp
- quota checker
- Consider requesting more common software space
  obtained 50 GB more
NERSC project space
- Verify MBII-snapshots is in HPSS and remove from project area
  Found the content in the desc hsi area, and viewed the contents of some snap.tar via htar -tf CS/MBII-DMO-snapshots/snapdir_074.tar Also noticed that the group_xxx.tar files are duplicated in the Archive home directory. Contacted Mustafa to sort that out and remove duplicates.
Data Portal
- MET updates
  - Meet with Anders and Seth
Harvard ATS
- Check that ccs reboot restarts the MySQL, LocalRestServices, and Trending
- Check in with Nick about the Lakeshore
Fermi
- Try GR using repoman
- Documentation
Discuss Gaudi for nEXO
https://gaudi.web.cern.ch/gaudi/releases/v30r0/
http://lcgsoft.web.cern.ch/lcgsoft/release/
https://gitlab.cern.ch/gaudi/Gaudi
Attend Stanford IT Unconference
https://itunconference.stanford.edu/sessions/remote

Week of Nov 20th 2017

L2 pipeline DC1 rerun
NERSC software installations
- py3 shared stack installation
- finish phosim 3.7.2 installation
- dmtcp intel installation
- quota checker
- cosmosis installation
  - check in on GH issue https://github.com/LSSTDESC/WLPipe/issues/31
Cover Vendor Ingests M-W
Data Portal
- plots
- check in with Tony about his changes to load the page more quickly
Harvard ATS
- Determine who will deal with the Lakeshore drivers
  https://jira.slac.stanford.edu/projects/LSSTCCSDRIVER/issues/LSSTCCSDRIVER-227
  https://github.com/lsst-camera-ccs/org-lsst-ccs-drivers-lakeshore/tree/LSSTCCSDRIVER-227
  https://www.lakeshore.com/Documents/336_Manual.pdf
- Grab Owens updates and install on CCS machine
- Direct Nick to make updates as needed.
- Fix mounting of data drive on ccs machine
- Try a version of the TS subsystem on the ccs machine and set up groovy & properties files
- Check monochromator plays well with the TS subsystem
Fermi
- Try a GR build with repoman
- Documentation
- Inquire about pointlike updates in CVS on Slack

Week of Nov 13th 2017

Harvard ATS
- Create user accounts and ccs sudo account which grants sudo access to all users
- Set up directory structure following IR2 model
- Install DMstack v14
- Install eotest compatible with v14
- document the above on the Confluence page ATS Test Stands
- Set up Trending Database
  - MariaDB installation
L2 pipeline DC1 rerun https://github.com/LSSTDESC/SSim_DC1/issues/47
- Meet with Jim on Monday
  - Write up Notes
- Contact James P
- Try to run Simon's Cookbook
- Chat with Tony
NERSC Software Installations
- Shared Stack
  - first try failed due to conflicts with libiconv installed presumably for
- Phosim 3.7.2 bug fix for haswell gcc & intel, KNL gcc & intel
- DMTCP intel build
- Help out with cosmosis installation https://github.com/LSSTDESC/WLPipe/issues/31
Data Portal
- Chat with Tony about plots
- Plots
  - Bad Channels
  - read noise
NERSC Project Space
- Test MBII-snapshot HPSS copy and remove from project area

Week of Nov 6th 2017

QA Testing
- Investigate cause of differences between Lauren's plots from NCSA and those generated at NERSC
  - look at 'conda list' on both NCSA and NERSC (no differences discovered)
  - generate cal_refs as was done at NCSA
  - try gcc 6 build as is being done at NCSA
- Determine the cause of differences between ref catalogs at NCSA and NERSC
L2 Pipeline
Data Portal
- Finish off first round of Richard's plots
  - bad channels
  - read_noise
NERSC Project Space
- test MBII-snapshot HPSS copy
ATS
- prep for Tony & Owen's visit
Installations
- SLAC
  - Py3 on SLAC RH6-64 for Joanne to test out
- NERSC
  - finish testing NERSC's cfitsio installation
  - install phosim 3.7.1 for Cori haswell gcc & intel, KNL gcc & intel
    - ask Glenn to install module files
  - install phosim 3.7.2 for Cori haswell gcc & intel, KNL gcc & intel
  - Shared Stack Installation using w.2017.41
    - Query DESCQA and CCL groups
    - py2 w.2017.41 installation
    - py3 w.2017.41 installation

Week of Oct 30th 2017

Data Portal
- Plots
DC2 prep
- QA testing
  - w.2017.41 installation plus branches of obs_lsstSim and pipe_analysis
  - Run visitAnalysis, coaddAnalysis, etc
NERSC Workflow work
NERSC Project Space
- Work with Mustafa to get MBII-snapshot backed up
NERSC Installations
- w.2017.? stack installation plus sims - waiting on new sims release using the camera geometry updates
- Phosim 3.7
  - Testing NERSC cfitsio installation
SLAC Installations
- descsoft w.2017.? stack plus sims installation
CCS
- See if there is any prep work necessary for Tony's visit later in Nov
  - Request Nick set up new box with CentOS7
  - set up user accounts and groups
- Determine if Mike is still coming out ~Nov 1st
Attend SuperCDMS meeting concerning their potential move to NERSC.

Week of Oct 23rd 2017

Data Portal
- Richard's Plots - bad channels (in progress)
- Look at new Report Labels and determine what the DP still requires from the Reports Table
  Jira
  server SLAC National Accelerator Laboratory
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1190
- Make sure we request a restart of Tomcat
NERSC Workflow work
- Try out HTCondor and Pegasus on cori21
- Take a look at Jim's imsim script
NERSC Project Space
- Set the old project directory to read-only (in progress)
  - claim desc ownership of the whole directory space
- Clear out all directories where the owners have indicated it is fine to do so
- Determine the end state of merging old project and projecta areas and handling of www
- Set up disk quota monitor
- Coordinate HPSS storage of MBII-snapshots
NERSC installations
- Phosim v3.7
  - Coordinate with NERSC to get cfitsio v3410 installed (in progress)
  - Rebuild using NERSC supplied cfitsio
  - Coordinate with Glenn to set up module files
- Set up software for DC2 testing
- v14_0 lsst_distrib + sims 2.4.0
SLAC installations
- Set up same imsim at SLAC as used at NERSC for KNL testing (in progress)
- v14_0 on RH6-64 public (on hold pending resolution to obs_base testCameraMapper.py unit test failure)
  - camera area
  - descsoft
  - test eotest branch
- v14_0 on IR-2
Document KNL vs Haskell mkl imsim testing at NERSC
- Initial mkl tests indicate no huge improvement - putting testing on back burner for now
CCS - still waiting to hear from Peter, I cannot access their TS3 machine to install and test a new version of the test stand subsystem

Week of Oct 16th 2017

Richard's plots in DP
- Complete Sensor Arrival Time plot
- Create JSON
NERSC installations
- Phosim v3.7
  - Try out NERSC supplied FFTW
  - Request new cfitsio v3410 installation required by Phosim
  - Complete installation of all flavors: haswell-gcc, haswell-intel, knl-gcc, knl-intel
    - Waiting on cfitsio v3410 before knl-intel will be available
NERSC Project Space
- Notify all users of the need to migrate
- Handle all help requests - migrating some directories to the new projecta area
- Install disk quota monitor
- Check on old project status and prepare to set it to read-only
- Coordinate backup of nkhandai data to tape with Mustafa
Install latest version of teststand subsystem at Harvard there is a machine made available - waiting on response from Peter
Keep an eye out for v14 DMstack

Week of October 9th 2017

Continue work on Richard's plots in DP
Jira
server SLAC National Accelerator Laboratory
columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
key LSSTTD-1165
- Released Data Portal 0.5.7
Testing imSim built on KNL
- Continue MKL tests at NERSC once the system recovers from the weekend outage
- Install the same version of the stack at SLAC to allow similar testing with and without MKL
- Attempt build of DMstack using the intel compiler at NERSC
NERSC Project space
- Install disk quota monitor
- Coordinate backup of nkhandai data to tape with Mustafa
- Prepare to set the old project area as read-only
- Send mail to all owners of files on the old project space
NERSC HPSS SRU overallocation
Install latest version of datacat-utilities 0.0.8 at SLAC
Install the latest teststand snapshot at Harvard to support use of the monochromator when Peter returns.
- Contacted Peter - I can't connect to the archon machine to install the test stand snapshot
- Investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly create a harvard main routine.
Keep an eye out for the v14 DMstack release and prepare to install on IR-2 and NERSC
- This release will include binary installations.. I'm thinking we may still want to install from source, perhaps that's most useful on IR-2, but we'll see.

Week of October 2nd 2017

Release DataPortal 0.5.6 which will include
- Addition of Grade and Procurement labels for the Sensor Acceptance pages
- RTM assignment
  Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1121
- NCR "Ready to close"
  Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1144
Implement Richard's plots in the DP starting from some of Tony's examples for trending and Richard's python code.
Continue testing KNL performance using galsim and imsim
- Attempt to compile galsim and possibly all of DMstack with the intel compiler
- Run longer tests using the MKL libraries on Haswell and KNL
- Rebuild imsim with MKL, using an updated meas_base
Install an initial shared stack in the new software area at NERSC for testing as discussed at last week's "Shared Stack" meeting
NERSC Project Space
- Set up an initial quota monitor at NERSC
- Re-connect with Mustafa to get the mkhandai 42 TB archived to tape.
Investigate an upgrade of the monochromator driver at Harvard and confirm that to achieve what Peter plans to do - to run his ATS work in the lab alongside most of the devices from his TS3 setup.. is possible by running both the teststand and rafts subsystem simultaneously.
- Install latest SNAPSHOT of test stand at Harvard and investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly creating a harvard main routine.
Keep an eye out for the v14 release of the DMstack and prepare to install at SLAC/IR-2 and NERSC.
Though this is assigned to Jim, so perhaps I have no need to worry about SLAC:
Jira
server SLAC National Accelerator Laboratory
serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
key LSSTTD-1157

Snowmass https://snowmass21.org/computational/start

Week of 16 November

CO
- write agenda, attend, potentially run and note take for CO meeting
- set up Zoom #s for Sprint week
NERSC
- shut down desc-python:bleed scheduled build for now
DESCDM
- write agenda, announce, and run DESC DM/DA mtg
- continue checking warps, include new DR2 warps produced at CC
CC
- attend SBPC update and note take
- Write up DESC newsletter entry
- Help finish off SNC draft cha
Fermi
SLAC
nEXO
- attend software meeting

Week of 9 November

CO
- attend CS meeting
- Help create lsstdesc-env conda-forge package
NERSC
- install intake in desc-python:bleed
- follow up on CVMFS problems with sqlite3
- Run a proper DR6 Gen2 ingest
- Create SPIN Postgres
- Test SPIN website
DESCDM
- handle DESC DM/DA meeting
- Discuss GCR future with Yao & Joanne
- Check on DR6-WFD warps
- Look at DR2 object catalogs to regenerate the ones fixed by the finished tract/patch processing
CC
- announce, attend and run CC meeting
- handle OMP amendment voting
- handle PM amendement voting
  - edit draft collaboration announcement
- Write up DESC newsletter entry
Fermi
SLAC
- attend ADASS as possible
nEXO

Week of 2 November

CO
- attend CS meeting
NERSC
DESCDM
- DESC DM/DA meeting
CC
Fermi
SLAC
nEXO

Week of 26 October

CO
- Contact James P & George B for CO meeting
- Write agenda, attend and note take for CO meeting
- Upgrade GCRCatalogs to v1.0.2 for all python environments
- Review DC2-analysis tutorials and automate beavis running
NERSC
- attend GPU webinar
- Update desc-stack-weekly for w_2020_44
  - automate those builds
- Follow up on Jupyter issues on Monday
- Deal with reported issues with MPI and nbodykit in desc-python env
- Help SN group upgrade to GCC 9.3.0
- Address question concerning permissions
DESC DM
- Announce, run, write agenda, for DESC DM/DA
- Complete initial set of DR2 object catalogs
- Copy matched catalogs to shared area at nersc https://github.com/LSSTDESC/desc-help/issues/49
CC
- Help write up slides, update agenda for DESC-wide Software Policy meeting
- Follow up on SBPC DESC-wide meeting
- Schedule November CC meeting
- ask MC to send out full membership announcement to collab
- Prepare and send email to DESC members reminding them to update DB entries
- Follow up on SNC
- Prepare OpaVote for ratification
- Prepare Collaboration-wide comment period for FM amendment
SLAC
Fermi
nEXO

Week of 19 October

CO
- attend CS meeting
- finish Gateways seminars
- Continue viewing JupyterCon
NERSC
- try out Globus Spin app
- update user on how to intall nbodykit
- Address various desc-help requests
DESC DM
- announce, write agenda, note take and run DESC DM meeting
- Produce object catalogs for all DR2 tracts where reasonable
CC
- attend, run CC Mtg
- Update SRPC page
- Announce SP DESC-wide meeting
SLAC
Fermi
nEXO
- attend software meeting

Week of 12 October

CO
- attend PS meeting
- attend Gateways
- attend JupyterCon
- attend Rubin Bootcamp
- attend SSim meeting
- Update Jupyter at NERSC Confluence page
NERSC
DESCDM
- attend TJP meeting to help present DC2 data availability
- write agenda, note take, attend DESC DM/DA
CC
- Stand up CCNC
- Prepare for Oct 19 meeting
- OMP ratfication
SLAC
Fermi
nEXO

Week of 5 October

CO
- Attend JupyterCon tutorials
- Attends parts of Snowmass
- Q4 update
NERSC
- Follow up on Humna's questions about trouble using desc-stack during Cori outage
- Post NERSC Cori updates to desc-nersc
- Check on HPSS storage use
- Work out issues connecting to jupyterhub with CSCRATCH unavailable.
- descqa website problems
DESCDM
- Attend ParslFest
- Follow up on DR2 WWT data
CC
- Announce potential CCNC members to CC and gain approval
  - send invites
- Announce CC Oct meeting, update DESC calendar, alert SP team, write agenda
SLAC
Fermi
nEXO
- Test SDF for nEXO user using singularity
- Set up new nEXO SDF space
- Start up build to set up a private docker image including nexo-offline to test with Yee

Week of 28 September

CO
- Attend CS meeting
- Prepare for CO meeting
- Attend DIA meeting
NERSC
- set up data for WWT
- check on desc-bleed
- set up re-useable Gen2/Gen3 repo at NERSC
- refresh small dataset on HPSS
DESCDM
- Write agenda, announce, note take, attend DESC DM/DA meeting
- Back up DR2 calexps just in case
CC
- Get DESC-wide meeting set up for SRPC and SBPC
- Stand up CCNC
- Work on scheduling Oct CC Meeting
SLAC
Fermi
nEXO
- set up nexo-offline on SDF with Yee

Week of 21 September

CO
- Attend SSim meeting
NERSC
- Set up Transfer for WWT
- Transfer DC2 Run2.2i to lsstdev
- check on desc-bleed
DESC DM
- Announce, write agenda, attend DESC DM meeting
- Grab updated Gen2 calibration flats for Gen3 testing
- set up Gen2/Gen3 repo at NERSC
- Move on with Gen3 testing at NERSC
CC
- Sept CC meeting
- MRPC Meeting
- Provide newsletter update
- Announce Software Policy draft
SLAC
Fermi
nEXO
- Work with Yee to set up nexo sims on SDF
- Software meeting

Week of 14 September

CO
- write agenda, take notes for CO meeting
  - announce meeting and invite DMers
NERSC
- Set up transfer for WWT
- Update desc-python:bleed and turn automatic installation back on
DESC DM
- write agenda, attend, take notes for Wednesday meeting
- Get DR6 WFD annoucement out the door
- organize warps and put them on CSCRATCH
- fix up dr2 calexps from Run2.2i
- regenerate tract2visit sqlite DB
CC
- set up Sept mtg time
- discuss SRP edits
- Push on CCNC and SNC meetings
- Contact 2019 SNC about survey
Fermi
Slac
nEXO
- attend Slac computing discussion on Thursday
- attend software meeting
- respond to issues submitting jobs on GPUs
- Work with Yee to set up some modules on SDF
  - Figure out a good production version using LLNL's install

Week of 7 September

CO
- attend CS meeting
NERSC
- add ceci to desc-python
- add ceci to desc-stack
DESC DM
- write agenda, attend, note take for DESC DM meeting
- Keep warp copying going
- Update scripts to find the warps left to copy and try globus instead
- Chat with Tom about warps
- Set up tract2visit mapper for DR2
- Find uband noPSF config discussions
CC
- Set up Sept CC MTG poll
- Push along SNC and CCNC
SLAC
Fermi
nEXO

Week of 31 August

CO
NERSC
- check on descdm quota
DESC DM
CC
- Aug CC meeting notes for collaboration
- Get back to meetings committee concerning their question
- get MRPC meeting set up
SLAC
Fermi
nEXO
- attend software meeting
- Try out GH Actions

Week of 17 August

CO
- https://github.com/LSSTDESC/desc-help/issues/35
NERSC
- help SN set up a conda environment
  - meet with Bruno and Rick
  - inquire about TxPipe
- look at running sn-pipe in tutorial notebooks
DESC DM
- agenda, attend DESC DM/DA meeting
- continue pulling over Run2.2i DR6 data from CC
- Prepare for DR6 release
  - back up pixel level data
  - remove unwanted tracts and patches
- Reach agreement on what is DR6d
- Update DC2 calibration headers to adhere to new CALIB_ID format
- Y1 warps for DR2
- add /usr/bin/time to shifter image for parsl DR2 runs
- data transfer to lsst-dev all of Run2.2 and Run3.1
- Clean up CSCRATCH
CC
- announce, write agenda, attend Aug Meeting
- write up DESC-wide minutes
- complete MRPC note and send
- Put out one more call for CCNC and SNC members
- Questionnaire Survey
SLAC
- glastsys password
- performance review
Fermi
nEXO
- attend software meeting
- test free github area
- look at open source chat tools to replace Slack

Week of 3 August

CO
- attend leadership meeting to update SRM
- export Zoom stats
- Size of DC2 files
- fverify Run3.1i sim files
NERSC
- update sn repo wth NERSC instructions
- test GHI
- setup new SN min
- rsync coadds again, consider cron
DESC DM
- Copy DR6 coadd data to NERSC
CC
- poll for August meeting
Fermi
nEXO
- attend software meeting
- add more users to nexo UNIX group

Week of 27 July

CO
- Complete Q3 report
NERSC
- Reorg Run1.2x data to reside on CFS
- Back up Run3.0i data on HPSS and remove from CFS
- Finish cleaning up my CSCRATCH
- Clean up desc's CSCRATCH
- Setup automated running of DC2Analysis tutorials at NERSC
- add selenium and python-snappy to desc-python https://github.com/LSSTDESC/desc-help/issues/31
  - need to fix up desc-python-bleed
DESC DM
- Upgrade desc-stack-weekly for next DM weekly, which hopefully fixes their conda install issues
- Reinstall Run2.2i Gen3 repo using latest DM weekly
- Update tutorial notebook for Gen3
- IPP eups
- setup rsync of DR6 WFD coadds
- Store Run2.2i WFD in HPSS
  - Y1
  - Y2
  - Y3
  - extract unneeded Y4 and Y5 files
  - Y4
  - Y5
CC
- write up June monthly meeting notes
- schedule August meeting
  - send out poll
SLAC
Fermi
nEXO
- Evaluate open source chat options

Week of 20 July

Attend collaboration meeting
Install new desc-python-dev to downgrade pyarrow for desc-pyspark
complete Gen3 Run2.2i demo repo
Set up desc-stack-weekly-latest for w_2020_27 to be used with Gen3 repo
Write up notes for others to run notebook examining Gen3 repo
nEXO
- respond to Samuele's questoins about CVMFS
- respond to Vladimir's data proposal

Week of 13 July

CO
- attend WL meeting
- Write up builder nomination letters
- Announce, attend, note take CO meeting
DESC DM
- announce, attend, note take DESC DM/DA meeting
NERSC
- set up Run2.2i Gen3 repo
- Fix entrypoint for desc-bleed
CC
SLAC
Fermi
nEXO
- attend software meeting
- create, test and install new nexo-python docker image at SLAC and announce
- set up nexo-base cpuonly pytorch container
  - reproduce success
- Update Singularity instructions to include how to submit jobs to bsub ( and slurm)

Week of 6 July

CO
- ask Debbie to speak at next CO meeting
- attend first half of DIA meeting
DESC DM
- Write agenda, note take and attend DESC DM / DA
NERSC
- Back up Run3.0 sim data to HPSS
- Back up truth catalogs to HPSS
- finish setting up dustmaps
- add v20 stack-sims image
- setup Gen3 repo in postgres (started)
- back up catalogs https://docs.google.com/presentation/d/1GErsSx1zNKtshnfWocCgUPxbmBtTkSuIHKUzAbx9Ae0/edit#slide=id.g73aaf4c5c5_0_3
- Copy out July 7th version of metacal DR6a merged tract parquet files
- Update desc-python & desc-stack for pyarrow, dask, and distributed: https://github.com/LSSTDESC/desc-help/issues/25
CC
- Write up June meeting notes for DESC
- Check on black lives matter Slack channel
- follow up on update to OMP
- Follow up on updates to Membership Policy
Fermi
SLAC
nEXO
- attend Slac computing discussion
- attend software meeting
- Update nexo-base to use py3.7
- Update nexo-base to install pytorch cpuonly
- Update nexo-base to install pytorch with cudatoolkit == 10.0 for SLAC compatibility
- Update singularity doc to include installation at SLAC instructions
- Update Singularity instructions to include how to submit jobs to bsub ( and slurm)

Week of 29 June

CO
- Finish up slides for DC2 update for WL and present
- Help plan Gen3 sessions at collab meeting
DESC DM
- Announce, write agenda, and note take DESC DM meeting
NERSC
- Count up data sizes of all Run 2.2 and 3.0 FITS
- fverify all Run3.0 files, update sqlite tracking db and notify Fabio we are ready to transfer
- Fix up desc-bleed and make sure build is scheduled and cron job set up to import into shifter
- set up rsync of object catalog area at CC (set up cron job to continue daily updates)
- set up rsync of metacal
- finish setting up dustmaps
- add v20 stack-sims image
- set up DR2 calexps for descdm
- Clean up descim CSCRATCH
- Back up truth catalogs to HPSS
CC
- Send Membership Committee Annual report to CC
- Follow up on updates to Membership Policy
- Check on SBPC draft to make sure it is not publically available.
- Follow up with SRPC to make their google doc draft as a draft
SLAC
Fermi
nEXO
- add more users to GitHub repo
- add more users to nexo UNIX group and look into ongoing inconsistent group membership on RHEL6 and centos7
- inquire about SLAC backup policies

Week of 22 June

CO
- attend CI interviews
- set up desc-stack-sims for v20 - but first perhaps sort out which sims weekly to use with it (22 or 24)
- set up desc-stack for v20 just create the tag, don't make it the productoin desc-stack
DESC DM
- scrape metacal logs and ?,?_nImage.fits for exposure info
- Setup notebook to look at data
NERSC
- clean up descim area
- back up truth catalogs
- rsync object catalogs and coadd from CC
- setup rsync cron job
- Handle desc-help #22 copying files for Sam
- set up cron job for desc-python-bleed
CC
- Write up June meeting minutes for DESC
SLAC
Fermi
nEXO
- attend software meeting

Week of 15 June

CO
- attend CS meeting
- attend DESC-wide meeting
- attend DC2 meeting
DESC DM
- setup CI for desc-drp-stack
  - add test for numba
  - add pyccl test for desc-stack
- set up desc-stack weekly images
- Look at GH actions environment differences: https://github.com/actions/virtual-environments/blob/ubuntu18/20200604.1/images/linux/Ubuntu1804-README.md
- merge PR#221 in obs_lsst
- Look into adding configs to IPP
- Join Snowmass mailing lists and slack channels
- add dustmaps packages to desc-stack (and possibly desc-python)
NERSC
- https://github.com/LSSTDESC/desc-help/issues/19
CC
- revise EDI committee member invite
- send EDI committee invites
- Set up June meeting and write agenda
- announce EDI committee
- attend CC meeting
- Update trello
- Retrieve GoogleForm response and send to CC
SLAC
- add DESC computing doc to KIPAC github
Fermi
nEXO
- Livermore workshop
- Meet with Lisa
- Update SLAC wiki page to fix broken links
- Try setting up LHC libraries again
- Update SLAC wiki page doc

note stack-jupyter:prod was pointing to v19_0_0-v3r1 moving to v19_0_0-v6r1

Week of 8 June

CO
- attend SSim meeting
DESC DM
- Set up and attend DR2 processing meeting
- FInish setting up DR2 repo
- Meet, set up agenda, note take, to discuss DR2 processing at NERSC
- Update Run3.0i Confluence page to include pointers to data and software versions
- Validate Run3.0 y1-y3 fits files
- Update desc-drp-stack image to include llvmlite 0.30.0
NERSC
- Create ticket to help user with UNIX group membership
CC
- Call Dawson vote for EDI
- Announce end of vote for PubBoard
- Create poll for June Meeting
- Help draft strike statement
SLAC
Fermi
- Cancel software meeting
- Rebuild 20-10-04-gr07 to compare lib/<variant> vesus lib structure obtained from Jenkins versus local build
nEXO
- attend sims meeting
- add another user to the nexo group
- closeout SLAC IT ticket concerning group membership across rhel6 and centos7
- Investigate NEST submodule failure - the branch with the commit was deleted
- Test Nest update PR 141 and merge and inform Jason Bane of fix

Week of 29 May

CO
- Write up agenda, attend and note take for CO meeting
- Attend Leadership meeting
- Listen in to WL meeting
- Reorg DC2 Confluence page: DC2
- DC2 Hack Day June 5th
  - Go over Confluence Doc
  - check on desc-dm-dc2 repo and update any areas to use CFS and add in Run2.2i
  - Check on DC2-Analysis
  - Transfer DR6 object catalogs to NERSC
- Provide desc-stack-weekly
DESC DM
- Announce, write agenda, note take DESC DM meeting
- set up new desc-drp-stack repo and got new v19-v2 docker image and wrote up doc for Tom
- Started setting up new DR2 repo, need to request new quota and figure out how much room DR2 might take up.
NERSC
- Attend Super facility Brown Bag
- Transfer registry.sqlite3 file from CC after DR6 calexp transfer
CC
- Post May CC meeting Confluence Page
SLAC
- Attend Virtual Residence when possible: http://www.oscer.ou.edu/virtualresidency2020.php
Fermi
- annouce software meeting
- Check on new "externals" directory for M.E.
nEXO
- Attend sims meeting
- Copy over more chargeSim files to SLAC
- Set up Doxygen for Lisa
- Provide example running 2 instance of the same service
- Test CMake ExternalData_OBJECT_STORES
- Announce SLAC FTP site
- Provide incident example
- send Tony Dockerfile
- update Dockerfile for Doxygen

Week of 25 May 2020

CO
- Attend CS meeting
- Clean up desc-python doc and yaml files
DESC DM
- Announce, write agenda, note-take DESC DM
NERSC
- Request full copy of DR6 calexps from CC
- Follow up on outstanding NERSC account requests
- Attend Superfacility Brown Bag - SPIN
- Request early access to rancher 2
CC
- Finalize MCPs registration wording
- Summarize May meeting minutes for DESC
- Provide CC summary for DESC newsletter
SLAC
Fermi
- Annouce software meeting
nEXO
- copy chargesim data files to SLAC
- Set up doxygen for Lisa
- Advertise FTP area and updated CMake files

Week of 18 May 2020

CO
DESC DM
- Announce, write agenda, write notes for DESC DM meeting
NERSC
- https://github.com/LSSTDESC/desc-help/issues/17
- Add GCRCatalgos master to desc-python-dev
- Attend SuperFacility Brown Bag
- Attend SIGEUF
- Update desc-stack and desc-python with new v0.17.0 GCRCatalogs
- Inform DESC user when over Computing allocation
CC
- attend CC meeting (note take on the side)
- WRite up DESC summary for meeting
- Work out PubBoard rotation
- Respond to Membership Committee email concerning annual report
- Update draft for note to Meetings Committee concerning upcoming registration
- update CC trello
- Write up possible update to MCP request on registation form
SLAC
Fermi
- announce s/w meeting
nEXO
- Meet with Lisa
- Poke Tony about Jenkins builds and web accessible area for doxygen output
- Set up new doxygen for Lisa and test it out

Week of 11 May 2020

CO
- Announce, write agenda, and note take for CO Meeting
DESC DM
- Announce, write agenda, and note take DESC DM meeting
- Refresh HSC Gen3 example at NERSC
- Demonstrate gen2togen3 conversion using DMstack w_2020_19 and Run2.2i data
NERSC
- Continuing move to shared area
  - copied PhotoZDC2 catalogs: https://github.com/LSSTDESC/gcr-catalogs/issues/424
  - Copy old and new Buzzard catalogs
  - descqa/v1
  - SSim data
- Transition WGs to CFS
  - CL
  - CS
  - LSS - created new area and requested current owners move content
  - PZ
  - SSim
  - TJP - removed content as agreed
  - WL - created new area and in contact with conveners to determine who will handle the data move
- Prepare for WG NERSC migration
  - Announce on desc-announce
  - Alert all WGs
  - Back up old and new descqa areas
- attend NERSC brown bag series - New Data Management
- Inquire about NERSC DB services
CC
- write up agenda for May 18th meeting, post entry on DESC Calendar
- attend SBPC DESC-wide meeting
- write up draft announcement of SRPC status
Fermi
- announce Fermi Software meeting
- Check for L1 branches
SLAC
nEXO
- attend sims meeting
- populate new SLAC FTP area
- add new user to nexo group at SLAC
- add new users to GH organization
- Update CMake to use new FTP area
  - test at SLAC
  - test at LLNL
  - test on laptop

Week of 4 May 2020

CO
- Alert WG conveners and DC2 analysis czars about upcoming CO discussion concerning validation efforts
- Complete Q2 report
- attend leadership meeting
- attend CS meeting
DESC DM
- Update GitHub workflows for DC2-imsim docker images, to work around disk space issues on GH Actions
- Announce, write agenda, attend, note take for DESC DM meeting
NERSC
- Install new gcr-catalogs 0.16.1
- Help write up draft CFS directory organization
- Test use of desc account in Globus
- Move Buzzard-4 into new CFS shared area
- attend External User Group meeting
CC
- Set up WhenIsGood poll for next CC meeting
- write agenda for next meeting
- Contact SRPC to determine next steps
SLAC
- Weekly SLAC meeting
Fermi
- announce offline s/w meeting
nEXO
- meet with Lisa
- Add nexo-base:current tag and update documentation
- Attend sims meeting
- Look at Fireworks
  - https://www.nersc.gov/assets/Uploads/JainNERSCUserDay2.pdf
  - Debbie's PDF

Week of 27 April 2020

CO
- attend SSims meeting
- inquire about missing w_2019_42 sims on CVMFS
  - https://github.com/airnandez/sw-lsst-eu/issues/3
- request v19 lsst_distrib be pinned on CVMFS
- Test reported issues with ipympl at NERSC and CC
- Upgrade desc-python to include tabulate, flake8, new sacc, healsparse
- Report Missing sims w_2019_42 on CVMFS: https://github.com/airnandez/sw-lsst-eu/issues/4
- Confirm removal of dev installs on CVMFS: https://github.com/airnandez/sw-lsst-eu/issues/5
- Write up quarterly report
DESC DM
- https://jira.lsstcorp.org/browse/DM-24517 Keep an eye on progress
- Finish reorg of Y4/Y5 checkpoint files
  - Check final counts
- Copy Y4/Y5 checkpoint files to tape
- Announce, write agenda, note take and run DESC DM meeting
NERSC
- Transition WG to CFS
  - CL
  - CS
  - LSS
  - PZ
  - SSim
  - TJP
  - WL
- desc-help
  - https://github.com/LSSTDESC/desc-help/issues/12
CC
- Write up message to Membership Committee
- Review scheduled council activities for April/May/June
Fermi
- announce, attend and note take for software meeting
- Merge astro changes from igrf_updates branch into L1 and test in Jenkins
- Release GR including new astro
SLAC
nEXO
- attend sims meeting
- set up new area for shared data files
- Update Cmake build to down data files
- Update CMake to require sniper >= 1.3
- add new nexo user to nexo group at SLAC
- add new nexo user to nexo group at SLAC

Week of 20 Apr 2020

CO
- announce, run, and note-take CO meeting
- Update Zoom infor for all Computing meetings in Calendar and some Confluence pages
DESC DM
- https://jira.lsstcorp.org/browse/DM-24517 Keep an eye on progress
- DESC DM meeting
NERSC
- Attend NUG on 4/22
- desc-help
  - https://github.com/LSSTDESC/desc-help/issues/12
- Finish reorg of Y4/Y5 checkpoint files
  - inquire how to handle Grid duplicates
CC
Fermi
- announce Fermi software meeting
- Help Nicola will additional GR/astro building questions
SLAC
nEXO
- attend sims meeting
- Add new users to GH and provide instructions to apply for accounts at SLAC & LLNL
- Locate new location for data files at SLAC?
  - ticket opened
- Address SNIper filtering to cut down output file size
- Update Cmake build to down data files
- Update CMake to require sniper >= 1.3

Week of 13 Apr 2020

CO
- attend CS meeting
- update sacc in desc-python
- update gcr-catalogs in desc-python
- fix up pip includes in desc-python
DESC DM
- announce, write up agenda, note take DESC DM mtg
NERSC
- crack all y4 y5 files and extract checkpoints
- Store y02-missing instance catalogs to HPSS
- Store cosmoDC2 to HPSS
- Store y02-missing instance catalogs to HPSS
- Store cosmoDC2 to HPSS
CC
- Look over notes for meetings committee
- release April Zoom mtg notes
Fermi
- announce, attend, note take for software meeting
- inquire about status of new astro
- help Nicola get going with developer tools
  - Tag GRcontainerSettings to capture 2018 changes on master
  - Test new Jenkins build
  - Install miniconda2 and add latest fermi-repoman and scons
  - Test command line build of GR
SLAC
nEXO
- attend sims meeting
- Finish setting up new docker/singularity image and documentation
- Update nexo-base image on GH Actions
- Continue trying to get nexo builds with LCG libraries to work

Week of 6 Apr 2020

CO
- attend SSims meeting
- Attend DC2 meeting
- Attend Leadership meeting
- test updated desc-python with fast3tree fix
  - release v0.0.24
- update sacc in desc-python
- fix up pip includes in desc-python
DESC DM
NERSC
- Confirm Y4-Y5 raw transfer was successful and inform Fabio
  - rollback 11 failed jobs and redo transfer to HPSS
  - Confirm success
- Extract Y4 Y5 checkpoint files, organize and push into HPSS
- Contact projecta www owners and tell them about projecta transition
- Contact WG conveners about NERSC projecta => CFS transition
- Start up Science Gateway page: https://confluence.slac.stanford.edu/x/X4Z2E
- Checklist projecta => CFS transition: https://github.com/LSSTDESC/desc-help/issues/11
CC
- Write agenda, send reminder and attend monthly CC meeting
- write up DESC summary
- Help Dominique pinpoint issues cloning DMstack env on CVMFS at CC
SLAC
- Complete Cyber security Training
Fermi
- Announce, attend, and take notes for Offline S/W meeting
nEXO
- attend sims meeting
- Test updated nexo-base docker image removing NEST, going to Py3 and LLNL externals
- setup SLAC with LCG 96python3

Week of 30 Mar 2020

CO
- Write up agenda, run CO meeting
  - discuss desc-python
- Announce new Confluence page for DESC envs on desc-leadership
- write up adding user packages to desc-python
  - test nbodykit
- write up adding user packages to desc-python
  - test nbodykit
- Help CC figure out how to install and maintain desc-python
DESC DM
- Validate Y2-hole FITS files and coordinate transfer to CC
- announce, write agenda and run DESC DM mtg
NERSC
- Meeting with Rollins
- Investigate nbody dependencies
- Transfer y4-y5 raw files to HPSS
- finish setting up testdata_ci_desc_gec repo
  - empty
  - Run2.2i data
CC
- Set March/April meeing, write agenda
Fermi
- Announce, attend, note take Offline S/W meeting
SLAC
nEXO
- attend sims meeting
- New nexo-base docker image with NEST and latest sniper
- Provided updated externals at SLAC
  - find fix for ROOT compilation error with gcc 4.8.5 and LZMA
    - Get SLAC to install lzma-devel
    - move to LCG externals

Week of 23 Mar 2020

CO
- ask Erin to present at next CO meeting
- attend WL meeting and present DC2 update
- Write up confluence page listing available DESC environments and POCs
- Attend CS meeting
DESC DM
- announce, write agenda and run DESC DM mtg
- set up testdata_ci_desc repo with a portion of Run2.2i data
- Help coordinate Y4/Y5 transfer from CC with Fabio
- Work out updating the SQLite tracking DB for Y4/Y5 and Y2 hole
- Help coordinate Y2 hole transfer to CC
- Write up wiki describing how to set up a DM Gen2 repo
NERSC
- Ask about jupyter widgets
- release new conda-forge based desc-python
- Test new desc-python and adding user packages
CC
- ask meetings committee about data retention
- plan next April CC meeting
SLAC
Fermi
nEXO
- attend sim meeting
- Merge updates to lightmapsvc
- Help new nexo user set up and use docker and latest nexo-offline

Week of 16 Mar 2020

CO
- Prepare slides for Monday's WL meeting
DESC DM
- Attend DM Alg Workshop remotely Tues - Thur 8:45 AM - 2:30 PM PT
- Validate Y4-WFD fits files and request transfer to CC
- Update DC2 Run2.2i Confluence doc for DR3
NERSC
- Move Y4 WFD checkpoint files to HPSS
- Copy Y2 Hole instance catalogs to HPSS
- Respond to Rollins questions about use of key/value API within jupyter notebooks
- deploy new desc-stack production bersion vased on v19
CC
- Update newsletter Feb/March summary
SLAC
- Complete Annual Security Training
Fermi
nEXO
- Attend simulation meeting
- Help Ako with Sniper Questions
- Update LightMapSvc to fix example algorithm to allow setup once during initialize
- Update LightMapSvc to retrieve data from TH2D
- Complete LLNL Security Training

Week of 9 Mar 2020

CO
- Announce, write agenda, note take and CO meeting
DESC DM
- Announce, write agenda, run, note take DESC DM meeting
NERSC
- Pull Y2 WFD instance catalogs from tape
CC
SLAC
- Attend Data Science Training
Fermi
nEXO
- Attend simulation meeting
- Chat with Lisa about Confluence, JIRA, GitHub, etc
- Open SLAC ticket to help nexo user set default shell to bash
- Test git submodules on nexo-offline

Week of 2 Mar 2020

CO
- Finish CI slides
- Present CI work at DESC seminar
- attend CS meeting
- update desc-dc2-dm-data repo with DR3 directory
- start gcr-catalogs updates for object and meta catalogs
- Update desc-help README to provide pointers to other help faciilties
DESC DM
- announce, write up agenda, run, and note take for DESC DM mtg
NERSC
- Clear off Y1 and Y2 chkpoints, freeing 146 TB for y4-wfd transfer from scratch
- Prepare subset of DR4 on CSCRATCH for general DESC release
- Request ownership transfer of DR3 drp data
- Update Confluence doc for NERSC account
- Test ability of DR3 drp data on CFS to run on batch - fails, same locking issue as projecta
CC
SLAC
Fermi
- attend software week sessions
- Test Jenkins builds of GR
  - Report problems with GR builds to Brian
  - retest Jenkins manual triggering of builds
  - Test binary tarball at SLAC - seems to work
nEXO
- attend software meeting
- Inquire about transfer speeds between IHEP and SLAC
- Provide skeleton light map service

Week of of 24 Feb 2020

CO
- Prepare slide for Computing Infrastructure Talk at DESC Seminar
- attend SSIM meeting
DESC DM
- Attend DC2 meeting
NERSC
- Set up space for DR3 transfer from CC
- update desc-stack to resolve: https://github.com/LSSTDESC/desc-help/issues/2
- Open NERSC ticket concerning users with trouble starting up jupyter
CC
- Attend and note take for the meeting
- Update Trello with CC activities
- Write up DESC Summary of Feb mtg
- Write up Feb DESC newsletter submission
- Fix up CC Confluece area containing Council Minutes - past minutes were spread between 2 separate pages.
SLAC
Fermi
nEXO
- attend Tuesday software meeting
- Inquire at SLAC to use gridFTP for data transfer
- Work with SLAC to gain CVMFS access to IHEP filesystem

Week of 17 Feb 2020

CO
- write agenda, note take for CO meeting
DESC DM
- write agenda, run meeting,note take
NERSC
- add graphviz to desc-python
- Store Y3-wfd checkpoint files on HPSS
CC
- Start up agenda page
- Poll for meeting time
- Follow up on
  - PubBoard Rotation
  - SBPC
SLAC
Fermi
- announce, note take for software meeting
nEXO
- help track down memory leak
- attend sims meeting
- install updated external libs at SLAC
- update cmake install to handle downloading of ancillary files

Week of 10 Feb 2020

CO
- attend DESC leadership mtg and take notes
- announce new desc-help github repo
- attend and note take for CS mtg
DESC DM
NERSC
- Copy Y4 instance catalogs back off tape to CSCRATCH
- Set up 1 TB area for PLASTICC
  - submit NERSC ticket to sort out how to set quota
- Refresh NERSC Getting Started Confluence page with new NERSC links
- install w_2020_06 in shifter for Michael
- address ipyml issue in jupyter
- contact NERSC about htar compression
CC
- attend SRPC - CC meeting and take notes
SLAC
nEXO
- Look over commit history on nexo-offline branches
- add outside collaborator to one nexo repo

Week of 3 Feb 2020

CO
- Generate new stack-sims-cat using w_2019_42 and adding in pyarrow for Bryce & Joanne
- Attend SSim mtg
DESC DM
- Gen3 meeting
NERSC
- transfer y02 instCat to hpss
- transfer y05 instCat to hpss
- transfer y06 instCat to hpss
- Request increase to descim CSCRATCH quota
- Inquire about new user LSST group membership
- Inquire about PI notification concerning users over job allocation
- Transfer checkpoint files to HPSS
  - Y1
  - Y2
  - Y3
CC
- set up meeting with SRPC
SLAC
- Attend group meeting
Fermi
- announce, attend, notetake Software mtg
nexo
- attend software meeting
- Write MC filter Alg
  - PR
- Coordinate SLAC adding JuNO path to CVMFS
- Meet with Lisa
- Add new SNiPER service to read in ROOT TH2 LightMap

Week of 27 Jan 2020

CO
- write up agenda, run, and note take during CO meeting
- Set up desc-help repo and explore issue templates
DESC DM
- write up agenda, note take during DESC DM & DA meeting
- validate Y3 sim data, update tracking DB and request transfer to CC
NERSC
- Consider moving projecta content to CFS
- Look over CFS content from project area
CC
- meet with Yao to discuss to do list
Fermi
nEXO
- attend software meeting

Week of 20 Jan 2020

CO
- Set up workflow for gcr-catalogs to create desc-stack:test and desc-python:test
- Set up workflow to move desc-stack:prod to desc-stack:old
DESC DM
- Set up mtg with Ben, Tom, Jim, Adrian
- Pull out y4 instance catalogs for James P to transfer to Grid
NERSC
CC
- Organize, pick up, deliver food for cc-mgmt meeting
- Set up mtg with Yao & Eric
Fermi
- announced cancellation of Software mtg this week
nEXO
- Update NEST to latest master in nexo-base:latest
  - testing failed due to missing valgrind which seems to be required

Week of 13 Jan 2020

CO
- Provide GCRCatalog workflow triggered off master to produce desc-stack and desc-python test envs
- Set up new desc-stack repo in LSSTDESC using v19.0.0
- Introduce Docker build of desc-python in LSSTDESC
- Update desc-dc2-dm-data repo with all Run2.1i and Run2.2i data locations
- Update desc-dc2-dm-data repo with all Run2.1i and Run2.2i data locations
DESC DM
- Validate straggler y1-wfd and y2-wfd visits, update tracking DB and request transfer
- Write up slides, attend WL meeting and provide DESC processing update
NERSC
- Prepare new CFS area for DESC with directories and permissions
  - Inquire about area ownership: https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=7fb44483db1600907d2a300f7c961932%26sysparm_view=ess
- Refresh Run2.1i CSCRATCH copy
- Refresh Run1.2i CSCRATCH copy
- Address inode issue for descdm CSCRATCH
- Start moving agn_ckpts to HPSS
  - inquire about stalled HPSS jobs: https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=586cd487db5600907d2a300f7c961926%26sysparm_view=ess
  - y1-wfd
  - y2-wfd
- Finish moving instance catalogs to HPSS
  - y02
  - y05
  - y06
DESC CC
- Update CC plenary slides
- Handle food for CC-Mgmt meeting
- Remind desc members to review draft SRP and provide comments
- Help plan CC Town Hall
  - zoom recording?
Fermi
- Announce, note take and attend weekly software meeting
nEXO
- attend software meeting
- Provide fresh externals installation at SLAC for latest nexo-offline
- Meet with Samuele to discuss skeleton recon SNiPER alg
- Write up draft organization for charge and light reconstruction

Week of 6 Jan 2020

CO
- Write agenda, announce, and notetake for CO meeting
- Provide Debbie some advance dates of DESC needs
- Q1 report
- survey paper validation section
- Prepare for Collab Mtg
- Follow up on DDF checklist
DESC DM
- Validate y2-wfd and update sqlite tracking db and request transfer to CC
- Announce, write agenda, attend and note take for DESC DM meeting
- Transfer tract 3828 and 3829 to lsst-dev
  - and ancillary data including CALIB, ref_cats
- Free up space in descim CSCRATCH
  - y4
  - y3
  - y1
  - y2
  - y5
- Free up space in projecta
  - agn-checkpoints for y1-wfd
NERSC
- Refresh RUn1.2p
- Refresh Run1.2i
- Cron job for CSCRATCH?
Collaboration Council
- Start preparing CC update for Collaboration Mtg
- Help plan TownHall
Fermi
- Announce, attend, and note take offline software meeting
nEXO
- Attend software meeting
- Read, respond, and provide Fermi's Gaudi experience for Vladimir's project overview

Week of 16 Dec 2019

CO
- Section 7.2 DC2 survey paper (cross-validation)
- Attend Wednesday Leadership Mtg
DESC DM
- Ingest Run2.1i reference catalog using v19.0.0 and obs_lsst dc2/run2.2 branch
NERSC
- Update desc-stack, making sure it pin python 3.7
- Finish SPIN implementation for running integration tests using CVMFS installations
- Help Anze figure out how to use emacs after loading desc-python
Fermi
- Create any necessary JIRAs for upcoming GR astro update
nEXO
- attend Sunday evening telecon with SNiPER representatives

Week of 9 Dec 2019

CO
- Attend SSim meeting
DESC DM
- write up agenda, run, and note take DESC DM meeting
- validate y1-wfd fits and centroids
  - Request transfer to CC
NERSC
- Attend SPIN Hack day
- Attend NUG: https://www.nersc.gov/users/NUG/teleconferences/nug-webinar-dec-12-2019/
- Test gcr-catalogs uninstall and clean up old catalogs files in desc-python
  - update desc-python
  - updated desc-stack
Fermi
nEXO
- attend 3 hour (until had to go) workflow meeting
- Attend software meeting

Week of 2 Dec 2019

CO
- Announce, write agenda, run, and note take CO meeting
DESC DM
- Announce and write up agenda for DESC DM meeting
- Update setup script for IPP pipelines
- Update obs_lsst dc2/run2.2 branch for w_2019_47
NERSC
- attend SPIN training Thus and Friday afternoons
Fermi
- announce software meeting
nEXO
- attend software meeting
- Read Vladimir's work plan and comment

Week of 18 Nov 2019

CO
- prepare list of current desc-python packages to circulate, investigate CC's conda env, and open GH issue
- Consult conveners concerning who should offer GCR analysis commentary at next mtg
DESC DM
- Announce, write up agenda, note take and run DESC DM meeting
- Provide Tom and Ben an updated dc2-run2.1i docker image with astropy 3.2.3
NERSC
- Deal with Michael's questions about building against CVMFS using recent stacks
  - Upload and test recent LSSTSQRE docker image into Shifter at NERSC and worked out building
  - Tested CVMFS at CC
  - Work with DM to determine how to set ABI flags for scons builds of local packages
- Dealing with private DM NERSC questions: Bruno, Humna
- Continue to work on mrau's inode use - down from 96% to 69%
- Move more files to HPSS
- Update desc-python-dev to include CLMM and cluster toolkit.
  - indicated that we need release tags on both before I add this to desc-python
  - Note the need for GSL and adding gsl-config to PATH
Fermi
- Announce, attend and note take for software meeting
nEXO
- attend software/sims meeting
- rebase Brian's PR and merge because no one knows how to use git on nexo
- add doxyfile for Tony to test Doxygen generation in Jenkins
- update nexo-base env to include new NEST master
- - Add GitHub actions workflow
  - Release new docker image

Week of 11 Nov 2019

CO
- PR, tag, and create docker image for lsstdesc/dc2-imsim:Run2.2i-production-v1
- attend data access meeting
DESC DM
- Attend DM bootcamp Tues - Thur 2-5 PM ET
NERSC
- Deal with projecta inode quota which is currently at 96%
- continue moving files to HPSS
Fermi
- announce and note take for offline s/w mtg
nEXO
- NEST PR
  - test build at SLAC
  - Open new GH issue for cmake build failure

Week of 4 Nov 2019

CO
- write agenda, announce and note take for CO mtg
- Update Nov, Dec, Jan DESC calendar for upcoming holidays
- Prepare new stack-sims-cat docker image for instance catalog generation
DESC DM
- ~~write agenda, announce and note take DESC DM meeting~~
NERSC
- Install pyCCL 2.0.1 in desc-stack and desc-python
- Meet with Debbie to go through NERSC user survey
- Continue moving data onto HPSS
Fermi
- announce mtg
nEXO
- attend software meeting and provide update
- Help Caio with SLAC install
- Deal with doc PR

Week of 25 Oct 2019

CO
- attend Data Access meeting and note take
- attend CS meeting
- write draft DDF checklist
- Transfer Grid data from CSCRATCH to projecta space
- continue transferring data to HPSS from projecta
- Generate new sims docker containers as needed
  - imSim v0.6.1
DESC DM
- process ALCF, Cori, and Grid validation test data, through makeFpSummary
NERSC
Fermi
- announce, attend, and note take for software meeting
nEXO
- Continue answering C++ and git questions
- setup external library area at SLAC using same versions as LLNL
- attend software meeting

Week of 14 Oct 2019

CO
- Write up Agenda and run and note take for CO mtg
- Attend Data Access meeting
- Attend SRM milestones meeting
DESC DM
- Edit Parsl DRP planning doc
- setup up empty Run2.1.1i repo for testing on CSCRATCH and HPSS
- Transfer NCSA Run2.1.1i processing to NERSC
NERSC
- Transfer more data to HPSS
Fermi
nEXO
- Handle C++ questions
- Start to Investigate strax

Week of 7 Oct 2019

CO
- Attend Grid meeting
- Attend CS meeting
DESC DM
- Announce, write up agenda, note take, and run DESC DM meeting
- Help Tom out with DM questions
NERSC
- Help Bela to run shifter images in batch
- Start transferring more data to HPSS
Fermi
- Announce, attend, and note take for software mtg
- Handle M.E.'s questions concerning new GR release build
- Remove Charlotte's name from the workbook where necessary, per her request
nEXO
- Attend software meeting
- Deal with C++ questions
- Test ROOT writing performance to address Soud's inquiries

Week of 30 Sept 2019

CO
DESC DM
- Oct 2nd meeting agenda and note taking
- Meet with Ben and Tom to discuss Parsl
- Discuss Data Transfer with Fabio, Antonio, Dominique, Tom
- Review Johann's PR from dc2/run2.1
- Create new obs_lsst branch for dc2/run2.2
- Finish writing Quarterly Report
NERSC
Fermi
- attend, note take for offline software
- opt build for GR 20-10-04-gr06
nEXO
- software meeting

Week of 23 Sept 2019

CO
- Prepare agenda and run CO meeting
DESC DM
- Finish no sky correction Run2.1.1i process and make it available to Jim
- Update weekly 39 docker image to remove current tag
NERSC
- Migrate more data to hpss
- Prepare to copy instance catalogs back off tape
Fermi
- Build new L1 GR
- See that Jenkins is fixed to build L1
- offline software meeting
nEXO
- attend sims meeting
- prep singluarity image for OSG
- update docker image with new externals

Week of 16 Sept 2019

DESC DM
- instance catalogs for Run2.1i do not need to be regeneratd? how does that affect what is stored to HPSS?
- take a look at throughputs tags in docker image
- Generate new imsim Docker image with new Galsim
- Set up agenda and run DESC DM portion of Wednesday meeting
- set up docker image for use in SRS pipeline
- validate DDF data from Run2.1i y5
CO
- Make Confluence pages consistent and update content
- Update DESC Calendar
- SChedule Data Transfer Discussion
  - draft agenda: Sep 30 10 AM PT
NERSC
- set up instance catalog generation software
- NaMaster in desc-python
- update for GCR-catalogs
- Help Bruno & Bob access Run2.1i data
- Coordinate with NERSC to see if we can use Globus transfers into HPSS and transfer owernship
- Continue copying instance catalogs and Run2.1i data to HPSS to free space

Week of 3 Sept 2019

DM-DC2
- finish processing 4 tracts of Run2.1.1i through multiband
- run merge_tract_cat to produce HDF files on all the 14 tracts of Run2.1.1i
- rerun processing with skyCorrection
- finish documenting running at NERSC
- Converse with Ben as questions arise as he works thorugh implementing DRP in parsl
DESC
- Check on imsim Docker image and the throughputs tag
- Install NaMaster in docker image
NERSC
- Check on CSCRATCH before outage and transfer to tape
  - Back up Run2.1.1i 4 tract nersc processing
  - recopy RUn1.2i and reapply symlink fix
  - Create back up of Run2.1.1i, Run2.1i on projecta for users to use during outage
- Inquire about Jupyter inaccessiability
SLAC
- install DMstack v18.1.0 for IR2
  Jira
  server SLAC National Accelerator Laboratory
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1433
Fermi
- Announce, run, and note take Fermi mtg Sept 4th
- Tag L1 fitsGen after rebasing with master and create a new L1 GR tag
nEXO
- write up tutorial for python access
- retrieve all requirements for OSG (onhold)
- meet with Lisa
- merge Emma's WT branch into issue_31 after resolving conflicts
- Present "How to create a Data Object and write it out in SNiPER"
- attend nEXO software/sim meeting

PCW

Catch up with Scott and Chris about the throughputs tags

Week of Aug 5 2019

DM-DC2
DESC
- Retrieve Y2 DR1B instance catalogs from HPSS
- Attend DC2 validation/next steps meeting
- Document
NERSC
- Poke NERSC about file system status
- Demonstrate running SRS workflow at NERSC
- Continue clearing projecta space
SLAC
Fermi
- announce, attend and note take for Fermi Offline mtg
nexo
- attend sims/software meetign
- present update on nexo-offline status
- Help update doc on Docker
- contact Tony about setting up Jenkins builds
- Work through Save All Events problem in nexo-offline
- finish testing cmake in docker
- Look at pySniper algorithm

Week of July 29 2019

DM-DC2
- announce, write agenda and note take for weekly meeting
DESC
- attend DC2 meeting
NERSC
- Recover from Cori upgrade
  - Update desc-python to use gcc 7.3.0
  - Adjust setup_current_stack and setup_current_sims to use new gcc 7.3.0 build of w_2019_19 on CVMFS
    - test new gcc 7.3.0 build
SLAC
Fermi
nEXO
- attend sims/software meeting
- Run nexo-offline and nEXO_MC using TPCVesselU.mac
- attend LLBL cmke training remotely

Week of July 22 2019

DM-DC2
- Open JIRA concerning throughputs and semantic tagging
  - https://jira.lsstcorp.org/browse/SIM-2620
- Ask Michael about identifying a tract to retain on projecta from instance cats through object catalogs
- Retrieve Y6 instance catalogs from HPSS for James P for use in grid processing
- Write up Draft Confluence page for DESC DM Change Requests
  - DESC Change Requests for DM
DESC
- add NaMaster to desc-stack
- consider adding sacc to desc-stack
- add cl-classmod environment to desc-stack? or some subset.. or update the classmod env
NERSC
- install new gcr-catalogs 0.14.1 in desc-python & desc-stack
- Help Humna with her ongoing desc-stack problems
  - https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=48035aeadbbe7300200f7d321f961962%26sysparm_view=ess
- Continue moving WFD instance catalogs to HPSS
SLAC
Fermi
- Announce, attend, and note-take for Fermi Offline Meeting
nEXO
- attend software/sims meeting

week of DESC Meeting

DM-DC2
- Help set up Tom to parsl sfd
- New confluence page for CCB DESC requests
Python
- NaMaster
- sacc
- DIA kernel
NERSC
- Install gcr-catalogs 0.13.1 and 0.14.0 into desc-python and desc-nersc for Hack Day
- Check on ability to use later gcc version with CVMFS installation
- Doc use of jupyter compute nodes
- Fix up DESCPYTHONPATH
- doc use of batch for desc-stack
- instance catalog from tape Y6

Week of July 1 - 12 2019

DM-DC2
- Write agenda, run and note take DM-DC2 meeting for both July 1 & 8th
DESC
NERSC
- update lsstdesc/nersc to remove PYTHONSTARTUP for desc-stack
- Document the use of DESCPYTHONPATH
SLAC
Fermi
nEXO

Week of June 24 2019

DM-DC2
- Listen to recording of weekly meeting and transcribe notes
- Plan any DM-DC2 sessions for collaboration meeting
- Document new DR naming convention
DESC
- Deal with docker removal from lsst-dc10, find another place to build
NERSC
- Continue clearing out the instance catalogs to HPSS
- Quality check Y5 simulated data and update the tracking DB
SLAC
Fermi
- announce, attend, and note take Fermi Offline Software meeting
nEXO
- attend collaboration meeting at Stony Brook
- add Cards to nexo-offline master and u/heather999/issue_31
- Testing running docker image with nEXO_MC
- Run nexo-offline and nexo_MC TPC jobs for regression testing
- Take a look at historgrams generated
- Keep track of Soud's progress with Clustering and Reconstruction port

Week of June 17 2019

DM-DC2
- https://jira.lsstcorp.org/browse/DM-19985
DESC
- Prepare new w_2019_23 stack sims docker image for imsim group
NERSC
- Continue moving instance catalogs to HPSS
- Alert IN2P3 when there is enough space for y1-y2 calexp and coadds
SLAC
- attend stand-up meeting
Fermi
- announce, take notes, attend offline software meeting
nEXO
- Complete new nexo-base docker image include NEST2.0
- write up slides for collaboration meeting

Week of June 10 2019

DM-DC2
- Write up agenda and run DM-DC2 meeting
- quality check y3-wfd sim data and update tracking db
- Help coordinate transfer for new calexp-v1 to NERSC from IN2P3
- Create branch and PR for forcedPhotCcd_metadata template https://github.com/lsst/obs_lsst/pull/116
DESC
- attend data access meeting
- Prepare updated desc-stack-dev and install at NERSC
- Test desc-stack-dev at NERSC using DC2-analysis notebooks, resulting in the following issues
  - https://github.com/LSSTDESC/gcr-catalogs/issues/327
  - https://github.com/LSSTDESC/DC2-analysis/issues/87
  - https://github.com/LSSTDESC/DC2-analysis/issues/86
  - https://github.com/LSSTDESC/DC2-analysis/issues/85
  - additionally identified the need to add seaborn and extinction ot the docker image
NERSC
- Consider increasing desc CSCRATCH quota
- Copy and remove DC1 deepCoadd to tape from projecta
- Check over y1-y3 instance catalogs on HPSS
  - remove y1-y3 from projecta
- move y7-10 to HPSS
- Help demonstrate cvmfs problems at NERSC
SLAC
- attend group meeting
Fermi
- announce, note take and attend software meeting
nEXO
- attend software meeting
- report on nexo-offline status at software meeting
- Update nexo-env to handle NEST2.0
- Create new docker base image with NEST2.0
- send Samuele/Lisa slide for collab meeting
- check with Tony about status of Jenkins builds

Week of May 28 2019

DM-DC2
- https://github.com/LSSTDESC/ImageProcessingPipelines/issues/92
- Coordinate transfer of 6 months DM processed data from IN2P3
- Set up 6 month data at NERSC under CSCRATCH with the appropriate CALIB, ref_cats
- Make Run1.2x repos readable by later versions of the DMstack
  - https://lsstc.slack.com/archives/C978LTJGN/p1557771460413100
DESC
- Create stack-sims:w_2019_19-sims_w_2019_19 image for use at NERSC
NERSC
- create new docker image based on w_2019_19 plus all desc python packages
  - include stile
  - install at NERSC as desc-stack-dev
  - Update desc-stack-prod
- Update BF Kernel cp_pipe installation for w_2019_19 after Tom's reports that the older installation using w_2019_17 no longer worked
- continue transferring data into NERSC
  - https://github.com/LSSTDESC/DC2-production/issues/343
  - DC2 Run2.1i instance catalogs
  - Remove DC1/DC1-imsim/dithered and undithered
  - Remove DC1/astrometry
  - Start cleaning DC1/DM/DC1-imsim-dithered
    - icExp contains just temp files
    - compare random calexp and deepCoadd fits files using fdiff with the copy in HPSS
SLAC
- attend group stand up meeting
Fermi
nEXO
- attend software meeting
- Get some more info about Simulation/DetSim/Nest
- Add

Week of May 20 2019

DM-DC2
- Write agenda, take notes, and run DM-DC2 meeting
- Welcome Dino and send along some getting started links
DESC
- Open up a new ticket for Darren White
- Attend DC2 meeting and lead discussion concerning what data needs to be available for the analysis groups
NERSC
- Continue transferring data into HPSS
- Open GitHub issue to discuss what can be removed from projecta
SLAC
Fermi
- Announce and note take for offline meeting
nEXO
- attend sims meeting
- setup Soud with u/heather999/issue_31 branch at SLAC
- contact Tony about getting some Jenkins builds going before collab meeting
- Contact Pere and Gabriele C about NEST2 linking errors against G4
- Poke Jason and Samuele about nEXO_MC PR for NEST

Week of May 13 2019

DM-DC2
- Write agenda, take notes,and run DM-DC2 meeting
- Ingest new ref catalog and deliver to IN2P3
- update obs_lsst configs according to Slack conversation: https://lsstc.slack.com/archives/C978LTJGN/p1557935315430600
- inquire about using --show history https://lsstc.slack.com/archives/C2K97E3L5/p1557940644050600
  - https://jira.lsstcorp.org/browse/DM-19804
DESC
- attend data access mtg
NERSC
- Move files from projecta to HPSS to reduce disk use on projecta
  - DC2_ImSim/Run2.0i
  - DC2_ImSim_UKTest
  - Twinkles1
- ask Richard to increase "desc" HPSS quota
SLAC
- install parsl on IR2 in v17_gcc7 env
Fermi
nEXO
- attend software meeting

Week of May 6 2019

DM-DC2
- Write agenda, take notes, and run DM-DC2 meeting
- Update visit DB to mark files and visits tables as quality_controlled and quality_completed, respectively
- Create updated CALIB products tarball including BF kernels
- Update Antonio's visit DB to note that all Y2-WFD visits and files have been quality controlled.
DESC
- attend CI meeting
- attend DC2 meeting
  - review Dominique's slides for DC2 mtg
- attend 3x2 mtg
  - help Francois access Run1.2i force photometry data - only to learn that newer DMstacks cannot deal with the older repos.
- Deal with user questions on #desc-nersc
- Start uploading DC2 data into HPSS to free projecta space
NERSC
- install stile
- Install new python environment compatible with recent DMstacks
- Create new docker image for sims w_2019_17 which will be the base of new jupyter kernels
Fermi
- announce, take notes and attend Fermi offline s/w mtg
nEXO
- attend sims meeting
- Clean up merge of nEXO_MC and nexo-offline
  - compile on SLAC with gcc 7.3 and LCG 94a
  - compile with gcc 4.8.5
  - run some tests
  - submit PR

Week of Apr 29 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Check y2 Run2.1i data for incomplete files
- Ferry sky correction configs into master of lsst/obs_lsst
  - rebase, update comments, squash commits
- Update dc2/run2.1 lsst/obs_lsst branch for use at IN2P3
- Rerun skyCorrection for eyeballer visits
  - Due to decision to skip skyCorrection, this attempt was abandoned
- Generate new set of BF kernels
  - first attempt failed
DESC
- Poked DM about weekly release note page which wasn't working: https://lsst-web.ncsa.illinois.edu/~swinbank/changelog_weekly.html
NERSC
- Update setup scripts at NERSC to use CVMFS
- Update confluence documentation for NERSC software installations
- Install updated cp_pipe for BF kernel generation
- Investigate inode use on /global/project/projectdirs/m1727
  - Poke owners of files
SLAC/Camera
Fermi
- Inquire about software meeting
nEXO
- Test CMake build of SNiPER
- Test CMake build of nexo-offline
- Test CMT build of nexo-offline against CMake build of SNiPER
- Set up LCG 94a at SLAC

Week of Apr 15 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Deliver CALIB to IN2P3
- Check 2y Run2.1i data for incomplete files
- test updated sky configs and running makeFpSummary
DESC
NERSC
- Clear out Javier's copy of DC1 undithered data
- Start transfer of Run1.1 and Run1.2 data into HPSS
- Update gcr-catalog in jupyter kernels and underlying shifter image
SLAC/Camera
Fermi
- announce, attend, and note take for Fermi software meeting
- Help Bill get set up with GR at SLAC for debugging
nEXO
- attend software meeting
- Review Git and Code review docs

Week of Apr 8 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Regnerate BF flats with w_2019_14
- update config and ingest new Reference catalogconfig and ingest new Reference catalog
- Rerun constructDark with doCrosstalkBeforeAssemble=False
- Rerun constructFlat with doCrosstalkBeforeAsseble=False
- Set up updated CALIB products using w_2019_14 and setting isr.doCrosstalkBeforeAssemble=False
- Update obs_lsst with isr.doCrosstalkBeforeAssemble=False https://jira.lsstcorp.org/browse/DM-19218
- Try out sky construction using Jim's sky frames
- Try sky construction again with flats available
- Get sky configs updates into obs_lsst
- Go through outstanding LSSTDESC/ImageProcessingPipelines issues and close where possible
- Work with Johann to clean up the IPP/workflows/srs/dm_configs area and ultimately deleted it
- Reviewed the reworked SFD pipeline SRS workflow with Johann Set up updated CALIB products using w_2019_14 and setting isr.doCrosstalkBeforeAssemble=False
DESC
- Attend and present at CI meeting
- Attend Rachel's Computing WG meeting
NERSC
- Get HPSS Project area set up
- Determine HPSS quota request
- Figure out Katrin's HPSS use
- Assess DESC user HPSS useage
- set up DM repo in CSCRATCH/descdm/DC2/Run2.1i area
- Point James Perry to Intel Python: https://docs.nersc.gov/programming/high-level-environments/python/
SLAC/Camera
Fermi
- announce, attend, and note take for Fermi software meeting
nEXO
- attend software meeting and present update
- Try out LCG view installation at SLAC and update nEXO wiki doc
- Complete LLNL security training
- Contact LLNL to get my account activated

Week of Apr 1 2019

DM-DC2
- Write agenda, take notes and run DM-DC2 mtg
- Script to find all potential incomplete Run2.1i visits in the first production run
- Write up todo list of DRP processing of Run2.1i-y2wfd
- Schedule mtg with Katrin, Jim, Rachel, Dominique, Fabio, Johann concerning Run2.1i processing
- meeting with Katrin,Jim, Rachel, Dominique, Fabio about Run2.1i processing
DESC
- Q2 report
- Attend Data access meeting
NERSC
- Arrange Run1.1p, 1.2p, 1.2i, 2.0i data and tabulate disk use
- Prepare HPSS disk request
- Put in HPSS project area request
- Set up new conda area based on updated DMstack dependencies
SLAC/Camera
Fermi
- announce, attend, and note take for Fermi Software Meeting
nEXO
- Attend sims meeting
- merge https://github.com/nEXO-collaboration/nEXO_MC/pull/10
- send Alex some details about SNiPER wrapping of python
- Ask SNiPER developers about XmlObjDesc
- Improve docker images
- Update SLAC instllation
- try out CMake

Week of Mar 25 2019

DM-DC2
- write agenda, take notes, and run DM-DC2 mtg
- Generate flats calibration products for Run2.1i
  - regenerate R22 S11 using updated configs
  - test run1.2i R22 S11 flat raw using same stack w_2019_09
  - test run2.1i 80k flats
- generate subset of bf flats
- Ask Johann for metacal summary
- Create RFC or Confluence page to request DM support for DC2: https://lsstc.slack.com/archives/C978LTJGN/p1553708211293400
DESC
- send out announcement concerning NERSC jupyter
NERSC
- Set up Yadu with a copy of ci_hsc at NERSC: https://github.com/lsst/ci_hsc
- check on mcnanna usage
- Continue arranging the /global/projecta/projectdirs/lsst/production area for Run1.2p and Run1.2i
- Pull out DC1 undithered calexp from HPSS for Javier
  - deal with desc account being over quota in CSCRATCH
SLAC/Camera
- inquire about BNL computing status
- SISA Information Security I
Fermi
- announce, attend, and note take for Fermi Software Meeting
nEXO
- attend Tuesday software meeting
- summarize decision concerning nexo-env https://github.com/nEXO-collaboration/nexo-env/issues/9
  - close
- merge https://github.com/nEXO-collaboration/nEXO_MC/pull/13
- merge https://github.com/nEXO-collaboration/nEXO_MC/pull/10
- send Alex some details about SNiPER wrapping of python
- Ask SNiPER developers about XmlObjDesc
- Improve docker images
- Update SLAC instllation
- try out CMake

Week of Mar 18 2019

DM-DC2
- write agenda, take notes, and attend DM-DC2 mtg
- Fix bf flat headers
- https://jira.lsstcorp.org/browse/DM-18551
- https://jira.lsstcorp.org/browse/DM-18575
- https://jira.lsstcorp.org/browse/RFC-583
- Install latest updated cp_pipe master from Craig
- Take a look at the BF kernels in Michael's notebook
  - https://lsstc.slack.com/archives/C978LTJGN/p1553090956192800?thread_ts=1553046434.190200&cid=C978LTJGN
DESC
- Attend Data access TF meeting
- Build and release new lsstdesc/stack-sims docker image w_2019_10_1 and patch w_2019_10 due to shebangtron
- Finish CI Q1 report
NERSC
- update jupyter doc https://github.com/LSSTDESC/nersc/issues/9
- Request descim account access for Ben Clifford
- Continue cleaning up data storage on projecta
SLAC/Camera
- Completely shut down elog bot on SLACK
- Update glastsys password and escrow
Fermi
- attend and note take offline software meeting
nEXO
- attend sims meeting
- attempt to set up 2009 mac with ElCapitan
  - attempts to upgrade to later MacOS failed
  - Attempt to install a current ROOT failed
- Set up VM with MacOS

Week of Mar 11 2019

DM-DC2
- Write agenda, take notes, and attend DM-DC2 mtg
- Generate BF flats
  - tarball and store on projecta
- set up CSCRATCH area with CALIB, ref_cats and bf flats
DESC
- Attend and present at CI meeting
- Attend DC2 meeting
- update jupyter doc for jupyter-dev changes
  - test after maintenance is completed on 3/15
- Build new w.2019.10 sims docker image
NERSC
- Help Anita with sncosmo and sims jupyter kernels
  https://lsstc.slack.com/archives/C2U2K05JR/p1551995963014800
- install parsl 0.7.2 in desc-python
- Deal with ongoing permission issues due to mixed ownership in /global/projecta/projectdirs/lsst/production/DC2_ImSim/Run2.1i/instCat
SLAC/Camera
- shut down eLog slackbot
- Install DMstack v17 + a few additional python packages and setup constraint file
  - Jira
    server SLAC National Accelerator Laboratory
    serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
    key LSSTTD-1387
Fermi
- Find SLAC anaconda 3 installation
  - send note to SLAC unix-admin about public installation
nEXO
- attend and apparently run software meeting

Week of Mar 4 2019

DM-DC2
- Write agenda, take notes, and attend DM-DC2 meeting
- Github issue for tracking handling of DC2 data
- Take another look at running SingleFrameDriver at NERSC
- Generate darks and biases for Run2.1i
  - first get cvmfs caching fixed at NERSC
    https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=2abc3055db8c33407cf774608c961912%26sysparm_view=ess
- BF Kernel Generation
  - test updated configs
    https://github.com/LSSTDESC/ImageProcessingPipelines/issues/82#issuecomment-469824632
  - send Tom some instructions
DESC
NERSC
- Help mishak with AUP
- add parsl 0.7 to desc-stack
- CCL installation instructions?
  - https://github.com/LSSTDESC/CCL/issues/587#issuecomment-468841051
SLAC/Camera
- Complete SLAC Computing Security Training
Fermi
- s/w mtg and note taking
nEXO
- attend sims meeting
- contact Sniper developers about how to best embed python in Sniper algs
- Write up to do list based on software workshop notes
- Check on Venkatesh and docker set up

Week of Feb 18, 2019

DM-DC2
- Create workflow for imsim biases and darks
- run test through patch of Run1.2i
  - ingest
  - sfd
  - coadd
  - multiband
DESC
- CI quarterly report
- set up Parsl discussion with Jim/Katrin/Tom/Yadu/Ben/Antonio
NERSC
- check into mweiser problems accessing lsst resources
- Inquire about globus and collab use
- Inquire about HPSS project directory
- add new ccl and postgressql to docker and jupyter kernels
- Finish copying DC2 data to projecta
- request increase in inode quota of descdm $CSCRATCH
- Clean up $CSCRATCH descdm
- update desc-python and desc-stack with new gcr-catalogs release in time for collaboration mtg
- Inform NERSC of Hack Day and planned Tuesday tutorials using jupyter-dev
SLAC/Camera
- Data portal updates for NCRs
Fermi
nEXO
- Test cmake
- attend sims meeting
- release nexo-ofline and install at SLAC and create docker image
- meet with Lisa about workshop agenda
- Demo Doxygen for Lisa
- meet with Lisa to go over git branches on nexo_MC
- PR updates to Guofu's optical photon changes

Week of Feb 11, 2019

DM-DC2
- Prepare agenda, attend and note take for the DM-DC2 mtg
- Set Tom up with Run1.2i calibration products
- Create workflow for imsim biases and darks
- Contact Scott about adding obs_lsst to lsst_sims
- Coordinate merge of https://jira.lsstcorp.org/browse/DM-17875
- Meeting with Jim & Katrin
  - Discuss disk space needs for Run2.1i (plan to add 1 Pb per year for now)
    - How much data will come back from IN2P3?
  - Where will Run2.1i processing occur?
DESC
- CI quarterly report
NERSC
- Install new CCL release
- add new ccl and postgressql lib to docker & jupyter kernels
- Finish copying the DC2 data into projecta and prepare count of all contents
- Plan HPSS use
SLAC/Camera
- Data Portal Update for NCRs
- install pyds9 on lsst-dc10 and maybe also afs
  - sent inquiry to Jim and Richard
Fermi
- Attend and take notes for offline meeting
nEXO
- attend software meeting
- create list of open branches in nexo_MC
- Ask SNiPER developers about the ability to submit jobs via the python config scripts
- Prepare discussion on documentation for next software meeting: Doxygen & ROOT's THtml
- Respond to Samuele's Gaudi questions
- test cmake build of sniper against nexo-offline
- install py3 at SLAC
- RElease new nexo-offline and install at SLAC & create docker/singularity containers
  - OSG?
- Document a tutorial for nexo-offline for workshop
- Lessons learned?

...

Week of Feb 4 2019

DM-DC2
- Prepare agenda, attend and note take for DM-DC2 mtg
- Help Tom set up a suitable stack for BF kernel generation testing
- Test BF kernel generation by running the data produced through a modified version of Michael's notebook
- Set Tom up with Ru1.2i calibration data
- Test obs_lsst master using Run2.1 data
- Create workflow for imsim biases and darks generation
DESC
- Continue moving DC2 data to production area at NERSC
- Prepare a space request for HPSS based on DC2 needs
- Attend Data-Access meeting
- CI quarterly report
- Finish looking over Joanne's postgres notebooks
- Send in Travel request
NERSC
- Ask NERSC about setting up a shared HPSS area accessible via globus
- Install latest pyCCL on desc-python and in shifter images
- Document and promote use of CVMFS at NERSC
SLAC/Camera
- contact Arash about setting up a DB for elog storage
Fermi
- announce, set up agenda, attend and take notes for s/w mtg
nEXO
- attend sims meeting
- Meet with Lisa to plan Software Workshop

Week of Jan 28 2019

DM-DC2
- Prepare agenda, attend, and note take for DM-DC2 mtg
- Finish off https://jira.lsstcorp.org/browse/DM-16472
- Merge https://github.com/lsst/obs_lsst/pull/51
- Help Tom set up a suitable stack to exercise MIchael WV's BF notebook
DESC
- Prepare NERSC space discussion and attend CI mtg
- Attend DC2 mtg
- Prepare DM-DC2 XWG Confluence page
NERSC
- Prepare space request for HPSS based on DC2 needs
- Request IN2P3 transfer Run1.2p raw repackaged data back to NERSC
- set up psycog2 at NERSC
SLAC/Camera
- eLog - check new FNAL account and contact Margherita about transferring data to external DB
- contact Arash about a DB?
- alert camera community about impending elog shut down
Fermi
- Announce, attend, and take notes for offline mtg
nEXO
- Help SLAC users set up nEXO and EXO env
- check out Gaudi

Week of Jan 21 2019

DM-DC2
- Reprocess requests 28 visits through minidrp
- Prepare for new Run2.1i branch of obs_lsst
DESC
- attend ssims meeting
- Review Joanne's postgres notebook
NERSC
- Write up proposal for DC2 disk space use
- Try Exposure Checker
- Update jupyter kernels set up to add current working directory to PYTHONPATH
- update desc-dc2-dm-data for Run1.2p_v4
- Copy Run1.2p_v4 to CSCRATCH
- Build and release new docker image that supports Run1.2p_v4
SLAC
Fermi
- Attend and take notes for Offline mtg
nEXO
- Attend Sims meeting
- Write up Communications tools page
- Help SLAC users set up nEXO and EXO env

Week of Jan 14 2019

DM-DC2
- Prepare agenda, run and take minutes for DM-DC2 mtg
- Merge minidrp Run2.1i into IPP repo
- Chat with Tom about pipeline help
DESC
- Prepare NERSC disk space use slides and attend CI mtg
- Write up NERSC DC2 disk space use
NERSC
- Set up collaboration accounts
SLAC
Fermi
- S/W mtg
nEXO
- Look into Gaudi
- Install Anaconda Python
- Provide installation instructions
- Getting started at SLAC doc
- Install nEXO_MC and nexo_offline at SLAC
- Fast Charge

Week of Dec 17

DM-DC2
- Prepare agenda, run and take minutes for DM-DC2 mtg
- Test Run2,0i minidrp workflow
- Transfer IN2P3 Run1.2i configs to lsst-dev
- Get copies of CALIB and ref_cats
- Fix up handling of config.isr.doCrosstalk in obs_lsst
DESC
- Prepare Run1.2i/p update and attend CI meeting
NERSC
SLAC
Fermi
- Attend software meeting
nEXO
- Attend sims meeting
- Attend analysis discussion
- Contact Sara about docker
- Take another look at migrating FastCharge to nexo-offline
- Contact Vladimir about CMake progress
- Test singularity image at OSG

Week of Dec 10

DM-DC2
- Prepare agenda, run and note take for DM-DC2 mtg
- Run1.2i processing
- Prepare Run2.1i minidrp workflow
DESC
- nersc user mailing list?
- docker image for sims+mkl
- New docker image sims 2_13_1
NERSC
SLAC
Fermi
- write agenda and notetake for offline meeting
nEXO
- complete travel request
- help Sara with Docker
- Poke G4 guy about speed of light G4Exceptions

Week of Dec 3

DM-DC2
- Prepare agenda, run and note take for DM-DC2 mtg
- ~~Test new biases and darks~~
- Tarball and send Johann new biases and darks, request CALIB directory be transferred to NERSC
- Run1.2i processing on KNL
- rebase and merge PR#41 obs_lsst
DESC
- nersc user mailing list?
- new sims image 2_13_0
  - tag obs_lsst for Run2.1i
- docker image for sims+mkl
- attend and present 1.2i/2.0i processing update CI mtg
NERSC
- install new jupyter-kernels
- update desc-python with GCRCatalogs 0.9.3
- update desc-stack with GCRCatalogs 0.9.3
- update desc-python and desc-stack with GCR 0.8.7
SLAC
Fermi
- Help with recon root reading
nEXO
- sims meeting
- meet with Lisa
- setup nexo-env so it can work with base nexo external image
- Help Sara with Docker
- Test for G4Exception using Guofu's mac file on v0r0p1
- Merge zepeng's chargesim updates
  - new release tag and docker image

Week of Nov 26

DM-DC2
- prepare agenda, run, and take notes for DM-DC2 mtg
- Test newly generated biases and darks
- look at obs_lsst PRs
- Run1.2i processing
DESC
- nersc user mailing list?
- docker image for sims+mkl
NERSC
- update desc-python to include emcee
SLAC
Fermi
- s/w meeting
- Help with Recon root files
nEXO
- ~~Meeting with Lisa~~
- Merge Guofu's updates and tag new release
- Create new v0r1p0 Docker images
- create external only base for docker images

...

Week of Nov 19

DM-DC2
- prepare agenda and run, take notes for DM-DC2 mtg
- Regenerate darks and biases for Run1.2p using phosim v3.7.9
- Test generating bias file and sky flat using haswell build vs knl build at NERSC
- continue Run1.2i processing
- make sure all outstanding obs_lsstCam PRs are merged
  - figure out default setting for xtalk for Ru1.2p versus Run1.2i
DESC
- ask Charlotte about setting up a nersc user mailing list
- Update stack-sims images to keep test files and retain eigen includes
NERSC
SLAC
Fermi
- weekly s/w mtg
nEXO
- singularity at SLAC
- update wiki doc
- generate test data for Guofu's update
- attend sims mtg

Week of Nov 12

DM-DC2
- continue processing Run1.2i
- Maintain copy of IN2P3 on NERSC $CSCRATCH
DESC
- ask Charlotte about setting up a nersc user mailing list
- Field ANL questions about docker and stack+sims building
NERSC
- create new jupyter-kernels repo
- Update desc-python at NERSC
- Get my.nersc.gov login fixed after enabling MFA
SLAC
Fermi
- weekly s/w meeting
nEXO
- meet with Lisa
- singluarity at SLAC
- ask Raymond about OSG
  - still not fixed
- Test Guofu's updates

Week of Nov 5

DM-DC2
- Prepare and run DM-DC2 mtg
- Repackage ALL of Run1.2i data
DESC
- Attend CI meeting
- Document DESCPYTHONPATH for jupyter
NERSC
- Transfer NCSA Run1.2i data to NERSC?
- Update desc-python at NERSC
- Add GCR==0.8.6
SLAC
- Attend weekly stand up meeting
nEXO
- sims meeting
- Check on Guofu's testing of his updates in nexo-offline
- singularity at SLAC
- doc use of SLAC in nexo WIki

Week of Oct 8

DM-DC2
- Prepare and run DM-DC2 Meeting
- Chat with Leanne and attend meeting with Leanne, Johann, Bob, Dominique
- Process Run1.2i using w_2018_39 + obs_lsstCam dc2/run1.2-v2
DESC
- Create w_2018_30-run1.2 docker image + DESC python
NERSC
- take a look at dask-jobqueue
- Help write up problems/request for upcoming ERCAP application
SLAC
nEXO
- attend Sims meeting
- Release new Docker and singularity image for testing
- Port optical updates in nEXO_MC to nexo-offline
  - Ask Guofu to test
  - Make a new Docker image
Fermi
- run and attend s/w meeting

Week of Oct 1

DM-DC2
- Prepare and run DM-DC2 meeting
- test obs_lsstCam master + w_2018_39 ingest
- Run calexps and checkCcdAstrometry on two visits shared between Run1.2i and Run1.2p
DESC
- attend data access meeting on Monday
NERSC
- Check on availability of /usr/bin/time on Cori compute nodes
SLAC
- Help Johann get started with jupyter on IR2
- Install Anaconda 5.2.0 for python 3.6
nEXO
- meet with Lisa on Wednesday
- Create updated docker image of nexo-offline
  - Create Singularity image
Fermi
- s/w meeting
- Respond to W's questions about fitsGen & astro updates

Week of Sept 24

DM-DC2
- Prepare and run DM-DC2 Meeting
- Update SRS workflow
- Regenerate calibration data for Run1.2p
  - repackage
  - ingest
- test creation of multiple ingest repo
  - merge separate repos
- Revisit darks/bias/flats generation from a couple of months ago
- Stay on top of obs_lsstCam updates
  - merge xtalk PR
  - Contact Robert about merging outstanding PRs
  - Contact Robert about obs_lsstCam=>obs_lsst migration
  - Update run1.2 branch
  - Contact Robert about run_visit updates for jointcal
DESC
- attend CI meeting
- Provide CV
- Attend DC2 meeting and take minutes
NERSC
- Create new w_2018_38 docker image for MWV
- Work out problems with movable stack builds
- Check on /usr/bin/time on Cori and Edison and contact NERSC
SLAC
- Deal with IR2 stack installs and python questions
  - Document on Confluence
Fermi
- attend s/w meeting
nEXO
- attend sims meeting
- install latest sniper-offline at SLAC
- Check on OSG installation
- Meet with Lisa on Wednesdays
- Check in with Raymond about OSG

Week of Sept 17

DM-DC2
- Prepare and run DM-DC2 Meeting
- Test and update new SRS workflow
  - move to Edison
    - cannot run on Edison - chatted with Tom, sent a note to Tony, Brian, and W
      - Tried Tony's suggested fix and it seems to be working
  - move back to Cori
  - write up for Warren
- Stay on top of obs_lsstCam updates
  - Create run2.0 branch
  - Update run1.2 branch
- Create scripts for merging ingested repos
DESC
NERSC
- Handle permissions on $CSCRATCH/../desc/DC2/Run2.0i
- Work out kinks setting up movable stack builds
SLAC
- Reinstall v16 binaries and test updating python install
Fermi
- Tag evtClassDefs
- attend s/w meeting
nEXO

Week of Sept 10

DM-DC2
- Prepare and run DM-DC2 meeting
- Help sort out using ingestDriver from within a container
  https://lsstc.slack.com/archives/C2K97E3L5/p1536670464000100
- Contact Tom G about pilot Edison jobs
- Test running ingest on Edison
- Coordinate with Johann to get an updated workflow
DESC
- Finished creating new lsst_sims docker image over the weekend and uploaded to NERSC
- Prepare and attend DC2 meeting
- copy UK Run1.2i data to projecta and test use of ACLs
NERSC
- Fix ACls on the projecta production area
- Coordinate with NERSC to sort out problems with cvmfs: https://nersc.service-now.com/nav_to.do?uri=incident.do?sys_id=725125d4dbaca3407d2a300f7c9619ee%26sysparm_view=ess
- Try to sort out source builds at NERSC on SCRATCH and migrate to /global/common/software https://lsstc.slack.com/archives/C4RKBLK33/p1536682237000100?thread_ts=1534435430.000100&cid=C4RKBLK33
  - still a work in progress, need to fix up miniconda installation
SLAC
- Install DMstack v16 on IR2
- Inquire about IR2 install of devtoolset-6
nEXO
- attend sims meeting
- Help nexo user use Centos7
- inquire about data disk filling up at SLAC
- convert FastCharge sim to Sniper
Fermi
- Attend C&A meeting

Week of Sept 3

DM-DC2
- Prepare and run DM-DC2 meeting on Tuesday
- Validation processing of Run1.2i
  - generate amplifier files from original eimages
  - run ingest on Cori Haswell and Cori KNL
  - run singleFrameDriver and report errors
  - run makeFpSummary
  - Try visualizeVisit
    - update obs_lsstCam to include necessary template
  - Keep documentation updated
  - Answer Tom U's questions about ingestDriver
- Coordinate obs_lsstCam branch and create Run1.2 tag
DESC
- Create new lsst_sims docker image for Jim and upload to NERSC
NERSC
- Inquire about desc access to globus
- Clean up inode use on /projecta
SLAC
- install w_2018_26 for Emily
- Handle vendor ingests (5 so far)
nEXO
Fermi
- software meeting
- check celestialSources used in GR L1 versus master
- contact Luis about updates to eblAtten

Week of Aug 20

DM-DC2
- Prepare and run DM-DC2 meeting
NERSC
- install phosim 3.7.16
SLAC
nEXO
Fermi
- Software Meeting

Week of August 13

DM-DC2
- Prepare and run DM-DC2 meeting
- Attend CI meeting
NERSC
- finish coadd processing Run1-2iTest
- Test use of DMstack Shifter image in jupyter-dev
- update jupyter-dev confluence pages
- fix inode use on projecta
- finish testing relocatable Stack weekly build
SLAC
- release and install latest datacat-utilities on SLAC public and IR2
nEXO
- attend Sims meeting
- Install nEXO_MC and test running FastCharge
Fermi
- respond to Katie Chapman's Fermipy helpsoft issue
- Update workbook pages to remove POI form

Week of July 16th

DM-DC2
- Prepare and run DM-DC2 meeting
NERSC
- Update shared python installation
  - GCRCatalogs
  - new CCL
    - track down reported notebook failure
    - track down reported notebook failure
    - reinstall at NERSC pyccl 0.3.5
SLAC
- Follow up on laptop purchase
- Release and install datacat-utiltiies-0.1.1
- Release and install datacat-utiltiies-0.1.1
- Install sims 2_9_0 at SLAC
nEXO
- Sims Meeting
- Test Docker image in Singularity
Fermi
- Offline S/W meeting

Week of July 9th

DM-DC2
- Prepare and run weekly DM-DC2 meeting
- Prepare for XWG DM-DC2 Meeting
NERSC
- Create symlink copy of IN2P3 data in $SCRATCH
- Help update Jupyter documentation and contact NERSC about jupyterLab
- Update CCL NERSC doc https://github.com/LSSTDESC/CCL/wiki/Using-and-Installing-CCL-on-NERSC
- Contact Debbie & Wahid about GPFS AFM
- Install sims 2_9_0 from source
- Install docker image of sim 2_9_0 at NERSC
- Continue testing weekly install cron job
- Contact NERSC about missing gcc/6.3.0 module
SLAC
- Follow up on laptop purchase
nEXO
- Meet with Lisa
- contact Raymond about fast charge
- contact Alex about fast charge (email sent)
- Gain access to LLNL account
- Pursue Singularity container
  - Install Singularity on CentOS7
  - Try using our docker image within Singularity
- Document Docker container
- Set up version control for SNiPER framework
- Check OSG external library installation and send email to Benedikt
Fermi
- Offline Software meeting

Week of June 25

DM-DC2
- Write up agenda, take notes, and run DM-DC2 meeting
- Create NERSC's raw directory for Run1.2p data transferred from IN2P3 in /global/projecta/projectdirs/lsst/global/in2p3/Run1.2p/w_2018_18
- Merge and close the makeFpSummary PR on LSSTDESC/obs_lsstSim
- Work through remaining issues, merge and close config updates https://github.com/LSSTDESC/obs_lsstSim/pull/16
- Push along updates to the Cookbook and write up a similar doc with an overview à la HSC
- Prepare to run mini-DRP on Run1.2i
- Attend DC2 prep meeting
- Attend DC2 data access meeting
- Contact Dominique F about providing an update at the DC2 meeting on July 6th
NERSC
- Monitor NERSC ticket concerning GPFS AFM
- Inquire about monitoring SCRATCH space for upcoming purges
- Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
- Fix weekly builds of DMstack to use limited number of cores and skip w_2018_24, since it will not build due at NERSC
- NERSC Ticket to work out details of weekly DMstack builds
- Update CCL doc for collab meeting
- Clean up disk space in /global/common/software
- Fix inode use on projecta space
- New sims 2_9_0 release, docker image
- Install sims 2_9_0 at NERSC
- Contact owner of galsim files installed under our w.2018.13-sims_2.7.0 installation in /global/common/software/lsst
SLAC
- Check on set up of datacat-utilities
nEXO
- Respond to CMT=>CMake project for SNiPER
- Ping Marco (Gaudi) about their CMT=>CMake experience
Fermi
- Verify that we can checkout L1 tags using --force and update doc
- Create new GR FDM-10 build:
  Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key FDM-10

Week of June 18

DM-DC2
- Write up agenda, take notes, and run DM-DC2 meeting
- Keep up with Johann's modifications to the SRS workflow
- Create NERSC's raw directory for Run1.2p data transferred from IN2P3 in /global/projecta/projectdirs/lsst/global/in2p3/Run1.2p/w_2018_18
- Merge and close the makeFpSummary PR on LSSTDESC/obs_lsstSim
- Work through remaining issues, merge and close config updates https://github.com/LSSTDESC/obs_lsstSim/pull/16
- Try running ingestCalib again using the phosim v3.7.9 produced biases and darks
- Push along updates to the Cookbook and write up a similar doc with an overview à la HSC
- Prepare to run mini-DRP on Run1.1i
NERSC
- Review ingestDriver.py slow runtime at NERSC including the use of --batch-type when using the SRS workflow
- Investigate load use of weekly installs and modify the cron job to limit the number of cores utilized.
- update to GCRCatalogs 0.7.8 in the shared python environment as requested by Yao
SLAC
- Install DMstack v15 for IR2, py3
- Install w_2018_23 for summer grad student
nEXO
- Write up slides for Docker intro for collaboration meeting and present
- Write up slides for SNiPER update and present
- Listen in to Simulation session 9PM-1:30AM
Fermi
- Weekly software meeting

Week of Jun 11 2018

DM-DC2
- Write up agenda, run, and take notes for weekly DM-DC2 meeting
NERSC
- update fast3tree installation for Yao to allow writing to the install directory
SLAC
nEXO
- Complete docker image for collaboration meeting
- Inform collaborators and help get them set up before the meeting
Fermi
- New L1 Release tag for GR See JIRA FDM-5
- Build new L1 Release tag at SLAC for Philippe to use

Week of Jun 4 2018

DM-DC2
- CI Meeting update
- Prepare agenda, run DM-DC2 meeting, and take notes
- Try constructFlat at NERSC
  - ingest subset of Run1-2p data
- ~~Doodle Poll to find new meeting time~~
  - chat with Dominique & Johann first
- ~~Ingest all Run1-2p data at NERSC~~
  - Merge latest IN2P3 workflow into NERSC workflow
  - Start up the jobs
NERSC
- investigate installing new python module Yao requested
- Talk to Joe Z about his miniconda env at NERSC to see about pulling it into the shared python env
- Reinstall stack weeklies when the next python update for DM is released
- deal with inode use on /global/projecta/projectdirs/lsst/
  - Issues is associated with the phosim SEDs for every visit, these files must be tarred.
SLAC
nEXO
- Data Task Force Meeting Wednesday
- Prepare proper docker image with tutorials
Fermi
- merge slac doc in github
- Offline Software meeting
- Help Philippe make L1 updates to TkrDigi
- get a new GR L1 release tag made

Week of May 28 2018

DM-DC2
NERSC
- Finish slides for seminar and present
- Test Javier's notebook
- Fix py3-4.3.21 installation with numpy and numexpr
- install phosim v3.7.15 for Haskell-Intel & KNL-intel
- Install CCL for Yao in the shared python area
SLAC
- Handle vendor ingests while Tom is away
nEXO
- Attend sims meeting and report on docker status
- Update docker image with tutorials
Fermi
- Help Philippe get set up with github and new JIRA area
- Write up initial doc for developing at SLAC: https://github.com/fermi-lat/doc/blob/heather999-patch-1/slacdev.rst

Week of May 21 2018

DM-DC2
- prepare for weekly meeting and take notes
- Send Bob Armstrong details for getting connected to NERSC and IN2P3
- Rerun processEimage test for v3.7.14 to capture log
- Prepare update for CI meeting
- Report unit test failure in LSSTDESC/obs_lsstSim
  - Fix the failure
- Arrange meeting with IN2P3 folks
- Attend DM-Stack-Club
- Attend and report at DC2 Meeting
- Rerun bias and darks using phosim v3.7.14
NERSC
- install latest parsl
- set up new external python environment
- Prepare slides for seminar
- Set up new docker image for new sims release and upload to shifter
- Install new sims release from source
- Transfer DC1 data from HPSS to SCRATCH
  - http://www.nersc.gov/users/storage-and-file-systems/hpss/advanced-usage-examples/hsi-tape-ordering-script/
  - http://www.nersc.gov/users/computational-systems/cori/running-jobs/advanced-running-jobs-options/
  - Confirmed /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered is the full data set stored in HPSS
- Help user build CCL paper and updated the CCL wiki https://github.com/LSSTDESC/CCL/wiki/Using-and-Installing-CCL-on-NERSC
nEXO
- update docker image to add in tutorials
SLAC
- order new laptop - waiting on response from I&T
- Handle vendor ingests while Tom is away
Fermi

Week of May 14 2018

DM-DC2
- set up agenda and write up notes for Monady's DM-DC2 meeting
NERSC
- Check getting started at NERSC documentation
- Contact Michael & Javier about the NERSC seminar May 30th
- Run phosim v3.7.9 and v3.7.14 test data through miniDRP
- Check on kubernetes availability at NERSC jupyterhub and set up meeting
nEXO
- update docker image to remove sudo and add in tutorials
- coordinate archiving nexo-sniper
SLAC
- set up order for new laptop
Fermi

Week of May 9 2018

DM-DC2
- Setup agenda and write up notes for Monday's DM-DC2 meeting
- Create darks using phosim 3.7.9
NERSC
- Install new versions of phosim as they appear
nEXO
- Attend sims meeting and make presentation concerning SNiPER visit
- Update docker image with tokens
- contact Raymond about timing correlation
SLAC
- Set up order for new laptop
- Requested and tested git lfs install at SLAC for nexo
Fermi
- Attend software meeting and write up notes

Week of April 30 2018

DM-DC2
- Set up agenda and write up notes for Monday's DM-DC2 meeting
- Rerun one processEimage job on one core on KNL using a source build of the stack
- Send Salman and Katrin directions about the steps to run "mini-DRP" so they can try it at ANL
- Set up full pipeline at NERSC to run on KNL
- Finish creating calibration products and document
- Prepare Monday's CI update
NERSC
- Update python environment to include potables
- Create "current" symlink to point to the current python installation
- Test and document setting up jupyter-dev using a shared stack install
- Install phosim v3.7.10 for both haswell-intel and knl-intel
- install phosim v3.7.11 for both haswell-intel and knl-intel
nEXO
- Set up Docker image for nexo-offline + SNiPER
- Provide daily summaries on the Slack meeting channel
- Test new SNiPER using the last collaboration meeting examples
- Understand how to implement Time Correlation
- Attend Tuesday Update meeting with nexo collaborators
SLAC
- Help Tom M set up python on his laptop
- Update elog Slack app to add a new channel as requested by Aaron

...

DM-DC2
- Set up agenda and write up notes for Monday's DM-DC2 meeting
- Rerun one processEimage job on one core on KNL as requested by Salman
- Send Salman directions about the steps to setup the DMstack so they can try it at ANL
- Provide NERSC update for DC2 meeting
- Attend DC2 meeting
NERSC
- Create an area of IN2P3 to transfer their data products
nEXO
- Install new version of SNiPER for centos7
  - request new tbb system library be installed
- Send some initial SLAC usage instructions to the SNiPER developers
SLAC
- Help Steve ssh tunnel to new IR2 jupyterhub

Week of Apr 16th 2018

DM-DC2
- In support of SkyCorrection Task
  - Generate darks using phosim v3.7.9
  - Play with constructFlat.
- Rerun miniDRP task using KNL and produce some timing information
- Update LSSTDESC/obs_lsstSIm for Phosim v3.7.9
  - Tagged Run1.1p obs_lsstSim fork
  - Successfully updated the obs_lsstSIm files using the bias output
  - Work through unit test failures
  - PR
NERSC
- Clear out out Stack builds as advertised on Slack
- Install parts in shared python env + weeklies + v15
  - Consider setting up a common shared lib area
- Install a modern dmtcp on knl-intel and haswell-intel
  - multilib install failed, went back to 64 bit only for now - which should be fine for phosim
nEXO
- Prepare for travel to SLAC
- Help SNiPER Developers get set up with SLAC accounts
- Install new version of SNiPER at SLAC

Week of Apr 9 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- HistogramSvc
- Write up SLAC SNiPER installation instructions on wiki
- Send Lisa some journal references for Gaudi
DM-DC2
- Prepare for Task Force Meeting and write up Live Notes
- Complete initial run of first Run 1.2p visits through minidrp
- Create new dc2 Shifter image with room for a NERSC volume
- Create dc2 Shifter image using a dc2 tag to try to avoid version mismatch errors
- update workflow to use Shifter and submit PR
- Create w_2018_14 + LSSTDESC/obs_lsstSim Shifter image
- Test pipeline with w_2018_14+LSSTDESC/obs_lsstSim Shifter
- update NERSC workflow to try out processEimage on rafts
NERSC
- Prepare updates for CI meeting
- Attend Ci meeting
- Send Katrin some information about how the set up NERSC for DESC
- Ask NERSC about anticipated need to recompile software after the scheduled Cori OS update on Apr 25-26
- Install DMstack v15 in the common software area
- Request to remove some older stack installations at NERSC
SLAC
- Write up portal section in Overleaf
- Complete Research integrity course
Aux Telescope

Week of Apr 2 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- HistogramSvc
- Attend Simulations Meeting
- Write up SLAC SNiPER installation instructions on wiki
DM-DC2
- Set up new Docker image which includes patched obs_lsstSIm
- Prep slides and attend SSim meeting
NERSC
- Check on weekly builds and document their new location
- Create new docker image for new sims 2_7_0 release
- Install sims 2_7_0 from source in the common software area
SLAC
- Complete CyberSecurity Training
- Write up portal section in Overleaf
- get antivirus going on home centos7 to allow VPN connection required to deploy portal
Aux Telescope

Week of Mar 26 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- HistogramSvc
DM-DC2
- Prep for Task Force Meeting
- Write up Live Notes
- Contact Dominque B concerning processEimage failure
  - Now fixed with new config file
  - add this config file to github
- Write update for Friday's DC2 meeting
- Write update for CI meeting
- Start up new mini-DRP task in SRS pipeline
  - Run ingestData for all 66 visits from Run 1.1p
  - Run processEimage (obtain time and memory consumption stats)
SLAC
- Fix LSSTTD-1257
- Deploy data portal 0.5.14
NERSC
- move weekly builds to common software area by updating cron job
Aux Telescope
- On Hold until we hear from Kirk

Week of Mar 19 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
- Attend Fermi Meeting
nEXO
- Fix build of ROOT on centos7, missing libEve and libRGL
- Finish building nexo SNiPER on centos7
- Work on HistogramSvc
- Attend simulations meeting
DM-DC2
- DM-DC2 Meeting prep
  - DM-DC2 Meeting Agendas
- Attend SSims meeting
- Write up Live notes
- Extend DRP Workflow to run processEimage
- Update DRP Workflow to run processEimage for multiple visits
- Meet with Nicolas to discuss DRP
SLAC
- Implement and deploy new run search box
  - Jira
    server SLAC National Accelerator Laboratory
    columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
    key ETCCB-463
- Update my own centos7 desktop to use SLAC VPN, NetBeans, etc so I can continue supporting the data portal after the VPN transition at SLAC.
- Run vendor data ingests
NERSC
- Install TreeCorr
- Install new GCRCatalogs in all conda environments
- Move weekly stack installs
  - Clean up old contrib area
  - request another 100 GB of space
  - update cron job
Aux Telescope
- Wait to hear from Kirk

Week of Mar 12 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
nEXO
- Fix build of ROOT on centos7, missing libEve and libRGL
- Finish building nexo SNiPER on centos7
- Work on HistogramSvc
DM-DC2
- Update Nicolas' ImageProcessingPipelines branch for NERSC
- Demonstrate running SRS workflow for ingest
NERSC
- Work out production installation of lsst_sims + w_2018_09 using the shared python env
SLAC
- Create some documentation to use jupyterlab at SLAC
- Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1256
Aux Telescope

Week of Mar 5 2018

Fermi
- Finish off creating L1 branches for all GR packages
- Get Brian to fully clean up between GR jenkins builds - see JIRA
- Figure out release process for L1
- Look at documentation: old Workbook, Github, Confluence..
nEXO
- Attend sims meeting
- Work with SLAC IT to get all necessary system libraries installed on centos7 to build nEXO's externals
  - libXpm-devel
  - bzip2-devel
  - Motif
  - Motif-devel
  - mesa-libGLU mesa-libGLU-devel
- Work through build problems with Coin3d 3.1.3 (an external lib)
  https://github.com/nEXO-collaboration/nexo-env/issues/1
- Finish building nexo SNiPER on centos7
- Work on HistogramSvc
- Create some initial dev doc in their wiki
DM-DC2
- Attend and take notes for task force meeting
- Attend SSims meeting Thursday
DESC
- Create docker image using new lsst_sims + w_2018_09
NERSC
- Install new lsst_sims + w_2018_09
- Add note about lsst_distrib + lsst_sims on Confluence: NERSC Software Installations
- Install CCL At NERSC in the shared DESCQA python env
  - GH Issue https://github.com/LSSTDESC/CCL/issues/346
  - Updated CCL NERSC Wiki: https://github.com/LSSTDESC/CCL/wiki/Using-and-Installing-CCL-on-NERSC
- Work out production installation of lsst_sims + w_2018_09 using the shared python env
SLAC
- Continue watching jupyterlab progress
- Create some documentation to use jupyterlab at SLAC
- Update Fermi and DESC SLAC Account instructions - maybe nEXO's too
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1256
ATS At Harvard
- Attend test stand tutorial 9 AM - 10 AM PT Wednesday

Week of Feb 26 2018

Fermi
- get Brian to fully clean up between GR Jenkins builds - created JIRA
- get L1 GR to build
- figure out how to clean up workspaces in Jenkins on my own.
- Tag first GlastRelease release in github GlastRelease-20-10-04-gr03 only change is to the xxxRootData packages
- Figure out the release process
nEXO
- send an email update to Lisa
- update Slack channel about Raymond's discussions with SNiPER developers
- Work on HistogramSvc for ROOT files
DM-DC2
- Run Monday Meeting and write minutes
- Update GitHub issues
- Contact Chris W about flats
- Talk to Tom about setting up directory for transfer to IN2P3
- test ingestDriver.py at NERSC
  https://github.com/LSSTDESC/ImageProcessingPipelines/issues/6
- Update IPP repo with new mini-DRP Epic
  https://app.zenhub.com/workspace/o/lsstdesc/imageprocessingpipelines/boards?repos=112563170
- Create new SRS pipeline task to run ingest on eImages
  http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=50591002
- Update SRS pipeline task to handle ref cat ingest
- Update SRS pipeline task to run processEimage
- attend DC2 meeting
NERSC
- Poke NERSC about jupyter-dev outage - now fixed
SLAC
- Watch JupyterLab demo install and keep Steve informed
- Try out eT DB updates to prep for Joanne's departure
- Update DESC and Fermi documentation on getting a SLAC account
- Check on Tom M's reported problem with the status of some CCD-011 bits in the DP
  - Fixed
    Jira
    server SLAC National Accelerator Laboratory
    serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
    key LSSTTD-1220
ATS At Harvard

Week of Feb 19 2018

Fermi
- Finish Anaconda 3 install
- Fix up L1 builds on Jenkins
- Fermi Meeting
nEXO
- Start new HistogramSvc for ROOT files
- Simulation Meeting Thursday
DM-DC2
- First Task Force Meeting
- Migrate DC2 Cookbook issues to IPP (ImageProcessingPipeline) repo
- Create new issues in IPP repo
NERSC
- sort out HPSS allocation
SLAC
- Help David L access his account
ATS at Harvard
- Meeting with Tony & Owen on Tuesday
- Go to Harvard on Friday

Week of Feb 12 2018

Fermi
- attend offline meetings!
- Finish Anaconda3 installation
- Work out distribution of externals in the new regime
- Install swig 3.0.12 RH6 and RH7 on GLAST_EXT
- work out L1 build in Jenkins
nEXO
- Meet with sims people
NERSC Support
- re-re-install cosmosis to help out the WL folks who wanted the latest along the develop branch
- Reach out to Joe to encourage tagging of cosmosis and provide some feedback on their setup script for NERSC
- Reach out to wl pipeline group to ask about cosmosis installations.
- Start up automated weekly stack installations
- Document weekly stack installations
- Send Joe Z updates to the setup-cosmosis-nersc script
- Help Erfan get going with cosmosis at NERSC
SLAC support
- Read over Joanne's docs
- Pow-wow with Joanne over the DP and eT and the eT DB
- set up docker for Jim on IR2
- Update DESC and Fermi documentation on getting a SLAC account
DRP/L2 Processing
- Task Force
  - Phil mentioned that "they" had to write up a charge - figure out what that means
    - Modify charge after receiving suggestions
  - At Nicolas' suggestion reach out to Fabio H concerning weekly stack installations
ATS at Harvard
- Attend CCS meeting to find out how best to support Harvard
- Contact Harvard to sort out list of devices & plans for their use for testing
- Meet with Kirk
- write up minutes from Kirk/Nick meeting and prepare to iterate with Kirk

Week of Feb 5 2018

DRP/L2 Processing
SLAC DESC support
NERSC support
- set up automated installs : https://github.com/lsst-dm/shared-stack
- Help Alex set up web data access: http://www.nersc.gov/users/data-analytics/science-gateways/
  https://github.com/pmelchior/des-exp-checker
Other
- Read over Joanne's docs: http://corner.readthedocs.io/en/latest/install.html
- Write up IN2P3 account Confluence page
- Update DMstack install on Harvard AUX Telescope machine
nexo
- Attend Sims meeting
- Schedule time to meet Lisa next week
- Try out Raymond's example using SLAC install
- finish SLAC installation
Fermi
- Finish Anaconda 3 installation at SLAC
- work out distribution of externals for end users

Week of Jan 29nd 2018

Data Portal
- support Joanne as she learns about the plotting
DRP/L2 Processing
- Check on James P
- Get a copy of the DC1 WF going
- Update WF to run Jim's latest imSim Data
- Learn how to deal with pipeline issue in JMX dealing with large number of jobs stuck in submitted state
- Run Jim's imSim data through SLAC WF (first attempt in progress)
- Prep to run using Phosim 1.1p
- Document using the Slac WF and add scripts to a GH repo in LSSTDESC area
- attend Friday dc2 meeting
SLAC support
- review, release, and install the latest version of datacat-utilities
- Read over Joanne's documentation
- -~~automate weekly install of DMstack binaries for camera-~~
- install datacat-utilities on lsst-dc10
NERSC support
- Upload latest SQRE DMstack (w_2018_03) Docker image into Shifter
- Install w_2018_03 plus sims from source
- Create w_2018_03+sims Shifter Image (debugging issue with afw - issue identified.. fixing it is something else)
  - create sims Shifter image the long way, without using lsstsqre's base image
- Install WeakLensingDeblending for Javier.
Random LSST Camera and DESC items
- SSim meeting
- Meet with Wil on Friday
- Write up notes from meeting
- Look over DESC architects confluence page
nEXO
- reinstall SNiPER in the new disk space
- try out examples
- sign up for the collaboration mailing list
Fermi
- Investigate trouble with AcdDigi in L1 branch
- Find a way to upgrade Anaconda external for ST
- Install miniconda py3
- Contact Helen

Week of Jan 22nd 2018

Data Portal
- Consider rerunning all the ITL sensors in need of updated MET ingest and MET 05 by hand
- plots - commit latest code and write up intro for Joanne
- CCB request and deploy MET updates for the DP
- Deploy DP 0.5.11
SLAC support
- automate weekly install of DMstack binaries for camera (Jim doesn't seem to feel this is a priority)
- Read over Joanne's documentation
NERSC support
- update DESCQA install to include tensor flow
- write up instructions for w_2018_01 shifter install
- automate installation of shifter images
- Create docker image of new sims release based on w_2018_04
- Reintall cosmosis for WL group due to updated FFTW module, trying cray-fftw this time
- Send instructions to Josh to set up his own test galsim using the extlibs we already have set up for Cori KNL gcc
- install Phosim 3.7.7 and 3.7.8 for knl-intel and haswell-intel
- meet with Mustafa about HPSS
- Fix Cosmosis installation
Workflows/L2 DC2 Pipeline
- send plea to Tony
- Meeting with Tom about using the SLAC pipeline + pilots
- work out setting up test Pipeline2 job reusing DC1 L2 task and a handful of visits
- Prep for CI meeting
Fermi
- Work through errors with AcdDigi on L1 branch
- Set up new Anaconda external
- Work out with Brian the requirements for setting up externals these days
nEXO
- get Tony an disk space estimate for SLAC installs
- get some disk space from Tony at SLAC
- set up SNiPER at SLAC in a temp location (in progress)
- attend Tuesday sims meeting

Week of Jan 15th 2018

Data Portal
- revisit plots
- Holding off MET updates until new traveler to reingest ITL sensors is available
datacat-utilities
- Install new release 0.0.9 at SLAC
SLAC support
- automate weekly installs of DMstack binaries for camera
NERSC Support
- update DESCQA install to include tensorflow
- Write up instructions for w_2018_01 shifter install
- automate installation of shifter images
- Add WeakLensingDeblending
- Check on CCL installation
Workflows/L2 DC2 Pipeline
- DC2 meeting Friday Jan 19th
- Slim down docker image which includes sims - now down to 3 GB on DockerHub https://hub.docker.com/r/heather999/test/tags/
- Slim down docker image which includes sims - now down to 3 GB on DockerHub https://hub.docker.com/r/heather999/test/tags/
Fermi
- Set up draft f2f meeting page Fermi Software Week Winter 2018
- Continue working on GR L1 branch - completed checkout of L1 code successfully
- Work through errors with AcdDigi on L1 branch
nEXO
- Read over nEXO at OSG simulation notes https://ntpc.ucllnl.org/nEXO/index.php/Simulations_on_the_OSG

Week of Jan 8th 2018

DataPortal
- finish off ITL-3800C-368 ingest after sorting out http authentication errors
- revisit plots
- Hold off on MET updates until new traveler to reingest ITL sensors is available.
datacat-utilities
- Install new release
SLAC suport
- Install w_2018_01 binaries lsst_distrib for camera
- Work out automated weekly installs via cron (u1 is 97% full, ~990 GB left)
- Install Merlin's eotest branch
NERSC Support
- Update DESCQA install to include tensorflow
- Try w_2018_01 shifter install (in progress)
- Testing w_2018_01 shifter install and write up instructions
- Add WeakLensingDeblending https://lsstc.slack.com/archives/G8RU8C4VD/p1515604937000489
Workflows/L2 DC2 pipeline
- Set up lsst_sims docker image for NERSC & uploaded to Dockerhub
- Try another lsst_sims docker image using lsstsqre image as the base rather than centos7 (initially failed, have new plans to retry)
Fermi
- ~~Attend Wednesday software meeting~~
- Finish setting up GR L1 branch https://github.com/fermi-lat/GlastRelease
- Test L1 build in Jenkins https://fermi-lat.github.io/doc/workflow.html
- Try interactive build of GR L1 branch
nEXO & SNiPER
- Attend Tuesday software meeting
- Contact SNiPER team about Open Source License
- Apply for OSG and LLNL accounts
- Read over nEXO at OSG simulation notes https://ntpc.ucllnl.org/nEXO/index.php/Simulations_on_the_OSG

Week of Dec 18th 2017

DataPortal
- MET updates
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1221
- Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1220
Workflows/L2 DC2 pipeline
- Reproduce error James P sees
- Repeat experiment with multiple visits
- Repeat experiment using Head of ImageProcessingPipeline repo
- Potentially meet with Jim & James this week
- Catch up on Parsl progress
CI
- Set up new Docker image to use with imSIm's travis builds
- Create a smaller Docker images for use with travis
NERSC Support
- Shared Stack
- JupyterHub documentation and simplification
- clean old project space by removing MBII-snapshots
- Request reduction in old project space quota to 40 TB
imSim
Fermi
- Set up GR L1 branch in GitHub
- Test build of L1 branch
SNiPER
- Read up on SNiPER
- Meet with Lisa K on Tuesday 5 PM Eastern
- send feedback on documentation
- set up user accounts on Wiki, LLNL, OSG
- talk to Tony about nExo

Week of Dec 11th 2017

Data Portal
- Deploy ETCCB-429
- MET updates - almost complete
CCS ATS
- Install mvn and associated libraries and get Nick going with Lakeshore driver
Workflows/L2 DC2 pipeline
- Send James P a link to the recordings from last week
- Repeat test using Jim's parsl wf at NERSC
CI
- Docker image for use in travis builds
imSIM
NERSC Support
- Fresh Shared Stack installation for KNL - stuck working out disk space in the common software area – started
- dmtcp installation
- Update jupiter-dev documentation
- quota checker
Fermi
- Try GR with repoman
- Documentation

Week of Nov 27th 2017

L2 pipeline DC1 rerun
- Finish off write up of chat with Tony
- Ask James P about remote participation during Spring Week
  M & W 8 - 12 central time
- Try a run using Simon's cookbook
- Meet with Jim & James P
  - write up notes
- Try Parsl pipeline at NERSC
NERSC software installations
- Shared Stack installation - verify that the DESCQA and py3 install is working
  - Add CCL
- Install new sims 2.4.2 using DMstack w.2017.46 Haswell-gcc6 using py3
- Ask about cosmosis needs
  - Install cosmosis, TreeCorr, and MLZ in projecta area under the WL WG directory
- Install phosim 3.7.3
- dmtcp
- quota checker
- Consider requesting more common software space
  obtained 50 GB more
NERSC project space
- Verify MBII-snapshots is in HPSS and remove from project area
  Found the content in the desc hsi area, and viewed the contents of some snap.tar via htar -tf CS/MBII-DMO-snapshots/snapdir_074.tar Also noticed that the group_xxx.tar files are duplicated in the Archive home directory. Contacted Mustafa to sort that out and remove duplicates.
Data Portal
- MET updates
  - Meet with Anders and Seth
Harvard ATS
- Check that ccs reboot restarts the MySQL, LocalRestServices, and Trending
- Check in with Nick about the Lakeshore
Fermi
- Try GR using repoman
- Documentation
Discuss Gaudi for nEXO
https://gaudi.web.cern.ch/gaudi/releases/v30r0/
http://lcgsoft.web.cern.ch/lcgsoft/release/
https://gitlab.cern.ch/gaudi/Gaudi
Attend Stanford IT Unconference
https://itunconference.stanford.edu/sessions/remote

Week of Nov 20th 2017

L2 pipeline DC1 rerun
NERSC software installations
- py3 shared stack installation
- finish phosim 3.7.2 installation
- dmtcp intel installation
- quota checker
- cosmosis installation
  - check in on GH issue https://github.com/LSSTDESC/WLPipe/issues/31
Cover Vendor Ingests M-W
Data Portal
- plots
- check in with Tony about his changes to load the page more quickly
Harvard ATS
- Determine who will deal with the Lakeshore drivers
  https://jira.slac.stanford.edu/projects/LSSTCCSDRIVER/issues/LSSTCCSDRIVER-227
  https://github.com/lsst-camera-ccs/org-lsst-ccs-drivers-lakeshore/tree/LSSTCCSDRIVER-227
  https://www.lakeshore.com/Documents/336_Manual.pdf
- Grab Owens updates and install on CCS machine
- Direct Nick to make updates as needed.
- Fix mounting of data drive on ccs machine
- Try a version of the TS subsystem on the ccs machine and set up groovy & properties files
- Check monochromator plays well with the TS subsystem
Fermi
- Try a GR build with repoman
- Documentation
- Inquire about pointlike updates in CVS on Slack

Week of Nov 13th 2017

Harvard ATS
- Create user accounts and ccs sudo account which grants sudo access to all users
- Set up directory structure following IR2 model
- Install DMstack v14
- Install eotest compatible with v14
- document the above on the Confluence page ATS Test Stands
- Set up Trending Database
  - MariaDB installation
L2 pipeline DC1 rerun https://github.com/LSSTDESC/SSim_DC1/issues/47
- Meet with Jim on Monday
  - Write up Notes
- Contact James P
- Try to run Simon's Cookbook
- Chat with Tony
NERSC Software Installations
- Shared Stack
  - first try failed due to conflicts with libiconv installed presumably for
- Phosim 3.7.2 bug fix for haswell gcc & intel, KNL gcc & intel
- DMTCP intel build
- Help out with cosmosis installation https://github.com/LSSTDESC/WLPipe/issues/31
Data Portal
- Chat with Tony about plots
- Plots
  - Bad Channels
  - read noise
NERSC Project Space
- Test MBII-snapshot HPSS copy and remove from project area

Week of Nov 6th 2017

QA Testing
- Investigate cause of differences between Lauren's plots from NCSA and those generated at NERSC
  - look at 'conda list' on both NCSA and NERSC (no differences discovered)
  - generate cal_refs as was done at NCSA
  - try gcc 6 build as is being done at NCSA
- Determine the cause of differences between ref catalogs at NCSA and NERSC
L2 Pipeline
Data Portal
- Finish off first round of Richard's plots
  - bad channels
  - read_noise
NERSC Project Space
- test MBII-snapshot HPSS copy
ATS
- prep for Tony & Owen's visit
Installations
- SLAC
  - Py3 on SLAC RH6-64 for Joanne to test out
- NERSC
  - finish testing NERSC's cfitsio installation
  - install phosim 3.7.1 for Cori haswell gcc & intel, KNL gcc & intel
    - ask Glenn to install module files
  - install phosim 3.7.2 for Cori haswell gcc & intel, KNL gcc & intel
  - Shared Stack Installation using w.2017.41
    - Query DESCQA and CCL groups
    - py2 w.2017.41 installation
    - py3 w.2017.41 installation

Week of Oct 30th 2017

Data Portal
- Plots
DC2 prep
- QA testing
  - w.2017.41 installation plus branches of obs_lsstSim and pipe_analysis
  - Run visitAnalysis, coaddAnalysis, etc
NERSC Workflow work
NERSC Project Space
- Work with Mustafa to get MBII-snapshot backed up
NERSC Installations
- w.2017.? stack installation plus sims - waiting on new sims release using the camera geometry updates
- Phosim 3.7
  - Testing NERSC cfitsio installation
SLAC Installations
- descsoft w.2017.? stack plus sims installation
CCS
- See if there is any prep work necessary for Tony's visit later in Nov
  - Request Nick set up new box with CentOS7
  - set up user accounts and groups
- Determine if Mike is still coming out ~Nov 1st
Attend SuperCDMS meeting concerning their potential move to NERSC.

Week of Oct 23rd 2017

Data Portal
- Richard's Plots - bad channels (in progress)
- Look at new Report Labels and determine what the DP still requires from the Reports Table
  Jira
  server SLAC National Accelerator Laboratory
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1190
- Make sure we request a restart of Tomcat
NERSC Workflow work
- Try out HTCondor and Pegasus on cori21
- Take a look at Jim's imsim script
NERSC Project Space
- Set the old project directory to read-only (in progress)
  - claim desc ownership of the whole directory space
- Clear out all directories where the owners have indicated it is fine to do so
- Determine the end state of merging old project and projecta areas and handling of www
- Set up disk quota monitor
- Coordinate HPSS storage of MBII-snapshots
NERSC installations
- Phosim v3.7
  - Coordinate with NERSC to get cfitsio v3410 installed (in progress)
  - Rebuild using NERSC supplied cfitsio
  - Coordinate with Glenn to set up module files
- Set up software for DC2 testing
- v14_0 lsst_distrib + sims 2.4.0
SLAC installations
- Set up same imsim at SLAC as used at NERSC for KNL testing (in progress)
- v14_0 on RH6-64 public (on hold pending resolution to obs_base testCameraMapper.py unit test failure)
  - camera area
  - descsoft
  - test eotest branch
- v14_0 on IR-2
Document KNL vs Haskell mkl imsim testing at NERSC
- Initial mkl tests indicate no huge improvement - putting testing on back burner for now
CCS - still waiting to hear from Peter, I cannot access their TS3 machine to install and test a new version of the test stand subsystem

Week of Oct 16th 2017

Richard's plots in DP
- Complete Sensor Arrival Time plot
- Create JSON
NERSC installations
- Phosim v3.7
  - Try out NERSC supplied FFTW
  - Request new cfitsio v3410 installation required by Phosim
  - Complete installation of all flavors: haswell-gcc, haswell-intel, knl-gcc, knl-intel
    - Waiting on cfitsio v3410 before knl-intel will be available
NERSC Project Space
- Notify all users of the need to migrate
- Handle all help requests - migrating some directories to the new projecta area
- Install disk quota monitor
- Check on old project status and prepare to set it to read-only
- Coordinate backup of nkhandai data to tape with Mustafa
Install latest version of teststand subsystem at Harvard there is a machine made available - waiting on response from Peter
Keep an eye out for v14 DMstack

Week of October 9th 2017

Continue work on Richard's plots in DP
Jira
server SLAC National Accelerator Laboratory
columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
key LSSTTD-1165
- Released Data Portal 0.5.7
Testing imSim built on KNL
- Continue MKL tests at NERSC once the system recovers from the weekend outage
- Install the same version of the stack at SLAC to allow similar testing with and without MKL
- Attempt build of DMstack using the intel compiler at NERSC
NERSC Project space
- Install disk quota monitor
- Coordinate backup of nkhandai data to tape with Mustafa
- Prepare to set the old project area as read-only
- Send mail to all owners of files on the old project space
NERSC HPSS SRU overallocation
Install latest version of datacat-utilities 0.0.8 at SLAC
Install the latest teststand snapshot at Harvard to support use of the monochromator when Peter returns.
- Contacted Peter - I can't connect to the archon machine to install the test stand snapshot
- Investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly create a harvard main routine.
Keep an eye out for the v14 DMstack release and prepare to install on IR-2 and NERSC
- This release will include binary installations.. I'm thinking we may still want to install from source, perhaps that's most useful on IR-2, but we'll see.

Week of October 2nd 2017

Release DataPortal 0.5.6 which will include
- Addition of Grade and Procurement labels for the Sensor Acceptance pages
- RTM assignment
  Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1121
- NCR "Ready to close"
  Jira
  server SLAC National Accelerator Laboratory
  columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
  serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
  key LSSTTD-1144
Implement Richard's plots in the DP starting from some of Tony's examples for trending and Richard's python code.
Continue testing KNL performance using galsim and imsim
- Attempt to compile galsim and possibly all of DMstack with the intel compiler
- Run longer tests using the MKL libraries on Haswell and KNL
- Rebuild imsim with MKL, using an updated meas_base
Install an initial shared stack in the new software area at NERSC for testing as discussed at last week's "Shared Stack" meeting
NERSC Project Space
- Set up an initial quota monitor at NERSC
- Re-connect with Mustafa to get the mkhandai 42 TB archived to tape.
Investigate an upgrade of the monochromator driver at Harvard and confirm that to achieve what Peter plans to do - to run his ATS work in the lab alongside most of the devices from his TS3 setup.. is possible by running both the teststand and rafts subsystem simultaneously.
- Install latest SNAPSHOT of test stand at Harvard and investigate creating a harvard.groovy (perhaps based on TS8bench or TS7) and possibly creating a harvard main routine.
Keep an eye out for the v14 release of the DMstack and prepare to install at SLAC/IR-2 and NERSC.
Though this is assigned to Jim, so perhaps I have no need to worry about SLAC:
Jira
server SLAC National Accelerator Laboratory
serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546
key LSSTTD-1157

...

Confluence and Jira now require federated login. Read more.

Child pages

Page History

Versions Compared

Old Version 605

New Version Current

Key

Week of 12 April 2021

Week of 29 March 2021

Week of 15 March 2021

Week of 8 March 2021

Week of 1 March 2021

Week of 22 Feb 2021

Week of 15 February 2021

Week of 1 February 2021

Week of 25 January 2021

Week of 18 January 2021

Jan

Week of 7 December

Week of 30 November

Week of 16 November

Week of 9 November

Week of 2 November

Week of 26 October

Week of 19 October

Week of 12 October

Week of 5 October

Week of 28 September

Week of 21 September

Week of 14 September

Week of 7 September

Week of 31 August

Week of 17 August

Week of 3 August

Week of 27 July

Week of 20 July

Week of 13 July

Week of 6 July

Week of 29 June

Week of 22 June

Week of 15 June

Week of 8 June

Week of 29 May

Week of 25 May 2020

Week of 18 May 2020

Week of 11 May 2020

Week of 4 May 2020

Week of 27 April 2020

Week of 20 Apr 2020

Week of 13 Apr 2020

Week of 6 Apr 2020

Week of 30 Mar 2020

Week of 23 Mar 2020

Week of 16 Mar 2020

Week of 9 Mar 2020

Week of 2 Mar 2020

Week of of 24 Feb 2020

Week of 17 Feb 2020

Week of 10 Feb 2020

Week of 3 Feb 2020

Week of 27 Jan 2020

Week of 20 Jan 2020

Week of 13 Jan 2020

Week of 6 Jan 2020

Week of 16 Dec 2019

Week of 9 Dec 2019

Week of 2 Dec 2019

Week of 18 Nov 2019

Week of 11 Nov 2019

Week of 4 Nov 2019

Week of 25 Oct 2019

Week of 14 Oct 2019

Week of 7 Oct 2019

Week of 30 Sept 2019

Week of 23 Sept 2019

Week of 16 Sept 2019

Week of 3 Sept 2019

PCW

Week of Aug 5 2019

Week of July 29 2019