DMstack updated python env

as of w_2018_35 (perhaps 34) we're using the updated python environment.

DESC DRP data storage at NERSC

cp -as /global/projecta/projectdirs/lsst/global/in2p3 $SCRATCH/DC2/global

created real copies of repositoryCFG.yaml files for Run1.1p and Run1.2p

 

Moving GitHub issues: https://github-issue-mover.appspot.com

 

Fixed problem when a user at NERSC has their own local python packages installed under HOME and trying to use a different conda env:

https://github.com/conda/conda/issues/448


git recover deleted file

https://stackoverflow.com/questions/953481/find-and-restore-a-deleted-file-in-a-git-repository

Binary lsst_distrib and lsst_sims at SLAC

scl enable devtoolset-6 bash
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
curl -LO https://raw.githubusercontent.com/lsst/lsst/w.2018.26/scripts/newinstall.sh  (get the right version of newinstall)
bash newinstall.sh -ct
source loadLSST.bash
eups distrib install -t w_2018_26 lsst_distrib --nolocks
eups distrib install -t sims_2_9_0 lsst_sims --nolocks
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py
  adding conda install ipython jupyter pylint 
edit loadLSST.bash to include "source scl_source enable devtoolset-6"
 
Remember the extra downloads for lsst_sims
setup lsst_sims
bash /global/common/software/lsst/cori-haswell-gcc/stack/w.2018.26_sim2.9.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/sims_catUtils/2.9.0.sims/support_scripts/get_kepler_light_curves.sh
bash /global/common/software/lsst/cori-haswell-gcc/stack/w.2018.26_sim2.9.0/stack/miniconda3-4.3.21-10a4fa6/Linux64/sims_catUtils/2.9.0.sims/support_scripts/get_mdwarf_flares.sh


Binary lsst_distrib and lsst_sims at SLAC

scl enable devtoolset-6 bash
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
curl -LO https://raw.githubusercontent.com/lsst/lsst/w.2018.26/scripts/newinstall.sh  (get the right version of newinstall)
bash newinstall.sh -ct
source loadLSST.bash
eups distrib install -t sims lsst_distrib --nolocks
eups distrib install -t sims lsst_sims --nolocks
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py
  adding conda install ipython jupyter pylint


Binary DMstack installations at SLAC

scl enable git19 bash
scl enable devtoolset-6 bash
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
curl -OL https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh (check this)
bash newinstall.sh -ct
source loadLSST.bash
eups distrib install -t w_2018_01 lsst_distrib  (installed w_2018_23 June 19th 2018)
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py
  adding conda install ipython jupyter pylint

Using EUPS to set up tags

https://dev.lsstcorp.org/trac/wiki/EupsTips#Tags

Defin eotest-merlin as a possible tag in the stack's startup.py via the line:
hooks.config.Eups.globalTags += ["dc2","dc2-18"]


source loadLSST.bash
setup lsst_distrib
git clone git@github.com:lsst-dm/eotest.git
cd eotest
git checkout tickets/DM-11348
eups declare -r . eotest -t eotest-merlin
setup -j -r .
scons
scons install
scons declare

Then another user comes along and does

source loadLSST.bash

setup lsst_distrib

setup eotest -t eotest-merlin


https://lsstc.slack.com/archives/C2K97E3L5/p1515774127000703 Note to SLACK asking how to use tags

eups repo: https://github.com/RobertLuptonTheGood/eups

Docker images for NERSC

Found some other references to setting up docker images of sims.

Also large: https://hub.docker.com/r/oboberg/maf/

Found James P's dockerfiles: https://lsst-uk.atlassian.net/wiki/spaces/HOME/pages/141328388/LSST+Simulation+Docker+Image which are using the LSSTSQR as a base.

Trying out an install of the weekly w_2018_01 into NERSC shifter via shifterimg pull lsstsqre/centos:7-stack-lsst_distrib-w_2018_01

Trying to build sims against lsstsqre/centos:7-stack-lsst_distrib-w_2017_46 doesn't work, maybe due to gcc.. trying again with w_2018_01

 

2018 Oct 4

Installing newer Anaconda 5.2.0 on SLAC NFS for us with the datacat tools.  Chose version based on the miniconda currently used in the stack weeklies (miniconda 4.5.4) which is close to the release of Anaconda 5.2.0.. hopefully both include python 3.6.

completed the install.. waiting for new datacat compatible with py3 before making datacat-utilities available with py3.6

2018 July 20

set up imdim repackager branch at NERSC

source setup_w_2018_26-sims_2_9_0.sh

setup lsst_sims

eups declare -r . imsim -t repackager

setup imdim -t repackager

scons

2018 July 19

imsim, DMstack set up at SLAC on gpfs bullet machines

Using RH6, gcc6.3.1, ran into trouble installing from source where the pipe_tasks unit tests fail, apparently due to using too many cores https://lsstc.slack.com/archives/C4RKBLK33/p1531952389000043

The binary builds of lsst_distrib and lsst_sims do not work out of the box, if using galsim.  Had to edit a meta_data.py file: https://lsstc.slack.com/archives/D2R2L88CW/p1532014239000182

https://lsstc.slack.com/archives/C4RKBLK33/p1531953371000087

 

  • No labels