2017 Notes

# Usage: source loadLSST.bash

source scl_source enable devtoolset-3

/nfs/farm/g/desc/u1/

1) Login to corigrid.nersc.gov as yourself

2) module load globus

3) myproxy-logon -s nerscca.nersc.gov

Enter your password

4) gsissh localhost -l desc

[heather@LSST-PC92228 rest]$ python hack.py prod Prod getResults getResultsJH-SR-RTM-noise.json ""

16 Nov 2017

vnc at Harvard https://www.howtoforge.com/vnc-server-installation-on-centos-7
still starting up the vncserver on reboot for now 

15 Nov 2017

MariaDB [(none)]> create user 'ccs'@localhost identified by 'lsst';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* to 'ccs'@localhost WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> CREATE USER 'ccs'@'%' IDENTIFIED BY 'lsst';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'ccs'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> create database atsccsdb;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> GRANT ALL on atsccsdb.* to 'ccs' IDENTIFIED BY 'lsst';
Query OK, 0 rows affected (0.00 sec)

Nov 9 2017

installing Shared Stack at NERSC

confirmed DESCQA uses py2 , and no one seems to object to gcc6 

installed miniconda 2 4.3.21 (looked at newinstall), using lsstsw package list

doing a fresh installation of lsst_distrib w_2017_41 and then followed by the release sims or perhaps something else..

Nov 6 2017

Installing MIniconda3 - 4.2.12 based on the version we get from NERSC for v14  This is for Joanne on RHEL6-64 to allow testing of py3 

py3-4.2.12

NERSC

py3-4.2.12 installation miniconda followed by

conda install --file conda_packages-linux-64.txt

This sets up the DM set of conda packages.  From this root environment, we can set up DM, DESCQA, CCL.. etc? hopefully

Then set up w.2017.41 against this installation using gcc6, to achieve a shared stack.

For QA testing

copied Jim's data off NCSA using tar -cvzf data.tar.gz -C dirwheredatais .

to avoid full path in the tarball

Oct 30 2017

Oct 26, 2017

V14 DMstack release. 

IR2

#!/bin/bash
cd /lsst/dh/software/centos7-gcc48/stack/v14_py2
curl -OL https://raw.githubusercontent.com/lsst/lsst/14.0/scripts/newinstall.sh
bash newinstall.sh -b -2 -ct
source loadLSST.bash
echo 'loadLSST.bash complete'
eups distrib install -t v14_0 lsst_distrib
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
mkdir site
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py
conda install -y ipython
conda install -y jupyter
conda install -y pylint

sudo -u du mkdir stack  (previously installs were conda installations)

sudo -u du chmod 2775 stack

sudo -u du bash newinstall.sh -2 -ct

source loadLSST.bash

granted myself w privileges via the group and then proceeded to do the

eups distrib install -t v14_0 lsst_distrib

where I"ll chown the whole directory structure back to dh user.

Need to set permissions for shared stack.

scl enable git19 bash
scl enable devtoolset-3 bash
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
bash newinstall.sh -ct
source loadLSST.bash
eups distrib install -t v14_0 lsst_distrib
curl -sSL https://raw.githubusercontent.com/lsst/shebangtron/master/shebangtron | python
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py
  adding conda install ipython jupyter pylint which upgrades numexpr to a nomkl version (which probably makes sense - why doesn't DM do that??)
could not install mysql-python, due to its dependency on py2

Miniconda3-4.2.12-Linux-x86_64.sh

unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
bash newinstall.sh

[ 75/106 ]  obs_base 14.0 ... 
***** error: from /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/Linux64/obs_base-14.0/build.log:
  File "tests/testCameraMapper.py", line 403, in tearDown
    shutil.rmtree(self.ROOT)
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/python/miniconda3-4.2.12/lib/python3.5/shutil.py", line 474, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/python/miniconda3-4.2.12/lib/python3.5/shutil.py", line 416, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/python/miniconda3-4.2.12/lib/python3.5/shutil.py", line 414, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'b'
----------------------------------------------------------------------
Ran 24 tests in 8.955s
FAILED (errors=1, expected failures=1)
The following tests failed:
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/Linux64/obs_base-14.0/obs_base-14.0/tests/.tests/testCameraMapper.py.failed
1 tests failed
scons: *** [checkTestStatus] Error 1
scons: building terminated because of errors.
+ exit -4
eups distrib: Failed to build obs_base-14.0.eupspkg: Command:
    source "/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/eups/2.1.4/bin/setups.sh"; export EUPS_PATH="/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/stack/miniconda3-4.2.12-7c8e67"; (/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/Linux64/obs_base-14.0/build.sh) >> /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/Linux64/obs_base-14.0/build.log 2>&1 4>/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v14/stack/miniconda3-4.2.12-7c8e67/EupsBuildDir/Linux64/obs_base-14.0/build.msg 
exited with code 252

Oct 24, 2017

Setting up SLAC install of imsim used at NERSC for KNL testing

skipping unit testing due to failure at SLAC

eups declare sims_catUtils 2.3.6.sims -r .
setup -r .
scons

2017 Oct 11

SRU allocation exhausted.  Found Top 7 users via NIM:

https://nim.nersc.gov  set Repository to m1727 in upper right

click on "User Status by Repo"

Individual User reports can be generated by clicking on "Reports"=>Daily Use/Repo

fill in the search filter

Users control their Project Percentage

PIs control user quotas

SRUs are charged for reading, writing, and storage

http://www.nersc.gov/users/storage-and-file-systems/hpss/hpss-charging/

2017 Oct 5

plots

python hack.py prod Prod getResults getHardwareInstances.json ""

====

Rebuilding meas_base using Mario's updates to allow use of the MKL

https://github.com/lsst/meas_base/compare/master...mjuric:mkl-support?expand=1

had to update Mario's code to use the new log package, replacing pex_logging

https://github.com/lsst/log/blob/master/include/lsst/log/Log.h

copied the meas_base code from the failed installation and copied into my own directory and removed build products

eups declare meas_base 13.0-20-g03c8adc+9 -r .
setup -r .
icons

2017 Oct 4

Pegasus @NERSC

http://www.nersc.gov/users/data-analytics/data-analytics-2/python/python-on-cori-knl

KNL Hack Day

NERSC KNL Hack Day Notes

Group Manager interface: http://java.freehep.org/svn/repos/srs/show/org-srs-groupmanager/trunk/jar/src/main/java/org/srs/groupmanager/GroupManager.java?revision=HEAD

2017 Sept 25

David Thomas desires a SLAC DMstack lsst_distrib, lsst_sims + phosim installation.  Trying our w_2017_26_py3 under the descsoft account first.

https://github.com/lsst/lsstsw/tree/master/etc

conda create --name base --file conda3_package-linux-64.txt from the lsstsw package

2017 Sept 22

Chat with Tom about NERSC space

• Ongoing issue with inode allocation
  • How many nodes are used per file on average?
  • Request 20 M ?
  • Already used 92% of 5M with 1/3 of 140 TB
• Disk use in production
  • 17 TB DC1_imsim - some cleaning up may be possible, but it does not account for much space
  • DM 21 TB
  • DC1_phosim3 46 TB
• WG directories
  • Create a dedicated user for project writing, that everyone is a member of. This allows everyone to clean up, but eliminates accountability.
  • Tom wonders how Posix extended ACLs handle setting permissions when dealing with unpacking tar files
  • Contact NERSC for suggestions on managing our project space and if the GPFS tools to manage quotas are available.
  • Monitoring quotas via a script run 1/day, assuming it doesn't run too long.  Plot user vs time
• SCRATCH
  • How to manage purging?  Maybe a monthly cron job that copies and unpacks a tarball
• Software area
  • For batch only
  • manage who has the ability to install s/w
  • Establish purging policy - retain production software or archive to HPSS or rely on the ability to rebuild at will
  • Is the Anaconda module useful to us?
• Emptying old project space
  • Tom put in a question to Mustafa to see how soon NERSC needs us out, there may be no real pressure
  • Float the idea of setting the old area read-only on Oct 1st with the goal of emptying it by Jan 1st

2017 Sept 21

Project portal at NERSC - troubles with DESCQ, still pointing to the old area.  Should figure out when we should point to the new one.

Doc NERSC disk quota tools... and monitory disk and inode use

https://my.nersc.gov/data-mgt.php

Project space write up

Clean up contrib area

Prep for KNL Hack Day

Check on running shifter image at NERSC

python hack.py prod Prod getContainingHardware getHardwareContain.json

[heather@LSST-PC92228 ~]$ python hack.py dev Prod getResults getMissing.json "jrb"

2017 Sept 5

Installing DMstack w_2017_26 using Miniconda py3, plus latest eups 2.1.4 which allows read-only stack for use by other users

scl enable git19 bash
scl enable devtoolset-3 bash
curl -LO https://raw.githubusercontent.com/lsst/lsst/w.2017.24/scripts/newinstall.sh
edit version of EUPS
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
bash newinstall.sh -3
source loadLSST.bash
eups distrib install -t w_2017_26 lsst_distrib --nolocks
eups distrib install -t sims_2_3_6 lsst_sims --nolocks
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py

2017 Sept 1

trouble with v13_0 on ir2 box lsst-dc10

  LSSTTD-1124 - Getting issue details... STATUS
Looking back at:  LSSTTD-922 - Getting issue details... STATUS

[v13_0] sudo -u dh env "PATH=$PATH" conda install --name v13_0 pyqt=4.11
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /lsst/dh/software/centos7-gcc48/anaconda/py-2.7-4.3.14/envs/v13_0:
The following NEW packages will be INSTALLED:
    harfbuzz:   0.9.39-2         
    pango:      1.40.3-1         
The following packages will be DOWNGRADED:
    matplotlib: 2.0.2-np111py27_0 --> 1.5.1-np111py27_0
    pyqt:       5.6.0-py27_2      --> 4.11.4-py27_4    
    qt:         5.6.2-4           --> 4.8.7-4  

2017 Aug 5

some links

http://modules.sourceforge.net/tcl/modulefile.html

https://www.hpcwire.com/2017/04/21/system-path-exascale-shows-world-many-cores-masses-works/

https://docs.python.org/2/library/pdb.html

https://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

2017 Jul 14

On base python installation

setfacl -R -d -m group:last:rx blah

setfacl -R -m group:lsst:rx blah

edit .condarc file:

allow_softlinks: False

envs_dirs:

    - /global/common/cori/contrib/lsst/apps/anaconda/py2-envs

source /blah/blah/bin/activate

conda create --copy --clone root --name DESCQA

poked around conda issues, it seems conda was updated to improve support for multi-users
https://github.com/conda/conda/issues/1329 

2017 Jul 13

setfacl -R -d -m group:last:rx blah

conda create --copy --clone root --name DESCQA

Do the ACLs set up allow others to play with their env?

as of conda 4.3.X multi-user env should honor permissions

setfacl -m group:lsst:rx blah

Set up .condarc in root anaconda directory and disallow softlinks

Also want to set pkg directory

2017 Jun 13

Slack bots and elog 

setup private SLACK channel on lsstc:  hmktest with bot user hmkelogtest

set up simple python installation with pip, virtualenv (try conda) and pip installed "slackclient" which has a python api

Miniconda install

pip install slackclient

conda install pyyaml  (see an interesting discussion here: https://github.com/ContinuumIO/anaconda-issues/issues/1205)

create bot user and save its Slack token

Also need user id for this bot to check which messages are directed to it.

Then set up eLog connection using an xml user account on elog, and a new Slack entry form which stores an author.

References: https://www.fullstackpython.com/blog/build-first-slack-bot-python.html

2017 Jun 2

Reinstalling using the dh user on IR2

sudo -u dh bash Miniconda.blah
sudo -u dh env "PATH=$PATH" conda install --file conda2_packages_linux64-v13.txt
sudo -u dh env "PATH=$PATH" conda create --copy -c http://conda.lsst.codes/stack/0.13.0 --name v13_0 lsst-distrib nose python=2

create loadLSST.bash
export PATH=/lsst/dh/software/centos7-gcc48/anaconda/py-2.7-4.3.14/bin:${PATH}
source activate v13_0
source eups-setups.sh
setup lsst_distrib
PS1="[v13_0] "

in the test subdirectory, I installed the demo code and created some scripts to run demo as the dh user
sudo -u dh ./runtest.sh 
sudo -u dh ./runcheck.sh  

2017 May 31

Installing DMstack v13_0 vanilla on IR2 using conda

Miniconda installation 4.3.14
conda install --file conda_packages-linux-64.txt
conda config --add channels http://conda.lsst.codes/stack/0.13.0
conda create --copy --name v13_0 python=2
source activate v13_0
conda install -c http://conda.lsst.codes/stack/0.13.0 --copy lsst-distrib
conda install nose

2017 May 26

Ingested vendor data for E2V-CCD250-277 after recreating the tar.bz2 file to replace the darks files.

2017 May 25

python hack.py prod Prod getResults jsonFile.json

Building DMstack v13 plus obsSim updates on Cori KNL

source installScripts/loadmodules_13.0_knl.sh
curl -OL https://raw.githubusercontent.com/lsst/lsst/13.0/scripts/newinstall.sh
bash newinstall.sh  (let them install miniconda)
source loadLSST.bash
eups distrib install -t v13_0 lsst_distrib --nolocks

2017 May 24

Testing DMstack building on KNL

phoSim on KNL test project

http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/

module swap PrgEnv-intel PrgEnv-gnu
module swap craype-haswell craype-mic-knl

2017 May 23

Trying to test the eT API via curl or directly in a browser doesn't seem to work.  I was able to use Warren's hack.py script instead and feed it json as input.

2017 May 12

command of the day: xdg-open  to open a png file from the command line on Linux - works at NERSC.

Started Linting

2017 May 11

Learned more about adding new conda packages to a DM installation.  Remember that conda environments do not allow additional "conda installs" must set up everything at creation time or use "pip install".

Ended up dumping spec file and creating a conda environment by adding the one additional conda package we desired.  Note that I had to find a version of this package, basemap, that was compatible with numpy 1.11.* via conda info basemap

conda list --explicit > specfile.txt
conda create --name rb --file specfile.txt
source activate rb

Learned about conda version pattern matching: https://conda.io/docs/spec.html#package-match-specifications and pinning https://conda.io/docs/faq.html#pinning-packages https://www.continuum.io/blog/developer/advanced-features-conda-part-1

store pinned file in conda-meta/pinned to use it, filled by doing conda list --export > pinned and then editing via vim :%s/=/ /g

astropy 1.2.1 np111py27_0
bottleneck 1.2.0 np111py27_0
cairo 1.14.6 0
conda-env 2.6.0 0
cycler 0.10.0 py27_0
cython 0.25.2 py27_0
enum34 1.1.6 py27_0
fontconfig 2.12.1 1
freetype 2.5.5 1
future 0.16.0 py27_0
glib 2.50.2 1
harfbuzz 0.9.39 1
libffi 3.2.1 1
libgfortran 3.0.0 1
libiconv 1.14 0
libpng 1.6.27 0
libxml2 2.9.4 0
matplotlib 1.5.1 np111py27_0
nomkl 1.0 0

2017 May 10

Managing conda environments for Twinkles & Dm at NERSC  in this GitHub issue: https://github.com/LSSTDESC/Twinkles/issues/448

Creating a new env: conda create --name <blah> --clone root

which creates a new env in Linux64/miniconda2/3.19.0.lsst4/envs
source activate <blah>
source deactivate <blah>

Pinning, requires creating a "pinned" file in envs/blah/conda-meta
format for versioning here: https://conda.io/docs/spec.html#package-match-specifications

nomkl versions are preferred in DM (numpy, scipy, numexpr)

conda list --export will show me what is in the current root environment, where we should pin to precisely what we receive from DM
Also see package list: https://github.com/lsst/lsstsw

2017 May 1

merged Brian's PR on the datacat-utilties repo into a new branch.  We weren't ready for master quite yet, not until we're ready to do with datacat 0.5

https://github.com/lsst-camera-dh/datacat-https://github.com/lsst-camera-dh/datacat-utilities/tree/datacat0.5utilities/tree/datacat0.5

git checkout -b brianv0-u/bvan/dc_0.5_client_fixes master
git pull https://github.com/brianv0/datacat-utilities.git u/bvan/dc_0.5_client_fixes
Created a new branch in GitHub
git checkout <newbranch>
git merge --no-ff brianv0-u/bvan/dc_0.5_client_fixes
git push

2017 Apr 17

VirtualBox CentOs7 installation to test ST

created 32 gb virtual harddrive plus 2GB memory

sudo dhclient to kick start internet connection
sudo yum install libXtst
sudo yum install gcc
sudo yum install glibc-devel (already installed)
curl -OL http://www-glast.stanford.edu/cgi-bin/viewcvs/SCons-installer/src/installer.py?revision=1.10&view=markup
rename installer.py
python installer.py -c install --package ScienceTools --versionType RELEASE --version 11-05-02 --variant Optimized --os redhat7-x86_64-64bit-gcc48 -e $HOME/extLib -i $HOME/ST/

2017 Apr 13

Created miniconda python 2.7.12 on rh7.  Modified the files to use GLAST_EXT environment variable rather than full SLAC path.. this doesn't work, the env variable is rejected.  Instead, users would have to run this on their own download and adjust the paths for their own machines.  Did some research about relocating conda installs:  github.com/conda/conda/issues/1845

perl -p -i -e 's!/afs/slac.stanford.edu/glast/ground/GLAST_EXT/redhat7-x86_64-64bit-gcc48!\$GLAST_EXT!g' `grep -rl /afs/slac/stanford.edu/glast/ground/GLAST_EXT/redhat7-x86_64-64bit-gcc48/python/2.7.12 /afs/slac/glast/ground/GLAST_EXT/redhat7-x86_64-64bit-gcc48/python/2.7.12-mini`

SEDs

Once DC1 phosim production is complete, we should review the SED installation and perhaps use those that come with the more recent sims release 2_3_4

Trying to keep SLAC and NERSC phosim installs in synch.

SIMS_SED_LIBRARY_DIR points to the installation directory

At NERSC we seem to be using: cp -as /global/common/cori/contrib/lsst/lsstDM/w.2016.20/lsstsw/stack/Linux64/sims_sed_library/12.0/* .
and at SLAC: cp -as /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/w.2016.40-sims_2.3.1/Linux64/sims_sed_library/2016.01.26/* .

https://lsstc.slack.com/archives/D30KX84QJ/p1486681636000011

2017 Apr 11

installing lsst_apps v13_0 rather than lsst_distrib to avoid building problems with galsim when building against our own python

datacat-utilities reorganization complete on branch.  Testing by clone 2.5.0 anaconda at SLAC:  conda create --name hmk --clone root

2017 Mar 30

scl enable git19 bash
scl enable devtoolset-3 bash
export PATH=/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/anaconda/4.3.1/bin:$PATH
curl -OL https://raw.githubusercontent.com/lsst/lsst/13.0/scripts/newinstall.sh
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
bash newinstall.sh
source loadLSST.bash
eups distrib install lsst_distrib -t v13_0 --nolocks
echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py

 Same error even after updating my python for nomkl

bash-4.1$ eups distrib install lsst_distrib -t v13_0 --nolocks
   [  1/93 ]  apr 1.5.2                                                  done. 
   [  2/93 ]  cfitsio 3360.lsst4                                         done. 
   [  3/93 ]  doxygen 1.8.5.lsst1 (already installed)                    done. 
   [  4/93 ]  eigen 3.2.5.lsst2                                          done. 
   [  5/93 ]  fftw 3.3.4.lsst2                                           done. 
   [  6/93 ]  gsl 1.16.lsst3                                             done. 
   [  7/93 ]  mariadbclient 10.1.21.lsst1                                done. 
   [  8/93 ]  minuit2 5.34.14                                            done. 
   [  9/93 ]  mpich 3.2.lsst1                                            done. 
   [ 10/93 ]  python 0.0.6 (already installed)                           done. 
   [ 11/93 ]  swig 3.0.10                                                done. 
   [ 12/93 ]  xpa 2.1.15.lsst3                                           done. 
   [ 13/93 ]  activemqcpp 3.9.0.lsst2+3                                  done. 
   [ 14/93 ]  apr_util 1.5.4                                             done. 
  [ 15/93 ]  astropy 0.0.1.lsst2+2 ... 
             Using externally provided astropy v1.3.
   [ 15/93 ]  astropy 0.0.1.lsst2+2                                      done. 
   [ 16/93 ]  boost 1.60.lsst1+1                                         done. 
   [ 17/93 ]  mpi 0.0.1+2                                                done. 
  [ 18/93 ]  numpy 0.0.3+1 ... 
             Using externally provided numpy v1.11.3.
   [ 18/93 ]  numpy 0.0.3+1                                              done. 
   [ 19/93 ]  pykg_config 1.3.0+1                                        done. 
   [ 20/93 ]  python_d2to1 0.2.12.lsst2+1                                done. 
   [ 21/93 ]  python_future 0.15.2+2                                     done. 
   [ 22/93 ]  python_mysqlclient 1.3.7.lsst1+8                           done. 
   [ 23/93 ]  python_psutil 4.1.0+3                                      done. 
   [ 24/93 ]  pyyaml 3.11.lsst1+3                                        done. 
   [ 25/93 ]  scons 2.5.0.lsst2+1 (already installed)                    done. 
   [ 26/93 ]  sqlalchemy 1.0.8.lsst3+3                                   done. 
   [ 27/93 ]  wcslib 5.13.lsst1                                          done. 
   [ 28/93 ]  astrometry_net 0.67.123ff3e.lsst1+1                        done. 
   [ 29/93 ]  esutil 0.6.0+1                                             done. 
   [ 30/93 ]  log4cxx 0.10.0.lsst7                                       done. 
  [ 31/93 ]  matplotlib 0.0.4 ... 
             Using externally provided matplotlib v2.0.0.
   [ 31/93 ]  matplotlib 0.0.4                                           done. 
   [ 32/93 ]  mpi4py 2.0.0+2                                             done. 
   [ 33/93 ]  ndarray 1.3.0                                              done. 
  [ 34/93 ]  scipy 0.0.1.lsst1+3 ... 
             Using externally provided scipy v0.19.0.
   [ 34/93 ]  scipy 0.0.1.lsst1+3                                        done. 
   [ 35/93 ]  sconsUtils 13.0 (already installed)                        done. 
   [ 36/93 ]  stsci_distutils 0.3.7.lsst1+2                              done. 
   [ 37/93 ]  tmv 0.73+5                                                 done. 
   [ 38/93 ]  astrometry_net_data 10.0+89                                done. 
   [ 39/93 ]  base 13.0                                                  done. 
   [ 40/93 ]  lmfit 0.9.3+4                                              done. 
   [ 41/93 ]  lsst 13.0 (already installed)                              done. 
   [ 42/93 ]  psfex 13.0                                                 done. 
   [ 43/93 ]  pyfits 3.4.0+7                                             done. 
  [ 44/93 ]  galsim 1.4.1.lsst3 ... 

***** error: from /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/EupsBuildDir/Linux64/galsim-1.4.1.lsst3/build.log:
TMV version is 0.73
Using TMV_LINK file: /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/Linux64/tmv/0.73+5/share/tmv/tmv-link
     -L/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/Linux64/tmv/0.73+5/lib -ltmv -lpthread -fopenmp
Checking for correct TMV linkage... (this may take a little while)
Checking for correct TMV linkage... yes
Checking if we can build against Python... 
Unable to build a python loadable module using the python executable:
/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/anaconda/4.3.1/bin/python,
the library name libpython2.7.so,
and the libdir /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/anaconda/4.3.1/lib.
Nor did any of the known variations on this work.
If these are not the correct library name or location, you can tell scons
the correct names to use with the flags EXTRA_LIB_PATH and/or EXTRA_LIBS.

Please fix the above error(s) and rerun scons.
Note: you may want to look through the file INSTALL.md for advice.
Also, if you are having trouble, please check the INSTALL FAQ at 
   https://github.com/GalSim-developers/GalSim/wiki/Installation%20FAQ

+ exit -4
eups distrib: Failed to build galsim-1.4.1.lsst3.eupspkg: Command:
    source /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/eups/bin/setups.sh; export EUPS_PATH=/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0; (/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/EupsBuildDir/Linux64/galsim-1.4.1.lsst3/build.sh) >> /nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/EupsBuildDir/Linux64/galsim-1.4.1.lsst3/build.log 2>&1 4>/nfs/farm/g/lsst/u1/software/redhat6-x86_64-64bit-gcc44/DMstack/v13_0/EupsBuildDir/Linux64/galsim-1.4.1.lsst3/build.msg 
exited with code 252

2017 Mar 29

Continued to pursue Minuit2 externals for RH6 and RH7.  I had trouble because I forgot to updated containerSettings/externals.scons.  There was also a conflict due to the reuse of the libMinuit2 library which ROOT also includes - I renamed the nonROOT Minuit2 as libMinuit2nonroot.so  Also had to add link dependences on lm and lgomp

Continuing problems compiling DMstack v13_0 from source using our own anaconda python install. Needed to add pyfits, future and more importantly nomkl to pull in the right versions of numpy and scipy

https://community.lsst.org/t/failure-to-build-v13-0-from-source-with-user-installed-anaconda/1750/4

2017 Mar 28

git 1.9 now installed on SLAC Rh6-64

ksa@rhel6-64a $ scl enable git19 bash 
[ksa@rhel6-64a ~]$ git --version 
git version 1.9.4 
Or 
ksa@rhel6-64a $ bash 
[ksa@rhel6-64a ~]$ source /opt/rh/git19/enable 
[ksa@rhel6-64a ~]$ git --version 
git version 1.9.4 

2017 Mar 27

JH log files at SLAC  (dev) /nfs/farm/g/lsst/u1/jobHarness/jh_stage-test/e2v-CCD/E2V-CCD250-182/4770D/vendorIngest/v0/2758

2017 Mar 24

Exploring why conda install of DMstack v13_0 isn't running on SLAC RH6-64.  Seems to be due to glibc.
https://community.lsst.org/t/failure-to-run-demo-using-v13-0-release/1741/9

Tom asked about dmtcp 2.4.5 on Edison.  It looks like GLIBCXX 3.4.20 was removed, opened a ticket at NERSC to inquire.  Here's the error Tom reported:
   /global/common/edison/contrib/lsst/dmtcp/dmtcp-2.4.5/bin/dmtcp_launch: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /global/common/edison/contrib/lsst/dmtcp/dmtcp-2.4.5/bin/dmtcp_launch)

strings /usr/lib64/libstdc++.so.6|grep GLIBCXX
strings /global/common/cori/contrib/lsst/dmtcp/2.4.5/bin/dmtcp_launch | grep GLIBCXX

2017 Mar 23

Installed DMstack v13_0 at SLAC in a new shorter path due to ongoing problems with paths >80 chars.  

install miniconda  Miniconda 4.3.1 currently
conda install -c http://conda.lsst.codes/stack/0.13.0 lsst-distrib
source eups-setups.sh
eups path 0
edit var/opt/eups/site/startup.py to add  hooks.config.site.lockDirectoryBase = None

2017 Mar 20

lsst-dc10

trying another conda install:  sudo conda create -y -q -c http://conda.lsst.codes/stack/0.12.0 -n v12_0a python=2 lsst-distrib

sudo conda install pyqt=4.11 which resulted in downgrading a few packages (which was probably a good thing)

(v12_0a) [heather@lsst-dc10 installScript]$ sudo env "PATH=$PATH" conda install pyqt=4.11
Fetching package metadata .........
Solving package specifications: .
Package plan for installation in environment /lsst/dh/software/centos7-gcc48/anaconda/py-2.7:
The following NEW packages will be INSTALLED:
    harfbuzz:   0.9.39-2                
    pango:      1.40.3-1                
The following packages will be UPDATED:
    astropy:    1.3-np112py27_0          --> 1.3.1-np111py27_0       
    scipy:      0.19.0-np112py27_nomkl_0 [nomkl] --> 0.19.0-np111py27_nomkl_0 [nomkl]
The following packages will be DOWNGRADED due to dependency conflicts:
    matplotlib: 2.0.0-np112py27_0        --> 1.5.1-np111py27_0       
    numpy:      1.12.0-py27_nomkl_0      [nomkl] --> 1.11.3-py27_nomkl_0      [nomkl]
    pyqt:       5.6.0-py27_2             --> 4.11.4-py27_4           
    qt:         5.6.2-3                  --> 4.8.7-4                 
Proceed ([y]/n)? y
numpy-1.11.3-p 100% |################################| Time: 0:00:00  39.40 MB/s
astropy-1.3.1- 100% |################################| Time: 0:00:00  47.97 MB/s
scipy-0.19.0-n 100% |################################| Time: 0:00:00  67.29 MB/s
harfbuzz-0.9.3 100% |################################| Time: 0:00:00  33.85 MB/s
pango-1.40.3-1 100% |################################| Time: 0:00:00  33.59 MB/s
qt-4.8.7-4.tar 100% |################################| Time: 0:00:00  65.81 MB/s
pyqt-4.11.4-py 100% |################################| Time: 0:00:00  41.16 MB/s
matplotlib-1.5 100% |################################| Time: 0:00:00  55.45 MB/s

this allowed me to fix the v12_0 environment.  Now missing libXext.so.6, installed via:  sudo yum install libXext-1.3.3-3.el7.x86_64

sudo yum install tcl-devel-8.5.13-8.el7.x86_64

sudo yum install libSM-1.2.2-2.el7.x86_64

sudo yum install libXrender-0.9.8-2.1.el7.x86_64

sudo -u dh release/bin/install.py --inst_dir /lsst/dh/software/centos7-gcc48/dev/lssttd-922 --hj_folders T08 release/packageLists/IR2_JH_versions.txt

 Installed datacat by copying it (scp) from rhel6-64 as well as the config.cfg file.  Reset group ownership to "lsst-ccs"
sudo -u dh scp -r heather@rhel6-64:/u/gl/srs/datacat/dev/0.4 .
sudo -u dh scp -r heather@rhel6-64:/nfs/farm/g/lsst/u1/software/datacat/config.cfg .

2017 Mar 16

Setting up lsst-dc10

bash

export PATH=<mynewpython>:$PATH

sudo env "PATH=$PATH" conda create --name v12_0 python=2

source activate v12_0

sudo env "PATH=$PATH" conda install -c http://conda.lsst.codes/stack/0.12.0 lsst-distrib

2017 Mar 14

Going back to set up DMstack v12_0 for users on SLAC public.  Installing from scratch rather than attempting to copy the existing the environment

first had to update the conda channels in .condarc to make sure I was pointing just to the 0.12.0 area: http://conda.lsst.codes/stack/0.12.0

conda create --name lsst python=2
source activate lsst
conda install lsst-distrib 

Assuming this works - then set up any additional python modules people might need:  nose, datacat?, mysql, satrapy

failed on swig so did "conda install swig" and tried again with conda install lsst-distrib

The following packages will be DOWNGRADED due to dependency conflicts:

swig: 3.0.10-0 --> 3.0.2-0

Proceed ([y]/n)? y

Started over fresh, with a new miniconda installation and then did the lsst-distrib install.. in a shorter path.

conda install nose, ipython, jupiter, mysql-python

2017 Mar 10

Another update to twinkles_#435 branch of obs_lsstSim

setup obs_lsstSim 2017.02.08.twinkles-2-g89ee635 -t v13_0

2017 Mar 9

creating copy of DM v12_0 JH camera installation.  First creating a users version on SLAC public and the on the IR2 cluster.  Found that the channels for the stack have changed to include a version id, such as 0.12.0.  First created an explicit package list and then edited to update the channel.  Also modified my .condarc to remove the eupsforge channel which I think is outdated.

conda list --explicit > file.txt
edit file.txt to update stack channel paths
point to my version of python and set up a new environment
conda create --new v12_0 --file file.txt
this may not take care of the pip installed packages (I could not export a yml due to an old version of conda used with the original v12_0

conda create --name v12_0 --file dm12-explicit2.txt   in my case
then installed git and then had to restart the installation:
conda install --name v12_0 --file dm12-explicit.txt 

Succeeded except for one package:

PaddingError: Placeholder of length '80' too short in package git-2.6.4-0.
The package must be rebuilt with conda-build > 2.0.

 Tried conda install git=2.6.4-0, but failed in the same way
conda install git  instead which worked and installed 2.11.1-0 

In the end this failed - with a pre-link error on afw

2017 Mar 9

MySQL & Oracle notes

http://stackoverflow.com/questions/233870/how-can-i-create-a-copy-of-an-oracle-table-without-copying-the-data

2017 Mar 8

setting up lsst-dc10

Need to add gcc and anaconda python, then set up DM_v12 based on the conda list we have

[heather@lsst-dc10 dh]$ yum whatprovides gcc
yum whatprovides glibc-devel
conda list --explicit > dm12-explicit.txt
conda list > dm12.txt  // includes pip packages

2017 Mar 6

Install lsst_distrib on Cori, Edison and SLAC rhel6-64

curl -OL https://raw.githubusercontent.com/lsst/lsst/13.0/scripts/newinstall.sh
unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
bash newinstall.sh
source loadLSST.bash
eups distrib install lsst_distrib -t v13_0 --nolocks

/global/common/cori/contrib/lsst/lsstDM/patchRepos/obs_lsstSim
git clone https://github.com/lsst/obs_lsstSim
git checkout twinkles_#435
setup -r . -t v13_0
scons
scons install
scons declare
setup obs_lsstSim 2017.02.08.twinkles-1-g480d760 -t v13_0

2017 Mar 1

Updating Edison with phoSim 3.6.1 and DMstack dc1, dc1-a for Tom due to planned Cori outages.

/global/common/cori/contrib/lsst/lsstDM/patchRepos/obs_lsstSim
git clone https://github.com/lsst/obs_lsstSim
git checkout twinkles_#435
setup -r . -t dc1
scons
scons install
scons declare
setup obs_lsstSim 2017.02.08 -t dc1

2017 Feb 23

added xlrd to SLAC v12_0 python so that vendor ingest would work with the new updates from Jim.  Needed to use "pip install xlrd"

2017 Feb

bash-4.1$ conda install --override-channels -c conda-forge xlrd

Scott defined a new twinkles tag

`eups distrib install lsst_sims -t twinkles` `eups distrib install lsst_apps -t twinkles`
For future use, this version can be re-installed with `eups distrib install lsst_* -t twinkles_2017_02_08`.

2017 Jan 19

Jim requested a fresh installation of DMstack for his imSim work. We are on the bullet machines, to access this area:
/gpfs/slac/kipac/fs1/g/desc/imsim_deep/DMstack/LSST_Stack_2017-01-19

unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH
curl -OL https://raw.githubusercontent.com/lsst/lsst/12.0/scripts/newinstall.sh
scl enable devtoolset-3 bash
bash newinstall.sh
source loadLSST.bash
eups distrib install -t sims_2_3_3 lsst_apps --nolocks 
eups distrib install -t sims lsst_sims --nolocks

Allowed DM installer to install its own python.

bash-4.1$ conda install --override-channels -c conda-forge nose
bash-4.1$ conda install --override-channels -c conda-forge ipython

2017 Jan 13

Install twinkles_stack at SLAC using the descsoft account

Installed Anaconda Python via miniconda plus a conda list from our py 2.7.11 installation used for the camera on rhel6-64

unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH

curl -OL https://raw.githubusercontent.com/lsst/lsst/12.0/scripts/newinstall.sh

scl enable devtoolset-3 bash
bash newinstall.sh