Jan 19 2018
Rerunning this test using a w.2018.01 shifter image on a KNL interactive node at NERSC
Found that pipe_analysis is not available in w.2018.01 - it's not released yet. Probably need to check it out and build against the shifter image. Skipping this for now.
Just ran the step that generates the reference catalog
heatherk@cori01:/global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018> salloc -N 1 --qos=interactive --image=lsstsqre/centos:7-stack-lsst_distrib-w_2018_01 -t 04:00:00 -L SCRATCH -C haswell
salloc: Pending job allocation 9635886
salloc: job 9635886 queued and waiting for resources
salloc: job 9635886 has been allocated resources
salloc: Granted job allocation 9635886
salloc: Waiting for resource configuration
salloc: Nodes nid00036 are ready for job
heatherk@nid00036:/global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018> shifter
sh-4.2$ bash
bash-4.2$ source /opt/lsst/software/stack/loadLSST.bash
bash-4.2$ setup obs_lsstSim
bash-4.2$ ingestReferenceCatalog.py /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018 /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018/dc1_reference_catalog_8deg_radius.txt.gz -C /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018/config/IngestIndexedReferenceTask.py
root INFO: Loading config overrride file '/opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/obs_lsstSim/14.0-7-g885f9b1+2/config/IngestIndexedReferenceTask.py'
CameraMapper INFO: Loading exposure registry from /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018/registry.sqlite3
root INFO: Running: /opt/lsst/software/stack/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_tasks/14.0-33-gee53dea0+2/bin/ingestReferenceCatalog.py /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018 /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018/dc1_reference_catalog_8deg_radius.txt.gz -C /global/cscratch1/sd/heatherk/qa_dm_11452/jim/extractedJan2018/config/IngestIndexedReferenceTask.py
root WARN: Unable to retrieve exit status ('NoneType' object is not iterable); assuming success
Nov 7 2017
Mismatches between the plots in my case and Lauren's
reran ingestReferenceCatalog after copying Jim's copy of the data on NCSA back over to NERSC
Logging into lsst-dev01.ncsa
https://community.lsst.org/t/changes-to-shared-stacks-on-developer-infrastructure/2350
Conda list NCSA stack3_20171021 followed by NERSC's for w.2017.41, no meaningful differences
heatherk@cori06:/global/cscratch1/sd/heatherk/qa_dm_11452> conda list # packages in environment at /global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/python/miniconda3-4.3.21: # asn1crypto 0.22.0 py36_0 astropy 2.0.1 np113py36_0 bottleneck 1.2.1 np113py36_0 certifi 2016.2.28 py36_0 cffi 1.10.0 py36_0 conda 4.3.27 py36h2866c0b_0 conda-env 2.6.0 0 cryptography 1.8.1 py36_0 cycler 0.10.0 py36_0 cython 0.26 py36_0 dbus 1.10.20 0 expat 2.1.0 0 fontconfig 2.12.1 3 freetype 2.5.5 2 future 0.16.0 py36_1 glib 2.50.2 1 gst-plugins-base 1.8.0 0 gstreamer 1.8.0 0 icu 54.1 0 idna 2.5 py36_0 jpeg 9b 0 libffi 3.2.1 1 libgcc 5.2.0 0 libgfortran 3.0.0 1 libiconv 1.14 0 libpng 1.6.30 1 libxcb 1.12 1 libxml2 2.9.4 0 matplotlib 2.0.2 np113py36_0 nomkl 1.0 0 numexpr 2.6.2 np113py36_nomkl_0 [nomkl] numpy 1.13.1 py36_nomkl_0 [nomkl] openblas 0.2.19 0 openssl 1.0.2l 0 packaging 16.8 py36_0 pandas 0.20.3 py36_0 pcre 8.39 1 pip 9.0.1 py36_1 py 1.4.34 py36_0 pycosat 0.6.2 py36_0 pycparser 2.17 py36_0 pyopenssl 17.0.0 py36_0 pyparsing 2.1.4 py36_0 pyqt 5.6.0 py36_2 pytest 3.2.1 py36_0 python 3.6.2 0 python-dateutil 2.6.1 py36_0 pytz 2017.2 py36_0 pyyaml 3.12 py36_0 qt 5.6.2 5 readline 6.2 2 requests 2.14.2 py36_0 ruamel_yaml 0.11.14 py36_1 scipy 0.19.1 np113py36_nomkl_0 [nomkl] setuptools 36.4.0 py36_0 sip 4.18 py36_0 six 1.10.0 py36_0 sqlalchemy 1.1.13 py36_0 sqlite 3.13.0 0 tk 8.5.18 0 wheel 0.29.0 py36_0 xz 5.2.2 1 yaml 0.1.6 0 zlib 1.2.8 3 heatherk@cori06:/glob
Testing the scripts
visitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452 --id visit=1993939 --tract=0 --config doApplyUberCal=False analysis.visitClassFluxRatio=0.925 refObjLoader.ref_dataset_name="cal_ref_cat"
compareVisitAnalysis.py /global/projecta/projectdirs/lsst/production/DC1/DM/DC1-imsim-dithered --rerun /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --rerun2 /global/cscratch1/sd/heatherk/qa_dm_11452/rerun --id visit=1993939 --tract=0 --config doApplyUberCal1=False doApplyUberCal2=False --config refObjLoader.ref_dataset_name="cal_ref_cat" --no-versions
git clone https://github.com/lsst/obs_lsstSim
cd obs_lsstSim |
git checkout tickets/DM-11452 |
setup -v -r . -j |
scons opt=3 -j4 |
cd .. |
added
scons delcare
and then to use it later: setup obs_lsstSim w.2017.43-2-g701b7b4 -t w_2017_41
https://github.com/LSSTDESC/SSim_DC1/issues/46
https://jira.lsstcorp.org/browse/DM-11452
Adding pipe_analysis https://github.com/lsst-dm/pipe_analysis
desc@cori17:/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.41/qa-dm-11452> git clone https://github.com/lsst-dm/pipe_analysis.git
following directions in the DM-11452 JIRA and then:
adding scons install
scons declare
1f6d490
Setting up
w.2017.41
Use a recent weekly which includes the camera geometry updates, plus py3, plus updated python packages like matplotlib 2.x
setup NERSC env curl -LO https://raw.githubusercontent.com/lsst/lsst/master/scripts/newinstall.sh unset LSST_HOME EUPS_PATH LSST_DEVEL EUPS_PKGROOT REPOSITORY_PATH bash newinstall.sh -3 source loadLSST.bash eups distrib install -t w_2017_41 lsst_distrib --nolocks echo 'hooks.config.site.lockDirectoryBase = None' >> site/startup.py
Install branches of obs_lsstSim and pipe_analysis
References
Overview talk