DC2 Run2.2i DR2 warps (Y1)

sqlite3 -noheader /sps/lssttest/dataproducts/desc/DC2/Run2.2i/v19.0.0-v1/rerun/run2.2i-coadd-wfd-dr6-v1/tracts_mapping.sqlite3 "select filter, tract, patch, visit from overlaps where filter='g' AND visit<=262622"

Created a python script to parse the resulting output files to produce a sensible list of files that needed to be copied to NERSC.

 

python script 
import os, sys
import argparse
import glob 
parser = argparse.ArgumentParser(description='A tutorial of argparse!')
parser.add_argument("--infile", required=True, type=str, help="input file")
parser.add_argument("--outfile", required=True, type=str, help="output file")
args = parser.parse_args()
with open(args.infile) as inputf:
    dirlist = inputf.readlines()
dirlist = [x.strip() for x in dirlist]    
with open(args.outfile, 'w') as f:
    for x in dirlist :
#        print (x)
        band, tract, patch, visit = x.split('/')
#        print (band, tract, patch, visit)
        warpFile="/sps/lssttest/dataproducts/desc/DC2/Run2.2i/v19.0.0-v1/rerun/run2.2i-coadd-wfd-dr6-v1-u/deepCoadd/warp-"+band+"-"+tract+"-"+patch+"-"+visit+".fits\n"
        matchFile="/sps/lssttest/dataproducts/desc/DC2/Run2.2i/v19.0.0-v1/rerun
/run2.2i-coadd-wfd-dr6-v1-u/deepCoadd/psfMatchedWarp-"+band+"-"+tract+"-"+patch
+"-"+visit+".fits\n"
        f.write(warpFile)
        f.write(matchFile)

 

SN python install

Spoke to Rick K and Bruno

Agreed we would set up /global/common/software/lsst/common/miniconda/sn-py as a root environment, Rick doesn't trust conda environments yet, nor do they want to use containers.

I will admin the area and grant Rick and Kara P full admin privileges via ACLs

Interested in setting up SN testing - should look at TxPipe

Ask that we add a SN-README to the sn-py area to remind people how to request new or updated packages via desc-help

Send notifications to the sn-pipelines channel

Should set up a jupyter kernel that SN can use, but not meant for general DESC use yet.

 

 

 

stoppoing point July 17

desc-dev env:

Executing transaction: \ b' \nFor Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.\nTo enable it, please set the environmental variable OMPI_MCA_opal_cuda_support=true before\nlaunching your MPI processes. Equivalently, you can set the MCA parameter in the command line:\nmpiexec --mca opal_cuda_support 1 ...\n \n

 

 

Stopping point July 10th

announce CO meeting

Gen3 - need to redo with latest bootstrap script

nEXO look at latest cpuonly image

quarterly

builder doc

CC - meeting summary and OMP

collab meeting - CO list

 

 

Copying Run 2.2i DR2 to CSCRATCH - using Globus CMD line tools

calexp:Transfer ID is 02fa31e4-bba4-11ea-8f22-0a21f750d19b label is descdm_2020-07-01_07_06_08

icSrc: Transfer ID is 63766d02-bba5-11ea-9a42-0255d23c44ef label is descdm_2020-07-01_07_16_00
singleFrameDriver Transfer ID is 96794832-bba5-11ea-8f22-0a21f750d19b label is descdm_2020-07-01_07_17_25

skyCorr Transfer ID is c5cc64b6-bba5-11ea-9a42-0255d23c44ef label is descdm_2020-07-01_07_18_45

src Transfer ID is f8cf204c-bba5-11ea-bef5-0e716405a293 label is descdm_2020-07-01_07_20_10

srcMatch: Transfer ID is 6fe6b302-bba6-11ea-8f22-0a21f750d19b label is descdm_2020-07-01_07_23_30

 

 

 

dotfile python package: https://pypi.org/project/python-dotenv/

stopping point

missing sims_w_2020_22 tag?  and setting up desc-python-bleed to point to right yaml file

y1-wfd 12 TB

y2-WFD 12 TB

y3-wfd  14 TB include centroids and all fits files

y4-wfd 15 TB (no centroids)

y5-wfd 24 TB

Run 2.2i:
Y01: 12 TB, Y02: 12 TB, Y03: 14 TB, Y04: 15 TB, Y05 24 TB = 77 TB
Run 3.0i:
Y01 1.7 TB, Y02: 1.3 TB, Y03: 1.6 TB, Y04: 1.4 TB = 6 TB

y5  1.8 TB


Setting up rsync object catalogs

rsync -avzhe ssh --progress --include 'object_tract_*.parquet' --include '*/' --exclude '*' hkelly@cca.in2p3.fr:/sps/lssttest/dataproducts/desc/DC2/Run2.2i/v19.0.0-v1/dpdd/run2.2i-wfd-dr6/object_table_summary . | & tee rsync-hmk.log

Look at TXpipe later

https://github.com/joezuntz/txpipe-docker/blob/master/txpipe-base/Dockerfile

How to get a list of available DM docker images;

 wget https://registry.hub.docker.com/v1/repositories/lsstsqre/centos/tags -O - | tr -d '[]" ' | tr '}' '\n' | awk -F: '{print $3}' | grep 7-stack-lsst_distrib-w_2020_


3 June 2020

new conda-forge weeklies have trouble allowing updates - apparently due to pinning some packages like astropy which may have incompatible version strings?

 

use conda list --export to create the pinned file

 

29 May 2020

conda env export --no-builds

 

For Monday - new Desc-stack

updated sims docker image

22 May 2020

Issues with pyccl in desc-stack docker image

Illegal instruction

https://github.com/LSSTDESC/CCL/issues/694

https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners

Error seen when running conda list pyccl:

https://github.com/conda/conda/issues/8616

https://github.com/tensorflow/tensorflow/issues/17411

will pin to camb 1.1.2 in desc-stack until we move to conda-forge support DMstack

 

 

SNiPER incident

check ROOT files

 

respond to Ben's and James - empty repo tarball

  • No labels