3 June 2024
MIgration of DECam
Backed up the registry and all the data to /global/cfs/cdirs/lsst/production/backups
starting from v3, need to upgrade to v4, then v5, then v7
v4 migration notes: https://confluence.lsstcorp.org/display/DM/2023-12-14+Updating+dimensions-config+to+v4+for+USDF+embargo
source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/loadLSST-ext.bash w_2024_21
setup lsst_distrib
daf_butler_migrate setup -k -r .
REPO=/global/cfs/cdirs/lsst/production/gen3/DECam/DECAT/repo/butler.yaml
(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login40:/pscratch/sd/d/descdm/schema-migration-v7> butler migrate show-current --butler $REPO
attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382
collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)
datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3
datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)
dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)
dimensions-config: daf_butler 3 -> c5ae3a2cd7c2
opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)
butler migrate stamp$REPO (needed to stamp first)
butler migrate upgrade $REPO 9888256c6a18
Checking - looks good
(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login40:/pscratch/sd/d/descdm/schema-migration-v7> butler migrate show-current --butler $REPO
attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382
collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)
datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3
datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)
dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)
dimensions-config: daf_butler 4 -> 9888256c6a18
opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)
Now upgrade to v5
butler migrate upgrade $REPO 2a8a32e1bec3
Checking looks good
butler migrate show-current --butler $REPO
attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382
collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)
datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3
datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)
dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)
dimensions-config: daf_butler 5 -> 2a8a32e1bec3
opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)
now upgrade to v7
butler migrate upgrade $REPO 352c30854bb0
(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login40:/pscratch/sd/d/descdm/schema-migration-v7> butler migrate upgrade $REPO 352c30854bb0
lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Checking that this is an unmodified daf_butler universe 5 repo
lsst.1fae088c80b6_py INFO: Generating data for day_obs table from exposure_table
lsst.1fae088c80b6_py INFO: Loading instrument definition DECam from class lsst.obs.decam.DarkEnergyCamera
lsst.1fae088c80b6_py INFO: Creating day_obs table
lsst.1fae088c80b6_py INFO: Populating day_obs table
lsst.1fae088c80b6_py INFO: Creating instrument index for day_obs table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference day_obs table
lsst.1fae088c80b6_py INFO: Creating group table
lsst.1fae088c80b6_py INFO: Populating group table
lsst.1fae088c80b6_py INFO: Creating instrument index for group table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference group table
lsst.1fae088c80b6_py INFO: Updating dimensions.json in ButlerAttributes
lsst.352c30854bb0_py INFO: Checking that this is an unmodified daf_butler universe 6 repo
lsst.352c30854bb0_py INFO: Adding can_see_sky column to exposure table
lsst.352c30854bb0_py INFO: Populating can_see_sky column
lsst.352c30854bb0_py INFO: ...can_see_sky values were set for all exposure records.
lsst.352c30854bb0_py INFO: Updating dimensions.json in ButlerAttributes
Checking looks good:
butler migrate show-current --butler $REPO
attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382
collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)
datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3
datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)
dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)
dimensions-config: daf_butler 7 -> 352c30854bb0 (head)
opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)
Checking exposure table
butler query-dimension-records --limit 10 $REPO exposure
instrument id day_obs group physical_filter obs_id exposure_time dark_time observation_type observation_reason seq_num seq_start seq_end target_name science_program tracking_ra tracking_dec sky_angle azimuth zenith_angle has_simulated can_see_sky timespan (TAI)
---------- ------- -------- ------- ----------------------------- ------------------- ------------- ----------- ---------------- ------------------ ------- --------- ------- ----------------- --------------- ------------------ ------------------ --------- ------- ------------ ------------- ----------- ------------------------------------------
DECam 1186271 20230321 1186271 N540 DECam c0014 5403.2 210.0 ct4m20230321t200459 100.0 81.76089 dome flat unknown 1186271 1186271 1186271 DomeFlat-N540-100 2023A-716082 49.128741550558175 20.693085581480673 90.0 0.0415 50.94 False False [2023-03-21T20:05:36, 2023-03-21T20:07:16)
DECam 1186273 20230321 1186273 N540 DECam c0014 5403.2 210.0 ct4m20230321t200909 100.0 101.2095399 dome flat unknown 1186273 1186273 1186273 DomeFlat-N540-100 2023A-716082 50.17257490142825 20.694835681778226 90.0 0.0415 50.94 False False [2023-03-21T20:09:46, 2023-03-21T20:11:26)
DECam 1186275 20230321 1186275 N540 DECam c0014 5403.2 210.0 ct4m20230321t201320 100.0 81.74192 dome flat unknown 1186275 1186275 1186275 DomeFlat-N540-100 2023A-716082 51.21645825298984 20.696585781208885 90.0 0.0415 50.94 False False [2023-03-21T20:13:57, 2023-03-21T20:15:37)
DECam 1186277 20230321 1186277 N540 DECam c0014 5403.2 210.0 ct4m20230321t201730 100.0 81.7875199 dome flat unknown 1186277 1186277 1186277 DomeFlat-N540-100 2023A-716082 52.2644916053075 20.698391713457994 90.0 0.0415 50.94 False False [2023-03-21T20:18:07, 2023-03-21T20:19:47)
31 May 2024
Actual migration on roman-desc-sims
...