5 June 2024

MIgration of Run2.2i and Run3.1i

2.2i

REPO=/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/butler.yaml

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.1 -> 1601d5973bf8

dimensions-config: daf_butler 5 -> 2a8a32e1bec3

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

butler migrate upgrade $REPO 352c30854bb0

lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Checking that this is an unmodified daf_butler universe 5 repo
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/python/lsst/daf/butler_migrate/cli/cmd/commands.py", line 142, in upgrade
script.migrate_upgrade(*args, **kwargs)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/python/lsst/daf/butler_migrate/script/migrate_upgrade.py", line 92, in migrate_upgrade
command.upgrade(cfg, revision, sql=sql)
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/command.py", line 403, in upgrade
script.run_env()
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/script/base.py", line 583, in run_env
util.load_python_file(self.dir, "env.py")
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
module = load_module_py(module_id, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
spec.loader.exec_module(module) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/_alembic/env.py", line 72, in <module>
run_migrations_online()
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/_alembic/env.py", line 66, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
self.get_context().run_migrations(**kw)
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
step.migration_fn(**kw)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/dimensions-config/1fae088c80b6.py", line 49, in upgrade
_validate_initial_dimension_universe(ctx)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/dimensions-config/1fae088c80b6.py", line 84, in _validate_initial_dimension_universe
ctx.attributes.validate_dimensions_json(5)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/python/lsst/daf/butler_migrate/butler_attributes.py", line 228, in validate_dimensions_json
raise err
ValueError: dimensions.json stored in database does not match expected daf_butler universe version 5.
Differences:

---
+++
@@ -109,12 +109,12 @@
"type": "int"
},
{
- "doc": "Earliest sequence number that might be related to this exposure.\n",
+ "doc": "The sequence number of the first exposure of the visit that contains this exposure.",
"name": "seq_start",
"type": "int"
},
{
- "doc": "Oldest sequence number that might be related to this exposure.\n",
+ "doc": "The sequence number of the final exposure of the visit that contains this exposure.",
"name": "seq_end",
"type": "int"
},
@@ -157,7 +157,7 @@
"type": "float"
},
{
- "doc": "Azimuth of the telescope at the start of the exposure in degrees. Can be NULL for observations that are not on sky, or for observations where the azimuth is not relevant.\n",
+ "doc": "Azimuth of the telescope at the start of the exposure in degrees. Can be NULL for observations that are not on sky, or for observations where the azimuth is not relevant.",
"name": "azimuth",
"type": "float"
},
@@ -167,7 +167,7 @@
"type": "float"
},
{
- "doc": "True if this exposure has some content that was simulated. This can be if the data itself were simulated or if parts of the header came from simulated systems, such as observations in the lab that are recorded as on-sky.\n",
+ "doc": "True if this exposure has some content that was simulated. This can be if the data itself were simulated or if parts of the header came from simulated systems, such as observations in the lab that are recorded as on-sky.",
"name": "has_simulated",
"type": "bool"
}
@@ -196,7 +196,7 @@
"type": "int"
},
{
- "doc": "The preferred visit system for this instrument.\n",
+ "doc": "The preferred visit system for this instrument.",
"name": "visit_system",
"type": "int"
},
@@ -366,7 +366,7 @@
"type": "int"
},
{
- "doc": "The sequence number of the first exposure that is part of this visit.\n",
+ "doc": "The sequence number of the first exposure that is part of this visit.",
"name": "seq_num",
"type": "int"
},
@@ -394,12 +394,12 @@
"type": "string"
},
{
- "doc": "Approximate azimuth of the telescope in degrees during the visit. Can only be approximate since it is continually changing during an observation and multiple exposures can be combined from a relatively long period.\n",
+ "doc": "Approximate azimuth of the telescope in degrees during the visit. Can only be approximate since it is continually changing during an observation and multiple exposures can be combined from a relatively long period.",
"name": "azimuth",
"type": "float"
},
{
- "doc": "Approximate zenith angle in degrees during the visit. Can only be approximate since it is continuously changing during an observation and multiple exposures can be combined from a relatively long period.\n",
+ "doc": "Approximate zenith angle in degrees during the visit. Can only be approximate since it is continuously changing during and observation and multiple visits can be combined from a relatively long period.\n",
"name": "zenith_angle",
"type": "float"
}
@@ -435,7 +435,7 @@
}
},
"visit_system": {
- "doc": "A system of self-consistent visit definitions, within which each exposure should appear at most once.\nA visit may belong to multiple visit systems, if the logical definitions for those systems happen to result in the same set of exposures - the main (and probably only) example is when a single-snap visit is observed, for which both the \"one-to-one\" visit system and a \"group by header metadata\" visit system will define the same single-exposure visit.\n",
+ "doc": "A system of self-consistent visit definitions, within which each exposure should appear at most once.\nA visit may belong to multiple visit systems, if the logical definitions for those systems happen to result in the same set of exposures - the main (and probably only) example is when a single-snap visit is observed, for which both the \"one-to-one\" visit system and a \"group by header metadata\" visit system will define the same single-exposure visit. ",
"keys": [
{
"name": "id",
@@ -459,7 +459,7 @@
},
"visit_system_membership": {
"always_join": true,
- "doc": "A many-to-many join table that relates visits to the visit_systems they belong to.\n",
+ "doc": "A many-to-many join table that relates visits to the visit_systems they belong to.",
"populated_by": "visit",
"requires": [
"visit",
Repositories originally created at dimension universe 1 or earlier may have incorrect documentation strings.
Re-run butler migrate with the flag '--options allow_dimension_universe_mismatch=1' to bypass this check.
This will overwrite any customizations made to the dimension universe.

This triggered the exemption noted here in the Rubin doc

Step 4b: Possible failure recovery needed for older repositories

For older repositories, the migration command in step 4a may fail with the following error:

ValueError: dimensions.json stored in database does not match expected daf_butler universe version 5.

This can happen because we recently began validating the configuration during migration, and there can be some minor discrepancies in older repositories.

If this occurs, review the output to make sure that only documentation strings were incorrect, for example lines like:

-          "doc": "Earliest sequence number that might be related to this exposure.\n",
+          "doc": "The sequence number of the first exposure of the visit that contains this exposure.",

If there are any changes that do not begin with "doc", do not attempt to finish the migration. Stop and ask for further guidance.

If everything is OK, run the following command to complete the migration:

butler migrate upgrade <repo> 352c30854bb0 --options allow_dimension_universe_mismatch=1

Rerunning with: butler migrate upgrade $REPO 352c30854bb0 --options allow_dimension_universe_mismatch=1

output

lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Generating data for day_obs table from exposure_table
lsst.1fae088c80b6_py INFO: Loading instrument definition LSSTCam-imSim from class lsst.obs.lsst.LsstCamImSim
lsst.1fae088c80b6_py INFO: Creating day_obs table
lsst.1fae088c80b6_py INFO: Populating day_obs table
lsst.1fae088c80b6_py INFO: Creating instrument index for day_obs table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference day_obs table
lsst.1fae088c80b6_py INFO: Creating group table
lsst.1fae088c80b6_py INFO: Populating group table
lsst.1fae088c80b6_py INFO: Creating instrument index for group table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference group table
lsst.1fae088c80b6_py INFO: Updating dimensions.json in ButlerAttributes
lsst.352c30854bb0_py INFO: Checking that this is an unmodified daf_butler universe 6 repo
lsst.352c30854bb0_py INFO: Adding can_see_sky column to exposure table
lsst.352c30854bb0_py INFO: Populating can_see_sky column
lsst.352c30854bb0_py INFO: ...can_see_sky values were set for all exposure records.
lsst.352c30854bb0_py INFO: Updating dimensions.json in ButlerAttributes

Checking looks good:

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.1 -> 1601d5973bf8

dimensions-config: daf_butler 7 -> 352c30854bb0 (head)

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

butler query-dimension-records --limit 10 $REPO exposure

instrument id day_obs group physical_filter obs_id exposure_time dark_time observation_type observation_reason seq_num seq_start seq_end target_name science_program tracking_ra tracking_dec sky_angle azimuth zenith_angle has_simulated can_see_sky timespan (TAI)

------------- --- -------- ----- --------------- ------ ------------- --------- ---------------- ------------------ ------- --------- ------- ----------- --------------- ----------------- ------------------ ------------------ ------- ------------------ ------------- ----------- ------------------------------------------

Run3.1i

REPO=/global/cfs/cdirs/lsst/production/gen3/DC2/Run3.1i/repo/butler.yaml

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.1 -> 1601d5973bf8

dimensions-config: daf_butler 5 -> 2a8a32e1bec3

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

Same anticipated error as Run2.2i

butler migrate upgrade $REPO 352c30854bb0
lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Checking that this is an unmodified daf_butler universe 5 repo
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/python/lsst/daf/butler_migrate/cli/cmd/commands.py", line 142, in upgrade
script.migrate_upgrade(*args, **kwargs)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/python/lsst/daf/butler_migrate/script/migrate_upgrade.py", line 92, in migrate_upgrade
command.upgrade(cfg, revision, sql=sql)
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/command.py", line 403, in upgrade
script.run_env()
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/script/base.py", line 583, in run_env
util.load_python_file(self.dir, "env.py")
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
module = load_module_py(module_id, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
spec.loader.exec_module(module) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/_alembic/env.py", line 72, in <module>
run_migrations_online()
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/_alembic/env.py", line 66, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/runtime/environment.py", line 948, in run_migrations
self.get_context().run_migrations(**kw)
File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/conda/envs/lsst-scipipe-8.0.0-exact-ext/lib/python3.11/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
step.migration_fn(**kw)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/dimensions-config/1fae088c80b6.py", line 49, in upgrade
_validate_initial_dimension_universe(ctx)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/migrations/dimensions-config/1fae088c80b6.py", line 84, in _validate_initial_dimension_universe
ctx.attributes.validate_dimensions_json(5)
File "/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate/python/lsst/daf/butler_migrate/butler_attributes.py", line 228, in validate_dimensions_json
raise err
ValueError: dimensions.json stored in database does not match expected daf_butler universe version 5.
Differences:

---
+++
@@ -109,12 +109,12 @@
"type": "int"
},
{
- "doc": "Earliest sequence number that might be related to this exposure.\n",
+ "doc": "The sequence number of the first exposure of the visit that contains this exposure.",
"name": "seq_start",
"type": "int"
},
{
- "doc": "Oldest sequence number that might be related to this exposure.\n",
+ "doc": "The sequence number of the final exposure of the visit that contains this exposure.",
"name": "seq_end",
"type": "int"
},
@@ -157,7 +157,7 @@
"type": "float"
},
{
- "doc": "Azimuth of the telescope at the start of the exposure in degrees. Can be NULL for observations that are not on sky, or for observations where the azimuth is not relevant.\n",
+ "doc": "Azimuth of the telescope at the start of the exposure in degrees. Can be NULL for observations that are not on sky, or for observations where the azimuth is not relevant.",
"name": "azimuth",
"type": "float"
},
@@ -167,7 +167,7 @@
"type": "float"
},
{
- "doc": "True if this exposure has some content that was simulated. This can be if the data itself were simulated or if parts of the header came from simulated systems, such as observations in the lab that are recorded as on-sky.\n",
+ "doc": "True if this exposure has some content that was simulated. This can be if the data itself were simulated or if parts of the header came from simulated systems, such as observations in the lab that are recorded as on-sky.",
"name": "has_simulated",
"type": "bool"
}
@@ -196,7 +196,7 @@
"type": "int"
},
{
- "doc": "The preferred visit system for this instrument.\n",
+ "doc": "The preferred visit system for this instrument.",
"name": "visit_system",
"type": "int"
},
@@ -366,7 +366,7 @@
"type": "int"
},
{
- "doc": "The sequence number of the first exposure that is part of this visit.\n",
+ "doc": "The sequence number of the first exposure that is part of this visit.",
"name": "seq_num",
"type": "int"
},
@@ -394,12 +394,12 @@
"type": "string"
},
{
- "doc": "Approximate azimuth of the telescope in degrees during the visit. Can only be approximate since it is continually changing during an observation and multiple exposures can be combined from a relatively long period.\n",
+ "doc": "Approximate azimuth of the telescope in degrees during the visit. Can only be approximate since it is continually changing during an observation and multiple exposures can be combined from a relatively long period.",
"name": "azimuth",
"type": "float"
},
{
- "doc": "Approximate zenith angle in degrees during the visit. Can only be approximate since it is continuously changing during an observation and multiple exposures can be combined from a relatively long period.\n",
+ "doc": "Approximate zenith angle in degrees during the visit. Can only be approximate since it is continuously changing during and observation and multiple visits can be combined from a relatively long period.\n",
"name": "zenith_angle",
"type": "float"
}
@@ -435,7 +435,7 @@
}
},
"visit_system": {
- "doc": "A system of self-consistent visit definitions, within which each exposure should appear at most once.\nA visit may belong to multiple visit systems, if the logical definitions for those systems happen to result in the same set of exposures - the main (and probably only) example is when a single-snap visit is observed, for which both the \"one-to-one\" visit system and a \"group by header metadata\" visit system will define the same single-exposure visit.\n",
+ "doc": "A system of self-consistent visit definitions, within which each exposure should appear at most once.\nA visit may belong to multiple visit systems, if the logical definitions for those systems happen to result in the same set of exposures - the main (and probably only) example is when a single-snap visit is observed, for which both the \"one-to-one\" visit system and a \"group by header metadata\" visit system will define the same single-exposure visit. ",
"keys": [
{
"name": "id",
@@ -459,7 +459,7 @@
},
"visit_system_membership": {
"always_join": true,
- "doc": "A many-to-many join table that relates visits to the visit_systems they belong to.\n",
+ "doc": "A many-to-many join table that relates visits to the visit_systems they belong to.",
"populated_by": "visit",
"requires": [
"visit",
Repositories originally created at dimension universe 1 or earlier may have incorrect documentation strings.
Re-run butler migrate with the flag '--options allow_dimension_universe_mismatch=1' to bypass this check.
This will overwrite any customizations made to the dimension universe.

descdm@perlmutter:login35:/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate> butler migrate upgrade $REPO 352c30854bb0 --options allow_dimension_universe_mismatch=1

lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Generating data for day_obs table from exposure_table
lsst.1fae088c80b6_py INFO: Loading instrument definition LSSTCam-imSim from class lsst.obs.lsst.LsstCamImSim
lsst.1fae088c80b6_py INFO: Creating day_obs table
lsst.1fae088c80b6_py INFO: Populating day_obs table
lsst.1fae088c80b6_py INFO: Creating instrument index for day_obs table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference day_obs table
lsst.1fae088c80b6_py INFO: Creating group table
lsst.1fae088c80b6_py INFO: Populating group table
lsst.1fae088c80b6_py INFO: Creating instrument index for group table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference group table
lsst.1fae088c80b6_py INFO: Updating dimensions.json in ButlerAttributes
lsst.352c30854bb0_py INFO: Checking that this is an unmodified daf_butler universe 6 repo
lsst.352c30854bb0_py INFO: Adding can_see_sky column to exposure table
lsst.352c30854bb0_py INFO: Populating can_see_sky column
lsst.352c30854bb0_py INFO: ...can_see_sky values were set for all exposure records.
lsst.352c30854bb0_py INFO: Updating dimensions.json in ButlerAttributes

Checking looks good

butler query-dimension-records --limit 10 $REPO exposure

instrument id day_obs group physical_filter obs_id exposure_time dark_time observation_type observation_reason seq_num seq_start seq_end target_name science_program tracking_ra tracking_dec sky_angle azimuth zenith_angle has_simulated can_see_sky timespan (TAI)

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.1 -> 1601d5973bf8

dimensions-config: daf_butler 7 -> 352c30854bb0 (head)

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

3 June 2024

MIgration of DECam

Backed up the registry and all the data to /global/cfs/cdirs/lsst/production/backups

starting from v3, need to upgrade to v4, then v5, then v7

v4 migration notes: https://confluence.lsstcorp.org/display/DM/2023-12-14+Updating+dimensions-config+to+v4+for+USDF+embargo

source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/loadLSST-ext.bash w_2024_21

setup lsst_distrib

daf_butler_migrate setup -k -r .

REPO=/global/cfs/cdirs/lsst/production/gen3/DECam/DECAT/repo/butler.yaml

(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login40:/pscratch/sd/d/descdm/schema-migration-v7> butler migrate show-current --butler $REPO
attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382
collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)
datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3
datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)
dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)
dimensions-config: daf_butler 3 -> c5ae3a2cd7c2
opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

butler migrate stamp $REPO (needed to stamp first)
butler migrate upgrade $REPO 9888256c6a18

Checking - looks good
(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login40:/pscratch/sd/d/descdm/schema-migration-v7> butler migrate show-current --butler $REPO
attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382
collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)
datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3
datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)
dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)
dimensions-config: daf_butler 4 -> 9888256c6a18
opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)


Now upgrade to v5

butler migrate upgrade $REPO 2a8a32e1bec3

Checking looks good

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)

dimensions-config: daf_butler 5 -> 2a8a32e1bec3

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

now upgrade to v7

butler migrate upgrade $REPO 352c30854bb0

(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login40:/pscratch/sd/d/descdm/schema-migration-v7> butler migrate upgrade $REPO 352c30854bb0
lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Checking that this is an unmodified daf_butler universe 5 repo
lsst.1fae088c80b6_py INFO: Generating data for day_obs table from exposure_table
lsst.1fae088c80b6_py INFO: Loading instrument definition DECam from class lsst.obs.decam.DarkEnergyCamera
lsst.1fae088c80b6_py INFO: Creating day_obs table
lsst.1fae088c80b6_py INFO: Populating day_obs table
lsst.1fae088c80b6_py INFO: Creating instrument index for day_obs table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference day_obs table
lsst.1fae088c80b6_py INFO: Creating group table
lsst.1fae088c80b6_py INFO: Populating group table
lsst.1fae088c80b6_py INFO: Creating instrument index for group table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference group table
lsst.1fae088c80b6_py INFO: Updating dimensions.json in ButlerAttributes
lsst.352c30854bb0_py INFO: Checking that this is an unmodified daf_butler universe 6 repo
lsst.352c30854bb0_py INFO: Adding can_see_sky column to exposure table
lsst.352c30854bb0_py INFO: Populating can_see_sky column
lsst.352c30854bb0_py INFO: ...can_see_sky values were set for all exposure records.
lsst.352c30854bb0_py INFO: Updating dimensions.json in ButlerAttributes

Checking looks good:

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)

dimensions-config: daf_butler 7 -> 352c30854bb0 (head)

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

Checking exposure table

butler query-dimension-records --limit 10 $REPO exposure

instrument id day_obs group physical_filter obs_id exposure_time dark_time observation_type observation_reason seq_num seq_start seq_end target_name science_program tracking_ra tracking_dec sky_angle azimuth zenith_angle has_simulated can_see_sky timespan (TAI)

---------- ------- -------- ------- ----------------------------- ------------------- ------------- ----------- ---------------- ------------------ ------- --------- ------- ----------------- --------------- ------------------ ------------------ --------- ------- ------------ ------------- ----------- ------------------------------------------

DECam 1186271 20230321 1186271 N540 DECam c0014 5403.2 210.0 ct4m20230321t200459 100.0 81.76089 dome flat unknown 1186271 1186271 1186271 DomeFlat-N540-100 2023A-716082 49.128741550558175 20.693085581480673 90.0 0.0415 50.94 False False [2023-03-21T20:05:36, 2023-03-21T20:07:16)

DECam 1186273 20230321 1186273 N540 DECam c0014 5403.2 210.0 ct4m20230321t200909 100.0 101.2095399 dome flat unknown 1186273 1186273 1186273 DomeFlat-N540-100 2023A-716082 50.17257490142825 20.694835681778226 90.0 0.0415 50.94 False False [2023-03-21T20:09:46, 2023-03-21T20:11:26)

DECam 1186275 20230321 1186275 N540 DECam c0014 5403.2 210.0 ct4m20230321t201320 100.0 81.74192 dome flat unknown 1186275 1186275 1186275 DomeFlat-N540-100 2023A-716082 51.21645825298984 20.696585781208885 90.0 0.0415 50.94 False False [2023-03-21T20:13:57, 2023-03-21T20:15:37)

DECam 1186277 20230321 1186277 N540 DECam c0014 5403.2 210.0 ct4m20230321t201730 100.0 81.7875199 dome flat unknown 1186277 1186277 1186277 DomeFlat-N540-100 2023A-716082 52.2644916053075 20.698391713457994 90.0 0.0415 50.94 False False [2023-03-21T20:18:07, 2023-03-21T20:19:47)

31 May 2024

Actual migration on roman-desc-sims

 butler migrate stamp $REPO
(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login39:/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate> butler migrate upgrade $REPO 352c30854bb0
lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Checking that this is an unmodified daf_butler universe 5 repo
lsst.1fae088c80b6_py INFO: Generating data for day_obs table from exposure_table
lsst.1fae088c80b6_py INFO: Loading instrument definition LSSTCam from class lsst.obs.lsst.LsstCam

lsst.1fae088c80b6_py INFO: Creating day_obs table
lsst.1fae088c80b6_py INFO: Populating day_obs table
lsst.1fae088c80b6_py INFO: Creating instrument index for day_obs table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference day_obs table
lsst.1fae088c80b6_py INFO: Creating group table
lsst.1fae088c80b6_py INFO: Populating group table
lsst.1fae088c80b6_py INFO: Creating instrument index for group table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference group table
lsst.1fae088c80b6_py INFO: Updating dimensions.json in ButlerAttributes
lsst.352c30854bb0_py INFO: Checking that this is an unmodified daf_butler universe 6 repo
lsst.352c30854bb0_py INFO: Adding can_see_sky column to exposure table
lsst.352c30854bb0_py INFO: Populating can_see_sky column
lsst.352c30854bb0_py INFO: WARNING: No default value for can_see_sky is known for the following observation types:
['flat']
Exposure records with these observation types will have a NULL can_see_sky.
lsst.352c30854bb0_py INFO: Updating dimensions.json in ButlerAttributes
(lsst-scipipe-8.0.0-exact-ext) descdm@perlmutter:login39:/pscratch/sd/d/descdm/schema-migration-v7/daf_butler_migrate>

30 May 2024

Schema migration v5→v7 on roman-desc-sims

https://confluence.lsstcorp.org/display/~salnikov/2024-05-15+Instructions+for+migrating+to+dimension+universe+7

Discussion on Slack: https://lsstc.slack.com/archives/C01FBUGM2CV/p1717090889474719

Will use LSST Sci Pipelines w_2024_21

Used globus to copy the full contents of /global/cfs/cdirs/lsst/production/gen3/roman-desc-sims/repo to /global/cfs/cdirs/lsst/productions/backups/roman-desc-sims-copy2/repo

roman_desc repo is in the desc_dm_gen3_spin database (while decam is in a separate database)
Looking at the list of schemas in desc_dm_gen3_spin:

desc_dm_gen3_spin=# \dn

                List of schemas

            Name            |      Owner       

----------------------------+------------------

 bfk_tests_w_2021_38        | gen3_devel

 dr6_run31i_wfd             | gen3_devel

 gen3_tiny_repo             | gen3_devel

 hk_w_2023_21               | desc_roman_devel

 hmk_lsst_test              | gen3_devel

 jchiang_4849_w_2021_16     | gen3_devel

 jchiang_bfk_gen_w_2021_15  | gen3_devel

 jchiang_bfk_test_w_2021_16 | gen3_devel

 jchiang_w_2021_10          | gen3_devel

 jchiang_w_2021_11          | gen3_devel

 jchiang_w_2021_12          | gen3_devel

 public                     | postgres

 roman_desc_sims_w_2024_07  | gen3_devel

 rubin_roman_w_2023_22      | desc_roman_devel

roman_desc_sims_w_2024_07 | gen3_devel is our target

Setting up a clone of our registry and pointing to our new data

shifter --image=postgres:13.8-alpine --module=none psql -h desc-schema-v7-test-loadbalancer.desc-co-migration-test.production.svc.spin.nersc.org -U postgres

now setting up btree extension by doing:

CREATE EXTENSION IF NOT EXISTS btree_gist;

Check that btree_gist was properly installed: To get list of extensions: /dx

then went on to create the database and a new user:

postgres=# create database desc_dm_gen3_spin;

CREATE DATABASE

postgres=# create user gen3_devel with password 'mypassword';

CREATE ROLE

postgres=# grant all on database desc_dm_gen3_spin to gen3_devel;

GRANT

postgres=# create user desc_roman_devel with password 'mypassword';

CREATE ROLE

postgres=# grant all on database desc_dm_gen3_spin to desc_roman_devel;

GRANT

ERROR: must be member of role "desc_roman_devel"

Fixed by doing

GRANT desc_roman_devel TO gen3_devel;

postgres=# create user gen3_user with password 'mypassword';

grant connect,temp on database desc_dm_gen3_spin to gen3_user;

Now dump the contents from the original db into the new one

shifter --image=postgres:13.8-alpine --module=none psql -h desc-schema-v7-test-loadbalancer.desc-co-migration-test.production.svc.spin.nersc.org -U gen3_devel -p 5432 -d desc_dm_gen3_spin -f ./postgres-desc_dm_gen3_spin-backup-30-05-2024.sql |& tee clone-for-schema-migration-v7.log

editing butler.yaml to point to my cloned postgres registry

Set up w_2024_21: source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_21/loadLSST-ext.bash w_2024_21

check out daf_migration

git clone https://github.com/lsst-dm/daf_butler_migrate.git cd daf_butler_migrate

setup -k -r .

scons

Checking status of existing REPO

REPO=/global/cfs/cdirs/lsst/production/backups/roman-desc-sims/repo/butler.yaml

butler migrate show-current --butler $REPO

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.1 -> c0e4f9ab4a5d (head)

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)

dimensions-config: daf_butler 5 -> 2a8a32e1bec3

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

Looks good we're at version 5 as expected

butler migrate upgrade $REPO 352c30854bb0

fails so we have to stamp first

butler migrate stamp $REPO

(lsst-scipipe-8.0.0-exact-ext) heatherk@perlmutter:login19:/pscratch/sd/h/heatherk/schema-migration-v7> butler migrate upgrade $REPO 352c30854bb0
lsst.1fae088c80b6_py INFO: Locking exposure table
lsst.1fae088c80b6_py INFO: Checking that this is an unmodified daf_butler universe 5 repo
lsst.1fae088c80b6_py INFO: Generating data for day_obs table from exposure_table
lsst.1fae088c80b6_py INFO: Loading instrument definition LSSTCam from class lsst.obs.lsst.LsstCam
lsst.1fae088c80b6_py INFO: Creating day_obs table
lsst.1fae088c80b6_py INFO: Populating day_obs table
lsst.1fae088c80b6_py INFO: Creating instrument index for day_obs table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference day_obs table
lsst.1fae088c80b6_py INFO: Creating group table
lsst.1fae088c80b6_py INFO: Populating group table
lsst.1fae088c80b6_py INFO: Creating instrument index for group table
lsst.1fae088c80b6_py INFO: Updating exposure table to reference group table
lsst.1fae088c80b6_py INFO: Updating dimensions.json in ButlerAttributes
lsst.352c30854bb0_py INFO: Checking that this is an unmodified daf_butler universe 6 repo
lsst.352c30854bb0_py INFO: Adding can_see_sky column to exposure table
lsst.352c30854bb0_py INFO: Populating can_see_sky column
lsst.352c30854bb0_py INFO: WARNING: No default value for can_see_sky is known for the following observation types:
['flat']
Exposure records with these observatio

butler query-dimension-records --limit 10 $REPO exposure

instrument id day_obs group physical_filter obs_id exposure_time dark_time observation_type observation_reason seq_num seq_start seq_end target_name science_program tracking_ra tracking_dec sky_angle azimuth zenith_angle has_simulated can_see_sky timespan (TAI)

---------- ------------- -------- ------------- --------------- -------------------- ------------- --------- ---------------- ------------------ ------- --------- ------- ----------- --------------- ------------------ ------------------- --------- ------- ------------ ------------- ----------- ------------------------------------------

LSSTCam 5024011700060 20240117 5024011700060 g_6 IM_P_20240117_000060 500.0 500.0 flat flat 60 60 60 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T03:29:59, 2024-01-17T03:38:20)

LSSTCam 5024011700061 20240117 5024011700061 g_6 IM_P_20240117_000061 500.0 500.0 flat flat 61 61 61 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T03:38:29, 2024-01-17T03:46:50)

LSSTCam 5024011700062 20240117 5024011700062 g_6 IM_P_20240117_000062 500.0 500.0 flat flat 62 62 62 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T03:46:59, 2024-01-17T03:55:20)

LSSTCam 5024011700063 20240117 5024011700063 g_6 IM_P_20240117_000063 500.0 500.0 flat flat 63 63 63 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T03:55:29, 2024-01-17T04:03:50)

LSSTCam 5024011700064 20240117 5024011700064 g_6 IM_P_20240117_000064 500.0 500.0 flat flat 64 64 64 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T04:03:59, 2024-01-17T04:12:20)

LSSTCam 5024011700065 20240117 5024011700065 g_6 IM_P_20240117_000065 500.0 500.0 flat flat 65 65 65 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T04:12:29, 2024-01-17T04:20:50)

LSSTCam 5024011700066 20240117 5024011700066 g_6 IM_P_20240117_000066 500.0 500.0 flat flat 66 66 66 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T04:20:59, 2024-01-17T04:29:20)

LSSTCam 5024011700067 20240117 5024011700067 g_6 IM_P_20240117_000067 500.0 500.0 flat flat 67 67 67 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T04:29:29, 2024-01-17T04:37:50)

LSSTCam 5024011700068 20240117 5024011700068 g_6 IM_P_20240117_000068 500.0 500.0 flat flat 68 68 68 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T04:37:59, 2024-01-17T04:46:20)

LSSTCam 5024011700069 20240117 5024011700069 g_6 IM_P_20240117_000069 500.0 500.0 flat flat 69 69 69 UNKNOWN calibration 199.68134610137426 -34.544668655603914 None None None True None [2024-01-17T04:46:29, 2024-01-17T04:54:50)

n types will have a NULL can_see_sky.
lsst.352c30854bb0_py INFO: Updating dimensions.json in ButlerAttributes

LOOKS Good,

checking

REPO=/global/cfs/cdirs/lsst/production/backups/roman-desc-sims/repo/butler.yaml

12 April 2024

butler transfer-datasets

Patricia ran some SSI processing in the test postgres, and it would be nice to migrate that over to the production version

source setup.sh w_2024_13

using query-collections to see Patricia's data

REPO=/global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/butler-schema-migration.yaml

butler query-collections $REPO

butler query-datasets --collections=u/plarsen/*_schemamigration* $REPO

butler transfer-datasets --collections=u/plarsen/*_schemamigration* --register-dataset-types $REPOSRC $REPODEST |& tee testing-transfer-2.log

lsst.daf.butler.direct_butler INFO: Transferring 4967824 datasets into Butler(collections=(), run=None, datastore='file:///global/cfs/cdirs/lsst/production/gen3/DC2/Run2.2i/repo/', registry='PostgreSQL@desc_dm_gen3_spin:run22i_w_2021_48')
lsst.daf.butler.datastores.fileDatastore INFO: Skipped transfer of 4862860 datasets already present in datastore
Number of datasets transferred: 4902592

15 March 2024

Looking at all repos in our Postgres DB

DC2

Run2.2i

repo and run22i_repo

Need to update from schema version 0 => 5

Testing out Run2.2i repo migration

butler migrate show-current --butler $REPO --namespace daf_butler

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.1 -> 1601d5973bf8

dimensions-config: daf_butler 0 -> f3bcee34f344

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

butler migrate show-history dimensions-config

9888256c6a18 -> 2a8a32e1bec3 (dimensions-config-daf_butler, dimensions-config) (head), Migration script for dimensions.yaml namespace=daf_butler version=5.

c5ae3a2cd7c2 -> 9888256c6a18 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=4.

bf6308af80aa -> c5ae3a2cd7c2 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=3.

380002bcbb26 -> bf6308af80aa (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=2.

f3bcee34f344 -> 380002bcbb26 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=1.

3e2891b82110 -> f3bcee34f344 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=0.

<base> -> 3e2891b82110 (dimensions-config-daf_butler, dimensions-config), This is an initial pseudo-revision of the 'dimensions-config' tree.

butler migrate stamp --namespace daf_butler $REPO

butler migrate show-current --namespace daf_butler $REPO

1601d5973bf8

2101fbf51ad3

f3bcee34f344

77e5b803ad3f (head)

8c57494cabcc (head)

f22a777cf382

a07b3b60e369 (head)

butler migrate upgrade --namespace daf_butler $REPO 380002bcbb26

butler migrate upgrade --options has_simulated=1 $REPO bf6308af80aa

butler migrate upgrade $REPO c5ae3a2cd7c2

butler migrate upgrade $REPO 9888256c6a18

butler migrate upgrade $REPO 2a8a32e1bec3

butler migrate show-current --namespace daf_butler $REPO

2101fbf51ad3

77e5b803ad3f (head)

8c57494cabcc (head)

f22a777cf382

a07b3b60e369 (head)

1601d5973bf8

f3bcee34f344

Run3.1i

repo and run31i_repo

Need to update from schema version 0 => 5

butler migrate show-current --butler $REPO --namespace daf_butler

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.1 -> 1601d5973bf8

dimensions-config: daf_butler 0 -> f3bcee34f344

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

butler migrate show-history dimensions-config

9888256c6a18 -> 2a8a32e1bec3 (dimensions-config-daf_butler, dimensions-config) (head), Migration script for dimensions.yaml namespace=daf_butler version=5.

c5ae3a2cd7c2 -> 9888256c6a18 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=4.

bf6308af80aa -> c5ae3a2cd7c2 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=3.

380002bcbb26 -> bf6308af80aa (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=2.

f3bcee34f344 -> 380002bcbb26 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=1.

3e2891b82110 -> f3bcee34f344 (dimensions-config-daf_butler, dimensions-config), Migration script for dimensions.yaml namespace=daf_butler version=0.

<base> -> 3e2891b82110 (dimensions-config-daf_butler, dimensions-config), This is an initial pseudo-revision of the 'dimensions-config' tree.

butler migrate stamp --namespace daf_butler $REPO

butler migrate upgrade --namespace daf_butler $REPO 380002bcbb26

DECam

postgresql://desc-postgres-loadbalancer.desc-co.production.svc.spin.nersc.org:5432/desc_decam_spin

user decam_devel

REPO=/global/cfs/cdirs/lsst/production/gen3/DECam/DECAT/repo

butler migrate show-current --butler $REPO

Currently on schema v3

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)

dimensions-config: daf_butler 3 -> c5ae3a2cd7c2

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

roman_desc_sims

REPO=/global/cfs/cdirs/lsst/production/gen3/roman-desc-sims/repo

Up to date with Schema version 5

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.1 -> c0e4f9ab4a5d (head)

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.2 -> 035dcf13ef18 (head)

dimensions-config: daf_butler 5 -> 2a8a32e1bec3 (head)

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

13 March 2024

output from \d+ run22i_w_2021_36.visit_definition;

+--------------+-----------------------+-----------+----------+--------------+-------------+

|--------------+-----------------------+-----------+----------+--------------+-------------|

+--------------+-----------------------+-----------+----------+--------------+-------------+

Indexes:

"visit_definition_pkey" PRIMARY KEY, btree (instrument, visit_system, exposure)

"visit_definition_fkidx_instrument" btree (instrument)

"visit_definition_fkidx_instrument_exposure" btree (instrument, exposure)

"visit_definition_fkidx_instrument_visit" btree (instrument, visit)

"visit_definition_fkidx_instrument_visit_system" btree (instrument, visit_system)

Foreign-key constraints:

"fkey_visit_definition_exposure_instrument_id_instrumen_21a4dc77" FOREIGN KEY (instrument, exposure) REFERENCES run22i_w_2021_36.exposure(instrument, id)

"fkey_visit_definition_instrument_name_instrument" FOREIGN KEY (instrument) REFERENCES run22i_w_2021_36.instrument(name)

"fkey_visit_definition_visit_instrument_id_instrument_visit" FOREIGN KEY (instrument, visit) REFERENCES run22i_w_2021_36.visit(instrument, id)

"fkey_visit_definition_visit_system_instrument_id_instr_26df1493" FOREIGN KEY (instrument, visit_system) REFERENCES run22i_w_2021_36.visit_system(instrument, id)

Has OIDs: no

Trying the schema migration again

source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_10/loadLSST.bash

setup lsst_distrib

cd daf_butler_migrate

setup -k -r .

REPO=/global/cfs/cdirs/lsst/production/temp/run22i_repo/

butler migrate show-current --butler $REPO --namespace daf_butler

attributes: lsst.daf.butler.registry.attributes.DefaultButlerAttributeManager 1.0.0 -> f22a777cf382

collections: lsst.daf.butler.registry.collections.synthIntKey.SynthIntKeyCollectionManager 2.0.0 -> 8c57494cabcc (head)

datasets: lsst.daf.butler.registry.datasets.byDimensions._manager.ByDimensionsDatasetRecordStorageManagerUUID 1.0.0 -> 2101fbf51ad3

datastores: lsst.daf.butler.registry.bridge.monolithic.MonolithicDatastoreRegistryBridgeManager 0.2.0 -> a07b3b60e369 (head)

dimensions: lsst.daf.butler.registry.dimensions.static.StaticDimensionRecordStorageManager 6.0.0 -> 87a30df8c8c5

dimensions-config: daf_butler 0 -> f3bcee34f344

opaque: lsst.daf.butler.registry.opaque.ByNameOpaqueTableStorageManager 0.2.0 -> 77e5b803ad3f (head)

butler migrate show-history dimensions-config

9888256c6a18 -> 2a8a32e1bec3 (dimensions-config, dimensions-config-daf_butler) (head), Migration script for dimensions.yaml namespace=daf_butler version=5.

c5ae3a2cd7c2 -> 9888256c6a18 (dimensions-config, dimensions-config-daf_butler), Migration script for dimensions.yaml namespace=daf_butler version=4.

bf6308af80aa -> c5ae3a2cd7c2 (dimensions-config, dimensions-config-daf_butler), Migration script for dimensions.yaml namespace=daf_butler version=3.

380002bcbb26 -> bf6308af80aa (dimensions-config, dimensions-config-daf_butler), Migration script for dimensions.yaml namespace=daf_butler version=2.

f3bcee34f344 -> 380002bcbb26 (dimensions-config, dimensions-config-daf_butler), Migration script for dimensions.yaml namespace=daf_butler version=1.

3e2891b82110 -> f3bcee34f344 (dimensions-config, dimensions-config-daf_butler), Migration script for dimensions.yaml namespace=daf_butler version=0.

<base> -> 3e2891b82110 (dimensions-config, dimensions-config-daf_butler), This is an initial pseudo-revision of the 'dimensions-config' tree.

butler migrate stamp --namespace daf_butler $REPO

checked visit_definitions in pgcli, and there are still foreign keys defined.

pgcli postgresql://gen3_devel@hmk-postgres-loadbalancer.desc-co-migration-test.production.svc.spin.nersc.org:5432/desc_dm_gen3_spin

butler migrate show-current --namespace daf_butler $REPO

87a30df8c8c5

2101fbf51ad3

f3bcee34f344

77e5b803ad3f (head)

f22a777cf382

8c57494cabcc (head)

a07b3b60e369 (head)

butler migrate upgrade --namespace daf_butler $REPO 380002bcbb26

no output, and the visit_definition still looks good in pgcli

butler migrate show-current $REPO

8c57494cabcc (head)

f22a777cf382

a07b3b60e369 (head)

87a30df8c8c5

77e5b803ad3f (head)

2101fbf51ad3

380002bcbb26