Page History
...
Mike Browne writes: I think Valerio's comments above are pretty spot on. The integer is essentially a version/commit number. It uniquely identifies everything at a particular moment in time... you change stuff, you get a new key/number. We're saving a complete history, with the idea if it gets too much at some point, someone in the future will write a history-pruning tool.
Two Suggested New Tools
Merging Schema Changes
There is a working example of schema update in psdaq/psdaq/configdb/hsd_config_update.py
issue: we don't handle merging of schema changes to the config objects
ric's figurative proposal:
- script to read db and write json file
- modify ts_config_store.py to put the new schema in dbase
- third script applies settings from json file to the dbase ignoring the ones
that are dissimilar
(get_config.py has some of the tools to do this)
cpo proposal:
- one script (configdb_modify.py) that reads the configdb json, modifies it (with python code)
and writes it back (a plug or a minus: we no longer have a script that knows the official
state of the schema: the database itself is the source of truth).
- if we ever lost the database we no longer know the configdb schemas
Another proposal (from Ric):
- Change the *_config_store.py scripts to take another command line argument that selects between creating a new configdb entry or modifying an existing one
- If creating, behave as now
- if modifying:
- read the configdb entry into a json string (get_config.py: get_config())
- apply the modified schema coded elsewhere in *config_store.py to configdb
- use the json string from above to update the values of the new configdb entry (get_config.py: update_config() ?? )
- elements that are in the configdb entry but not in the json string are ignored
- elements that are not in the configdb entry but are in the json string are ignored
- The RO elements like
help
and version numbers should be governed by the new schema code and not by the previous DB contents - Delta configs (e.g. CALIB configdb entries) may need special handling to avoid turning a delta config into a base config and vice versa
- Delta configs are recognized through the presence of a
_cfgTypeRef
element
- Delta configs are recognized through the presence of a
item 1:
ts_config_V1:
config[group][0][rate]=10Hz
[1][rate]=100Hz
[2][rate]=100Hz
need a procedure to produce:
ts_config_V2:
config[group][0][rate]=10Hz
[2][rate]=100Hz
or
ts_config_V2:
config[group][0][rate]=10Hz
[1][rate]=100Hz
[2][rate]=100Hz
[4][rate]=100Hz
Rename/Remove Old Configdb Objects
ability remove and hide (in the control gui) detector configurations that aren't needed anymore
(hide: prepend an "_"?, rename timing_1 to _timing_1?)
rename = delete+create
Ric added: Maybe this should be thought of as archiving unneeded ConfigDb objects and we should therefore provide a way to unarchive them for when archiving is done erroneously or we change our mind. Need to consider how to handle the case when, for example, timing_1 is archived to _timing_1 and then a new timing_1 is created.
Mike Browne implemented the following rename/delete functionality. Murali is implementing a web interface on top of this:
Code Block | ||
---|---|---|
| ||
c2 = cdb.configdb(mdb.server, "SXR", create=True, root=dbname)
c2.add_alias("FOO") # 0
c2.transfer_config("AMO", "BEAM", "evr0", "FOO", "evr3") # 1
with pytest.raises(Exception):
c2.transfer_config("AMO", "BEAM", "evr0", "FOO", "evr3")
print("Configs:")
c2.print_configs()
cfg1 = c.get_configuration("BEAM", "evr0")
cfg2 = c2.get_configuration("FOO", "evr3")
# These should be the same except for the detector names!
del cfg1['detName:RO']
del cfg2['detName:RO']
assert cfg1 == cfg2
c2.rename_device("evr3", "evr7", "FOO")
cfg3 = c2.get_configuration("FOO", "evr7")
# These should be the same except for the detector names!
del cfg3['detName:RO']
assert cfg2 == cfg3
with pytest.raises(ValueError):
c2.remove_device("evr6", "FOO")
c2.remove_device("evr7", "FOO")
with pytest.raises(ValueError):
cfg4 = c2.get_configuration("FOO", "evr7") |
Rename/Remove Discussion
with Chris Ford, Murali, Matt, cpo on April 17, 2024
Murali highlighted some important questions with the new rename/remove features. Here they are, with answers from the group. I believe the conclusions were unanimous:
- do we want to update all old keys or only the latest key?
- the consensus was that updating only the latest key did a better job of preserving the history (e.g. for comparison with xtc values). there was a weak counter-argument that the "configdb rollback" command would not work as naturally (would need to know both the old name and the new name).
- the detector name (e.g. trigger_0) shows up twice: once as a database-lookup key ("device") and once in the document with typed-json ("detName:RO", which is useful for converting to xtc). Do we want to edit both?
- Yes, it is more natural to edit both. Matt pointed out that the daq code may overwrite detName:RO, but we weren't certain, and in any event it feels more elegant to have it be correct in the database
- do we want to create a new key with the changes?
- Yes, because this does a better job of preserving the history
- do we want to create a new document with the new detName:RO?
- Yes, because this does a better job of preserving the history
- what do we do if the user tries to rename to name that already exists in the current key?
- We should raise an error, which would force the user to explicitly remove the existing conflicting name before doing the rename.
Removal is more straightforward: a new key will be created with the appropriate device removed, but history will be preserved in the old keys.
These answers will require changes to Mike Browne's code which Murali has kindly agreed to do.
CLI Overview
Code Block | ||
---|---|---|
| ||
$ configdb -h
usage: configdb [-h] [--url URL] [--root ROOT] {cat,rm,cp,mv,history,rollback,ls} ...
configuration database CLI
positional arguments:
{cat,rm,cp,mv,history,rollback,ls}
cat print a configuration
rm remove a configuration
cp copy a configuration (EXAMPLE: configdb cp --create --write tmo/BEAM/timing_0 newhutch/BEAM/timing_0)
mv rename a configuration (EXAMPLE: configdb mv --write tst/BEAM/timing_45 timing_46)
history get history of a configuration
rollback rollback configuration to a specific key
ls list directory contents
optional arguments:
-h, --help show this help message and exit
--url URL configuration database connection
--root ROOT configuration database root (default: configDB) |
Configdb Server Logs
The configdb server runs on a subset of psdm[01-04], where multiple nodes are used in a "load balancer" manner. You have login to each of these nodes and look for logfiles in /u1/psdm/logs/configdb* to find the one that contains the transaction you are interested in.
Sample curl Commands
From Murali (for the dev configdb).
Code Block |
---|
Get configuration for a device
curl -s -u "tstopr:passwordremoved" "https://pswww.slac.stanford.edu/ws-auth/devconfigdb/ws/configDB/get_configuration/tst/BEAM/trigger_0/"
Rename device
curl -s -u "tstopr:passwordremoved" "https://pswww.slac.stanford.edu/ws-auth/devconfigdb/ws/configDB/rename_device/tst/BEAM/trigger_0/?newname=trigger_1"
Remove device
curl -s -u "tstopr:passwordremoved" "https://pswww.slac.stanford.edu/ws-auth/devconfigdb/ws/configDB/remove_device/tst/BEAM/trigger_0/" |