Sometimes the instrument specialists give us new calibrations, usually just a dead strip list for TKR , 3 or three at a time for CAL (asym, MeV/DAC, peds) at a time for CAL. They These need to be put in a standard place, registered in the calibration database, and set active. This requires picking a time when validity switches from the previous set to the new one. We want this to happen between runs. The gaps between non-SAA runs are short, and CALDB doesn't do leap seconds, so you should put the transitions near the middle of an SAA passage - the first one after the last run that we have data for.
The original update procedure used the rdbGUI and required various information to be input by hand. Since the rdbGUI would will not work on >RHEL6 systems, a new Python update script was written in 2022 to automate the procedure.
In general, it is probably better to do the updates early in the week rather than later in case there are any problems.
There are three Python scripts created for the database updates:
...
The update script performs various checks and will exit with an error if any of these conditions fail:
The scripts are can be access accessed in the conda environment called "calibrator" that is available on both the rhel6 and centos7 machines. This environment is also available on the SDF S3DF (with a slightly different activation command), but since the calibration files aren't (currently) accessible from the SDFup-to-date on S3DF, the update script won't work (it will give an error). Also, the pipeline wouldn't know to look in the S3DF directory. The two database viewing scripts will work on S3DF, though.
The scripts only recognize the four types of calibration files we currently update: CAL_Asym, CAL_MevPerDac, CAL_Ped, and TKR_DeadChan. Other types can be added easily if needed in the future.
...
These are things that you will need to do before you can run the calibration database update script. The old calibrator account that was used in the old procedure stores its password in an obsolete and insecure manner that will not work with most modern MySQL interfaces (such as the one used by the Python script). While the database admins may update it in the future, for now, you will need to get your own account.
Create a MySQL options file in your home directory called ~/.my_cal.cnf
.
Create a MySQL options file in your home directory called ~/.my_cal.cnf
.
Change the permissions so that you are the only one who can read it (chmod 600
).
Add the following lines to the file:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
[client]
host=glastCalibDB.slac.stanford.edu
user=<your database user name>
password=<your database password>
database=calib (for testing use calib_test) |
name>
password=<your database password>
database=calib |
You can use the calib_test database instead of the calib database for testing. You may need to enclose your password in quotes if it contains certain special characters (like #).
If you don't have an individual account, you can now use the 'calibrator' account as was done with the rdbGUI. This was not possible when the scripts were written because the Python module would not work with the old password format used by MySQL 5.5, but since the upgrading the MySQL 8 (and resetting the password), it works. The password is the same as the user name but with an "8" in place of the second "a".
These are the steps that you will have to do to update the database for a new calibration file(s).
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
bash: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.sh
(t)csh: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.cshrc
$LATCalibRoot
directory. If not, ask Tom Glanzman.$LATCalibRoot
will be set for you.$LATCalibRoot
directory. If not, ask Tom Glanzman.$LATCalibRoot/TKR
$LATCalibRoot/CAL/p7repro
S3DF: source /sdf/group/fermi/sw/conda/bin/activate calibrator
RHEL6: source
/nfs/farm/g/glast/software/conda/bin/activate
calibrator
Run the update_l1_calibration_database.py
script with the time in UTC for the new calibration(s) to take effect and the names of the one or more calibration files. It will update the database and print out the last two rows for each calibration type, so you can see the new file and the updated vend (stop) time of the previous calibration. Here's an example that updates all the usual CAL files:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
(calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 22:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Rows changed/added in database for CAL_Asym: +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing fit_proton_calib_664m_668m_bigsum.calMPD.xml. Rows changed/added in database for CAL_MevPerDac: +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing pedavr_664m_668m.xml. Rows changed/added in database for CAL_Ped: +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated. |
If there are any problems, the script will print out an error message and stop, e.g.,:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
(calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-22 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: The table has 1 row(s) with same calibration file. +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Exiting script. (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: Input date is before current UTC time. |
The script also has a noaction (aka dry-run) option that will show you what it would have done but not actually update the database, e.g.,
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
(calibrator) [horner@cent7a] ~ % update_l1_calibration_database.py "2022-04-26 23:00:00" LAT_BadStrips_64.xml -n Processing LAT_BadStrips_64.xml. No action set. Would have run SQL command: insert into metadata_v2r1 (instrument, calib_type, flavor, data_fmt, data_size,vstart, vend, locale, fmt_version, completion, proc_level, creator, uid, data_ident, enter_time) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) with values: ('LAT', 'TKR_DeadChan', 'L1current', 'XML', 66985, '2022-04-26 23:00:00', '2037-01-01 00:00:00', 'SLAC', 'v2r1', 'OK', 'PROD', 'update_l1_calibration_database.py', 'horner', '$(LATCalibRoot)/TKR/LAT_BadStrips_64.xml', datetime.datetime(2022, 4, 26, 20, 43, 44, 121595, tzinfo=datetime.timezone.utc)) No action set. Would have run command: update metadata_v2r1 set vend=%s where ser_no = %s with values: ('2022-04-26 23:00:00', 1287) No action set. No rows changed. Last entry is: +--------+------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+------------------------------------------+---------------------+---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 | 2037-01-01 00:00:00 | 2021-11-02 16:32:38 | +--------+------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated. |
You can then check the database with the show_l1_selection.py script. For example, after you run the update script but before the calibration start time, it will show the previous calibration file being selected, but you can see the vend time is set to your start time.
...
If you set the time you're asking about (with the -s or --start option) to after the start time you set, then it will show the new calibration file as the one the pipeline will chose choose (note in this example the tracker file was not updated).
...
Get the ID of the next run by using XTime or similar tool to convert start time of the next run after the SAA pass into MET.
Go to data processing page and click on the L1Proc bar for the run after the change should have happened.
...
then that calibration file was not actually copied to the TKR or CAL directories described above. Copy it there and then either contact someone to rollback the process or do it yourself following the instructions in Things to know while on-call for Data Processing. In this case, a command line rollback of delivery 210427007:
/afs/slac.stanford.edu/u/gl/glast/pipeline-II/prod/pipeline -m PROD rollbackStream --minimum 'L1Proc[210427007]'
...
Further troubleshooting instructions will be added as issues come uparise.