Sometimes the instrument specialists give us new calibrations, usually just a dead strip list for TKR or three at a time for CAL (asym, MeV/DAC, peds). These need to be put in a standard place, registered in the calibration database, and set active. This requires picking a time when validity switches from the previous set to the new one. We want this to happen between runs. The gaps between non-SAA runs are short, and CALDB doesn't do leap seconds, so you should put the transitions near the middle of an SAA passage - the first one after the last run that we have data for.
The original update procedure used the rdbGUI and required various information to be input by hand. Since the rdbGUI will not work on >RHEL6 systems, a new Python update script was written in 2022 to automate the procedure.
In general, it is probably better to do the updates early in the week rather than later in case there are any problems.
There are three Python scripts created for the database updates:
You can see the range of options provided by each script by just giving it the -h (or --help) option.
The update script performs various checks and will exit with an error if any of these conditions fail:
The scripts are can be access in the conda environment called "calibrator" that is available on both the rhel6 and centos7 machines. This environment is also available on the SDF (with a slightly different activation command), but since the calibration files aren't (currently) accessible from the SDF, the update script won't work (it will give an error). The two database viewing scripts will work, though.
The scripts only recognize the four types of calibration files we currently update: CAL_Asym, CAL_MevPerDac, CAL_Ped, and TKR_DeadChan. Other types can be added easily if needed in the future.
These are things that you will need to do before you can run the calibration database update script. The old calibrator account that was used with rdbGUI stores its password in an obsolete and insecure manner that will not work with most modern MySQL interfaces (such as the one used by the Python script). While the database admins may update it in the future, for now, you will need to get your own account.
Create a MySQL options file in your home directory called ~/.my_cal.cnf
.
Change the permissions so that you are the only one who can read it (chmod 600
).
Add the following lines to the file:
[client] host=glastCalibDB.slac.stanford.edu user=<your database user name> password=<your database password> database=calib (for testing use calib_test) |
You may need to enclose your password in quotes if it contains certain special characters (like #).
These are the steps that you will have to do to update the database for a new calibration file(s).
bash: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.sh
(t)csh: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.cshrc
$LATCalibRoot
directory. If not, ask Tom Glanzman.$LATCalibRoot/TKR
$LATCalibRoot/CAL/p7repro
source /nfs/farm/g/glast/software/conda/bin/activate calibrator
Run the update_l1_calibration_database.py
script with the time in UTC for the new calibration(s) to take effect and the names of the one or more calibration files. It will update the database and print out the last two rows for each calibration type, so you can see the new file and the updated vend (stop) time of the previous calibration. Here's an example that updates all the usual CAL files:
(calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 22:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Rows changed/added in database for CAL_Asym: +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing fit_proton_calib_664m_668m_bigsum.calMPD.xml. Rows changed/added in database for CAL_MevPerDac: +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing pedavr_664m_668m.xml. Rows changed/added in database for CAL_Ped: +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated. |
If there are any problems, the script will print out an error message and stop, e.g.,:
(calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-22 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: The table has 1 row(s) with same calibration file. +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Exiting script. (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: Input date is before current UTC time. |
The script also has a noaction (aka dry-run) option that will show you what it would have done but not actually update the database, e.g.,
(calibrator) [horner@cent7a] ~ % update_l1_calibration_database.py "2022-04-26 23:00:00" LAT_BadStrips_64.xml -n Processing LAT_BadStrips_64.xml. No action set. Would have run SQL command: insert into metadata_v2r1 (instrument, calib_type, flavor, data_fmt, data_size,vstart, vend, locale, fmt_version, completion, proc_level, creator, uid, data_ident, enter_time) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) with values: ('LAT', 'TKR_DeadChan', 'L1current', 'XML', 66985, '2022-04-26 23:00:00', '2037-01-01 00:00:00', 'SLAC', 'v2r1', 'OK', 'PROD', 'update_l1_calibration_database.py', 'horner', '$(LATCalibRoot)/TKR/LAT_BadStrips_64.xml', datetime.datetime(2022, 4, 26, 20, 43, 44, 121595, tzinfo=datetime.timezone.utc)) No action set. Would have run command: update metadata_v2r1 set vend=%s where ser_no = %s with values: ('2022-04-26 23:00:00', 1287) No action set. No rows changed. Last entry is: +--------+------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+------------------------------------------+---------------------+---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 | 2037-01-01 00:00:00 | 2021-11-02 16:32:38 | +--------+------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated. |
You can then check the database with the show_l1_selection.py script. For example, after you run the update script but before the calibration start time, it will show the previous calibration file being selected, but you can see the vend time is set to your start time.
(calibrator) [horner@rhel6-64l] ~ % show_l1_selection.py Doing database lookups for CAL_Asym First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1294 | +--------+ Second query to get file name returns: +----------+-------------+----------------------------------------------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+----------------------------------------------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | +----------+-------------+----------------------------------------------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | +---------------------+---------------------+ Doing database lookups for CAL_MevPerDac First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1293 | +--------+ Second query to get file name returns: +----------+-------------+--------------------------------------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+--------------------------------------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | +----------+-------------+--------------------------------------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | +---------------------+---------------------+ Doing database lookups for CAL_Ped First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1292 | +--------+ Second query to get file name returns: +----------+-------------+--------------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+--------------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | +----------+-------------+--------------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | +---------------------+---------------------+ Doing database lookups for TKR_DeadChan First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1291 | +--------+ Second query to get file name returns: +----------+-------------+------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/TKR/LAT_BadStrips_64.xml | +----------+-------------+------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ |
If you set the time you're asking about (with the -s or --start option) to after the start time you set, then it will show the new calibration file as the one the pipeline will chose (note in this example the tracker file was not updated).
(calibrator) [horner@rhel6-64l] ~ % show_l1_selection.py -s "2022-04-22 00:00:00" Doing database lookups for CAL_Asym First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1295 | +--------+ Second query to get file name returns: +----------+-------------+----------------------------------------------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+----------------------------------------------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | +----------+-------------+----------------------------------------------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ Doing database lookups for CAL_MevPerDac First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1296 | +--------+ Second query to get file name returns: +----------+-------------+--------------------------------------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+--------------------------------------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | +----------+-------------+--------------------------------------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ Doing database lookups for CAL_Ped First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1297 | +--------+ Second query to get file name returns: +----------+-------------+--------------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+--------------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | +----------+-------------+--------------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ Doing database lookups for TKR_DeadChan First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1291 | +--------+ Second query to get file name returns: +----------+-------------+------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/TKR/LAT_BadStrips_64.xml | +----------+-------------+------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ |
You can also use the view_l1_calibration_database.py to see the status of the database:
(calibrator) [horner@rhel6-64l] ~ % view_l1_calibration_database.py Latest entries in database for CAL_Asym: +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Latest entries in database for CAL_MevPerDac: +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Latest entries in database for CAL_Ped: +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ Latest entries in database for TKR_DeadChan: +--------+------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+------------------------------------------+---------------------+---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 | 2022-02-17 23:00:00 | 2022-02-17 11:47:00 | | 1291 | $(LATCalibRoot)/TKR/LAT_BadStrips_64.xml | 2022-02-17 23:00:00 | 2037-01-01 00:00:00 | 2022-02-17 11:47:00 | +--------+------------------------------------------+---------------------+---------------------+---------------------+ |
These are the steps you should do after running a successful update.
Get the ID of the next run by using XTime or similar tool to convert start time of the next run after the SAA pass into MET.
Go to data processing page and click on the L1Proc bar for the run after the change should have happened.
This section is to help troubleshoot any errors.
The new script should prevent errors like this, but if the L1Proc deliveries start to fail after the calibration change takes effect, look at the logs mentioned above and search for any errors. If you see something like:
XmlBaseCnv FATAL Unable to parse document $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_634m_638m_bigsum.calMPD.xml aka /afs/slac/g/glast/ground/releases/calibrations//CAL/p7repro/fit_proton_calib_634m_638m_bigsum.calMPD.xml
then that calibration file was not actually copied to the TKR or CAL directories described above. Copy it there and then either contact someone to rollback the process or do it yourself following the instructions in Things to know while on-call for Data Processing. In this case, a command line rollback of delivery 210427007:
/afs/slac.stanford.edu/u/gl/glast/pipeline-II/prod/pipeline -m PROD rollbackStream --minimum 'L1Proc[210427007]'
was all that was needed to successfully process the delivery.
Further troubleshooting instructions will be added as issues arise.