Date: Fri, 29 Mar 2024 02:07:15 -0700 (PDT) Message-ID: <1567182920.8322.1711703235265@co.atl-prod-01> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_8321_1663600390.1711703235261" ------=_Part_8321_1663600390.1711703235261 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Sometimes the instrument specialists give us new calibrations, usually just a dead strip list for TKR or three = at a time for CAL (asym, MeV/DAC, peds). These need to be put in a standard= place, registered in the calibration database, and set active. This requir= es picking a time when validity switches from the previous set to the new o= ne. We want this to happen between runs. The gaps between non-SAA runs are = short, and CALDB doesn't do leap seconds, so you should put the transitions= near the middle of an SAA passage - the first one after the last run that = we have data for.
The original update procedure used the rdbGUI and required various infor= mation to be input by hand. Since the rdbGUI will not work on >RHE= L6 systems, a new Python update script was written in 2022 to automate the = procedure.
In general, it is probably better to do the updates early in the week ra= ther than later in case there are any problems.
There are three Python scripts created for the database updates:
You can see the range of options provided by each script by just giving = it the -h (or --help) option.
The update script performs various checks and will exit with an error if= any of these conditions fail:
The scripts can be accessed in the conda environment called "calibrator"= that is available on both the rhel6 and centos7 machines. This environment= is also available on the S3DF (with a slightly = different activation command), but sinc= e the calibration files aren't (currently) up-to-date on S3DF, the update s= cript will give an error. Also, the pipeline wouldn't know to look in= the S3DF directory. The two database viewing scripts will work on S3DF, th= ough.
The scripts only recognize the four types of calibration files we curren= tly update: CAL_Asym, CAL_MevPerDac, CAL_Ped, and TKR_DeadChan. = Other types can be added easily if needed in the future.
These are things that you will need to do before you can run the calibra= tion database update script.
Create a MySQL options file =
in your home directory called ~/.my_cal.cnf
.
Change the permissions so that you are the only one who can read it =
(chmod 600
).
Add the following lines to the file:
[client] host=3DglastCalibDB.slac.stanford.edu user=3D<your database user name> password=3D<your database password> database=3Dcalib (for testing use calib_test)
You may need to enclose your password in quotes if it contains cer= tain special characters (like #).
If you don't have an individual account, you can now use the 'calibrator= ' account as was done with the rdbGUI. This was not possible when the scrip= ts were written because the Python module would not work with the old passw= ord format used by MySQL 5.5, but since the upgrading the MySQL 8 (and rese= tting the password), it works. The password is the same as the user n= ame but with an "8" in place of the second "a".
These are the steps that you will have to do to update the database for = a new calibration file(s).
bash: source /afs/slac.stanford.edu/g/glast/ground/scripts/gro=
up.sh
(t)csh: source /afs/slac.stanford.edu/g/glast/ground/scripts/g=
roup.cshrc
$LATCalibRoot directory. If not, ask Tom Glanzman.
$LATCalibRoot directory. If not, ask Tom Glanzman.
$LATCalibRoot/TKR
$LATCalibRoot/CAL/p7repro
S3DF: source /sdf/group/fermi/sw/conda/bin/activate calibrator=
RHEL6: source /nfs/farm/g/glast/software/conda/bin/activate ca=
librator
Run the update_l1_calibration_database.py
script with t=
he time in UTC for the new calibration(s) to take effect and the names of t=
he one or more calibration files. It will update the database and pri=
nt out the last two rows for each calibration type, so you can see the new =
file and the updated vend (stop) time of the previous calibration. Here's a=
n example that updates all the usual CAL files:
(calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database= .py "2022-04-21 22:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.= xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Rows changed/added in database for CAL_Asym: +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ | ser_no | data_ident = | vstart | vend | update_t= ime | +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.g= cr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 1= 4:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.g= cr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 1= 4:12:07 | +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ Processing fit_proton_calib_664m_668m_bigsum.calMPD.xml. Rows changed/added in database for CAL_MevPerDac: +--------+-----------------------------------------------------------------= ---------+---------------------+---------------------+---------------------= + | ser_no | data_ident = | vstart | vend | update_time = | +--------+-----------------------------------------------------------------= ---------+---------------------+---------------------+---------------------= + | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.ca= lMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 = | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.ca= lMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 = | +--------+-----------------------------------------------------------------= ---------+---------------------+---------------------+---------------------= + Processing pedavr_664m_668m.xml. Rows changed/added in database for CAL_Ped: +--------+--------------------------------------------------+--------------= -------+---------------------+---------------------+ | ser_no | data_ident | vstart= | vend | update_time | +--------+--------------------------------------------------+--------------= -------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23= :00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22= :30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+--------------= -------+---------------------+---------------------+ Calibration database successfully updated.
If there are any problems, the script will print out an error mess= age and stop, e.g.,:
(calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database= .py "2022-04-22 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.= xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: The table has 1 row(s) with same calibration file. +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ | ser_no | data_ident = | vstart | vend | update_t= ime | +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.g= cr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 1= 4:12:07 | +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ Exiting script. (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022= -04-21 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_p= roton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: Input date is before current UTC time.
The script also has a noaction (aka dry-run) option that will show= you what it would have done but not actually update the database, e.g.,
(calibrator) [horner@cent7a] ~ % update_l1_calibration_database.py= "2022-04-26 23:00:00" LAT_BadStrips_64.xml -n Processing LAT_BadStrips_64.xml. No action set. Would have run SQL command: insert into metadata_v2r1 (instrument, calib_type, flavor, data_fmt, data_s= ize,vstart, vend, locale, fmt_version, completion, proc_level, creator, uid= , data_ident, enter_time) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s= ,%s) with values: ('LAT', 'TKR_DeadChan', 'L1current', 'XML', 66985, '2022-04-26 23:00:00', '= 2037-01-01 00:00:00', 'SLAC', 'v2r1', 'OK', 'PROD', 'update_l1_calibration_= database.py', 'horner', '$(LATCalibRoot)/TKR/LAT_BadStrips_64.xml', datetim= e.datetime(2022, 4, 26, 20, 43, 44, 121595, tzinfo=3Ddatetime.timezone.utc)= ) No action set. Would have run command: update metadata_v2r1 set vend=3D%s where ser_no =3D %s with values: ('2022-04-26 23:00:00', 1287) No action set. No rows changed. Last entry is: +--------+------------------------------------------+---------------------+= ---------------------+---------------------+ | ser_no | data_ident | vstart |= vend | update_time | +--------+------------------------------------------+---------------------+= ---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 |= 2037-01-01 00:00:00 | 2021-11-02 16:32:38 | +--------+------------------------------------------+---------------------+= ---------------------+---------------------+ Calibration database successfully updated.
You can then check the database with the show_l1_selection.py script.&nb= sp; For example, after you run the update script but before the calibration= start time, it will show the previous calibration file being selected, but= you can see the vend time is set to your start time.
(calibrator) [horner@rhel6-64l] ~ % show_l1_selection.py=20 Doing database lookups for CAL_Asym First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1294 | +--------+ Second query to get file name returns: +----------+-------------+-------------------------------------------------= ---------------------------------+ | data_fmt | fmt_version | data_ident = | +----------+-------------+-------------------------------------------------= ---------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_65= 8m_662m_bigsum.gcr_asym_hist.xml | +----------+-------------+-------------------------------------------------= ---------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | +---------------------+---------------------+ Doing database lookups for CAL_MevPerDac First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1293 | +--------+ Second query to get file name returns: +----------+-------------+-------------------------------------------------= -------------------------+ | data_fmt | fmt_version | data_ident = | +----------+-------------+-------------------------------------------------= -------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658= m_662m_bigsum.calMPD.xml | +----------+-------------+-------------------------------------------------= -------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | +---------------------+---------------------+ Doing database lookups for CAL_Ped First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1292 | +--------+ Second query to get file name returns: +----------+-------------+-------------------------------------------------= -+ | data_fmt | fmt_version | data_ident = | +----------+-------------+-------------------------------------------------= -+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml= | +----------+-------------+-------------------------------------------------= -+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | +---------------------+---------------------+ Doing database lookups for TKR_DeadChan First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1291 | +--------+ Second query to get file name returns: +----------+-------------+------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/TKR/LAT_BadStrips_64.xml | +----------+-------------+------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+
If you set the time you're asking about (with the -s or --start option) = to after the start time you set, then it will show the new calibration file= as the one the pipeline will choose (note in this example the tracker file= was not updated).
(calibrator) [horner@rhel6-64l] ~ % show_l1_selection.py -s "2022-= 04-22 00:00:00" Doing database lookups for CAL_Asym First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1295 | +--------+ Second query to get file name returns: +----------+-------------+-------------------------------------------------= ---------------------------------+ | data_fmt | fmt_version | data_ident = | +----------+-------------+-------------------------------------------------= ---------------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_66= 4m_668m_bigsum.gcr_asym_hist.xml | +----------+-------------+-------------------------------------------------= ---------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ Doing database lookups for CAL_MevPerDac First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1296 | +--------+ Second query to get file name returns: +----------+-------------+-------------------------------------------------= -------------------------+ | data_fmt | fmt_version | data_ident = | +----------+-------------+-------------------------------------------------= -------------------------+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664= m_668m_bigsum.calMPD.xml | +----------+-------------+-------------------------------------------------= -------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ Doing database lookups for CAL_Ped First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1297 | +--------+ Second query to get file name returns: +----------+-------------+-------------------------------------------------= -+ | data_fmt | fmt_version | data_ident = | +----------+-------------+-------------------------------------------------= -+ | XML | v2r1 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml= | +----------+-------------+-------------------------------------------------= -+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+ Doing database lookups for TKR_DeadChan First query to get ser_no returns: +--------+ | ser_no | +--------+ | 1291 | +--------+ Second query to get file name returns: +----------+-------------+------------------------------------------+ | data_fmt | fmt_version | data_ident | +----------+-------------+------------------------------------------+ | XML | v2r1 | $(LATCalibRoot)/TKR/LAT_BadStrips_64.xml | +----------+-------------+------------------------------------------+ Third query to get vstart and vend returns: +---------------------+---------------------+ | vstart | vend | +---------------------+---------------------+ | 2022-02-17 23:00:00 | 2037-01-01 00:00:00 | +---------------------+---------------------+
You can also use the view_l1_calibration_database.py to see the sta= tus of the database:
(calibrator) [horner@rhel6-64l] ~ % view_l1_calibration_database.p= y=20 Latest entries in database for CAL_Asym: +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ | ser_no | data_ident = | vstart | vend | update_t= ime | +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.g= cr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 1= 4:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.g= cr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 1= 4:12:07 | +--------+-----------------------------------------------------------------= -----------------+---------------------+---------------------+-------------= --------+ Latest entries in database for CAL_MevPerDac: +--------+-----------------------------------------------------------------= ---------+---------------------+---------------------+---------------------= + | ser_no | data_ident = | vstart | vend | update_time = | +--------+-----------------------------------------------------------------= ---------+---------------------+---------------------+---------------------= + | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.ca= lMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 = | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.ca= lMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 = | +--------+-----------------------------------------------------------------= ---------+---------------------+---------------------+---------------------= + Latest entries in database for CAL_Ped: +--------+--------------------------------------------------+--------------= -------+---------------------+---------------------+ | ser_no | data_ident | vstart= | vend | update_time | +--------+--------------------------------------------------+--------------= -------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23= :00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22= :30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+--------------= -------+---------------------+---------------------+ Latest entries in database for TKR_DeadChan: +--------+------------------------------------------+---------------------+= ---------------------+---------------------+ | ser_no | data_ident | vstart |= vend | update_time | +--------+------------------------------------------+---------------------+= ---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 |= 2022-02-17 23:00:00 | 2022-02-17 11:47:00 | | 1291 | $(LATCalibRoot)/TKR/LAT_BadStrips_64.xml | 2022-02-17 23:00:00 |= 2037-01-01 00:00:00 | 2022-02-17 11:47:00 | +--------+------------------------------------------+---------------------+= ---------------------+---------------------+
These are the steps you should do after running a successful update.
Get the ID of the next run by using XTime or similar tool to convert start time of the next run after= the SAA pass into MET.
Go to data processing page and c= lick on the L1Proc bar for the run after the change should have happened.= p>
This section is to help troubleshoot any errors.
The new script should prevent errors like this, but if the L1Proc delive= ries start to fail after the calibration change takes effect, look at the l= ogs mentioned above and search for any errors. If you see something = like:
XmlBaseCnv FATAL Unable to parse document $(LATCalibRoot)/CAL/p7repro/fi= t_proton_calib_634m_638m_bigsum.calMPD.xml aka /afs/slac/g/glast/ground/rel= eases/calibrations//CAL/p7repro/fit_proton_calib_634m_638m_bigsum.calMPD.xm= l
then that calibration file was not actually copied to the TKR or CAL dir=
ectories described above. Copy it there and then either contact someo=
ne to rollback the process or do it yourself following the instructions in =
Th=
ings to know while on-call for Data Processing. In this case, a command line rollback of delivery 21042=
7007:
/afs/slac.stanford.edu/u/gl/glast/pipeline-II/prod/pipeline -m PROD roll= backStream --minimum 'L1Proc[210427007]'
was all that was needed to successfully process the delivery.
Further troubleshooting instructions will be added as issues arise.<= /em>