Page History

Sometimes the instrument specialists give us new calibrations, usually just a dead strip list for TKR , 3 or three at a time for CAL (asym, MeV/DAC, peds) at a time for CAL. They These need to be put in a standard place, registered in the calibration database, and set active. This requires picking a time when validity switches from the previous set to the new one. We want this to happen between runs. The gaps between non-SAA runs are short, and CALDB doesn't do leap seconds, so you should put the transitions near the middle of an SAA passage - the first one after the last run that we have data for.

...

In general, it is probably better to do the updates early in the week rather than later in case there are any problems.

Database Update Scripts

There are three Python scripts created for the database updates:

• update_l1_calibration_database.py - This is the script that actually does the database update.  Technically, it's the only one you really need.  You give it the UTC start time for the calibration to take effect and the names of the new calibration filesfile(s).  It adds a new row in the database table for the each file and changes the end time of the previous calibration file to the start time of the new file.
• show_l1_selection.py - This runs executes the exact same three database queries the L1 pipeline does when getting the each type of calibration file.  The first query gets the "ser_no", which is basically the row number in the table.  The second gets the "data_ident" (file name) and a few other parameters for that the ser_no, and the third prints out the start and end times ("vstart" and "vend") for using that calibration file.  By default, the script will show it for the current time (in UTC) but you can set it to another time.  This is useful for checking what the pipeline will choose before and after the start time you give it.
• view_l1_calibration_database.py - You can use this script to more generally view prints the contents of the database table.  By default, it will print out the last two rows for each of the four calibration types we update (which should be the previous and current calibrations).  The script also has various options to change the rows and columns output.

...

The update script performs various checks and will exit with an error if any of these conditions fail:

• It checks that The calibration files exist where they are supposed to.
• It checks that the The time you give it is valid.  If it is It should have the correct format (YYYY-MM-DD HH:MM:SS) and cannot be in the past or more than 24 hours in the future, it will give an error.  The 24 hours is an arbitrary limit just to make sure you're got giving it a very wrong date by accident.
• It checks if a The calibration file with the same name is not already in the database (based on the file name), so you can't accidently accidentally enter a duplicate a file.
• For tracker files, it checks that the version number in the file name is higher than the current version. 
• For the other non-tracker files, it checks that the file name numbers are 6 greater than for the previous calibration file values, e.g., pedavr_658m_662m.xml should be followed by pedavr_664m_668m.xml .For non-tracker files, it checks that and the two numbers in the file name are 4 apart. Each file is meant to cover 6 megaseconds and are created from three 2 Ms files, so 240m-244m is made of the 240m, 242m, and 244m files.  These two checks are taken from information on the file name on the page Working Group on Calibration Issues. They can be relaxed if they end up being too strict.

The scripts are can be access accessed in the conda environment called "calibrator" that is available on both the rhel6 and centos7 machines. This environment is also available on the SDF S3DF (with a slightly different activation command), but since the calibration files aren't (currently) accessible from the SDFup-to-date on S3DF, the update script won't work (it will give an error).  Also, the pipeline wouldn't know to look in the S3DF directory. The two database viewing scripts will work on S3DF, though.

The scripts only recognize the four types of calibration files we currently update: CAL_Asym,  CAL_MevPerDac, CAL_Ped, and TKR_DeadChan.  Other types can be added easily if needed in the future.

...

These are things that you will need to do before you can run the calibration database update script. The old calibrator account that was used in the old procedure stores its password in an obsolete and insecure manner that will not work with most modern MySQL interfaces (such as the one used by the Python script).  While the database admins may update it in the future, for now, you will need to get your own account.

1. Create a MySQL options file in your home directory called ~/.my_cal.cnf.

   1. This file will contain the database access parameters the Python scripts will use.
   2. You can use a different file with the -o parameter. It can be useful to have different files for
   Send a message to db-admin@slac.stanford.edu and cc horner@slac.stanford.edu to ask for an account on the MySQL server glastCalibDB.slac.stanford.edu with the same permissions on
   1. the calib and calib_test databases
   as the horner account.   They should send you a password.
   

   Create a MySQL options file in your home directory called ~/.my_cal.cnf.

   1. This file will contain the database access parameters the Python scripts will use, although you can give them a different file with the -o parameter. Be sure to include the full path (e.g., ~/) for the file.

   2. Change the permissions so that you are the only one who can read it (chmod 600).

2. Add the following lines to the file:

   Code Block
   language text title cnf file collapse true
   [client] host=glastCalibDB.slac.stanford.edu user=<your database user name> password=<your database password> database=calib (for testing use

   You can use the calib_test

   )

   database instead of the calib database for testing. You may need to enclose your password in quotes if it contains certain special characters (like #).

Updating the Database

These are the steps that you will have to do to update the database for a new calibration file(s).

If you don't have an individual account, you can now use the 'calibrator' account as was done with the rdbGUI. This was not possible when the scripts were written because the Python module would not work with the old password format used by MySQL 5.5, but since the upgrading the MySQL 8 (and resetting the password), it works.  The password is the same as the user name but with an "8" in place of the second "a". 

Updating the Database

These are the steps that you will have to do to update the database for a new calibration file(s).

1. If there is not already an existing issue for the update, start one in the If there is not already an existing issue for the update, start one in the JIRA PII tracker.  Here is an example that can be used as a template:

   Jira
   server SLAC National Accelerator Laboratory columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key PII-454

   based off the ones they use for on-board hot strip masking updates (e.g.,

   Jira
   server SLAC National Accelerator Laboratory columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key OBCONF-204

   ). 
2. Figure out the transition time:
   1. Go to the data processing page.
      
      1. Find the most recent run. It will be at the top, unless deliveries have arrived out of order.
      2. Click on the run number.
      3. Note the start time.
   2. Go to the mission planning timeline
      1. Find the first SAA passage that is still in the future processing-wise.
         1. NOTE: use an SAA passage that is in the future in UTC, not just processing-wise. The update script will reject an update attempt applied to the UTC past.
      2. Pick a Pick a time that is in the middle of the SAA passage. That is when we will start the new calibration.
      3. You may also want to note the start time of the next run after the SAA.  You will want to send that information to the data monitors list (see below) when you inform them of the new calibration.
3. Setup your environment:  :
   1. On RHEL6 or CentOS: 
      
      1. Log into a SLAC machine. 
      2. If one is not already sourced in your .bashrc or .cshrc then:
         

         1. bash: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.sh

         2. (t)csh: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.cshrc

      3. This will set environment variables like $LATCalibRoot where the calibration files are stored.
      4. Make sure that you have write permissions to the $LATCalibRoot directory. If not, ask Tom Glanzman.
   2. On S3DF:
      1. Log into S3DF and then the fermi-devl node.
      2. If you have .profile.d files sourced in your .bash_profile as recommened in the S3DF Cheatsheet, then $LATCalibRoot will be set for you.
      3. Make sure that you have write permissions to the $LATCalibRoot directory. If not, ask Tom Glanzman.
4. Copy files to the "normal" place (if the calibration file creators haven't already):
   1. TKR: $LATCalibRoot/TKR
   2. CAL: $LATCalibRoot/CAL/p7repro
5. Start the calibrator conda environment.  Note that this uses the Fermitools/Conda Shared Installation at SLAC, which only works with the bash shell.
   

   1. S3DF: source /sdf/group/fermi/sw/conda/bin/activate calibrator

   2. RHEL6: source /nfs/farm/

   3. source /nfs/farm/

      g/glast/software/conda/bin/activate

      calibrator

      calibrator

      1. You should probably see your terminal prompt change to include "(calibrator)".

6. Run the update_l1_calibration_database.py script with the time in UTC for the new calibration(s) to take effect and the names of the one or more calibration files.  It will update the database and print out the last two rows for each calibration type, so you can see the new file and the updated vend (stop) time of the previous calibration. Here's an example that updates all the usual CAL files:

   Code Block
   language text title Example run collapse true
   (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 22:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Rows changed/added in database for CAL_Asym: +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing fit_proton_calib_664m_668m_bigsum.calMPD.xml. Rows changed/added in database for CAL_MevPerDac: +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing pedavr_664m_668m.xml. Rows changed/added in database for CAL_Ped: +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated.

   If there are any problems, the script will print out an error message and stop, e.g.,:

   Code Block
   language text title Error Examples collapse true
   (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-22 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: The table has 1 row(s) with same calibration file. +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Exiting script. (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: Input date is before current UTC time.

   The script also has a noaction (aka dry-run) option that will show you what it would have done but not actually update the database, e.g.,

   Code Block
   language text title Example of dry-run collapse true
   (calibrator) [horner@cent7a] ~ % update_l1_calibration_database.py "2022-04-26 23:00:00" LAT_BadStrips_64.xml -n Processing LAT_BadStrips_64.xml. No action set. Would have run SQL command: insert into metadata_v2r1 (instrument, calib_type, flavor, data_fmt, data_size,vstart, vend, locale, fmt_version, completion, proc_level, creator, uid, data_ident, enter_time) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) with values: ('LAT', 'TKR_DeadChan', 'L1current', 'XML', 66985, '2022-04-26 23:00:00', '2037-01-01 00:00:00', 'SLAC', 'v2r1', 'OK', 'PROD', 'update_l1_calibration_database.py', 'horner', '$(LATCalibRoot)/TKR/LAT_BadStrips_64.xml', datetime.datetime(2022, 4, 26, 20, 43, 44, 121595, tzinfo=datetime.timezone.utc)) No action set. Would have run command: update metadata_v2r1 set vend=%s where ser_no = %s with values: ('2022-04-26 23:00:00', 1287) No action set. No rows changed. Last entry is: +--------+------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+------------------------------------------+---------------------+---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 | 2037-01-01 00:00:00 | 2021-11-02 16:32:38 | +--------+------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated.

You can then check the database with the show_l1_selection.py script.  For example, after you run the update script but before the calibration start time, it will show the previous calibration file being selected, but you can see the vend time is set to your start time.

...

If you set the time you're asking about (with the -s or --start option) to after the start time you set, then it will show the new calibration file as the one the pipeline will chose choose (note in this example the tracker file was not updated).

...

was all that was needed to successfully process the delivery.

Changing the Vstart Time

The script currently has no way to fix problems, e.g., you realized you put in the wrong vstart time.  These types of problems can be fixed manually.  For example, to change the vstart time, you can do something like this:

mysql -h glastCalibDB.slac.stanford.edu -u calibrator -p
use calib;
select ser_no,vstart,data_ident from metadata_v2r1;
update metadata_v2r1 set vstart = '2024-08-21 01:30:00' where ser_no = 1340;
update metadata_v2r1 set vstart = '2024-08-21 01:30:00' where ser_no = 1341;
update metadata_v2r1 set vstart = '2024-08-21 01:30:00' where ser_no = 1342;
update metadata_v2r1 set vend = '2024-08-21 01:30:00' where ser_no = 1336;
update metadata_v2r1 set vend = '2024-08-21 01:30:00' where ser_no = 1337;
update metadata_v2r1 set vend = '2024-08-21 01:30:00' where ser_no = 1338;

This will log into the MySQL server (you will need the calibrator password given earlier) and switch to the 'calib' database.  You can view the contents of the table metadata_v2r1, and then change the 'vstart' times of the latest files, and the 'vend' time of the previous files.  You can get the 'ser_no' values from the output of the 'select' command or from the Python scripts as shown above.

Further troubleshooting instructions will be added as issues ariseFurther troubleshooting instructions will be added as issues come up.