Page History

...

• The calibration files exist where they are supposed to.
• The time you give it is valid.  It should have the correct format (YYYY-MM-DD HH:MM:SS) and cannot be in the past or more than 24 hours in the future.  The 24 hours is an arbitrary limit just to make sure you're got giving it a very wrong date by accident.
• The calibration file is not already in the database (based on the file name), so you can't accidentally enter a duplicate.
• For tracker files, the version number in the file name is higher than the current version. 
• For non-tracker files, the file name numbers are 6 greater than for the previous calibration file values, e.g., pedavr_658m_662m.xml should be followed by pedavr_664m_668m.xml and the two numbers in the file name are 4 apart. Each file is meant to cover 6 megaseconds and are created from three 2 Ms files, so 240m-244m is made of the 240m, 242m, and 244m files.  These two checks are taken from information on the file name on the page Working Group on Calibration Issues. They can be relaxed if they end up being too strict.

The scripts are can be access accessed in the conda environment called "calibrator" that is available on both the rhel6 and centos7 machines. This environment is also available on the SDF S3DF (with a slightly different activation command), but since the calibration files aren't (currently) accessible from the SDFup-to-date on S3DF, the update script won't work (it will give an error).  Also, the pipeline wouldn't know to look in the S3DF directory. The two database viewing scripts will work on S3DF, though.

The scripts only recognize the four types of calibration files we currently update: CAL_Asym,  CAL_MevPerDac, CAL_Ped, and TKR_DeadChan.  Other types can be added easily if needed in the future.

...

These are things that you will need to do before you can run the calibration database update script. The old calibrator account that was used with rdbGUI stores its password in an obsolete and insecure manner that will not work with most modern MySQL interfaces (such as the one used by the Python script).  While the database admins may update it in the future, for now, you will need to get your own account.

1. Create a MySQL options file in your home directory called ~/.my_cal.cnf.

   1. This file will contain the database access parameters the Python scripts will use.
2. Send a message to db-admin@slac.stanford.edu and cc horner@slac.stanford.edu to ask for an account on the MySQL server glastCalibDB.slac.stanford.edu with the same permissions on the calib and calib_test databases as the horner account.   They should send you a password.
   

3. Create a MySQL options file in your home directory called ~/.my_cal.cnf.

   1. This file will contain the database access parameters the Python scripts will use.
   2. You can use a different file with the -o parameter. It can be useful to have different files for the calib and calib_test databases. Be sure to include the full path (e.g., ~/) for the file.

   3. Change the permissions so that you are the only one who can read it (chmod 600).

4. Add the following lines to the file:

   Code Block
   language text title cnf file collapse true
   [client] host=glastCalibDB.slac.stanford.edu user=<your database user name> password=<your database password> database=calib (for testing use calib_test)

   You may need to enclose your password in quotes if it contains certain special characters (like #).

   name> password=<your database password> database=calib (for testing use calib_test)

   You may need to enclose your password in quotes if it contains certain special characters (like #). 

If you don't have an individual account, you can now use the 'calibrator' account as was done with the rdbGUI. This was not possible when the scripts were written because the Python module would not work with the old password format used by MySQL 5.5, but since the upgrading the MySQL 8 (and resetting the password), it works.  The password is the same as the user name but with an "8" in place of the second "a". 

Updating the Database

These are the steps that you will have to do to update the database for a new calibration file(s).

1. If there is not already an existing issue for the update, start one in the JIRA PII tracker.  Here is an example that can be used as a template:

   Jira
   server SLAC National Accelerator Laboratory columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key PII-454

   based off the ones they use for on-board hot strip masking updates (e.g.,

   Jira
   server SLAC National Accelerator Laboratory columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key OBCONF-204

   ). 
2. Figure out the transition time:
   1. Go to the data processing page.
      
      1. Find the most recent run. It will be at the top, unless deliveries have arrived out of order.
      2. Click on the run number.
      3. Note the start time.
   2. Go to the mission planning timeline
      1. Find the first SAA passage that is still in the future processing-wise.
         1. NOTE: use an SAA passage that is in the future
         processing-wise
         1. in UTC, not just processing-wise. The update script will reject an update attempt applied to the UTC past.
      2. Pick a time that is in the middle of the SAA passage. That is when we will start the new calibration.
      3. You may also want to note the start time of the next run after the SAA.  You will want to send that information to the data monitors list (see below) when you inform them of the new calibration.
3. Setup your environment:  :
   1. On RHEL6 or CentOS: 
      
      1. Log into a SLAC machine. 
      2. If one is not already sourced in your .bashrc or .cshrc then:
         

         1. bash: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.sh

         2. (t)csh: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.cshrc

      3. This will set environment variables like $LATCalibRoot where the calibration files are stored.
      4. Make sure that you have write permissions to the $LATCalibRoot directory. If not, ask Tom Glanzman.
   2. On S3DF:
      1. Log into S3DF and then the fermi-devl node.
      2. If you have .profile.d files sourced in your .bash_profile as recommened in the S3DF Cheatsheet, then $LATCalibRoot will be set for you.
      3. Make sure that you have write permissions to the $LATCalibRoot directory. If not, ask Tom Glanzman.
4. Copy files to the "normal" place (if the calibration file creators haven't already):
   1. TKR: $LATCalibRoot/TKR
   2. CAL: $LATCalibRoot/CAL/p7repro
5. Start the calibrator conda environment.  Note that this uses the Fermitools/Conda Shared Installation at SLAC, which only works with the bash shell.
   

   1. S3DF: source /sdf/group/fermi/sw/conda/bin/activate calibrator

   2. RHEL6: source

      source

      /nfs/farm/g/glast/software/conda/bin/activate

      calibrator

6. Run the update_l1_calibration_database.py script with the time in UTC for the new calibration(s) to take effect and the names of the one or more calibration files.  It will update the database and print out the last two rows for each calibration type, so you can see the new file and the updated vend (stop) time of the previous calibration. Here's an example that updates all the usual CAL files:

   Code Block
   language text title Example run collapse true
   (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 22:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Rows changed/added in database for CAL_Asym: +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing fit_proton_calib_664m_668m_bigsum.calMPD.xml. Rows changed/added in database for CAL_MevPerDac: +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing pedavr_664m_668m.xml. Rows changed/added in database for CAL_Ped: +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated.

   If there are any problems, the script will print out an error message and stop, e.g.,:

   Code Block
   language text title Error Examples collapse true
   (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-22 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: The table has 1 row(s) with same calibration file. +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Exiting script. (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: Input date is before current UTC time.

   The script also has a noaction (aka dry-run) option that will show you what it would have done but not actually update the database, e.g.,

   Code Block
   language text title Example of dry-run collapse true
   (calibrator) [horner@cent7a] ~ % update_l1_calibration_database.py "2022-04-26 23:00:00" LAT_BadStrips_64.xml -n Processing LAT_BadStrips_64.xml. No action set. Would have run SQL command: insert into metadata_v2r1 (instrument, calib_type, flavor, data_fmt, data_size,vstart, vend, locale, fmt_version, completion, proc_level, creator, uid, data_ident, enter_time) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) with values: ('LAT', 'TKR_DeadChan', 'L1current', 'XML', 66985, '2022-04-26 23:00:00', '2037-01-01 00:00:00', 'SLAC', 'v2r1', 'OK', 'PROD', 'update_l1_calibration_database.py', 'horner', '$(LATCalibRoot)/TKR/LAT_BadStrips_64.xml', datetime.datetime(2022, 4, 26, 20, 43, 44, 121595, tzinfo=datetime.timezone.utc)) No action set. Would have run command: update metadata_v2r1 set vend=%s where ser_no = %s with values: ('2022-04-26 23:00:00', 1287) No action set. No rows changed. Last entry is: +--------+------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+------------------------------------------+---------------------+---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 | 2037-01-01 00:00:00 | 2021-11-02 16:32:38 | +--------+------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated.

...

If you set the time you're asking about (with the -s or --start option) to after the start time you set, then it will show the new calibration file as the one the pipeline will chose choose (note in this example the tracker file was not updated).

...