Page History

...

1. Create a MySQL options file in your home directory called ~/.my_cal.cnf.

   1. This file will contain the database access parameters the Python scripts will use.
   2. You can use a different file with the -o parameter. It can be useful to have different files for the calib and calib_test databases. Be sure to include the full path (e.g., ~/) for the file.

   3. Change the permissions so that you are the only one who can read it (chmod 600).

2. Add the following lines to the file:

   Code Block
   language text title cnf file collapse true
   [client] host=glastCalibDB.slac.stanford.edu user=<your database user name> password=<your database password> database=calib (for testing use

   You can use the calib_test

   )

   database instead of the calib database for testing. You may need to enclose your password in quotes if it contains certain special characters (like #). 

...

1. If there is not already an existing issue for the update, start one in the JIRA PII tracker.  Here is an example that can be used as a template:

   Jira
   server SLAC National Accelerator Laboratory columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key PII-454

   based off the ones they use for on-board hot strip masking updates (e.g.,

   Jira
   server SLAC National Accelerator Laboratory columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 1b8dc293-975d-3f2d-b988-18fd9aec1546 key OBCONF-204

   ). 
2. Figure out the transition time:
   1. Go to the data processing page.
      
      1. Find the most recent run. It will be at the top, unless deliveries have arrived out of order.
      2. Click on the run number.
      3. Note the start time.
   2. Go to the mission planning timeline
      1. Find the first SAA passage that is still in the future processing-wise.
         1. NOTE: use an SAA passage that is in the future in UTC, not just processing-wise. The update script will reject an update attempt applied to the UTC past.
      2. Pick a time that is in the middle of the SAA passage. That is when we will start the new calibration.
      3. You may also want to note the start time of the next run after the SAA.  You will want to send that information to the data monitors list (see below) when you inform them of the new calibration.
3. Setup your environment:
   1. On RHEL6 or CentOS: 
      
      1. Log into a SLAC machine. 
      2. If one is not already sourced in your .bashrc or .cshrc then:
         

         1. bash: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.sh

         2. (t)csh: source /afs/slac.stanford.edu/g/glast/ground/scripts/group.cshrc

      3. This will set environment variables like $LATCalibRoot where the calibration files are stored.
      4. Make sure that you have write permissions to the $LATCalibRoot directory. If not, ask Tom Glanzman.
   2. On S3DF:
      1. Log into S3DF and then the fermi-devl node.
      2. If you have .profile.d files sourced in your .bash_profile as recommened in the S3DF Cheatsheet, then $LATCalibRoot will be set for you.
      3. Make sure that you have write permissions to the $LATCalibRoot directory. If not, ask Tom Glanzman.
4. Copy files to the "normal" place (if the calibration file creators haven't already):
   1. TKR: $LATCalibRoot/TKR
   2. CAL: $LATCalibRoot/CAL/p7repro
5. Start the calibrator conda environment.  Note that this uses the Fermitools/Conda Shared Installation at SLAC, which only works with the bash shell.
   

   1. S3DF: source /sdf/group/fermi/sw/conda/bin/activate calibrator

   2. RHEL6: source /nfs/farm/g/glast/software/conda/bin/activate calibrator

6. Run the update_l1_calibration_database.py script with the time in UTC for the new calibration(s) to take effect and the names of the one or more calibration files.  It will update the database and print out the last two rows for each calibration type, so you can see the new file and the updated vend (stop) time of the previous calibration. Here's an example that updates all the usual CAL files:

   Code Block
   language text title Example run collapse true
   (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 22:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Rows changed/added in database for CAL_Asym: +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1294 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_658m_662m_bigsum.gcr_asym_hist.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing fit_proton_calib_664m_668m_bigsum.calMPD.xml. Rows changed/added in database for CAL_MevPerDac: +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1293 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_658m_662m_bigsum.calMPD.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1296 | $(LATCalibRoot)/CAL/p7repro/fit_proton_calib_664m_668m_bigsum.calMPD.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Processing pedavr_664m_668m.xml. Rows changed/added in database for CAL_Ped: +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ | 1292 | $(LATCalibRoot)/CAL/p7repro/pedavr_658m_662m.xml | 2022-02-17 23:00:00 | 2022-04-21 22:30:00 | 2022-04-21 14:12:07 | | 1297 | $(LATCalibRoot)/CAL/p7repro/pedavr_664m_668m.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+--------------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated.

   If there are any problems, the script will print out an error message and stop, e.g.,:

   Code Block
   language text title Error Examples collapse true
   (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-22 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: The table has 1 row(s) with same calibration file. +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ | 1295 | $(LATCalibRoot)/CAL/p7repro/fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml | 2022-04-21 22:30:00 | 2037-01-01 00:00:00 | 2022-04-21 14:12:07 | +--------+----------------------------------------------------------------------------------+---------------------+---------------------+---------------------+ Exiting script. (calibrator) [horner@rhel6-64l] ~ % update_l1_calibration_database.py "2022-04-21 18:30:00" fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml fit_proton_calib_664m_668m_bigsum.calMPD.xml pedavr_664m_668m.xml Processing fit_gcrhists_lkhd_664m_668m_bigsum.gcr_asym_hist.xml. Error: Input date is before current UTC time.

   The script also has a noaction (aka dry-run) option that will show you what it would have done but not actually update the database, e.g.,

   Code Block
   language text title Example of dry-run collapse true
   (calibrator) [horner@cent7a] ~ % update_l1_calibration_database.py "2022-04-26 23:00:00" LAT_BadStrips_64.xml -n Processing LAT_BadStrips_64.xml. No action set. Would have run SQL command: insert into metadata_v2r1 (instrument, calib_type, flavor, data_fmt, data_size,vstart, vend, locale, fmt_version, completion, proc_level, creator, uid, data_ident, enter_time) values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s) with values: ('LAT', 'TKR_DeadChan', 'L1current', 'XML', 66985, '2022-04-26 23:00:00', '2037-01-01 00:00:00', 'SLAC', 'v2r1', 'OK', 'PROD', 'update_l1_calibration_database.py', 'horner', '$(LATCalibRoot)/TKR/LAT_BadStrips_64.xml', datetime.datetime(2022, 4, 26, 20, 43, 44, 121595, tzinfo=datetime.timezone.utc)) No action set. Would have run command: update metadata_v2r1 set vend=%s where ser_no = %s with values: ('2022-04-26 23:00:00', 1287) No action set. No rows changed. Last entry is: +--------+------------------------------------------+---------------------+---------------------+---------------------+ | ser_no | data_ident | vstart | vend | update_time | +--------+------------------------------------------+---------------------+---------------------+---------------------+ | 1287 | $(LATCalibRoot)/TKR/LAT_BadStrips_63.xml | 2021-11-02 13:40:00 | 2037-01-01 00:00:00 | 2021-11-02 16:32:38 | +--------+------------------------------------------+---------------------+---------------------+---------------------+ Calibration database successfully updated.

...

If you set the time you're asking about (with the -s or --start option) to after the start time you set, then it will show the new calibration file as the one the pipeline will chose choose (note in this example the tracker file was not updated).

...