...

MOOT Key Mismatch between on-board LAT and pipeline expectation

From: https://www-glast.stanford.edu/protected/mail/opsprob/10894.html

1) Change the alias and mootkey in glastops_acqsummary (Oracle)

• for SQL commands, see: https://www-glast.stanford.edu/protected/mail/opsprob/10899.html

2) Change the config_fk (mootkey) in mood's Acq_summary (mySQL)

This will cause an error in the logs like:

MootSvc FATAL Hw key 8517 in data does not match hw key belonging to Moot Config #3063

This has happened in 2023, 2017, and 2013 due to a problem with an update to the hot strip mask on the satellite, e.g., the change was reverted but the pipeline was expecting the MOOT key for the new configuration.  Following what was first done in 2013, the fix is to update the databases to change the MOOT key for the failed runs back to the old value and rollback the failures. Other stuff mentioned in the original thread doesn't seem necessary.

1) Change the moot_key in glastops_acqsummary in the Oracle database.

• Setup an ISOC environment terminal and then run sqlplus \@isocflight to access the database.  You will need standard pipeline maintainer permissions.
• For SQL commands, see:
• It's straightforward to do for someone with write access to the MySQL database.
• for SQL commands, see https://www-glast.stanford.edu/protected/mail/opsprob/1089810899.html

>>>>>>>> These will allow the analyses to run
3) Disable the MootSvc check with MootSvc.ExitOnFatal = false

• see: https://www-glast.stanford.edu/protected/mail/opsprob/10905.html

>>>>>>>> Otherwise, the FATAL Hw Key error *will* occur
Then repipe.

...

2) Change the config_fk (mootkey) in MOOD's Acq_summary in the MOOD/FMX MySQL database.

• To access the database, you will need to get a password from the secret/fpd/fermi/isoc vault.  Richard may need to put in a ticket for you to get permission. In 2023, we used Steve Tether's account (tether) since the "magician" account is overpowered for what we need (it's basically the super-user account).
• For SQL commands, see https://www-glast.stanford.edu/protected/mail/opsprob/

...

• 10898.html.

3) Rollback the affected deliveries

• If they are in a Failed state, you can just do a standard rollback.

• If findChunks is stuck, then bkill the process and wait for the reaper to get it. Most of them should then auto rollback without intervention.

History

• The MOOT Key Mismatch problem occurred on 2023 July 18. See the Fermi news item for 2023 July 20 at https://fermi.gsfc.nasa.gov/ssc/library/news/
  • 15 LPA runs from the LAT had to have their MOOT keys manually fixed in Oracle and MySQL before they could be processed correctly through L1:
    • 711402686, 711405073, 711411046, 711416992, 711422925, 711428850, 711434773, 711440537, 711446088, 711451782, 711457476, 711463169, 711468863, 711474557, 711478257
• The MOOT Key Mismatch problem occurred on 2017 November 30. See: https://www-glast.stanford.edu/protected/mail/opsprob/10893.html

...