January 2024 GPS Outage

No, the GPS receiver had a maximum drift of ~milliseconds relative to the spacecraft clock of ~60 milliseconds at 12:40 UTC.  However, the spacecraft clock should have been drifting away from truth at a relatively constant rate during the outage, and at then end of the outage (when the GPS receiver was once again locked and effectively providing timing very close to truth) the offset was indeed ~18 microseconds.  I misread the exponent of the offset at 12:40, and confused the ~60 milliseconds with ~600 milliseconds.

From: eric.j.siskind@nasa.gov via the list <latopslist@glast4.stanford.edu>
Sent: Monday, January 29, 2024 2:22 PM
To: Cameron, Rob <rac@slac.stanford.edu>; maldera@to.infn.it; stefano.ciprini.asdc@gmail.com; datamonlist@glast4.stanford.edu; latopslist@glast.stanford.edu <latopslist@glast4.stanford.edu>
Cc: David A. Smith <smith@cenbg.in2p3.fr>
Subject: [EXTERNAL] [BULK] [LATOPS] RE: [DATAMON] about 12 hours (Dec. 25, 3:30AM-3:30PM UTC) of lost contact with satellites and related GPS alarms in related ( 727886207 727882166 727876478 727870789 727865101 727859412 727853724 727847949 727842527).

What is the source of the assertion that the time drift at the end of the outage was ~18 microseconds?  To me it appears that it drifted off by half a second, but I may be missing something.

From: Cameron, Rob <rac@slac.stanford.edu>
Sent: Monday, January 29, 2024 1:03 PM
To: maldera@to.infn.it; stefano.ciprini.asdc@gmail.com; datamonlist@glast4.stanford.edu; latopslist@glast.stanford.edu <latopslist@glast4.stanford.edu>
Cc: Siskind, E. J. <ejs@slac.stanford.edu>; David A. Smith <smith@cenbg.in2p3.fr>
Subject: Re: [DATAMON] about 12 hours (Dec. 25, 3:30AM-3:30PM UTC) of lost contact with satellites and related GPS alarms in related ( 727886207 727882166 727876478 727870789 727865101 727859412 727853724 727847949 727842527).

Hi Simone,

Your proposal for flagging is ok with me.

• Rob

-- 

Robert Cameron; Fermi LAT ISOC; SuperCDMS SNOLAB; 650-898-7620

From: Simone Maldera <maldera@to.infn.it>
Date: Monday, January 29, 2024 at 8:40 AM
To: Cameron, Rob <rac@slac.stanford.edu>, stefano.ciprini.asdc@gmail.com <stefano.ciprini.asdc@gmail.com>, datamonlist@glast4.stanford.edu <datamonlist@glast4.stanford.edu>, latopslist@glast.stanford.edu <latopslist@glast4.stanford.edu>
Cc: Siskind, E. J. <ejs@slac.stanford.edu>, David A. Smith <smith@cenbg.in2p3.fr>
Subject: Re: [DATAMON] about 12 hours (Dec. 25, 3:30AM-3:30PM UTC) of lost contact with satellites and related GPS alarms in related ( 727886207 727882166 727876478 727870789 727865101 727859412 727853724 727847949 727842527).

Hi all,

I am just adding some plots of the outage from the DQM.

In particular the SWHWUDLGPSSUBSEC variable indicates that at the end of the GPS outage the time drift was ~18microseconds.

If everybody agrees I  will set the GPS flag to "bad" and  DATA_QUAL=2  for these runs.

Simone

On 29/01/24 08:25, rac@slac.stanford.edu via the list wrote:

To follow up on this GPS outage on 2024-01-25: it was similar to a previous GPS outage that occurred on 2023-02-22.

In that case in 2022, a decision was made on how to flag the affected runs in DQM, as follows:

set the ft2 quality flag to 2 for the affected time interval (the same value that was used in the past for a timing issue).

We will also mark these runs as "good with bad parts" and set the GPS flag to "bad" in the run quality database.

Note: I recall that steps were taken by the FOT in the February 2023 GPS outage to minimize the drift and inaccuracy of the timing and position data from Fermi during the outage, which should have also presumably been applicable to the 25 January 2024 outage. But the 2023 data would not have been useful for fast pulsar studies, and so had to be flagged in DQM.

Other people will likely want to correct what I said above…

Robert Cameron; Fermi LAT ISOC; SuperCDMS SNOLAB; 650-898-7620

From: rac@slac.stanford.edu via the list <latopslist@glast4.stanford.edu>
Date: Sunday, January 28, 2024 at 7:09 PM
To: stefano.ciprini.asdc@gmail.com <stefano.ciprini.asdc@gmail.com>, datamonlist@glast4.stanford.edu <datamonlist@glast4.stanford.edu>, latopslist@glast.stanford.edu <latopslist@glast4.stanford.edu>
Subject: [LATOPS] Re: [DATAMON] about 12 hours (Dec. 25, 3:30AM-3:30PM UTC) of lost contact with satellites and related GPS alarms in related ( 727886207 727882166 727876478 727870789 727865101 727859412 727853724 727847949 727842527).

Hi Stefano et al,

Fermi’s GPS receiver had an outage on January 25. I attach a plot showing the number of tracked and visible GPS satellites seen/recognized by Fermi’s GPS receiver, clearly showing the outage.

The FOT at GSFC effectively performed a soft reset of the GPS receiver to clear the problem, by reloading tracking initialization data.

I realize that when you stated December 25 in your email title, you meant January 25.

The FOT used this event to refine its response procedures. So in the event of future re-occurrences of this sort of problem, hopefully the duration time of future outages might be smaller.

Rob

-- 

Robert Cameron; Fermi LAT ISOC; SuperCDMS SNOLAB; 650-898-7620

From: stefano.ciprini.asdc@gmail.com via the list <datamonlist@glast4.stanford.edu>
Date: Sunday, January 28, 2024 at 10:13 AM
To: datamonlist@glast4.stanford.edu <datamonlist@glast4.stanford.edu>
Subject: [DATAMON] about 12 hours (Dec. 25, 3:30AM-3:30PM UTC) of lost contact with satellites and related GPS alarms in related ( 727886207 727882166 727876478 727870789 727865101 727859412 727853724 727847949 727842527).

Hello

because of is not possible to continue to write single email
spamming woth the same kind of alarms, please
find here a summary of screeshot of alarma for all the run
interested by the lsot of contact with satellites
and the related GPS issue, that spans about 12 hours
form Dec. 25, 3:30AM  to about Dec. 25 3:30PM UTC
( runs 727886207   727882166   727876478    727870789   727865101
727859412   727853724   727847949   727842527).

Plot with alarmas in attachments for these runs cited above.

I am not so sure I can mark these run as good for this gps' reason
BUT i prelimi9nary marked them as good...

Sorry if there is some confusio in my email
but seems a busy dqm and tired weekend.
(also on shift as fa).
Cheers
SC