This page contains information concerning tests of the absolute times associated with GLAST LAT events.
Motivation: Important satellite missions, amongst which are USA, Chandra, XMM, and CGRO, have had problems with the absolute times assigned to events. See the anecdotes at the end of the page. The 10 uS timing precision required for the GLAST LAT is _essential_for good pulsar science. In addition to extensive testing of the different links in the timing chain, we have proposed an end-to-end test which will be the main focus of this page.
(30 May 2022) NUSTAR's on-orbit timestamps were temperature dependent, and varied by 2 or 3 ms. Now corrected for in Bachetti et al. 2021, with 60 microsecond residuals.
Timing Calibration of the NuSTAR X-Ray Telescope https://ui.adsabs.harvard.edu/abs/2021ApJ...908..184B/abstract
Today I also learned that there's something called https://iachec.org with a sub-category called https://iachec.org/timing/
IACHEC – International Astronomical Consortium for High-Energy Calibration
that points you to https://wikis.mit.edu/confluence/display/iachec/Timing where they don't seem to be aware that Fermi LAT timing is better than just our Crab-radio offset from 2011.
(April 3, 2008) -- added https://confluence.slac.stanford.edu/display/CAL/Comments+in+JIRA+GRINF-38
(Saturday 27 October 2007) Post Dynamic Testing CPT -- Eric Grove just spent the week in Arizona. This might be the last time we use the NRL scintillator plus Bordeaux GPS setup! Two key things are new: 1) Eric Siskind's telemetry analyses have evolved to the point where we now know pretty clearly exactly how the clocks have been performing before looking at the time differences. Siskind's predictions don't tell us if the absolute time is right. But they do tell us if the system is performing as expected, and our experience is that when performing as expected, the times are right. 2) the test procedure is now much much smoother than in the past -- Siskind's prescriptions of what should be done (with priceless input from me & Grove of course) combined with Grove's masterly proc writing skills have made the whole process fast & smooth. I've been getting the data analysed within 12 hours of the runs, thanks to Anders' et al marvelous pipeline making the SvacTuples so reliably.
One little hitch: I subtract 2 seconds from the Bordeaux GPS times to match the LAT times, but it should only be -1 to make the GPS UTC line up with MET.
Anyway... standard procedure is a) 15 minutes of muons with GPS lock ; b) 15 minutes where we've commanded the satellite to require more GPS satellites than it can see ; c) 15 minutes where we disconnect the cable from the satellite GPS antennas. This is repeated for the two redundant sets of spacecraft and LAT electronics. See the eLog for the run numbers on the plots for additional details.
The result is sub-microsecond agreement between LAT and standalone GPS times, with a sub-microsecond rms, when you have GPS lock. And drifts less than 2 or 3 microseconds in the 900 second runs for "unlocked" runs, compared to the requirement of 1 microsecond
These six plots are the two sets of three runs acquired this week. I was at Jodrell Bank radio telescope for LAT pulsar timing in the middle of it all -- phew!
(Monday 20 August 2007) INTEGRAL and XMM have had timing problems at the 300 us level since a MOC software upgrade in June 2005 caused them to tend to use predicted orbital positions instead of actual positions. While GLAST won't suffer from a problem exactly like this -- our times and positions are derived quite differently -- it is sobering to see, once again, that accidents do happen. Here is the link to a presentation summarizing their issue. (provided by Lucien Kuiper of SRON.)
(Friday 17 August 2007, still Montreal): Eric G and Patty S were up all night in Arizona -- it took 9 hours of perserverence to collect one hour of good muon data. In particular, GPS lock didn't get recovered the way it was thought to be recovered, and a whole crew is looking into that. BUT THE POINT IS that these B-side data LOOK REALLY GREAT. Here are plots from the three runs: 30 minutes with GPS lock 077015632, and 15 minutes each with GPS unlocked twice successively, 077015633 and 077015634.
(Sunday 12 August 2007, from Montreal): Runs '155773,4,5,6 were supposed to test the B-side, but the s/c GPS and clocks were mis-configured and the data are ugly. However, runs '15593,4,5,6 were repeats of side A and not only did we once again see good dates properly jammed to MET in run '15593, but we also saw +/- 2 microsecond precision for UNlocked data. Another important milestone. Uh... actually '93 is only good for the last 100 seconds since it takes a good 1000 seconds for the UDL to settle down to the right time after the GPS is turned on. Tomorrow, Eric G returns to Phoenix and, now that a broad audience agrees on how to configure and what to test, side B data will be taken once again. We expect to confirm that locked and unlocked dates are good, and that the B-side is also good.
(Wednesday 25 July 2007) Spacecraft Flight Software got upgraded a week or two ago, after EMI but before the CPT. LAT FSW went to version 1.0 about the same time. Eric G and Patty S went to Arizona, and after a long, hot, trying week of delay acquired NEW MUON TIME TEST DATA. Run 077015287 is GPS lock, '288 is without GPS lock. With GPS lock, things look truly peachy: look at SASSySC15287.gif.When GPS is locked, LAT timing looks swell (nota bene I had to subtract 1 second from the Bordeaux VME GPS times to get a match -- the s/c "jamming" procedure at s/c power-up is still not perfect...).
On the other hand, without GPS lock, we see a drift of 85 ns per second, compared to the requirement of <+/- 10 ns/s, as shown inSASSySC15288.gif. Eric Siskind and I have had numerous e-mail exchanges, we think that the UDL software algorithm that calculates the average PPS interval (in 100 ns UDL ticks) could maybe be improved a little... Anders, Pat Hascall, and Bryson Lee have been making very valuable contributions.
Furthermore: s/c telemetry shows the GPS flag bit changing state between the two runs, but in the LAT root files the bit is 0 for both runs. This is probably due to the fact that v1.0 LAT FSW was running, but the ground decompression code was still at v0.9.0, and this will get cleaned up shortly. There does however remain some confusion about the definition of the GPS lock bit at the s/c level that still needs to get cleared up.
Drew Roberts was taught to run the muon telescope by Patty & Eric, and they acquired run '15349 for practice. It is side 2, GPS lock. It looks wunnerful: SASSySC15349.gif .
Preliminary conclusions: the rock bottom bare minimum for pulsar science now seems to be in hand ; some continued turbulence regarding operation during loss of GPS lock is still improving.
SpaceCraft and MeritTuple times are indeed MET not UTC: Discussion around https://jira.slac.stanford.edu/browse/PULS-31lead to clarification of the times that come from telemetry, pass through the MeritTuple, and wind up in the FT1 file. The times will not have leapsecond jumps. At present it seems that they will be synched to UTC when the S/C gets powered on for the last time, in which case they would be one second off of TT (because of the 12-31-2005 leapsecond) although that could get cleaned up between now and launch. During the exchanges, Julie sent two PowerPoints and a draft .doc which are illuminating, so I archive them here: PowerPoint, better PowerPoint, and rough draft text (11 April 2007).
PASS/FAIL guidelines: the document at ftp://www.cenbg.in2p3.fr/astropart/Smith/Pulsars/SASS/MuonTimeTestProcMarch07.pdfis a 4 page "streamlined" version of the November and February test reports, intended to spell out more clearly & briefly to a test operator or busy engineer just how the tests are performed and what the criteria for success are (3 April 2007).
Here is my presentation to the Calibration & Analysis face-to-face meetingat GSFC on Monday, 26 March.
Click here for the datasheet of the Viceroy GPS receiverused by GLAST.
Norman Rioux sent an e-mail message outlining prospects for mitigation. Tuesday, 20 March we had a long telecon discussing GD's proposed fixes, more complex than just inverting the PPS signal polarity.
Here is my presentation to the Friday 16 March Sci Ops VRVS meeting, at ftp://www.cenbg.in2p3.fr/astropart/Smith/Pulsars/SASS/SOmeetMarch2007.pdf
At the end I asked about GPS diagnostics in the spacecraft telemetry, so Gary Godfrey sent me the list of the 263 variables being monitored.
First real test Eric and Patty took 8 data runs in Arizona (Thursday-Friday, 22-23 February 2007) . Dave analysed them in Bordeaux. For dT=(BdxTime-LATtime) we see an ugly sawtooth pattern ramping from 0 to -1 ms with a period of 290 seconds, when the GPS is locked, for both the satellite A and B sides. When the GPS is unlocked, instead of wrapping back to zero at -1ms, it ramps down to 10's of ms on the one side, and up to 10's of ms on the other side. Also, the LAT time is 1 second later than UTC. In short, something is very wrong.
Based on the information available to him, Eric Siskind favors the hypothesis that the PPS polarity out of the S/C GPS is inverted relative to the required input of the S/C UDL board (UplinkDownLink board, pronounced "oodle"). The information has been provided to General Dynamics, following a telecon between us and the NASA project office on March 7th. It is unknown at this time whether GD agrees with the diagnosis, and whether signal polarity is software configurable or would instead require a cable modification.
A rough report describing our findings is at ftp://www.cenbg.in2p3.fr/astropart/Smith/Pulsars/SASS/End2EndFeb07.pdf
Plans for the real test (Friday, 2 February 2007) This coming Tuesday, while we're at Stanford for the 1st GLAST Symposium, we (=Smith Dumora Grove Thayer) will have a telecon (at 6:30 in the morning, yipes) with folks from Goddard to define details of the final test. It could happen with s/c CPT beginning 21 February.
Summary report of November tests: * *http://www.cenbg.in2p3.fr/ftp/astropart/Smith/Pulsars/SASS/TimingEnd2End.pdfalso known as LAT-TD-08777-03.
More news from Arizona (Saturday, 18 November 2006) Unbolted the top muon paddle and moved it right up next to the LAT. Here is text with more details, and here are the paddle timestamp files. ( Wednesday 22 November) We have a lovely signal, with an interesting feature that we want to discuss with VSC experts after Thanksgiving. We've started a writeup. Instructions for setting up and running the test equipment are in this document.
News from Arizona (Wednesday, 15 November 2006) It took me 46 hours to get from Bordeaux to Phoenix, due to overbooking and an unforeseen night at Paris airport, but my luggage arrived intact and everything worked right out of the box. Patty and Eric had the muon telescope humming smoothly. This afternoon we moved it all into the HiBay clean room and took about 40' of muons along with LAT muon runs 077012947, and 8. The GPS antenna cable quickly became a non-issue due to the excellent help of Spectrum RF expert Alan Ames. I have updated Output.C to fix the conceptual bug that I mentioned in my presentation last Friday (see Seth's notes). The date lists from the Bordeaux VME are in files SASS8.dat and SASS9.dat which are in SASS.tar.gzalong with some other stuff. The 3 runs are still being reconstructed in the pipeline so analysis is on hold. SASS8 ran from UTC 23:00:31.06 to 23:12:23 and SASS9 ran from 23:15:27.6 to 23:42:37.3 . SASS9 thus corresponds exactly to 077012948 whereas SASS8 covers the last 9 minutes of '47.
The SLAC VME GPS card finally arrived! (Friday evening, 10 November 2006, at 5pm...) So after the SO meeting, took it downstairs, plugged it in, waited a few minutes for the antenna to pick up the satellites and... voila! Timestamps. Here's a sample output. More tomorrow...
Saturday-- Denis put both the new and old VME GPS cards into the crate, each with its own antenna, and triggered both of them in parallel. The microsecond digits always agree, while the 0.1 microsecond digits vary by a couple of counts. Furthermore, we repeated the "good to a second?" check using the bureau of standards web page. Denis made a lovely plot
that shows the two GPS's disagreeing by a whole microsecond before finally agreeing at the <200 ns level after 126 seconds.
The manufacturer recommends a patch on the new board. It's not on the one that SLAC sent us, Gregg tells me that it is on their other cards. Tech support told me that the patch is for 8-bit reads and writes (we only do 16-bits) and for interuppts (we interuppt the crate controller not the GPS card) and so I conclude that we can forget about the patch.
Everything looks good, we're feeling pretty comfortable. Next hurdle -- will my baggage make it past the airline?
Sci. Ops. VRVS meeting, 10 November 2006: Seth & Eduardo invited me to summarize the end-to-end test plan in 10 minutes at their Friday meeting. Here is the presentation. I refer to a nice macro written by Anders which is here in a .tar file.
Details of VME GPS modules and their antennas: In Bordeaux, we have two datum bc637gps modules. The manual is at this URL: http://www.symmetricom.com/media/pdf/manuals/man-8500-0019.pdf The 15-pin Trimble connector is described in Table 6-1, which is on the 54th page.
According to the VSC (Virtual SpaceCraft) documentation, LAT-TD-05601, http://www-glast.slac.stanford.edu/IntegrationTest/ONLINE/docs/VSC.pdf, the GPS module is a datum TTM637VME. There's a drawing on page 43, and, especially, the reference (#20) on page 4. Here it is. On page 6-1 (the 57th page) it says that if you opt for the version with an ACE GPS module then the front-panel antenna connector is an SMB, which connects to a GPS ANT. However, if you chose a Acutime 2000 Smart Antenna, then you have a DB-15 connector. That is, the SLAC VME GPS module could have been purchased with the same antenna as we have in Bordeaux.
We just (20 october) found a document that details the differences between the CELESTE GPS and the SLAC VSC GPS, it is http://www.symmetricom.com/Media/pdf/ttm-pcn/0401-01-TTM-PCN.pdf.
Details of VME crate and its controller: Eric wants to be sure to send the right kind of VME crate from NRL to SASS, so here is documentation on our crate controller and here it is for one of our typical crates. Nota bene we do not need the CERN Jaux connector.
Phone call with Eric, 13 October 2006: Spectrum has an 8-way splitter for the GPS cable on their roof. What comes down into the HiBay is a BNC-like twin-ax cable, connector part number Ponoma 4958. Spectrum says that they will adapt their connector to our needs. However, I reminded Eric that our Trimble antenna has a DB-15 connector (like for the VGA connector on your laptop) which has 4 lines for RS-422 serial communication, a 1 PPS line, ground lines, and +12 volts that supplies the active elements in the antenna itself. Hence, a BNC adapter ain't enough.
Here are some of the options:
- Eric says he's talked to some good guys but not yet to the local GPS guru. So there may be cables like ours.
- Eric says that the Spectrum guy says that re-radiating the GPS signals into the HiBay is an option. Golly, sounds cool. Then we could use our antenna indoors. Eric intends to find out whether this is really a possibility.
- The datum VME GPS that is in the SLAC VSC crate is very very similar to the two datum bc635's we have in Bordeaux, except that it connects to its antenna via a coaxial line. So presumably the active components are on the VME board and not in the antenna. We in Bordeaux believe that our acquisition code would read out the VSC GPS with little or no modifications. SLAC has spares, and Gregg Thayer says that loaning us one is plausible. We wish they'd mail us one, with one of their antennas too.
- Alternatively, the SLAC VSC GPS could be put into the same VME crate as the Bordeaux one, and the two P2 connectors can be connected with a flat jumper cable behind the back plane. This is how we have both of our VME GPS's see our single antenna here in Bordeaux. In the twin-Bordeaux case we believe that the slave VME module gets the current hour:minute:second via RS-422 through the P2 connector. If the Bordeaux VME card were married to the VSC VME card would their still be the same RS-422 exchange? Dunno, it's not clear in the documentation. Would like to try before coming to Spectrum.
Meanwhile, Denis has bought a DB-15 cable and is experimenting...
Minutes of telecon 11 October 2006: DS will go to Arizona the week of November 12th, to shake down and install the system. The NASA project office has approved our test. Spectrum has GPS cables and will get the right kind of connector for us. The muon telescope will sit right next to and slightly below the LAT. Thetest could be during the 2nd week of December.
Minutes of telecon 26 September 2006 : click here.
Current projects: Status of VME GPS in Bordeaux (this was up-to-date mid-september but now obsolete...) --
- We latch times with a trigger signal, and interuppt the processor to provoke readout with the same signal, and all that works nicely.
- We compare our times with http://nist.time.gov/timezone.cgi?UTC/s/0/java and are convinced that our dates are accurate at the 1 second level.
- We are hatching schemes to validate our dates to the microsecond level, either using a reference at an observatory or with a completely different GPS syste. A possible example is http://www.rfsolutions.co.uk/acatalog/GPS_Evaluation_Kit.htmlIt's been ordered, don't know when it will come.
- We got a spare VME GPS module from our former CAT now HESS colleagues in Paris it, we tested it, and it works.
Shopping list of things to prepare before going to NRL ; and then to Arizona End2End.docDraft of list of things to prepare. We could be ready to travel on very short notice now.
Absolute Time Calculation in LAT data:
13 October 2006 -- Found & fixed a bug this week! Here is what I sent to Anders. He and Warren seem to agree with me. The reason that Anders' algorithm wasn't quite right was that the pertinent lines in the DFI document were unclear, and a comment in LsfTime.h in digiRootData is wrong
From smith@cenbg.in2p3.fr Fri Oct 13 18:17:31 2006
To: borgland@slac.stanford.edu
( ... irrelevant stuff deleted ...)
I believe that I've been making a mistake in calculating my absolute times all these months, by a more than a second, for some large fraction of events. I used your May 26 slide 9 algorithm. What's wrong with that is that the GEM ticks counter doesn't get set to zero when the TimeTone gets updated.
Below is my piece of code: my old way commented, and my new proposition. How do you & Warren calculate absolute time. Do you agree with me?
float fraction ;
int RefDiff = TTCGemTimeTicks-TTPGemTimeTicks ;
if (TTPGemTimeTicks>TTCGemTimeTicks) RefDiff = TTCGemTimeTicks+(33554432-TTPGemTimeTicks) ;
if (RefDiff == 0)
// 75 in May became 51 in September, and 58 on Oct 8th.
/* 10 October 2006 -- I fear that the next line, taken from Anders' 26 May 2006 talk (slide 9), is wrong. The ticks don't get reset when a new pps comes in. Try the line after, instead.
fraction=TTCTTicks/(float)RefDiff ;
*/
int TDiff=(TTCTTicks-TTCGemTimeTicks) ;
if (TTCGemTimeTicks>TTCTTicks) TDiff = TTCTTicks+(33554432-TTCGemTimeTicks) ;
fraction=(float)TDiff/(float)RefDiff ;
( So, the absolute time is TTCTSecs+fraction )
Data Analysis:
Using FSW muons acquired at NRL on 27 May, 2006 (run 77005390) I practiced how I plan to analyse the data we'll acquire. Here is the output plot:
At left is cos(theta)sin(phi) vs cos(theta)cos(phi) for those of the first 13000 events having exactly one reconstructed track. (In svac.root syntax that would be AllMuDir->Fill(Tkr1EndDir[0], Tkr1EndDir[1]) ; ). I faked a corresponding list of VME GPS time stamps from the muon telescope by taking the LAT times and adding1 µS per event to them (an implausible and catastrophic scenario intended mainly to give some grist to my code). I then pretended that the scintillator paddles were at the place in space corresponding to the spot in the middle plot, which I reconstructed by requiring <100 µs between the LAT time and my fudged time. Note the presence of a few accidentals. Finally, in the plot on the right, I made a spatial cut and plotted the time difference (for a larger sample than for my training sample).
Since the May data had a fair number of strangenesses in the ContextLsf variables, I re-ran the code for some post-TVAC "door open" data from GMT 2006-09-08 11:13:43 (run 77010099). The detector is horizontal. The left-most plot is amusing, I didn't fill the other two. I can explain the funny structure -- can you?
As for the weirdnesses -- they didn't reappear. I spoke with Anders, he said "FSW hasn't changed between the two data sets, those bugs are known and sporadic, you were just lucky on the recent data". Last but certainly not least: here is my code in a gzipped tar file.
Background material:
Links to science requirement document ; to measurements made by J.G. Thayer ; to plot and text made by G. Godfrey ;
Presentation to Spectrum (General Dynamics) made at 22 August I&T meeting by Neil Johnson, prepared by David Smith.
Presentation made in SVAC IA meeting, 16 June 2006 by Smith. It builds on an IA presentation by Anders Borgland on 26 May. The thread continued with presentations by both Anders and Warren Focke on 30 June ; and Warren again on 14 July. You can find all these at the IA meetings page, http://www-glast.slac.stanford.edu/IntegrationTest/SVAC/Instrument_Analysis/Meetings/agenda.html
Link to Bordeaux DAQ note that was prepared for the Beamtest workshop.
Anecdotes: Here are timing failures on six major missions.
USA: The GPS kept getting stuck once on orbit and had to be reset a few times a day. The reason was that sometimes the satellite would go through GPS beams so intense that it confused the receivers. Furthermore, the speed with which the satellite would move relative to GPS's took it far from the design-regime considered for ground-based GPS's. Note that our proposed GLAST test would not have caught this problem. (Thanks to Michael Lovelette for this story).
XMM: Jean Ballet tells me that two years elapsed before absolute phases were really right, and that there were a series of problems. Recall that XMM has a CCD -- it's not one photon per event. Natalie Webb of the CESR-Toulouse kindly provided me with this, Proc. SPIE 5165, 85-95 (2004). XMM does not use GPS times in orbit (maybe because the eccentric 48 hour orbit took it too far away? I dunno), although the ground-stations did. OBT=On Board Time is set in a way more akin to what CGRO did. The problems were all electronics related, as I understand it. A table lists the 5 different kinds of anomalies.
INTEGRAL: see "20 August 2007" comment towards the top of this page. Orbital imprecisions, due to ground software shared by INTEGRAL and XMM, caused 300 us problems.
CHANDRA: For one of the two instruments on board, the HRC, the time stamp of a given event was that of the previous event. On board filters remove events, so obtaining the right date for a given event was impossible. The solution was to reduce the trigger area significantly, so as to reduce the event rate, so as to allow transmitting all events to the ground. This story documented in e.g. S. Murray et al, ApJ 568:226-231 (2002) and references therein.
Compton GRO: This was in the days before GPS. An on-board clock was set to absolute time using a reference like the one at the Bureau of Standards in Colorado, sent up to the instrument keeping close track of transit times. Events were assembled into packets on board, and the packets were grouped into a "major packet", to which a time stamp was afixed. These packets were sent to the ground by telemetry.The problem was that the time stamp was from the preceding packet! And so the time was off by more than a second. At first the engineers didn't quite believe the scientists, and the scientists weren't confident enough (yet) to be pushy. Further, once the time stamp was right, there was still an absolute phase problem, because the dispersion measure of Vela had changed by more than a millisecond (!) since the days of SAS. (Thanks to Dave Thompson for this story). Thanks further to Dave for digging up this old e-mail that details the story further.
ROSAT: Excerpt from http://www.mporzio.astro.it/~gianluca/phdthesis/node28.html , provided by Olaf Reimer: " A problem was also found in the timing of individual events, due to the processing software (Briel et al. 1994). The origin of the problem was the spacecraft clock reset which followed the spacecraft tumbling incident of 1991 Jan. 25. All PSPC data after that time are affected. The problem leads to relative shift of 1s between adjacent PSPC events. Although the problem is not cumulative, throughout the data there may be relative time shifts of the order of 1s. The effect of the problem is the presence of spurious power spectrum peaks on second and subsecond time scales. Unfortunately, there is no algorithm for correcting bad data. However data after the second revision (Rev2) processing are corrected."