This page describes the
Symptoms of the failure
On 27 Jul 2010 (Day 208), at 1:55 PM EDT, Anders Borgland wrote:
Starting with run 301753824 we have two problems:
1/
We do not see any signal in the high energy diode in tower 4, layer X1, column 4, + side. From Eric Siskind: "Whether the failure is within the diode itself or in the GCFE electronics chain (HE preamp or slow shaper) is currently unknown". You can read the whole thread here:https://www-glast.stanford.edu/protected/mail/datamon/4835.html
2/ While problem 1/ affects all (high energy) events in that channel, GCR events (4-range and zero-suppressed events) tickle FSW bug 1156:
https://jira.slac.stanford.edu/browse/FSW-1156
This means that about 10 events per run will fail in the decompression. Because of the way the Halfpipe works we lose the complete datagram for each of these events. Since a datagram contains about 110 events we are currently losing about 1100 events per run. This corresponds to about 2.5 seconds of data for each 90 minute run.
The FSW group have a fix for bug 1156 and will upload a new build asap.
Note that currently there is no failure mode in CalRecon so events from this channel is not treated in any special way. NRL is working on this.
It should also be noted that the problem was caught immediately by two separate parts of the Data monitoring. The automatic alarms caught both the missing datagrams and the missing signal from the diode. These runs are marked as 'GOOD' by the DQM shifter, but with a comment attached to them. Obviously we will have to live with the missing diode signal from now on.
Some of you will not have failed to notice the irony that it's GCR events tickling FSW bug 1156 (hint: SSC-258)
anders
What has failed?
Some text here.
Consequences of this failure
On 27 Jul 2010 (Day 208), at 2:15 PM EDT, J. Eric Grove wrote:
Additional clarification:
We do not see any signal in the high energy diode in tower 4, layer X1, column 4, + side. From Eric Siskind: "Whether
[]
Note that currently there is no failure mode in CalRecon so events from this channel is not treated in any special way. NRL is working on this.
What this means is that:
- Any photon that MISSES this one crystal is not affected. It is correctly and properly reconstructed.
- Any photon that deposits LESS THAN about 1 GeV in this one crystal is not affected. It is correctly and properly reconstructed.
- Any photon that deposits MORE THAN about 1 GeV in this one crystal has an incorrect energy and position measurement in this one crystal, and therefore has an incorrect reconstructed incident energy and direction. The level of error in reconstructed incident energy and direction is surely energy-dependent, and I don't have an estimate of the magnitude yet.
In the above sentences, "photon" means "any event that is not read out by Trigger Engine 4, i.e. any event that is not read out in 4-range, zero-suppressed mode". I used the word photon to focus the discussion.
The combination of cases (1) and (2) covers the overwhelming majority of photons in the LAT dataset, so most events are perfectly fine, but clearly we need to implement a fix for this particular failure in the code that reconstructs crystal energy and position.
Eric
Changes necessary to recon
As of July 2010, CAL recon code contained no mitigation against failures, despite our having discussed adding such code for the last 10+ years. This failure requires that we do something.