Thoughts about CAL data monitoring are collected here.
Date: Wed, 24 Jan 2007 17:06:39 -0500
From: J. Eric Grove <eric.grove@nrl.navy.mil>
To: Borgland Anders <borgland@slac.stanford.edu>,
Alexandre Chekhtman <chehtman@ssd5.nrl.navy.mil>,
Hascall Patrick <hascallp@slac.stanford.edu>
Subject: more CAL plots to add to digi report
Anders buddy,
One of the things we've talked about adding to the digi report is a pedestal summary. But we could also add something that would supplement the calf_mu_trend test in the CAL CPT. Are you still in charge of digi reports?
1. Pedestal, 4 histograms for every LPA run.
Accumulate 12000 histograms (1536 logs x 2 ends x 4 energy ranges) from periodic triggers!! For each log end, fit gaussians to the LEX8 and HEX8 histograms and accumulate the LEX8 and HEX8 gaussian sigma values into histograms. Discard the centroid because it's meaningless. For each log end, calculate the mean and rms of the LEX1 and HEX1 values in the 5 bins surrounding the modes, and accumulate the LEX1 and HEX1 rms values into histograms (and pretend those are gaussian sigmas). Discard the centroid because it's meaningless.
In the digi report, plot 4 histograms (LEX8 sigma, LEX1 sigma, HEX8 sigma, HEX1 sigma), each with 3072 entries, and let root calculate the mean and rms of the pedestal sigmas.
This gives an overall measure of CAL noise, and a sense of whether there are two channels or two hundred channels that are out of
family, for every run. It's similar to something we do in the calu_pedestals_ci test in the LAT/CAL CPT.
2. Optical gain, 3 histograms for every LPA run.
Accumulate 4500 histograms (1536 logs x 3 ratios) from all 4-range readouts that aren't periodic triggers. There aren't many of those events in ground 22x runs, but there are in ground 71x runs, and there will be many on orbit. The three ratios are
i. LEX1(plus face) / LEX1(minus face) = P/M ~ 1
ii. LEX1(plus face) / HEX8 (plus face) = P/p ~ 6
iii. LEX1(minus face) / HEX8(minus face) = M/m ~ 6
Note that each of those quantities is pedestal subtracted, otherwise the ratio doesn't make any sense. And each one needs a cut that rejects an event from this histogram if it's <50 ADC bins (to avoid binning errors). For each of the 1536 logs, fit three gaussians and accumulate the centroid into 3 histograms, one for each of the ratios.
Alternatively, you could do this in energy space from the CAL tuple, but I'll bet there's a causality issue with creating the recon report before the CAL tuple or something.
In the digi (or recon) report, plot the 3 centroid histograms (P/M, P/p, M/m), each with 1536 entries, and let root calculate the mean and rms of the ratio centroids.
This gives an overall measure of whether there were diode bond failures. It won't tell us which CDE is bad, and there are some
systematic problems with certain channels on each AFEE board (remember David figured that out while he was at SLAC?).
It won't add much run time to the digi reporting to calculate 16,000 gaussian fits, will it? 
Just a reminder of things to add to your queue.
Thanks,
Eric
Another message
From: grove@ssd5.nrl.navy.mil
Subject: Additions to digi report for CAL
Date: 17 July 2006 5:48:46 PM EDT
To: borgland@slac.stanford.edu, neil.johnson@nrl.navy.mil
Anders,
How about the following additions to the CAL plots for the digi report?
Average ped-subtracted pulse height (LEX8 data only, say)
Average energy (all ranges, but first-range-only for the 4-range readouts)
Average plus/minus ratio, pedestal-subtracted ratios please! (LEX8 only?, and be sure to put 100 ADC bin threshold on each end before
taking ratio)
RMS of plus/minus ratio (same qualifiers: >100 bins ped-subtracted, LEX8 only)
Range occupancy (i.e. 4 plots of fraction of readouts in each range, averaged over xtals in a layer)
Probably others....
Eric
Here are some thoughts about Trigger monitoring.
From: eric.grove@nrl.navy.mil
Subject: trigger monitoring
Date: 26 January 2007 9:47:32 AM EST
To: kocian@slac.stanford.edu
Hi Martin,
I just read through your trigger monitoring slides on confluence.
The real reason I wrote is that I have a few ideas of rate monitoring, so these are comments to pp 5 and 6.
1. Rate v. geomag latitude is fine, but so is rate v. McIlwain L parameter. And I guess what I'd also really do is create a strip chart of the McIlwain L value to run in parallel with the trigger rates.
2. We need to monitor and display on timescales shorter than rate per day or rate per orbit. For CGRO/OSSE, we used 4-sec and 16-sec rates (depending on the data src, but mostly 16-sec rates) in various energy bands and detector components (e.g. spectrometer, shield, and particle anticoincidence det were shown separately). That's because there are interesting celestial and terrestrial phenomena occurring on much shorter timescales that trigger monitors might see. Two examples:
a. solar flares: gammas, particles, geomagnetic disturbances from solar particle events, etc. Features have timescales of seconds, with total event durations of minutes to hours.
b. particle precipitation events, including occasional events from a couple strong radio transmitters (e.g. there's one in Australia). Total event durations are seconds to ~1 minute. GBM will likely interpret these as GRBs.
I'd make strip charts of various trigger rates with, say, 10 or 15-sec binning, with total duration of each plot either the total time span of data in a downlink or a 24-hr day. Heck, probably both. Each downlink would create its monitor plot, and then maybe they could summarized at the end of the day with a single, merged plot.
3. The trigger rates for strip charting that come to mind are below, and I guess I'd derive say 15-sec avgs of each for plotting. I'm listing a lot of plots so presumably we need some hierarchical way to view them.
a. Sent, Discard, Prescaled, and Deadzone rates from the GEM LRS counters.
a. the livetime or deadtime fraction.
b. the 16 trigger engines, derived from the events leaked either by PFC or DFC filters.
c. Bill's favorite monitor of particles, the TKR && ROI && (don'tCare) rate, derived from the leaked events.
d. each of the 8 GEM trigger primitives, derived from the leaked events.
e. each of the individual tower TKR, CALLO, and CALHI trigger rates from the trigger vector, again derived from the leaked events.
f. rates of events accepted by GFC, MFC, HFC, so that'd crudely be the rates of gammas, MIPs, and heavy ions, this time derived from the the events passed by those filters, of course, rather than by leakage.
I guess that the leaked events are the best sample to use to derive rates. Need to think about that.
I like your "Fraction of chg particles over total rate" idea. Maybe ratios of some of these other quantities would be good too.
For the SAA monitor ("trigger rate during last min before and after SAA"), I think the items I've listed in 3 cover that, and I really think we absolutely need continuous rate plots – not just before/after SAA. Another thing that would be useful for the SAA monitor would be avg CAL hit occupancy: the SAA activates 128I in the CsI, which has a ~30-min beta decay with ~2 MeV endpoint energy. Since that endpoint is close to the CAL zero supp threshold, we should see CAL occupancy a bit higher right after SAA exit and decaying with 30-min timescale to the nominal occupancy. Again I guess we select the leaked events to calculate occupancy.
Come to think of it, maybe strip charts of ACD, CAL, and TKR occupancy are useful in general. OK, it's not trigger rate, but it's related to the sources of triggers.
I'd also overlay times of ITS (Immediate Trigger Signal) messages from the GBM, LAT burst alert messages, etc. And indicate the intervals of LAT pointing, whether that means just the pitch from the zenith while we're doing sky survey or the intervals of three-axis pointing (the point there is that the particle exposure is a function of s/c attitude too).
I'll dig up the email I sent to Eric Charles with the daily monitoring plots we used for OSSE, and I'll forward that to you.
Eric
Here's a reminder of the contents of the CGRO/OSSE Daily Standard Plots
See "page 4" of the Daily Standard Plots here.
See "page 5" of the Daily Standard Plots here.
From: grove@ssd5.nrl.navy.mil
Subject: Some GRO daily monitoring plots
Date: 3 November 2006 7:15:22 PM EST
To: echarles@slac.stanford.edu, borgland@slac.stanford.edu
Cc: eduardo@slac.stanford.edu
Eric,
Here's a teaser of the OSSE on-orbit daily standard plots, sent to you without enough explanation. We made a series of plots in the production data processing for every single day of the 9-year mission as part of the data integrity and instrument health monitoring process. Both plots are for day 91/271, i.e. the 271st day of 1991, the entire day. On "page 4" you're seeing count rates in energy bands from detector 4 of 4. On "page 5" you're seeing count rates in ancillary particle detectors. At the top of both pages is a context timeline, with geomagnetic rigidity, marks for SAA times, GRB messages from BATSE (the equivalent of GBM, from the same guys at Marshall), marks for the "primary" and "secondary" sources in each orbit (OSSE viewed two targets per orbit, nominally on complementary sides of the Earth).
Look at page 5.
The SAA appears on page 5 in the CPMPR and CPMEL (Charge Particle Monitor Proton and Electron detector) time series. See that 6 of the 16 orbits each day have a significant SAA passage, with 2 relatively modest SAA passes. Note that they are marked also by the bold bars in the Rigidity time series. BTW, the CPM was a small plastic scintillator (3/4" diameter, 3/4" long) turned on at all times.
Note from the CPD$R* plots (the Charge Particle Detectors 1 through 4) that the orbital particle rate modulation is bigger on non-SAA orbits than on SAA orbits, i.e. the particles that aren't trapped in the belts (i.e. aren't in the SAA) are more strongly modulated on those orbits that don't pass through the SAA. This is an interesting consequence of a 28 deg orbit from an Eastward launch. BTW, the CPD was a plastic scintillator paddle, about 24" in diameter, over the aperture of each OSSE spectroscopy detector (4 spec detectors, so 4 plastic CPDs) used in the trigger logic as an active veto.
Note that by chance (ok, by design) I chose a day with a minor problem with the SAA boundary. See the sharp spikes in the "CPD" rates near 20 hrs. Comparing the times of those spikes with the heavy black lines in the "RIGID" plot (the heavy lines are ground-defined duration of the SAA pases) it's apparent that the Eastern edge of SAA needed to be extended a bit – i.e. the rates were still high at the end of the SAA pass.
Look at page 4.
Here you see the same Rigidity panel, plus a bunch of other rates associated with OSSE spectroscopy detector #4.
The first panel below the rigidity is SHIELD$R4. These are total veto count rates in the four, thick CsI shield segments that surrounded the main spectroscopy detector. The threshold was about 10 or 20 keV, as I recall.
The DTL$R4 panel shows the deadtime in the low-energy channel. I've forgotten the units, I'm embarrased to say.
Next is the neutral particle rate (mostly utter nonsense). The OSSE detectors were NaI-CsI phoswich detectors, with the CsI(Na) acting as an active shield to the rear of the NaI(Tl), which was used as the spectroscopy detector. The detectors were fairly massive: 13" diameter, 4" thick NaI and 3" thick CsI (or maybe I have the Na and Cs thicknesses backwards, it's been a while). From the pulse shape, one can deduce the depth of interaction (or at least deduce whether the scintillation came from NaI, CsI, or a mix). Neutrons have a different pulse shape than gammas, electrons, or protons, and this channel was meant as a neutron monitor. It's heavily polluted with overspill from real gammas, so really all you were looking for here was a big, huge spike above the wandering trend. Ignore it.
The bottom 5 panels are counts per 16 sec (I believe those are the right units) in each of 5 energy bands. For example, the bottom panel ".05 C/4" is the rate of 50-100 keV photons in detector 4, counts per 16 sec. Note the large exponential decays in the bottom three bands, i.e. up to the 300 keV to 3 MeV band. This is the decay of iodine-128 activated in the SAA from the 127I in NaI and CsI. It's a beta+ emitter with a 1.5 or 2 MeV endpoint and a 30ish minute lifetime. Note that we're saved a little from this activation in GLAST by having the CAL zero suppression at 2 MeV, and by the fact that each CsI xtal is a lot smaller than the OSSE phoswich.
In the 3-10 MeV and >10 MeV bands, note the orbital modulation again. Obviously it's not the gammas that are modulated here, but the particles. You're looking here at prompt decays from passing GCRs and residual light.
Unlike in the GLAST band, in the nuclear gamma regime (say 50 keV to 10 MeV), observations are overwhelmingly dominated by local background. The instrument is a lot hotter than the sky. So to observe an astrophysical source, we pointed the collimated detectors at a source for 2 minutes, then scanned off source by a couple collimator attenuation lengths (say 5 deg), then chopped back to source, then off, etc etc etc for 9 long years. There were only a handful of sources that were visible in these 16-sec samples (Crab nebula, ~6 BH binary transients, and one or two neutron star transients, as I recall). So when we saw the 2-minute chopping in these standard plots, we knew we would detect an extremely bright source in the detailed spectroscopy analysis. The other dozens of OSSE sources were visible only in incoherent sums of source-minus-background over one or two weeks of observing.
Eric