Thoughts about CAL data monitoring are collected here
Compare to thoughts I added to https://confluence.slac.stanford.edu/display/DC2/Monitoring
on 5 Oct 2006.
Note: I did a lot of rearranging of this section on 13 Mar, but I've added only a few items, mostly in last two sections before the email.
Daily standard plots
This includes a lot more than just CAL info, but there are so many parallel features among the instruments that we might as well list them all.
I haven't specified the sampling time for these strip charts. Not all will be the same. Some could be done every 10 sec, but others require longer summing. TBD TBR.
Environment monitoring
Strip charts of temperatures, say a couple grid points, mean of inner TKRs (top cable temp), mean of corner TKRs, mean of inner CAL baseplates, mean of corner CAL baseplates, a couple ACD temps. 5-min averages are fine.
Strip chart of LAT hardware Configuration state. I'm not sure quite how to plot this, but plot identity of SIU and EPUs powered, identity of power feeds, .... I'll flesh this out later.
Strip chart of LAT Mode: Quiescent, Physics, SAA, TOO, AR, HOLD. Again, not quite sure how to plot this, maybe a color-coded line.
Strip charts of vertical cut-off rigidity or McIlwain L parameter (not very interesting when the LAT is stationary).
Strip charts of attitude information: pitch angle WRT local zenith, RA and dec of Z, ...
Strip charts of location information: s/c RA and ec, s/c lat and lon, ...
Event classification information, background rates
Strip charts of livetime, deadtime.
Strip charts of trigger sent, discard, prescale, and deadzone rates.
Strip charts of TKR, CAL-LO, CAL-HI, and CNO trigger req rates from the GEM Condition Summary word. In addition, strip chart of TKR && ROI (mostly particles) and TKR && !ROI rates (mostly not particles). Maybe do all 8 trigger sources.
Strip charts of TKR, CAL-LO, and CAL-HI trigger req rates from each tower, taken from the GEM trigger vector. Strip charts of CNO trigger req rates from each board.
Strip charts of trigger engine rates, reconstructed from dowlinked data that satisfies the event filters. I realize this is biased info.
Strip charts of rates passed by filters (GFC, MFC, HFC, and DFC). It will take some effort to reconstruct this from the filter bits, I suppose. Note that the DFC has two instances. The first passes periodic triggers, so perhaps its strip chart isn't interesting. But the second passes the "unbiased" TKR && ROI && (don'tCare) trigger rate prescaled by 250:1, so it'll show ~20 Hz average, with good modulation through the orbit.
Detector information
Strip charts of total TKR, total CAL, total ACD occupancy rates (e.g. strip hits per unit time). Same plots as TKR EMI/EMC test, now in pipeline.
Subpages for strip charts of TKR and CAL occupancy rates by Tower. Same plots as TKR EMI/EMC test, now in pipeline.
Strip chart of sigma of overall CAL LEX8 and HEX8 pedestal width distributions calculated in Item 1 of 24 Jan 2007 email message below. Measures average noise and its variation through the day, through time.
Note: see comment on time binning below.
Strip chart of mean and rms of the 3 ratio centroids (P/M, P/p, M/m) for CAL calculated in Item 2 of 24 Jan 2007 email message below. Measures global PDA bond stability through time.
Note: see comment on time binning below.
Strip chart of median raw CAL energy sum for events with non-zero CAL energy sum.
Monitoring CAL quantities in the first LPA runs
The LISOC will work to keep a correct, public copy of the timeline in slacspace. The times and dates I've written here will probably not be right, so please go to slacspace to see the real timeline. You'll need your Windows password, and remember your username is preceded by "SLAC\" (e.g. I'm SLAC\grove). The URL is https://slacspace.slac.stanford.edu/sites/ISOC/MP/Lists/LEO%20%20Activation%20Timeline/Science%20Ops.aspx
You'll be able to do these tasks with a combination of monitoring plots from LISOC and s/w written by Zach and Sasha, described here.
A useful overview of the plots is at https://confluence.slac.stanford.edu/display/ISOC/Review+of+CAL+Data+monitoring .
In basic form, here's the list of tasks.
0. For each task below, record the CAL AFEE temperatures
For each task and each dataset below, it's important to understand what the AFEE board temperatures are. There are reported in the LISOC data monitoring. See instructions in the next section. Compare to the AFEE temperatures during Observatory cold thermal balance, when the LAC calibration data were acquired.
*The CAL AFEE temperatures can be displayed on the Telemetry Trending -> Health and Safety -> Temperature -> Calorimeter and pick the LHKT#CALAF## value you need where # is the tower number (in Hex) and ## is 0 = X+, 1 = Y+,
2 = X-, ..... Beware: this page looked different last week so the way to access these variables might change too.
If you want to grab the temperatures with a command line go to the ISOC utilities and use the MnemRet.py script.
1. From the initial LPA runs, determine whether there are any serious problems with zero suppression or CAL triggering.
The useful datasets are
a. the runs we start via PROC requests on L+15. They'll be conSciOps_noCal, calibOps, and nomSciOps
b. the short 5-min runs of all LPA configs on L+16, including all of the fleCalib*, fheCalib*, and lacCalib* configs.
c. the longer calibOps and conSciOps_noCal runs on L+16.
The tools here are mostly LISOC monitoring plots.
From those runs, we need to know absolutely ASAP whether there are any problems with LAC, FLE, or FHE. We need that information at the MOC within hours after the data come out of the pipeline, if we're to have any chance to fix them before the FLE and FHE timing-in data are acquired. Sorry about that. Please communicate this information to the Shift Coordinator (Rob or Eduardo or ?) at the Mission Support Room at SLAC absolutely as soon as you can, so that he can relay to us in the MOC at GSFC.
Phone at LISOC MSR: +1 650-926-7900, 7901, 7902
Phone near LAT stations at GSFC MOC: +1 301-286-0866
Go to the GLAST Data Quality Monitoring pages http://glast-ground.slac.stanford.edu/DataQualityMonitoring/.
Select an appropriate time interval, and click on the run ID.
Click on the "Expert" mode in the upper right corner of the left-hand panel. When you're in "Expert" mode, that word will show "Shifter", and when you're in "Shifter mode, that word will show "Expert", just to be confusing.
To find the number of trigger requests per tower in this run or sum of runs, click on
Root > Digi > GEM > TriggerVectors > CAL_HI map(tower)
Root > Digi > GEM > TriggerVectors > CAL_LO map(tower)
Of course, if one GCFE is hot, one of the towers will be significantly out of family.
To see whether CALLO or CALHI TREQ rate is too high, click on
(hmmm, I'm not sure what to do, yet)
To find the distribution of CAL hit occupancy, click on
Root > Digi > CAL > Num. logs hit (LAT)
Root > Digi > CAL > Num. logs hit (tower)
If zero suppression is bad, the minimum occupancy won't be zero. If it's really bad, the peak will be somewhere above zero. Find out which tower is causing the problem, or if all towers are.
To see a 2D histogram of the average hit occupancy per layer and tower, click on
Root > Digi > CAL > CAL Hit map (tower,layer)
or its complement
Root > Digi > CAL > Missed logs map (tower,layer)
Volunteers for this activity: Fred Piron, Eric Nuss, Benoit Lott, Sasha Chekhtman. Thierry Reposeur & Damien Parent, at SLAC. Dave Smith in Bordeaux. Veronique Pelassa. Please add your name.
2. From the initial LPA runs, "measure" the LAC, FLE, and FHE thresholds.
The useful datasets are a, b, and c above plus
d. the calibOps runs on L+18
e. the First Light Image runs of nomSciOps on L+19 to L+22.
The tools here are code from Zach and Sasha. See Zach's presentation.
Again, please don't wait until datasets d and e are acquired to start.
Volunteers for this activity: Berrie Giebels, David Sanchez, Sasha Chekhtman. Please add your name.
link to LAC, FLE, and FHE thresholds page
3. From the initial LPA runs, measure pedestals, look for temperature dependence.
This is really the first step of the calibGenCAL calibration process.
The useful datasets are a, b, c, d, and e above.
Volunteers for this activity: Aous Abdo, Berrie Giebels, David Sanchez, Sasha Chekhtman. Please add your name.
Pedestal Studies (comparison to cold TVac)
CAL+Peds+TVacPedestal Studies (comparison to Run r0236323982_v000)
4. From the LAC, FLE, and FHE calibration data acquired on L+23 and L+24, measure the LAC, FLE, and FHE threshold DAC calibration curves.
Here the useful datasets are the lacCalib*, fleCalib*, and fheCalib* acquisitions. Many of you have contributed to the analysis code for these data. See Zach's presentation.
Description of the simulated data for the CAL FLE/FHE threshold calibration has been added to https://confluence.slac.stanford.edu/display/ISOC/Detector+Calibration+Sequenceafter the item "Monitoring and control plots"
I'll have a better estimate for you shortly, but you have less than 3 days to analyze those data and deliver threshold DAC calibration curves. Isn't this fun?
Volunteers for this activity: Sasha Chekhtman.
General monitoring thoughts are below
Compare to thoughts I added to https://confluence.slac.stanford.edu/display/DC2/Monitoring on 5 Oct 2006.
Note: I did a lot of rearranging of this section on 13 Mar, but I've added only a few items, mostly in last two sections before the email.
Daily standard plots
This includes a lot more than just CAL info, but there are so many parallel features among the instruments that we might as well list them all.
I haven't specified the sampling time for these strip charts. Not all will be the same. Some could be done every 10 sec, but others require longer summing. TBD TBR.
Environment monitoring
Strip charts of temperatures, say a couple grid points, mean of inner TKRs (top cable temp), mean of corner TKRs, mean of inner CAL baseplates, mean of corner CAL baseplates, a couple ACD temps. 5-min averages are fine.
Strip chart of LAT hardware Configuration state. I'm not sure quite how to plot this, but plot identity of SIU and EPUs powered, identity of power feeds, .... I'll flesh this out later.
Strip chart of LAT Mode: Quiescent, Physics, SAA, TOO, AR, HOLD. Again, not quite sure how to plot this, maybe a color-coded line.
Strip charts of vertical cut-off rigidity or McIlwain L parameter (not very interesting when the LAT is stationary).
Strip charts of attitude information: pitch angle WRT local zenith, RA and dec of Z, ...
Strip charts of location information: s/c RA and ec, s/c lat and lon, ...
Here are a few more, added 10 Oct 2007:
Strip chart of Day/Night flag, so we know whether LAT was in the sun at each instant.
Strip chart of zenith angle to Earth's limb. Maybe also some measure of arc length of limb within nominal LAT FOV.
Event classification information, background rates
Strip charts of livetime, deadtime.
Strip charts of trigger sent, discard, prescale, and deadzone rates.
Strip charts of TKR, CAL-LO, CAL-HI, and CNO trigger req rates from the GEM Condition Summary word. In addition, strip chart of TKR && ROI (mostly particles) and TKR && !ROI rates (mostly not particles). Maybe do all 8 trigger sources.
Strip charts of TKR, CAL-LO, and CAL-HI trigger req rates from each tower, taken from the GEM trigger vector. Strip charts of CNO trigger req rates from each board.
Strip charts of trigger engine rates, reconstructed from dowlinked data that satisfies the event filters. I realize this is biased info.
Strip charts of rates passed by filters (GFC, MFC, HFC, and DFC). It will take some effort to reconstruct this from the filter bits, I suppose. Note that the DFC has two instances. The first passes periodic triggers, so perhaps its strip chart isn't interesting. But the second passes the "unbiased" TKR && ROI && (don'tCare) trigger rate prescaled by 250:1, so it'll show ~20 Hz average, with good modulation through the orbit.
Detector information
Strip charts of total TKR, total CAL, total ACD occupancy rates (e.g. strip hits per unit time). Same plots as TKR EMI/EMC test, now in pipeline.
Subpages for strip charts of TKR and CAL occupancy rates by Tower. Same plots as TKR EMI/EMC test, now in pipeline.
Strip chart of sigma of overall CAL LEX8 and HEX8 pedestal width distributions calculated in Item 1 of 24 Jan 2007 email message below. Measures average noise and its variation through the day, through time.
Note: see comment on time binning below.
Strip chart of mean and rms of the 3 ratio centroids (P/M, P/p, M/m) for CAL calculated in Item 2 of 24 Jan 2007 email message below. Measures global PDA bond stability through time.
Note: see comment on time binning below.
Strip chart of median raw CAL energy sum for events with non-zero CAL energy sum.
Subpages with strips charts of raw CAL energy sum by tower. Same calc as previous item, but by tower.
Post-recon end-to-end sanity checks (added by Dave Smith, 25 September 2007) (see also 2nd and 3rd slides of this presentation).
Extrapolate TKR track to CAL. For cleanly hit CsI logs, make histograms of dN/dE and fit with Landau. Histogram Delta X,Y of TKR, CAL positions and fit with gaussian. Store the lists of fit results. Easy to compare histogram of values with a template histogram ; or difference between current values and reference values (Monte Carlo or gold-plated certified reference data run), and thus flag outliers automatically. Easy to make a single summary page. This tests the RDB metadata database contents at the very end of the RECON chain. Dave S did this with his code during I&T, now that GCR.ROOT exists it is probably smarter to use that. We (Fred, and Dave Thierry Damien) will practice during Ops Sim1, then advise David Paneque how to best add some such to the monitoring variablesSubpages with strips charts of raw CAL energy sum by tower. Same calc as previous item, but by tower.
Daily (or orbit) sums or summaries
...
Added 6 Apr 07 Occupancy ratio summary. The intent is to find specific GCFEs for which the current occupancy has changed from their typical occupancy. See occupancy ratio summary discussion@Occupancy Occupancy Ratio Summary comment box below for further details.
Added 6 Apr 07 Pedestal noise ratio summary. The intent is to find specific GCFEs for which the pedestal width has changed from its typical value. See noise ratio summary discussion@Noise Noise Ratio Summary comment box below for further details.
Added 25 June 07 A set of 2D histograms designed to detect suspect CAL threshold settings. See Suspect CALLO and CALHI Finder comment box below for further details.
This section contains mail messages
...
For the SAA monitor ("trigger rate during last min before and after SAA"), I think the items I've listed in 3 cover that, and I really think we absolutely need continuous rate plots – - not just before/after SAA. Another thing that would be useful for the SAA monitor would be avg CAL hit occupancy: the SAA activates 128I in the CsI, which has a ~30-min beta decay with ~2 MeV endpoint energy. Since that endpoint is close to the CAL zero supp threshold, we should see CAL occupancy a bit higher right after SAA exit and decaying with 30-min timescale to the nominal occupancy. Again I guess we select the leaked events to calculate occupancy.
...
Here's a reminder of the contents of the CGRO/OSSE Daily Standard Plots
| Anchor | ||||
|---|---|---|---|---|
|
| Anchor | ||||
|---|---|---|---|---|
|
See "page 4" of the Daily Standard Plots here.
...
Note that by chance (ok, by design) I chose a day with a minor problem with the SAA boundary. See the sharp spikes in the "CPD" rates near 20 hrs. Comparing the times of those spikes with the heavy black lines in the "RIGID" plot (the heavy lines are ground-defined duration of the SAA pases) it's apparent that the Eastern edge of SAA needed to be extended a bit – - i.e. the rates were still high at the end of the SAA pass.
...