Page History

...

It includes some discussion of mitigation of the failure of the HE channels of a single GCFE.  There are 3072 GCFEs in CAL, two for each of the 1536 xtals.  The problem occurred in the run starting at MET = 301753824, which is MJD 55402.52108796, or 2010 Jul 25 at 12:30:22 UTC, or 2010 day 206 at 12:30:22 UTC.  For more details and related discussion, see ?CAL July 2010 channel failure.

Changes necessary to recon

CalXtalRecAlg and related objects will be changed for Pass 8 (and perhaps earlier) for modify the xtal response in case of failures and for saturated readouts.

Crystal status word

We are adding status bits to xtal recon to indicate that the energy and position information returned by a xtal have been modified, along with some information about why the modification was necessaryAs of July 2010, CAL recon code contained no mitigation against failures, despite our having discussed adding such code for the last 10+ years.  This failure requires that we do something.  The intent here is to fix the reconstruction of events that hit this a xtal at the level of xtal recon so that the failure and fix are transparent to downstream energy reconstruction and clustering.  We're adding status bits to xtal recon to indicate that the energy and position information returned by a xtal with compromised readout have been modified.  Any reconstruction  Reconstruction code that follows xtal recon could make use of those bits to accept/reject that xtal or modify higher level recon algorithms, if the authors and architects of that code wish.On 4 Aug 2010 (Day 216), at 4:59 PM EDT, J. Eric Grove wrote:

All,

We at NRL had a very productive discussion with Tracy and Leon last week about how to mitigate against the failure in a CAL channel on 25 July. This email fleshes out a proposal for a new calibration DB quantity, how it'll be used, and requests your comments. And it requests that we start the process of adding this new calib quantity to the DB.

Over the years, we've discussed having a status word to indicate known h/w failures, and now we have a good reason to implement it. I wish we'd done it before. Our CAL online s/w used an "exception list" XML file to document failures or out-of-family conditions, and this is similar – but it's not XML.

Each of the 1536 xtals will have a 32-bit status word indicating its hardware and configuration status, and that 1536-element array will be stored as a calibration database quantity. The initial value of the status word for each event comes from the calibration database. In addition, some of the bits in that status word are used by xtal recon to indicate whether the energy and position calculated by xtal recon for a given event are good, need to be corrected, or have been corrected for the appropriate failure mode indicated by the hardware and config status bits.The bit map for the status word is given at the end of the email. Your comments or proposed modifications to the word are welcome.

The status word array needs to have an epoch of validity, like all other calib quantities. Until this failure, the status word for each xtal end was 0x0. Now we have one xtal with two bits set, the bits that indicate bad plus-face HEX8 and HEX1, and that condition began on 25 July.

...

(rev. 16 Feb 2010, Pass 8 discussion Grove, Bruel, Baldini, Usher)

Algorithms in CalXtalRecData

Algorithms for correcting current and not-unlikely future failures in FixXtalResp

On 5 Aug 2010 (Day 217), at 5:24 PM EDT, J. Eric Grove wrote:

Wiki MarkupChul,
Here I've detailed the actions for FixXtalResp in the cases we currently understand. I've added a new CalXtalRecData status word definition \ [at the bottom of this email\] adding Bill's corrected-energy bit and a bit to indicate that we've mitigated the failure.
Eric

case == bad plus-face HEX8 && bad plus-face HEX1 && ! plus-face HE autoranging disabled
/* this is the case for the current failure and current configuration */
/* if one of the HEX ranges is best range, fix it */
/* if plus-face first range == (HEX8 || HEX1),
	calculate E using the opposite face and the externally provided longitudinal position
	clear the bad energy bit
	set the externally provided longitudinal position bit
	set the failure-mitigation energy bit
*/


case == bad plus-face HEX8 && bad plus-face HEX1 && plus-face HE autoranging disabled
/* this is the case for the current failure, but it requires a configuration we have not used */
/* if LEX1 is best range and saturated, fix it */
/* if plus-face first range == LEX1 && plus-face > 4050 (i.e. saturated),
	calculate E using the opposite face and the externally provided longitudinal position
	clear the bad energy bit
	set the externally provided longitudinal position bit
	set the failure-mitigation energy bit
*/


case == bad plus-face LEX8 && bad plus-face LEX1 && bad plus-face HEX8 && bad plus-face HEX1
/* this is the case we may get to soon if this GCFE continues to degrade and the LE ranges fail */
/* in all cases,
	calculate E using the opposite face and the externally provided longitudinal position
	clear the bad energy bit
	set the externally provided longitudinal position bit
	set the failure-mitigation energy bit
*/

Algorithms for correcting for saturated xtals

This is the rough outline that we (Eric G., Philippe, Tracy and myself) agreed on during the F2F pass8 meeting in Pisa (Feb 2011).

1. Perform the start xtal rec step.
   • We'll put in all the new bits to keep track of what's happening, according to the scheme outlined above. Chul and Tracy will follow up on this and get it done.
2. If there's one or more saturated xtal(s), the first thing we'll do in CalRecon is a fit using the transverse information only with the full xtal collection (i.e. before the clustering) to be used to fix the position for saturated xtals. Luca B. and Philippe will take care of this.
   • This will require moving the current fitting routine (now living in CalRecon/src/Clustering/MomentsClusterInfo.cxx) in some common area (Tracy suggests CalRecon/src/Utilities/) to be used in more than one place. Luca B. will take care of this.
   • As a side project, the possibility of not using Minuit for the fit will be investigated, in order to gain in speed and avoid a dependency on ROOT. This is low-priority project that Luca B. will take a look into.
3. At this point (if there are saturated xtals) from CalRecon we will call a dedicated tool in CalXtalResponse to set the longitudinal position to our best value, based on the transverse fit (will need some coordination between all of us).
   • After this point the CalXtalRecData objects in the TDS will have a reasonable longitudinal position information even for the saturated xtals, and the correct information will propagate trasparently to the downstream code (with the proper status bit set).
4. The next step is the calorimeter clustering (i.e. the splitting of the xtal collection in clusters).
5. And after that we'll go through the standard transverse fit/moments analysis/classification on a cluster by cluster basis.
   • We decided to keep the CalMomentsData as the basic object the CAL moments analysis acts on. Luca B. will take care of propagating all the new information (i.e. the status bits) to the object.
   • Further possible adjustments to the xtals that need transverse fit at the cluster level will be done by calling the proper tool at this point.