Checking FT1 files in the context of the FT2 reprocessing

Maria-Elena reprocessed a complete set of files for a run for 634008661:

root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft2/gll_pt_p310_r0634008661_v000.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft2Seconds/gll_pt1s_p310_r0634008661_v000.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft1/gll_ph_p305_r0634008661_v000.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ls1/gll_ev_p305_r0634008661_v000.fit

Checks on run 634008661

• Philippe: I've compared the new FT1 file with the original file root://glast-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/prod/5.7/ft1/gll_ph_p305_r0634008661_v000.fit and the key variables (energy, direction, event class and types) are identical. The nb of events in the files are also identical.
• Don:

  I compared the original and new photon and event files that Maria Elena created. The data in the photon (FT1) files match exactly. The event files (LS1) have small differences in the PtAlt, PtLat, and PtMagLat columns. Looking at a few examples, the differences are in the fourth significant figure or lower (e.g., 537.501 vs 537.494 in PtAlt). I assume this is because of the change to the geodetic position calculation implemented for the FT2 files. I compared PtAlt values in the LS1 file against RAD_GEO values in the FT2 file for the reprocessed run and they are consistent, so that checks out. I supposed if we were to do everything properly we would need to reprocess all the event files, too, but I think we can just document the issue and move forward since the differences are small.

Later she created a few more FT1 files for several runs:

root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft1/gll_ph_p305_r0600163450_v000.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft1/gll_ph_p305_r0601783627_v002.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft1/gll_ph_p305_r0634002957_v000.fit
root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/ft1/gll_ph_p305_r0633997253_v000.fit

Don: I found some differences this time.  While the 600163450 and 601783627 files are identical to the latest versions received at the FSSC,  the last two files both have differences in the column 18 (LIVETIME) values starting at rows 11859 and 20150, respectively.   For instance for 634002957

The first number is the for "old" file while second is the "new" file.  Every row after this seems to have a different value for LIVETIME.  The other run 633997253 shows the same behavior.  The fdiff utility doesn't report any other differences in the data (e.g, the number of photons in the files are the same).

According to the confluence page On FT1 Livetime: "So for now, the LIVETIME values in the FT1 files should be ignored. Again, they are not currently used by the Science Tools, and in particular do not affect calculations of exposure, which are made with the livetime accumulations in FT2 files."

Checking (RA,DEC) directions: (Philippe)

Following Seth's advice, the following plots were updated using the Haversine formula, which removed a peak at 1e-6.

The 2 runs with significant differences are the ones reported by Don with the weird LIVETIME differences.

Since the (theta,phi) values don't change, the differences occur when going from instrument to sky coordinates. But the differences are on the event basis so it doesn't look like a calibration difference between the 2 processings.

There is no correlation between the angular differences and the LIVETIME differences.

List of differences

The fact that for some events there is a difference in one of the RA,DEC variables and not in the other one is very strange.

I note that some numbers appear several times. Very weird !

And:

• 3.814697265e-06 * 2 = 7.6293945e-06
• 3.814697265e-06 * 3 = 1.5258789e-05
• 3.814697265e-06 * 8 = 3.0517578e-05

Merit files

I've compared:

• original=root://glast-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/prod/5.7/merit/r0634002957_v001_merit.root
• new=root://glast-test-rdr.slac.stanford.edu//glast/Data/Flight/Level1/LPA/dev/5.9/merit/r0634002957_v000_merit.root

And I see the same differences for the variable FT1Ra and FT1Dec.

Pieces of code (/nfs/farm/g/glast/u52/repomanBuild/redhat6-x86_64-64bit-gcc44/GlastRelease/20-10-04-gr07/)

The variables FT1RA and FT1Dec are filled in AnalysisNtuple/src/FT1Alg.cxx:

• gps = astro::GPS::instance();

• double etime(header→time());

• gps→time(etime);

• glastDir= Hep3Vector(CTBBestXDir, CTBBestYDir, CTBBestZDir);
  • this information is the same in both merit files
• SkyDir sdir( gps->toSky(glastDir) );
• m_ft1ra = sdir.ra();
• m_ft1dec = sdir.dec();

The gps object lives in astro/src/GPS.cxx:

• astro::SkyDir GPS::toSky(const CLHEP::Hep3Vector& latdir, double met)
  • CLHEP::Hep3Vector tdir( - (m_currentPoint.rotation() * m_alignment * latdir) );
• void GPS::update(double inputTime)
  • //this function calculates all the relevant position and orientation info
  • the differences could come from this part of the code

Plots with merit files from run 634002957:

Other important differences with the MeritTuple Pt variables

While answering Seth's request about providing the times of the problematic events, I've started comparing more variables and I've discovered other differences concerns some Pt variables in the MeritTuple:

• PtPos[3] = current orbit position
• PtRax, PtDecx = equatorial coordinates for orientation of S/C x-axis
• PtRaz, PtDecz = equatorial coordinates for orientation of S/C z-axis

As can be seen in the pieces of code below, these variables are a copy of some quantities of the gps object. Unfortunately they do not include the quaternion information.

These quantities are the result of some interpolation between values computed from the FT2 file. So we expect them to change continuously with time. One way to check it is to look at the difference between 2 consecutive events w.r.t. to the delta time between these events:

One can see:

• a "main" correlation branch, with clear sub-branches for the PtRaDec x and z directions
• some events with a difference (graph y-axis) = 0
  • it occurs often that the quantity does not change during several consecutive events
• some events scattered away from the main branch ()
  • they correspond to when the information goes back to normal

The non-differences between consecutive events are not exactly the same between the processing and reprocessing merit files. For the position information for instance:

• processing && reprocessing : 3612 evts
• processing && !reprocessing : 466 evts
• !processing && reprocessing: 464 evts

In one processing flavor, there is not a 100% correlation between the deltas of these variables (delta pos=0 does not imply delta S/C z-axis direction = 0, and vice versa) but if one delta is 0, then the fraction of delta=0 in the other variables is larger.

I've dumped the Merit information in a text file (for the run 634002957, starting at 1st event with (Ra,Dec) diff between processings) with the following information:

1345961 id 7012702 t 634005394.600154 as 0.000000 0.000000 d 1.224745 ft2 d 0.000000 1.224745 x 0.000008 0.000008 z 0.000018 0.000018

• first row = row number in MeritTuple
• id = EvtEvent_Id
• t = PtTime
• as = (Ra,Dec) angsep and cal (Ra,Dec) angsep between processings
• d = PtPos distance between processings
• ft2 d = PtPos distance w.r.t. to FT2 file information for processing and reprocessing
• x = S/C x-axis ang sep w.r.t. to FT2 file information for processing and reprocessing
• z = S/C z-axis ang sep w.r.t. to FT2 file information for processing and reprocessing

If the event is bad (the maximum of as is >0 or if d>0), the line starts with:

• !D! if as=0 and d>0
• !A! if as>0 and d=0
• !B! if both as=0 and d=0

To write the text file, I loop over the merit events, loading the FT2 second information when needed (so that START<PtTime<STOP; and a line is written starting with "Loading new ft2 second") and I write the information as soon as there is a bad events (and also the previous events when we start having bad events) and I stop writing it after 100 good evts in a row (so that we can see these problems mix with the good ones).

The bad events come in bunches but in 2 flavors:

• consecutive bad events (e.g. the first bunch in the text file: 100 bad events in a row)
  • occuring right after a new FT2 second has been loaded
  • for one of the processing, the Pt variables stick to the values in the FT2 file
• bad and good events mixed together: e.g. second bunch in the text file: ~25 bad events among good ones
  • not occuring right after a new FT2 second has been loaded
  • the Pt variables don't stick to the values in the FT2 file (but it doesn't mean that they agree between processings)

The main conclusion is that there is something wrong regarding the PtVariables , possibly related to the FT1 Ra,Dec direction of the events, in both processings !

Related pieces of code:

AnalysisNtuple/src/PointingInfo::execute( const astro::GPS& gps)

• start = gps.time();
• CLHEP::Hep3Vector pos_km = gps.position();
• CLHEP::Hep3Vector location = 1.e3* pos_km;
• sc_position[0] = location.x(); sc_position[1] = location.y(); sc_position[2] = location.z();
• ra_zenith = gps.zenithDir().ra(); dec_zenith= gps.zenithDir().dec();
• ra_scx = gps.xAxisDir().ra(); dec_scx = gps.xAxisDir().dec();
• ra_scz = gps.zAxisDir().ra(); dec_scz = gps.zAxisDir().dec();

The interpolation is performed during in void GPS::update(double inputTime)

• // if history, just ask the history to interpolate the table, or whatever
• if(m_rockType == HISTORY || m_rockType == HISTORY_X_EAST ){
•      m_currentPoint = (*m_history)(inputTime);

The last command is defined in const astro::PointingInfo& PointingHistory::operator()(double time)const throw(TimeRangeError)

• m_currentPoint = h1.interpolate(h2, prop, time);

where m_currentPoint contains the following quantities:

• CLHEP::Hep3Vector m_position; ///< position (km)
• Quaternion m_q; ///< orientaion of the SC: xaxis direction
• EarthCoordinate m_earth; ///< Earth position: (lat, lon)

The interpolation is done independently for these quantities (see Seth's comment on the quaternion interpolation)

More on Pt variables and the question on correlation with Ra,Dec difference primary problem

The following plots show the time of problematic events (either Ra,Dec diff between processings or evts with PtPos or S/C x or z axis identical to prev evt). One can clearly see that there is no 100% correlation between these problems.

I was misled by run 634002957 and the corresponding txt file in which I wrote information starting from the first evt with Ra,Dec diff between processings. So I missed all the evts with PtPos or S/C x or z axis identical to prev evt at the beginning of the run for which there is no Ra,Dec diff between processings.

But nevertheless some spikes do correspond.