Content

On this page:

Other material

This is the page for the code review containing a presentation (MIND: slide 7 and 8 are OUTDATED) and other material. Mind that the code have been updated following the code review, thus for the most updated documentation see the Doxygen of the current version.
Description of the old code (here)
Expected accuracy for position interpolation (here)

Why a new code

The old code (ft2Util), written by Andrea Tramacere, has performed very well during the first 3 years of the mission. But it has been patched a lot of times under the pressure of time, and it is now quite difficult to understand and to maintain. Thus we decided to rewrite it, taking into account all the things we learned during these 3 years.

Programmatic differences

The implementations are completely different, but from a conceptual point of view there are only few main differences:

Handling of data gaps during the livetime computation: Fermi data on-ground can have two different kinds of gaps: gaps in the Digi file or gaps in the Merit file. The former type can be the results of a copy/download problem from the spacecraft or the ground stations, while the latter can be caused by a problem during the reconstruction (the Recon step). Both of them can also result from problems during the merging of different chunks of data. Note that all these problems are usually temporary, and gaps are usually filled with a roll back of the appropriate processing steps. The old code used to try to compensate for those gaps, by subtracting the corresponding deadtime from the livetime of the affected bin(s). The new code instead isolates the time span affected by these problems in single bins, which will have zero livetime, and thus will generate bad time intervals which will be excluded by downstream software (like the Science Tools). Every gap is defined by the last good events around it, so the start (stop) time of the gap is defined as the time stamp of the last event before (the first event after) the gap, plus (minus) a configurable offset, to avoid the exclusion of the events surrounding the gap.
Sun coordinates: the new code now compute the Sun coordinates and saves them in the output FT2 file
Exploit all the attitude information: the attitude information is issued by the spacecraft at 5 Hz, and it is formatted in ATT messages available on-ground in the so-called "Magic 7" files. The position information instead is issued by the spacecraft at 1 Hz. The old code used only those ATT messages issued contemporaneously to ORB messages, discarding the others. The new code instead keeps all the information. So when interpolation is needed, the old code have less information than the new one, slightly degrading the quality of the interpolations. The difference is anyway unimportant in most of the circumstances, leading to very small improvement in the quality of the interpolations and extrapolations of the satellite attitude.
Check for proper input: the new code check that both the Digi file both the Merit file are ordered by event ID and time, and throws an exception if this is not the case. Out of order events can result from merging or download problem.

Interpolation and extrapolation

The Magic 7 file can sometimes contain gaps, that is, time interval inside a run where no position or attitude information are available. In this case the code will interpolate (or extrapolate, if the gaps are at the beginning or at the end of a run) the known information. The attitude interpolation/extrapolation is obtained by performing a SLERP between the closest known quaternion, which basically means to assume a constant rotational velocity for the spacecraft between the two known attitudes. Regarding the position, the interpolation are performed using the same algorithm already implemented in gtbary (you can find a description here). Position extrapolation is instead performing by fitting the closest known position before and after the gap with a quadratic polynomial, and using the fit to extrapolate forward or backward. Mind that the extrapolation should be very rare, since the Magic 7 file usually covers much more than the run interval.

Executables

makeFT2.exe

To create an FT2 file you can use the makeFT2.exe executable. There are two flavours of FT2 files: the "fake" FT2 file, and the normal FT2 file. The former lacks the livetime information, and it is generated inside the pipeline at the chunk level to be used as input to Recon. The latter provides the livetime information, and it is generated inside the pipeline at the run level.
To produce an FT2 file you will always need a Magic 7 file. If you want to compute also the livetime ("normal" FT2 file), you will need also a Digi file, a Merit file, and a DigiGAP file. You can download all those files for a particular run from the data catalog.
This executable will normally produce a FT2 file with 1s time binning. If, when producing a normal FT2 file, there are gaps in the data files, as described above, some bins in the FT2 file will have a duration < 1 s, and livetime=0. These time intervals will be treated by downstream software as Bad Time Intervals (BTIs) and they will be excluded from scientific analysis.

Configuration

The program read a configuration file contained in "data/configuration.rc". This is the content of the file, with the comments describing each parameter:

#This file contains the configuration for the ft2Util_2 package            
#Every line beginning with "#" is a comment.                               
#Every blank line is ignored                                               

#Package info

packageVersion             = v0r0p0
packageName                = ft2Util_2
packageAuthor              = G.Vianello
packageAuthorEmail         = giacomov@slac.stanford.edu

#The Julian date of the start of the mission
JulianDateMissionStart     = 211845067200.0 

#If the difference between two time stamps is less than NULL_TIME_DIFFERENCE
#the two time stamps are considered to be the same (in MET, that is, in seconds)
NULL_TIME_DIFFERENCE       = 1E-6                                               

#This is the number of ticks occurring between the closing of the window
#for physics events and the issue of the sweep event, at the end of a run
sweepEventsDelayEnd        = 20000                                       

#This is the number of ticks needed to complete a single command in the
#flight software                                                       
ticksPerCommand            = 196                                       

#This is the roll over number for the onboard clock
RollOver                   = 33554432.0            

#This is the nominal conversion factor from ticks to second for the 
#onboard clock                                                      
nominalTicksToSecond       = 5E-8                                   

#The following three numbers are the angles (in arcsec) for the default boresight                                                                                 
#correction, which is the rotation which translates the spacecraft pointing into 
#the LAT pointing                                                                
Rx                         = -170
Ry                         = -173
Rz                         = -491

#When the Magic 7 file covers more than the time requested in the command line,
#all the messages referring to more than magic7ReadPadding seconds
#before and after the requested interval will be ignored. The others are kept
#to allow a safe extrapolation for the position and attitude, if needed.
magic7ReadPadding          = 30

#This flag enable/disable the check for gaps in the Digi and Merit files
#(turn this to "no" to speed up the process, if you already know the files are ok.
#You can also use a command line option to do this)
verify                     = yes

#When there is one or more missing events either in the Digi either in the Merit,
#the code produces a bad time interval going from t1+deadPad to t2-deadPad,
#where t1 and t2 are respectively the time stamp of the event before the gap
#and the time stamp of the event after the gap. So deadPad is a padding to avoid
#the loosing of the events before and after the gap because of the GTI filtering
#(in micro seconds)
deadPad                    =  1

How to use

makeFT2.exe

makeFT2.exe : produce the FT2 file for the Fermi mission.

(See http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html#SpacecraftFile)                                      

Usage: 

-Produce the 'fake' FT2 file (without livetime):

makeFT2.exe -ft2start [start time] -ft2stop [stop time]
            -m7file [Magic 7 file] -ft2file [output FT2 file]
            [-clobber ['yes' or 'no'] ]                      

-Produce a regular FT2 file (with livetime):

makeFT2.exe -ft2start [start time] -ft2stop [stop time]
            -m7file [Magic 7 file] -ft2file [output FT2 file]
            -digifile [Digi file] -meritfile [Merit file]    
            -gapfile [Digi gap file]
            [-clobber ['yes' or 'no'] -verify ['yes' or 'no']
             -dataquality [DQ flag] -latconfig [LAT config value]
             -templateFT2 [FT2 template] -version [version number] ]

Parameters:

 -ft2start         Start time for the output FT2 file (MET)
 -ft2stop          Stop time for the output FT2 file (MET)
 -m7file           Input Magic 7 file
 -ft2file          Name for the output FT2 file
 -digifile         Input Digi file
 -meritfile        Input Merit file
 -gapfile          Input Digi gap file
 -clobber          (yes|no) Overwrite an existing output? (default = no)
 -dataquality      Data Quality flag to set (default = 2)
 -latconfig        LAT configuration status (default = 1)
 -templateFT2      Override the default template file for the output FT2
 -version          Version number for the file (integer,default=1)
 -configFile       User-provided configuration file path
 -verify           (yes|no) Verify Merit and Digi for gaps (default=yes)
 -help             Prints this help

mergeFT2.exe

This program merge bins from 1s FT2 file producing a 30s FT2 file, or a FT2 file with the specified binning. Note that some bins in the output file could have a different duration than the target binning, since the program will prematurely cut bins for example if LAT_MODE or LAT_CONFIG change value, or if it encounters bins with livetime=0 in the input FT2 file. In any case the program will never produce bins with a duration greater than the target binning.

How to use

mergeFT2.exe

mergeFT2.exe : Merge bins from 1s FT2 file producing a 30s FT2 file, or a FT2 file with the specified binning.

(See http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html#SpacecraftFile)

mergeFT2.exe -inputFT2 [input 1s FT2 file] -outputFT2 [output FT2 file]
            -version [version int]
            [-dt [time step] -clobber ['yes' or 'no']

Parameters:

 -inputFT2         Input 1s FT2 file
 -outputFT2        Name for the output FT2 file
 -version          Version to be written in the header (integer, default=1)
 -dt               Bin size for the output file*
 -clobber          (yes|no) Overwrite an existing output? (default = no)
 -help             Prints this help

* Note that some bins in the output file could have a different duration than 'dt', since the program will prematurely cut bins for example if LAT_MODE or LAT_CONFIG change value, or if it encounters bins with livetime=0 in the input FT2 file

Validation

Here I collect the results of the testing I performed, comparing the output generated by the old ft2Util code with the output generated by my code.

I used the version of the old code currently used in the PROD pipeline (v1r2p32), and I compiled my code and the old one against GlastRelease-v15r47p12gr21.

Output testing

Procedure

I selected 100 runs (from 323423607 to 323989797), then for each run I submitted a different jobs to the farm. Each job performed these steps:

Download the Digi, Merit, Magic7L1 and DigiGap files using the datacatalog, placing them in the /scratch directory of the batch worker.

Run the old code and mine, with the following command lines:

Old version:

makeFT2Entries.exe -M7File $magic7 \
                   -FT2_fits_File ${filenameroot}_andrea.fits \
		   -DigiTstart $start \
                   -DigiTstop $stop \
		   -new_tpl $AndreaTemplate \
                   -DigiFile $digifile \
                   -MeritFile $meritfile \
                   -Gaps_File $digigapfile

New version:

makeFT2.exe -m7file $magic7 \
            -ft2file ${filenameroot}_io.fits \
            -ft2start $start \
            -ft2stop $stop \
            -digifile $digifile \
            -meritfile $meritfile \
            -gapfile $digigapfile

(without specifying a template, the new version uses the template provided by the current version of fitsGen)
$start and $stop are respectively the time stamp of the first and the last event in the Digi file.

Run a python script which matches the entries in the two FT2 files and produces another FITS file containing the fractional difference for each quantity for each entry.
Move back the produced files to my user disk

When every job finished, I simply merged all the FITS files produced by the python script in one big FITS file, and generate the plots.

Results

The following plots show the results (mind the scales on the y axes, usually the differences are very small!). For every quantity q (like LAT_GEO, LON_GEO, RA_SCX, DEC_SCX...) I computed (q_old-q_new)/q_old , that is, the "fractional difference".


	a	b	c	d

e	f	g	h	i

l	m	n	o	p

q	r	s	t	u

v	z

As you can see, there are differences, but they are very small.

Attitude and orbital quantities

The very small differences between the two codes arise mainly from two different strategies adopted for the interpolation/extrapolation: the old code uses only ATT messages arriving in sync with an ORB message (that is, one time per second), discarding the others; the new code, instead, uses all the available ATT messages. Thus, the new code should provide better interpolation/extrapolation. Anyway, the differences are really tiny, and I can't think of any application where they could really matter...

Livetime

I show here only the results concerning the live time:


a	b	c

In the first panel there is the fractional difference for LIVETIME: as you can see, there are differences up to a few percent (~5%). The second panel shows a zoom of a region: we can clearly see a small growing trend inside each and every run, and some occasional larger "flares". As you can see in the third panel, those flares seem symmetrically distributed around the zero line, with a upward fluctuation followed by a downward fluctuation of the same amount. But, two fluctuations can combine giving a third fluctuation which is the opposite of the algebraic sum of the former two. This also means that this should NOT usually be a problem for analysis using 30s FT2 files, or in general spanning more than few seconds, because those problems will cancel out.

To investigate the issue, I've produced a Digi file where I've overwritten the livetime counter in each event with its elapsed time counter. This means a dead time of zero, thus every bin in the FT2 file (except the first and the last one) must have LIVETIME = 1. Then, I run both codes on this Digi file, with the following results:

Old code

New code

a

b

As you can see, while my code gives exactly 1 for every bin, the old code has the same behavior as with the real file, and it's clearly wrong, returning sometimes a LIVETIME greater than the elapsed time. As a further check, I repeated the same procedure imposing a dead time fraction of 0.5 (livetime=elapsed/2), obtaining the same results:

Old code

New code

c

d

Gap handling

Procedure

I modified one Digi file and one Merit file deliberately inserting gaps, and I check if the code can find them and handle them correctly.

In particular, I downloaded from the data catalog a Digi and a Merit file for run 327356667 ("r0327356667_v001_digi.root" and "r0327356667_v120_merit.root"), and I select the events with 100000 < event ID < 700000 in both files. Then I inserted these gaps:

I removed from the Digi file all events between 81.3 and 84.6 seconds from the time stamp of the first event in the file (a total of 2609 events removed). The last event before the gap has eventID = 145973 and the first event after the gap has eventID = 159384. I also overwritten the livetime counter making it equal to the elapsed time counter (thus, deadtime = 0). This is saved in "data/r0327356667_digi_gaps.root".
I removed from the Merit file the events with these IDs: 199996 (which is comprised between event 199993 and event 200002), 2000031 (which is comprised between event 200029 and event 200035), and all the events with 300544 < event ID < 300858. This is saved in "data/r0327356667_v001_merit_crashes.root".
I inserted by hand two fake gaps in the digi Gap file, excluding events in these ranges of event IDs: 129504-181395 and 273973-280562. This is saved in "data/r0327356667_v001_digiGap_2gaps.txt"

All the files with the gaps are available in the /data directory of the package, as well as the ROOT macros used to produce them. Note that they contain only the variables used in the FT2 code (to reduce their size).

The following is the output of the code used to produce the FT2 file with time bin of 1 s. As you can see, it finds all the gaps. There are a couple of "INTERPOLATION NEEDED" messages, which are expected, and correspond to the beginning and the end of the bad time intervals (which is also the beginning of the bin after the bad time interval):


Input parameters: 
--------------------------------------------------
Desired start time for the FT2        :327356689.88605165482
Desired stop time for the FT2         :327356838.64119803905
Magic 7 file                          :../data/r0327356667_v001_magic7L1.txt
Output FT2 file                       :test.fits
Input Digi file                       :../data/r0327356667_digi_gaps.root
Input Merit file                      :../data/r0327356667_v001_merit_crashes.root
Input Digi Gap file                   :../data/r0327356667_v001_digiGap_2gaps.txt
Overwrite an existing FT2 file        :yes
Data quality flag                     :2
LAT configuration                     :1
Template for the FT2 file             :/afs/slac/g/glast/ground/glastsoft/AUTO_GLAST/arch/os/SCONS_BUILDS/Optimized/ScienceTools/09-25-01/fitsGen/data/ft2.tpl
Version number for the FT2 file       :1


Configuration: 
(from file /a/wain006/g.glast.u55/giacomov/ft2/users/giacomov/ft2Util_2/v0r0p0/data/configuration.rc)
--------------------------------------------------
packageName                           :ft2Util_2
packageVersion                        :v0r0p0
packageAuthor                         :G.Vianello
packageAuthorEmail                    :giacomov@slac.stanford.edu
JulianDateMissionStart                :211845067200
NULL_TIME_DIFFERENCE                  :1e-06
sweepEventsDelayEnd                   :20000
ticksPerCommand                       :196
RollOver                              :33554432
nominalTicksToSecond                  :5e-08
Rx                                    :-170
Ry                                    :-173
Rz                                    :-491
magic7ReadPadding                     :30
verify                                :yes
deadPad                               :1
Magic 7 file ../data/r0327356667_v001_magic7L1.txt opened

Magic 7 content:
--------------------------------------------------
N. of ATT messages                    :1044
Time of the first ATT message (MET)   :327356660
Time of the last ATT message (MET)    :327356868.6
N. of ORB messages                    :209
Time of the first ORB message (MET)   :327356660.6
Time of the last ORB message (MET)    :327356868.6

First concurrent ORB and ATT messages at time (MET):   327356660.6

INFO: Ft2::Ft2():: There are 29 time bins before the requested start covered by the Magic7 file which are not needed

Rebuilding Digi index... done
Building Merit index... done

Livetime::Livetime(): the input Digi file lacks a sweep event at the beginning. This means it does not span an entire run.
Livetime input:
--------------------------------------------------
N. of Digi events                     :110325
Time of the first Digi event (MET)    :327356689.88605
ID of the first Digi event            :100001
Time of the last Digi event (MET)     :327356838.6412
ID of the last Digi event             :699997
N. of Merit events                    :112874
Time of the first Merit event (MET)   :327356689.88605
Time of the last Merit event (MET)    :327356838.6412
Sweep events at the beginning:        :0
Sweep events at the end:              :0

Rebuilding Digi index... done
Building Merit index... done


Gap file ../data/r0327356667_v001_digiGap_2gaps.txt parsed:
--------------------------------------------------
N. of gaps for this run               :2
gaps:
327356697.132590115070343 - 327356709.861321747303009
327356732.6005374193191528 - 327356734.2353370785713196


Checking Merit and Digi for errors or gaps: 

    0 % completed
   10 % completed
GapHandler: gap in the Merit file between event 199993 and event 200002
You lost 1 event (these events are contained in a bad time interval, which will be excluded by downstream software)

GapHandler: gap in the Merit file between event 200029 and event 200035
You lost 1 event (these events are contained in a bad time interval, which will be excluded by downstream software)

   20 % completed
   30 % completed
GapHandler: gap in the Merit file between event 300544 and event 300858
You lost 58 event (these events are contained in a bad time interval, which will be excluded by downstream software)

   40 % completed
   50 % completed
GapHandler: gap in the Digi file between event 429501 and event 442840
You lost 2609 event (these events are contained in a bad time interval, which will be excluded by downstream software)

   60 % completed
   70 % completed
   80 % completed
   90 % completed

WARNING: There are gaps either in the Merit either in the Digi file (or both).


Bad time intervals:
327356697.132590115070343 - 327356709.861321747303009
327356732.6005374193191528 - 327356734.2353370785713196
327356714.4137212038040161 - 327356714.4157781600952148
327356714.4220960736274719 - 327356714.423194944858551
327356739.1916109323501587 - 327356739.2655935287475586
327356771.1854096651077271 - 327356774.4867381453514099

Now parse the file and compute the quantities to be saved in the FT2 file:
---------------------------------------------------------------------------
    0 % completed
WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356689.88605


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356689.88605
Performing SLERP between 327356689.8 and 327356690 to get attitude at time 327356689.88605 (fraction: 0.43025824011911 s )

WARNING: Magic7::getMode(): no ORB message at time 327356689.88605 . Returning the last known mode (5), reported in the ORB message at time 327356689.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356689.88605 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356689.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356697.13259


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356697.13259
Performing SLERP between 327356697 and 327356697.2 to get attitude at time 327356697.13259 (fraction: 0.66295061486665 s )

WARNING: Magic7::getMode(): no ORB message at time 327356697.13259 . Returning the last known mode (5), reported in the ORB message at time 327356696.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356697.13259 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356696.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356709.86132


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356709.86132
Performing SLERP between 327356709.8 and 327356710 to get attitude at time 327356709.86132 (fraction: 0.3066086951857 s )

WARNING: Magic7::getMode(): no ORB message at time 327356709.86132 . Returning the last known mode (5), reported in the ORB message at time 327356709.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356709.86132 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356709.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.41372


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.41372
Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.41372 (fraction: 0.068606121872396 s )

WARNING: Magic7::getMode(): no ORB message at time 327356714.41372 . Returning the last known mode (5), reported in the ORB message at time 327356713.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356714.41372 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6

   10 % completed
WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.41578


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.41578
Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.41578 (fraction: 0.078890900876307 s )

WARNING: Magic7::getMode(): no ORB message at time 327356714.41578 . Returning the last known mode (5), reported in the ORB message at time 327356713.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356714.41578 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.4221


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.4221
Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.4221 (fraction: 0.11048046100605 s )

WARNING: Magic7::getMode(): no ORB message at time 327356714.4221 . Returning the last known mode (5), reported in the ORB message at time 327356713.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356714.4221 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.42319


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.42319
Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.42319 (fraction: 0.11597481585149 s )

WARNING: Magic7::getMode(): no ORB message at time 327356714.42319 . Returning the last known mode (5), reported in the ORB message at time 327356713.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356714.42319 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6

   21 % completed
WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356732.60054


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356732.60054
Performing SLERP between 327356732.6 and 327356732.8 to get attitude at time 327356732.60054 (fraction: 0.002686977546631 s )

WARNING: Magic7::getMode(): no ORB message at time 327356732.60054 . Returning the last known mode (5), reported in the ORB message at time 327356732.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356732.60054 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356732.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356734.23534


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356734.23534
Performing SLERP between 327356734.2 and 327356734.4 to get attitude at time 327356734.23534 (fraction: 0.17668546299252 s )

WARNING: Magic7::getMode(): no ORB message at time 327356734.23534 . Returning the last known mode (5), reported in the ORB message at time 327356733.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356734.23534 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356733.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356739.19161


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356739.19161
Performing SLERP between 327356739 and 327356739.2 to get attitude at time 327356739.19161 (fraction: 0.9580547188553 s )

WARNING: Magic7::getMode(): no ORB message at time 327356739.19161 . Returning the last known mode (5), reported in the ORB message at time 327356738.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356739.19161 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356738.6

   31 % completed
WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356739.26559


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356739.26559
Performing SLERP between 327356739.2 and 327356739.4 to get attitude at time 327356739.26559 (fraction: 0.32796772289084 s )

WARNING: Magic7::getMode(): no ORB message at time 327356739.26559 . Returning the last known mode (5), reported in the ORB message at time 327356738.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356739.26559 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356738.6

   41 % completed
   52 % completed
WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356771.18541


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356771.18541
Performing SLERP between 327356771 and 327356771.2 to get attitude at time 327356771.18541 (fraction: 0.92704838079502 s )

WARNING: Magic7::getMode(): no ORB message at time 327356771.18541 . Returning the last known mode (5), reported in the ORB message at time 327356770.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356771.18541 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356770.6

WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356774.48674


----------------------
INTERPOLATIONS NEEDED

Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356774.48674
Performing SLERP between 327356774.4 and 327356774.6 to get attitude at time 327356774.48674 (fraction: 0.43369074256641 s )

WARNING: Magic7::getMode(): no ORB message at time 327356774.48674 . Returning the last known mode (5), reported in the ORB message at time 327356773.6


WARNING: Magic7::getInSAA():  no ORB message at time 327356774.48674 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356773.6

   62 % completed
   72 % completed
   83 % completed
   93 % completed
FT2 time bins:
--------------------------------------------------
N. of bins                            :145
Beginning (MET)                       :327356689.88605
End (MET)                             :327356838.6412

This is a plot of the livetime in the output FT2 file:

As you can see, most bins have livetime = 1. The bin corresponding to the gaps have livetime = 0, while the bins surrounding the gaps have 1 < livetime < 0.

This is a table summarizing the gap and the corresponding bin in the FT2 file:

Event before and after the gap	Corresponding time interval directly read from the original Digi file	FT2 file start-stop	Gap type
129504 - 181395	327356697.1325891 - 327356709.8613227	327356697.1325901 - 327356709.8613217	Gap from the Digi gap file
273973 - 280562	327356732.6005364 - 327356734.2353380	327356732.6005374 - 327356734.2353370	Gap from the Digi gap file
199993 - 200002	327356714.4137201 - 327356714.4157791	327356714.4137212 - 327356714.4157781	Gap in the Merit
200029 - 200035	327356714.4220950 - 327356714.4231959	327356714.4220960 - 327356714.4231949	Gap in the Merit
300544 - 300858	327356739.1916099 - 327356739.2655945	327356739.1916109 - 327356739.2655935	Gap in the Merit
429501 - 442840	327356771.1854086 - 327356774.4867391	327356771.1854096 - 327356774.4867381	Gap in the Digi

As you can see, everything is right.

Now, using mergeFT2.exe to rebin the FT2 file gives this:

./mergeFT2.exe -inputFT2 test.fits -outputFT2 test_30s.fits -clobber yes                             

Input parameters: 
---------------------------------------------
Input FT2 file                             : test.fits
Output FT2 file                            : test_30s.fits
Overwrite an existing FT2 file             : yes          
File version                               : 1            
Time step for the output FT2 file          : 30           


* Entry number 1:
  WARNING: livetime=0 at 327356697.13259. Stopping there. 
  327356689.88605 ........ 327356697.13259                
  Elapsed time 7.24654, Livetime = 7.24654 (100 %)        

* Entry number 2:
  Starting from livetime=0 at 327356709.86132
  327356697.13259 . 327356709.86132          
  Elapsed time 12.7287, Livetime = 0 (0 %)   

* Entry number 3:
  WARNING: livetime=0 at 327356714.41372. Stopping there. 
  327356709.86132 ..... 327356714.41372                   
  Elapsed time 4.5524, Livetime = 4.5524 (100 %)          

* Entry number 4:
  Starting from livetime=0 at 327356714.41578
  327356714.41372 . 327356714.41578          
  Elapsed time 0.00205696, Livetime = 0 (0 %)

* Entry number 5:
  WARNING: livetime=0 at 327356714.4221. Stopping there. 
  327356714.41578 . 327356714.4221                       
  Elapsed time 0.00631791, Livetime = 0.00631792 (100 %) 

* Entry number 6:
  Starting from livetime=0 at 327356714.42319
   327356714.4221 . 327356714.42319          
  Elapsed time 0.00109887, Livetime = 0 (0 %)

* Entry number 7:
  WARNING: livetime=0 at 327356732.60054. Stopping there. 
  327356714.42319 .................... 327356732.60054    
  Elapsed time 18.1773, Livetime = 18.1773 (100 %)        

* Entry number 8:
  Starting from livetime=0 at 327356734.23534
  327356732.60054 . 327356734.23534          
  Elapsed time 1.6348, Livetime = 0 (0 %)    

* Entry number 9:
  WARNING: livetime=0 at 327356739.19161. Stopping there. 
  327356734.23534 ...... 327356739.19161                  
  Elapsed time 4.95627, Livetime = 4.95627 (100 %)        

* Entry number 10:
  Starting from livetime=0 at 327356739.26559
  327356739.19161 . 327356739.26559          
  Elapsed time 0.0739826, Livetime = 0 (0 %) 

* Entry number 11:
  327356739.26559 .............................. 327356768.6
  Elapsed time 29.3344, Livetime = 29.3344 (100 %)

* Entry number 12:
  WARNING: livetime=0 at 327356771.18541. Stopping there.
      327356768.6 ... 327356771.18541
  Elapsed time 2.58541, Livetime = 2.58541 (100 %)

* Entry number 13:
  Starting from livetime=0 at 327356774.48674
  327356771.18541 . 327356774.48674
  Elapsed time 3.30133, Livetime = 0 (0 %)

* Entry number 14:
  327356774.48674 .............................. 327356803.6
  Elapsed time 29.1133, Livetime = 29.1133 (100 %)

* Entry number 15:
      327356803.6 .............................. 327356833.6
  Elapsed time 30, Livetime = 30 (100 %)

* Entry number 16:
      327356833.6 ...... 327356838.6412
  Elapsed time 5.0412, Livetime = 5.0412 (100 %)

As you can see the code stops when it encounter bins with livetime=0, and keep them isolated. Thus, even if the target bin time is 30 s, there is only one bin that long (# 15). The others are prematurely stopped because of the gaps. But, if we change the target binning to 5 s, much more bins will have the target duration:

mergeFT2.exe -inputFT2 test.fits -outputFT2 test_30s.fits -clobber yes -dt 5

Input parameters: 
---------------------------------------------
Input FT2 file                             : test.fits
Output FT2 file                            : test_30s.fits
Overwrite an existing FT2 file             : yes          
File version                               : 1            
Time step for the output FT2 file          : 5            


* Entry number 1:
  327356689.88605 ..... 327356694.6
  Elapsed time 4.71395, Livetime = 4.71395 (100 %)

* Entry number 2:
      327356694.6 ... 327356697.13259
  Elapsed time 2.53259, Livetime = 2.53259 (100 %)

* Entry number 3:
  327356697.13259 . 327356709.86132
  Elapsed time 12.7287, Livetime = 0 (0 %)

* Entry number 4:
  WARNING: livetime=0 at 327356714.41372. Stopping there. 
  327356709.86132 ..... 327356714.41372                   
  Elapsed time 4.5524, Livetime = 4.5524 (100 %)          

* Entry number 5:
  Starting from livetime=0 at 327356714.41578
  327356714.41372 . 327356714.41578          
  Elapsed time 0.00205696, Livetime = 0 (0 %)

* Entry number 6:
  WARNING: livetime=0 at 327356714.4221. Stopping there. 
  327356714.41578 . 327356714.4221                       
  Elapsed time 0.00631791, Livetime = 0.00631792 (100 %) 

* Entry number 7:
  Starting from livetime=0 at 327356714.42319
   327356714.4221 . 327356714.42319          
  Elapsed time 0.00109887, Livetime = 0 (0 %)

* Entry number 8:
  327356714.42319 ..... 327356718.6
  Elapsed time 4.17681, Livetime = 4.17681 (100 %)

* Entry number 9:
      327356718.6 ..... 327356723.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 10:
      327356723.6 ..... 327356728.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 11:
      327356728.6 ..... 327356732.60054
  Elapsed time 4.00054, Livetime = 4.00054 (100 %)

* Entry number 12:
  Starting from livetime=0 at 327356734.23534
  327356732.60054 . 327356734.23534          
  Elapsed time 1.6348, Livetime = 0 (0 %)    

* Entry number 13:
  327356734.23534 ...... 327356739.19161
  Elapsed time 4.95627, Livetime = 4.95627 (100 %)

* Entry number 14:
  Starting from livetime=0 at 327356739.26559
  327356739.19161 . 327356739.26559          
  Elapsed time 0.0739826, Livetime = 0 (0 %) 

* Entry number 15:
  327356739.26559 ..... 327356743.6
  Elapsed time 4.33441, Livetime = 4.33441 (100 %)

* Entry number 16:
      327356743.6 ..... 327356748.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 17:
      327356748.6 ..... 327356753.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 18:
      327356753.6 ..... 327356758.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 19:
      327356758.6 ..... 327356763.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 20:
      327356763.6 ..... 327356768.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 21:
      327356768.6 ... 327356771.18541
  Elapsed time 2.58541, Livetime = 2.58541 (100 %)

* Entry number 22:
  Starting from livetime=0 at 327356774.48674
  327356771.18541 . 327356774.48674          
  Elapsed time 3.30133, Livetime = 0 (0 %)   

* Entry number 23:
  327356774.48674 ..... 327356778.6
  Elapsed time 4.11326, Livetime = 4.11326 (100 %)

* Entry number 24:
      327356778.6 ..... 327356783.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 25:
      327356783.6 ..... 327356788.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 26:
      327356788.6 ..... 327356793.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 27:
      327356793.6 ..... 327356798.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 28:
      327356798.6 ..... 327356803.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 29:
      327356803.6 ..... 327356808.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 30:
      327356808.6 ..... 327356813.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 31:
      327356813.6 ..... 327356818.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 32:
      327356818.6 ..... 327356823.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 33:
      327356823.6 ..... 327356828.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 34:
      327356828.6 ..... 327356833.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 35:
      327356833.6 ..... 327356838.6
  Elapsed time 5, Livetime = 5 (100 %)

* Entry number 36:
      327356838.6 . 327356838.6412
  Elapsed time 0.041198, Livetime = 0.041198 (100 %)

Child pages

The new FT2 code

Content

Other material

Why a new code

Programmatic differences

Interpolation and extrapolation

Executables

makeFT2.exe

Configuration

How to use

mergeFT2.exe

How to use

Validation

Output testing

Procedure

Results

Attitude and orbital quantities

Livetime

Gap handling

Procedure