Content

On this page:

Other material

• This is a presentation of the code for the code review (MIND: slide 7 and 8 are OUTDATED!)
• Description of the old code (here)
• ft2Util_2 in cvs (here)
• Expected accuracy for position interpolation (here)

Why a new code

The old code (ft2Util), written by Andrea Tramacere, has performed very well during the first 3 years of the mission. But it has been patched a lot of times under the pressure of time, and it is now quite difficult to understand and to maintain. Thus we decided to rewrite it, taking into account all the things we learned during these 3 years.

Programmatic differences

The implementations are completely different, but from a conceptual point of view there are only few main differences:

• Handling of data gaps during the livetime computation: Fermi data on-ground can have two different kinds of gaps: gaps in the Digi file or gaps in the Merit file. The former type can be the results of a copy/download problem from the spacecraft or the ground stations, while the latter can be caused by a problem during the reconstruction (the Recon step). Both of them can also result from problems during the merging of different chunks of data. Note that all these problems are usually temporary, and gaps are usually filled with a roll back of the appropriate processing steps. The old code used to try to compensate for those gaps, by subtracting the corresponding deadtime from the livetime of the affected bin(s). The new code instead isolates the time span affected by these problems in single bins, which will have zero livetime, and thus will generate bad time intervals which will be excluded by downstream software (like the Science Tools). Every gap is defined by the last good events around it, so the start (stop) time of the gap is defined as the time stamp of the last event before (the first event after) the gap, plus (minus) a configurable offset, to avoid the exclusion of the events surrounding the gap.
• Sun coordinates: the new code now compute the Sun coordinates and saves them in the output FT2 file
• Exploit all the attitude information: the attitude information is issued by the spacecraft at 5 Hz, and it is formatted in ATT messages available on-ground in the so-called "Magic 7" files. The position information instead is issued by the spacecraft at 1 Hz. The old code used only those ATT messages issued contemporaneously to ORB messages, discarding the others. The new code instead keeps all the information. So when interpolation is needed, the old code have less information than the new one, slightly degrading the quality of the interpolations. The difference is anyway unimportant in most of the circumstances, leading to very small improvement in the quality of the interpolations and extrapolations of the satellite attitude.
• Check for proper input: the new code check that both the Digi file both the Merit file are ordered by event ID and time, and throws an exception if this is not the case. Out of order events can result from merging or download problem.

Interpolation and extrapolation

The Magic 7 file can sometimes contain gaps, that is, time interval inside a run where no position or attitude information are available. In this case the code will interpolate (or extrapolate, if the gaps are at the beginning or at the end of a run) the known information. The attitude interpolation/extrapolation is obtained by performing a SLERP between the closest known quaternion, which basically means to assume a constant rotational velocity for the spacecraft between the two known attitudes. Regarding the position, the interpolation are performed using the same algorithm already implemented in gtbary (you can find a description here). Position extrapolation is instead performing by fitting the closest known position before and after the gap with a quadratic polynomial, and using the fit to extrapolate forward or backward. Mind that the extrapolation should be very rare, since the Magic 7 file usually covers much more than the run interval.

Executables

makeFT2.exe

To create an FT2 file you can use the makeFT2.exe executable. There are two flavours of FT2 files: the "fake" FT2 file, and the normal FT2 file. The former lacks the livetime information, and it is generated inside the pipeline at the chunk level to be used as input to Recon. The latter provides the livetime information, and it is generated inside the pipeline at the run level.
To produce an FT2 file you will always need a Magic 7 file. If you want to compute also the livetime ("normal" FT2 file), you will need also a Digi file, a Merit file, and a DigiGAP file. You can download all those files for a particular run from the data catalog.
This executable will normally produce a FT2 file with 1s time binning. If, when producing a normal FT2 file, there are gaps in the data files, as described above, some bins in the FT2 file will have a duration < 1 s, and livetime=0. These time intervals will be treated by downstream software as Bad Time Intervals (BTIs) and they will be excluded from scientific analysis.

Configuration

The program read a configuration file contained in "data/configuration.rc". This is the content of the file, with the comments describing each parameter:

#This file contains the configuration for the ft2Util_2 package            
#Every line beginning with "#" is a comment.                               
#Every blank line is ignored                                               

#Package info

packageVersion             = v0r0p0
packageName                = ft2Util_2
packageAuthor              = G.Vianello
packageAuthorEmail         = giacomov@slac.stanford.edu

#The Julian date of the start of the mission
JulianDateMissionStart     = 211845067200.0 

#If the difference between two time stamps is less than NULL_TIME_DIFFERENCE
#the two time stamps are considered to be the same (in MET, that is, in seconds)
NULL_TIME_DIFFERENCE       = 1E-6                                               

#This is the number of ticks occurring between the closing of the window
#for physics events and the issue of the sweep event, at the end of a run
sweepEventsDelayEnd        = 20000                                       

#This is the number of ticks needed to complete a single command in the
#flight software                                                       
ticksPerCommand            = 196                                       

#This is the roll over number for the onboard clock
RollOver                   = 33554432.0            

#This is the nominal conversion factor from ticks to second for the 
#onboard clock                                                      
nominalTicksToSecond       = 5E-8                                   

#The following three numbers are the angles (in arcsec) for the default boresight                                                                                 
#correction, which is the rotation which translates the spacecraft pointing into 
#the LAT pointing                                                                
Rx                         = -170
Ry                         = -173
Rz                         = -491

#When the Magic 7 file covers more than the time requested in the command line,
#all the messages referring to more than magic7ReadPadding seconds
#before and after the requested interval will be ignored. The others are kept
#to allow a safe extrapolation for the position and attitude, if needed.
magic7ReadPadding          = 30

#This flag enable/disable the check for gaps in the Digi and Merit files
#(turn this to "no" to speed up the process, if you already know the files are ok.
#You can also use a command line option to do this)
verify                     = yes

#When there is one or more missing events either in the Digi either in the Merit,
#the code produces a bad time interval going from t1+deadPad to t2-deadPad,
#where t1 and t2 are respectively the time stamp of the event before the gap
#and the time stamp of the event after the gap. So deadPad is a padding to avoid
#the loosing of the events before and after the gap because of the GTI filtering
#(in micro seconds)
deadPad                    =  1

How to use

makeFT2.exe

makeFT2.exe : produce the FT2 file for the Fermi mission.

(See http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html#SpacecraftFile)                                      

Usage: 

-Produce the 'fake' FT2 file (without livetime):

makeFT2.exe -ft2start [start time] -ft2stop [stop time]
            -m7file [Magic 7 file] -ft2file [output FT2 file]
            [-clobber ['yes' or 'no'] ]                      

-Produce a regular FT2 file (with livetime):

makeFT2.exe -ft2start [start time] -ft2stop [stop time]
            -m7file [Magic 7 file] -ft2file [output FT2 file]
            -digifile [Digi file] -meritfile [Merit file]    
            -gapfile [Digi gap file]
            [-clobber ['yes' or 'no'] -verify ['yes' or 'no']
             -dataquality [DQ flag] -latconfig [LAT config value]
             -templateFT2 [FT2 template] -version [version number] ]

Parameters:

 -ft2start         Start time for the output FT2 file (MET)
 -ft2stop          Stop time for the output FT2 file (MET)
 -m7file           Input Magic 7 file
 -ft2file          Name for the output FT2 file
 -digifile         Input Digi file
 -meritfile        Input Merit file
 -gapfile          Input Digi gap file
 -clobber          (yes|no) Overwrite an existing output? (default = no)
 -dataquality      Data Quality flag to set (default = 2)
 -latconfig        LAT configuration status (default = 1)
 -templateFT2      Override the default template file for the output FT2
 -version          Version number for the file (integer,default=1)
 -configFile       User-provided configuration file path
 -verify           (yes|no) Verify Merit and Digi for gaps (default=yes)
 -help             Prints this help

mergeFT2.exe

This program merge bins from 1s FT2 file producing a 30s FT2 file, or a FT2 file with the specified binning. Note that some bins in the output file could have a different duration than the target binning, since the program will prematurely cut bins for example if LAT_MODE or LAT_CONFIG change value, or if it encounters bins with livetime=0 in the input FT2 file. In any case the program will never produce bins with a duration greater than the target binning.

How to use

mergeFT2.exe

mergeFT2.exe : Merge bins from 1s FT2 file producing a 30s FT2 file, or a FT2 file with the specified binning.

(See http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html#SpacecraftFile)

mergeFT2.exe -inputFT2 [input 1s FT2 file] -outputFT2 [output FT2 file]
            -version [version int]
            [-dt [time step] -clobber ['yes' or 'no']

Parameters:

 -inputFT2         Input 1s FT2 file
 -outputFT2        Name for the output FT2 file
 -version          Version to be written in the header (integer, default=1)
 -dt               Bin size for the output file*
 -clobber          (yes|no) Overwrite an existing output? (default = no)
 -help             Prints this help

* Note that some bins in the output file could have a different duration than 'dt', since the program will prematurely cut bins for example if LAT_MODE or LAT_CONFIG change value, or if it encounters bins with livetime=0 in the input FT2 file

Validation

Here I collect the results of the testing I performed, comparing the output generated by the old ft2Util code with the output generated by my code.

I used the version of the old code currently used in the PROD pipeline (v1r2p32), and I compiled my code and the old one against GlastRelease-v15r47p12gr21.

Output testing

Procedure

I selected 100 runs (from 323423607 to 323989797), then for each run I submitted a different jobs to the farm. Each job performed these steps:

...

When every job finished, I simply merged all the FITS files produced by the python script in one big FITS file, and generate the plots.

Results

The following plots show the results (mind the scales on the y axes, usually the differences are very small!). For every quantity q (like LAT_GEO, LON_GEO, RA_SCX, DEC_SCX...) I computed (q_old-q_new)/q_old , that is, the "fractional difference".

...

As you can see, there are differences, but they are very small.

Attitude and orbital quantities

The very small differences between the two codes arise mainly from two different strategies adopted for the interpolation/extrapolation: the old code uses only ATT messages arriving in sync with an ORB message (that is, one time per second), discarding the others; the new code, instead, uses all the available ATT messages. Thus, the new code should provide better interpolation/extrapolation. Anyway, the differences are really tiny, and I can't think of any application where they could really matter... 

Livetime

I show here only the results concerning the live time:

...

Gap handling

Procedure

I modified one Digi file and one Merit file deliberately inserting gaps, and I check if the code can find them and handle them correctly.

...