Content
On this page:
Other material
- This is a presentation of the code for the code review (MIND: slide 7 and 8 are OUTDATED!)
- Description of the old code (here)
- ft2Util_2 in cvs (here)
- Expected accuracy for position interpolation (here)
Why a new code
The old code (ft2Util), written by Andrea Tramacere, has performed very well during the first 3 years of the mission. But it has been patched a lot of times under the pressure of time, and it is now quite difficult to understand and to maintain. Thus we decided to rewrite it, taking into account all the things we learned during these 3 years.
Programmatic differences
The implementations are completely different, but from a conceptual point of view there are only few main differences:
- Handling of data gaps during the livetime computation: Fermi data on-ground can have two different kinds of gaps: gaps in the Digi file or gaps in the Merit file. The former type can be the results of a copy/download problem from the spacecraft or the ground stations, while the latter can be caused by a problem during the reconstruction (the Recon step). Both of them can also result from problems during the merging of different chunks of data. Note that all these problems are usually temporary, and gaps are usually filled with a roll back of the appropriate processing steps. The old code used to try to compensate for those gaps, by subtracting the corresponding deadtime from the livetime of the affected bin(s). The new code instead isolates the time span affected by these problems in single bins, which will have zero livetime, and thus will generate bad time intervals which will be excluded by downstream software (like the Science Tools). Every gap is defined by the last good events around it, so the start (stop) time of the gap is defined as the time stamp of the last event before (the first event after) the gap, plus (minus) a configurable offset, to avoid the exclusion of the events surrounding the gap.
- Sun coordinates: the new code now compute the Sun coordinates and saves them in the output FT2 file
- Exploit all the attitude information: the attitude information is issued by the spacecraft at 5 Hz, and it is formatted in ATT messages available on-ground in the so-called "Magic 7" files. The position information instead is issued by the spacecraft at 1 Hz. The old code used only those ATT messages issued contemporaneously to ORB messages, discarding the others. The new code instead keeps all the information. So when interpolation is needed, the old code have less information than the new one, slightly degrading the quality of the interpolations. The difference is anyway unimportant in most of the circumstances, leading to very small improvement in the quality of the interpolations and extrapolations of the satellite attitude.
- Check for proper input: the new code check that both the Digi file both the Merit file are ordered by event ID and time, and throws an exception if this is not the case. Out of order events can result from merging or download problem.
Interpolation and extrapolation
The Magic 7 file can sometimes contain gaps, that is, time interval inside a run where no position or attitude information are available. In this case the code will interpolate (or extrapolate, if the gaps are at the beginning or at the end of a run) the known information. The attitude interpolation/extrapolation is obtained by performing a SLERP between the closest known quaternion, which basically means to assume a constant rotational velocity for the spacecraft between the two known attitudes. Regarding the position, the interpolation are performed using the same algorithm already implemented in gtbary (you can find a description here). Position extrapolation is instead performing by fitting the closest known position before and after the gap with a quadratic polynomial, and using the fit to extrapolate forward or backward. Mind that the extrapolation should be very rare, since the Magic 7 file usually covers much more than the run interval.
Executables
makeFT2.exe
To create an FT2 file you can use the makeFT2.exe executable. There are two flavours of FT2 files: the "fake" FT2 file, and the normal FT2 file. The former lacks the livetime information, and it is generated inside the pipeline at the chunk level to be used as input to Recon. The latter provides the livetime information, and it is generated inside the pipeline at the run level.
To produce an FT2 file you will always need a Magic 7 file. If you want to compute also the livetime ("normal" FT2 file), you will need also a Digi file, a Merit file, and a DigiGAP file. You can download all those files for a particular run from the data catalog.
This executable will normally produce a FT2 file with 1s time binning. If, when producing a normal FT2 file, there are gaps in the data files, as described above, some bins in the FT2 file will have a duration < 1 s, and livetime=0. These time intervals will be treated by downstream software as Bad Time Intervals (BTIs) and they will be excluded from scientific analysis.
Configuration
The program read a configuration file contained in "data/configuration.rc". This is the content of the file, with the comments describing each parameter:
#This file contains the configuration for the ft2Util_2 package #Every line beginning with "#" is a comment. #Every blank line is ignored #Package info packageVersion = v0r0p0 packageName = ft2Util_2 packageAuthor = G.Vianello packageAuthorEmail = giacomov@slac.stanford.edu #The Julian date of the start of the mission JulianDateMissionStart = 211845067200.0 #If the difference between two time stamps is less than NULL_TIME_DIFFERENCE #the two time stamps are considered to be the same (in MET, that is, in seconds) NULL_TIME_DIFFERENCE = 1E-6 #This is the number of ticks occurring between the closing of the window #for physics events and the issue of the sweep event, at the end of a run sweepEventsDelayEnd = 20000 #This is the number of ticks needed to complete a single command in the #flight software ticksPerCommand = 196 #This is the roll over number for the onboard clock RollOver = 33554432.0 #This is the nominal conversion factor from ticks to second for the #onboard clock nominalTicksToSecond = 5E-8 #The following three numbers are the angles (in arcsec) for the default boresight #correction, which is the rotation which translates the spacecraft pointing into #the LAT pointing Rx = -170 Ry = -173 Rz = -491 #When the Magic 7 file covers more than the time requested in the command line, #all the messages referring to more than magic7ReadPadding seconds #before and after the requested interval will be ignored. The others are kept #to allow a safe extrapolation for the position and attitude, if needed. magic7ReadPadding = 30 #This flag enable/disable the check for gaps in the Digi and Merit files #(turn this to "no" to speed up the process, if you already know the files are ok. #You can also use a command line option to do this) verify = yes #When there is one or more missing events either in the Digi either in the Merit, #the code produces a bad time interval going from t1+deadPad to t2-deadPad, #where t1 and t2 are respectively the time stamp of the event before the gap #and the time stamp of the event after the gap. So deadPad is a padding to avoid #the loosing of the events before and after the gap because of the GTI filtering #(in micro seconds) deadPad = 1
How to use
makeFT2.exe makeFT2.exe : produce the FT2 file for the Fermi mission. (See http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html#SpacecraftFile) Usage: -Produce the 'fake' FT2 file (without livetime): makeFT2.exe -ft2start [start time] -ft2stop [stop time] -m7file [Magic 7 file] -ft2file [output FT2 file] [-clobber ['yes' or 'no'] ] -Produce a regular FT2 file (with livetime): makeFT2.exe -ft2start [start time] -ft2stop [stop time] -m7file [Magic 7 file] -ft2file [output FT2 file] -digifile [Digi file] -meritfile [Merit file] -gapfile [Digi gap file] [-clobber ['yes' or 'no'] -verify ['yes' or 'no'] -dataquality [DQ flag] -latconfig [LAT config value] -templateFT2 [FT2 template] -version [version number] ] Parameters: -ft2start Start time for the output FT2 file (MET) -ft2stop Stop time for the output FT2 file (MET) -m7file Input Magic 7 file -ft2file Name for the output FT2 file -digifile Input Digi file -meritfile Input Merit file -gapfile Input Digi gap file -clobber (yes|no) Overwrite an existing output? (default = no) -dataquality Data Quality flag to set (default = 2) -latconfig LAT configuration status (default = 1) -templateFT2 Override the default template file for the output FT2 -version Version number for the file (integer,default=1) -configFile User-provided configuration file path -verify (yes|no) Verify Merit and Digi for gaps (default=yes) -help Prints this help
mergeFT2.exe
This program merge bins from 1s FT2 file producing a 30s FT2 file, or a FT2 file with the specified binning. Note that some bins in the output file could have a different duration than the target binning, since the program will prematurely cut bins for example if LAT_MODE or LAT_CONFIG change value, or if it encounters bins with livetime=0 in the input FT2 file. In any case the program will never produce bins with a duration greater than the target binning.
How to use
mergeFT2.exe mergeFT2.exe : Merge bins from 1s FT2 file producing a 30s FT2 file, or a FT2 file with the specified binning. (See http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/LAT_Data_Columns.html#SpacecraftFile) mergeFT2.exe -inputFT2 [input 1s FT2 file] -outputFT2 [output FT2 file] -version [version int] [-dt [time step] -clobber ['yes' or 'no'] Parameters: -inputFT2 Input 1s FT2 file -outputFT2 Name for the output FT2 file -version Version to be written in the header (integer, default=1) -dt Bin size for the output file* -clobber (yes|no) Overwrite an existing output? (default = no) -help Prints this help * Note that some bins in the output file could have a different duration than 'dt', since the program will prematurely cut bins for example if LAT_MODE or LAT_CONFIG change value, or if it encounters bins with livetime=0 in the input FT2 file
Validation
Here I collect the results of the testing I performed, comparing the output generated by the old ft2Util code with the output generated by my code.
I used the version of the old code currently used in the PROD pipeline (v1r2p32), and I compiled my code and the old one against GlastRelease-v15r47p12gr21.
Output testing
Procedure
I selected 100 runs (from 323423607 to 323989797), then for each run I submitted a different jobs to the farm. Each job performed these steps:
- Download the Digi, Merit, Magic7L1 and DigiGap files using the datacatalog, placing them in the /scratch directory of the batch worker.
- Run the old code and mine, with the following command lines:
- Old version:
makeFT2Entries.exe -M7File $magic7 \ -FT2_fits_File ${filenameroot}_andrea.fits \ -DigiTstart $start \ -DigiTstop $stop \ -new_tpl $AndreaTemplate \ -DigiFile $digifile \ -MeritFile $meritfile \ -Gaps_File $digigapfile
- New version:
(without specifying a template, the new version uses the template provided by the current version of fitsGen)
makeFT2.exe -m7file $magic7 \ -ft2file ${filenameroot}_io.fits \ -ft2start $start \ -ft2stop $stop \ -digifile $digifile \ -meritfile $meritfile \ -gapfile $digigapfile
$start and $stop are respectively the time stamp of the first and the last event in the Digi file.
- Old version:
- Run a python script which matches the entries in the two FT2 files and produces another FITS file containing the fractional difference for each quantity for each entry.
- Move back the produced files to my user disk
When every job finished, I simply merged all the FITS files produced by the python script in one big FITS file, and generate the plots.
Results
The following plots show the results (mind the scales on the y axes, usually the differences are very small!). For every quantity q (like LAT_GEO, LON_GEO, RA_SCX, DEC_SCX...) I computed (qold-qnew)/qold , that is, the "fractional difference".
There are no images attached to this page. |
As you can see, there are differences, but they are very small.
Attitude and orbital quantities
The very small differences between the two codes arise mainly from two different strategies adopted for the interpolation/extrapolation: the old code uses only ATT messages arriving in sync with an ORB message (that is, one time per second), discarding the others; the new code, instead, uses all the available ATT messages. Thus, the new code should provide better interpolation/extrapolation. Anyway, the differences are really tiny, and I can't think of any application where they could really matter...
Livetime
I show here only the results concerning the live time:
There are no images attached to this page. |
In the first panel there is the fractional difference for LIVETIME: as you can see, there are differences up to a few percent (~5%). The second panel shows a zoom of a region: we can clearly see a small growing trend inside each and every run, and some occasional larger "flares". As you can see in the third panel, those flares seem symmetrically distributed around the zero line, with a upward fluctuation followed by a downward fluctuation of the same amount. But, two fluctuations can combine giving a third fluctuation which is the opposite of the algebraic sum of the former two. This also means that this should NOT usually be a problem for analysis using 30s FT2 files, or in general spanning more than few seconds, because those problems will cancel out.
To investigate the issue, I've produced a Digi file where I've overwritten the livetime counter in each event with its elapsed time counter. This means a dead time of zero, thus every bin in the FT2 file (except the first and the last one) must have LIVETIME = 1. Then, I run both codes on this Digi file, with the following results:
Old code |
New code |
||
---|---|---|---|
|
|
As you can see, while my code gives exactly 1 for every bin, the old code has the same behavior as with the real file, and it's clearly wrong, returning sometimes a LIVETIME greater than the elapsed time. As a further check, I repeated the same procedure imposing a dead time fraction of 0.5 (livetime=elapsed/2), obtaining the same results:
Old code |
New code |
||
---|---|---|---|
|
|
Gap handling
Procedure
I modified one Digi file and one Merit file deliberately inserting gaps, and I check if the code can find them and handle them correctly.
In particular, I downloaded from the data catalog a Digi and a Merit file for run 327356667 ("r0327356667_v001_digi.root" and "r0327356667_v120_merit.root"), and I select the events with 100000 < event ID < 700000 in both files. Then I inserted these gaps:
- I removed from the Digi file all events between 81.3 and 84.6 seconds from the time stamp of the first event in the file (a total of 2609 events removed). The last event before the gap has eventID = 145973 and the first event after the gap has eventID = 159384. I also overwritten the livetime counter making it equal to the elapsed time counter (thus, deadtime = 0). This is saved in "data/r0327356667_digi_gaps.root".
- I removed from the Merit file the events with these IDs: 199996 (which is comprised between event 199993 and event 200002), 2000031 (which is comprised between event 200029 and event 200035), and all the events with 300544 < event ID < 300858. This is saved in "data/r0327356667_v001_merit_crashes.root".
- I inserted by hand two fake gaps in the digi Gap file, excluding events in these ranges of event IDs: 129504-181395 and 273973-280562. This is saved in "data/r0327356667_v001_digiGap_2gaps.txt"
All the files with the gaps are available in the /data directory of the package, as well as the ROOT macros used to produce them. Note that they contain only the variables used in the FT2 code (to reduce their size).
The following is the output of the code used to produce the FT2 file with time bin of 1 s. As you can see, it finds all the gaps. There are a couple of "INTERPOLATION NEEDED" messages, which are expected, and correspond to the beginning and the end of the bad time intervals (which is also the beginning of the bin after the bad time interval):
Input parameters: -------------------------------------------------- Desired start time for the FT2 :327356689.88605165482 Desired stop time for the FT2 :327356838.64119803905 Magic 7 file :../data/r0327356667_v001_magic7L1.txt Output FT2 file :test.fits Input Digi file :../data/r0327356667_digi_gaps.root Input Merit file :../data/r0327356667_v001_merit_crashes.root Input Digi Gap file :../data/r0327356667_v001_digiGap_2gaps.txt Overwrite an existing FT2 file :yes Data quality flag :2 LAT configuration :1 Template for the FT2 file :/afs/slac/g/glast/ground/glastsoft/AUTO_GLAST/arch/os/SCONS_BUILDS/Optimized/ScienceTools/09-25-01/fitsGen/data/ft2.tpl Version number for the FT2 file :1 Configuration: (from file /a/wain006/g.glast.u55/giacomov/ft2/users/giacomov/ft2Util_2/v0r0p0/data/configuration.rc) -------------------------------------------------- packageName :ft2Util_2 packageVersion :v0r0p0 packageAuthor :G.Vianello packageAuthorEmail :giacomov@slac.stanford.edu JulianDateMissionStart :211845067200 NULL_TIME_DIFFERENCE :1e-06 sweepEventsDelayEnd :20000 ticksPerCommand :196 RollOver :33554432 nominalTicksToSecond :5e-08 Rx :-170 Ry :-173 Rz :-491 magic7ReadPadding :30 verify :yes deadPad :1 Magic 7 file ../data/r0327356667_v001_magic7L1.txt opened Magic 7 content: -------------------------------------------------- N. of ATT messages :1044 Time of the first ATT message (MET) :327356660 Time of the last ATT message (MET) :327356868.6 N. of ORB messages :209 Time of the first ORB message (MET) :327356660.6 Time of the last ORB message (MET) :327356868.6 First concurrent ORB and ATT messages at time (MET): 327356660.6 INFO: Ft2::Ft2():: There are 29 time bins before the requested start covered by the Magic7 file which are not needed Rebuilding Digi index... done Building Merit index... done Livetime::Livetime(): the input Digi file lacks a sweep event at the beginning. This means it does not span an entire run. Livetime input: -------------------------------------------------- N. of Digi events :110325 Time of the first Digi event (MET) :327356689.88605 ID of the first Digi event :100001 Time of the last Digi event (MET) :327356838.6412 ID of the last Digi event :699997 N. of Merit events :112874 Time of the first Merit event (MET) :327356689.88605 Time of the last Merit event (MET) :327356838.6412 Sweep events at the beginning: :0 Sweep events at the end: :0 Rebuilding Digi index... done Building Merit index... done Gap file ../data/r0327356667_v001_digiGap_2gaps.txt parsed: -------------------------------------------------- N. of gaps for this run :2 gaps: 327356697.132590115070343 - 327356709.861321747303009 327356732.6005374193191528 - 327356734.2353370785713196 Checking Merit and Digi for errors or gaps: 0 % completed 10 % completed GapHandler: gap in the Merit file between event 199993 and event 200002 You lost 1 event (these events are contained in a bad time interval, which will be excluded by downstream software) GapHandler: gap in the Merit file between event 200029 and event 200035 You lost 1 event (these events are contained in a bad time interval, which will be excluded by downstream software) 20 % completed 30 % completed GapHandler: gap in the Merit file between event 300544 and event 300858 You lost 58 event (these events are contained in a bad time interval, which will be excluded by downstream software) 40 % completed 50 % completed GapHandler: gap in the Digi file between event 429501 and event 442840 You lost 2609 event (these events are contained in a bad time interval, which will be excluded by downstream software) 60 % completed 70 % completed 80 % completed 90 % completed WARNING: There are gaps either in the Merit either in the Digi file (or both). Bad time intervals: 327356697.132590115070343 - 327356709.861321747303009 327356732.6005374193191528 - 327356734.2353370785713196 327356714.4137212038040161 - 327356714.4157781600952148 327356714.4220960736274719 - 327356714.423194944858551 327356739.1916109323501587 - 327356739.2655935287475586 327356771.1854096651077271 - 327356774.4867381453514099 Now parse the file and compute the quantities to be saved in the FT2 file: --------------------------------------------------------------------------- 0 % completed WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356689.88605 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356689.88605 Performing SLERP between 327356689.8 and 327356690 to get attitude at time 327356689.88605 (fraction: 0.43025824011911 s ) WARNING: Magic7::getMode(): no ORB message at time 327356689.88605 . Returning the last known mode (5), reported in the ORB message at time 327356689.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356689.88605 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356689.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356697.13259 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356697.13259 Performing SLERP between 327356697 and 327356697.2 to get attitude at time 327356697.13259 (fraction: 0.66295061486665 s ) WARNING: Magic7::getMode(): no ORB message at time 327356697.13259 . Returning the last known mode (5), reported in the ORB message at time 327356696.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356697.13259 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356696.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356709.86132 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356709.86132 Performing SLERP between 327356709.8 and 327356710 to get attitude at time 327356709.86132 (fraction: 0.3066086951857 s ) WARNING: Magic7::getMode(): no ORB message at time 327356709.86132 . Returning the last known mode (5), reported in the ORB message at time 327356709.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356709.86132 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356709.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.41372 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.41372 Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.41372 (fraction: 0.068606121872396 s ) WARNING: Magic7::getMode(): no ORB message at time 327356714.41372 . Returning the last known mode (5), reported in the ORB message at time 327356713.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356714.41372 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6 10 % completed WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.41578 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.41578 Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.41578 (fraction: 0.078890900876307 s ) WARNING: Magic7::getMode(): no ORB message at time 327356714.41578 . Returning the last known mode (5), reported in the ORB message at time 327356713.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356714.41578 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.4221 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.4221 Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.4221 (fraction: 0.11048046100605 s ) WARNING: Magic7::getMode(): no ORB message at time 327356714.4221 . Returning the last known mode (5), reported in the ORB message at time 327356713.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356714.4221 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356714.42319 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356714.42319 Performing SLERP between 327356714.4 and 327356714.6 to get attitude at time 327356714.42319 (fraction: 0.11597481585149 s ) WARNING: Magic7::getMode(): no ORB message at time 327356714.42319 . Returning the last known mode (5), reported in the ORB message at time 327356713.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356714.42319 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356713.6 21 % completed WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356732.60054 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356732.60054 Performing SLERP between 327356732.6 and 327356732.8 to get attitude at time 327356732.60054 (fraction: 0.002686977546631 s ) WARNING: Magic7::getMode(): no ORB message at time 327356732.60054 . Returning the last known mode (5), reported in the ORB message at time 327356732.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356732.60054 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356732.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356734.23534 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356734.23534 Performing SLERP between 327356734.2 and 327356734.4 to get attitude at time 327356734.23534 (fraction: 0.17668546299252 s ) WARNING: Magic7::getMode(): no ORB message at time 327356734.23534 . Returning the last known mode (5), reported in the ORB message at time 327356733.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356734.23534 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356733.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356739.19161 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356739.19161 Performing SLERP between 327356739 and 327356739.2 to get attitude at time 327356739.19161 (fraction: 0.9580547188553 s ) WARNING: Magic7::getMode(): no ORB message at time 327356739.19161 . Returning the last known mode (5), reported in the ORB message at time 327356738.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356739.19161 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356738.6 31 % completed WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356739.26559 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356739.26559 Performing SLERP between 327356739.2 and 327356739.4 to get attitude at time 327356739.26559 (fraction: 0.32796772289084 s ) WARNING: Magic7::getMode(): no ORB message at time 327356739.26559 . Returning the last known mode (5), reported in the ORB message at time 327356738.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356739.26559 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356738.6 41 % completed 52 % completed WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356771.18541 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356771.18541 Performing SLERP between 327356771 and 327356771.2 to get attitude at time 327356771.18541 (fraction: 0.92704838079502 s ) WARNING: Magic7::getMode(): no ORB message at time 327356771.18541 . Returning the last known mode (5), reported in the ORB message at time 327356770.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356771.18541 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356770.6 WARNING: Magic7::getPosition(): interpolation needed to get position at time 327356774.48674 ---------------------- INTERPOLATIONS NEEDED Magic7::getQuaternion(): interpolation needed to get quaternion at time 327356774.48674 Performing SLERP between 327356774.4 and 327356774.6 to get attitude at time 327356774.48674 (fraction: 0.43369074256641 s ) WARNING: Magic7::getMode(): no ORB message at time 327356774.48674 . Returning the last known mode (5), reported in the ORB message at time 327356773.6 WARNING: Magic7::getInSAA(): no ORB message at time 327356774.48674 . Returning the last known value for the in_SAA flag (0), reported in the ORB message at time 327356773.6 62 % completed 72 % completed 83 % completed 93 % completed FT2 time bins: -------------------------------------------------- N. of bins :145 Beginning (MET) :327356689.88605 End (MET) :327356838.6412
This is a plot of the livetime in the output FT2 file:
As you can see, most bins have livetime = 1. The bin corresponding to the gaps have livetime = 0, while the bins surrounding the gaps have 1 < livetime < 0.
This is a table summarizing the gap and the corresponding bin in the FT2 file:
Event before and after the gap |
Corresponding time interval |
FT2 file start-stop |
Livetime in the FT2 file |
Gap type |
---|---|---|---|---|
129504 - 181395 |
327356697.1325891 - 327356709.8613227 |
327356697.1325901 - 327356709.8613217 |
0.000000 |
Gap from the Digi gap file |
273973 - 280562 |
327356732.6005364 - 327356734.2353380 |
327356732.6005374 - 327356734.2353370 |
0.000000 |
Gap from the Digi gap file |
199993 - 200002 |
327356714.4137201 - 327356714.4157791 |
327356714.4137212 - 327356714.4157781 |
0.000000 |
Gap in the Merit |
200029 - 200035 |
327356714.4220950 - 327356714.4231959 |
327356714.4220960 - 327356714.4231949 |
0.000000 |
Gap in the Merit |
300544 - 300858 |
327356739.1916099 - 327356739.2655945 |
327356739.1916109 - 327356739.2655935 |
0.000000 |
Gap in the Merit |
429501 - 442840 |
327356771.1854086 - 327356774.4867391 |
327356771.1854096 - 327356774.4867381 |
0.000000 |
Gap in the Digi |
As you can see, everything is right.
Now, using mergeFT2.exe to rebin the FT2 file gives this:
./mergeFT2.exe -inputFT2 test.fits -outputFT2 test_30s.fits -clobber yes Input parameters: --------------------------------------------- Input FT2 file : test.fits Output FT2 file : test_30s.fits Overwrite an existing FT2 file : yes File version : 1 Time step for the output FT2 file : 30 * Entry number 1: WARNING: livetime=0 at 327356697.13259. Stopping there. 327356689.88605 ........ 327356697.13259 Elapsed time 7.24654, Livetime = 7.24654 (100 %) * Entry number 2: Starting from livetime=0 at 327356709.86132 327356697.13259 . 327356709.86132 Elapsed time 12.7287, Livetime = 0 (0 %) * Entry number 3: WARNING: livetime=0 at 327356714.41372. Stopping there. 327356709.86132 ..... 327356714.41372 Elapsed time 4.5524, Livetime = 4.5524 (100 %) * Entry number 4: Starting from livetime=0 at 327356714.41578 327356714.41372 . 327356714.41578 Elapsed time 0.00205696, Livetime = 0 (0 %) * Entry number 5: WARNING: livetime=0 at 327356714.4221. Stopping there. 327356714.41578 . 327356714.4221 Elapsed time 0.00631791, Livetime = 0.00631792 (100 %) * Entry number 6: Starting from livetime=0 at 327356714.42319 327356714.4221 . 327356714.42319 Elapsed time 0.00109887, Livetime = 0 (0 %) * Entry number 7: WARNING: livetime=0 at 327356732.60054. Stopping there. 327356714.42319 .................... 327356732.60054 Elapsed time 18.1773, Livetime = 18.1773 (100 %) * Entry number 8: Starting from livetime=0 at 327356734.23534 327356732.60054 . 327356734.23534 Elapsed time 1.6348, Livetime = 0 (0 %) * Entry number 9: WARNING: livetime=0 at 327356739.19161. Stopping there. 327356734.23534 ...... 327356739.19161 Elapsed time 4.95627, Livetime = 4.95627 (100 %) * Entry number 10: Starting from livetime=0 at 327356739.26559 327356739.19161 . 327356739.26559 Elapsed time 0.0739826, Livetime = 0 (0 %) * Entry number 11: 327356739.26559 .............................. 327356768.6 Elapsed time 29.3344, Livetime = 29.3344 (100 %) * Entry number 12: WARNING: livetime=0 at 327356771.18541. Stopping there. 327356768.6 ... 327356771.18541 Elapsed time 2.58541, Livetime = 2.58541 (100 %) * Entry number 13: Starting from livetime=0 at 327356774.48674 327356771.18541 . 327356774.48674 Elapsed time 3.30133, Livetime = 0 (0 %) * Entry number 14: 327356774.48674 .............................. 327356803.6 Elapsed time 29.1133, Livetime = 29.1133 (100 %) * Entry number 15: 327356803.6 .............................. 327356833.6 Elapsed time 30, Livetime = 30 (100 %) * Entry number 16: 327356833.6 ...... 327356838.6412 Elapsed time 5.0412, Livetime = 5.0412 (100 %)
As you can see the code stops when it encounter bins with livetime=0, and keep them isolated. Thus, even if the target bin time is 30 s, there is only one bin that long (# 15). The others are prematurely stopped because of the gaps. But, if we change the target binning to 5 s, much more bins will have the target duration:
mergeFT2.exe -inputFT2 test.fits -outputFT2 test_30s.fits -clobber yes -dt 5 Input parameters: --------------------------------------------- Input FT2 file : test.fits Output FT2 file : test_30s.fits Overwrite an existing FT2 file : yes File version : 1 Time step for the output FT2 file : 5 * Entry number 1: 327356689.88605 ..... 327356694.6 Elapsed time 4.71395, Livetime = 4.71395 (100 %) * Entry number 2: 327356694.6 ... 327356697.13259 Elapsed time 2.53259, Livetime = 2.53259 (100 %) * Entry number 3: 327356697.13259 . 327356709.86132 Elapsed time 12.7287, Livetime = 0 (0 %) * Entry number 4: WARNING: livetime=0 at 327356714.41372. Stopping there. 327356709.86132 ..... 327356714.41372 Elapsed time 4.5524, Livetime = 4.5524 (100 %) * Entry number 5: Starting from livetime=0 at 327356714.41578 327356714.41372 . 327356714.41578 Elapsed time 0.00205696, Livetime = 0 (0 %) * Entry number 6: WARNING: livetime=0 at 327356714.4221. Stopping there. 327356714.41578 . 327356714.4221 Elapsed time 0.00631791, Livetime = 0.00631792 (100 %) * Entry number 7: Starting from livetime=0 at 327356714.42319 327356714.4221 . 327356714.42319 Elapsed time 0.00109887, Livetime = 0 (0 %) * Entry number 8: 327356714.42319 ..... 327356718.6 Elapsed time 4.17681, Livetime = 4.17681 (100 %) * Entry number 9: 327356718.6 ..... 327356723.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 10: 327356723.6 ..... 327356728.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 11: 327356728.6 ..... 327356732.60054 Elapsed time 4.00054, Livetime = 4.00054 (100 %) * Entry number 12: Starting from livetime=0 at 327356734.23534 327356732.60054 . 327356734.23534 Elapsed time 1.6348, Livetime = 0 (0 %) * Entry number 13: 327356734.23534 ...... 327356739.19161 Elapsed time 4.95627, Livetime = 4.95627 (100 %) * Entry number 14: Starting from livetime=0 at 327356739.26559 327356739.19161 . 327356739.26559 Elapsed time 0.0739826, Livetime = 0 (0 %) * Entry number 15: 327356739.26559 ..... 327356743.6 Elapsed time 4.33441, Livetime = 4.33441 (100 %) * Entry number 16: 327356743.6 ..... 327356748.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 17: 327356748.6 ..... 327356753.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 18: 327356753.6 ..... 327356758.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 19: 327356758.6 ..... 327356763.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 20: 327356763.6 ..... 327356768.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 21: 327356768.6 ... 327356771.18541 Elapsed time 2.58541, Livetime = 2.58541 (100 %) * Entry number 22: Starting from livetime=0 at 327356774.48674 327356771.18541 . 327356774.48674 Elapsed time 3.30133, Livetime = 0 (0 %) * Entry number 23: 327356774.48674 ..... 327356778.6 Elapsed time 4.11326, Livetime = 4.11326 (100 %) * Entry number 24: 327356778.6 ..... 327356783.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 25: 327356783.6 ..... 327356788.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 26: 327356788.6 ..... 327356793.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 27: 327356793.6 ..... 327356798.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 28: 327356798.6 ..... 327356803.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 29: 327356803.6 ..... 327356808.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 30: 327356808.6 ..... 327356813.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 31: 327356813.6 ..... 327356818.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 32: 327356818.6 ..... 327356823.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 33: 327356823.6 ..... 327356828.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 34: 327356828.6 ..... 327356833.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 35: 327356833.6 ..... 327356838.6 Elapsed time 5, Livetime = 5 (100 %) * Entry number 36: 327356838.6 . 327356838.6412 Elapsed time 0.041198, Livetime = 0.041198 (100 %)