Jump to updates of 30 June 2006
...
Jump to updates of 29 Jun 2010
Jump to updates of 19 Aug 2008
Jump to updates of 20 Nov 2007
LAT Photons (FT1,
...
LS-002)
...
[formerly
...
Event
...
Summary
...
Data]
This is a fundamental data product used by the science tools. The current definition (version of July 25, 2006 as of this writing) is posted on the Guidelines for Science Tools Design page maintained by Masa Hirayama.
...
(Digel) This is specified as a 32-bit integer. Right now, most likely the event IDs will be assigned by the LAT as a time stamp. Most likely we will need more than 32 bits to represent them, and of course they should look more or less like some kind of integer representation of the value in TIME. Right now, I don't know what to recommend for EVENT_ID.
(Foschini) Yes, it appears that EVENT_ID is a duplication of TIME column. It can be removed.
...
Proposed resolution: Find out how the event numbers will be assigned onboard. From the ISOC-FSW ICD it isn't obvious to me. The EVENT_ID column will be distinct from time and should stay.
[anders] The extended GEM sequence counter (which is your event ID) is assigned onboard by FSW and it's a 64-bit number. Note that to define an event
uniquely you need the combination Run ID and Event ID.
7. RECON_VERSION column
(Digel) RECON_VERSION is defined as a 2-byte integer to define the version of reconstruction (and classification) algorithm applied for a given event. This is probably more general than we need. In principle, a given events file could have events processsed with different versions of the software, but in practice, I do not expect that this will happen. I recommend that we make RECON_VERSION a header variable instead of a column.
...
Proposed resolution: I'm not sure. We probably could adopt a 'v1r2p3' string for designating the version of reconstruction - say from the version of Gleam used to make the reconstruction, but I don't think that any versioning has been worked out yet for event classification.
[anders] There is also makeFT1 which is part of ScienceTools. And for completeness you may also want the version of the Halfpipe which merges and time orders the events. In any case, I suggest you explicitly use 'GlastRelease-v1r2p3' instead of just 'v1r2p3' to avoid any confusion as to what the version number refers to.
8. CALIB8. CALIB_VERSION column
(Digel) As for RECON_VERSION we do not need this as a column. Actually, I'd recommend combining it with RECON_VERSION.
Proposed resolution: Actually, on second thought, I do not recommend combining it with RECON_VERSION, but this is another example where I don't think we know yet how calibrations will be versioned. I'll at least ask the ISOC.
[anders] Currently we do not have any versioning of the calibration files. Also note that each calibration file is updated independently of any other. They will also have different validity periods. There is an ID number associated with each calibration type. This number currently only lives in the calibration meta-database.
9. IMGOODCALPROP, IMVERTEXPROB, IMCOREPROB, IMPSFERRPRED, CALENERGYSUM9. IMGOODCALPROP, IMVERTEXPROB, IMCOREPROB, IMPSFERRPRED, CALENERGYSUM, CALTOTRLN, and IMGAMMAPROB columns
...
Proposed resolution: Add the LIVETIME column.
[anders] Long story short: Only the livetime per run will be available.
14. Event summary++
(Digel) LS-002 is an event summary, intended to have information for higher-level analyses. The GSSC expects (reasonably) to receive this summary for every event that is telemetered to the ground (and that is not discarded early in the process as obviously background to save disk space at the ISOC).
...
Proposed resolution: Keep the time units as seconds. I need to look into TIERRELA and TIERABSO still.
16. Name of tables
(Foschini) Presently there is only one keyword to identify the template of EVENTS and GTI data structure. To take into account that templates can change (particularly during the first months after the launch), it is better to add a keyword EXTVERS or something similar with the version number of the used template. Moreover, the data structure of LAT and GBM can require different keywords, so I propose to give at the EXTNAME more complicate names to take into account the different uses: for example, for the LAT events there can be EXTNAME='GLAST-LAT_EVT' and the corresponding GTI can be EXTNAME='GLAST-LAT-GTI'.
(Ballet) I don't think it is a good idea to change the names. The INSTRUME and TELESCOP keywords are there to indicate which intruments we are dealing with. Better keep standard names for the EVENTS and GTI extensions, as was done for previous missions such as Chandra or XMM-Newton. I agree to the EXTVERS suggestion.
Proposed resolution: Keep the extension names as EVENTS and GTI.
17. GTI
(Foschini) I suggest to add a keyword GTI_NAME to the GTI header to explain the origin of GTI: for example, there could be GTI due to attitude, telemetry, or other. The final GTI can be a merging of the whole types of GTI.
The keyword TELAPSE in GTI should indicate the elapsed time of the whole observation (see the notes at n. 15 Time Issues) and not the difference of GTI STOP-START.
I suggest also to add two more columns in GTI table, OBTSTART,OBTSTOP, with the onboard time corresponding to START,STOP columns.
Perhaps it could be useful also to add two more columns with START and STOP in UTC.
Note: the keyword HDUCLAS2 for GTI header (if kept, see n. 2 HDUCLASS keywords) I think should be 'STANDARD'.
(Hirayama) About the TELAPSE keyword, I remember somebody (most likely a member of the HEASARC FITS Working Group) told us the definition is "time between START of the first GTI and STOP of the last," which is the current FT1 definition. However, I couldn't find such statement on the web. Instead, HFWG Recommendation R11 states "TELAPSE is the time interval (in seconds) obtained as difference between the start and stop times of an observation." It sounds like they assume a pointed observation, as usual, where the assumption is not quite appropriate for GLAST. My guess is that, when they explained the definition to us, they gave us a traditional explanation that works for pointed observations, without much consideration on a continuously scanned observation that GLAST will perform, although we should confirm it with the HEASARC people before we conclude so.
If my understanding is correct, I would agree that TELAPSE = TSTOP - TSTART and should apppear in an EVENTS extension, too. In the current definition, it shows up only in a GTI extension because it is defined as a derived quantity from the contents of a GTI extension.
For GTI_NAME, if we need it at all, it should cooperate with DSS keywords, I think. In order to compute and revise GTI's upon data subselection, not all DSS keywords will be taken into account of. So, one solution could be (although I don't think it is pretty) to list DSS keyword numbers that are used to compute GTI's in GTI_NAME keyword value.
For OBTSTART and OBTSTOP, I don't think it is a good idea. Noting GTI's will change upon data subselection, a data subselection tool must update OBTSTART and OBTSTOP keyword values, too. The GLAST tools can be modified to do so, but external tools such as XSELECT will not. At the least, that necessitates a special handling of a GLAST FT1 file, wihch is different from event files of other astrophysics missions.
I think we should adhere to a standard GTI format, unless a deviation from it will significantly improve users' benefits.
For START and STOP in UTC, I simply don't know how to do it. A moment in time in the UTC system cannot be expressed by a single number because of leap seconds, if I understand correctly. Also, for the same reason as for OBTSTART and OBTSTOP, I don't think inventing a new GTI extension format is not a good idea, either.
(Foschini) The OBTSTART,OBTSTOP keywords are useful only in case of problems, rather than for a direct use. That is, if there are problems in time correlation or anything else, the only way to reconstruct on ground the events sequence is to have the on board time (that is the only direct measurement of time of an event) and restart again the conversion to user friendly time values. So it is just to have a backup option.
[anders] Minor detail: The start and end time of the run (ultimately deciding the livetime) may not be the time of the first and last events. The reason is that the instrument is alive for a short time before the first event and also after the last event. This is not captured in the livetime counters. I may have a solution to this using 'sweep' events which are issued just before and after the first and last physics triggers. The advantage is that there is a fixed and calculable time between the sweep event and when the instrument is alive/dead (independently of when the first trigger arrives).
16. Name of tables
(Foschini) Presently there is only one keyword to identify the template of EVENTS and GTI data structure. To take into account that templates can change (particularly during the first months after the launch), it is better to add a keyword EXTVERS or something similar with the version number of the used template. Moreover, the data structure of LAT and GBM can require different keywords, so I propose to give at the EXTNAME more complicate names to take into account the different uses: for example, for the LAT events there can be EXTNAME='GLAST-LAT_EVT' and the corresponding GTI can be EXTNAME='GLAST-LAT-GTI'.
(Ballet) I don't think it is a good idea to change the names. The INSTRUME and TELESCOP keywords are there to indicate which intruments we are dealing with. Better keep standard names for the EVENTS and GTI extensions, as was done for previous missions such as Chandra or XMM-Newton. I agree to the EXTVERS suggestion.
Proposed resolution: Keep the extension names as EVENTS and GTI.
17. GTI
(Foschini) I suggest to add a keyword GTI_NAME to the GTI header to explain the origin of GTI: for example, there could be GTI due to attitude, telemetry, or other. The final GTI can be a merging of the whole types of GTI.
The keyword TELAPSE in GTI should indicate the elapsed time of the whole observation (see the notes at n. 15 Time Issues) and not the difference of GTI STOP-START.
I suggest also to add two more columns in GTI table, OBTSTART,OBTSTOP, with the onboard time corresponding to START,STOP columns.
Perhaps it could be useful also to add two more columns with START and STOP in UTC.
Note: the keyword HDUCLAS2 for GTI header (if kept, see n. 2 HDUCLASS keywords) I think should be 'STANDARD'.
(Hirayama) About the TELAPSE keyword, I remember somebody (most likely a member of the HEASARC FITS Working Group) told us the definition is "time between START of the first GTI and STOP of the last," which is the current FT1 definition. However, I couldn't find such statement on the web. Instead, HFWG Recommendation R11 states "TELAPSE is the time interval (in seconds) obtained as difference between the start and stop times of an observation." It sounds like they assume a pointed observation, as usual, where the assumption is not quite appropriate for GLAST. My guess is that, when they explained the definition to us, they gave us a traditional explanation that works for pointed observations, without much consideration on a continuously scanned observation that GLAST will perform, although we should confirm it with the HEASARC people before we conclude so.
If my understanding is correct, I would agree that TELAPSE = TSTOP - TSTART and should apppear in an EVENTS extension, too. In the current definition, it shows up only in a GTI extension because it is defined as a derived quantity from the contents of a GTI extension.
For GTI_NAME, if we need it at all, it should cooperate with DSS keywords, I think. In order to compute and revise GTI's upon data subselection, not all DSS keywords will be taken into account of. So, one solution could be (although I don't think it is pretty) to list DSS keyword numbers that are used to compute GTI's in GTI_NAME keyword value.
For OBTSTART and OBTSTOP, I don't think it is a good idea. Noting GTI's will change upon data subselection, a data subselection tool must update OBTSTART and OBTSTOP keyword values, too. The GLAST tools can be modified to do so, but external tools such as XSELECT will not. At the least, that necessitates a special handling of a GLAST FT1 file, wihch is different from event files of other astrophysics missions.
I think we should adhere to a standard GTI format, unless a deviation from it will significantly improve users' benefits.
For START and STOP in UTC, I simply don't know how to do it. A moment in time in the UTC system cannot be expressed by a single number because of leap seconds, if I understand correctly. Also, for the same reason as for OBTSTART and OBTSTOP, I don't think inventing a new GTI extension format is not a good idea, either.
(Foschini) The OBTSTART,OBTSTOP keywords are useful only in case of problems, rather than for a direct use. That is, if there are problems in time correlation or anything else, the only way to reconstruct on ground the events sequence is to have the on board time (that is the only direct measurement of time of an event) and restart again the conversion to user friendly time values. So it is just to have a backup option.
For UTC keywords, these are generally expressed as a string: e.g. "2005-11-22T11:43:00". In this case, it is just for the end user, to have something more friendly than JD or anything else. To make the conversion, it is used a time correlation, that is a function that linearly correlates the UTC to the onboard time.For UTC keywords, these are generally expressed as a string: e.g. "2005-11-22T11:43:00". In this case, it is just for the end user, to have something more friendly than JD or anything else. To make the conversion, it is used a time correlation, that is a function that linearly correlates the UTC to the onboard time.
Proposed resolution: There's a lot here. I don't feel strongly about TELAPSE but I recommend that we do not adopt it because as Masa pointed out, it is not so relevant for scanning observations. I need to think more about OBTSTART and OBTSTOP. I am reminded that according to the letter of the law for GTIs, our servers should return corresponding GTIs for each query.
...
Proposed resolution: Include EVENT_CLASS as Jim describes in place of CONVERSION_LAYER and as a way to encode other classes that we come up with. After discussion with Jim, I think we should also have a column called something like CONVERSION that would indicate, e.g., by value 1,2, or 3 whether the event is FRONT, BACK, or CAL-ONLY. This would be in addition to EVENT_CLASS, although until we have other ways defined to discriminate other kinds of response functions it will have the same value.
----
| Anchor | ||||
|---|---|---|---|---|
|
(30 June 2006) Here are a few more issues that have arisen in the run-up to Ground Readiness Test 5, which has the ISOC transferring a realistic FT1 file to the GSSC, who will verify that it is consistent in every testable way with the detailed specifications in the Science Data Products File Format Document (Word file). The FFD has not been finalized and several loose ends have been recognized regarding the specification of the FT1 headers and columns.
...
These issues are resolved. See STGEN-30. Wiki Markup
| Anchor | ||||
|---|---|---|---|---|
|
...
20 November 2007
The current version of the FT1 template is here. Since the last update 13 months ago we have gained more experience with the kinds of analyses we be making with the FT1 files and the L1 pipeline has reached an advanced stage of development. Some aspects of the FT1 files need to be updated as a result.
23. GPS_OUT
(Digel) In the current iteration of the definition, we had assumed that whether or not the absolute times are based on the 1 pulse per second signal being synched with the spacecraft's GPS clock could be indicated with one flag in the header of the file. Anders points out that we really should be carrying this as a column in the FT1 files because the answer can be different for different events. GPS_OUT as a header keyword could be interpreted to indicate whether any event in the FT1 file had a time based on the internal oscillator, but that in itself would not be particularly informative.
For reasons that I don't quite grasp right now, correcting event times after GPS lock has been re-established for oscillator drift during periods when lock was lost is not really practical. Also, except in extreme circumstances (e.g., the fastest ms pulsar considered over a very long time range) the potential drifts are probably negligible. So the GPS_OUT flag being set for a given event would be just a warning to be careful.
Anders also suggests that we consider including the number of seconds since GPS lock was lost as a way of allowing users to estimate the relative qualities of the time accuracy. These times could range up to thousands of seconds in the worst case
I propose that we make GPS_OUT be a column as a 2-byte integer; 0 means that the event time was assigned while the GPS system was in lock. Values >=1 indicate the number of seconds since lock was lost. Anders points out that this quantity is always an integer, the number of 1 pulse-per second counts lost.
24. Replacing RECON_VERSION and CALIB_VERSION
(Digel) These columns are 2-byte and 3x 2-byte quantities that were intended to represent the version of Gleam and the versions of the ACD, CAL, and TKR calibrations (however exactly they are defined) that were used in the processing.
I propose to replace these with a single 2-byte integer PIPELINE_VERSION that is indexed to the configuration of the L1 pipeline (versions of Gleam and other software and versions of the calibrations) that were active at the time that a given event was run through the pipeline. When the L1 pipeline is under configuration control, these versions will be carefully tracked.
Anders: "We don't really have such a uber-version number yet. But we do need one."
With reprocessing, we'll want a way to tell whether a given event or file of events are in their most current processing versions. We could maintain a table or graph of most current PIPELINE_VERSION vs. event time.
25. Revisiting CONVERSION_TYPE and EVENT_CLASS
(Digel) These are 2-byte integer columns that define whether an event converted in the front or back section of the tracker and what event class the event satisfies.
As we currently manage the response functions, front and back are de facto different event classes. The EVENT_CLASS column is intended to make other distinctions, such as classes A and B for DC2. Right now for Pass5 we have still another column (CTBCLASSLEVEL) that is used to specify the event classes - so EVENT_CLASS is not actually being used.
What should do here depends in part on whether we rearrange the Pass 5 IRFs so that each event can be a member of one and only one event class; currently that is not the case.
In the future we can look forward to having still different sets of response functions - obviously - that will correspond to different versions of reconstruction and classification. In principle we could keep incrementing EVENT_CLASS (if we had a table someplace that mapped the response functions and processing versions into EVENT_CLASS values). Or we could have some kind of string column related to the name of the IRF that the event qualifies for. Or we could not try to protect people from themselves and let EVENT_CLASS have different meanings for different sets of response functions (possibly including a header keyword to name the response function family that applies to the file).
I don't have a recommendation about which option would work best.
| Anchor | ||||
|---|---|---|---|---|
|
...
19 August 2008
26. Including GEOMAG_LAT
(Digel) This is a quantity that depends on position of the spacecraft and time, and so years ago was relegated to the FT2 file. The thinking at the time was that filtering on geomagnetic quantities could be done with GTIs using gtmktime. Now that geomagnetic latitude appears to be of more direct interest to the analysis, having it available in FT1 files would be a convenience. The quantity would not have to be geomagnetic latitude per se - the current implementation derives it from McIlwain L - but either one or the other should be in the FT1 files.
(Jim C) If this is needed for correlation studies with other FT1 quantities, then I suppose there is a reason to have it here, although it would make more sense just to use the merit file directly. However, if this is needed for making selections for ranges of geomagnetic latitude for use in analyses with IRF-based analyses, whether using the ScienceTools or not, then one really should use gtmktime, since the GTIs enable one to compute the livetimes.
(McEnery) Geomagnetic latitude is already included in the event data definition (which is FT1 + a bunch of extra variables). Someone who wants to work with a fits file (and the FT1 quantities) could just use the event data directly.
| Anchor | ||||
|---|---|---|---|---|
|
...
29 June 2010
27. Including PROC_VER keyword
(Digel) This is summarizing what was worked out between Maria-Elena and Don Horner to allow the FSSC to handle receiving reprocessed data without interrupting ingest of output from the L1 pipeline. This 'dual ingest pipeline' scheme will allow reprocessed data to be transferred in the background, for little or no downtime when a switch is made to a new version. Incidentally the naming scheme of the FT1 files is also being modified to include the processing version as part of the name; that change is not covered here but will be documented in the File Format Document. The addition would go in LS1 as well. The keyword will go into the primary headers of the files
The value starts at 0 and will be incremented (perhaps not necessarily sequentially) in reprocessings.
This has been implemented in the File Format Document
28. Including PASS_VER keyword
(Digel) This keyword would have a numerical value indicating the analysis pass used to classify the events (e.g., Pass 6 or Pass 7.3), and would be used by the Science Tools to avoid mixing FT1 files with different analysis pass values. Like the above, this keyword would be implemented in a reprocessing. Initial thinking had been that this would need to be a new column, but Jim is thinking that having the Science Tools make a consistency check using just a header keyword (e.g., if given a list of input files) would not be difficult. This keyword would be set by makeFT1 and the value to assign would be passed to makeFT1. It would go in the EVENTS extensions of FT1 (and also LS1) files
(JC, 09Jul2010) This keyword will probably have a string value rather than be a number. It will be taken from the xml definition for the event classes.
This has been implemented in the File Format Document (22 July 2010 update, pdf to current version . Note the complete lack of version control on this site).
| Anchor | ||||
|---|---|---|---|---|
|
...
22 July 2010
27. Making EVENT_CLASS a long integer
(Digel) This is just to note that for Pass 7.3, EVENT_CLASS will need to be changed to a 4-byte integer (J in FITS binary table format) from a 2-byte integer (I). This is because in Pass 7.3 the event class designator will become a bit mask in order to allow complete flexibility in specifying which classes each event belongs to. With Pass 7.3, the event classes will not all be nested, so simple range selections on EVENT_CLASS will no longer work.
This has been implemented in the File Format Document (22 July 2010 update, pdf to current version . Note the complete lack of version control on this site).