...
(Hirayama) I don't remember why those keywords were introduced, but there are some records in topic 009 and topic 010 of the latest topics of FT1 and FT2, which might be of your interest.
5. MC_TRUTH keyword
(Digel) I think that the original intent was that this keyword indicate whether the data are entirely Monte Carlo truth values (i.e., actual directions, energies, etc.). We do not actually have files like this, and I propose removing this keyword. gtobsssim does add a column MC_SRC_ID to the files it generates, but this is easy enough to check for without the MC_TRUTH keyword.
(Stephens) Actually I thought this was a flag like the old PSR_COLS keyword and indicated that there were additional MC columns in the file. As we don't use this I think it can go away.
(Hirayama) Adding a new column to an FT1 file for a special case may not be appreciated by the HEASARC people. I remember they don't like "variants" of an FT1 file, mainly for archival purposes. I think their discussion was like: "for the GLAST team members (who know the file contents very well), it is just one additional column, but for the HEASARC members (who do NOT want to deal with the file contents as much as possible), it is a different file format." Probably it is better to talk to them about it again, once we have our conclusion about this topic, rather than trusting my rusty memory.
6. EVENT_ID column
(Digel) This is specified as a 32-bit integer. Right now, most likely the event IDs will be assigned by the LAT as a time stamp. Most likely we will need more than 32 bits to represent them, and of course they should look more or less like some kind of integer representation of the value in TIME. Right now, I don't know what to recommend for EVENT_ID.
(Foschini) Yes, it appears that EVENT_ID is a duplication of TIME column. It can be removed.
(Hirayama) The topic is also outlined in topic 010 as well as other range problems that might interest you. Also, note that it may not be very easy to identify an event by TIME column after gtbary overwrites its contents for a barycentric correction.
7. RECON_VERSION column
(Digel) RECON_VERSION is defined as a 2-byte integer to define the version of reconstruction (and classification) algorithm applied for a given event. This is probably more general than we need. In principle, a given events file could have events processsed with different versions of the software, but in practice, I do not expect that this will happen. I recommend that we make RECON_VERSION a header variable instead of a column.
Also, somebody needs to think of how we will translate versions of the geometry, calibration files, reconstruction, and classification, into some fairly compact representation. Certainly we will be closely keeping track of this information for Level 1 processing, and will have a database someplace that can tell us.
8. CALIB_VERSION column
(Digel) As for RECON_VERSION we do not need this as a column. Actually, I'd recommend combining it with RECON_VERSION.
9. IMGOODCALPROP, IMVERTEXPROB, IMCOREPROB, IMPSFERRPRED, CALENERGYSUM, CALTOTRLN, and IMGAMMAPROB columns
(Digel) These columns relate to the classifications of the events and date to DC1, in particular to implementing the Atwood cuts for background rejection. Many things came together at once in the runup to DC1 and we ended up incorporating each of these variables in the event summary file. The DC1 response functions were derived post-Atwood cuts.
For DC2, we will have a cleaner way to specify the results of the event classification, although I do not know what it is yet. We will have some kind of analog of IMGAMMAPROB. Ideally, we'll also have distilled other classifications into a few sets that map into response functions that we'll provide (e.g., front, back, and cal-only events, with good_pdf, good_energy, or dont_care).
What do you recommend here?
Speaking of classification, will we also have a flag like 'HEAVY_CR'?
(Foschini) The basic problem of these and other keywords is to define the boundaries of the data that are saved into a certain file. After a fruitfull email exchange with Seth and Julie about the observing modes, I suggest that it can be useful to separate the data obtained from slews and those from pointings. Also in survey mode, the spacecraft is not continuously slewing, but rocking between two pointings, lasting one orbit on each pointing. The pointing direction can change, but what is important is that also in survey mode the spacecraft is pointing toward a certain direction for one orbit. On the other hand, independently on the reason for which the spacecraft is moving (slew, repoint, other), the data acquired in this mode require a different treatment with respect to the data from pointings. So, we have a sequence of pointings-slews-pointings-slews... both in survey and GO modes.
I think that it could be useful to use the change between pointing and slew as boundary for the files: in this way, a lot of keywords, like RA_SCX, DEC_SCX, RA_SCZ, DEC_SCZ, OBJECT and even OBS_ID, are automatically defined (for slews these keywords could refer to the middle value), and so that also for time keywords. This could be also useful to avoid an excessive load of the hardware for the science analysis: several single small files are better than one or few single huge files.
Please note that the RA_SCX, DEC_SCX, RA_SCZ, DEC_SCZ keywords refer to the spacecraft (i.e. star tracker), but the boresight of the LAT should be calculated by applying a rototranslation from the star tracker position. The latter can be always improved, so that it is always better to have the starting point, that is the position from the star tracker.
5. MC_TRUTH keyword
(Digel) I think that the original intent was that this keyword indicate whether the data are entirely Monte Carlo truth values (i.e., actual directions, energies, etc.). We do not actually have files like this, and I propose removing this keyword. gtobsssim does add a column MC_SRC_ID to the files it generates, but this is easy enough to check for without the MC_TRUTH keyword.
(Stephens) Actually I thought this was a flag like the old PSR_COLS keyword and indicated that there were additional MC columns in the file. As we don't use this I think it can go away.
(Hirayama) Adding a new column to an FT1 file for a special case may not be appreciated by the HEASARC people. I remember they don't like "variants" of an FT1 file, mainly for archival purposes. I think their discussion was like: "for the GLAST team members (who know the file contents very well), it is just one additional column, but for the HEASARC members (who do NOT want to deal with the file contents as much as possible), it is a different file format." Probably it is better to talk to them about it again, once we have our conclusion about this topic, rather than trusting my rusty memory.
6. EVENT_ID column
(Digel) This is specified as a 32-bit integer. Right now, most likely the event IDs will be assigned by the LAT as a time stamp. Most likely we will need more than 32 bits to represent them, and of course they should look more or less like some kind of integer representation of the value in TIME. Right now, I don't know what to recommend for EVENT_ID.
(Foschini) Yes, it appears that EVENT_ID is a duplication of TIME column. It can be removed.
(Hirayama) The topic is also outlined in topic 010 as well as other range problems that might interest you. Also, note that it may not be very easy to identify an event by TIME column after gtbary overwrites its contents for a barycentric correction.
7. RECON_VERSION column
(Digel) RECON_VERSION is defined as a 2-byte integer to define the version of reconstruction (and classification) algorithm applied for a given event. This is probably more general than we need. In principle, a given events file could have events processsed with different versions of the software, but in practice, I do not expect that this will happen. I recommend that we make RECON_VERSION a header variable instead of a column.
Also, somebody needs to think of how we will translate versions of the geometry, calibration files, reconstruction, and classification, into some fairly compact representation. Certainly we will be closely keeping track of this information for Level 1 processing, and will have a database someplace that can tell us.
8. CALIB_VERSION column
(Digel) As for RECON_VERSION we do not need this as a column. Actually, I'd recommend combining it with RECON_VERSION.
9. IMGOODCALPROP, IMVERTEXPROB, IMCOREPROB, IMPSFERRPRED, CALENERGYSUM, CALTOTRLN, and IMGAMMAPROB columns
(Digel) These columns relate to the classifications of the events and date to DC1, in particular to implementing the Atwood cuts for background rejection. Many things came together at once in the runup to DC1 and we ended up incorporating each of these variables in the event summary file. The DC1 response functions were derived post-Atwood cuts.
For DC2, we will have a cleaner way to specify the results of the event classification, although I do not know what it is yet. We will have some kind of analog of IMGAMMAPROB. Ideally, we'll also have distilled other classifications into a few sets that map into response functions that we'll provide (e.g., front, back, and cal-only events, with good_pdf, good_energy, or dont_care).
What do you recommend here?
Speaking of classification, will we also have a flag like 'HEAVY_CR'?
((Burnett) The "IM" prefix was inserted by me when I absorbed Bill's values. Since it stands for Insightful Miner, and it is not clear that we will always use exactly Bill's trees for this, I would suggest that we drop it from the names. "COREPROB" is confusing, it really means goodpsfprob. And there is no longer anything to correspond to IMPSFERRPED.
...
(Hirayama) About the TELAPSE keyword, I remember somebody (most likely a member of the HEASARC FITS Working Group) told us the definition is "time between START of the first GTI and STOP of the last," which is the current FT1 definition. However, I couldn't find such statement on the web. Instead, HFWG Recommendation R11 states "TELAPSE is the time interval (in seconds) obtained as difference between the start and stop times of an observation." It sounds like they assume a pointed observation, as usual, where the assumption is not quite appropriate for GLAST. My guess is that, when they explained the definition to us, they gave us a traditional explanation that works for pointed observations, without much consideration on a continuously scanned observation that GLAST will perform, although we should confirm it with the HEASARC people before we conclude so.
If my understanding is correct, I would agree that TELAPSE = TSTOP - TSTART and should apppear in an EVENTS extension, too. In the current definition, it shows up only in a GTI extension because it is defined as a derived quantity from the contents of a GTI extension.
For GTI_NAME, if we need it at all, it should cooperate with DSS keywords, I think. In order to compute and revise GTI's upon data subselection, not all DSS keywords will be taken into account of. So, one solution could be (although I don't think it is pretty) to list DSS keyword numbers that are used to compute GTI's in GTI_NAME keyword value.
the HEASARC FITS Working Group) told us the definition is "time between START of the first GTI and STOP of the last," which is the current FT1 definition. However, I couldn't find such statement on the web. Instead, HFWG Recommendation R11 states "TELAPSE is the time interval (in seconds) obtained as difference between the start and stop times of an observation." It sounds like they assume a pointed observation, as usual, where the assumption is not quite appropriate for GLAST. My guess is that, when they explained the definition to us, they gave us a traditional explanation that works for pointed observations, without much consideration on a continuously scanned observation that GLAST will perform, although we should confirm it with the HEASARC people before we conclude so.
If my understanding is correct, I would agree that TELAPSE = TSTOP - TSTART and should apppear in an EVENTS extension, too. In the current definition, it shows up only in a GTI extension because it is defined as a derived quantity from the contents of a GTI extension.
For GTI_NAME, if we need it at all, it should cooperate with DSS keywords, I think. In order to compute and revise GTI's upon data subselection, not all DSS keywords will be taken into account of. So, one solution could be (although I don't think it is pretty) to list DSS keyword numbers that are used to compute GTI's in GTI_NAME keyword value.
For OBTSTART and OBTSTOP, I don't think it is a good idea. Noting GTI's will change upon data subselection, a data subselection tool must update OBTSTART and OBTSTOP keyword values, too. The GLAST tools can be modified to do so, but external tools such as XSELECT will not. At the least, that necessitates a special handling of a GLAST FT1 file, wihch is different from event files of other astrophysics missions.
I think we should adhere to a standard GTI format, unless a deviation from it will significantly improve users' benefits.
For START and STOP in UTC, I simply don't know how to do it. A moment in time in the UTC system cannot be expressed by a single number because of leap seconds, if I understand correctly. Also, for the same reason as for For OBTSTART and OBTSTOP, I don't think it is inventing a new GTI extension format is not a good idea. Noting GTI's will change upon data subselection, a data subselection tool must update OBTSTART and OBTSTOP keyword values, too. The GLAST tools can be modified to do so, but external tools such as XSELECT will not. At the least, that necessitates a special handling of a GLAST FT1 file, wihch is different from event files of other astrophysics missions.
I think we should adhere to a standard GTI format, unless a deviation from it will significantly improve users' benefits.
, either.
(Foschini) The OBTSTART,OBTSTOP keywords are useful only in case of problems, rather than for a direct use. That is, if there are problems in time correlation or anything else, the only way to reconstruct on ground the events sequence is to have the on board time (that is the only direct measurement of time of an event) and restart again the conversion to user friendly time values. So it is just to have a backup option.
For UTC keywords, these are generally expressed as a string: e.g. "2005-11-22T11:43:00". In this case, it is just for the end user, to have something more friendly than JD or anything else. To make the conversion, it is used a time correlation, that is a function that linearly correlates the UTC to the onboard timeFor START and STOP in UTC, I simply don't know how to do it. A moment in time in the UTC system cannot be expressed by a single number because of leap seconds, if I understand correctly. Also, for the same reason as for OBTSTART and OBTSTOP, I don't think inventing a new GTI extension format is not a good idea, either.