Science Tools Working Group
The current release of the Science Tools is now v9r11.
Here are the differences from v9r10. v9r11 includes the Pass 6 v3 IRFS that include the effect of ghosts and some additional functionality described below.
Data products: Reprocessing is still in planning.
Databases and related utilities
In you use the AstroServer, you should have noted Tony's note this week about the 4 runs previously flagged as bad have been removed from the server.
Likelihood analysis
From Jim, regarding new functionality in v9r11 of the Science Tools:
- phi-dependence in the Likelihood packages. Note that none of the IRFs have the phi-dependence in them yet, and when they do, it will only be for the effective area
- At Jean's request, Pat and I have added access to the MINOS functionality for the Minuit and NewMinuit optimizers. This will use the profile likelihood method for determining 1-sigma confidence intervals. The interface still needs some development, but knowledgeable users (e.g., Johann C.-T., et al) can make use of it.
GRB tools
No news
Pulsar tools
Masa reports continuing to work on updated documentation for the pulsar tools. As of Monday evening he was ready to start updating the User Workbook.
No news yet about how and whether D4 and the pulsar tools will be modified for handling the real-life ephemeridies that the Bordeaux group has been receiving from the pulsar timing consortium; for some pulsars very-high order time derivatives are included.
Observation simulation
Jim reports that v9r11 includes the previously-reported "fix to astro wherein the unneeded IGRF calculations are skipped so that gtobssim runs a factor of 3 faster."
User interface and infrastructure (& utilities)
No news
Source Catalog
Last week most of the discussion was related to the 6-month source list that Jean has developed and a presentation by Keith on monitoring cumulative source significances.
Science Tools Working Group
The current release of the Science Tools remains v9r10.
Eric Winter reports that he is working on an FSSC port of v9r10. Part of this is to streamline the process.
An incremental release of the Science Tools is likely soon, to include the Pass 6 v3 IRFS that include the effect of ghosts. This does not mean that these IRFs have been blessed for publishing results but it is time to shake them down. Some early results were presented in the C&A meeting yesterday.
Data products: No new news about reprocessing; Richard scheduled a meeting on reprocessing planning ('many-ringed circus') later this morning.
Databases and related utilities
No news
Likelihood analysis
No news
GRB tools
No news
Pulsar tools
From Masa: "Nothing new in the pulsar tools development. I am still writing a new document describing what the pulsar tools do in detail."
Last week saw some discussion about how the real-life radio ephemerides provided by the timing consortium could be handled by the Science Tools (pulsar tools and ephemeris database). It turns out that at least some pulsars are coming in with timing solutions with several frequency derivatives. D4 can handle up to 2nd derivatives only. The pulsar group has worked around this with TEMPO 2.
Observation simulation
No news
User interface and infrastructure (& utilities)
No news
Source Catalog
Last week most of the discussion was related to the 6-month source list that Jean has developed and a first look at source associations.
Science Tools Working Group
The current release of the Science Tools remains v9r10.
Eric Winter has sent some news regarding the FSSC distribution of Science Tools. Their distribution is based on v9r8p2. They are getting feedback (bug reports so far related to the FSSC distributions only). The list of platforms they are supporting with binary distributions is growing:
SL 4.4 32-bit
SL 4.4 64-bit
SL 5.2 32-bit
SL 5.2 64-bit
OS X 10.4 PPC
OS X 10.4 Intel
OS X 10.5 PPC
OS X 10.5 Intel
CentOS 5.2 32-bit
Debian 5.0 32-bit
Fedora 10.0 32-bit
OpenSUSE 11.1 32-bit
Ubuntu 8.04 32-bit
Ubtuntu 8.10 32-bit
Ubuntu 8.04 64-bit
Ubtuntu 8.10 64-bit
See the Science Tools update for Feb. 10 for a link to the distribution page at the FSSC.
Data products: No new news about reprocessing.
Databases and related utilities
No news
Likelihood analysis
Jim is maintaining a kind of diary of Likelihood Usage Notes that you should pay attention to. The updates for this week relate to likelihood analysis with diffuse sources. pyLikelihood can now provide fluxes, energy fluxes, and upper limits for diffuse sources. Also, the diffuse response calculation (gtdiffrsp) is now accurate for discrete diffuse sources defined via spatial templates. The user needs to take care to make the image size commensurate with the diffuse source. These updates are in Likelihood v14r3 and pyLikelihood v1r10p1 (not yet in a release).
GRB tools
No news
Pulsar tools
No news
Observation simulation
No news
User interface and infrastructure (& utilities)
No news
Source Catalog
Last week most of the discussion was related to the generation of the 6-month source list that Jean is starting to undertake.
Science Tools Working Group
The current release of the Science Tools remains v9r10. Nico Giglietto has reported a problem with installing rh9_gcc32 and rh9_gcc32opt builds of this release - under investigation; Navid reports that it is probably just a download error.
Data products: No new news about reprocessing. Jean has recommended that the data products include the version of the Science Tools that generated them; this may be a use for the CREATOR keyword that is part of the specification.
Databases and related utilities
No news
Likelihood analysis
No news.
GRB tools
No news
Pulsar tools
No news - Masa is working on improvements to the documentation for the pulsar tools.
Observation simulation
From Jim: "Since gtobssim doesn't make use of the magnetic coordinate-related quantities computed in the astro::EarthCoordinate class, I asked Leon to modify that class so that the IGRField::compute(...) function is not called unless those quantities are explicitly requested. As a result of these changes, gtobssim runs about a factor of 3 faster. (astro v3r7p1, ST HEAD1.717)"
User interface and infrastructure (& utilities)
No news
Source Catalog
Source variability was the topic last week. Toby also described an investigation of the shapes of source location regions. Gino presented a first look at the 6-month sources.
Science Tools Working Group
The current release of the Science Tools is v9r10. Here are the differences from v9r9. This is an incremental release but if you are a user of likelihood you will want to switch to the new release.
The first public distribution of the Science Tools was released on Feb. 6 by the FSSC. You will see that they have binary distributions for several versions of Linux and Mac OS X. Eric W. reports that Ubuntu builds have also just been made. The FSSC distribution is based on v9r8p2 of the Science Tools; the public distribution is a subset of the packages of our builds.
Data products: No new news about reprocessing
Databases and related utilities
No news
Likelihood analysis
v9r10 includes Steve Fegan's speed-up of the evaluations of the likelihood function. Speed-ups seem to be by a factor of ~2. This is really like something for nothing, and Jim points out that it should be especially noticeable in gtfindsrc.
Jim reports that it also includes the fix for the gtsrcmaps offset issue found by Jean Ballet, who found that the diffuse model for the Galactic plane was displaced by ~0.5 pixel after convolution in gtsrcmaps.
And he reports that v9r10 addresses interface requests from Jean for pyLikelihood. This includes allowing users to select plot colors and line styles.
GRB tools
No news
Pulsar tools
No news - Masa is still working on other FSSC tasks.
Observation simulation
No news
User interface and infrastructure (& utilities)
Jim points out that v9r10 includes a new version of the tip package that allows large (>2^31 row) FITS tables, if you want to use files that large. This is not currently working on the Windows builds.
Jim also points out that v9r10 includes handling of phi-dependence of IRFs in irfs/latResponse and irfs/handoff_response. Azimuth dependence is not currently included in IRFs distributed with the Science Tools, or in the evaluation of livetime cubes, but that is coming.
Source Catalog
The Bright Source List paper is out and the main table is available in various forms from the FSSC.
Science Tools Working Group
The current release of the Science Tools remains v9r9. The first public distribution of the Science Tools will be released on Feb. 6 by the FSSC. I believe that these are based on v9r8p2 of the Science Tools; the public distribution is a subset of the packages of our builds. Again, the FSSC distributions will support a wider variety of platforms.
Data products: Still not much news about the reprocessing; Tom G. is implementing and testing a pipeline task for reprocessing (at the Merit level, I believe) for Pass 7.
Databases and related utilities
No news
Likelihood analysis
No news
GRB tools
No news
Pulsar tools
No news - Masa has been working on other FSSC-based tasks over the last week.
Observation simulation
No news
User interface and infrastructure (& utilities)
In the C&A meeting yesterday, Riccardo gave an update on Pass 6 v3 IRFs, which will include phi dependence of the effective area (at least).
Source Catalog
Bright Source List day and night.
Science Tools Working Group
The current release of the Science Tools remains v9r9. Chris reports that at the FSSC the Science Tools to be publicly released on Feb. 6 are undergoing internal testing. The FSSC distributions will support a wider variety of platforms. One outstanding issue that you may have seen in JIRA relates to segmentation faults with the optimizers in some 64-bit builds; John, Pat, Jim and Toby have worked on tracking this down.
Data products: Little news about the reprocessing; for FT2 reprocessing; signs of progress are appearing in JIRAS for L1 processing.
Databases and related utilities
No news
Likelihood analysis
- Stephen Fegan made some speed improvements in the likelihood evaluation, using caching to avoid recomputing certain values. The speed-ups are up to a factor of 3 for some cases. (Likelihood v14r2).
- I (JC) finally made the changes to ensure that the same spectral models are available to gtlike, gtmodel and the pyLikelihood interface. (Likelihood v14r2p1, pyLikelihood v1r9p6)
- I've made all the necessary changes to use the phi-dependence in the IRFs. (Likelihood v14r2p1)
GRB tools
No news.
Pulsar tools
From Masa: "Right now we are writing a detailed description of exactly what computations pulsar tools perform, which has been desired for a long time by now. But it has been taking a longer time than I expected. It is just difficult to explain enough, but not too much."
Observation simulation
No news
User interface and infrastructure (& utilities)
Jim has done ground work to get the Science Tools ready for handling azimuth-dependent IRFs. (As far as I know we are first, or primarily, going to be allowing for azimuthal dependence of the effective area.) Riccardo described in C&A and Weekly Analysis meetings last week the schedule for implementation of new IRFs. From Toby "I'm working on introducing phi dependence to the exposure computation. (To go with the enhancement of the IRFS.) It is at least a week from completion, but I already know that simply computing the phi associated with each pixel doubles the computation time."
Source Catalog
Bright Source List day and night. Jean has also done some interesting work on the dependence of source properties on the diffuse emission model.
Science Tools Working Group
The current release of the Science Tools remains v9r9.
Again, the most important difference from v9r8p3 is in how gtdiffrsp manages the diffuse response columns. If you have invested time in calculating diffuse responses already, you may want to hold off on updating to v9r9, at least for running gtdiffrsp. Aside from gtdiffrsp, the new versions of tools will read the old format FT1 files, but the new gtdiffrsp cannot add columns old format FT1 files.
Data products: No news about the reprocessing of the FT2 files to remake them with the updated format. At this point they will not be produced until after the New Year.
Databases and related utilities
No news
Likelihood analysis
No news. From the LATEST builds, it looks like plotting options for model counts spectra are being enhanced in pyLikelihood.
GRB tools
No news
Pulsar tools
Masa reports that he is continuing to update the user-level documentation for the tools.
Observation simulation
No news
User interface and infrastructure (& utilities)
No news
Source Catalog
Last week source associations and populations in the 3-month source list were discussed.
Science Tools Working Group
Since December 4, the current release version of the Science Tools is v9r9. Jim reports that the new version has the changes in how diffuse response columns are named that were described last week. It also has fixes for Likelihood and gtselect to handle large (>2 Gbyte) files; see below. For the record, here are the differences from v9r8p3.
If you have invested time in calculating diffuse responses already, you may want to hold-off on updating to v9r9, at least for running gtdiffrsp. Aside from gtdiffrsp, the new versions of tools will read the old format FT1 files, but the new gtdiffrsp cannot add columns old format FT1 files.
Data products: Still no news about the reprocessing of the FT2 files to remake them with the updated format.
Databases and related utilities
No news
Likelihood analysis
From Jim:
- modifications to enable large file-support recently added to tip STGEN-83@jira (Likelihood v14r0p2)
- added function to python interface to allow one to plot the model counts for a given source using a specified color (pyLikelihood v1r9p2, ST LATEST only)
GRB tools
No news
Pulsar tools
Masa reports that he is continuing to update the user-level documentation for the tools. Some of the updates are in the CVS repository at SLAC, but they are not yet in the User Workbook.
Observation simulation
No news
User interface and infrastructure (& utilities)
From Jim:
- gtselect
- modifications to enable large file-support recently added to tip STGEN-83@jira (dataSubselector v6r3p5, fitsGen v4r2p1)
Note that there is still a bug in cfitsio that James was planning to patch (see jira comments).
From Eric W.: (on news from the FSSC)
Dave is working on test scripts and data that we will use for our brutal but thorough testing regime in preparation for our release on 15 February. John is collecting and integrating all of the various test programs from the SLAC-derived Science Tools code tree for use in a quick-check build-time test (i.e. a "hmake test" target). James has been executing a variety of bug fixes. And I've been running down problems getting pyLikelihood to build and install properly under the HEADAS system, which has forced me to take the plunge and learn a lot about how SWIG works when gluing C++ code to Python.
Source Catalog
Last week Toby presented news about resolving the pointfit-related issues that Jean noticed in the Catalog pipeline analysis of the 3-month data set. Elizabetta presented the current version of the source catalog database tables that are being designed and implemented. And we had some more discussion about source associations and identifications.
Science Tools Working Group
The current release version of the Science Tools is v9r8p3. Jim reports "ST v9r8p3 was tagged to gather up the changes in ST HEAD before including the gtdiffrsp column name changes in the next major release." See below for details regarding the pending changes to gtdiffrsp. The differences from v9r8p2 are mostly small but include some important bug fixes.
Data products: No news about the reprocessing of the FT2 files to remake them with the updated format; progress took a Thanksgiving holiday.
The pending changes to gtdiffrsp above will regularize the names of the columns added to the FT1 files. Presently the names of the columns that gtdiffrsp adds are the conjoined names of the IRFs and the diffuse source name in the XML file. In the new version that Jim has put together, the column names will be simple and uniform, but the headers of the files will include new diffuse response keywords that contain the information formerly encoded in the column names. This will make FT1 files that include diffuse responses something that can be accepted by the FSSC, which has to live by HEASARC conventions for naming columns in FITS binary tables.
Databases and related utilities
No news
Likelihood analysis
From Jim:
- I improved the upper limit calculation following a bug-report from Jean (pyLikelihood v1r9p1, ST v9r8p3)
- There had been a suggestion to make the default "strategy" for NewMinuit be 2 (instead of 1). This change was made in optimizers v2r14 and made part of ST v9r8p2, but subsequent tests revealed this to produce unreliable results in some cases, so it was changed back to 1 in optimizers v2r14p1 and ST v9r8p3.
- The new diffuse response column conventions are implemented in Likelihood v14r0p1 and fitsGen v4r2 (both still in ST LATEST).
GRB tools
No news
Pulsar tools
No news; Masa reports that he is updating the user-level documentation for the tools. This is going on behind the scenes so far - not yet in the User Workbook.
Observation simulation
No news
User interface and infrastructure (& utilities)
No news. Eric W. reports that the FSSC builds of the parts of v9r8p2 that will be supported for Guest Investigators by the FSSC are still under test. Dave Davis estimates that a week or two of testing/fixing will be required to make the builds work reliably on all of the platforms they intend to support (which include Mac OS X).
Source Catalog
The Catalog group met last week and talked about the 3-month source list and 'unassociated' sources.
Science Tools Working Group
The current release version of the Science Tools remains v9r8p2. The RM seems to be having trouble testing the VC8 version but it has at least built.
Data products: Anders is marshaling the effort to reprocess old FT2 files, coordinate redelivery to the FSSC (and Data Catalog and Astro Server), and update L1proc to make FT2 files in the new format. which has a few additional columns added to the old format.
Databases and related utilities
No news
Likelihood analysis
No news
GRB tools
No news
Pulsar tools
No news
Observation simulation
No news
User interface and infrastructure (& utilities)
Eric W. has made builds on a variety of platforms of the (large) subset of the Science Tools that the FSSC plans to support for guest investigators:
32-bit & 64-bit SL 4.4 (gcc 3.4)
32-bit & 64-bit SL 5.2 (gcc 4.1)
Intel & PPC OS X 10.4
Intel & PPC OS X 10.5
These are not yet available at SLAC. The specific packages built for the FSSC distribution are listed in the requirements file for the ST_dist package
Source Catalog
The Catalog group met last week: some discussion of multiwavelength follow up along with a lot of discussion of the 3-month source list.
Science Tools Working Group
The current release version (since November 1) of the Science Tools is now v9r8p2. This includes a number of mostly-small updates from v9r8p1. As Dave Davis wrote in a note to scisoftlist on Oct. 16, according to the freeze and testing plan at the FSSC, v9r8p2 is the target version for public release in February. (Keep in mind that the FSSC will not be releasing all of the packages that are built with Science Tools; the list is in the ST_dist package.)
Data products: The proposed update to the Science Data Products File Format Document for the changes to FT2 (LS-005) has been incorporated by David B. in the next revision of the document. Andrea has developed a version of ft2Util to support generating updated FT2 files. A plan for regenerating FT2 files with this format, and delivering them to the FSSC in concert with switching over to the new version of ft2Util in L1Proc needs to be settled on.
Databases and related utilities
No news
Likelihood analysis
In the current release Jim has added an option to choose relative or absolute convergence tolerance in gtlike. This functionality still needs to be propagated to gttsmap and gtfindsrc.
GRB tools
No news
Pulsar tools
From Masa: "In the pulsar tools development, the pulsar tools were made ready for the public release after bug fixes and minor improvements, and are included in ScienceTools v9r8p2. I will be updating the Workbook to match the versions of the pulsar tools in this package."
Observation simulation
John V. is planning to assemble a complete set of example files for the various inputs to gtorbsim. These will be primarily for testing purposes, but will be made available to us and (I think) integrated with the documentation for the tool.
User interface and infrastructure (& utilities)
Toby has updated the astro package to account for the leap second that happened at the end of 2005 and to handle the one that is planned for the end of 2008. The most noticeable aspect of this may be in the DATE-OBS and DATE-END strings in the headers of FT1 and FT2 files. These are the human-readable dates that until now have been 1 second off.
Source Catalog
The Catalog group met last week; Toby described enhancements to pointlike "to use the normalization of the background as an independent estimator of the fraction of observed events in the ROI that are not background". Catalog pipeline analysis of the September data and studies of associations with September sources were also discussed.
Science Tools Working Group
The current release version (since October 22) of the Science Tools is now v9r8p1. The driver for the update from v9r8 was the fix for gtobssim simulations using merged flight FT2 files; see below.
Data products: The proposed update to the Science Data Products File Format Document for the changes to FT2 (LS-005) has been sent to David B. for consideration for the next revision of the document.
Databases and related utilities
No news
Likelihood analysis
No news
GRB tools
No news
Pulsar tools
From Masa: "In the pulsar tool development, we fixed two JIRA issues (PULS-48 and PULS-49), and made minor improvements in error messages, prompts, Doxygen documents, and so on. The latest tagged versions as of this writing are timeSystem v6r1, pulsarDb v8r1, pulsePhase v8r1, and periodSearch v10r3. Currently we are working on the errors in their unit tests that the release manager is showing."
Observation simulation
As Jim reported last week, there was a logic error in the code introduced in observationSim v8r2 to compute livetime information for merged FT2 files. See OBS-13. This is fixed in observationSim v8r3 (in the current release of Science Tools).
John V. is looking into whether we have the most-current documentation in the Workbook for gtorbsim, and into the possibility of providing examples of the various input files to go with the documentation.
User interface and infrastructure (& utilities)
No news
Source Catalog
The Catalog group met last week (2x including a rump session on Friday); the topics were quite wide ranging, from studies of sources detected to potential associations in other astronomical catalogs.
Science Tools Working Group
The current release version of the Science Tools is now v9r8. Here is the list from the Release Manager of the differences.
We had a Science Tools meeting last Wednesday (11 attendees). Much of the discussion was about how to make relevant parts of the User Workbook documentation available to guest investigators next year via the FSSC, while keeping the documentation in synch. The freeze/test/release plan for the Science Tools to be released by the FSSC was also discusssed. Basically, the freeze will be the end of this month; Dave Davis sent a schedule for what happens next to scisoftlist.
Data products: Regarding the planned additions to FT2 contents, Andrea has modified ft2util to provide the new columns. The new version is being tested.
Databases and related utilities
No news.
Likelihood analysis
No specific news, although you will be reminded from the RM link above that a number of useful updates have accumulated since the last release.
You will notice that in v9r8 gtselect now prompts for a zenith angle limit. Providing one is a good idea even if you are analyzing only survey-mode data, especially because sun avoidance and other maneuvers can bring the horizon into the FOV even during these times. Currently 105 deg is commonly used. If you don't want to make a cut on zenith angle, specify the limit as 180 deg.
GRB tools
No news
Pulsar tools
The new release incorporates all of the various updates that have been described in these reports over the last couple of months.
Masa has noticed a bug in gtbary in the new release; see the details in his posting PULS-49. Under certain circumstances, gtbary ignores (and does not report) errors in computing geocentric and barycentric times. The outputs are correct if the inputs are correct, but if you run gtbary as a standalone tool you don't get any warnings if you come up against this bug.
Masa reports that this does not affect any of the other pulsar tools. The current workaround for gtbary is to continue to use the v9r7p1 version [or live dangerously].
Observation simulation
Reminder: Jim has updated gtobssim to allow it to process merged flight FT2 files. This is a great convenience for simulating as-flown pointing and livetime histories; formerly the gaps between the runs required simulations to be made run-by-run and then merged. See the Science Tools Update for October 14.
STOP PRESS There was a logic error in the code introduced in observationSim v8r2 to compute livetime information for merged FT2 files. See obs-13@jira. This is fixed in observationSim v8r3.
In v9r8 the gtorbsim tool is now the orbit/attitude simulator that derives from the Tako planning system in the FSSC. The simulation is somewhat idealized but it can make, e.g., pointed observations with earth avoidance and survey-mode observations with a user-specified rocking profile. The inputs are described in the User Workbook, although we do not have examples yet of all of the input files. The simulator was written by Giuseppe Romeo, and it is currently supported by John Vernaleo; Analia Cillis wrote the documentation.
User interface and infrastructure (& utilities)
No news
Source Catalog
The Catalog group did not meet last week; next meeting is tomorrow.
Science Tools Working Group
The current release version of the Science Tools remains v9r7p1 - 2+ months of stability. In the current release candidate (HEAD1.690) a number of updates for Likelihood and the pulsar tools (see below) are pending; see this summary from the Release Manager.
Data products: The planned additions to FT2 contents are posted are closer to implementation; Andrea is working on modifications to ft2util and will produce some test files using a new ft2.tpl file. The revised format will be submitted as a change to the File Format Document for the LAT science data products
Databases and related utilities
No news.
Likelihood analysis
No news
GRB tools
No news
Pulsar tools
From Masa: "In the pulsar tool development, we added a geocentric option to gtbary. Two JIRA issues (PULS-46 and 47) were fixed. Some tools add leapsecfile parameter (hidden) to specify a leap second file, just in case a user want it other than a system default. As a result, all the pulsar tools change their version numbers. The latest versions as of this writing is timeSystem v6, pulsarDb v8, pulsePhase v8, and periodSearch v10r2p1." [N.B. These updates are not yet in a release version of Science Tools]
Observation simulation
There was an unannounced change in the flux package in how it handles FT2 files with gaps. Now it apparently silently interpolates across those gaps instead of raising an exception as it had before. I've modified gtobssim to return zero livetime if a candidate event arrives during a gap. The upshot is that gtobssim can now be run reliably on FT2 files with gaps. (observationSim v8r2)
User interface and infrastructure (& utilities)
No news
Source Catalog
Last week's discussion included a comparison of pointlike and gtlike results for the August source list. Juergen also described his use of gtlike to obtain model-independent spectra. Benoit showed a very interesting sensitivity map for August. Will not meet this week.