Science Tools Update, November 1, 2007
Science Tools Working Group
Did not meet this week.
On Wednesday, Jim released v9r3p1 of the Science Tools. Here are the differences from v9r3. It has some bug fixes in the pulsar tools (see below), but the driving factor was switching to a rebuilt version of ROOT that did not expect to find itself on SLAC Linux. The problem was pointed out by Nicola O..
Data products: Julie mentioned yesterday the extended version of the event summary (FT1) files. These are planned to be a large superset of the FT1 files in terms of contents (~200 quantities vs. ~18 for FT1) and may also be a superset in terms of the events included. The additional variables will be quantities from the Merit files, and these files will be delivered routinely to the GSSC to facilitate the extra-Science Tools use of the data. Julie has a makeFT1 dictionary file to make the extended version of FT1 and plans to use it this week to make example files from the new 55-day run.
Databases and related utilities
No news.
Likelihood analysis
Jim reports that the likelihood tools now check the IRF names in the diffuse response columns of FT1 files. Not doing this had become a more obvious potential problem with the advent of Pass 5 IRFs, which currently have 3 classes.
GRB tools
No news
Pulsar tools
Masa and James fixed the PULS-41 (gtpphase ignores the phi0 parameter) and PULS-42 (negative phases possible) issues.
Masa is working on updating documentation for the User Workbook.
Observation simulation
Jim reports that the new RadialSource source has been tested in Gleam.
Jim fixed the JIRA issue OBS-12, that gtobssim had been ignoring the LIVETIME column in input pointing histories. Riccardo pointed out this problem.
User interface and infrastructure (& utilities)
No news
Source Catalog
Met again this week. Jean presented results from a new look at the test pattern data sets with likelihood analysis. One interesting result was the flux limit as a function of source spectral index. He also updated his comparison of source detection algorithms on obssim2 data, drawing interesting conclusions from studying which sources were detected by only one of the algorithms. Jean also investigated the effect of removing sources below the detection threshold in the 'time' part of the pipeline analysis. This is fairly closely analagous to what is done for source monitoring in ASP. He found that the effects on the measured fluxes for the remaining sources were typically very small. He did not measure the CPU time but by halving the number of sources the expectation is that the CPU time required decreased by a factor of 4.
Science Tools Working Group
Did not meet this week.
On Wednesday, Jim released v9r3 of the Science Tools. Here are the differences from v9r2p2. The biggest news is that the new versions of the pulsar tools have been included (see below), but the release also includes important fixes (e.g., in gtburstfit) and enhancements (e.g., speed of gtselect - see below).
Emmanuel will make a MacOSX build (by hand) of v9r3.
Data products: No new news.
Databases and related utilities
No development news.
Likelihood analysis
Jim has worked out a filtering expression for gtselect and gtmktime that applies "zenith angle cuts to FT1 data without incurring additional excessive computational cost in the likelihood analysis." In general this is a combination of applying zenith angle cuts on FT1 files and also creating GTIs to remove time intervals when the horizon would cross the ROI (or source region). Depending on the pointing history and the zenith angle range of the ROI, the time ranges actually removed can be very small. Jim is working on evaluating and refining the approach.
GRB tools
James resolved 2 of the 3 the JIRA issues relating to GRB tools that were opened during the Ops Sim and these fixes are in the new release of the Science Tools
Pulsar tools
This week Masa and James released the extensively updated pulsar tools that they have been preparing for some time. This is a major milestone. Among other changes, the tools include barycentering on-the-fly, although gtbary remains available. As mentioned last week, alternative solar system ephemerides can now also be specified. Masa plans to start work on updating the reference pages and tutorials in the Workbook. If you can't wait, he has already updated the Doxygen documentation.
Marianne traced the unexpected phase shifts of the light curves of the bright pulsars in the Ops Sim data to what turned out to be a bug in gtpphase. Masa describes the issue and a workaround in the JIRA issue PULS-41 he opened for it.
Observation simulation
Jim reports that he has "implemented a RadialSource source object that allows for diffuse sources with azimuthal symmetry to be easily modeled. This will be used to model dark matter sources for Gleam and gtobssim simulations." This will sure beat using FITS images to define the source extents and profiles.
Jim also "made minor modifications to the SpectralTransient source class to allow for neutron components to be modeled in Solar Flare simulations."
And he "modified gtobssim to allow for the SAA to be disabled to support GRB sensitivity studies." This uses an environment variable DISABLE_SAA.
Max R. reports that he is working on the code for PulsarSpectrum to make it faster and more readable, and he is also extensively testing the code.
User interface and infrastructure (& utilities)
Jim "added a function that calls cfitsio directly to enable efficient copying of FITS files that are filtered by standard extended filename expressions. This function is now used by gtselect and gtmktime and has improved execution speed for FT1 files of size >2 M events by factors of >30.
James reports that he "met with Bryan Irby and Eric Winter to discuss the status of the Science Tools port into the HEADAS build system. They determined a way forward in which the Science Tools would be treated architecturally exactly like all other current missions (Swift and Suzaku). The plan is still to distribute the Science Tools independently of the rest of HEADAS for the time being."
Source Catalog
Met this week. Jean presented results from comparisons of source lists from different source detection algorithms applied to the obssim2 data. Depending on region of the sky (low vs. high) and probably also the duration of the data set different algorithms perform best.
We started a discussion about updating the concept of the data flow to the Catalog analysis and on the timeline early in the mission for producing successively refined source lists and preliminary source catalogs. The principal issue discussed was the role of ASP. Iteration of the catalog analysis with the refinement of the diffuse emission model also needs some more specific attention, but that is a topic for another meeting.
Not sure where else this belongs: Juergen has made significant updates to gtsrcid in the current release - simplifying access to external catalogs and allowing for elliptical confidence regions. The documentation in the Workbook is now somewhat out of date for this tool.
Science Tools Working Group
Met this week (11 attendees).
Since October 5, the current version of ScienceTools is v9r2p2. Here are the minor differences from v9r2p1. Jim removed Mc variables from Pass5_classifier - they weren't actually used in the classifying and the change was needed to process the Ops Sim data sets. It isn't clear yet whether we'll have a new release of the Science Tools to go along with the release of the 55-day Interleaved data set sometime next week. Whether or not we do is not vitally important.
Data products: No new news.
Databases and related utilities
No development news.
Likelihood analysis
Jim has fixed a bug that Jean noticed in the Python version of likelihood, relating to how a stored best-fit result was retained even if the source model changed.
GRB tools
No development news. Some JIRA issues relating to GRB tools were opened last week during the Ops Sim. The most serious of these is that gtburstfit is not currently working under Linux. James is working on resolving the issues.
Pulsar tools
Masa reports that he and James have updated the interface to the pulsar tools to include the prompting options needed now that barycentering-on-the-fly has been implemented. He is updated the doxygen documentation in the mainpage.h files. Then Masa will work on modifying the documentation for the workbook. One result of this changes in prompting is that users will be able to specify the solar system ephemeris that they want. This is not going to be needed by casual users but could be very important for the right user.
Observation simulation
No direct news. Valerie is updating the GBM simulation code, which is not part of the Science Tools.
User interface and infrastructure (& utilities)
James reports that he "addressed several issues in Ape and HEADAS related to file access checking performed on file (type "f") parameters. Specifically, not all access modes (read, write, file exists, file doesn't exist) were being checked correctly in various combinations. The code now checks all cases and behaves identically on Windows and on Linux/OSX. He also rationalized prompt redirections; under certain circumstances, Ape's prompts properly go directly to the user's terminal device instead of to the standard output. Ape was behaving differently in two cases: with and without readline."
Eric reports that converting the Science Tools from CMT to hmake (HEASARC) is not as much fun as it sounds. Discussion on this topic during the meeting was more or less along the lines of producing hmake files routinely for the Science Tools - manually (I suppose) or possibly with Scons - so that a complicated and delicate conversion script would not be needed at the GSSC end
Source Catalog
Did not meet this week, but did meet last week. Jean has compiled lists of sources detected in the obssim2 data by several algorithms (none of which searched for sources that were bright for only part of the year). He is developing a quantitative way to compare their performance; this is complementary to the Test Pattern studies. Ludovic demonstrated that gtfindsrc and pointfind (UW) both work well for refining initial guesses for positions of sources. pointfind has a speed advantage and will be included as a preprocessing step before source analysis in the Catalog pipeline.
Science Tools Working Group
Met this week.
Since Sunday, the current version of ScienceTools is v9r2p1. Here are the differences from v9r2. Jim fixed a problem in makeFT1 and Navid has made some updates in the facilities package.
Chris confirmed yesterday that the GLAST Users Committee has agreed to paricipate in a 'beta test' of the Science Tools, starting in mid-December and running for 6 weeks. Chris and the GSSC also hope to recruit some HEASARC people and I think also LAT people to participate. Looking ahead to the beta test, Chris is organizing a review of documentation of the tools (reference pages, tutorials, and Cicerone) and Analia has been working on filling in some topics in the latter. Chris and the GSSC will produce a template for more-detailed documentation of the tools in the Reference Pages section of the Workbook.
Data products: No new news.
Databases and related utilities
No development news.
Likelihood analysis
Not sure where this belongs - Igor and Troy joined the Science Tools meeting this week to describe the developments they have in mind to support all-sky model fitting (likelihood analysis). gtlike cannot make all-sky convolutions accurately and for some very important applications, like tuning parameters of GALPROP there's no substitute for fitting the whole sky. Their first efforts, with Gulli J., will be to add a HEALPix binning capability to gtbin. They are also finding/implementing HEALPix convolution algorithms for fast all-sky convolutions with LAT PSFs. Jim has discussed with Gulli moving the convolution aspects of Likelihood into its own package; HEALPix-based convolutions could be used to advantage by gtlike and other Science Tools and there's some hope that we'll get there.
GRB tools
No development news.
Pulsar tools
Masa reports that barycentering on the fly is implemented in the HEAD version of the Science Tools, but the needed changes to the parameter files are still coming along.
Observation simulation
No direct news. Nicola fixed a problem in GRBobsmanager last week related to what happens during initialization if a GRB happens to be in progress at the start of a run. The symptom had been that the starting time for the run got reset back to the start of the GRB.
User interface and infrastructure (& utilities)
James described work on porting the Science Tools to build under Windows at the GSSSC/HEASARC, his plan of approach was modified. Jim suggested that rather than work on getting the hmake system to make Visual Studio files, the Windows versions just be built to use Cygwin. That route was rejected by HEASARC some time ago but apparently use of Cygwin has worked itself into HEASARC-supported software. It sounds like James will pursue the Cygwin approach. Eric W. has got the Linux builds with hmake almost working.
Source Catalog
Did not meet this week. In the meantime, Jean is collecting results from analysis of obssim2 data using the source detection algorithms that are under study.