External Library Installations
This past week (9/29/08 - 10/3/08), I have completed the following installations for third party python packages, and external library installation, for both the rhel4 and rhel3 machines:
- python 2.5.1-gl built against againstTcl/Tk 8.4
- 4Suite-XML
- ipython
- numpy
- pyfits
-OmniOrb
There is one catch. During our core meeting last week, it was decided that we are going to go along with including matplotlib and SciPy. I installed matplotlib for the rhel3 machines; however, matplotlib requires a version of numpy more current than the one I installed. pyfits needs numpy... So, I have to reinstall numpy and pyfits for all of the machines in order to install matplotlib and Scipy.
So far, SciPy and matplotlib (in addition to the latest versions of numpy and pyfits) have been installed for the rhel3 machines. SciPy seems to be functioning properly. I imported it, and python did not generate any errors.
GR-SCons
Obf
The remaining packages that did not build require Obf and OmniOrb. Most of the packages depend on Obf, so I configured externals.scons to incorporate Obf in Scons builds. I ran into problems with Obf itself:
(1) The linker was throwing errors about undefined references among Obf libraries
-- Some of the libraries needed where listed in the Obf requirements file (thus I had already listed them in externals.scons for Obf)
-- Some libraries needed were not listed in the Obf requirements file, but contained in the Obf's "lib/" directory. Thus, I added those additional libraries to externals.scons.
(2) "/usr/bin/ld: warning: libstdc+.so.5, needed by /afs/slac/g/glast/ground/GLAST_EXT/redhat4-i686-32bit/obf/B1-1-3/gcc34/lib/libcdm.so, may conflict with libstdc+.so.6"
For the first set of errors, I kept playing around with those and even considering that maybe the order of the libraries specified on the ld command line could have been out of place. Each time I resolve one, a new set of undefined references pop up.
Fortunately, I was able to eliminate ALL of the errors generated by Obf, by setting the "-RPATH" for LD (which is the runtime library search path) to point to the location of Obf's "lib/" directory. It resolved the errors, but Navid told me that it is not portable, so that solution is unacceptable. I suspect that these errors are strongly tied into the second error (2). For (2), I suspect that libcdm needs libstdc+.so.5, but something else needs libstdc.so.6. Both libstdc.so.6 and libstdc+.so.5 are in /usr/lib. I have not resolved that one yet. I moved to OmniOrb...
OmniOrb
I configured Scons to include OmniOrb (modifying the externals.scons file). I have encountered an error with OmniOrb itself:
/afs/slac/g/glast/ground/GLAST_EXT/redhat4-i686-32bit/OmniOrb/4.1.2/gcc34/include/omniORB4/omniORB.h:659: error: class `LOCATION_FORWARD' does not have any field named `pd_objref'
/afs/slac/g/glast/ground/GLAST_EXT/redhat4-i686-32bit/OmniOrb/4.1.2/gcc34/include/omniORB4/omniORB.h:659: error: 'const class LOCATION_FORWARD' has no member named 'pd_objref'
In file included from /afs/slac/g/glast/ground/GLAST_EXT/redhat4-i686-32bit/OmniOrb/4.1.2/gcc34/include/omniORB4/omniORB.h:670
- then a lot of undefined references... I have to look into this more.
NAGIOS
The Short Story:
I had to make a fix to a particular plugin due to a request from Tony.
-Fix: A timeout from the jmx query which checks the job control daemons should be considerer a non-reportable error for Nagios. Report timeouts from other jmx queries.
- My solution (plan, design, debug, and testing included ):
- I added a help menu, and additional command option (--disable-timer) for the check_jmx plugin
- I changed the check_command for the job control daemons to take the --disable-timer switch
- Tested with Nagios
What was the Issue?
- I ported some service checks from Tony's standalone monitoring program to Nagios, but our check-status results sometimes did not agree
- I did some poking around and realized the source of the issue was in his code
- I broke down the options and presented them to Tony, Navid, & Richard. Tony's decision was to choose the above-mentioned "fix" to resolve the problem.
I also had to make a minor change to the IP address for one of the collaborator's websites (hiroshima), which had changed so Nagios was complaining.
Other Misc/Time consuming activities:
I had two cumbersome forms to fill out
(1) Travel Authorization form for my trip to SLAC
- I filled it out, submitted it for approval and it was sent back to me to fix since I supposedly completed the wrong form. I did a completely new form which wanted some unusual information. I ran into problems, and begin dialogue with the person who told me to send the form. Eventual a few hours of confusion, I sent it back and double-checked with her that she sent me the right form. It turns out that I did originally have the correct form, and I was accidentally sent the wrong form (which she thought was the correct form).
(2) I had to complete paper work for a background check for my new badge here at Goddard. I was resent the email about 2 weeks ago because a lot of people/contractors never received it. That form took a very good deal of time late last week, up until today. However, the form is a big priority so I can continue to have access to Goddard this month.
Nagios Storm
About two weeks ago, a lot of services threw critical alerts over night. It was an unusual amount of errors. Richard asked me to look into that situation. I have been doing so, off and on. I see that their were a lot of Socket connection timeouts for the same machines that threw critical alerts for other services so their seems to be the cause-and-effect errors (socket-connection timeouts hints at ....?). I have yet to generate the formal report of that incident. I have been rather swamped which has stunned progress with this one, but I am pulling out of the quick sand!