Externals Upgrade 2012

ROOT v5.34

Currently looking at v5.34.03

Trouble in System tests - all of the tests are taking longer to execute.  Particular in the ROOT Analysis step, not in actually running Gleam.  Some of the runs time out in the xlong queue.

For tests v20r7p0-rc4, the following tests timed out:  VerticalGamma100MeV, VerticalGamma1GeV, VerticalMuon1GeV, VerticalProton1GeV, http://glast-ground.slac.stanford.edu/SystemTests/?releaseVersionId=14021

Looking at the times for the other runs:

It would appear we are getting stuck in the Root Analysis portion of the tests.

All of the failed runs had completed their GR simulations, so the ROOT files were available.  Repeated the ROOT Analysis step, stand alone, using the same version of ROOT, and all the system tests scripts and submitting to the same machine (fell) that was used originally.  All the runs failed to complete in the time allotted to the xlong queue.

Comparing the results of running ROOT's stress test between v5.34.03 and v5.26

heather@yakut14 $ root -b \
************************************ ** W E L C O M E to R O O T ** **
Version 5.34/03 27 October 2012
** ** You are welcome to visit our Web site
** http://root.cern.ch ** *
*************************************
ROOT 5.34/03 (tags/v5-34-03@46856, Oct 27 2012, 23:19:27 on linux)
CINT/ROOT C/C++ Interpreter version 5.18.00, July 2, 2010
Type ? for help. Commands must be C++ statements.Enclose multiple statements between { }.
root* 0 *.L stress.cxx
root* 1 *stress(1000)
*********************************************************
Starting R O O T - S T R E S S test suite with 1000 events
************************************************************
Test 1 : Functions, Random Numbers, Histogram Fits............. OK{}
Test 2 : Check size & compression factor of a Root file........ OK{}
Test 3 : Purge, Reuse of gaps in TFile......................... OK{}
Test 4 : Test of 2-d histograms, functions, 2-d fits........... OK{}
Test 5 : Test graphics & Postscript............................ OK{}
Test 6 : Test subdirectories in a Root file.................... OK{}
Test 7 : TNtuple, selections, TCut, TCutG, TEventList.......... OK{}
Test 8 : Trees split and compression modes..................... OK{}
Test 9 : Analyze Event.root file of stress 8................... OK{}
Test 10 : Create 10 files starting from Event.root.............. OK{}
Test 11 : Test chains of Trees using the 10 files............... OK{}
Test 12 : Compare histograms of test 9 and 11................... OK{}
Test 13 : Test merging files of a chain......................... OK{}
Test 14 : Check correct rebuilt of Event.root in test 13........ OK{}
Test 15 : Divert Tree branches to separate files................ OK{}
Test 16 : CINT test (3 nested loops) with LHCb trigger..........OK
*******************************************************
SYS: Linux yakut14 2.6.18-308.13.1.el5 #1 SMP Thu Jul 26 05:49:19*
SYS: "Red Hat Enterprise Linux Client release 5.8 (Tikanga)"
************************************************************
stress : Total I/O = 552.7 Mbytes, I = 404.5, O = 148.2stress : Compr I/O = 381.1 Mbytes, I = 276.7, O = 104.5
stress : Real Time = 82.77 seconds Cpu Time = 81.54 seconds
********************************************************
ROOTMARKS = 620.3 * Root5.34/03 20121027/2308

heather@yakut14 $ root -b
ROOT 5.26/00a (tags/v5-26-00a@32053, Jan 19 2010, 12:43:52 on linux)
CINT/ROOT C/C++ Interpreter version 5.17.00, Dec 21, 2008
Type ? for help. Commands must be C++ statements.
Enclose multiple statements between { }.
root [0] .L stress.cxx
root [1] stress(1000)
******************************************************************
*  Starting  R O O T - S T R E S S test suite with 1000 events
******************************************************************
Test  1 : Functions, Random Numbers, Histogram Fits............. OK
Test  2 : Check size & compression factor of a Root file........ OK
Test  3 : Purge, Reuse of gaps in TFile......................... OK
Test  4 : Test of 2-d histograms, functions, 2-d fits........... OK
Test  5 : Test graphics & Postscript............................ OK
Test  6 : Test subdirectories in a Root file.................... OK
Test  7 : TNtuple, selections, TCut, TCutG, TEventList.......... OK
Error in <TUnixSystem::DynamicPathName>: $(ROOTSYS)/test/libEvent[.so | .sl | .dl | .a | .dll] does not exist in /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib:/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib:/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib/root:.:/afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib::/afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/cint/cint/stl
Error in <TUnixSystem::DynamicPathName>: test/libEvent[.so | .sl | .dl | .a | .dll] does not exist in /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib:/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib:/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib/root:.:/afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib::/afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/cint/cint/stl
Test  8 : Trees split and compression modes..................... ===>stress8 will try to build the libEvent library
g++  -O2 -Wall -fPIC -pthread -m32 -I/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/include -c Event.cxx
Generating dictionary EventDict.cxx...
rootcint -f EventDict.cxx -c Event.h EventLinkDef.h
g++  -O2 -Wall -fPIC -pthread -m32 -I/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/include -c EventDict.cxx
g++ -shared -O2 -m32 Event.o EventDict.o -o  libEvent.so
libEvent.so done
g++  -O2 -Wall -fPIC -pthread -m32 -I/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/include -c MainEvent.cxx
g++ -O2 -m32 MainEvent.o Event.o EventDict.o -L/afs/slac/g/glast/ground/GLAST_EXT/redhat5-i686-32bit-gcc41/ROOT/v5.26.00a-gl5/gcc41/lib -lCore -lCint -lRIO -lNet -lHist -lGraf -lGraf3d -lGpad -lTree -lRint -lPostscript -lMatrix -lPhysics -lMathCore -lThread -pthread -lm -ldl -rdynamic  -o Event
Event done
OK
Test  9 : Analyze Event.root file of stress 8................... OK
Test 10 : Create 10 files starting from Event.root.............. OK
Test 11 : Test chains of Trees using the 10 files............... OK
Test 12 : Compare histograms of test 9 and 11................... OK
Test 13 : Test merging files of a chain......................... OK
Test 14 : Check correct rebuilt of Event.root in test 13........ OK
Test 15 : Divert Tree branches to separate files................ OK
Test 16 : CINT test (3 nested loops) with LHCb trigger.......... OK
******************************************************************
*  SYS: Linux yakut14 2.6.18-308.13.1.el5 #1 SMP Thu Jul 26 05:49:19
*  SYS: "Red Hat Enterprise Linux Client release 5.8 (Tikanga)"
******************************************************************
stress    : Total I/O =  549.8 Mbytes, I =  406.3, O = 143.5
stress    : Compr I/O =  388.0 Mbytes, I =  284.7, O = 103.2
stress    : Real Time =  74.65 seconds Cpu Time =  66.10 seconds
******************************************************************
*  ROOTMARKS = 765.2   *  Root5.26/00a  20100119/1117

******************************************************************

There doesn't seem to be anything intrinsically slower about v5.34.03.

Trying a fresh rebuild of v5.34.03 on rhel5-32

No Difference

Running a "read-only" job without filling histograms

This runs quite smoothly and quickly - so likely the issue is not with reading/writing, but rather some other aspect of the scripts

Re-running tests in xxl queue

In preparation for the next GR release, it would be nice to have a complete set of system tests results, I think xxl will allow all the tests to complete.  This required a small tweak to use "xxl" from "xlong" in $SYSTESTS/src/exec/runSystests_SCons.pl

Building systest scripts via ACLIC

The system tests scripts have been historically run via the interpreter.  To enable more debugging and to perhaps speed and clean things up.  Work has started to build the RootTreeAnalysis used in the system via ACLIC.  A number of errors and oddities are being fixed in the hopes that this might speed things up again.

Required some tweaks to reconRootData includes...still having trouble getting the RootTreeAnalysis class recognized after the shared library is created.