...

• Any file without an extension is treated as a test script. This script will be installed and run.
  • WARNING: beware of editors such as emacs that leave backup files such as myscript~, they will be treated as a separate test
• Any file that looks like C or C++ code (.c, .cpp extension, etc) is treated as a test program. It will be compiled, installed, and run.
• If a test script or program returns non-zero, it failed and scons will report this.

Other files in the test directory Files with extensions that do not look like C/C++ code are ignored.

We will start with some simple examples to demonstrate the testing mechanism. Then we will look at examples using testing frameworks. In practice, developers will most likely want to use testing frameworks.

Python Unit Test

Add the file MyPkg/test/myfirst

#!@PYTHON@
import sys
if __name__ == '__main__':
 print "Running mymyfirst test. - it should fail:"
 sys sys.exit(1)

When you do scons test, you will get the output

...

This is because the script returned something non-zero. If you look at the file build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst.utest you see the output of the script, i.

The syntax @PYTHON@ is explained in the SConsTools page.

e:

$ cat build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst.utest
Running myfirst test - it should fail:

This is not very informative, but demonstrates the underlying mechanism by which the system is told if a test passed or failed. Below we'll see how the testing frameworks provide more information in the output.

The syntax @PYTHON@ is explained in the SConsTools page.

Change the sys.exit(1) to sys.exit(0) and the test will Change the sys.exit(1) to sys.exit(0) and the test will succeed. You can also take out the sys.exit line, be default Python will return 0 after the script runs.

...

#!@PYTHON@

import sys
import unittest

class MyTest( unittest.TestCase ) :

    def setUp(self) :
        """ 
        Method called to prepare the test fixture. This is called immediately 
        before calling the test method; any exception raised by this method 
        will be considered an error rather than a test failure.  
        """
        pass

    def tearDown(self) :
        """
        Method called immediately after the test method has been called and 
        the result recorded. This is called even if the test method raised 
        an exception, so the implementation in subclasses may need to be 
        particularly careful about checking internal state. Any exception raised 
        by this method will be considered an error rather than a test failure. 
        This method will only be called if the setUp() succeeds, regardless 
        of the outcome of the test method. 
        """
        pass

    def test_mytesttestMyTestOne(self):
        a=3
        b=4
        self.assertEqual(a,b)

if __name__ == '__main__':
  unittest.main(argv=[sys.argv[0], '-v'])

...

********************************************************************************************************
*** Unit test failed, check log file build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest ***
********************************************************************************************************
scons: *** [build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest] Error 256
scons: building terminated because of errors.
psana1302:~/rel/unitTestTutorial $ cat build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest
test_mytesttestMyTest (__main__.MyTest) ... FAIL
======================================================================
FAIL: test_mytest testMyTest (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "build/x86_64-rhel5-gcc41-opt/MyPkg/usingnext_python_frameworktest", line 31, in test_mytesttestMyTest
    self.assertEqual(a,b)
AssertionError: 3 != 4
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)

Refer to the documentation https://docs.python.org/2/library/unittest.html for more information on unittest.

Boost unit_test Framework

For an example of using the boost C++ unit test framework, create the file MyPkg/test/using_boost_framework.cpp with the contents (this is mostly copied from the boost website:

I believe the mechanism by which Python unittest works is

• The script we wrote runs unittest.main and passes it the script name. We also pass the verbose argument so there will be more output in the logfile.
• The unittest module then inspects the contents of the script, finds a class that is derived from unittest.TestCase.
• It then creates an instance of this class, calling the setUp() function
• Next it looks for any method of the class that starts with "test" these are the "test methods" referred to in the comments for setUp and tearDown.
  • Other methods are ignored
• All of the test methods are called, finally tearDown is called (assuming all tests passed, but I'm not sure about that)

Refer to the documentation https://docs.python.org/2/library/unittest.html for more information on unittest and some good tutorials.

Boost unit_test Framework

For an example of using the boost C++ unit test framework, create the file MyPkg/test/using_boost_framework.cpp with the contents (this is mostly copied from the boost website:

/*

#define BOOST_TEST_MODULE MyTest
#include <boost/test/included/unit_test.hpp>

/**
 * Simple test suite for module psevt-unit-test.
 * See http://www.boost.org/doc/libs/1_36_0/libs/test/doc/html/index.html
 */


#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>

int add( int i, int j ) { return i+j; }

BOOST_AUTO_TEST_CASE( my_test )
{
    // seven ways to detect and report the same error:
    BOOST_CHECK( add( 2,2 ) == 4 );        // #1 continues on error

    BOOST_REQUIRE( add( 2,2 ) == 4 );      // #2 throws on error

    if( add( 2,2 ) != 4 )
      BOOST_ERROR( "Ouch..." );            // #3 continues on error

    if( add( 2,2 ) != 4 )
      BOOST_FAIL( "Ouch..." );             // #4 throws on error

    if( add( 2,2 ) != 4 ) throw "Ouch..."; // #5 throws on error

    BOOST_CHECK_MESSAGE( add( 2,2 ) == 4,  // #6 continues on error
                         "add(..) result: " << add( 2,2 ) );

    BOOST_CHECK_EQUAL( add( 2,2 ), 4 );      // #7 continues on error
}

BOOST_AUTO_TEST_CASE( my_test_fail )
{
    BOOST_CHECK_EQUAL( add( 2,2 ), 5 );      
}

The second test is designed to fail, and the output in the .utest log file is

psana1302:~/rel/unitTestTutorial $ cat $ cat build/x86_64-rhel5-gcc41-opt/MyPkg/using_boost_framework.utest
Running 2 test cases...
MyPkg/test/using_boost_framework.cpp(38): error in "my_test_fail": check add( 2,2 ) == 5 failed [4 != 5]

For more examples, one can look in the test subdirectories of packages like AppUtils, ConfigSvc, XtcInput, psana, psana_test, and Translator.

Nightly Build Considerations - External Test Data

...

    def test_mytest(self): 
        text = file('MyPkg/test/fixtures/myfile.txt','r').read()
        self.assertTrue(text.startswith('file text'))

External Test Data Location

For xtc files, we have a directory,

/reg/g/psdm/data_test

that was created expressly for the purpose of storing xtc test data. However we do not want to copy entire xtc files from the experiments into this location, we need to select the parts of the xtc file necessary for testing. The current organization of the data_test directory is

Keeping the test data files small makes the preparation of xtc test data tedious. One must identifying a part of the xtc file that you want to test and copying it out. Presently the largest xtc test file in data_test is about 1GB, which is bigger than it needs to be. For the testing that I have done, I typically want to run psana on a few datagrams in an xtc file to test how it parses a new type or handles some damaged data. I need to identifying the file offsets of the beginning and ends of those datagrams in the xtc file, as well as the beginning and ends of transition datagrams that make the xtc file correct. I have some tools in the psana_test package for this purpose. An example is below.

One thing to note, scons test only runs tests for packages that are part of the working release. It does not run tests for all the packages that are part of the base release. I think the simplest thing for looking for package data, is to just look in one place, using a path relative to the working release directory like above. This works for the nightly build. It won't work for developers that use the new sit_setup feature that allows them to change directories away from their working release, but I think we should just follow the convention that we run scons test from the working release directory. This is different than how packages work with files in the 'data' subdirectory, where they really should check is the package is part of the working/test release and then look in the base release.

External Test Data Location

A directory for test data has been set up here:

/reg/g/psdm/data_test

that was created expressly for the purpose of storing test data for the analysis releases. Presently it holds xtc files, and some calibration constant files. We do not want to copy entire xtc files from the experiments into this location as they are to big. We need to select the parts of the xtc file necessary for testing. The current organization of the data_test directory is

Keeping the test data files small makes the preparation of xtc test data tedious. One must identifying the parts of the xtc file that you need for your test. An xtc file is comprised of datagrams. Event data are in L1Accept datagrams, but a properly formatted xtc file includes transition datagrams that precede and follow the L1Accept datagrams. At the low most tedious level, making a small xtc file for testing involves identifying the beginning and ending offsets of the datagrams you need to make your file. I'll give an example below of how I do this with tools I've written. Other people are welcome to add examples of using other tools that they find easier to work with.

Once you have prepared some test data, you can either add it to the Translator subdirectory, or the multifile subdirectory, or create a new sub directory, maybe with your package name (like I did when I made the Translator subdirectory). If you want to add it to Translator or multifile, please contact me (davidsch) as these Once you have prepared some test data, you can either add it to the Translator subdirectory, or the multifile subdirectory, or create a new sub directory, maybe with your package name (like I did when I made the Translator subdirectory). If you want to add it to Translator or multifile, please contact me (davidsch) as these files have specific naming conventions and there are unit tests in the psana_test package that access them. Creating a new subdirectory requires less coordination, however if you think the test data is going to be useful to others, we should work together on it. One of the benefits of using the psana_test package, is I have a mechanism for checking in the md5 checksums of the test data into svn. This allows the unit tests to verify that the test data has not changed.

Using the psana_test package

Here I'll go through an example of using the psana_test package to prepare a new test file to test a new pdsdata type. Of general interest will be preparation of the test file, and functionality in the python library code in psana_test.

Preparing a small sample test file

Making a small xtc test file

This section covers making small xtc test files. Developers are welcome to add sections for other tools they find useful. Presently the largest xtc test file in data_test is about 1GB, which is bigger than it needs to be. I think we should be able to keep test files down to 20-100 MB, smaller files mean faster unit tests as well.

Using psana_test and xtclinedump

For the testing that I have done, I typically want to run psana on a few datagrams in an xtc file to test how it parses a new type or handles some damaged data. Suppose we Suppose we don't have unit tests to see how psana handles Epix100aConfig version1 and EpixElement verion version 2, and we have identified an experiment xtc file with these types, namely

...

One of the tools in psana_test will let me see which datagrams have epix. I can string together a unix command to see:is xtclinedump, some documentation is in the psana - Module Catalog. it is a line oriented header dump of xtc files. One can use grep to filter the output so one only sees the datagram headers (which include file offsets in the file) and any xtc headers for a type that has epix as a part of it. Here is a command line that lets me see that there is epix in datagram 5, and to see the offset of where datagram 6 begins. This will be the first part of the xtc I want to save in my small test file. I am also going to want to get some transitions at the end of the file to form a correct xtc file.

~/rel2/unitTestTutorial $ xtclinedump xtc /reg/d/psdm/xcs/xcsi0314/xtc/e524-r0213-s03-c00.xtc | grep -i "dg=\|epix" | head -10
dg=    1 offset=0x00000000 tp=Event sv=      Configure ex=1 ev=0 sec=54754FDF nano=1E31D0E8 tcks=0000000 fid=1FFFF ctrl=84 vec=0000 env=0000161C
xtc d=2  offset=0x00022F4C extent=00108834 dmg=00000 src=01003069,19002300 level=1 srcnm=XcsEndstation.0:Epix100a.0 typeid=84 ver=1 value=10054 compr=0 compr_ver=1 type_name=Epix100aConfig plen=1083424 payload=0x0B...
dg=    2 offset=0x0012B780 tp=Event sv=       BeginRun ex=0 ev=0 sec=54755EBD nano=35A7112E tcks=0000000 fid=1FFFF ctrl=06 vec=0000 env=000000D5
dg=    3 offset=0x0012B820 tp=Event sv=BeginCalibCycle ex=0 ev=0 sec=54755EBE nano=00D5EE07 tcks=0000000 fid=1FFFF ctrl=08 vec=0000 env=00000000
dg=    4 offset=0x0012C188 tp=Event sv=         Enable ex=0 ev=0 sec=54755EBE nano=0124366D tcks=0000000 fid=1FFFF ctrl=0A vec=0000 env=80000000

psana1302:/reg/d/psdm/xcs/xcsi0314/xtc $ xtclinedump xtc e524-r0213-s03-c00.xtc | grep -v -i epic | grep -v -i "type_name=Xtc" | head -40 | grep -i "dg=\|epix"
dg=    15 offset=0x000000000x0012C228 tp=Event sv=      Configure L1Accept ex=1 ev=01 sec=54754FDF54755EBE nano=1E31D0E805EE73D6 tcks=0000000005094A fid=1FFFF144F9 ctrl=848C vec=0000146F env=0000161C00000003
xtc d=2  offset=0x00022F4C0x0012C264 extent=001088340010A454 dmg=00000 src=01003069,19002300 level=1 srcnm=XcsEndstation.0:Epix100a.0 typeid=84 1 ver=1 value=1005410001 compr=0 compr_ver=1 type_name=Epix100aConfigXtc
xtc plend=1083424 payload=0x0B...
dg=    23  offset=0x0012B7800x0012C278 tpextent=Event0010A440 svdmg=00000 src=01003069,19002300      BeginRun ex=0 ev=0 sec=54755EBD nano=35A7112E tcks=0000000 fid=1FFFF ctrl=06 vec=0000 env=000000D5level=1 srcnm=XcsEndstation.0:Epix100a.0 typeid=75 ver=2 value=2004B compr=0 compr_ver=2 type_name=EpixElement plen=1090604 payload=0x00...
dg=    36 offset=0x0012B8200x00266284 tp=Event sv=BeginCalibCycle       L1Accept ex=01 ev=01 sec=54755EBE nano=00D5EE0706EE644D tcks=00000000050974 fid=1FFFF144FF ctrl=088C vec=00001475 env=0000000000000003
dg=    4xtc d=2  offset=0x0012C1880x002662C0 tpextent=Event0010A454 svdmg=00000         Enable ex=0 ev=0 sec=54755EBE nano=0124366D tcks=0000000 fid=1FFFF ctrl=0A vec=0000 env=80000000
dg=    5 offset=0x0012C228 tp=Event sv=       L1Accept ex=1 ev=1 sec=54755EBE nano=05EE73D6 tcks=005094A fid=144F9 ctrl=8C vec=146F env=00000003
xtc d=3  offset=0x0012C278 extent=0010A440 dmg=00000 src=01003069,19002300 level=1 srcnm=XcsEndstation.0:Epix100a.0 typeid=75 ver=2 value=2004B compr=0 compr_ver=2 type_name=EpixElement plen=1090604 payload=0x00...
dg=    6 offset=0x00266284src=01003069,19002300 level=1 srcnm=XcsEndstation.0:Epix100a.0 typeid= 1 ver=1 value=10001 compr=0 compr_ver=1 type_name=Xtc

so if I copy bytes [0,0x00266284) I will get up to the first datagram with an EpixElement in it. Next I take a look at just the datagram headers at the end of the file:

$ xtclinedump dg /reg/d/psdm/xcs/xcsi0314/xtc/e524-r0213-s03-c00.xtc | tail -5 
dg= 1204 offset=0x5C225D2C tp=Event sv=       L1Accept ex=1 ev=1 sec=54755EFA nano=01E1D524 tcks=00506D4 fid=1993E ctrl=8C vec=3089 env=00000003
dg= 1205 offset=0x5C35FDE8 tp=Event sv=       L1Accept ex=1 ev=1 sec=54755EBE54755EFA nano=06EE644D04DB98AE tcks=0050974 fid=144FF19950 ctrl=8C vec=1475308F env=00000003

so if I copy bytes [0,0x00266284) I will get the first datagram with an EpixElement in it. I'd also like to get the last datagrams in the file, for a proper end of the calib cyle, etc. These are

dg= 1206 offset=0x5C499EA0 tp=Event sv=        Disable ex=0 ev=0 sec=54755EFA nano=172A7720 tcks=0000000 fid=1FFFF ctrl=0B vec=0000 env=00000000
dg= 1207 offset=0x5C499F40 tp=Event sv=  EndCalibCycle ex=0 ev=0 sec=54755EFA nano=17A3F59A tcks=0000000 fid=1FFFF ctrl=09 vec=0000 env=00000000
dg= 1208 offset=0x5C499FE0 tp=Event sv=         EndRun ex=0 ev=0 sec=54755EFA nano=19054BF0 tcks=0000000 fid=1FFFF ctrl=07 vec=0000 env=00000000

So I also want bytes [0x5C499EA0, end of file), and of file). By using the unix command ls -l I can see the file length is 1548329088.

Part of the psana_test package is a library function to do thiscopy out portions from a source file to a destination file. From a Python shell, I do

...

After creating it, I take a look at it in the data_test directory and make sure the permissions look good for psrel, basically that it is world readable. I also remove my write permission to make it read only.

Make a Unit Test to Process Output

Suppose you have prepared a test file. Sometimes we are testing the output of Psana modules. It can be awkward to get that output into a testable form. One thing I do is run psana on test data using the Python subprocess package. I gather the stdout and stderr, looking for errors from psana as well as checking the module output. Sometimes I parse the module output looking for expected strings, sometimes I save the md5 of the entire output . Other things and test against that. Another thing you might want to do before running the test is check if the xtc file has changed, and check that the output is the same as the last time you tested it.

A useful psana module in the psana_test package is psana_test.dump (see psana - Module Catalog). This dumps information on all the types and data psana sees. We first run it on the above file and inspect the output. After satisfying ourselves that it is correct, we make a unit test as follows. We run the original xtc and the dump output through md5sum. We record these, and then write a unit test that does both md5 checks and compares them.

...