Introduction
The analysis release build system, SConsTools, provides a mechanism for integrating unit tests. Each package in the release, or a package that a user is developing, can have its own tests. All tests can be run by a simple command. Users may find this useful for testing their packages. For packages that psana developers add to the analysis release, these tests are automatically run during the nightly build. This page is primarily for psana developers, to go over how to add unit tests to test packages in the release during nightly builds. There are special considerations to make for tests that are part of the nightly build discussed below.
Creating a Package test directory
As an example, lets make a package with both a Python and a C++ unit test. For the example below, I am starting from the directory rel in my home directory. I make a new release, a new package in the release, and the crucial step is that I make a subdirectory called test in my package:
psanacs051:~/rel $ newrel ana-current unitTestTutorial
psanacs051:~/rel $ cd unitTestTutorial/
psanacs051:~/rel/unitTestTutorial $ sit_setup
psanacs051:~/rel/unitTestTutorial $ newpkg MyPkg
psanacs051:~/rel/unitTestTutorial $ mkdir MyPkg/test
Now when one does
scons test
You are building and running the test target. SConsTools will look in the test subdirectory for all packages. It looks for unit tests in these test subdirectories. It looks for:
- Any file without an extension is treated as a test script. This script will be installed and run.
- Any file that looks like C or C++ code (.c, .cpp extension, etc) is treated as a test program. It will be compiled, installed, and run.
- If a test script or program returns non-zero, it failed and scons will report this.
Other files in the test directory are ignored.
Python Unit Test
Add the file MyPkg/test/myfirst
#!@PYTHON@ import sys if __name__ == '__main__': print "Running my test." sys.exit(1)
When you do scons test, you will get the output
Running UnitTest: "build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst" ************************************************************************************ *** Unit test failed, check log file build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst.utest *** ************************************************************************************
This is because the script returned something non-zero. If you look at the file build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst.utest you see the output of the script.
The syntax @PYTHON@ is explained in the SConsTools page.
Change the sys.exit(1) to sys.exit(0) and the test will succeed. You can also take out the sys.exit line, be default Python will return 0 after the script runs.
C++ Unit Test
A simple C++ test would like this, create the file MyPkg/test/mysecond.cpp
#include <iostream>
int main() {
std::cout << "Cpp test" << std::endl;
return -1;
}This test will also fail. Note, scons test stops after the first test fails. If you have not changed myfirst to return 0, only one of myfirst and mysecond will be run before failure is reported.
Using Frameworks for Unit Tests
It is worthwhile to learn how to use a testing framework. Psana developers are encouraged to use unittest for Python, and boost::unit_test for C++ in order to be consistent with existing tests in the release. However this is not necessary. You can use whatever framework you like.
Python unittest Framework
Below is an example of using unittest with Python. Add the file MyPkg/test/using_python_framework with the following content:
#!@PYTHON@
import sys
import unittest
class MyTest( unittest.TestCase ) :
def setUp(self) :
"""
Method called to prepare the test fixture. This is called immediately
before calling the test method; any exception raised by this method
will be considered an error rather than a test failure.
"""
pass
def tearDown(self) :
"""
Method called immediately after the test method has been called and
the result recorded. This is called even if the test method raised
an exception, so the implementation in subclasses may need to be
particularly careful about checking internal state. Any exception raised
by this method will be considered an error rather than a test failure.
This method will only be called if the setUp() succeeds, regardless
of the outcome of the test method.
"""
pass
def test_mytest(self):
a=3
b=4
self.assertEqual(a,b)
if __name__ == '__main__':
unittest.main(argv=[sys.argv[0], '-v'])
after doing scons test, you will get failure. After looking at the utest output file, one will find
********************************************************************************************************
*** Unit test failed, check log file build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest ***
********************************************************************************************************
scons: *** [build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest] Error 256
scons: building terminated because of errors.
psana1302:~/rel/unitTestTutorial $ cat build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest
test_mytest (__main__.MyTest) ... FAIL
======================================================================
FAIL: test_mytest (__main__.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework", line 31, in test_mytest
self.assertEqual(a,b)
AssertionError: 3 != 4
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
Refer to the documentation https://docs.python.org/2/library/unittest.html for more information on unittest.
Boost unit_test Framework
For an example of using the boost C++ unit test framework, create the file MyPkg/test/using_boost_framework.cpp with the contents (this is mostly copied from the boost website:
#define BOOST_TEST_MODULE MyTest
#include <boost/test/included/unit_test.hpp>
/**
* Simple test suite for module psevt-unit-test.
* See http://www.boost.org/doc/libs/1_36_0/libs/test/doc/html/index.html
*/
#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>
int add( int i, int j ) { return i+j; }
BOOST_AUTO_TEST_CASE( my_test )
{
// seven ways to detect and report the same error:
BOOST_CHECK( add( 2,2 ) == 4 ); // #1 continues on error
BOOST_REQUIRE( add( 2,2 ) == 4 ); // #2 throws on error
if( add( 2,2 ) != 4 )
BOOST_ERROR( "Ouch..." ); // #3 continues on error
if( add( 2,2 ) != 4 )
BOOST_FAIL( "Ouch..." ); // #4 throws on error
if( add( 2,2 ) != 4 ) throw "Ouch..."; // #5 throws on error
BOOST_CHECK_MESSAGE( add( 2,2 ) == 4, // #6 continues on error
"add(..) result: " << add( 2,2 ) );
BOOST_CHECK_EQUAL( add( 2,2 ), 4 ); // #7 continues on error
}
BOOST_AUTO_TEST_CASE( my_test_fail )
{
BOOST_CHECK_EQUAL( add( 2,2 ), 5 );
}
The second test is designed to fail, and the output in the .utest file is
psana1302:~/rel/unitTestTutorial $ cat build/x86_64-rhel5-gcc41-opt/MyPkg/using_boost_framework.utest Running 2 test cases... MyPkg/test/using_boost_framework.cpp(38): error in "my_test_fail": check add( 2,2 ) == 5 failed [4 != 5]
#define BOOST_TEST_MODULE MyTest
#include <boost/test/included/unit_test.hpp>
/**
* Simple test suite for module psevt-unit-test.
* See http://www.boost.org/doc/libs/1_36_0/libs/test/doc/html/index.html
*/
#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>
int add( int i, int j ) { return i+j; }
BOOST_AUTO_TEST_CASE( my_test )
{
// seven ways to detect and report the same error:
BOOST_CHECK( add( 2,2 ) == 4 ); // #1 continues on error
BOOST_REQUIRE( add( 2,2 ) == 4 ); // #2 throws on error
if( add( 2,2 ) != 4 )
BOOST_ERROR( "Ouch..." ); // #3 continues on error
if( add( 2,2 ) != 4 )
BOOST_FAIL( "Ouch..." ); // #4 throws on error
if( add( 2,2 ) != 4 ) throw "Ouch..."; // #5 throws on error
BOOST_CHECK_MESSAGE( add( 2,2 ) == 4, // #6 continues on error
"add(..) result: " << add( 2,2 ) );
BOOST_CHECK_EQUAL( add( 2,2 ), 4 ); // #7 continues on error
}
BOOST_AUTO_TEST_CASE( my_test_fail )
{
BOOST_CHECK_EQUAL( add( 2,2 ), 5 );
}
Overview
Content Tools