Introduction
The analysis release build system, SConsTools, provides a mechanism for integrating unit tests. Each package in the release, or a package that a user is developing, can have its own tests. All tests can be run by a simple command. Users may find this useful for testing their packages. For packages that psana developers add to the analysis release, these tests are automatically run during the nightly build. This page is primarily for psana developers, to go over how to add unit tests to test packages in the release during nightly builds. There are special considerations to make for tests that are part of the nightly build discussed below.
Creating a Package test directory
As an example, lets make a package with both a Python and a C++ unit test. For the example below, I am starting from the directory rel in my home directory. I make a new release, a new package in the release, and the crucial step is that I make a subdirectory called test in my package:
psanacs051:~/rel $ newrel ana-current unitTestTutorial
psanacs051:~/rel $ cd unitTestTutorial/
psanacs051:~/rel/unitTestTutorial $ sit_setup
psanacs051:~/rel/unitTestTutorial $ newpkg MyPkg
psanacs051:~/rel/unitTestTutorial $ mkdir MyPkg/test
Now when one does
scons test
You are building and running the test target. SConsTools will look in the test subdirectory for all packages. It looks for unit tests in these test subdirectories. It looks for:
- Any file without an extension is treated as a test script. This script will be installed and run.
- Any file that looks like C or C++ code (.c, .cpp extension, etc) is treated as a test program. It will be compiled, installed, and run.
- If a test script or program returns non-zero, it failed and scons will report this.
Other files in the test directory are ignored.
Python Unit Test
Add the file MyPkg/test/myfirst
#!@PYTHON@ import sys if __name__ == '__main__': print "Running my test." sys.exit(1)
When you do scons test, you will get the output
Running UnitTest: "build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst" ************************************************************************************ *** Unit test failed, check log file build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst.utest *** ************************************************************************************
This is because the script returned something non-zero. If you look at the file build/x86_64-rhel5-gcc41-opt/MyPkg/myfirst.utest you see the output of the script.
The syntax @PYTHON@ is explained in the SConsTools page.
Change the sys.exit(1) to sys.exit(0) and the test will succeed. You can also take out the sys.exit line, be default Python will return 0 after the script runs.
C++ Unit Test
A simple C++ test would like this, create the file MyPkg/test/mysecond.cpp
#include <iostream>
int main() {
std::cout << "Cpp test" << std::endl;
return -1;
}
This test will also fail. Note, scons test stops after the first test fails. If you have not changed myfirst to return 0, only one of myfirst and mysecond will be run before failure is reported.
Using Frameworks for Unit Tests
Most people will use a framework for unit tests. You can use whatever framework you like. We have been using the standard Python unit test module, as well as the boost unit_test package.
Python unittest Framework
An example using the Python unit test framework would be
#!@PYTHON@ import sys import unittest class MyTest( unittest.TestCase ) : def setUp(self) : """ Method called to prepare the test fixture. This is called immediately before calling the test method; any exception raised by this method will be considered an error rather than a test failure. """ pass def tearDown(self) : """ Method called immediately after the test method has been called and the result recorded. This is called even if the test method raised an exception, so the implementation in subclasses may need to be particularly careful about checking internal state. Any exception raised by this method will be considered an error rather than a test failure. This method will only be called if the setUp() succeeds, regardless of the outcome of the test method. """ pass def test_mytest(self): a=3 b=4 self.assertEqual(a,b) if __name__ == '__main__': unittest.main(argv=[sys.argv[0], '-v'])
after doing scons test, you will get failure, and see looking at the utest output file will find
******************************************************************************************************** *** Unit test failed, check log file build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest *** ******************************************************************************************************** scons: *** [build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest] Error 256 scons: building terminated because of errors. psana1302:~/rel/unitTestTutorial $ cat build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework.utest test_mytest (__main__.MyTest) ... FAIL ====================================================================== FAIL: test_mytest (__main__.MyTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "build/x86_64-rhel5-gcc41-opt/MyPkg/using_python_framework", line 31, in test_mytest self.assertEqual(a,b) AssertionError: 3 != 4 ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1)
Boost unit_test Framework
exa,