3.10. Tests¶
In previous section we have checked that executable is working by finding it in binary tree and running it explicitly. If we have several executables or want to run the same executable with different parameters we can organize everything into test suite driven by CTest tool.
CMake documentation
Examples on GitHub
Creating two executables:
# Top-level CMakeLists.txt
cmake_minimum_required(VERSION 2.8)
project(foo)
add_executable(boo boo.cpp)
add_executable(bar bar.cpp)
enable_testing()
add_test(NAME boo COMMAND boo)
add_test(NAME bar COMMAND bar)
add_test(NAME bar-with-args COMMAND bar arg1 arg2 arg3)
Executable boo:
#include <iostream> // std::cout
int main() {
std::cout << "boo" << std::endl;
}
Executable bar:
#include <iostream> // std::cout
int main(int argc, char** argv) {
std::cout << "bar argc: " << argc << std::endl;
for (int i=1; i<argc; ++i) {
std::cout << "argv[" << i << "]: " << argv[i] << std::endl;
}
}
Testing allowed by enable_testing directive which must be
called in the root directory:
# Top-level CMakeLists.txt
cmake_minimum_required(VERSION 2.8)
project(foo)
add_executable(boo boo.cpp)
add_executable(bar bar.cpp)
enable_testing()
add_test(NAME boo COMMAND boo)
add_test(NAME bar COMMAND bar)
add_test(NAME bar-with-args COMMAND bar arg1 arg2 arg3)
Come up with some tests name and specify executable arguments if needed:
# Top-level CMakeLists.txt
cmake_minimum_required(VERSION 2.8)
project(foo)
add_executable(boo boo.cpp)
add_executable(bar bar.cpp)
enable_testing()
add_test(NAME boo COMMAND boo)
add_test(NAME bar COMMAND bar)
add_test(NAME bar-with-args COMMAND bar arg1 arg2 arg3)
Configure and build project:
[examples]> rm -rf _builds
[examples]> cmake -Htest-examples/simple -B_builds
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /.../examples/_builds
[examples]> cmake --build _builds
Scanning dependencies of target boo
[ 25%] Building CXX object CMakeFiles/boo.dir/boo.cpp.o
[ 50%] Linking CXX executable boo
[ 50%] Built target boo
Scanning dependencies of target bar
[ 75%] Building CXX object CMakeFiles/bar.dir/bar.cpp.o
[100%] Linking CXX executable bar
[100%] Built target bar
Enter _builds directory and use ctest tool to run all tests:
[examples]> cd _builds
[examples/_builds]> ctest
Test project /.../examples/_builds
Start 1: boo
1/3 Test #1: boo .............................. Passed 0.00 sec
Start 2: bar
2/3 Test #2: bar .............................. Passed 0.00 sec
Start 3: bar-with-args
3/3 Test #3: bar-with-args .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 0.02 sec
3.10.1. Multi-config testing¶
Note that for the
multi-configuration generators
you have to specify build type while running ctest. Otherwise no
tests will be run. Example of Visual Studio project:
[examples\_builds]> ctest
Test project C:/.../examples/_builds
Start 1: boo
Test not available without configuration. (Missing "-C <config>"?)
1/3 Test #1: boo ..............................***Not Run 0.00 sec
Start 2: bar
Test not available without configuration. (Missing "-C <config>"?)
2/3 Test #2: bar ..............................***Not Run 0.00 sec
Start 3: bar-with-args
Test not available without configuration. (Missing "-C <config>"?)
3/3 Test #3: bar-with-args ....................***Not Run 0.00 sec
0% tests passed, 3 tests failed out of 3
Total Test time (real) = 0.02 sec
The following tests FAILED:
1 - boo (Not Run)
2 - bar (Not Run)
3 - bar-with-args (Not Run)
Errors while running CTest
Just add -C Debug to test with Debug build type:
[examples\_builds]> ctest -C Debug
Test project C:/.../examples/_builds
Start 1: boo
1/3 Test #1: boo .............................. Passed 0.04 sec
Start 2: bar
2/3 Test #2: bar .............................. Passed 0.02 sec
Start 3: bar-with-args
3/3 Test #3: bar-with-args .................... Passed 0.01 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 0.09 sec
3.10.2. Verbose output¶
By default only Passed/Failed information is shown. You can control
tests output by -V/-VV options:
[examples/_builds]> ctest -VV
...
test 1
Start 1: boo
1: Test command: /.../examples/_builds/boo
1: Test timeout computed to be: 9.99988e+06
1: boo
1/3 Test #1: boo .............................. Passed 0.00 sec
test 2
Start 2: bar
2: Test command: /.../examples/_builds/bar
2: Test timeout computed to be: 9.99988e+06
2: bar argc: 1
2/3 Test #2: bar .............................. Passed 0.00 sec
test 3
Start 3: bar-with-args
3: Test command: /.../examples/_builds/bar "arg1" "arg2" "arg3"
3: Test timeout computed to be: 9.99988e+06
3: bar argc: 4
3: argv[1]: arg1
3: argv[2]: arg2
3: argv[3]: arg3
3/3 Test #3: bar-with-args .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 3
Total Test time (real) = 0.01 sec
3.10.3. Subset of tests¶
It is possible to run only subset of tests instead of all suite. For example
running all tests with bar pattern in name by using regular expression:
[examples/_builds]> ctest -R bar
Test project /.../examples/_builds
Start 2: bar
1/2 Test #2: bar .............................. Passed 0.00 sec
Start 3: bar-with-args
2/2 Test #3: bar-with-args .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.01 sec
Or only bar test:
[examples/_builds]> ctest -R '^bar$'
Test project /.../examples/_builds
Start 2: bar
1/1 Test #2: bar .............................. Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.01 sec