# Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
The Vespa test framework helps you write small applications to test C++ code. All interaction with the test framework is done with the use of macros.
The minimal test application looks like this:
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> TEST_MAIN() {}
minimal_test.cpp: info: running test suite 'minimal_test.cpp' minimal_test.cpp: info: summary --- 0 check(s) passed --- 0 check(s) failed minimal_test.cpp: info: CONCLUSION: PASS
The runnable application itself is called a test suite and inherits its name from the cpp file containing the TEST_MAIN macro. Each individual verification of some value is called a check. Checks can be put anywhere, but it is highly recommended that you put them inside tests. Tests are created by a family of macros. Another macro (TEST_RUN_ALL) is used to execute them.
Example with two tests, each containing a single check:
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> TEST("require something") { EXPECT_TRUE(true); } TEST("require something else") { EXPECT_TRUE(true); } TEST_MAIN() { TEST_RUN_ALL(); }
simple_test.cpp: info: running test suite 'simple_test.cpp' simple_test.cpp: info: status_for_test 'require something': PASS simple_test.cpp: info: status_for_test 'require something else': PASS simple_test.cpp: info: test summary --- 2 test(s) passed --- 0 test(s) failed simple_test.cpp: info: imported 2 passed check(s) from 1 thread(s) simple_test.cpp: info: summary --- 2 check(s) passed --- 0 check(s) failed simple_test.cpp: info: CONCLUSION: PASS
All checks are available in two variants. Those with the EXPECT_ prefix allow execution to continue even if a check fails. Those with the ASSERT_ prefix will terminate execution of the current test if it fails.
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> #include <stdexcept> void willThrow() { throw std::runtime_error("This failed"); } TEST("require that checks work") { EXPECT_TRUE(true); EXPECT_FALSE(false); EXPECT_EQUAL(3, 3); EXPECT_NOT_EQUAL(3, 4); EXPECT_APPROX(3.0, 3.1, 0.2); EXPECT_NOT_APPROX(3.0, 3.5, 0.2); EXPECT_LESS(3, 4); EXPECT_LESS_EQUAL(3, 3); EXPECT_GREATER(4, 3); EXPECT_GREATER_EQUAL(4, 4); EXPECT_EXCEPTION(willThrow(), std::runtime_error, "fail"); } TEST("this test will fail") { EXPECT_EQUAL(3, 4); } TEST_MAIN() { TEST_RUN_ALL(); }
checks_test.cpp: info: running test suite 'checks_test.cpp' checks_test.cpp: info: status_for_test 'require that checks work': PASS checks_test.cpp: ERROR: check failure #1: '3 == 4' in thread '0(1)' (checks_test.cpp:24) STATE[0]: 'this test will fail' (checks_test.cpp:23) lhs: 3 rhs: 4 checks_test.cpp: ERROR: status_for_test 'this test will fail': FAIL checks_test.cpp: info: test summary --- 1 test(s) passed --- 1 test(s) failed checks_test.cpp: info: imported 11 passed check(s) from 1 thread(s) checks_test.cpp: info: summary --- 11 check(s) passed --- 1 check(s) failed checks_test.cpp: info: CONCLUSION: FAIL
Checks involving comparison of values typically use == and < operators to compare values. Also; in order to be part of a comparison check, the value must support the << operator to print the value to a string stream in case the check fails.
Sometimes multiple tests wish to use the same predefined state as a starting point. This state is called a test fixture. Test fixtures are untangled from the actual tests and construction/destruction is used to handle their lifetime. When thinking of what can be used as a fixture, think of what can be put after new to create an object.
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> struct Fixture { int value; Fixture() : value(5) {} }; TEST_F("basic fixture", Fixture) { EXPECT_EQUAL(5, f1.value); } TEST_FFF("fancy fixtures", size_t(10), int(5), std::vector<int>(f1, f2)) { EXPECT_EQUAL(10u, f1); EXPECT_EQUAL(5, f2); ASSERT_EQUAL(10u, f3.size()); EXPECT_EQUAL(5, f3[7]); } TEST_MAIN() { TEST_RUN_ALL(); }
fixtures_test.cpp: info: running test suite 'fixtures_test.cpp' fixtures_test.cpp: info: status_for_test 'basic fixture': PASS fixtures_test.cpp: info: status_for_test 'fancy fixtures': PASS fixtures_test.cpp: info: test summary --- 2 test(s) passed --- 0 test(s) failed fixtures_test.cpp: info: imported 5 passed check(s) from 1 thread(s) fixtures_test.cpp: info: summary --- 5 check(s) passed --- 0 check(s) failed fixtures_test.cpp: info: CONCLUSION: PASS
One of the most novel features of the test framework is the ability to write multi-threaded tests. Multi-threaded tests are created by using test macros containing _MT and supplying a thread count. All threads will execute the block of code encapsulated by the test. The test fixtures are shared among all threads. In addition, each thread has a variable named num_threads containing the total number of threads running the test and a variable named thread_id identifying the thread.
The TEST_BARRIER() macro can be used inside the test to synchronize the threads. The macro will block execution of each thread invoking it until all threads have invoked it.
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> TEST_MT_F("multiple threads", 2, std::vector<size_t>(num_threads)) { ASSERT_EQUAL(num_threads, f1.size()); f1[thread_id] = thread_id; TEST_BARRIER(); if (thread_id == 0) { TEST_TRACE(); EXPECT_EQUAL(1u, f1[1]); } else { TEST_TRACE(); EXPECT_EQUAL(0u, f1[0]); } } TEST_MAIN() { TEST_RUN_ALL(); }
threads_test.cpp: info: running test suite 'threads_test.cpp' threads_test.cpp: info: trace: thread '0(2)' (threads_test.cpp:9) threads_test.cpp: info: trace: thread '1(2)' (threads_test.cpp:12) threads_test.cpp: info: status_for_test 'multiple threads': PASS threads_test.cpp: info: test summary --- 1 test(s) passed --- 0 test(s) failed threads_test.cpp: info: imported 4 passed check(s) from 2 thread(s) threads_test.cpp: info: summary --- 4 check(s) passed --- 0 check(s) failed threads_test.cpp: info: CONCLUSION: PASS
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> #include <vespa/vespalib/util/box.h> using namespace vespalib; void checkValues(const std::vector<int> &values, size_t n) { ASSERT_EQUAL(n, values.size()); for (size_t i = 0; i < n; ++i) { EXPECT_EQUAL(int(10 + (10 * i)), values[i]); } } TEST("require that boxes can be created and converted to vector") { Box<int> box; box.add(10).add(20).add(30); checkValues(box, 3); } TEST("require that boxes can be created in place") { checkValues(Box<int>().add(10).add(20).add(30), 3); } TEST("require that make_box works") { checkValues(make_box(10), 1); checkValues(make_box(10, 20), 2); checkValues(make_box(10, 20, 30), 3); checkValues(make_box(10, 20, 30, 40), 4); checkValues(make_box(10, 20, 30, 40, 50), 5); } TEST_MAIN() { TEST_RUN_ALL(); }
box_test.cpp: info: running test suite 'box_test.cpp' box_test.cpp: info: status_for_test 'require that boxes can be created and converted to vector': PASS box_test.cpp: info: status_for_test 'require that boxes can be created in place': PASS box_test.cpp: info: status_for_test 'require that make_box works': PASS box_test.cpp: info: test summary --- 3 test(s) passed --- 0 test(s) failed box_test.cpp: info: imported 28 passed check(s) from 1 thread(s) box_test.cpp: info: summary --- 28 check(s) passed --- 0 check(s) failed box_test.cpp: info: CONCLUSION: PASS
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> #include <vespa/vespalib/util/barrier.h> using namespace vespalib; struct Fixture { Barrier barrier; CountDownLatch latch; Fixture(size_t n) : barrier(n), latch(n) {} }; TEST_MT_F("require that barriers are satisfied by the appropriate number of threads", 3, Fixture(num_threads)) { if (thread_id == 0) { f1.latch.countDown(); EXPECT_FALSE(f.latch.await(250)); EXPECT_TRUE(f.barrier.await()); EXPECT_TRUE(f.latch.await(25000)); } else { EXPECT_TRUE(f1.barrier.await()); f1.latch.countDown(); } } TEST_MT_F("require that barriers can be used multiple times", 3, Fixture(num_threads)) { EXPECT_TRUE(f1.barrier.await()); EXPECT_TRUE(f1.barrier.await()); if (thread_id == 0) { f1.latch.countDown(); EXPECT_FALSE(f.latch.await(250)); EXPECT_TRUE(f.barrier.await()); EXPECT_TRUE(f.latch.await(25000)); } else { EXPECT_TRUE(f1.barrier.await()); f1.latch.countDown(); } } TEST_MT_F("require that barriers can be broken", 3, Fixture(num_threads)) { EXPECT_TRUE(f1.barrier.await()); if (thread_id == 0) { f1.latch.countDown(); EXPECT_FALSE(f.latch.await(250)); f1.barrier.destroy(); EXPECT_TRUE(f.latch.await(25000)); } else { EXPECT_FALSE(f1.barrier.await()); f1.latch.countDown(); } EXPECT_FALSE(f1.barrier.await()); } TEST_MT_F("require that barriers cannot be retroactively broken", 100, Barrier(num_threads)) { EXPECT_TRUE(f1.await()); f1.destroy(); EXPECT_FALSE(f1.await()); } TEST_MAIN() { TEST_RUN_ALL(); }
barrier_test.cpp: info: running test suite 'barrier_test.cpp' barrier_test.cpp: info: status_for_test 'require that barriers are satisfied by the appropriate number of threads': PASS barrier_test.cpp: info: status_for_test 'require that barriers can be used multiple times': PASS barrier_test.cpp: info: status_for_test 'require that barriers can be broken': PASS barrier_test.cpp: info: status_for_test 'require that barriers cannot be retroactively broken': PASS barrier_test.cpp: info: test summary --- 4 test(s) passed --- 0 test(s) failed barrier_test.cpp: info: imported 226 passed check(s) from 100 thread(s) barrier_test.cpp: info: summary --- 226 check(s) passed --- 0 check(s) failed barrier_test.cpp: info: CONCLUSION: PASS
// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. #include <vespa/vespalib/testkit/test_kit.h> #include <vespa/vespalib/util/dual_merge_director.h> using namespace vespalib; struct MySource : public DualMergeDirector::Source { bool typeA; std::string data; std::string diff; MySource(bool a, size_t num_sources, size_t source_id) : typeA(a), data(num_sources, '0'), diff(num_sources, '5') { if (source_id < num_sources) { data[source_id] = '1'; diff[source_id] = '6'; } } virtual void merge(Source &mt) { MySource &rhs = static_cast<MySource&>(mt); ASSERT_EQUAL(typeA, rhs.typeA); ASSERT_EQUAL(data.size(), rhs.data.size()); for (size_t i = 0; i < data.size(); ++i) { int d = (rhs.data[i] - '0'); data[i] += d; diff[i] += d; rhs.diff[i] -= d; } } void verifyFinal() const { EXPECT_EQUAL(std::string(data.size(), '1'), data); EXPECT_EQUAL(std::string(diff.size(), '6'), diff); } void verifyIntermediate() const { EXPECT_EQUAL(std::string(diff.size(), '5'), diff); } }; TEST_MT_F("require that merging works", 64, std::unique_ptr<DualMergeDirector>()) { for (size_t use_threads = 1; use_threads <= num_threads; ++use_threads) { MySource sourceA(true, use_threads, thread_id); MySource sourceB(false, use_threads, thread_id); if (thread_id == 0) { f1.reset(new DualMergeDirector(use_threads)); } TEST_BARRIER(); if (thread_id < use_threads) { f1->dualMerge(thread_id, sourceA, sourceB); } TEST_BARRIER(); if (thread_id == 0) { sourceA.verifyFinal(); sourceB.verifyFinal(); } else if (thread_id < use_threads) { sourceA.verifyIntermediate(); sourceB.verifyIntermediate(); } } } TEST_MAIN() { TEST_RUN_ALL(); }
dual_merge_director_test.cpp: info: running test suite 'dual_merge_director_test.cpp' dual_merge_director_test.cpp: info: status_for_test 'require that merging works': PASS dual_merge_director_test.cpp: info: test summary --- 1 test(s) passed --- 0 test(s) failed dual_merge_director_test.cpp: info: imported 12352 passed check(s) from 64 thread(s) dual_merge_director_test.cpp: info: summary --- 12352 check(s) passed --- 0 check(s) failed dual_merge_director_test.cpp: info: CONCLUSION: PASS
TEST_MAIN()
TEST_RUN_ALL()
TEST(name)
TEST_F(name, fixture)
TEST_FF(name, fixture1, fixture2)
TEST_FFF(name, fixture1, fixture2, fixture3)
TEST_MT(name, threads)
TEST_MT_F(name, threads, fixture)
TEST_MT_FF(name, threads, fixture1, fixture2)
TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)
IGNORE_TEST(name)
IGNORE_TEST_F(name, fixture)
IGNORE_TEST_FF(name, fixture1, fixture2)
IGNORE_TEST_FFF(name, fixture1, fixture2, fixture3)
IGNORE_TEST_MT(name, threads)
IGNORE_TEST_MT_F(name, threads, fixture)
IGNORE_TEST_MT_FF(name, threads, fixture1, fixture2)
IGNORE_TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)
ASSERT_TRUE(rc)
ASSERT_FALSE(rc)
ASSERT_EQUAL(a, b)
ASSERT_NOT_EQUAL(a, b)
ASSERT_APPROX(a, b, eps)
ASSERT_NOT_APPROX(a, b, eps)
ASSERT_LESS(a, b)
ASSERT_LESS_EQUAL(a, b)
ASSERT_GREATER(a, b)
ASSERT_GREATER_EQUAL(a, b)
ASSERT_EXCEPTION(statement, exception_type, msg_substr)
TEST_FATAL(msg)
EXPECT_TRUE(rc)
EXPECT_FALSE(rc)
EXPECT_EQUAL(a, b)
EXPECT_NOT_EQUAL(a, b)
EXPECT_APPROX(a, b, eps)
EXPECT_NOT_APPROX(a, b, eps)
EXPECT_LESS(a, b)
EXPECT_LESS_EQUAL(a, b)
EXPECT_GREATER(a, b)
EXPECT_GREATER_EQUAL(a, b)
EXPECT_EXCEPTION(statement, exception_type, msg_substr)
TEST_ERROR(msg)
TEST_BARRIER()
TEST_DO(doit)
TEST_STATE(msg)
TEST_TRACE()
TEST_FLUSH()
TEST_THREAD(name)
TEST_DEBUG(lhsFile, rhsFile)
TEST_MAIN_WITH_PROCESS_PROXY()