Vespa Test Framework Tutorial

The Vespa test framework helps you write small applications to test C++ code. All interaction with the test framework is done with the use of macros.

The minimal test application looks like this:

minimal_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>

TEST_MAIN() {}

minimal_test.cpp: info:  running test suite 'minimal_test.cpp'
minimal_test.cpp: info:  summary --- 0 check(s) passed --- 0 check(s) failed
minimal_test.cpp: info:  CONCLUSION: PASS

The runnable application itself is called a test suite and inherits its name from the cpp file containing the TEST_MAIN macro. Each individual verification of some value is called a check. Checks can be put anywhere, but it is highly recommended that you put them inside tests. Tests are created by a family of macros. Another macro (TEST_RUN_ALL) is used to execute them.

Example with two tests, each containing a single check:

simple_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>

TEST("require something") {
    EXPECT_TRUE(true);
}

TEST("require something else") {
    EXPECT_TRUE(true);
}

TEST_MAIN() { TEST_RUN_ALL(); }

simple_test.cpp: info:  running test suite 'simple_test.cpp'
simple_test.cpp: info:  status_for_test 'require something': PASS
simple_test.cpp: info:  status_for_test 'require something else': PASS
simple_test.cpp: info:  test summary --- 2 test(s) passed --- 0 test(s) failed
simple_test.cpp: info:  imported 2 passed check(s) from 1 thread(s)
simple_test.cpp: info:  summary --- 2 check(s) passed --- 0 check(s) failed
simple_test.cpp: info:  CONCLUSION: PASS

All checks are available in two variants. Those with the EXPECT_ prefix allow execution to continue even if a check fails. Those with the ASSERT_ prefix will terminate execution of the current test if it fails.

checks_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>
#include <stdexcept>

void willThrow() {
    throw std::runtime_error("This failed");
}

TEST("require that checks work") {
    EXPECT_TRUE(true);
    EXPECT_FALSE(false);
    EXPECT_EQUAL(3, 3);
    EXPECT_NOT_EQUAL(3, 4);
    EXPECT_APPROX(3.0, 3.1, 0.2);
    EXPECT_NOT_APPROX(3.0, 3.5, 0.2);
    EXPECT_LESS(3, 4);
    EXPECT_LESS_EQUAL(3, 3);
    EXPECT_GREATER(4, 3);
    EXPECT_GREATER_EQUAL(4, 4);
    EXPECT_EXCEPTION(willThrow(), std::runtime_error, "fail");
}

TEST("this test will fail") {
    EXPECT_EQUAL(3, 4);
}

TEST_MAIN() { TEST_RUN_ALL(); }

checks_test.cpp: info:  running test suite 'checks_test.cpp'
checks_test.cpp: info:  status_for_test 'require that checks work': PASS
checks_test.cpp: ERROR: check failure #1: '3 == 4' in thread '0(1)' (checks_test.cpp:24)
    STATE[0]: 'this test will fail' (checks_test.cpp:23)
lhs: 3
rhs: 4
checks_test.cpp: ERROR: status_for_test 'this test will fail': FAIL
checks_test.cpp: info:  test summary --- 1 test(s) passed --- 1 test(s) failed
checks_test.cpp: info:  imported 11 passed check(s) from 1 thread(s)
checks_test.cpp: info:  summary --- 11 check(s) passed --- 1 check(s) failed
checks_test.cpp: info:  CONCLUSION: FAIL

Checks involving comparison of values typically use == and < operators to compare values. Also; in order to be part of a comparison check, the value must support the << operator to print the value to a string stream in case the check fails.

Sometimes multiple tests wish to use the same predefined state as a starting point. This state is called a test fixture. Test fixtures are untangled from the actual tests and construction/destruction is used to handle their lifetime. When thinking of what can be used as a fixture, think of what can be put after new to create an object.

A single test can have multiple test fixtures.
The number of Fs in the test macro denotes the number of fixtures.
Inside the test, fixtures are available as f1, f2 and so forth.
Test fixtures can be parameterized with constructor parameters.
Checks can be performed inside test fixture constructors.
A test fixture constructor can take other test fixtures as input.

fixtures_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>

struct Fixture {
    int value;
    Fixture() : value(5) {}
};

TEST_F("basic fixture", Fixture) {
    EXPECT_EQUAL(5, f1.value);
}

TEST_FFF("fancy fixtures", size_t(10), int(5), std::vector<int>(f1, f2)) {
    EXPECT_EQUAL(10u, f1);
    EXPECT_EQUAL(5, f2);
    ASSERT_EQUAL(10u, f3.size());
    EXPECT_EQUAL(5, f3[7]);
}

TEST_MAIN() { TEST_RUN_ALL(); }

fixtures_test.cpp: info:  running test suite 'fixtures_test.cpp'
fixtures_test.cpp: info:  status_for_test 'basic fixture': PASS
fixtures_test.cpp: info:  status_for_test 'fancy fixtures': PASS
fixtures_test.cpp: info:  test summary --- 2 test(s) passed --- 0 test(s) failed
fixtures_test.cpp: info:  imported 5 passed check(s) from 1 thread(s)
fixtures_test.cpp: info:  summary --- 5 check(s) passed --- 0 check(s) failed
fixtures_test.cpp: info:  CONCLUSION: PASS

One of the most novel features of the test framework is the ability to write multi-threaded tests. Multi-threaded tests are created by using test macros containing _MT and supplying a thread count. All threads will execute the block of code encapsulated by the test. The test fixtures are shared among all threads. In addition, each thread has a variable named num_threads containing the total number of threads running the test and a variable named thread_id identifying the thread.

The TEST_BARRIER() macro can be used inside the test to synchronize the threads. The macro will block execution of each thread invoking it until all threads have invoked it.

threads_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>

TEST_MT_F("multiple threads", 2, std::vector<size_t>(num_threads)) {
    ASSERT_EQUAL(num_threads, f1.size());
    f1[thread_id] = thread_id;
    TEST_BARRIER();
    if (thread_id == 0) {
        TEST_TRACE();
        EXPECT_EQUAL(1u, f1[1]);
    } else {
        TEST_TRACE();
        EXPECT_EQUAL(0u, f1[0]);
    }
}

TEST_MAIN() { TEST_RUN_ALL(); }

threads_test.cpp: info:  running test suite 'threads_test.cpp'
threads_test.cpp: info:  trace: thread '0(2)' (threads_test.cpp:9)
threads_test.cpp: info:  trace: thread '1(2)' (threads_test.cpp:12)
threads_test.cpp: info:  status_for_test 'multiple threads': PASS
threads_test.cpp: info:  test summary --- 1 test(s) passed --- 0 test(s) failed
threads_test.cpp: info:  imported 4 passed check(s) from 2 thread(s)
threads_test.cpp: info:  summary --- 4 check(s) passed --- 0 check(s) failed
threads_test.cpp: info:  CONCLUSION: PASS

box_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>
#include <vespa/vespalib/util/box.h>

using namespace vespalib;

void checkValues(const std::vector<int> &values, size_t n) {
    ASSERT_EQUAL(n, values.size());
    for (size_t i = 0; i < n; ++i) {
        EXPECT_EQUAL(int(10 + (10 * i)), values[i]);
    }
}

TEST("require that boxes can be created and converted to vector") {
    Box<int> box;
    box.add(10).add(20).add(30);
    checkValues(box, 3);
}

TEST("require that boxes can be created in place") {
    checkValues(Box<int>().add(10).add(20).add(30), 3);
}

TEST("require that make_box works") {
    checkValues(make_box(10), 1);
    checkValues(make_box(10, 20), 2);
    checkValues(make_box(10, 20, 30), 3);
    checkValues(make_box(10, 20, 30, 40), 4);
    checkValues(make_box(10, 20, 30, 40, 50), 5);
}

TEST_MAIN() { TEST_RUN_ALL(); }

box_test.cpp: info:  running test suite 'box_test.cpp'
box_test.cpp: info:  status_for_test 'require that boxes can be created and converted to vector': PASS
box_test.cpp: info:  status_for_test 'require that boxes can be created in place': PASS
box_test.cpp: info:  status_for_test 'require that make_box works': PASS
box_test.cpp: info:  test summary --- 3 test(s) passed --- 0 test(s) failed
box_test.cpp: info:  imported 28 passed check(s) from 1 thread(s)
box_test.cpp: info:  summary --- 28 check(s) passed --- 0 check(s) failed
box_test.cpp: info:  CONCLUSION: PASS

barrier_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>
#include <vespa/vespalib/util/barrier.h>

using namespace vespalib;

struct Fixture {
    Barrier barrier;
    CountDownLatch latch;
    Fixture(size_t n) : barrier(n), latch(n) {}
};

TEST_MT_F("require that barriers are satisfied by the appropriate number of threads", 3, Fixture(num_threads)) {
    if (thread_id == 0) {
        f1.latch.countDown();
        EXPECT_FALSE(f.latch.await(250));
        EXPECT_TRUE(f.barrier.await());
        EXPECT_TRUE(f.latch.await(25000));
    } else {
        EXPECT_TRUE(f1.barrier.await());
        f1.latch.countDown();
    }
}

TEST_MT_F("require that barriers can be used multiple times", 3, Fixture(num_threads)) {
    EXPECT_TRUE(f1.barrier.await());
    EXPECT_TRUE(f1.barrier.await());
    if (thread_id == 0) {
        f1.latch.countDown();
        EXPECT_FALSE(f.latch.await(250));
        EXPECT_TRUE(f.barrier.await());
        EXPECT_TRUE(f.latch.await(25000));
    } else {
        EXPECT_TRUE(f1.barrier.await());
        f1.latch.countDown();
    }
}

TEST_MT_F("require that barriers can be broken", 3, Fixture(num_threads)) {
    EXPECT_TRUE(f1.barrier.await());
    if (thread_id == 0) {
        f1.latch.countDown();
        EXPECT_FALSE(f.latch.await(250));
        f1.barrier.destroy();
        EXPECT_TRUE(f.latch.await(25000));
    } else {
        EXPECT_FALSE(f1.barrier.await());
        f1.latch.countDown();
    }
    EXPECT_FALSE(f1.barrier.await());
}

TEST_MT_F("require that barriers cannot be retroactively broken", 100, Barrier(num_threads)) {
    EXPECT_TRUE(f1.await());
    f1.destroy();
    EXPECT_FALSE(f1.await());
}

TEST_MAIN() { TEST_RUN_ALL(); }

barrier_test.cpp: info:  running test suite 'barrier_test.cpp'
barrier_test.cpp: info:  status_for_test 'require that barriers are satisfied by the appropriate number of threads': PASS
barrier_test.cpp: info:  status_for_test 'require that barriers can be used multiple times': PASS
barrier_test.cpp: info:  status_for_test 'require that barriers can be broken': PASS
barrier_test.cpp: info:  status_for_test 'require that barriers cannot be retroactively broken': PASS
barrier_test.cpp: info:  test summary --- 4 test(s) passed --- 0 test(s) failed
barrier_test.cpp: info:  imported 226 passed check(s) from 100 thread(s)
barrier_test.cpp: info:  summary --- 226 check(s) passed --- 0 check(s) failed
barrier_test.cpp: info:  CONCLUSION: PASS

dual_merge_director_test.cpp

// Copyright Yahoo. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
#include <vespa/vespalib/testkit/test_kit.h>
#include <vespa/vespalib/util/dual_merge_director.h>

using namespace vespalib;

struct MySource : public DualMergeDirector::Source {

    bool typeA;
    std::string data;
    std::string diff;

    MySource(bool a, size_t num_sources, size_t source_id)
        : typeA(a),
          data(num_sources, '0'),
          diff(num_sources, '5')
    {
        if (source_id < num_sources) {
            data[source_id] = '1';
            diff[source_id] = '6';
        }
    }
    virtual void merge(Source &mt) {
        MySource &rhs = static_cast<MySource&>(mt);
        ASSERT_EQUAL(typeA, rhs.typeA);
        ASSERT_EQUAL(data.size(), rhs.data.size());
        for (size_t i = 0; i < data.size(); ++i) {
            int d = (rhs.data[i] - '0');
            data[i] += d;
            diff[i] += d;
            rhs.diff[i] -= d;
        }
    }
    void verifyFinal() const {
        EXPECT_EQUAL(std::string(data.size(), '1'), data);
        EXPECT_EQUAL(std::string(diff.size(), '6'), diff);
    }
    void verifyIntermediate() const {
        EXPECT_EQUAL(std::string(diff.size(), '5'), diff);
    }
};

TEST_MT_F("require that merging works", 64, std::unique_ptr<DualMergeDirector>()) {
    for (size_t use_threads = 1; use_threads <= num_threads; ++use_threads) {
        MySource sourceA(true, use_threads, thread_id);
        MySource sourceB(false, use_threads, thread_id);
        if (thread_id == 0) {
            f1.reset(new DualMergeDirector(use_threads));
        }
        TEST_BARRIER();
        if (thread_id < use_threads) {
            f1->dualMerge(thread_id, sourceA, sourceB);
        }
        TEST_BARRIER();
        if (thread_id == 0) {
            sourceA.verifyFinal();
            sourceB.verifyFinal();
        } else if (thread_id < use_threads) {
            sourceA.verifyIntermediate();
            sourceB.verifyIntermediate();
        }
    }
}

TEST_MAIN() { TEST_RUN_ALL(); }

dual_merge_director_test.cpp: info:  running test suite 'dual_merge_director_test.cpp'
dual_merge_director_test.cpp: info:  status_for_test 'require that merging works': PASS
dual_merge_director_test.cpp: info:  test summary --- 1 test(s) passed --- 0 test(s) failed
dual_merge_director_test.cpp: info:  imported 12352 passed check(s) from 64 thread(s)
dual_merge_director_test.cpp: info:  summary --- 12352 check(s) passed --- 0 check(s) failed
dual_merge_director_test.cpp: info:  CONCLUSION: PASS

Overall Execution Macros

TEST_MAIN()
TEST_RUN_ALL()

Test Creation Macros

TEST(name)
TEST_F(name, fixture)
TEST_FF(name, fixture1, fixture2)
TEST_FFF(name, fixture1, fixture2, fixture3)

TEST_MT(name, threads)
TEST_MT_F(name, threads, fixture)
TEST_MT_FF(name, threads, fixture1, fixture2)
TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)

IGNORE_TEST(name)
IGNORE_TEST_F(name, fixture)
IGNORE_TEST_FF(name, fixture1, fixture2)
IGNORE_TEST_FFF(name, fixture1, fixture2, fixture3)

IGNORE_TEST_MT(name, threads)
IGNORE_TEST_MT_F(name, threads, fixture)
IGNORE_TEST_MT_FF(name, threads, fixture1, fixture2)
IGNORE_TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)

Check Macros

ASSERT_TRUE(rc)
ASSERT_FALSE(rc)
ASSERT_EQUAL(a, b)
ASSERT_NOT_EQUAL(a, b)
ASSERT_APPROX(a, b, eps)
ASSERT_NOT_APPROX(a, b, eps)
ASSERT_LESS(a, b)
ASSERT_LESS_EQUAL(a, b)
ASSERT_GREATER(a, b)
ASSERT_GREATER_EQUAL(a, b)
ASSERT_EXCEPTION(statement, exception_type, msg_substr)
TEST_FATAL(msg)

EXPECT_TRUE(rc)
EXPECT_FALSE(rc)
EXPECT_EQUAL(a, b)
EXPECT_NOT_EQUAL(a, b)
EXPECT_APPROX(a, b, eps)
EXPECT_NOT_APPROX(a, b, eps)
EXPECT_LESS(a, b)
EXPECT_LESS_EQUAL(a, b)
EXPECT_GREATER(a, b)
EXPECT_GREATER_EQUAL(a, b)
EXPECT_EXCEPTION(statement, exception_type, msg_substr)
TEST_ERROR(msg)

Thread Macros

TEST_BARRIER()

State Tracking Macros

TEST_DO(doit)
TEST_STATE(msg)

Macros of Limited Use

TEST_TRACE()
TEST_FLUSH()
TEST_THREAD(name)
TEST_DEBUG(lhsFile, rhsFile)
TEST_MAIN_WITH_PROCESS_PROXY()