diff options
author | Jon Bratseth <bratseth@yahoo-inc.com> | 2016-06-15 23:09:44 +0200 |
---|---|---|
committer | Jon Bratseth <bratseth@yahoo-inc.com> | 2016-06-15 23:09:44 +0200 |
commit | 72231250ed81e10d66bfe70701e64fa5fe50f712 (patch) | |
tree | 2728bba1131a6f6e5bdf95afec7d7ff9358dac50 /vespalib/src/tests/tutorial/tutorial.html |
Publish
Diffstat (limited to 'vespalib/src/tests/tutorial/tutorial.html')
-rw-r--r-- | vespalib/src/tests/tutorial/tutorial.html | 628 |
1 files changed, 628 insertions, 0 deletions
diff --git a/vespalib/src/tests/tutorial/tutorial.html b/vespalib/src/tests/tutorial/tutorial.html new file mode 100644 index 00000000000..744dd9541e2 --- /dev/null +++ b/vespalib/src/tests/tutorial/tutorial.html @@ -0,0 +1,628 @@ +<!DOCTYPE html> +<!-- Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. --> +<html lang="en"> +<head> + <meta charset="utf-8" /> + <title>Vespa Test Framework Tutorial</title> +# Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + <style type="text/css"> + + body { + margin: 3em; + } + + body, p, div, div, span, dl, li, pre { + font-size: 16px; + line-height: 22px; + } + + header h1 { + font-size: 54px; + line-height: 1; + } + + section { + padding-top: 60px; + } + + ol.linenums { + padding-left: 20px; + } + + pre, pre * { + font-size: 14px !important; + } + + li.L0, li.L1, li.L2, li.L3, li.L4, li.L5, li.L6, li.L7, li.L8, li.L9 + { + color: #bbb; + list-style-type: decimal; + } + + h2 { + padding-top: 20px; + padding-bottom: 10px; + } + + pre.output { + border: 1px solid #888; + } + + code { + color: #444; + border: 0; + background-color: transparent; + } + + </style> +</head> +<body> + +<header> +<h1>Vespa Test Framework Tutorial</h1> +</header> + +<section> +<div class="page-header"> +<h1>Introduction</h1> +</div> + +<p> +The Vespa test framework helps you write small applications to test +C++ code. All interaction with the test framework is done with the use +of macros. +</p> + +<p> +The minimal test application looks like this: +</p> + +<div class="example" id="minimal"> +<h2>minimal_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> + +TEST_MAIN() {} +</pre> +<pre class="output"> +minimal_test.cpp: info: running test suite 'minimal_test.cpp' +minimal_test.cpp: info: summary --- 0 check(s) passed --- 0 check(s) failed +minimal_test.cpp: info: CONCLUSION: PASS +</pre> +</div> + +<p> +The runnable application itself is called a <strong>test +suite</strong> and inherits its name from the cpp file containing the +TEST_MAIN macro. Each individual verification of some value is called +a <strong>check</strong>. Checks can be put anywhere, but it is highly +recommended that you put them inside <strong>tests</strong>. Tests are +created by a family of macros. Another macro (TEST_RUN_ALL) is used to +execute them. +</p> + +<p> +Example with two tests, each containing a single check: +</p> + +<div class="example" id="simple"> +<h2>simple_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> + +TEST("require something") { + EXPECT_TRUE(true); +} + +TEST("require something else") { + EXPECT_TRUE(true); +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +simple_test.cpp: info: running test suite 'simple_test.cpp' +simple_test.cpp: info: status_for_test 'require something': PASS +simple_test.cpp: info: status_for_test 'require something else': PASS +simple_test.cpp: info: test summary --- 2 test(s) passed --- 0 test(s) failed +simple_test.cpp: info: imported 2 passed check(s) from 1 thread(s) +simple_test.cpp: info: summary --- 2 check(s) passed --- 0 check(s) failed +simple_test.cpp: info: CONCLUSION: PASS +</pre> +</div> +</section> + + +<section> +<div class="page-header"> +<h1>Checks</h1> +</div> + +<p> +All checks are available in two variants. Those with the +<strong>EXPECT_</strong> prefix allow execution to continue even if a +check fails. Those with the <strong>ASSERT_</strong> prefix will +terminate execution of the current test if it fails. +</p> + +<div class="example" id="checks"> +<h2>checks_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> +#include <stdexcept> + +void willThrow() { + throw std::runtime_error("This failed"); +} + +TEST("require that checks work") { + EXPECT_TRUE(true); + EXPECT_FALSE(false); + EXPECT_EQUAL(3, 3); + EXPECT_NOT_EQUAL(3, 4); + EXPECT_APPROX(3.0, 3.1, 0.2); + EXPECT_NOT_APPROX(3.0, 3.5, 0.2); + EXPECT_LESS(3, 4); + EXPECT_LESS_EQUAL(3, 3); + EXPECT_GREATER(4, 3); + EXPECT_GREATER_EQUAL(4, 4); + EXPECT_EXCEPTION(willThrow(), std::runtime_error, "fail"); +} + +TEST("this test will fail") { + EXPECT_EQUAL(3, 4); +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +checks_test.cpp: info: running test suite 'checks_test.cpp' +checks_test.cpp: info: status_for_test 'require that checks work': PASS +checks_test.cpp: ERROR: check failure #1: '3 == 4' in thread '0(1)' (checks_test.cpp:24) + STATE[0]: 'this test will fail' (checks_test.cpp:23) +lhs: 3 +rhs: 4 +checks_test.cpp: ERROR: status_for_test 'this test will fail': FAIL +checks_test.cpp: info: test summary --- 1 test(s) passed --- 1 test(s) failed +checks_test.cpp: info: imported 11 passed check(s) from 1 thread(s) +checks_test.cpp: info: summary --- 11 check(s) passed --- 1 check(s) failed +checks_test.cpp: info: CONCLUSION: FAIL +make: *** [test] Error 1 +</pre> +</div> + +<p> +Checks involving comparison of values typically use == and < +operators to compare values. Also; in order to be part of a comparison +check, the value must support the << operator to print the value +to a string stream in case the check fails. +</p> +</section> + + +<section> +<div class="page-header"> +<h1>Test Fixtures</h1> +</div> +<p> +Sometimes multiple tests wish to use the same predefined state as a +starting point. This state is called a test fixture. Test fixtures are +untangled from the actual tests and construction/destruction is used +to handle their lifetime. When thinking of what can be used as a +fixture, think of what can be put after <strong>new</strong> to create +an object. +</p> + +<ul> +<li>A single test can have multiple test fixtures.</li> +<li>The number of <strong>F</strong>s in the test macro denotes the number of fixtures.</li> +<li>Inside the test, fixtures are available as <strong>f1</strong>, <strong>f2</strong> and so forth.</li> +<li>Test fixtures can be parameterized with constructor parameters.</li> +<li>Checks can be performed inside test fixture constructors.</li> +<li>A test fixture constructor can take other test fixtures as input.</li> +</ul> + +<div class="example" id="fixtures"> +<h2>fixtures_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> + +struct Fixture { + int value; + Fixture() : value(5) {} +}; + +TEST_F("basic fixture", Fixture) { + EXPECT_EQUAL(5, f1.value); +} + +TEST_FFF("fancy fixtures", size_t(10), int(5), std::vector<int>(f1, f2)) { + EXPECT_EQUAL(10u, f1); + EXPECT_EQUAL(5, f2); + ASSERT_EQUAL(10u, f3.size()); + EXPECT_EQUAL(5, f3[7]); +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +fixtures_test.cpp: info: running test suite 'fixtures_test.cpp' +fixtures_test.cpp: info: status_for_test 'basic fixture': PASS +fixtures_test.cpp: info: status_for_test 'fancy fixtures': PASS +fixtures_test.cpp: info: test summary --- 2 test(s) passed --- 0 test(s) failed +fixtures_test.cpp: info: imported 5 passed check(s) from 1 thread(s) +fixtures_test.cpp: info: summary --- 5 check(s) passed --- 0 check(s) failed +fixtures_test.cpp: info: CONCLUSION: PASS +</pre> +</div> +</section> + + +<section> +<div class="page-header"> +<h1>Multi-Threaded Tests</h1> +</div> +<p> +One of the most novel features of the test framework is the ability to +write multi-threaded tests. Multi-threaded tests are created by using +test macros containing <strong>_MT</strong> and supplying a thread +count. All threads will execute the block of code encapsulated by the +test. The test fixtures are shared among all threads. In addition, +each thread has a variable named <strong>num_threads</strong> +containing the total number of threads running the test and a variable +named <strong>thread_id</strong> identifying the thread. +</p> + +<p> +The <strong>TEST_BARRIER()</strong> macro can be used inside the test +to synchronize the threads. The macro will block execution of each +thread invoking it until all threads have invoked it. +</p> + +<div class="example" id="threads"> +<h2>threads_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> + +TEST_MT_F("multiple threads", 2, std::vector<size_t>(num_threads)) { + ASSERT_EQUAL(num_threads, f1.size()); + f1[thread_id] = thread_id; + TEST_BARRIER(); + if (thread_id == 0) { + TEST_TRACE(); + EXPECT_EQUAL(1u, f1[1]); + } else { + TEST_TRACE(); + EXPECT_EQUAL(0u, f1[0]); + } +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +threads_test.cpp: info: running test suite 'threads_test.cpp' +threads_test.cpp: info: trace: thread '0(2)' (threads_test.cpp:9) +threads_test.cpp: info: trace: thread '1(2)' (threads_test.cpp:12) +threads_test.cpp: info: status_for_test 'multiple threads': PASS +threads_test.cpp: info: test summary --- 1 test(s) passed --- 0 test(s) failed +threads_test.cpp: info: imported 4 passed check(s) from 2 thread(s) +threads_test.cpp: info: summary --- 4 check(s) passed --- 0 check(s) failed +threads_test.cpp: info: CONCLUSION: PASS +</pre> +</div> +</section> + + +<section> +<div class="page-header"> +<h1>Real World Examples</h1> +</div> +<div class="example" id="../box"> +<h2>box_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> +#include <vespa/vespalib/util/box.h> + +using namespace vespalib; + +void checkValues(const std::vector<int> &values, size_t n) { + ASSERT_EQUAL(n, values.size()); + for (size_t i = 0; i < n; ++i) { + EXPECT_EQUAL(int(10 + (10 * i)), values[i]); + } +} + +TEST("require that boxes can be created and converted to vector") { + Box<int> box; + box.add(10).add(20).add(30); + checkValues(box, 3); +} + +TEST("require that boxes can be created in place") { + checkValues(Box<int>().add(10).add(20).add(30), 3); +} + +TEST("require that make_box works") { + checkValues(make_box(10), 1); + checkValues(make_box(10, 20), 2); + checkValues(make_box(10, 20, 30), 3); + checkValues(make_box(10, 20, 30, 40), 4); + checkValues(make_box(10, 20, 30, 40, 50), 5); +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +box_test.cpp: info: running test suite 'box_test.cpp' +box_test.cpp: info: status_for_test 'require that boxes can be created and converted to vector': PASS +box_test.cpp: info: status_for_test 'require that boxes can be created in place': PASS +box_test.cpp: info: status_for_test 'require that make_box works': PASS +box_test.cpp: info: test summary --- 3 test(s) passed --- 0 test(s) failed +box_test.cpp: info: imported 28 passed check(s) from 1 thread(s) +box_test.cpp: info: summary --- 28 check(s) passed --- 0 check(s) failed +box_test.cpp: info: CONCLUSION: PASS +</pre> +</div> +<div class="example" id="../barrier"> +<h2>barrier_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> +#include <vespa/vespalib/util/barrier.h> + +using namespace vespalib; + +struct Fixture { + Barrier barrier; + CountDownLatch latch; + Fixture(size_t n) : barrier(n), latch(n) {} +}; + +TEST_MT_F("require that barriers are satisfied by the appropriate number of threads", 3, Fixture(num_threads)) { + if (thread_id == 0) { + f1.latch.countDown(); + EXPECT_FALSE(f.latch.await(250)); + EXPECT_TRUE(f.barrier.await()); + EXPECT_TRUE(f.latch.await(25000)); + } else { + EXPECT_TRUE(f1.barrier.await()); + f1.latch.countDown(); + } +} + +TEST_MT_F("require that barriers can be used multiple times", 3, Fixture(num_threads)) { + EXPECT_TRUE(f1.barrier.await()); + EXPECT_TRUE(f1.barrier.await()); + if (thread_id == 0) { + f1.latch.countDown(); + EXPECT_FALSE(f.latch.await(250)); + EXPECT_TRUE(f.barrier.await()); + EXPECT_TRUE(f.latch.await(25000)); + } else { + EXPECT_TRUE(f1.barrier.await()); + f1.latch.countDown(); + } +} + +TEST_MT_F("require that barriers can be broken", 3, Fixture(num_threads)) { + EXPECT_TRUE(f1.barrier.await()); + if (thread_id == 0) { + f1.latch.countDown(); + EXPECT_FALSE(f.latch.await(250)); + f1.barrier.destroy(); + EXPECT_TRUE(f.latch.await(25000)); + } else { + EXPECT_FALSE(f1.barrier.await()); + f1.latch.countDown(); + } + EXPECT_FALSE(f1.barrier.await()); +} + +TEST_MT_F("require that barriers cannot be retroactively broken", 100, Barrier(num_threads)) { + EXPECT_TRUE(f1.await()); + f1.destroy(); + EXPECT_FALSE(f1.await()); +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +barrier_test.cpp: info: running test suite 'barrier_test.cpp' +barrier_test.cpp: info: status_for_test 'require that barriers are satisfied by the appropriate number of threads': PASS +barrier_test.cpp: info: status_for_test 'require that barriers can be used multiple times': PASS +barrier_test.cpp: info: status_for_test 'require that barriers can be broken': PASS +barrier_test.cpp: info: status_for_test 'require that barriers cannot be retroactively broken': PASS +barrier_test.cpp: info: test summary --- 4 test(s) passed --- 0 test(s) failed +barrier_test.cpp: info: imported 226 passed check(s) from 100 thread(s) +barrier_test.cpp: info: summary --- 226 check(s) passed --- 0 check(s) failed +barrier_test.cpp: info: CONCLUSION: PASS +</pre> +</div> +<div class="example" id="../dual_merge_director"> +<h2>dual_merge_director_test.cpp</h2> +<pre class="prettyprint linenums"> +// Copyright 2016 Yahoo Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. +#include <vespa/vespalib/testkit/test_kit.h> +#include <vespa/vespalib/util/dual_merge_director.h> + +using namespace vespalib; + +struct MySource : public DualMergeDirector::Source { + + bool typeA; + std::string data; + std::string diff; + + MySource(bool a, size_t num_sources, size_t source_id) + : typeA(a), + data(num_sources, '0'), + diff(num_sources, '5') + { + if (source_id < num_sources) { + data[source_id] = '1'; + diff[source_id] = '6'; + } + } + virtual void merge(Source &mt) { + MySource &rhs = static_cast<MySource&>(mt); + ASSERT_EQUAL(typeA, rhs.typeA); + ASSERT_EQUAL(data.size(), rhs.data.size()); + for (size_t i = 0; i < data.size(); ++i) { + int d = (rhs.data[i] - '0'); + data[i] += d; + diff[i] += d; + rhs.diff[i] -= d; + } + } + void verifyFinal() const { + EXPECT_EQUAL(std::string(data.size(), '1'), data); + EXPECT_EQUAL(std::string(diff.size(), '6'), diff); + } + void verifyIntermediate() const { + EXPECT_EQUAL(std::string(diff.size(), '5'), diff); + } +}; + +TEST_MT_F("require that merging works", 64, std::unique_ptr<DualMergeDirector>()) { + for (size_t use_threads = 1; use_threads <= num_threads; ++use_threads) { + MySource sourceA(true, use_threads, thread_id); + MySource sourceB(false, use_threads, thread_id); + if (thread_id == 0) { + f1.reset(new DualMergeDirector(use_threads)); + } + TEST_BARRIER(); + if (thread_id < use_threads) { + f1->dualMerge(thread_id, sourceA, sourceB); + } + TEST_BARRIER(); + if (thread_id == 0) { + sourceA.verifyFinal(); + sourceB.verifyFinal(); + } else if (thread_id < use_threads) { + sourceA.verifyIntermediate(); + sourceB.verifyIntermediate(); + } + } +} + +TEST_MAIN() { TEST_RUN_ALL(); } +</pre> +<pre class="output"> +dual_merge_director_test.cpp: info: running test suite 'dual_merge_director_test.cpp' +dual_merge_director_test.cpp: info: status_for_test 'require that merging works': PASS +dual_merge_director_test.cpp: info: test summary --- 1 test(s) passed --- 0 test(s) failed +dual_merge_director_test.cpp: info: imported 12352 passed check(s) from 64 thread(s) +dual_merge_director_test.cpp: info: summary --- 12352 check(s) passed --- 0 check(s) failed +dual_merge_director_test.cpp: info: CONCLUSION: PASS +</pre> +</div> +</section> + + +<section> +<div class="page-header"> +<h1>Macro Summary</h1> +</div> + +<h2>Overall Execution Macros</h2> +<ul> +<li><code>TEST_MAIN()</code></li> +<li><code>TEST_RUN_ALL()</code></li> +</ul> + +<h2>Test Creation Macros</h2> +<ul> +<li><code>TEST(name)</code></li> +<li><code>TEST_F(name, fixture)</code></li> +<li><code>TEST_FF(name, fixture1, fixture2)</code></li> +<li><code>TEST_FFF(name, fixture1, fixture2, fixture3)</code></li> +</ul> + +<ul> +<li><code>TEST_MT(name, threads)</code></li> +<li><code>TEST_MT_F(name, threads, fixture)</code></li> +<li><code>TEST_MT_FF(name, threads, fixture1, fixture2)</code></li> +<li><code>TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)</code></li> +</ul> + +<ul> +<li><code>IGNORE_TEST(name)</code></li> +<li><code>IGNORE_TEST_F(name, fixture)</code></li> +<li><code>IGNORE_TEST_FF(name, fixture1, fixture2)</code></li> +<li><code>IGNORE_TEST_FFF(name, fixture1, fixture2, fixture3)</code></li> +</ul> + +<ul> +<li><code>IGNORE_TEST_MT(name, threads)</code></li> +<li><code>IGNORE_TEST_MT_F(name, threads, fixture)</code></li> +<li><code>IGNORE_TEST_MT_FF(name, threads, fixture1, fixture2)</code></li> +<li><code>IGNORE_TEST_MT_FFF(name, threads, fixture1, fixture2, fixture3)</code></li> +</ul> + +<h2>Check Macros</h2> +<ul> +<li><code>ASSERT_TRUE(rc)</code></li> +<li><code>ASSERT_FALSE(rc)</code></li> +<li><code>ASSERT_EQUAL(a, b)</code></li> +<li><code>ASSERT_NOT_EQUAL(a, b)</code></li> +<li><code>ASSERT_APPROX(a, b, eps)</code></li> +<li><code>ASSERT_NOT_APPROX(a, b, eps)</code></li> +<li><code>ASSERT_LESS(a, b)</code></li> +<li><code>ASSERT_LESS_EQUAL(a, b)</code></li> +<li><code>ASSERT_GREATER(a, b)</code></li> +<li><code>ASSERT_GREATER_EQUAL(a, b)</code></li> +<li><code>ASSERT_EXCEPTION(statement, exception_type, msg_substr)</code></li> +<li><code>TEST_FATAL(msg)</code></li> +</ul> + +<ul> +<li><code>EXPECT_TRUE(rc)</code></li> +<li><code>EXPECT_FALSE(rc)</code></li> +<li><code>EXPECT_EQUAL(a, b)</code></li> +<li><code>EXPECT_NOT_EQUAL(a, b)</code></li> +<li><code>EXPECT_APPROX(a, b, eps)</code></li> +<li><code>EXPECT_NOT_APPROX(a, b, eps)</code></li> +<li><code>EXPECT_LESS(a, b)</code></li> +<li><code>EXPECT_LESS_EQUAL(a, b)</code></li> +<li><code>EXPECT_GREATER(a, b)</code></li> +<li><code>EXPECT_GREATER_EQUAL(a, b)</code></li> +<li><code>EXPECT_EXCEPTION(statement, exception_type, msg_substr)</code></li> +<li><code>TEST_ERROR(msg)</code></li> +</ul> + +<h2>Thread Macros</h2> +<ul> +<li><code>TEST_BARRIER()</code></li> +</ul> + +<h2>State Tracking Macros</h2> +<ul> +<li><code>TEST_DO(doit)</code></li> +<li><code>TEST_STATE(msg)</code></li> +</ul> + +<h2>Macros of Limited Use</h2> +<ul> +<li><code>TEST_TRACE()</code></li> +<li><code>TEST_FLUSH()</code></li> +<li><code>TEST_THREAD(name)</code></li> +<li><code>TEST_DEBUG(lhsFile, rhsFile)</code></li> +<li><code>TEST_MAIN_WITH_PROCESS_PROXY()</code></li> +</ul> +</section> + +<script type="text/javascript"> +$(function(){ + window.prettyPrint && prettyPrint(); +}); + +</script> + +</body> +</html> |