Add simple probabilistic query tracing and logging for streaming searches that time out

This currently only catches the cases where a query has not respected its
expected timeout and therefore has an actual timeout that is at least a
5x multiple of its expected one. Due to the response payload nature of
MessageBus traces it's not given that the trace will contain any
information from the backend nodes. But it is highly likely to contain
useful information from the client-local policy instances, as well as
resending info.

To test this very conservatively, only allows approximately 1 query to be
traced every 2 seconds and only dumps traces to logs once every 10 seconds.

1 files changed, 15 insertions, 0 deletions

diff --git a/container-search/src/main/java/com/yahoo/vespa/streamingvisitors/tracing/MonotonicNanoClock.java b/container-search/src/main/java/com/yahoo/vespa/streamingvisitors/tracing/MonotonicNanoClock.java
new file mode 100644
index 00000000000..1b3d766e266
--- /dev/null
+++ b/container-search/src/main/java/com/yahoo/vespa/streamingvisitors/tracing/MonotonicNanoClock.java
@@ -0,0 +1,15 @@
+// Copyright 2019 Oath Inc. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
+package com.yahoo.vespa.streamingvisitors.tracing;
+
+/*
+ * Clock which returns a monotonically increasing timestamp from an undefined epoch.
+ * The epoch is guaranteed to be stable within a single JVM execution, but not across
+ * processes. Should therefore only be used for relative duration tracking, not absolute
+ * wall clock time events.
+ */
+@FunctionalInterface
+public interface MonotonicNanoClock {
+
+    long nanoTimeNow();
+
+}