Add new Protobuf-based MessageBus DocumentAPI protocol

This adds an entirely new implementation of the internal MessageBus
DocumentAPI protocol, which shall be functionally 1-to-1 compatible
with the existing legacy protocol.

New protobuf schemas have been added to the top-level documentapi
module, which are separated into different domains of responsibility:

 * CRUD messages
 * Visiting messages
 * Data inspection messages

As well as a schema for shared, common message types.

Both C++ and Java protocol implementations separate serialization and
deserialization into a codec abstraction per message type, which
hides the boilerplate required for Protobuf buffer management. The
Java version is a tad more verbose due to generics type-erasure.

This protocol does _not_ currently support lazy (de-)serialization
in Java, as the existing mechanisms for doing so are inherently tied
to the legacy protocol version. Performance tests will decide if we
need to introduce such functionality to the new protocol version.

To avoid having the new protocol go live in production, this commit
changes the semantics of how MessageBus version reporting works (at
least for the near future); instead of reporting the current Vespa
_release_ version, it reports the highest supported _protocol_ version.
This lets us conditionally enable the new protocol by reporting a
MessageBus version greater than or equal to the protocol version _iff_
the protocol should be active.

The new protocol is disabled by default.

Other changes:
 * Protocol tests have been moved up one package directory level to
   be aligned with the actual package of the classes they test. This
   allows for using package-protected constructors in the serialization
   tests.
 * `DocumentDeserializer` now exposes the underlying document type
   repo/manager. This is done to detangle `Document`/`DocumentUpdate`
   deserialization from the underlying wire buffer management.
 * `RemoveLocationMessage` at long last contains a bucket space,
   which was forgotten when we initially added this concept to the
   other messages, and where the pain of adding it in later was
   too big (not so anymore!).

Unit tests for both C++ and Java have been hoisted from the legacy
test suite, cleaned up and extended with additional cases. The C++
tests use the old unit test kit and should receive a good follow-up
washing and GTest-rewrite.

**Important**: due to how MessageBus protocol versioning works, the
final protocol version is _not_ yet decided, as setting it requires
syncing against our build systems. A follow-up commit will assign
the final version as well as include all required binary test files.

3 files changed, 48 insertions, 7 deletions

diff --git a/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/RPCNetwork.java b/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/RPCNetwork.java
index 0373609e806..3ab62542ace 100644
--- a/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/RPCNetwork.java
+++ b/messagebus/src/main/java/com/yahoo/messagebus/network/rpc/RPCNetwork.java
@@ -2,7 +2,6 @@
 package com.yahoo.messagebus.network.rpc;
 
 import com.yahoo.component.Version;
-import com.yahoo.component.Vtag;
 import com.yahoo.concurrent.ThreadFactoryFactory;
 import com.yahoo.jrt.Acceptor;
 import com.yahoo.jrt.ListenFailedException;
@@ -301,15 +300,34 @@ public class RPCNetwork implements Network, MethodHandler {
         return false;
     }
 
+    private static Version deriveSupportedProtocolVersion() {
+        // This is a very leaky abstraction, but since MessageBus only exchanges versions
+        // (and not a set of supported protocols), we have to do this workaround.
+        // Disallow-version MUST be lower than that used as a protocol lower bound in
+        // DocumentProtocol.java and the exact same as that used in C++ for the same purposes.
+        // ... Or else!
+        // TODO remove this glorious hack once protobuf protocol is enabled by default
+        var maybeEnvVal = System.getenv("VESPA_MBUS_DOCUMENTAPI_USE_PROTOBUF");
+        if ("true".equals(maybeEnvVal) || "yes".equals(maybeEnvVal)) {
+            return new Version(8, 304); // _Allows_ new protobuf protocol
+        }
+        return new Version(8, 303); // _Disallows_ new protobuf protocol
+    }
+
+    private static final Version REPORTED_VERSION = deriveSupportedProtocolVersion();
+
     /**
-     * Returns the version of this network. This gets called when the "mbus.getVersion" method is invoked on this
-     * network, and is separated into its own function so that unit tests can override it to simulate other versions
-     * than current.
+     * Returns the (protocol) version of this network. This gets called when the "mbus.getVersion" method is invoked
+     * on this network, and is separated into its own function so that unit tests can override it to simulate other
+     * versions than current.
+     *
+     * Note that this version reflects the highest supported <em>protocol</em> version, and is not necessarily
+     * 1-1 with the actual Vespa release version of the underlying binary.
      *
      * @return the version to claim to be
      */
     protected Version getVersion() {
-        return Vtag.currentVersion;
+        return REPORTED_VERSION;
     }
 
     /**
diff --git a/messagebus/src/vespa/messagebus/network/rpcnetwork.cpp b/messagebus/src/vespa/messagebus/network/rpcnetwork.cpp
index cacd18430a7..f626e2c325b 100644
--- a/messagebus/src/vespa/messagebus/network/rpcnetwork.cpp
+++ b/messagebus/src/vespa/messagebus/network/rpcnetwork.cpp
@@ -18,6 +18,8 @@
 #include <vespa/vespalib/stllike/asciistream.h>
 #include <vespa/vespalib/util/size_literals.h>
 #include <vespa/vespalib/util/stringfmt.h>
+#include <cstdlib>
+#include <string_view>
 #include <thread>
 
 #include <vespa/log/log.h>
@@ -25,6 +27,7 @@ LOG_SETUP(".rpcnetwork");
 
 using vespalib::make_string;
 using namespace std::chrono_literals;
+using namespace std::string_view_literals;
 
 namespace mbus {
 
@@ -148,10 +151,26 @@ RPCNetwork::flushTargetPool()
     _targetPool->flushTargets(true);
 }
 
+namespace {
+
+[[nodiscard]] vespalib::Version derive_supported_protocol_version() {
+    // TODO remove this hilariously leaky abstraction once protobuf protocol is the default :D
+    // Disallow-version MUST be lower than that used as a protocol lower bound in documentprotocol.cpp
+    // and the exact same as that used in Java for the same purposes. Or else!
+    const char* maybe_env_val = getenv("VESPA_MBUS_DOCUMENTAPI_USE_PROTOBUF");
+    if (maybe_env_val && (("true"sv == maybe_env_val) || ("yes"sv == maybe_env_val))) {
+        return {8, 304}; // _Allows_ new protobuf protocol
+    }
+    return {8, 303}; // _Disallows_ new protobuf protocol
+}
+
+}
+
 const vespalib::Version &
 RPCNetwork::getVersion() const
 {
-    return vespalib::Vtag::currentVersion;
+    static vespalib::Version reported_version = derive_supported_protocol_version();
+    return reported_version;
 }
 
 void
diff --git a/messagebus/src/vespa/messagebus/network/rpcnetwork.h b/messagebus/src/vespa/messagebus/network/rpcnetwork.h
index 05ccaecb2c5..40590d4545f 100644
--- a/messagebus/src/vespa/messagebus/network/rpcnetwork.h
+++ b/messagebus/src/vespa/messagebus/network/rpcnetwork.h
@@ -98,11 +98,15 @@ private:
 
 protected:
     /**
-     * Returns the version of this network. This gets called when the
+     * Returns the (protocol) version of this network. This gets called when the
      * "mbus.getVersion" method is invoked on this network, and is separated
      * into its own function so that unit tests can override it to simulate
      * other versions than current.
      *
+     * Note that this version reflects the highest supported protocol version, and
+     * is not necessarily 1-1 with the actual Vespa release version of the
+     * underlying binary.
+     *
      * @return The version to claim to be.
      */
     virtual const vespalib::Version &getVersion() const;