Add new Protobuf-based MessageBus DocumentAPI protocol

This adds an entirely new implementation of the internal MessageBus
DocumentAPI protocol, which shall be functionally 1-to-1 compatible
with the existing legacy protocol.

New protobuf schemas have been added to the top-level documentapi
module, which are separated into different domains of responsibility:

 * CRUD messages
 * Visiting messages
 * Data inspection messages

As well as a schema for shared, common message types.

Both C++ and Java protocol implementations separate serialization and
deserialization into a codec abstraction per message type, which
hides the boilerplate required for Protobuf buffer management. The
Java version is a tad more verbose due to generics type-erasure.

This protocol does _not_ currently support lazy (de-)serialization
in Java, as the existing mechanisms for doing so are inherently tied
to the legacy protocol version. Performance tests will decide if we
need to introduce such functionality to the new protocol version.

To avoid having the new protocol go live in production, this commit
changes the semantics of how MessageBus version reporting works (at
least for the near future); instead of reporting the current Vespa
_release_ version, it reports the highest supported _protocol_ version.
This lets us conditionally enable the new protocol by reporting a
MessageBus version greater than or equal to the protocol version _iff_
the protocol should be active.

The new protocol is disabled by default.

Other changes:
 * Protocol tests have been moved up one package directory level to
   be aligned with the actual package of the classes they test. This
   allows for using package-protected constructors in the serialization
   tests.
 * `DocumentDeserializer` now exposes the underlying document type
   repo/manager. This is done to detangle `Document`/`DocumentUpdate`
   deserialization from the underlying wire buffer management.
 * `RemoveLocationMessage` at long last contains a bucket space,
   which was forgotten when we initially added this concept to the
   other messages, and where the pain of adding it in later was
   too big (not so anymore!).

Unit tests for both C++ and Java have been hoisted from the legacy
test suite, cleaned up and extended with additional cases. The C++
tests use the old unit test kit and should receive a good follow-up
washing and GTest-rewrite.

**Important**: due to how MessageBus protocol versioning works, the
final protocol version is _not_ yet decided, as setting it requires
syncing against our build systems. A follow-up commit will assign
the final version as well as include all required binary test files.

5 files changed, 14 insertions, 2 deletions

diff --git a/document/abi-spec.json b/document/abi-spec.json
index 899c107a242..ca06e2547d7 100644
--- a/document/abi-spec.json
+++ b/document/abi-spec.json
@@ -2724,7 +2724,8 @@
       "abstract"
     ],
     "methods" : [
-      "public abstract com.yahoo.io.GrowableByteBuffer getBuf()"
+      "public abstract com.yahoo.io.GrowableByteBuffer getBuf()",
+      "public abstract com.yahoo.document.DocumentTypeManager getTypeRepo()"
     ],
     "fields" : [ ]
   },
@@ -3015,6 +3016,7 @@
     ],
     "methods" : [
       "public final com.yahoo.document.DocumentTypeManager getDocumentTypeManager()",
+      "public com.yahoo.document.DocumentTypeManager getTypeRepo()",
       "public void read(com.yahoo.document.Document)",
       "public void read(com.yahoo.vespa.objects.FieldBase, com.yahoo.document.Document)",
       "public void read(com.yahoo.vespa.objects.FieldBase, com.yahoo.document.datatypes.FieldValue)",
diff --git a/document/src/main/java/com/yahoo/document/Document.java b/document/src/main/java/com/yahoo/document/Document.java
index 4bb93426994..294750f40f3 100644
--- a/document/src/main/java/com/yahoo/document/Document.java
+++ b/document/src/main/java/com/yahoo/document/Document.java
@@ -123,6 +123,7 @@ public class Document extends StructuredFieldValue {
     }
 
     public int getSerializedSize() throws SerializationException {
+        // TODO shouldn't this be createHead()?
         DocumentSerializer data = DocumentSerializerFactory.create6(new GrowableByteBuffer(64 * 1024, 2.0f));
         data.write(this);
         return data.getBuf().position();
@@ -135,6 +136,7 @@ public class Document extends StructuredFieldValue {
     public final int getApproxSize() { return 4096; }
 
     public void serialize(OutputStream out) throws SerializationException {
+        // TODO shouldn't this be createHead()?
         DocumentSerializer writer = DocumentSerializerFactory.create6(new GrowableByteBuffer(64 * 1024, 2.0f));
         writer.write(this);
         GrowableByteBuffer data = writer.getBuf();
diff --git a/document/src/main/java/com/yahoo/document/serialization/DocumentDeserializer.java b/document/src/main/java/com/yahoo/document/serialization/DocumentDeserializer.java
index 59849d88c28..f6e3a75c7b2 100644
--- a/document/src/main/java/com/yahoo/document/serialization/DocumentDeserializer.java
+++ b/document/src/main/java/com/yahoo/document/serialization/DocumentDeserializer.java
@@ -1,6 +1,7 @@
 // Copyright Vespa.ai. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
 package com.yahoo.document.serialization;
 
+import com.yahoo.document.DocumentTypeManager;
 import com.yahoo.io.GrowableByteBuffer;
 
 /**
@@ -17,5 +18,7 @@ public interface DocumentDeserializer extends DocumentReader, DocumentUpdateRead
      */
     GrowableByteBuffer getBuf();
 
+    DocumentTypeManager getTypeRepo();
+
 }
 
diff --git a/document/src/main/java/com/yahoo/document/serialization/DocumentSerializer.java b/document/src/main/java/com/yahoo/document/serialization/DocumentSerializer.java
index 9d52bc4aead..faaad95d5e1 100644
--- a/document/src/main/java/com/yahoo/document/serialization/DocumentSerializer.java
+++ b/document/src/main/java/com/yahoo/document/serialization/DocumentSerializer.java
@@ -15,6 +15,6 @@ public interface DocumentSerializer extends DocumentWriter, SpanNodeWriter, Anno
     /**
      * Returns the underlying buffer used for serialization.
      */
-    public GrowableByteBuffer getBuf();
+    GrowableByteBuffer getBuf();
 }
 
diff --git a/document/src/main/java/com/yahoo/document/serialization/VespaDocumentDeserializer6.java b/document/src/main/java/com/yahoo/document/serialization/VespaDocumentDeserializer6.java
index 5f24a2d8f60..b508f0d2c7c 100644
--- a/document/src/main/java/com/yahoo/document/serialization/VespaDocumentDeserializer6.java
+++ b/document/src/main/java/com/yahoo/document/serialization/VespaDocumentDeserializer6.java
@@ -91,6 +91,11 @@ public class VespaDocumentDeserializer6 extends BufferSerializer implements Docu
 
     final public DocumentTypeManager getDocumentTypeManager() { return manager; }
 
+    @Override
+    public DocumentTypeManager getTypeRepo() {
+        return manager;
+    }
+
     public void read(Document document) {
         read(null, document);
     }