Use fast updates when replica metadata is out of sync but document itself is in sync

When a bucket has replicas with mismatching metadata (i.e. they are out of sync),
the distributor will initiate a write-repair for updates to avoid divergence of
replica content. This is done by first sending a Get to all diverging replica
sets, picking the highest timestamp and applying the update locally. The updated
document is then sent out as a Put. This can be very expensive if document Put
operations are disproportionally more expensive than partial updates, and also
makes the distributor thread part of a contended critical path.

This commit lets `TwoPhaseUpdateOperation` restart an update as a "fast path"
update (partial updates sent directly to the nodes) if the initial read phase
returns the same timestamp for the document across all replicas.

It also removes an old (but now presumed unsafe) optimization where Get
operations are only sent to replicas marked "trusted" even if others are
out of sync with it. Since trustedness is a transient state that does not
persist across restarts or bucket handoffs, it's not robust enough to be
used for such purposes. Gets will now be sent to all out of sync replica
groups regardless of trusted status.

2 files changed, 11 insertions, 3 deletions

diff --git a/storageapi/src/vespa/storageapi/message/persistence.cpp b/storageapi/src/vespa/storageapi/message/persistence.cpp
index 0b641eea253..8bd4f648f30 100644
--- a/storageapi/src/vespa/storageapi/message/persistence.cpp
+++ b/storageapi/src/vespa/storageapi/message/persistence.cpp
@@ -207,13 +207,17 @@ GetCommand::print(std::ostream& out, bool verbose, const std::string& indent) co
     }
 }
 
-GetReply::GetReply(const GetCommand& cmd, const DocumentSP& doc, Timestamp lastModified)
+GetReply::GetReply(const GetCommand& cmd,
+                   const DocumentSP& doc,
+                   Timestamp lastModified,
+                   bool had_consistent_replicas)
     : BucketInfoReply(cmd),
       _docId(cmd.getDocumentId()),
       _fieldSet(cmd.getFieldSet()),
       _doc(doc),
       _beforeTimestamp(cmd.getBeforeTimestamp()),
-      _lastModifiedTime(lastModified)
+      _lastModifiedTime(lastModified),
+      _had_consistent_replicas(had_consistent_replicas)
 {
 }
 
diff --git a/storageapi/src/vespa/storageapi/message/persistence.h b/storageapi/src/vespa/storageapi/message/persistence.h
index 1fdd5c369bf..f2161421feb 100644
--- a/storageapi/src/vespa/storageapi/message/persistence.h
+++ b/storageapi/src/vespa/storageapi/message/persistence.h
@@ -217,11 +217,13 @@ class GetReply : public BucketInfoReply {
     DocumentSP _doc; // Null pointer if not found
     Timestamp _beforeTimestamp;
     Timestamp _lastModifiedTime;
+    bool _had_consistent_replicas;
 
 public:
     GetReply(const GetCommand& cmd,
              const DocumentSP& doc = DocumentSP(),
-             Timestamp lastModified = 0);
+             Timestamp lastModified = 0,
+             bool had_consistent_replicas = false);
     ~GetReply() override;
 
     const DocumentSP& getDocument() const { return _doc; }
@@ -231,6 +233,8 @@ public:
     Timestamp getLastModifiedTimestamp() const { return _lastModifiedTime; }
     Timestamp getBeforeTimestamp() const { return _beforeTimestamp; }
 
+    bool had_consistent_replicas() const noexcept { return _had_consistent_replicas; }
+
     bool wasFound() const { return (_doc.get() != 0); }
     void print(std::ostream& out, bool verbose, const std::string& indent) const override;
     DECLARE_STORAGEREPLY(GetReply, onGetReply)