Continue serving search queries when in Maintenance node state

Previously, entering maintenance state would implicitly deactivate
all buckets on the searchnode and cause empty responses to be returned
for searches.

However, container query dispatch uses async health pings to decide
which nodes to route queries to, so it would be possible for a node to
still be used for queries for a few seconds until the ping discovered
that the node should not be used. In the case of multiple groups without
multiple ready replicas within the group, this would cause transient
coverage loss since the dispatcher would not realize it should route
queries to other groups instead.

With this commit, maintenance edge behavior is changed as follows:

 - Buckets are _not_ deactivated when going from an available state
   to the maintenance state. However, they _are_ deactivate when going
   from maintenance state to an available state in order to avoid transient
   query duplicates immediately after the change.
 - Searches are executed as normal instead of returning empty replies
   when the node is in maintenance state.

The following behavior is intentionally _not_ changed:

 - The search interface is still marked as offline when in maintenance
   state, as this signals that the node should be taken out of rotation.
   In particular, it's critical that the RPC health ping response is
   explicitly tagged as having zero active docs when the search interface
   is offline, even though many buckets may now actually be active.
   Otherwise, queries would not be gracefully drained from the node.

1 files changed, 27 insertions, 0 deletions

diff --git a/persistence/src/tests/spi/clusterstatetest.cpp b/persistence/src/tests/spi/clusterstatetest.cpp
index 8a303c1a1ac..9f7507c992b 100644
--- a/persistence/src/tests/spi/clusterstatetest.cpp
+++ b/persistence/src/tests/spi/clusterstatetest.cpp
@@ -233,4 +233,31 @@ TEST(ClusterStateTest, can_infer_own_node_retired_state)
     EXPECT_TRUE(!node_marked_as_retired_in_state("distributor:3 storage:3 .1.s:r", d, 0));
 }
 
+namespace {
+
+bool
+node_marked_as_maintenance_in_state(const std::string& stateStr,
+                                    const lib::Distribution& d,
+                                    uint16_t node)
+{
+    lib::ClusterState s(stateStr);
+    ClusterState state(s, node, d);
+    return state.nodeMaintenance();
+}
+
+}
+
+TEST(ClusterStateTest, can_infer_own_node_maintenance_state)
+{
+    lib::Distribution d(lib::Distribution::getDefaultDistributionConfig(3, 3));
+
+    EXPECT_FALSE(node_marked_as_maintenance_in_state("distributor:3 storage:3", d, 0));
+    EXPECT_FALSE(node_marked_as_maintenance_in_state("distributor:3 storage:3 .0.s:i", d, 0));
+    EXPECT_FALSE(node_marked_as_maintenance_in_state("distributor:3 storage:3 .0.s:d", d, 0));
+    EXPECT_TRUE( node_marked_as_maintenance_in_state("distributor:3 storage:3 .0.s:m", d, 0));
+    EXPECT_FALSE(node_marked_as_maintenance_in_state("distributor:3 storage:3 .0.s:r", d, 0));
+    EXPECT_FALSE(node_marked_as_maintenance_in_state("distributor:3 storage:3 .0.s:m", d, 1));
+    EXPECT_FALSE(node_marked_as_maintenance_in_state("distributor:3 storage:3 .1.s:m", d, 0));
+}
+
 }