Ensure proper memory visibility on distributor stripe flush

Flushing the distributor stripes happens as part of process shutdown
and therefore takes place in the main thread. At this point we should
no longer receive any messages from the RPC/messaging subsystems, but
there may still be lingering replies to be purged. At this point, the
stripe threads are joined but the communication manager thread is not
(happens slightly after in the shutdown sequence).

We therefore have to explicitly form a mutex acquire/release pair to
ensure proper memory visibility between anything touched by the comm.
manager thread (within the mutex) and our main thread.

This should resolve a ThreadSanitizer error.

1 files changed, 6 insertions, 0 deletions

diff --git a/storage/src/vespa/storage/distributor/distributor_stripe.cpp b/storage/src/vespa/storage/distributor/distributor_stripe.cpp
index 782dd9d7e12..616fd77fdd7 100644
--- a/storage/src/vespa/storage/distributor/distributor_stripe.cpp
+++ b/storage/src/vespa/storage/distributor/distributor_stripe.cpp
@@ -127,6 +127,12 @@ void DistributorStripe::send_shutdown_abort_reply(const std::shared_ptr<api::Sto
 }
 
 void DistributorStripe::flush_and_close() {
+    // This function is called from a different thread than that of the stripe
+    // itself, so we need to take the same mutex to form a memory visibility pair.
+    // It is important that no flushing ever sends any _requests_, as these
+    // will most likely synchronously be bounced by the already shut down RPC
+    // layer, causing a deadlock when the response call chain arrives back here.
+    std::lock_guard lock(_external_message_mutex);
     for (auto& msg : _messageQueue) {
         if (!msg->getType().isReply()) {
             send_shutdown_abort_reply(msg);