From fb96b6e254ef367428374eb4f254c248666bcd11 Mon Sep 17 00:00:00 2001 From: Tor Brede Vekterli Date: Tue, 1 Dec 2020 09:34:29 +0000 Subject: Initial support for backend reindexing visitor functionality MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces the concept of a read-for-write visitor operation which blocks all mutating operations from starting for a bucket being visited. This read-for-write mode is used if (and only if) the visitor library being specified by the client is "reindexingvisitor". Since read-for-write visitors cannot race with concurrent write operations, starting such visitors are deferred until no further mutations are pending. Also adds a basic reindexingvisitor implementation to the content node which sends all documents as Puts containing a special TaS token that will let the operation through even if a bucket is locked. This token is cleared by the distributor before it is passed on to the content nodes. Note: this feature is not yet production ready. For now the following caveats apply: * Mutating vs non-mutating pending ops to a bucket are not tracked separately, so it’s possible to starve a reindexing visitor by sending constant pending read load, as read load is not blocked by the operation sequencer. * Ideal state operations towards locked buckets are not blocked, so it's possible for e.g. a split to be sent for a bucket that is being visited. --- storage/src/tests/distributor/CMakeLists.txt | 1 + 1 file changed, 1 insertion(+) (limited to 'storage/src/tests/distributor/CMakeLists.txt') diff --git a/storage/src/tests/distributor/CMakeLists.txt b/storage/src/tests/distributor/CMakeLists.txt index 96df1a842a4..810ffb550bf 100644 --- a/storage/src/tests/distributor/CMakeLists.txt +++ b/storage/src/tests/distributor/CMakeLists.txt @@ -32,6 +32,7 @@ vespa_add_executable(storage_distributor_gtest_runner_app TEST pendingmessagetrackertest.cpp persistence_metrics_set_test.cpp putoperationtest.cpp + read_for_write_visitor_operation_test.cpp removebucketoperationtest.cpp removelocationtest.cpp removeoperationtest.cpp -- cgit v1.2.3