diff options
author | Tor Brede Vekterli <vekterli@verizonmedia.com> | 2019-05-24 10:51:03 +0000 |
---|---|---|
committer | Tor Brede Vekterli <vekterli@verizonmedia.com> | 2019-05-24 10:55:58 +0000 |
commit | 31f6c321cb5b130486fe267eaf8ced6878a27319 (patch) | |
tree | 2135688631cb97a143fe1a4dc1bdc34cbb06560b /storage/src/tests/distributor/putoperationtest.cpp | |
parent | 871edfff3e3ce6601f7577c40faa399400de453b (diff) |
Bounce Puts when a node is unavailable in the pending cluster state
Avoids scheduling Put operations towards nodes that are available in the
current cluster state, but not in the one that is being async processed
in the background. Not only are operations to such nodes highly likely to
be doomed in the first place, we also risk triggering assertion failures
by code that does not expect bucket DB inserts to said nodes to ever
be rejected.
Since we're recently added explicit rejection of inserts to nodes that
are unavailable in the pending state, these assertions have the potential
for triggering in certain edge case scenarios.
Diffstat (limited to 'storage/src/tests/distributor/putoperationtest.cpp')
-rw-r--r-- | storage/src/tests/distributor/putoperationtest.cpp | 20 |
1 files changed, 20 insertions, 0 deletions
diff --git a/storage/src/tests/distributor/putoperationtest.cpp b/storage/src/tests/distributor/putoperationtest.cpp index 66ef13310a4..881ccb560b4 100644 --- a/storage/src/tests/distributor/putoperationtest.cpp +++ b/storage/src/tests/distributor/putoperationtest.cpp @@ -569,6 +569,26 @@ TEST_F(PutOperationTest, replica_not_resurrected_in_db_when_node_down_in_pending dumpBucket(bucket)); } +// TODO probably also do this for updates and removes +// TODO consider if we should use the pending state verbatim for computing targets if it exists +TEST_F(PutOperationTest, put_is_failed_with_busy_if_target_down_in_pending_state) { + setupDistributor(Redundancy(3), NodeCount(4), "version:1 distributor:1 storage:3"); + auto doc = createDummyDocument("test", "test"); + auto bucket = getExternalOperationHandler().getBucketId(doc->getId()); + addNodesToBucketDB(bucket, "0=1/2/3/t,1=1/2/3/t,2=1/2/3/t"); + getBucketDBUpdater().onSetSystemState( + std::make_shared<api::SetSystemStateCommand>( + lib::ClusterState("version:2 distributor:1 storage:4 .0.s:d .2.s:m"))); + _sender.clear(); + + sendPut(createPut(doc)); + EXPECT_EQ("", _sender.getCommands(true)); + EXPECT_EQ("PutReply(doc:test:test, BucketId(0x0000000000000000), " + "timestamp 100) ReturnCode(BUSY, " + "One or more target content nodes are unavailable in the pending cluster state)", + _sender.getLastReply(true)); +} + TEST_F(PutOperationTest, send_to_retired_nodes_if_no_up_nodes_available) { setupDistributor(Redundancy(2), NodeCount(2), "distributor:1 storage:2 .0.s:r .1.s:r"); |