summaryrefslogtreecommitdiffstats
path: root/storage/src
Commit message (Collapse)AuthorAgeFilesLines
* Don't let ignored bucket info reply be propagated out of distributorTor Brede Vekterli2021-11-303-44/+61
| | | | | | | | | | If a reply arrives for a preempted cluster state it will be ignored. To avoid it being automatically sent further down the storage chain we still have to treat it as handled. Otherwise a scary looking but otherwise benign "unhandled message" warning will be emitted in the Vespa log. Also move an existing test to the correct test fixture.
* Reduce default bucket delete pri and fix priority inheritance for merge deletionTor Brede Vekterli2021-11-304-2/+14
| | | | | | | | Since deletes are now async in the backend, make them FIFO-order with client feed by default to avoid swamping the queues with deletes. Also, explicitly inherit priority for bucket deletion triggered by bucket merging. This was actually missing previously and meant such deletes got the default, very low priority.
* Merge pull request #20211 from vespa-engine/vekterli/reduce-update-log-levelHenning Baldersheim2021-11-251-7/+7
|\ | | | | Reduce log level from error to warning for update inconsistency message
| * Reduce log level from error to warning for update inconsistency messageTor Brede Vekterli2021-11-251-7/+7
| | | | | | | | | | | | `error` level is for the "node is on fire and falling down an elevator shaft" type of messages, not for operation inconsistencies. Reduce to `warning` accordingly.
* | Update operation metrics for delayed or chained merge handler replies.Tor Egge2021-11-244-7/+123
|/
* Actually test maintenance -> down node state transitionTor Brede Vekterli2021-11-241-1/+1
|
* Ensure member variable is initializedTor Brede Vekterli2021-11-241-1/+2
|
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-246-39/+156
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* drop 'graceful' shutdown if config timed out during initHåvard Pettersen2021-11-241-0/+4
|
* Update merge latency metrics after async writes have completed.Tor Egge2021-11-234-16/+70
|
* Revert "Continue serving search queries when in Maintenance node state ↵Henning Baldersheim2021-11-236-156/+39
| | | | [run-systemtest]"
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-236-39/+156
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* GC unused codeHenning Baldersheim2021-11-202-16/+0
|
* Remove redundant sync_allHenning Baldersheim2021-11-201-1/+0
|
* Fix forward declaration of NodeSupportedFeatures.Tor Egge2021-11-191-1/+1
|
* Let removeAsync handle list of documents.Henning Baldersheim2021-11-187-26/+29
|
* Move BucketIdFactory to test fixture.Henning Baldersheim2021-11-186-21/+13
|
* Move removeLocation over to Asynchandler and issue all removes for one ↵Henning Baldersheim2021-11-176-54/+65
| | | | | | bucket before waiting for the replies. Prepare RemoveResult to contain more replies.
* Add noexcept specifier to NodeSupportedFeaturesRepo constructor.Tor Egge2021-11-172-2/+2
|
* We only the docid for remove and stat.Henning Baldersheim2021-11-161-2/+2
|
* Address PR commentsTor Brede Vekterli2021-11-152-11/+13
| | | | | | * Add `from_distributor()` utility function to `MergeBucketCommand` * Simplify boolean expression by moving sub-expression to own statement * Improve wording of config parameter
* Add configurable support for unordered merge forwardingTor Brede Vekterli2021-11-1237-99/+550
| | | | | | | | | | | | | | | | | | | | | | | Historically the MergeThrottler component has required a deterministic forwarding of merges between nodes in strictly increasing distribution key order. This is to avoid distributed deadlocks caused by ending up with two or more nodes waiting for each other to release merge resources, where releasing one depends on releasing the other. This works well, but has the downside that there's an inherent pressure of merges towards nodes with lower distribution keys. These often become a bottleneck. This commit lifts this ordering restriction, by allowing forwarded, unordered merges to immediately enter the active merge window. By doing this we remove the deadlock potential, since nodes will longer be waiting on resources freed by other nodes. Since the legacy MergeThrottler has a lot of invariant checking around strictly increasing merge chains, we only allow unordered merges to be scheduled towards node sets where _all_ nodes are on a Vespa version that explicitly understands unordered merges (and thus do not self- obliterate upon seeing one). To communicate this, full bucket fetches will now piggy-back version-specific feature sets as part of the response protocol. Distributors then aggregate this information internally.
* Remove implicit write to tmp as part of cluster controller health pingTor Brede Vekterli2021-11-101-24/+0
| | | | | | | | | | | This was a feature tracing its lineage back to ye olde VDS days where we had a dusty root drive holding the OS and Vespa and a dozen spinning rust drives storing the actual data. The health ping would confirm that the root drive was still functioning properly by triggering a write to it, as the node would swiftly bail if any of the I/O operations failed. Today's access patterns are wildly different so we'll detect disk problems quickly anyway, not to mention Vespa Cloud has disk health monitoring built in.
* Fix compilation when using gcc 9:Tor Egge2021-10-311-0/+1
| | | | | - Include header file for atomic when needed. - Use normal function template instead of abbreviated function template.
* Rename ISequencedTaskExecutor::sync() to sync_all().Tor Egge2021-10-281-1/+1
|
* Remove createBucketAsync() spi call when merging buckets. It is noTor Egge2021-10-281-4/+0
| | | | longer needed.
* Merge pull request #19735 from ↵Geir Storli2021-10-286-0/+112
|\ | | | | | | | | vespa-engine/toregge/handover-tracker-to-apply-bucket-diff-state-on-exceptions Handover tracker to ApplyBucketDiffState on exceptions.
| * Handover tracker to ApplyBucketDiffState on exceptions.Tor Egge2021-10-266-0/+112
| |
* | Update 2020 Oath copyrights.gjoranv2021-10-272-2/+2
| |
* | Update 2019 Oath copyrights.gjoranv2021-10-2724-24/+24
| |
* | Fix forward declarations.Tor Egge2021-10-271-1/+1
| |
* | ApplyBucketDiffState uses std::optional, add needed include.Tor Egge2021-10-271-0/+1
|/
* Merge pull request #19725 from ↵v7.489.25Henning Baldersheim2021-10-253-19/+38
|\ | | | | | | | | vespa-engine/vekterli/prioritize-forwarded-merges-in-throttler-queue Prioritize forwarded merges in MergeThrottler queue
| * Prioritize forwarded merges in MergeThrottler queueTor Brede Vekterli2021-10-253-19/+38
| | | | | | | | | | | | | | Rationale: merges that already are part of an active merge window are taking up logical resources on one or more nodes in the cluster and we should prefer completing these before starting new merges queued from distributors.
* | Adjust dummy persistence spi semantics towards proton spi semantics whenTor Egge2021-10-251-4/+4
|/ | | | | | | bucket doesn't exist: * getBucketInfo() returns success with empty bucket info * createIterator() returns success * iterate() returns empty complete result.
* Merge pull request #19717 from ↵Geir Storli2021-10-258-9/+49
|\ | | | | | | | | vespa-engine/toregge/delay-apply-bucket-diff-state-deletion-try-2 Delay deletion of ApplyBucketState.
| * Delay deletion of ApplyBucketState.Tor Egge2021-10-258-9/+49
| |
* | GC unused codeHenning Baldersheim2021-10-251-15/+0
| |
* | create/delete bucket will never throw.Henning Baldersheim2021-10-258-31/+16
| |
* | No need to wait for the result of async createBucket as the result is ↵Henning Baldersheim2021-10-252-23/+28
| | | | | | | | ignored anyway, and no later operations depends on the outcome. In addition they are sequenced as the master executor conducts all operation in fifo order.
* | Async createBucketHenning Baldersheim2021-10-2512-161/+180
|/
* Fix MergeStatus::check_delayed_error: Empty fail message is not an error.Tor Egge2021-10-221-1/+3
|
* Forward new config to merge handler.Tor Egge2021-10-225-3/+22
|
* Merge pull request #19671 from ↵Geir Storli2021-10-228-42/+227
|\ | | | | | | | | vespa-engine/toregge/delay-replies-for-async-apply-bucket-diff Delay replies for async apply bucket diff.
| * Add comment about _tracker->_reply and _delayed_reply in ApplyBucketDiffStateTor Egge2021-10-221-0/+1
| | | | | | | | pointing to the same reply.
| * Add class comment for MergeHandlerTest.Tor Egge2021-10-211-0/+4
| |
| * Delay replies for async apply bucket diff.Tor Egge2021-10-218-42/+222
| |
* | Add noexcept specifier to operation complete callback.Tor Egge2021-10-223-3/+3
|/
* Remove bad std::move.Tor Egge2021-10-201-1/+1
|
* Eliminate ApplyBucketDiffEntryResult.Tor Egge2021-10-2010-173/+128
|