aboutsummaryrefslogtreecommitdiffstats
path: root/storageapi
Commit message (Collapse)AuthorAgeFilesLines
* Address PR commentsTor Brede Vekterli2021-11-151-0/+1
| | | | | | * Add `from_distributor()` utility function to `MergeBucketCommand` * Simplify boolean expression by moving sub-expression to own statement * Improve wording of config parameter
* Add configurable support for unordered merge forwardingTor Brede Vekterli2021-11-125-6/+56
| | | | | | | | | | | | | | | | | | | | | | | Historically the MergeThrottler component has required a deterministic forwarding of merges between nodes in strictly increasing distribution key order. This is to avoid distributed deadlocks caused by ending up with two or more nodes waiting for each other to release merge resources, where releasing one depends on releasing the other. This works well, but has the downside that there's an inherent pressure of merges towards nodes with lower distribution keys. These often become a bottleneck. This commit lifts this ordering restriction, by allowing forwarded, unordered merges to immediately enter the active merge window. By doing this we remove the deadlock potential, since nodes will longer be waiting on resources freed by other nodes. Since the legacy MergeThrottler has a lot of invariant checking around strictly increasing merge chains, we only allow unordered merges to be scheduled towards node sets where _all_ nodes are on a Vespa version that explicitly understands unordered merges (and thus do not self- obliterate upon seeing one). To communicate this, full bucket fetches will now piggy-back version-specific feature sets as part of the response protocol. Distributors then aggregate this information internally.
* Update 2019 Oath copyrights.gjoranv2021-10-279-9/+9
|
* Fix node list printing for storage messages used by merging.Tor Egge2021-10-221-11/+12
|
* Update Verizon Media copyright notices.gjoranv2021-10-072-2/+2
|
* Update 2017 copyright notices.gjoranv2021-10-0781-81/+81
|
* allow generated PB files outside source treeArne H Juul2021-09-221-4/+4
|
* Avoid race condition regression introduced in #18179Tor Brede Vekterli2021-06-241-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | We would previously check for the presence of pending null-bucket `RequestBucketInfoCommand`s to determine if a pending cluster state was present. We would also attempt to block all bucket delete operations from starting if _any_ operation was pending towards that bucket on a given node, including bucket info requests. The former was rewritten to instead explicitly consider pending cluster state checks instead, as checking null buckets no longer works when using stripes. Unfortunately, due to a long-standing bug with message tracking of `RequestBucketInfoCommand`s, these would _always_ be marked as pending towards the null bucket. Since all ideal state ops would be blocked by null-bucket info requests, this would avoid starting any ideal state op as long as _any_ other op had an info request pending for the target node. This had the desirable (but not explicitly coded for) side effect of inhibiting bucket deletions from racing with half-finished merge operations. It also had the undesirable effect of needlessly blocking ops for completely unrelated buckets. With these changes, we now explicitly handle bucket info requests for single buckets in the `PendingMessageTracker`, allowing inhibition of deletions to work as expected. Also add an explicit check for pending info requests for all ideal state ops to mirror the old behavior (but now per-bucket instead of globally...!).
* Dispatch RequestBucketInfoReply for non-existing buckets to correct ↵Geir Storli2021-06-162-1/+11
| | | | distributor stripe.
* Dispatch get and visitor messages to correct distributor stripe.Geir Storli2021-06-032-0/+14
|
* Initial implementation and wiring of cross-stripe state and DB handlingTor Brede Vekterli2021-04-212-4/+9
| | | | | | | | | | | | | | Introduces the concept of stripe access guards, that ensure safe and non-concurrent access to the underlying state of all running distributor stripes. Also bring back a top-level `BucketDBUpdater` component responsible for managing cluster state/distribution config and all related bucket info fetching for the entire node as a whole. This component abstracts away all stripe-specific operations via the new guard interface. For now, only a single stripe can be used via the new code path, and by default the legacy code path (single stripe acts as an entire distirbutor) is used. New path may be enabled via (non-live) config, but is not yet production ready.
* - Change error handling so that both synchonous and asynchronous errors can ↵Henning Baldersheim2021-02-231-2/+2
| | | | | | | | be reported back from bucket executor. - Treat remapping as an error. - For lidspace compaction job iterator is reset and will be recreated on next invocation. - For bucketmove th ebucket is rechecked and either discarded or restarted.
* - Reduce visibility of ClusterState and Distribution.Henning Baldersheim2021-02-195-1/+7
|
* use size literals in storageArne Juul2021-02-151-1/+2
|
* Add noexcept and some other minor code health.Henning Baldersheim2021-02-022-23/+22
|
* Revert "Properly track execution of BucketTasks and provide sync() and ↵Henning Baldersheim2021-02-022-22/+23
| | | | order… "
* Properly track execution of BucketTasks and provide sync() and orderly shutdown.Henning Baldersheim2021-02-022-23/+22
|
* Revert "Implement BucketExecutor::sync."Henning Baldersheim2021-02-022-22/+23
|
* Implement BucketExecutor::sync.Henning Baldersheim2021-02-012-23/+22
|
* Include exception where needed and use std:.make_uniqueHenning Baldersheim2021-01-181-3/+1
|
* GC unused code and remove comment.Henning Baldersheim2021-01-181-1/+0
|
* Improve readability.Henning Baldersheim2021-01-181-208/+73
|
* GC unused hasSingleBucketId method.Henning Baldersheim2021-01-183-7/+4
|
* Remove storageframework dependency not needed.Henning Baldersheim2021-01-181-1/+0
|
* Merge pull request #15502 from vespa-engine/balder/inline-and-move-returncodeHenning Baldersheim2020-11-278-43/+31
|\ | | | | Inline some small constructors and also prefer moving the return code.
| * Inline some small constructors and also prefer moving the return code.Henning Baldersheim2020-11-278-43/+31
| |
* | Use explicit on the right places.Henning Baldersheim2020-11-272-2/+2
| |
* | GC LoadType and PriorityMapper.Henning Baldersheim2020-11-275-15/+3
| |
* | GC LoadTypeSetHenning Baldersheim2020-11-2712-67/+34
|/
* Add noexceptHenning Baldersheim2020-11-262-49/+49
|
* Comment removedHenning Baldersheim2020-11-261-1/+1
|
* Use correct storage cluster name.Henning Baldersheim2020-11-261-1/+1
|
* As we have have now removed the expensive Route member we can further ↵Henning Baldersheim2020-11-265-97/+78
| | | | | | | | | | compact the message objects. - Compact StorageMessageAddress to 16 bytes by - using reference to cluster name. - Use small enums for protocol and node type. - Avoid having StorageMessage as separate allocation. - Avoid default values
* Merge pull request #15466 from ↵Henning Baldersheim2020-11-254-33/+28
|\ | | | | | | | | vespa-engine/geirst/simplify-storage-message-address Simplify storage message address
| * Simplify hash calculation.Geir Storli2020-11-251-7/+2
| |
| * Create the mbus::Route on demand instead of storing it inside ↵Geir Storli2020-11-254-33/+33
| | | | | | | | | | | | StorageMessageAddress. Creating and deleting the route is expensive and not used with RPC for Storage API communication.
* | Handle different size of std::string.Tor Egge2020-11-251-4/+4
| |
* | Just return default constructed object.Henning Baldersheim2020-11-251-4/+4
| |
* | Track size of frequently used objects.Henning Baldersheim2020-11-241-0/+4
| |
* | Move the error description to a separate allocation as it is rarely used.Henning Baldersheim2020-11-245-30/+67
|/ | | | This reduces the size of frequently used objects.
* Remove load type from StorageMessageHenning Baldersheim2020-11-246-25/+39
|
* Remove load type from MergeStatus and Mergehandler.Henning Baldersheim2020-11-241-1/+1
|
* - Reorder members for compactness.Henning Baldersheim2020-11-232-8/+4
| | | | | - Avoid needing the definition of Error everywhere. - use std::make_xxx and other c++11 constructs.
* Avoid having mutating methods const.Henning Baldersheim2020-11-231-3/+3
|
* Steal the traces explicit and force moving of traces. Also hide access to ↵Henning Baldersheim2020-11-231-2/+3
| | | | the root.
* Only copy tracelevel.Henning Baldersheim2020-11-233-8/+2
|
* Use a std:.unique_ptr to make Trace a thin wrapper for TraceNode to make the ↵Henning Baldersheim2020-11-181-2/+2
| | | | happy path fast.
* Explicit copy construction of Trace.Henning Baldersheim2020-11-183-4/+6
|
* No need to copy an empty object into another empty object.Henning Baldersheim2020-11-182-2/+10
|
* Reorder members for smaller memory footprint.Henning Baldersheim2020-11-185-23/+28
|