aboutsummaryrefslogtreecommitdiffstats
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* Merge pull request #18287 from ↵Geir Storli2021-06-161-10/+2
|\ | | | | | | | | vespa-engine/geirst/fix-stripe-dispatch-of-request-bucket-info-reply Dispatch RequestBucketInfoReply for non-existing buckets to correct d…
| * Dispatch RequestBucketInfoReply for non-existing buckets to correct ↵Geir Storli2021-06-161-10/+2
| | | | | | | | distributor stripe.
* | Rename bucket db updater test that is only testing legacy mode.Geir Storli2021-06-164-96/+98
| |
* | Rename destributor test that is only testing legacy mode.Geir Storli2021-06-165-70/+72
|/
* Assert that the storage message has a valid bucket id for striping.Geir Storli2021-06-152-9/+11
|
* Add per stripe handling of ideal state metrics with aggregation on top.Geir Storli2021-06-159-36/+127
| | | | This is handled similarly to per stripe distributor metrics.
* Merge pull request #18247 from ↵Geir Storli2021-06-152-4/+20
|\ | | | | | | | | vespa-engine/toregge/aggregate-metrics-on-the-fly-when-adding-to-snapshot Aggregate distributor metrics when adding to snapshot.
| * Factor out common code to private helper member function.Tor Egge2021-06-142-8/+12
| |
| * Aggregate distributor metrics when adding to snapshot.Tor Egge2021-06-142-0/+12
| |
* | Measure queue size after element have been inserted, and stabilize test by ↵Henning Baldersheim2021-06-142-2/+3
|/ | | | waiting for full Q
* Only support legacy mode in getActiveIdealStateOperations().Geir Storli2021-06-112-23/+2
| | | | This function is only used by idealstatemanagertest in the context of testing a single stripe.
* Merge pull request #18219 from ↵Geir Storli2021-06-115-10/+88
|\ | | | | | | | | vespa-engine/toregge/aggregate-metrics-from-distributor-stripes-pass-2 Aggregate metrics from distributor stripes.
| * Remove stale comments.Tor Egge2021-06-112-5/+3
| | | | | | | | Reorder member variables.
| * Aggregate metrics from distributor stripes.Tor Egge2021-06-115-6/+86
| |
* | Merge pull request #18218 from ↵Geir Storli2021-06-111-0/+1
|\ \ | |/ |/| | | | | vespa-engine/geirst/getnodestate-command-in-distributor-main-thread Handle GetNodeStateCommand in distributor main thread when running in…
| * Handle GetNodeStateCommand in distributor main thread when running in new ↵Geir Storli2021-06-111-0/+1
| | | | | | | | stripe mode.
* | Provide correct stripe index to notify_stripe_wants_to_send_host_info.Tor Egge2021-06-111-1/+1
|/
* MinReplica counts the minimum bucket replication factor, so use std::min ↵Geir Storli2021-06-112-2/+4
| | | | instead of sum.
* Implement aggregation across distributor stripes for min replica, bucket ↵Geir Storli2021-06-1012-15/+316
| | | | spaces, and pending maintenance stats.
* Block ideal state ops when a pending cluster state is presentTor Brede Vekterli2021-06-0929-139/+223
| | | | | | | | | | | | Since distributor stripes no longer have access to the top-level pending message tracking info, it's no longer possible to infer if a pending cluster state is happening by looking at the sent messages. Instead, do this more generally (and efficiently) by looking at the potential pending cluster state directly. Rewire the `isBlocked` logic to take in an operation context instead of just a `PendingMessageTracker`, giving it access to a lot more relevant information.
* Route CreateVisitorCommand with too few used bits in the super bucket id to ↵Geir Storli2021-06-092-5/+26
| | | | | | a random distributor stripe. Such commands will eventually be bounced with WRONG_DISTRIBUTION when handled by the stripe.
* addValue -> set for gauge metric.Henning Baldersheim2021-06-071-1/+1
|
* Add queue size metricHenning Baldersheim2021-06-063-1/+5
|
* Merge pull request #18102 from ↵Geir Storli2021-06-031-0/+7
|\ | | | | | | | | vespa-engine/geirst/dispatch-get-and-visitor-messages-to-stripe Dispatch get and visitor messages to correct distributor stripe.
| * Dispatch get and visitor messages to correct distributor stripe.Geir Storli2021-06-031-0/+7
| |
* | Use a hash map for specs. If the request is a point lookup then just use a ↵Henning Baldersheim2021-06-022-4/+3
|/ | | | | | | | hash lookup. If it is a wildcard lookup iterate as earlier on. Also use vespalib::stringref in interface to avoid conversion. Use vespalib:string in the hash map to locate string in object aswe are still on old abi.
* Add proof of concept support for multiple distributor stripes.Geir Storli2021-06-016-23/+83
| | | | | | | | | The most basic functionality is now supported using multiple distributor stripes (and threads). Note that the following is (at least) still missing: * Stripe-separate metrics with top-level aggregation. * Aggregation over all stripes in misc functions in Distributor that currently is using the first stripe. * Handling of messages without bucket id in the top-level Distributor instead of using the first stripe.
* Merge pull request #18050 from ↵Geir Storli2021-06-016-7/+39
|\ | | | | | | | | vespa-engine/geirst/validate-distributor-stripes-config Add validation of the number of distributor stripes from config and a…
| * Add validation of the number of distributor stripes from config and add more ↵Geir Storli2021-06-016-7/+39
| | | | | | | | | | | | asserts. This ensures the number of stripes is a power of 2 and within MaxStripes boundary.
* | Minor code cleanupTor Brede Vekterli2021-05-312-34/+5
| |
* | Do not block global merges to nodes tagged as busyTor Brede Vekterli2021-05-313-5/+39
|/ | | | | | | | | | | | | | To avoid starvation of high priority global bucket merges, we do not consider these for blocking due to a node being "busy" (usually caused by a full merge throttler queue). This is for two reasons: 1. When an ideal state op is blocked, it is still removed from the internal maintenance priority queue. This means a blocked high pri operation will not be retried until the next DB pass (at which point the node is likely to still be marked as busy when there's heavy merge traffic). 2. Global bucket merges have high priority and will most likely be allowed to enter the merge throttler queues, displacing lower priority merges.
* Make merge_entries_into_db() work across multiple stripes by handling each ↵Geir Storli2021-05-285-12/+123
| | | | stripe in sequence.
* Add common utils to map from bucket key to stripe and calculcate number of ↵Geir Storli2021-05-287-12/+112
| | | | stripe bits.
* admin/slobrok.0 does not always exist ....... anymore.Henning Baldersheim2021-05-271-1/+1
|
* Add a couple of stripe work TODOsTor Brede Vekterli2021-05-262-0/+4
| | | | | | | | - Ideal state ops cannot look at null-bucket messages for determining if full bucket checks are pending when running in striped mode, as these are not handled by stripes when not in legacy mode. - State checker context should use ideal state cache instead of recomputing for every checked bucket (observed via `perf` in production).
* Remove extra braces from initializer list.Tor Egge2021-05-251-1/+1
|
* Merge pull request #17943 from vespa-engine/vekterli/minor-code-cleanupGeir Storli2021-05-2115-72/+41
|\ | | | | Minor cleanups in distributor maintenance handling code
| * Minor cleanups in distributor maintenance handling codeTor Brede Vekterli2021-05-2115-72/+41
| | | | | | | | No functional changes
* | GC unused codeHenning Baldersheim2021-05-211-26/+0
|/
* Make distributor timestamp generation thread safeTor Brede Vekterli2021-05-185-23/+42
| | | | | | | | | | | | | | New behavior: - Only allow time to travel forwards within a given distributor process' lifetime. This is a change from the old behavior, which would emit a warning to the logs and happily continue from a previously used second, possibly causing the distributor to reuse timestamps. - Try to detect cases where the wall clock has been transiently set far into the future--only to bounce back--by aborting the process if the current observed time is more than 120 seconds older than the highest observed wall clock time. This is an attempt to avoid generating _too_ many bogus future timestamps, as the distributor would otherwise continue generating timestamps within the highest observed second.
* Reduce direct use of DistributorStripeComponent.Geir Storli2021-05-1437-104/+129
|
* Merge pull request #17834 from vespa-engine/geirst/impl-bucket-space-state-mapGeir Storli2021-05-124-2/+143
|\ | | | | Implement class that provides mapping from bucket space to state for …
| * Implement class that provides mapping from bucket space to state for that space.Geir Storli2021-05-124-2/+143
| |
* | Merge pull request #17830 from ↵Tor Brede Vekterli2021-05-127-64/+178
|\ \ | |/ |/| | | | | vespa-engine/vekterli/add-initial-multi-stripe-support-to-access-guard Add initial multi stripe support to access guard [run-systemtest]
| * Add initial multi stripe support to access guardTor Brede Vekterli2021-05-127-64/+178
| | | | | | | | | | | | Still missing functionality for: - Merging bucket entries across stripes - Aggregating pending operation stats across stripes
* | Stop all stripe threads before starting shutdown (and closing) of the ↵Geir Storli2021-05-117-19/+31
|/ | | | | | | | | | storage link chain. This is required to avoid stripe threads being able to send up messages while the communication manager is being closed. Such messages will fail at the RPC layer (already closed) and an error reply is sent down from the communication manager. This triggers an assert in StorageLink::sendDown() which is already CLOSED.
* Extend TickableStripe interface to avoid direct access to DistributorStripe ↵Tor Brede Vekterli2021-05-118-27/+224
| | | | | | | internals Also lets us test guard functionality much more easily since its target is now fully mockable.
* Explicitly signal locking requirements in function signatureTor Brede Vekterli2021-05-102-3/+3
|
* Add timed batching of explicit host info sends triggered by stripesTor Brede Vekterli2021-05-108-12/+193
| | | | | | | | | | | | | | | | | | | | | Since distributor stripes may independently reach a conclusion that a `GetNodeState` reply containing new host info should be sent back to the cluster controller, implement basic rate limiting/batching of concurrent sends. Batching has two separate modes of operation: - If the node is initializing, host info will be sent immediately after _all_ stripes have reported in (they will always do this post-init). This is not timed, in order to minimize latency of bucket info being visible to the cluster controller. - If the node has already initialized, have a grace period of up to 1 second from the time the first stripe signals its intent to send host info until it's actually sent. This allows several stripes to complete their recovery mode and signal host info intents during this second. Batch time period is currently not configurable, may be done later if deemed useful or necessary.
* Propagate config to underlying bucket repos when storage distribution changes.Geir Storli2021-05-102-0/+14
| | | | | | | This fixes the following system tests when running with new distributor stripe mode: Capacity::test_capacity FlatToHierarchicTransitionTest::test_transition_implicitly_indexes_and_activates_docs_per_group HierarchDistr::test_app_change__PROTON