| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
| |
* Add `from_distributor()` utility function to `MergeBucketCommand`
* Simplify boolean expression by moving sub-expression to own statement
* Improve wording of config parameter
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Historically the MergeThrottler component has required a deterministic
forwarding of merges between nodes in strictly increasing distribution
key order. This is to avoid distributed deadlocks caused by ending up
with two or more nodes waiting for each other to release merge resources,
where releasing one depends on releasing the other. This works well,
but has the downside that there's an inherent pressure of merges towards
nodes with lower distribution keys. These often become a bottleneck.
This commit lifts this ordering restriction, by allowing forwarded,
unordered merges to immediately enter the active merge window. By doing
this we remove the deadlock potential, since nodes will longer be waiting
on resources freed by other nodes.
Since the legacy MergeThrottler has a lot of invariant checking around
strictly increasing merge chains, we only allow unordered merges to be
scheduled towards node sets where _all_ nodes are on a Vespa version
that explicitly understands unordered merges (and thus do not self-
obliterate upon seeing one). To communicate this, full bucket fetches
will now piggy-back version-specific feature sets as part of the response
protocol. Distributors then aggregate this information internally.
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We would previously check for the presence of pending null-bucket
`RequestBucketInfoCommand`s to determine if a pending cluster state
was present. We would also attempt to block all bucket delete operations
from starting if _any_ operation was pending towards that bucket on
a given node, including bucket info requests. The former was rewritten
to instead explicitly consider pending cluster state checks instead,
as checking null buckets no longer works when using stripes.
Unfortunately, due to a long-standing bug with message tracking of
`RequestBucketInfoCommand`s, these would _always_ be marked as pending
towards the null bucket. Since all ideal state ops would be blocked
by null-bucket info requests, this would avoid starting any ideal
state op as long as _any_ other op had an info request pending for
the target node. This had the desirable (but not explicitly coded for)
side effect of inhibiting bucket deletions from racing with half-finished
merge operations. It also had the undesirable effect of needlessly
blocking ops for completely unrelated buckets.
With these changes, we now explicitly handle bucket info requests for
single buckets in the `PendingMessageTracker`, allowing inhibition
of deletions to work as expected. Also add an explicit check for
pending info requests for all ideal state ops to mirror the old
behavior (but now per-bucket instead of globally...!).
|
|
|
|
| |
distributor stripe.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduces the concept of stripe access guards, that ensure safe and
non-concurrent access to the underlying state of all running distributor
stripes. Also bring back a top-level `BucketDBUpdater` component responsible
for managing cluster state/distribution config and all related bucket info
fetching for the entire node as a whole. This component abstracts away all
stripe-specific operations via the new guard interface.
For now, only a single stripe can be used via the new code path, and by
default the legacy code path (single stripe acts as an entire distirbutor)
is used. New path may be enabled via (non-live) config, but is not yet
production ready.
|
|
|
|
|
|
|
|
| |
be reported back from bucket executor.
- Treat remapping as an error.
- For lidspace compaction job iterator is reset and will be recreated on next invocation.
- For bucketmove th ebucket is rechecked and either discarded or restarted.
|
| |
|
| |
|
| |
|
|
|
|
| |
order… "
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|\
| |
| | |
Inline some small constructors and also prefer moving the return code.
|
| | |
|
| | |
|
| | |
|
|/ |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
compact the message objects.
- Compact StorageMessageAddress to 16 bytes by
- using reference to cluster name.
- Use small enums for protocol and node type.
- Avoid having StorageMessage as separate allocation.
- Avoid default values
|
|\
| |
| |
| |
| | |
vespa-engine/geirst/simplify-storage-message-address
Simplify storage message address
|
| | |
|
| |
| |
| |
| |
| |
| | |
StorageMessageAddress.
Creating and deleting the route is expensive and not used with RPC for Storage API communication.
|
| | |
|
| | |
|
| | |
|
|/
|
|
| |
This reduces the size of frequently used objects.
|
| |
|
| |
|
|
|
|
|
| |
- Avoid needing the definition of Error everywhere.
- use std::make_xxx and other c++11 constructs.
|
| |
|
|
|
|
| |
the root.
|
| |
|
|
|
|
| |
happy path fast.
|
| |
|
| |
|
| |
|