summaryrefslogtreecommitdiffstats
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* - Wire in the guard to make it evident that we have it when making changes ↵Henning Baldersheim2020-12-152-4/+6
| | | | | | that require it. - Clean up some old members and code not used any more.
* Gracefully abort outdated reindexing visitorsTor Brede Vekterli2020-12-147-23/+104
| | | | | | | | | | Instead of sending `REJECTED` when lock token does not match, send `TEST_AND_SET_CONDITION_FAILED` from distributors. The reindexing visitor will detect this and remap the failure code to `ABORTED` while simultaneously failing the backend visitor. `ABORTED` will be considered transient by the visiting client, allowing it to retry towards another distributor without the whole thing having to fail out.
* Merge pull request #15795 from ↵Tor Brede Vekterli2020-12-1425-63/+268
|\ | | | | | | | | vespa-engine/vekterli/use-random-bucket-lock-passthrough-token-for-reindexing-visitors Only let reindexing puts through locked bucket if their TaS token matches that of the lock
| * Only let reindexing puts through locked bucket if their token matches that ↵Tor Brede Vekterli2020-12-1125-63/+268
| | | | | | | | | | | | | | | | | | | | | | of the lock Avoids race condition edge case where reindexing puts from an outdated visitor may pass through a bucket lock intended for a newly created visitor operation Tokens are 128-bit values derived from a CSPRNG, so uniqueness is for all intents and purposes guaranteed.
* | Merge pull request #15798 from ↵Henning Baldersheim2020-12-1120-98/+100
|\ \ | | | | | | | | | | | | vespa-engine/balder/wire-cluster-name-as-reference-all-the-way - Wire the clustername through as a reference to avoid referencing te…
| * | - Wire the clustername through as a reference to avoid referencing temporaries.Henning Baldersheim2020-12-1120-98/+100
| |/
* | Merge pull request #15797 from ↵Geir Storli2020-12-115-25/+35
|\ \ | |/ |/| | | | | vespa-engine/geirst/optimize-two-phase-update-operation Avoid lookup bucket database entries two times when handling fast pat…
| * Avoid lookup bucket database entries two times when handling fast path ↵Geir Storli2020-12-115-25/+35
| | | | | | | | update operation.
* | Merge pull request #15788 from vespa-engine/toregge/extend-use-of-ownership-hashTor Egge2020-12-117-96/+86
|\ \ | | | | | | Extend use of bucket ownership hash.
| * | Extend use of bucket ownership hash.Tor Egge2020-12-117-96/+86
| |/
* | Merge pull request #15753 from ↵Tor Egge2020-12-118-147/+0
|\ \ | |/ |/| | | | | vespa-engine/toregge/remove-internal-bucket-join-command Remove InternalBucketJoinCommand and InternalBucketJoinReply.
| * Remove InternalBucketJoinCommand and InternalBucketJoinReply.Tor Egge2020-12-098-147/+0
| |
* | Avoid constructing a TraceNode (via ensureRoot) when tracing is disabled.Geir Storli2020-12-101-1/+1
| |
* | Make "events before wakeup" configurable for SharedRpcResources.Geir Storli2020-12-106-6/+13
| |
* | Bounce reindexing visitor with BUSY if merge is pending for bucketTor Brede Vekterli2020-12-095-0/+62
| | | | | | | | | | | | | | | | | | | | | | Since reindexing visitors take a bucket lock when they arrive and wait for pending ops to drain before they start, doing so when there's a pending merge risks starving the bucket for a long time. This is because merges may linger for a long time in the merge throttling queues in the cluster. By not starting such visitors if there is a pending merge, we avoid this edge case. Functionality is already in place to inhibit merges from starting if there's an active bucket lock present.
* | Remove BucketOperationLogger.Tor Egge2020-12-0913-512/+0
|/
* Remove outdated comment.Tor Egge2020-12-091-2/+0
|
* Cleanup remains of disk concept.Tor Egge2020-12-081-29/+24
|
* Remove leftovers from removed disk concept.Tor Egge2020-12-0813-141/+2
|
* Merge pull request #15725 from ↵Tor Brede Vekterli2020-12-0828-57/+199
|\ | | | | | | | | vespa-engine/vekterli/block-ideal-state-ops-towards-locked-buckets Block ideal state operations towards buckets that are locked
| * Block ideal state operations towards buckets that are lockedTor Brede Vekterli2020-12-0728-57/+199
| | | | | | | | | | | | | | Prevents ideal state ops from modifying buckets that are being used in a read-for-write context. Move `OperationSequencer` to main `Distributor` to more easily facilitate sharing of it across components.
* | Annotate config requiring restart, and properly handle the ones that do notHenning Baldersheim2020-12-073-19/+21
|/
* Add noexcept specifiers to non-throwing constructors and operators.Tor Egge2020-12-058-11/+11
|
* Merge pull request #15664 from ↵Geir Storli2020-12-0530-175/+292
|\ | | | | | | | | vespa-engine/geirst/decouple-external-operation-handler Decouple external operation handler
| * Remove explicit.Geir Storli2020-12-041-2/+2
| |
| * Use noexcept for functions that return a reference.Geir Storli2020-12-043-13/+13
| |
| * Remove dependency between ExternalOperationHandler and Distributor.Geir Storli2020-12-045-16/+29
| |
| * Remove dependency between ExternalOperationHandler and DistributorComponent.Geir Storli2020-12-0329-161/+265
| |
* | Use the super bucket key when resolving which RPC target to use to handle a ↵Geir Storli2020-12-045-10/+68
| | | | | | | | command.
* | Remove unused member variable.Tor Egge2020-12-031-1/+0
|/
* Merge pull request #15632 from ↵Henning Baldersheim2020-12-037-32/+159
|\ | | | | | | | | vespa-engine/toregge/faster-distributor-bucket-db-update Add process_update member function to BucketDatabase.
| * Add process_update member function to BucketDatabase. It is used forTor Egge2020-12-037-32/+159
| | | | | | | | updating an entry with a single lookup.
* | Revert "Use the super bucket key when resolving which RPC target to use to ↵Jon Marius Venstad2020-12-035-62/+11
| | | | | | | | ha…"
* | Merge pull request #15611 from ↵Tor Brede Vekterli2020-12-035-42/+153
|\ \ | | | | | | | | | | | | vespa-engine/vekterli/allow-starting-deferred-tasks-concurrently-with-reads Allow starting deferred tasks concurrently with reads
| * | Allow starting deferred tasks concurrently with pending read opsTor Brede Vekterli2020-12-023-16/+77
| | | | | | | | | | | | | | | | | | Since read ops aren't blocked by active bucket locks, a constant stream of read ops towards a bucket could otherwise starve deferred tasks from starting.
| * | Test edge case where a bucket is removed before starting deferred visitorTor Brede Vekterli2020-12-022-26/+76
| |/
* | Make function inline and noexcept.Geir Storli2020-12-031-1/+1
| |
* | Use the super bucket key when resolving which RPC target to use to handle a ↵Geir Storli2020-12-025-11/+62
|/ | | | command.
* Merge pull request #15594 from ↵Henning Baldersheim2020-12-029-46/+27
|\ | | | | | | | | vespa-engine/toregge/remove-distributor-component-trampoline-member-function Remove DistributorComponent trampoline member function.
| * Remove getIdealNodes() trampoline member function from DistributorComponent.Tor Egge2020-12-024-13/+5
| |
| * Use mutable hash maps in DistributorBucketSpace.Tor Egge2020-12-024-8/+8
| |
| * Remove DistributorComponent trampoline member function.Tor Egge2020-12-027-29/+18
| |
* | Merge pull request #15572 from ↵Tor Brede Vekterli2020-12-0226-146/+941
|\ \ | |/ |/| | | | | vespa-engine/vekterli/initial-backend-reindexing-visitor-support Initial support for backend reindexing visitor functionality
| * Add top-level commentsTor Brede Vekterli2020-12-022-3/+21
| |
| * Simplify code interacting with blocked sequencing handlesTor Brede Vekterli2020-12-023-9/+18
| |
| * Initial support for backend reindexing visitor functionalityTor Brede Vekterli2020-12-0126-146/+914
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Introduces the concept of a read-for-write visitor operation which blocks all mutating operations from starting for a bucket being visited. This read-for-write mode is used if (and only if) the visitor library being specified by the client is "reindexingvisitor". Since read-for-write visitors cannot race with concurrent write operations, starting such visitors are deferred until no further mutations are pending. Also adds a basic reindexingvisitor implementation to the content node which sends all documents as Puts containing a special TaS token that will let the operation through even if a bucket is locked. This token is cleared by the distributor before it is passed on to the content nodes. Note: this feature is not yet production ready. For now the following caveats apply: * Mutating vs non-mutating pending ops to a bucket are not tracked separately, so it’s possible to starve a reindexing visitor by sending constant pending read load, as read load is not blocked by the operation sequencer. * Ideal state operations towards locked buckets are not blocked, so it's possible for e.g. a split to be sent for a bucket that is being visited.
* | Remove BucketDBUpdater trampoline member function.Tor Egge2020-12-029-25/+9
| |
* | Trim down DistributorComponent member functions.Tor Egge2020-12-027-59/+25
| |
* | Move config setters directly onto main TransportConfig objectHenning Baldersheim2020-12-011-1/+1
| |
* | provide the config in the constructor only.Henning Baldersheim2020-12-011-4/+2
| |