aboutsummaryrefslogtreecommitdiffstats
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* Add comment with rationale for using atomic value updates instead of tree ↵Tor Brede Vekterli2022-03-141-0/+10
| | | | thawing/freezing
* Make B-tree bucket database values atomic to ensure well-defined accessTor Brede Vekterli2022-03-113-24/+98
| | | | | | | | | | | | | Existing implementation already used explicit acquire/release fences to ensure visibility from writer to concurrent readers, but the values written/read were not of an atomic type and thus _technically_ considered a data race. This commit adds an AtomicValueWrapper wrapper to vespalib which looks and acts much like the existing AtomicEntryRef, but for primitive types that are not related to EntryRefs. The bucket DB B-tree primitive u64 value type is replaced with an atomic wrapper and explicit memory fences are replaced with release stores and acquire loads on the atomic values themselves to ensure they form correct pairs between writer and readers.
* Reduce exposure of SymbolTable, Stash and other classes not necessary for ↵Henning Baldersheim2022-03-101-0/+1
| | | | users of Slime.
* Move BucketIdListResultHenning Baldersheim2022-03-092-2/+2
|
* Reduce visibility of document::DocumentHenning Baldersheim2022-03-0714-44/+33
|
* Reduce use of Identifiable for document::DatatTypeHenning Baldersheim2022-03-0310-2/+16
|
* Check cluster-wide state inside lock to match state writeTor Brede Vekterli2022-03-031-1/+3
|
* Merge pull request #21477 from ↵Henning Baldersheim2022-03-014-18/+22
|\ | | | | | | | | vespa-engine/balder/use-wait_until-to-reduce-need-for-sampling-time Use wait_until and a deadline instead of a timeout.
| * timeout_end -> deadlineHenning Baldersheim2022-03-014-10/+10
| |
| * Use wait_until and a deadline instead of a timeout.Henning Baldersheim2022-03-014-18/+22
| | | | | | | | This reduces the need to smaple the time significantly.
* | Make reconfiguring dynamic vs unlimited throttling not require restartingTor Brede Vekterli2022-03-016-35/+52
|/ | | | | | | | | | | | Instead of having one abstract throttler created from bootstrap config, explicitly create one dynamic and one unlimited throttler and allow for atomically switching between the two based on received config. The `MergeHandler` component will now always fetch the current throttler from the `FileStorHandler` instead of caching it at construction time. This commit removes the `restart` annotation on the existing throttler type config enums.
* Add a simple adjustTimeoutByDetectedHz to compute a suitable timeout/naptime ↵Henning Baldersheim2022-02-282-2/+2
| | | | | | based on base VESPA_TIMER_HZ to use for random waiting when doing a poll based appraoch.
* Merge pull request #21408 from ↵Tor Brede Vekterli2022-02-286-2/+55
|\ | | | | | | | | vespa-engine/vekterli/fix-storage-slobrok-threading-edge-cases Gracefully flush pending Slobrok task on content node RPC teardown [run-systemtest]
| * Explicitly wait for cross-node Slobrok visibility in test instead of sleepingTor Brede Vekterli2022-02-285-2/+50
| | | | | | | | | | | | | | Prior code had an ad-hoc (and undocumented) sleep that would hopefully delay the rest of the test until there was mutual Slobrok visibility, but which was not guaranteed and usually over-pessimized. Replace with explicit checks of Slobrok mirrors.
| * Gracefully flush pending Slobrok task on content node RPC teardownTor Brede Vekterli2022-02-251-0/+5
| | | | | | | | | | Avoids race between transport connection destruction in one FNET thread and processing unregistration task in another FNET thread.
* | Remove inlining warnings (storage).Tor Egge2022-02-261-0/+3
|/
* Sync executor to ensure tasks are run prior to stripe teardowns in testsTor Brede Vekterli2022-02-251-2/+11
|
* Revert "Revert "- Create the common transport and threadpool in the main loop.""Henning Baldersheim2022-02-241-1/+1
|
* Revert "- Create the common transport and threadpool in the main loop."Henning Baldersheim2022-02-241-1/+1
|
* Merge pull request #21376 from ↵Tor Brede Vekterli2022-02-248-9/+52
|\ | | | | | | | | vespa-engine/vekterli/configurable-per-merge-feed-op-throttling Make it possible to configure what aspects of merges should be throttled [run-systemtest]
| * Make it possible to configure what aspects of merges should be throttledTor Brede Vekterli2022-02-248-9/+52
| | | | | | | | | | | | | | | | | | | | Add live config for choosing whether merges should be throttled on a per-feed operation (`MergeHandler`) level, or on an `ApplyBucketDiff` persistence thread level. This is intended to be a temporary feature while we do experiments, so some liberties are taken with regards to how holes are punched in the various abstraction layers.
* | - Create the common transport and threadpool in the main loop.Henning Baldersheim2022-02-241-1/+1
|/ | | | | - Also use the common transport for config subscriptions. - Put The TransportConfig in the fnet namespace.
* Add more config for tuning the persistence throttle policyTor Brede Vekterli2022-02-231-6/+10
|
* Add backend metrics for new operation throttler statisticsTor Brede Vekterli2022-02-223-0/+6
|
* Make ConfigUri constructors explicit and use same context where possible in ↵Henning Baldersheim2022-02-2021-43/+34
| | | | proton.
* Add a base vespa timer frequency that is used as a base to control poll ↵Henning Baldersheim2022-02-172-2/+2
| | | | frequency of various tasks.
* Remove possibility to update waitTime, maxProcessTime and maxTicksBeforWait ↵Henning Baldersheim2022-02-173-11/+6
| | | | from config.
* Merge pull request #21227 from ↵Tor Brede Vekterli2022-02-161-2/+8
|\ | | | | | | | | vespa-engine/vekterli/fix-data-race-in-persistence-test-mock Fix data race for persistence provider test mock
| * Fix data race for persistence provider test mockTor Brede Vekterli2022-02-161-2/+8
| |
* | Use std::nullopt for empty optionals.Geir Storli2022-02-161-1/+1
| |
* | Tag persistence threads with cpu category WRITE and relevant operations as READ.Geir Storli2022-02-163-4/+20
| |
* | Tag visitor threads with cpu category READ.Geir Storli2022-02-162-2/+2
| |
* | Support setting cpu category on storageframework threads.Geir Storli2022-02-161-1/+1
|/
* Allow inhibiting all default bucket space merges if global merges pendingTor Brede Vekterli2022-02-1514-10/+205
| | | | | | | | | | | | | | | | | | If the cluster controller publishes a cluster state where at least one node is marked as maintenance in the default space but is up in the global space, global bucket merges are pending in the cluster. We want these merges to complete ASAP, and to avoid starvation caused by distributors that finish their global merges quicker than others (or whose global bucket ops get preempted), allow for _completely_ inhibiting merges to buckets in the default space when global buckets are pending. Requires a live config flag to be set to actually be taken into account in the bucket synchronization state checker. Flag is tracked per distributor bucket space (per stripe) and is automatically cleared once a cluster state is received without pending global merges.
* Merge pull request #21149 from ↵Tor Brede Vekterli2022-02-111-0/+6
|\ | | | | | | | | vespa-engine/vekterli/add-distributor-config-for-default-space-merge-inhibition Add config for default bucket space merge inhibition during global merges
| * Add config for default bucket space merge inhibition during global mergesTor Brede Vekterli2022-02-111-0/+6
| | | | | | | | Not wired to anything yet, but allows for feature flag to be created.
* | Move memory allocator argument to be last argument.Tor Egge2022-02-111-1/+1
| |
* | Add memory allocator to array store.Tor Egge2022-02-111-1/+1
|/
* - Generate mode modern c++ code.Henning Baldersheim2022-02-081-1/+1
| | | | | - Use existing predefined types. - Allow for better management by allowing large string vectors to be mmapped.
* Reduce code visibility and include only what you need from config library.Henning Baldersheim2022-02-0633-116/+143
|
* Don't set deprecated fields when aggregating visitor statisticsTor Brede Vekterli2022-01-312-35/+10
| | | | | | | | The 1st/2nd pass functionality has been deprecated for a long time, but unfortunately the documents/bytes visited stats have been wired to be returned as part of 2nd phase statistics instead of the regular higher-level fields. This commit changes this, but the serialization will still have to remain in place until Vespa 8.
* Merge pull request #20980 from ↵Henning Baldersheim2022-01-281-8/+11
|\ | | | | | | | | vespa-engine/geirst/tag-tasks-and-threads-with-cpu-category Tag tasks and threads with cpu category
| * Tag tls and operation response thread pools with cpu category WRITE.Geir Storli2022-01-281-8/+11
| |
* | Avoid assertion failure on aborted node startupTor Brede Vekterli2022-01-281-1/+3
|/ | | | | | | | | | | If a node fails to completely start up due to e.g. failures fetching bootstrap config, it's possible for the storage component chain to have been created but not yet completely initialized. Attempts to gracefully abort startup will still invoke the component `onClose()` methods, at which point the components must be able to handle a partially initialized state. Let `ModifiedBucketChecker` handle that its worker thread was not created when closing, as this will happen when `onOpen()` is never called.
* Support live reconfig of dynamic persistence throttlingTor Brede Vekterli2022-01-261-7/+23
| | | | | | | | Start moving internal `stor-server` throttling config to use struct config instead of separate fields, as this is more flexible and better matches how we configure throttling elsewhere. For now, let dynamic throttling be enabled via both the new and the old config enum. Config model will be updated to use the new config struct shortly.
* Support use of dynamic throttling in MergeThrottler componentTor Brede Vekterli2022-01-244-135/+182
| | | | | | | | | | | | | | | | | | | | | | Now always uses `DynamicThrottlePolicy` instead of `StaticThrottlePolicy`, but by default hardwires the min/max limits to _functionally_ act as if a static policy were in place. If dynamic throttling is enabled, forwarded (unordered) merges are now busy-bounced instead of immediately accepted into the pending window if the window size is exhausted. This so that dynamic throttling windows on nodes earlier in the forwarding chain can be sized down implicitly (thus the _dynamic_ aspect of it all). This is done to help control resource usage and content node queue sizes. Dynamic throttling may be enabled/disabled live via config. If enabled, the following (live) config is supported: * Minimum window size * Maximum window size * Window size increment More may be added (and defaults changed) as appropriate.
* Replace storage operation throttler with vespalib implementationTor Brede Vekterli2022-01-2114-409/+33
| | | | Also move the remaining throttler unit tests to vespalib.
* Merge pull request #20875 from ↵Tor Brede Vekterli2022-01-193-0/+28
|\ | | | | | | | | vespa-engine/vekterli/disallow-none-fieldset-for-visitors Expicitly reject "none"-fieldset for external visitors
| * Expicitly reject "none"-fieldset for external visitorsTor Brede Vekterli2022-01-193-0/+28
| | | | | | | | | | | | | | It does not make sense for a client to run a visitor that inherently cannot return any data at all. Explicitly check for, and reject, `[none]` fieldsets and emit a hopefully helpful error message to use `[id]` instead.
* | Reduce default split size from 32M to 16M.Henning Baldersheim2022-01-181-3/+3
| |