aboutsummaryrefslogtreecommitdiffstats
path: root/storage/src/tests/persistence
Commit message (Collapse)AuthorAgeFilesLines
* Avoid using a reserved identifier naming formatTor Brede Vekterli2023-10-252-52/+52
| | | | | | Identifiers of the form `_Uppercased` are considered reserved by the standard. Not likely to cause ambiguity in practice, but it's preferable to stay on the good side of the standard-gods.
* Rewire `FileStorManager` configTor Brede Vekterli2023-10-242-9/+16
|
* Rewire `ModifiedBucketChecker` configTor Brede Vekterli2023-10-242-12/+16
|
* Update copyrightJon Bratseth2023-10-0933-33/+33
|
* Remove dead code from FileStorManager unit test.Tor Egge2023-08-311-27/+0
|
* GC some unused internal storage message typesTor Brede Vekterli2023-06-161-15/+0
| | | | | Remnants of the "file per bucket on spinning disks" days and no longer used for anything.
* Clean up some todos.Henning Baldersheim2023-06-085-8/+8
|
* propagate create flag to test-and-set helperHåvard Pettersen2023-05-031-0/+22
|
* Add backend support for distributed condition evaluationTor Brede Vekterli2023-04-182-5/+88
| | | | | | | | | | | | Lets the "test" part of a test-and-set condition be evaluated locally on individual content nodes. Piggybacks on top of metadata-only Get operations, adding a new condition field to the request and a boolean match result to the response. Decouples the existing TaS utility code from being command-oriented, allowing it to be used in other contexts as well. Not yet wired through any protocols.
* Reduce creation of Document instances without DocumentTypeRepo.Geir Storli2023-03-131-6/+6
|
* remove document::RunnableHåvard Pettersen2023-02-201-17/+27
| | | | use std::thread directly instead
* use std::thread directlyHåvard Pettersen2023-02-151-1/+1
| | | | | | also add very simple ThreadPool class to run multiple threads at once make an effort to only join once
* stop using fastos thread more placesHåvard Pettersen2023-02-141-3/+1
| | | | | | | | | | - also stop using std::jthread - remove Active and Joinable interfaces - remove stop, stopped and slumber - remove currentThread - make start function static - override start for Runnable w/init or custom function - explicit stop/slumber where needed
* Remove redundant or pessimizing moves.Tor Egge2023-01-281-1/+1
|
* Change from typedef to using in storage C++ code.Geir Storli2022-12-213-8/+8
|
* Remove stacksize from the thread pools and thread executors.Henning Baldersheim2022-12-201-1/+1
|
* Improve error response when trying to use an imported field in a conditionTor Brede Vekterli2022-12-081-0/+12
| | | | | | | | | | | We don't support using imported fields in conditional mutations, so catch attempts at doing this during the field enumeration that is done as part of the condition evaluation. Would previously get an internal error response with an ugly stack trace since the exception would propagate up to a generic exception-to-response handler. Will now generate an `ILLEGAL_PARAMETERS` error response with a hopefully helpful error message.
* Force content node-internal bucket DB metric update during startupTor Brede Vekterli2022-08-301-0/+1
| | | | | | | | | | | | | | After initialization, the node will immediately start communicating with the cluster controller, exchanging host info. This host info contains a subset snapshot of the active metrics, which includes the total bucket count, doc count etc. It is critical that we must never report back host info _prior_ to having run at least one full sweep of the bucket database, lest we risk transiently reporting zero buckets held by the content node. Doing so could cause orchestration logic to perform operations based on erroneous assumptions. To avoid this, we explicitly force a full DB sweep and metric update prior to reporting the node as up. Since this function is called prior to the CommunicationManager thread being started, any CC health pings should also always happen after this init step.
* Add support for two-phase document garbage collectionTor Brede Vekterli2022-08-171-2/+92
| | | | | | | | | | | | | | If enabled, garbage collection is performed in two phases (metadata gathering and deletion) instead of just a single phase. Two-phase GC allows for ensuring the same set of documents is deleted across all nodes and explicitly takes write locks on the distributor to prevent concurrent feed ops to GC'd documents from potentially creating inconsistencies. Two-phase GC is only used _iff_ all replica content nodes support the feature _and_ it's enabled in config. An additional field has been added to the feature negotiation functionality to communicate support from content nodes to distributors.
* Add wrapper for <doc id, timestamp> tuple and update APIs to use thisTor Brede Vekterli2022-07-072-4/+4
| | | | | Feels more intuitive to have a tuple that implies "document foo at timestamp bar" rather than the current inverse of "timestamp bar with document foo".
* Remove '.sum' form vds sum metrics.Henning Baldersheim2022-06-083-8/+8
| | | | | | | | Remove '.sum' from metric names for storage node and also remove the average metrics for the same. Remove '.sum' from distributor metrics set and remove distributor average metrics. GC '.sum' from distributor metric names. Remove '.alldisks' from metric names and update tests. GC '.alldisks' from filestor metrics.
* Remove most use of vespalib::rmdirTor Egge2022-06-031-3/+3
|
* Fold storageapi into storage.Henning Baldersheim2022-05-191-1/+0
|
* Remove disabled unit test that triggered compiler warningsTor Egge2022-05-161-42/+0
| | | | (gcc 12 on aarch64 platform).
* GC unused Context parameterHenning Baldersheim2022-03-319-160/+98
|
* Remove copy constructors.Henning Baldersheim2022-03-283-57/+29
|
* Use both lvalue and rvalue specifier to avoid explicit std::move()Henning Baldersheim2022-03-282-3/+3
|
* Avoid the need for clone by using unique_ptr.Henning Baldersheim2022-03-282-5/+3
|
* Avoid need to copy/clone FieldUpdateHenning Baldersheim2022-03-272-10/+3
|
* Move BucketIdListResultHenning Baldersheim2022-03-092-2/+2
|
* Reduce use of Identifiable for document::DatatTypeHenning Baldersheim2022-03-034-1/+7
|
* Sync executor to ensure tasks are run prior to stripe teardowns in testsTor Brede Vekterli2022-02-251-2/+11
|
* Make ConfigUri constructors explicit and use same context where possible in ↵Henning Baldersheim2022-02-204-4/+4
| | | | proton.
* Fix data race for persistence provider test mockTor Brede Vekterli2022-02-161-2/+8
|
* Replace storage operation throttler with vespalib implementationTor Brede Vekterli2022-01-212-117/+0
| | | | Also move the remaining throttler unit tests to vespalib.
* Wire in control of whether taskLimit is hard.Henning Baldersheim2022-01-181-1/+1
|
* Remove sync apply bucket diff.Tor Egge2022-01-181-51/+34
|
* Support dynamic throttling of async persistence operationsTor Brede Vekterli2022-01-105-39/+156
| | | | | | | | | | | | | | | | | | | | | | | Adds an operation throttler that is intended to provide global throttling of async operations across all persistence stripe threads. A throttler wraps a logical max pending window size of in-flight operations. Depending on the throttler implementation, the window size may expand and shrink dynamically. Exactly how and when this happens is unspecified. Commit adds two throttler implementations: * An unlimited throttler that is no-op and never blocks. * A throttler built around the mbus `DynamicThrottlePolicy` and defers all window decisions to it. Current config default is to use the unlimited throttler. Config changes require a process restart. Offers both polling and (timed, non-timed) blocking calls for acquiring a throttle token. If the returned token is valid, the caller may proceed to invoke the asynchronous operation. The window slot taken up by a valid throttle token is implicitly freed up when the token is destroyed.
* Revert "Revert "Balder/refactor docentry""Henning Baldersheim2022-01-072-11/+14
|
* Revert "Balder/refactor docentry"Arnstein Ressem2022-01-072-14/+11
|
* - Flags -> Enum.Henning Baldersheim2022-01-062-10/+12
| | | | - Consistently use DocEntryList as type for std::vector<spi::DocEntry::UP>
* Use enum class for the flags.Henning Baldersheim2022-01-061-4/+4
|
* Simplify DocEntry to get a clean interface with multiple implementations, ↵Henning Baldersheim2022-01-061-0/+1
| | | | | | instead of an mutant. Also add tests for the different variations a DocEntry can have.
* _executor -> _threadHenning Baldersheim2021-12-091-2/+2
|
* Add init_fun to vespalib::Thread too to figure out what the thread is used for.Henning Baldersheim2021-12-091-1/+3
|
* Decrement persistence thread merge counter when syncronous processing is ↵Tor Brede Vekterli2021-12-031-0/+4
| | | | | | | | | | | | | | | | | | complete Add a generic interface for letting an operation know that the synchronous parts of its processing in the persistence thread is complete. This allows a potentially longer-running async operation to free up any limits that were put in place when it was taking up synchronous thread resources. Currently only used by merge-related operations (that may dispatch many async ops). Since we have a max upper bound for how many threads in a stripe may be processing merge ops at the same time (to avoid blocking client ops), we previously could effectively stall the pipelining of merges caused by hitting the concurrency limit even if all persistence threads were otherwise idle (waiting for prior async merge ops to complete). We now explicitly decrease the merge concurrency counter once the synchronous processing is done, allowing us to take on further merges immediately.
* Add metrics for active operations on service layer.Tor Egge2021-12-012-0/+151
|
* Update operation metrics for delayed or chained merge handler replies.Tor Egge2021-11-241-2/+76
|
* Actually test maintenance -> down node state transitionTor Brede Vekterli2021-11-241-1/+1
|
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-241-26/+83
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.