aboutsummaryrefslogtreecommitdiffstats
path: root/storage/src/tests/persistence/filestorage
Commit message (Collapse)AuthorAgeFilesLines
* Add fundamental metrics for async "remove by GID" SPI operationTor Brede Vekterli2023-11-151-0/+8
| | | | | | Tracks invocation count, latency and failures (although we don't expect to see any failures in this pipeline, since the remove ops logically happen atomically with the bucket iteration).
* Add and wire live config for selecting `DeleteBucket` behaviorTor Brede Vekterli2023-11-101-5/+21
| | | | By default the legacy behavior is used.
* Implement DeleteBucket with throttled per-document async removalTor Brede Vekterli2023-11-101-0/+10
| | | | | | | | | | | | | | | | | | | | | Previous (legacy) behavior was to immediately async schedule a full bucket deletion in the persistence backend, which incurs a very disproportionate cost when documents are backed by many and/or heavy indexes (such as HNSW). This risked swamping the backend with tens to hundreds of thousands of concurrent document deletes. New behavior splits deletion into three phases: 1. Metadata enumeration for all documents present in the bucket 2. Persistence-throttled async remove _per individual document_ that was returned in the iteration result. This blocks the persistence thread (by design) if the throttling window is not sufficiently large to accomodate all pending deletes. 3. Once all async removes have been ACKed, schedule the actual `DeleteBucket` operation towards the backend. This will clean up any remaining (cheap) tombstone entries as well as the meta data store. Operation reply is sent as before once the delete has completed.
* Avoid using a reserved identifier naming formatTor Brede Vekterli2023-10-251-50/+50
| | | | | | Identifiers of the form `_Uppercased` are considered reserved by the standard. Not likely to cause ambiguity in practice, but it's preferable to stay on the good side of the standard-gods.
* Rewire `FileStorManager` configTor Brede Vekterli2023-10-241-2/+5
|
* Rewire `ModifiedBucketChecker` configTor Brede Vekterli2023-10-242-12/+16
|
* Update copyrightJon Bratseth2023-10-0913-13/+13
|
* Remove dead code from FileStorManager unit test.Tor Egge2023-08-311-27/+0
|
* GC some unused internal storage message typesTor Brede Vekterli2023-06-161-15/+0
| | | | | Remnants of the "file per bucket on spinning disks" days and no longer used for anything.
* Clean up some todos.Henning Baldersheim2023-06-083-6/+6
|
* Reduce creation of Document instances without DocumentTypeRepo.Geir Storli2023-03-131-6/+6
|
* remove document::RunnableHåvard Pettersen2023-02-201-17/+27
| | | | use std::thread directly instead
* use std::thread directlyHåvard Pettersen2023-02-151-1/+1
| | | | | | also add very simple ThreadPool class to run multiple threads at once make an effort to only join once
* stop using fastos thread more placesHåvard Pettersen2023-02-141-3/+1
| | | | | | | | | | - also stop using std::jthread - remove Active and Joinable interfaces - remove stop, stopped and slumber - remove currentThread - make start function static - override start for Runnable w/init or custom function - explicit stop/slumber where needed
* Change from typedef to using in storage C++ code.Geir Storli2022-12-211-1/+1
|
* Remove stacksize from the thread pools and thread executors.Henning Baldersheim2022-12-201-1/+1
|
* Force content node-internal bucket DB metric update during startupTor Brede Vekterli2022-08-301-0/+1
| | | | | | | | | | | | | | After initialization, the node will immediately start communicating with the cluster controller, exchanging host info. This host info contains a subset snapshot of the active metrics, which includes the total bucket count, doc count etc. It is critical that we must never report back host info _prior_ to having run at least one full sweep of the bucket database, lest we risk transiently reporting zero buckets held by the content node. Doing so could cause orchestration logic to perform operations based on erroneous assumptions. To avoid this, we explicitly force a full DB sweep and metric update prior to reporting the node as up. Since this function is called prior to the CommunicationManager thread being started, any CC health pings should also always happen after this init step.
* Remove '.sum' form vds sum metrics.Henning Baldersheim2022-06-081-6/+6
| | | | | | | | Remove '.sum' from metric names for storage node and also remove the average metrics for the same. Remove '.sum' from distributor metrics set and remove distributor average metrics. GC '.sum' from distributor metric names. Remove '.alldisks' from metric names and update tests. GC '.alldisks' from filestor metrics.
* Fold storageapi into storage.Henning Baldersheim2022-05-191-1/+0
|
* GC unused Context parameterHenning Baldersheim2022-03-312-21/+12
|
* Move BucketIdListResultHenning Baldersheim2022-03-092-2/+2
|
* Reduce use of Identifiable for document::DatatTypeHenning Baldersheim2022-03-031-0/+4
|
* Sync executor to ensure tasks are run prior to stripe teardowns in testsTor Brede Vekterli2022-02-251-2/+11
|
* Make ConfigUri constructors explicit and use same context where possible in ↵Henning Baldersheim2022-02-203-3/+3
| | | | proton.
* Support dynamic throttling of async persistence operationsTor Brede Vekterli2022-01-101-17/+17
| | | | | | | | | | | | | | | | | | | | | | | Adds an operation throttler that is intended to provide global throttling of async operations across all persistence stripe threads. A throttler wraps a logical max pending window size of in-flight operations. Depending on the throttler implementation, the window size may expand and shrink dynamically. Exactly how and when this happens is unspecified. Commit adds two throttler implementations: * An unlimited throttler that is no-op and never blocks. * A throttler built around the mbus `DynamicThrottlePolicy` and defers all window decisions to it. Current config default is to use the unlimited throttler. Config changes require a process restart. Offers both polling and (timed, non-timed) blocking calls for acquiring a throttle token. If the returned token is valid, the caller may proceed to invoke the asynchronous operation. The window slot taken up by a valid throttle token is implicitly freed up when the token is destroyed.
* _executor -> _threadHenning Baldersheim2021-12-091-2/+2
|
* Add init_fun to vespalib::Thread too to figure out what the thread is used for.Henning Baldersheim2021-12-091-1/+3
|
* Actually test maintenance -> down node state transitionTor Brede Vekterli2021-11-241-1/+1
|
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-241-26/+83
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* Revert "Continue serving search queries when in Maintenance node state ↵Henning Baldersheim2021-11-231-83/+26
| | | | [run-systemtest]"
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-231-26/+83
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* Update 2019 Oath copyrights.gjoranv2021-10-271-1/+1
|
* Adjust dummy persistence spi semantics towards proton spi semantics whenTor Egge2021-10-251-4/+4
| | | | | | | bucket doesn't exist: * getBucketInfo() returns success with empty bucket info * createIterator() returns success * iterate() returns empty complete result.
* create/delete bucket will never throw.Henning Baldersheim2021-10-251-2/+2
|
* Async createBucketHenning Baldersheim2021-10-251-2/+2
|
* Only keep async variant to simplify what to implement and what fallback ↵Henning Baldersheim2021-10-181-3/+5
| | | | there are.
* Implement async delete bucket.Henning Baldersheim2021-10-181-2/+2
|
* Update Verizon Media copyright notices.gjoranv2021-10-071-1/+1
|
* Update 2017 copyright notices.gjoranv2021-10-0711-11/+11
|
* Report max address space used in attribute vector components from content ↵Geir Storli2021-08-201-29/+15
| | | | | | | nodes (proton) to the cluster controller. This is more generic than explicit address space values for enum store and multi value. This is used in the cluster controller to determine whether to block external feed.
* - Change error handling so that both synchonous and asynchronous errors can ↵Henning Baldersheim2021-02-231-14/+14
| | | | | | | | be reported back from bucket executor. - Treat remapping as an error. - For lidspace compaction job iterator is reset and will be recreated on next invocation. - For bucketmove th ebucket is rechecked and either discarded or restarted.
* - Reduce visibility of ClusterState and Distribution.Henning Baldersheim2021-02-192-0/+3
|
* use size literals in storageArne Juul2021-02-151-6/+7
|
* Make the noise level used when deciding whether to report resource usage ↵Geir Storli2021-02-041-12/+34
| | | | configurable.
* Revert "Properly track execution of BucketTasks and provide sync() and ↵Henning Baldersheim2021-02-021-34/+0
| | | | order… "
* Properly track execution of BucketTasks and provide sync() and orderly shutdown.Henning Baldersheim2021-02-021-0/+34
|
* Revert "Implement BucketExecutor::sync."Henning Baldersheim2021-02-021-34/+0
|
* Use conditional notify instead of sleep.Henning Baldersheim2021-02-011-0/+1
|
* Implement BucketExecutor::sync.Henning Baldersheim2021-02-011-0/+33
|
* Wire reporting of attribute resource usage all the way to the cluster ↵Geir Storli2021-01-291-0/+22
| | | | controller via the host info API.