aboutsummaryrefslogtreecommitdiffstats
path: root/storage/src/tests/storageserver
Commit message (Collapse)AuthorAgeFilesLines
* Fix typoTor Brede Vekterli8 days1-1/+1
|
* Adjust cluster state node count to avoid false negatives in testTor Brede Vekterli8 days1-1/+1
| | | | | | Cluster state should state N nodes and distribution config should state <N nodes in order to properly check that attempting to use node N fails due to the _config_ and not the _state_.
* Handle cluster state bundle with distribution config on content nodeTor Brede Vekterli9 days2-12/+165
| | | | | | | | | | | | | If a cluster state bundle contains distribution config, this is internally propagated via the `StateManager` component to all registered state listeners. One such state listener is `FileStorManager`, which updates the content node-internal bucket space repository. All `SetSystemStateCommand` and internal config-aware components (`StateManager` and `ChangedBucketOwnershipHandler`) now explicitly track whether the cluster controller provides distribution config, or if the internally provided config should be used (including fallback to internal config if necessary).
* Merge pull request #31675 from ↵Henning Baldersheim9 days1-1/+11
|\ | | | | | | | | vespa-engine/vekterli/decode-cluster-state-bundle-distribution-config-cpp Support decoding distribution config as part of cluster state bundles in C++
| * Support decoding distribution config as part of cluster state bundles in C++Tor Brede Vekterli12 days1-1/+11
| | | | | | | | | | | | Actually _encoding_ config in the same format as that used for decoding config payloads is not directly supported, so we do our own roundabout conversion as part of testing.
* | Adjust storage server document api converter unit test forTor Egge10 days1-1/+2
|/ | | | out of source builds.
* Rename storage library to vespa_storage.Tor Egge13 days2-3/+3
|
* Support enforcing strictly increasing state versions across cluster controllersTor Brede Vekterli2024-06-031-28/+98
| | | | | | | | | | | | | | | | | Adds a (live) config that specifies if content nodes and distributors shall reject cluster state versions that are lower than the one that is currently active on the node. This prevents "last write wins" ordering problems when multiple cluster controllers have partially overlapping leadership periods. In the name of pragmatism, we try to auto-detect the case where ZooKeeper state must have been lost on the cluster controller cluster, and accept the state even with lower version number. Otherwise, the content cluster would be effectively stalled until all its processes had been manually restarted. Adds wiring of live config to the `StateManager` component on both content and distributor nodes.
* Remove usages of deprecated DirConfig in storage unit testsTor Brede Vekterli2024-05-159-89/+100
| | | | | | Introduce a distinct `StorageConfigSet` which wraps the actual underlying config objects and exposes them through a unified `ConfigUri`.
* Remove legacy storage node root directory IO in unit testsTor Brede Vekterli2024-05-141-1/+0
| | | | | | | | | | Once upon a time, VDS roamed the lands. It used real disk IO as part of tests. Then came the meteor and in-memory dummy persistence took over. Now it is time for the fossils to be moved into a museum where they belong. Also make PID file writing conditional on a config that is set to `false` during unit testing (but `true` as default).
* Propagate "create if missing"-flag outside binary Update payload in protocolsTor Brede Vekterli2024-04-261-22/+40
| | | | | | | | | | | | | | | | | | | | Avoids potentially having to deserialize the entire update just to get to a single bit of information that is technically metadata existing orthogonally to the document update itself. To ensure backwards/forwards compatibility, the flag is propagated as a Protobuf `enum` where the default value is a special "unspecified" sentinel, implying an old sender. Since the Java protocol implementation always eagerly deserializes messages, it unconditionally assigns the `create_if_missing` field when sending and completely ignores it when receiving. The C++ protocol implementation observes and propagates the field iff set. Otherwise the flag is deferred to the update object as before. This applies to both the DocumentAPI and StorageAPI protocols.
* disable_queue_limits_for_chained_merges has long been true, GCHenning Baldersheim2024-02-021-14/+5
|
* GC void config from stor-bouncer.defHenning Baldersheim2024-02-011-65/+11
|
* Also memory limit throttle enqueued mergesTor Brede Vekterli2023-11-131-0/+33
| | | | | | | | | | | | | | This plugs the hole where merges could enter the active window even if doing so would exceeded the total memory limit, as dequeueing is a separate code path from when a merge is initially evaluated for inclusion in the active window. There is a theoretical head-of-line blocking/queue starvation issue if the merge at the front of the queue has an unrealistically large footprint and the memory limit is unrealistically low. In practice this is not expected to be a problem, and it should never cause merging to stop (at least one merge is always guaranteed to be able to execute). As such, not adding any kind of heuristics to deal with this for now.
* Wire HwInfo into MergeThrottler and use for auto-deduction of memory limitsTor Brede Vekterli2023-11-022-27/+105
| | | | | | | | Add config for min/max capping of deduced limit, as well as a scaling factor based on the memory available to the process. Defaults have been chosen based on empirical observations over many years, but having these as config means we can tune things live if it should ever be required.
* Add content node soft limit on max memory used by mergesTor Brede Vekterli2023-11-011-22/+138
| | | | | | | | | If configured, the active merge window is limited so that the sum of estimated memory usage for its merges does not go beyond the configured soft memory limit. The window can always fit a minimum of 1 merge regardless of its size to ensure progress in the cluster (thus this is a soft limit, not a hard limit).
* Simplify and reuse utility config functionTor Brede Vekterli2023-10-241-11/+7
|
* Pull up and out config of `ChangedBucketOwnershipHandler` componentTor Brede Vekterli2023-10-243-23/+19
|
* Wire config to MergeThrottler in from the outsideTor Brede Vekterli2023-10-242-13/+29
|
* Rewire Bouncer configuration flowTor Brede Vekterli2023-10-191-14/+19
| | | | | | | | | Removes own `ConfigFetcher` in favor of pushing reconfiguration responsibilities onto the components owning the Bouncer instance. The current "superclass calls into subclass" approach isn't ideal, but the longer term plan is to hoist all config subscriptions out of `StorageNode` and into the higher-level `Process` structure.
* Remove unused configurability of operation prioritiesTor Brede Vekterli2023-10-173-6/+6
| | | | | | | | | | | | | | | | | | As far as I know, this config has not been used by anyone for at least a decade (if it ever was used for anything truly useful). Additionally, operation priorities are a foot-gun at the best of times. The ability to dynamically change the meaning of priority enums even more so. This commit entirely removes configuration of Document API priority mappings in favor of a fixed mapping that is equal to the default config, i.e. what everyone's been using anyway. This removes a thread per distributor/storage node process as well as 1 mutex and 1 (presumably entirely unneeded `seq_cst`) atomic load in the message hot path. Also precomputes a LUT for the priority reverse mapping to avoid needing to lower-bound seek an explicit map.
* Wire `CommunicationManager` config from its owner rather than self-subscribingTor Brede Vekterli2023-10-161-40/+48
| | | | | | | | This moves the responsibility for bootstrapping and updating config for the `CommunicationManager` component to its owner. By doing this, a dedicated `ConfigFetcher` can be removed. Since this is a component used by both the distributor and storage nodes, this reduces total thread count by 2 on a host.
* Ensure internal messages are flushed before shutting down RPC subsystemTor Brede Vekterli2023-10-111-7/+36
| | | | | | | | | | This moves RPC shutdown from being the _first_ thing that happens to being the _last_ thing that happens during storage chain shutdown. To avoid concurrent client requests from the outside reaching internal components during the flushing phases, the Bouncer component will now explicitly and immediately reject incoming RPCs after closing and all replies will be silently swallowed (no one is listening for them at that point anyway).
* Revert "Ensure internal messages are flushed before shutting down RPC ↵Harald Musum2023-10-111-36/+7
| | | | subsystem, take 2"
* Ensure internal messages are flushed before shutting down RPC subsystemTor Brede Vekterli2023-10-111-7/+36
| | | | | | | | | | This moves RPC shutdown from being the _first_ thing that happens to being the _last_ thing that happens during storage chain shutdown. To avoid concurrent client requests from the outside reaching internal components during the flushing phases, the Bouncer component will now explicitly and immediately reject incoming RPCs after closing and all replies will be silently swallowed (no one is listening for them at that point anyway).
* Revert "Ensure internal messages are flushed before shutting down RPC subsystem"Tor Brede Vekterli2023-10-101-36/+7
|
* Merge pull request #28825 from ↵Henning Baldersheim2023-10-101-7/+36
|\ | | | | | | | | vespa-engine/vekterli/ensure-internal-messages-flushed-prior-to-rpc-shutdown Ensure internal messages are flushed before shutting down RPC subsystem
| * Ensure internal messages are flushed before shutting down RPC subsystemTor Brede Vekterli2023-10-061-7/+36
| | | | | | | | | | | | | | | | | | | | This moves RPC shutdown from being the _first_ thing that happens to being the _last_ thing that happens during storage chain shutdown. To avoid concurrent client requests from the outside reaching internal components during the flushing phases, the Bouncer component will now explicitly and immediately reject incoming RPCs after closing and all replies will be silently swallowed (no one is listening for them at that point anyway).
* | Update copyrightJon Bratseth2023-10-0920-20/+20
|/
* Remove remaining deprecated "revert" functionality from code baseTor Brede Vekterli2023-09-081-6/+0
| | | | | | Serialization code can safely be removed, as no revert-related messages have ever flown across the wire in the new serialization format.
* Let node info for cluster controller be explicit, and not a metric consumer.Henning Baldersheim2023-08-291-8/+3
|
* Ensure executor is synced after shutdownTor Brede Vekterli2023-06-051-1/+1
| | | | Also clarify/update some comments.
* Avoid blocking CommunicationManager thread during cluster state transitionsTor Brede Vekterli2023-06-021-32/+25
| | | | | | | | | | | | | | | | Incoming cluster state versions are not applied locally on a content node until all potentially conflicting operations running in the persistence threads have completed and all potentially conflicting operations in the persistence queues have been aborted. This can take a relatively long time when running LID space compactions etc via the persistence threads, and we'd risk blocking the main CommunicationManager thread (which handles all cluster controller communication) for prolonged periods of time. Move state blocking and internal state propagation to a dedicated task executor. The executor only has 1 thread, effectively turning it into an asynchronous FIFO executor.
* wire create flag from document api to storage apiHåvard Pettersen2023-05-081-0/+24
|
* Add capability checking to state API handlersTor Brede Vekterli2023-03-221-1/+1
| | | | | | | | | | | | | | | | | | This covers both the entry points from the `storagenode` and `searchnode` HTTP servers, though the former is mostly in the name of legacy support. Ideally, capability checking would exist as a property of the HTTP server (Portal) bindings, but the abstractions for the JSON request handling are sufficiently leaky that it ended up making more sense to push things further down the hierarchy. It's always a good thing to move away from using strings with implicit semantics as return types anyway. The `searchnode` state API handler mapping supports fine grained capabilities. The legacy `storagenode` state API forwarding does not; it uses a sledgehammer that expects the union of all possible API capability requirements.
* Reduce creation of Document instances without DocumentTypeRepo.Geir Storli2023-03-132-3/+3
|
* use ref_counted in fnetHåvard Pettersen2023-03-062-3/+3
| | | | | | also get rid of some cleanup functions on reference counted classes enable specifying low-level parameters to addref/subref (cnt/reserve)
* typesafe getLastProcessedTime tooHenning Baldersheim2023-03-011-1/+1
|
* Ensure we stay with system time until limitation has been lifted in a type ↵Henning Baldersheim2023-02-281-1/+1
| | | | safe way.
* Unify on steady timeHenning Baldersheim2023-02-281-1/+1
|
* Use vespalib::steady_time for getMilliSecTime to ensure no wraps around and ↵Henning Baldersheim2023-02-281-27/+11
| | | | safer code.
* avoid using fastos thread in searchcoreHåvard Pettersen2023-02-271-1/+0
| | | | also remove some left-behind includes
* Merge pull request #26175 from ↵Geir Storli2023-02-241-3/+1
|\ | | | | | | | | vespa-engine/havardpe/avoid-fastos-thread-in-storage avoid using fastos thread in storage
| * avoid using fastos thread in storageHåvard Pettersen2023-02-241-3/+1
| |
* | untangle messagebus from fastosHåvard Pettersen2023-02-241-0/+1
|/
* remove document::RunnableHåvard Pettersen2023-02-201-1/+1
| | | | use std::thread directly instead
* Reduce use of getSeconds further.Henning Baldersheim2023-02-081-3/+1
|
* Merge pull request #25887 from vespa-engine/balder/code-health-1Henning Baldersheim2023-02-071-1/+0
|\ | | | | General code healt, nodiscard, range loops etc
| * General code healt, nodiscard, range loops etcHenning Baldersheim2023-02-061-1/+0
| |
* | Use std::chrono based time for bucket manager time-to-work check.Henning Baldersheim2023-02-072-10/+7
| |