summaryrefslogtreecommitdiffstats
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* Misc cleanup and formatting of distributor codeTor Brede Vekterli2022-12-15100-675/+454
| | | | No functional changes
* Move MessageGuard out to storage/common sub-libraryTor Brede Vekterli2022-12-158-48/+60
|
* Add MBus trace entry when no replicas are found for a document bucketTor Brede Vekterli2022-12-131-7/+14
|
* Merge pull request #25199 from ↵Tor Brede Vekterli2022-12-091-1/+3
|\ | | | | | | | | vespa-engine/vekterli/propagate-update-internal-doc-repo-to-auto-created-doc Propagate doc repo in update to auto-created blank document instance [run-systemtest]
| * Propagate doc repo in update to auto-created blank document instanceTor Brede Vekterli2022-12-091-1/+3
| |
* | Avoid nullptr memcpy argument in protocol serialization.Tor Egge2022-12-091-2/+6
|/
* Improve error response when trying to use an imported field in a conditionTor Brede Vekterli2022-12-082-1/+21
| | | | | | | | | | | We don't support using imported fields in conditional mutations, so catch attempts at doing this during the field enumeration that is done as part of the condition evaluation. Would previously get an internal error response with an ugly stack trace since the exception would propagate up to a generic exception-to-response handler. Will now generate an `ILLEGAL_PARAMETERS` error response with a hopefully helpful error message.
* Add metric tracking internal `StateManager` state propagation latencyTor Brede Vekterli2022-12-082-0/+24
| | | | | | | | Adds a new top-level metric set with the single metric ``` vds.state_manager.invoke_state_listeners_latency ``` which tracks the latency for invoking the set of all state listeners.
* Limit to 1 response thread in test configTor Brede Vekterli2022-12-011-0/+1
| | | | Avoids any potential surprises caused by reply reordering in tests.
* GC old config valuesTor Brede Vekterli2022-12-011-40/+0
|
* Defer MBus DestinationSession registration during content node initTor Brede Vekterli2022-11-291-0/+9
| | | | | | | | | | | | | Creating a `DestinationSession` that is immediately registered as available for business means we may theoretically start receiving messages over the session even before the call returns to the caller. Either way there would be no memory barrier that ensures that `_messageBusSession` would be fully visible to the MessageBus threads (since it's written after return). To avoid this sneaky scenario, defer registration (and thus introduce a barrier) until _after_ we've initialized our internal member variables. This addresses a TSan warning.
* Be more explicit about previous GC timestamp in testTor Brede Vekterli2022-11-281-0/+3
|
* Avoid mass pending GC on config enable edgeTor Brede Vekterli2022-11-285-6/+136
| | | | | | | | | | | | If a system is running in a stable state with no GC enabled, per-bucket last GC timestamps in the DB will end up further and further in the past. If GC is then enabled in config, we must ensure that GC timestamps are reset to the current time to avoid suddenly ending up with _every single_ bucket having exceeded its GC deadline, causing pending GC en masse. Resetting is edge-triggered, so it should not happen if GC is enabled in both the old and new configs.
* Merge pull request #24999 from ↵Henning Baldersheim2022-11-258-9/+9
|\ | | | | | | | | vespa-engine/vekterli/make-internal-state-propagation-noexcept Make internal cluster/node state propagation noexcept
| * Make internal cluster/node state propagation noexceptTor Brede Vekterli2022-11-258-9/+9
| | | | | | | | | | It is not well defined what to do if an implementation of the propagation callback throws, so make it noexcept to core the process if it does happen.
* | Add shared constant for PAGE_SIZE.Geir Storli2022-11-251-2/+4
|/
* Drop support for old gtest.Tor Egge2022-11-184-5/+5
|
* Use emplace_backHenning Baldersheim2022-10-183-14/+12
|
* Use std::make_shared for StateCheckers.Henning Baldersheim2022-10-186-197/+119
|
* Rename trimHoldLists() -> reclaim_memory().Geir Storli2022-10-121-2/+2
|
* Rename transferHoldLists() -> assign_generation().Geir Storli2022-10-121-2/+2
|
* Rename first used generation -> oldest used generation.Geir Storli2022-10-121-1/+1
|
* Update test to handle the change to ↵Henning Baldersheim2022-09-303-3/+9
| | | | enable_metadata_only_fetch_phase_for_inconsistent_updates=true as default.
* Always doing three phase updates now.Henning Baldersheim2022-09-301-1/+1
|
* Remove StateManager handling of deprecated Initializing stateTor Brede Vekterli2022-09-263-58/+32
| | | | | | | Neither distributors nor content nodes ever report their state as Initializing as part of their startup sequence; they go straight from Down to Up. Remove complicated init progress delta reporting that is no longer needed.
* Reduce include and visibility of ptrholder.hHenning Baldersheim2022-09-232-6/+5
|
* Simplify state version checks by requiring exact version matchTor Brede Vekterli2022-09-223-45/+6
| | | | | | | | | | | The existing state unification logic was likely to help ensure that various distributor availability-states were treated as if they were simply Up, but the distributor has not been able to even _be_ in other available states than Up for many years. So it's effectively pointless. Remove unification entirely and instead require both the distributor and content node to be mutually in sync with the exact cluster state version.
* Merge pull request #24158 from ↵Tor Brede Vekterli2022-09-213-28/+109
|\ | | | | | | | | vespa-engine/vekterli/avoid-bucket-db-race-during-cluster-state-transition Avoid bucket DB race during content node cluster state transition [run-systemtest]
| * Avoid bucket DB race during content node cluster state transitionTor Brede Vekterli2022-09-213-28/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was possible for a distributor bucket fetch request to be processed _after_ a cluster state was enabled (and internally propagated) on the content node, but _before_ all side effects of this enabling were complete and fully visible. This could cause inconsistent information to be returned to the distributor, causing nodes to get out of sync bucket metadata. This commit handles such transition periods by introducing an implicit barrier between observing the incoming command and outgoing reply for a particular cluster state version. Upon observing the reply for a version, all side effects must already be visible since the reply is only sent once internal state processing is complete (both above and below the SPI). Until initiated and completed versions converge, requests are rejected and will be transparently retried by the distributors.
* | Merge pull request #24090 from ↵Henning Baldersheim2022-09-2011-27/+76
|\ \ | | | | | | | | | | | | vespa-engine/vekterli/do-not-inhibit-activation-under-maintenance-mode Do not inhibit bucket replica activation under maintenance mode [run-systemtest]
| * | typo index -> indexing + some minor c++11ification.Henning Baldersheim2022-09-203-16/+15
| | |
| * | Apply suggestions from code review Tor Brede Vekterli2022-09-161-3/+3
| | | | | | | | | | | | | | | Let naming better reflect underlying semantics Co-authored-by: Geir Storli <geirst@yahooinc.com>
| * | Support activation of replicas even if ideal node in maintenanceTor Brede Vekterli2022-09-152-7/+43
| | | | | | | | | | | | | | | | | | | | | Adds an internal feature support flag which communicates that the content node will not implicitly index a non-ideal replica marked explicitly as active. Activation during maintenance is only performed iff a content node has this flag set.
| * | Add internal content node feature for not implicitly indexing active bucketsTor Brede Vekterli2022-09-157-8/+22
| | |
* | | Avoid including smallvector in arrayref, then it is included 'everywhere'.Henning Baldersheim2022-09-161-0/+1
| |/ |/|
* | Allow path prefix alias for legacy content status pagesTor Brede Vekterli2022-09-165-24/+60
|/ | | | | | | | | | | This lets status pages that today are served at the root `/` path be aliased under `/contentnode-status/v1/`. Legacy paths continue working as before. Change existing status page absolute paths to relative to avoid having to care about this particular detail internally. Note: both distributor and search/storage node process status pages use the `/contentnode-status/v1/` prefix, as they're both technically processes that are part of a _logical_ content node.
* rewrite simple perl script in /bin/shArne Juul2022-09-141-13/+7
|
* Update test HTTP header expectationsTor Brede Vekterli2022-09-081-3/+3
|
* Merge pull request #23951 from ↵Henning Baldersheim2022-09-063-88/+46
|\ | | | | | | | | vespa-engine/vekterli/make-gc-work-with-parent-child-with-subset-indexed Make two-phase GC work for parent-child with subset of replicas indexed [run-systemtest]
| * Make two-phase GC work for parent-child with subset of replicas indexedTor Brede Vekterli2022-09-063-88/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous iteration of GC 1st phase candidate set computation required _all_ replicas to agree that a particular document should be removed for it to be passed on to the second phase. I.e. the intersection of all nodes' document sets. This does not work as expected when the GC expression references imported fields _and_ `searchable-copies` is less than `redundancy`, as the required index structures are not present across all replicas. The result was that eligible documents were never removed. This commit changes the candidate set semantics to instead use a union of document IDs, using the maximum observed timestamp in the case of conflicts for the same ID. This mirrors the end result of the legacy behavior, but does not require merging in order to propagate tombstones from the indexed replicas to those without. It also greatly simplifies the candidate computation code.
* | Escape internal cluster states on internal state manager status pageTor Brede Vekterli2022-09-051-5/+6
| |
* | Escape merge command stringification in MergeThrottler status pageTor Brede Vekterli2022-09-051-2/+5
| |
* | Escape relevant fields when rendering internal filestore status pagesTor Brede Vekterli2022-09-052-8/+14
| |
* | Escape relevant fields when rendering internal visitor status pageTor Brede Vekterli2022-09-053-27/+33
|/
* Remove unused methodHarald Musum2022-09-012-25/+0
|
* Force content node-internal bucket DB metric update during startupTor Brede Vekterli2022-08-306-1/+29
| | | | | | | | | | | | | | After initialization, the node will immediately start communicating with the cluster controller, exchanging host info. This host info contains a subset snapshot of the active metrics, which includes the total bucket count, doc count etc. It is critical that we must never report back host info _prior_ to having run at least one full sweep of the bucket database, lest we risk transiently reporting zero buckets held by the content node. Doing so could cause orchestration logic to perform operations based on erroneous assumptions. To avoid this, we explicitly force a full DB sweep and metric update prior to reporting the node as up. Since this function is called prior to the CommunicationManager thread being started, any CC health pings should also always happen after this init step.
* Merge pull request #23832 from ↵Tor Brede Vekterli2022-08-294-20/+64
|\ | | | | | | | | vespa-engine/vekterli/capability-filtering-of-content-status-pages Add capability filtering for content layer status pages and metrics [run-systemtest]
| * Unify to_string for PeerCredentials to match other typesTor Brede Vekterli2022-08-291-1/+1
| |
| * Add capability filtering for content layer status pages and metricsTor Brede Vekterli2022-08-294-20/+64
| | | | | | | | | | This currently only applies to the port exposed for the content node or distributor specific status pages and metrics export, not state V1.
* | remove unused doxygen setup filesArne Juul2022-08-291-994/+0
|/