summaryrefslogtreecommitdiffstats
path: root/storage
Commit message (Collapse)AuthorAgeFilesLines
* Use emplace_backHenning Baldersheim2022-10-183-14/+12
|
* Use std::make_shared for StateCheckers.Henning Baldersheim2022-10-186-197/+119
|
* Rename trimHoldLists() -> reclaim_memory().Geir Storli2022-10-121-2/+2
|
* Rename transferHoldLists() -> assign_generation().Geir Storli2022-10-121-2/+2
|
* Rename first used generation -> oldest used generation.Geir Storli2022-10-121-1/+1
|
* Update test to handle the change to ↵Henning Baldersheim2022-09-303-3/+9
| | | | enable_metadata_only_fetch_phase_for_inconsistent_updates=true as default.
* Always doing three phase updates now.Henning Baldersheim2022-09-301-1/+1
|
* Remove StateManager handling of deprecated Initializing stateTor Brede Vekterli2022-09-263-58/+32
| | | | | | | Neither distributors nor content nodes ever report their state as Initializing as part of their startup sequence; they go straight from Down to Up. Remove complicated init progress delta reporting that is no longer needed.
* Reduce include and visibility of ptrholder.hHenning Baldersheim2022-09-232-6/+5
|
* Simplify state version checks by requiring exact version matchTor Brede Vekterli2022-09-223-45/+6
| | | | | | | | | | | The existing state unification logic was likely to help ensure that various distributor availability-states were treated as if they were simply Up, but the distributor has not been able to even _be_ in other available states than Up for many years. So it's effectively pointless. Remove unification entirely and instead require both the distributor and content node to be mutually in sync with the exact cluster state version.
* Merge pull request #24158 from ↵Tor Brede Vekterli2022-09-213-28/+109
|\ | | | | | | | | vespa-engine/vekterli/avoid-bucket-db-race-during-cluster-state-transition Avoid bucket DB race during content node cluster state transition [run-systemtest]
| * Avoid bucket DB race during content node cluster state transitionTor Brede Vekterli2022-09-213-28/+109
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It was possible for a distributor bucket fetch request to be processed _after_ a cluster state was enabled (and internally propagated) on the content node, but _before_ all side effects of this enabling were complete and fully visible. This could cause inconsistent information to be returned to the distributor, causing nodes to get out of sync bucket metadata. This commit handles such transition periods by introducing an implicit barrier between observing the incoming command and outgoing reply for a particular cluster state version. Upon observing the reply for a version, all side effects must already be visible since the reply is only sent once internal state processing is complete (both above and below the SPI). Until initiated and completed versions converge, requests are rejected and will be transparently retried by the distributors.
* | Merge pull request #24090 from ↵Henning Baldersheim2022-09-2011-27/+76
|\ \ | | | | | | | | | | | | vespa-engine/vekterli/do-not-inhibit-activation-under-maintenance-mode Do not inhibit bucket replica activation under maintenance mode [run-systemtest]
| * | typo index -> indexing + some minor c++11ification.Henning Baldersheim2022-09-203-16/+15
| | |
| * | Apply suggestions from code review Tor Brede Vekterli2022-09-161-3/+3
| | | | | | | | | | | | | | | Let naming better reflect underlying semantics Co-authored-by: Geir Storli <geirst@yahooinc.com>
| * | Support activation of replicas even if ideal node in maintenanceTor Brede Vekterli2022-09-152-7/+43
| | | | | | | | | | | | | | | | | | | | | Adds an internal feature support flag which communicates that the content node will not implicitly index a non-ideal replica marked explicitly as active. Activation during maintenance is only performed iff a content node has this flag set.
| * | Add internal content node feature for not implicitly indexing active bucketsTor Brede Vekterli2022-09-157-8/+22
| | |
* | | Avoid including smallvector in arrayref, then it is included 'everywhere'.Henning Baldersheim2022-09-161-0/+1
| |/ |/|
* | Allow path prefix alias for legacy content status pagesTor Brede Vekterli2022-09-165-24/+60
|/ | | | | | | | | | | This lets status pages that today are served at the root `/` path be aliased under `/contentnode-status/v1/`. Legacy paths continue working as before. Change existing status page absolute paths to relative to avoid having to care about this particular detail internally. Note: both distributor and search/storage node process status pages use the `/contentnode-status/v1/` prefix, as they're both technically processes that are part of a _logical_ content node.
* rewrite simple perl script in /bin/shArne Juul2022-09-141-13/+7
|
* Update test HTTP header expectationsTor Brede Vekterli2022-09-081-3/+3
|
* Merge pull request #23951 from ↵Henning Baldersheim2022-09-063-88/+46
|\ | | | | | | | | vespa-engine/vekterli/make-gc-work-with-parent-child-with-subset-indexed Make two-phase GC work for parent-child with subset of replicas indexed [run-systemtest]
| * Make two-phase GC work for parent-child with subset of replicas indexedTor Brede Vekterli2022-09-063-88/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous iteration of GC 1st phase candidate set computation required _all_ replicas to agree that a particular document should be removed for it to be passed on to the second phase. I.e. the intersection of all nodes' document sets. This does not work as expected when the GC expression references imported fields _and_ `searchable-copies` is less than `redundancy`, as the required index structures are not present across all replicas. The result was that eligible documents were never removed. This commit changes the candidate set semantics to instead use a union of document IDs, using the maximum observed timestamp in the case of conflicts for the same ID. This mirrors the end result of the legacy behavior, but does not require merging in order to propagate tombstones from the indexed replicas to those without. It also greatly simplifies the candidate computation code.
* | Escape internal cluster states on internal state manager status pageTor Brede Vekterli2022-09-051-5/+6
| |
* | Escape merge command stringification in MergeThrottler status pageTor Brede Vekterli2022-09-051-2/+5
| |
* | Escape relevant fields when rendering internal filestore status pagesTor Brede Vekterli2022-09-052-8/+14
| |
* | Escape relevant fields when rendering internal visitor status pageTor Brede Vekterli2022-09-053-27/+33
|/
* Remove unused methodHarald Musum2022-09-012-25/+0
|
* Force content node-internal bucket DB metric update during startupTor Brede Vekterli2022-08-306-1/+29
| | | | | | | | | | | | | | After initialization, the node will immediately start communicating with the cluster controller, exchanging host info. This host info contains a subset snapshot of the active metrics, which includes the total bucket count, doc count etc. It is critical that we must never report back host info _prior_ to having run at least one full sweep of the bucket database, lest we risk transiently reporting zero buckets held by the content node. Doing so could cause orchestration logic to perform operations based on erroneous assumptions. To avoid this, we explicitly force a full DB sweep and metric update prior to reporting the node as up. Since this function is called prior to the CommunicationManager thread being started, any CC health pings should also always happen after this init step.
* Merge pull request #23832 from ↵Tor Brede Vekterli2022-08-294-20/+64
|\ | | | | | | | | vespa-engine/vekterli/capability-filtering-of-content-status-pages Add capability filtering for content layer status pages and metrics [run-systemtest]
| * Unify to_string for PeerCredentials to match other typesTor Brede Vekterli2022-08-291-1/+1
| |
| * Add capability filtering for content layer status pages and metricsTor Brede Vekterli2022-08-294-20/+64
| | | | | | | | | | This currently only applies to the port exposed for the content node or distributor specific status pages and metrics export, not state V1.
* | remove unused doxygen setup filesArne Juul2022-08-291-994/+0
|/
* Require DocumentAPI capabilities for content node doc API mbus callsTor Brede Vekterli2022-08-251-0/+3
|
* Add factory functions for creating capability requirement filtersTor Brede Vekterli2022-08-232-6/+2
| | | | Refactor existing request access filter creation to use these.
* Merge pull request #23750 from ↵Tor Brede Vekterli2022-08-231-0/+17
|\ | | | | | | | | vespa-engine/vekterli/add-cc-api-server-capability-filter Add capability filter to cluster controller API RPCs on content nodes
| * Add capability filter to cluster controller API RPCs on content nodesTor Brede Vekterli2022-08-231-0/+17
| |
* | Let compiler warn on missing switch cases for enumsTor Brede Vekterli2022-08-232-3/+3
|/ | | | Minor cleanup sweep.
* Merge pull request #23701 from vespa-engine/vekterli/two-phase-document-gcTor Brede Vekterli2022-08-2229-135/+955
|\ | | | | Add support for two-phase document garbage collection [run-systemtest]
| * Add support for two-phase document garbage collectionTor Brede Vekterli2022-08-1729-135/+955
| | | | | | | | | | | | | | | | | | | | | | | | | | | | If enabled, garbage collection is performed in two phases (metadata gathering and deletion) instead of just a single phase. Two-phase GC allows for ensuring the same set of documents is deleted across all nodes and explicitly takes write locks on the distributor to prevent concurrent feed ops to GC'd documents from potentially creating inconsistencies. Two-phase GC is only used _iff_ all replica content nodes support the feature _and_ it's enabled in config. An additional field has been added to the feature negotiation functionality to communicate support from content nodes to distributors.
* | Hide hash implementation details.Henning Baldersheim2022-08-193-4/+4
| |
* | Use xxhash for all BucketId hash sets and maps.Henning Baldersheim2022-08-193-43/+22
|/
* Avoid holding ref to last generated operationTor Brede Vekterli2022-07-191-2/+2
| | | | | Move ref away to avoid an unneeded refcount bump and avoid leaving behind a lingering strong reference to the last generated operation.
* Add capability request access filter to internal StorageAPI RPC methodTor Brede Vekterli2022-07-111-0/+4
| | | | | | | | This should always succeed today, as authz rules by default grant all capabilities. But since this is a very hot call path, we'll learn very quickly if the capability check incurs a measurable overhead; it is not expected to do so in practice (really just a virtual function call and a few bitwise ops).
* Merge pull request #23415 from ↵Henning Baldersheim2022-07-076-11/+11
|\ | | | | | | | | vespa-engine/vekterli/add-separate-id-and-timestamp-wrapper Add wrapper for <doc id, timestamp> tuple and update APIs to use this
| * Add wrapper for <doc id, timestamp> tuple and update APIs to use thisTor Brede Vekterli2022-07-076-11/+11
| | | | | | | | | | Feels more intuitive to have a tuple that implies "document foo at timestamp bar" rather than the current inverse of "timestamp bar with document foo".
* | Add back dispatch_on_encode and dispatch_on_decode that still has overridesHenning Baldersheim2022-07-071-0/+10
|/
* GC unused numThreads too.Henning Baldersheim2022-07-072-5/+0
|
* GC temporary dispatch flags.Henning Baldersheim2022-07-071-9/+0
|
* Never dispatch to network helper threads.Henning Baldersheim2022-07-061-2/+0
|