summaryrefslogtreecommitdiffstats
path: root/persistence
Commit message (Collapse)AuthorAgeFilesLines
* Revert "Revert "Unify access to assets needed during rank-setup.""Henning Baldersheim2022-09-071-2/+2
|
* Revert "Unify access to assets needed during rank-setup."Tor Egge2022-09-071-2/+2
|
* Unify access to assets needed during rank-setup.Henning Baldersheim2022-09-061-2/+2
|
* remove unused doxygen setup filesArne Juul2022-08-291-994/+0
|
* Add support for two-phase document garbage collectionTor Brede Vekterli2022-08-172-0/+30
| | | | | | | | | | | | | | If enabled, garbage collection is performed in two phases (metadata gathering and deletion) instead of just a single phase. Two-phase GC allows for ensuring the same set of documents is deleted across all nodes and explicitly takes write locks on the distributor to prevent concurrent feed ops to GC'd documents from potentially creating inconsistencies. Two-phase GC is only used _iff_ all replica content nodes support the feature _and_ it's enabled in config. An additional field has been added to the feature negotiation functionality to communicate support from content nodes to distributors.
* Don't add tombstone in dummy persistence when a newer entry already existsTor Brede Vekterli2022-08-172-5/+52
| | | | | | | This better mirrors how Proton actually works, since it's not a multi version store. Since only the highest timestamped entry for a document is the one that is ever considered on a node, there's no point in storing an explicit tombstone that cannot be referenced.
* Add wrapper for <doc id, timestamp> tuple and update APIs to use thisTor Brede Vekterli2022-07-079-12/+69
| | | | | Feels more intuitive to have a tuple that implies "document foo at timestamp bar" rather than the current inverse of "timestamp bar with document foo".
* Collapse persistencetypes into persistenceHenning Baldersheim2022-05-184-1/+121
|
* - Move persitence/spi/types.h under to persitence/spi/types.hHenning Baldersheim2022-05-183-3/+3
| | | | - Cut dependency to persistencetypes for searchlib.
* GC unused code and dependenciesHenning Baldersheim2022-05-141-5/+0
|
* Don't attempt to actually execute document moves from a cancelled bucket moverTor Brede Vekterli2022-05-122-3/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This prevents the following race condition where the bucket mover logic fails to notify the content layer that the bucket sub DB status has changed for a particular bucket: 1. Bucket state is changed over SPI, a mover is created and registered and a BucketTask is scheduled onto the persistence queues to actually do the document reads and finalize the move. 2. Before the bucket task is executed, bucket state is changed again over the SPI. A new mover is created, the old one is cancelled (tagging mover as not consistent) and another BucketTask is scheduled onto the persistence queues. Note: the old task still remains. 3. Old bucket task is executed and performs the actual document moving despite being cancelled. No notification is done towards the content layer since the mover was tagged as not being consistent. 4. New bucket task is executed and tries to move the same document set as the old mover. Since the documents are no longer present in the source document DB, the moves fail. This tags the mover as inconsistent and no notification is done. Bucket is automatically rechecked, but since all docs are already moved away there is nothing more to do and no subsequent mover is created. This means the "should notify?" edge is not triggered and the content layer remains blissfully unaware of any sub DB changes. This commit simply changes cancellation to actually inhibit document moves from taking place. This lets the preempting mover successfully complete its moves, thus triggering the notify-edge as expected.
* GC unused Context parameterHenning Baldersheim2022-03-3110-323/+259
|
* Remove copy constructors.Henning Baldersheim2022-03-281-108/+46
|
* Use both lvalue and rvalue specifier to avoid explicit std::move()Henning Baldersheim2022-03-281-3/+1
|
* Avoid the need for clone by using unique_ptr.Henning Baldersheim2022-03-281-2/+1
|
* Avoid need to copy/clone FieldUpdateHenning Baldersheim2022-03-271-49/+16
|
* Move BucketIdListResultHenning Baldersheim2022-03-094-20/+25
|
* Reduce visibility of document::DocumentHenning Baldersheim2022-03-071-0/+1
|
* Reduce use of Identifiable for document::DatatTypeHenning Baldersheim2022-03-032-0/+5
|
* Since we schedule the last chunk for commit in triggerSyncNow, we can assert ↵Henning Baldersheim2022-03-021-2/+2
| | | | | | that we will be fully synced on the next pull when it happens in the singleCommitter thread. That allows for further simplification.
* Revert "Revert "Balder/refactor docentry""Henning Baldersheim2022-01-0711-230/+320
|
* Revert "Balder/refactor docentry"Arnstein Ressem2022-01-0711-320/+230
|
* - Flags -> Enum.Henning Baldersheim2022-01-067-71/+65
| | | | - Consistently use DocEntryList as type for std::vector<spi::DocEntry::UP>
* Only care about size of payload. Also add payload containing only doctype ↵Henning Baldersheim2022-01-066-30/+74
| | | | and gid
* Use enum class for the flags.Henning Baldersheim2022-01-066-47/+44
|
* Simplify by avoid both DocumentSize and PersistedDocumentSize. That is the same.Henning Baldersheim2022-01-067-164/+153
|
* Simplify DocEntry to get a clean interface with multiple implementations, ↵Henning Baldersheim2022-01-0610-90/+156
| | | | | | instead of an mutant. Also add tests for the different variations a DocEntry can have.
* Declare noexcept move constructor and assignment for storage::spi::Result.Tor Egge2021-12-112-0/+4
|
* more descriptive name for header fileArne H Juul2021-12-021-1/+1
|
* track namespace move in documenttypes.defArne H Juul2021-12-022-4/+2
| | | | | | | * For C++ code this introduces a "document::config" namespace, which will sometimes conflict with the global "config" namespace. * Move all forward-declarations of the types DocumenttypesConfig and DocumenttypesConfigBuilder to a common header file.
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-243-25/+29
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* Continue serving search queries when in Maintenance node stateTor Brede Vekterli2021-11-243-10/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, entering maintenance state would implicitly deactivate all buckets on the searchnode and cause empty responses to be returned for searches. However, container query dispatch uses async health pings to decide which nodes to route queries to, so it would be possible for a node to still be used for queries for a few seconds until the ping discovered that the node should not be used. In the case of multiple groups without multiple ready replicas within the group, this would cause transient coverage loss since the dispatcher would not realize it should route queries to other groups instead. With this commit, maintenance edge behavior is changed as follows: - Buckets are _not_ deactivated when going from an available state to the maintenance state. However, they _are_ deactivate when going from maintenance state to an available state in order to avoid transient query duplicates immediately after the change. - Searches are executed as normal instead of returning empty replies when the node is in maintenance state. The following behavior is intentionally _not_ changed: - The search interface is still marked as offline when in maintenance state, as this signals that the node should be taken out of rotation. In particular, it's critical that the RPC health ping response is explicitly tagged as having zero active docs when the search interface is offline, even though many buckets may now actually be active. Otherwise, queries would not be gracefully drained from the node.
* Revert "Continue serving search queries when in Maintenance node state ↵Henning Baldersheim2021-11-233-58/+18
| | | | [run-systemtest]"
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-233-25/+29
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* Continue serving search queries when in Maintenance node stateTor Brede Vekterli2021-11-223-10/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, entering maintenance state would implicitly deactivate all buckets on the searchnode and cause empty responses to be returned for searches. However, container query dispatch uses async health pings to decide which nodes to route queries to, so it would be possible for a node to still be used for queries for a few seconds until the ping discovered that the node should not be used. In the case of multiple groups without multiple ready replicas within the group, this would cause transient coverage loss since the dispatcher would not realize it should route queries to other groups instead. With this commit, maintenance edge behavior is changed as follows: - Buckets are _not_ deactivated when going from an available state to the maintenance state. However, they _are_ deactivate when going from maintenance state to an available state in order to avoid transient query duplicates immediately after the change. - Searches are executed as normal instead of returning empty replies when the node is in maintenance state. The following behavior is intentionally _not_ changed: - The search interface is still marked as offline when in maintenance state, as this signals that the node should be taken out of rotation. In particular, it's critical that the RPC health ping response is explicitly tagged as having zero active docs when the search interface is offline, even though many buckets may now actually be active. Otherwise, queries would not be gracefully drained from the node.
* Let removeAsync handle list of documents.Henning Baldersheim2021-11-187-28/+68
|
* Move removeLocation over to Asynchandler and issue all removes for one ↵Henning Baldersheim2021-11-171-17/+17
| | | | | | bucket before waiting for the replies. Prepare RemoveResult to contain more replies.
* Adjust dummy persistence spi semantics towards proton spi semantics whenTor Egge2021-10-274-36/+96
| | | | | bucket doesn't exist: setActiveState(), put(), remove() creates bucket if it doesn't already exist.
* Adjust dummy persistence spi semantics towards proton spi semantics whenTor Egge2021-10-253-7/+56
| | | | | | | bucket doesn't exist: * getBucketInfo() returns success with empty bucket info * createIterator() returns success * iterate() returns empty complete result.
* Undo auto formatHenning Baldersheim2021-10-251-230/+217
|
* create/delete bucket will never throw.Henning Baldersheim2021-10-254-247/+244
|
* Async createBucketHenning Baldersheim2021-10-255-6/+14
|
* Add noexcept specifier to operation complete callback.Tor Egge2021-10-223-3/+3
|
* Only keep async variant to simplify what to implement and what fallback ↵Henning Baldersheim2021-10-186-130/+63
| | | | there are.
* Implement async delete bucket.Henning Baldersheim2021-10-186-31/+20
|
* Make setActiveState async.Henning Baldersheim2021-10-176-10/+24
|
* Factor out CatchResultHenning Baldersheim2021-10-154-19/+49
|
* Revert "- Refactor and use CatchResult in the PersistenceEngine in ↵Henning Baldersheim2021-10-154-49/+19
| | | | preparatio…"
* - Refactor and use CatchResult in the PersistenceEngine in preparation for ↵Henning Baldersheim2021-10-154-19/+49
| | | | making more moretaions async.
* Update Verizon Media copyright notices.gjoranv2021-10-0713-13/+13
|