aboutsummaryrefslogtreecommitdiffstats
path: root/persistence
Commit message (Collapse)AuthorAgeFilesLines
* Add persistence provider conformance test for metadata iterationTor Brede Vekterli2023-11-081-14/+44
|
* Include doc type name and GID in metadata iteration resultsTor Brede Vekterli2023-11-082-2/+2
| | | | | | | | | | | | | | Document type is fetched from the associated `IPersistenceHandler` on-demand; it is assumed the lifetime of the pointer must be valid for the entire lifetime of the iterator itself, as the latter holds a valid handler snapshot. For simplicity, it's possible to _not_ pass in a handler, in which case the doc type name will be implicitly empty. Some expected `DocEntry` sizes have been adjusted, as we now report the size of the document type and GID alongside the base type size.
* Test remove by gid for nonexisting gid and for gid with tombstone.Tor Egge2023-11-073-3/+31
|
* Compare all elements of DocTypeGidAndTimestamp.Tor Egge2023-11-061-2/+5
|
* Add removeByGidAsync() to spi.Tor Egge2023-11-068-0/+197
|
* Update copyrightJon Bratseth2023-10-0960-60/+60
|
* GC void ignoreNewFeatures flag always false.Henning Baldersheim2023-08-091-1/+1
|
* Reduce creation of Document instances without DocumentTypeRepo.Geir Storli2023-03-131-2/+2
|
* re-apply "remove fastos"Håvard Pettersen2023-03-011-1/+0
| | | | This reverts commit 003f019d7579e49f4ec7609ef8eac26ada6ae753.
* Revert "remove fastos"Harald Musum2023-02-281-0/+1
|
* remove fastosHåvard Pettersen2023-02-281-1/+0
|
* Avoid excessive inlining.Henning Baldersheim2023-02-112-42/+54
|
* Change from typedef to using in persistence C++ code.Geir Storli2022-12-218-10/+10
|
* Remove stacksize from the thread pools and thread executors.Henning Baldersheim2022-12-201-1/+1
|
* Revert "Revert "Unify access to assets needed during rank-setup.""Henning Baldersheim2022-09-071-2/+2
|
* Revert "Unify access to assets needed during rank-setup."Tor Egge2022-09-071-2/+2
|
* Unify access to assets needed during rank-setup.Henning Baldersheim2022-09-061-2/+2
|
* remove unused doxygen setup filesArne Juul2022-08-291-994/+0
|
* Add support for two-phase document garbage collectionTor Brede Vekterli2022-08-172-0/+30
| | | | | | | | | | | | | | If enabled, garbage collection is performed in two phases (metadata gathering and deletion) instead of just a single phase. Two-phase GC allows for ensuring the same set of documents is deleted across all nodes and explicitly takes write locks on the distributor to prevent concurrent feed ops to GC'd documents from potentially creating inconsistencies. Two-phase GC is only used _iff_ all replica content nodes support the feature _and_ it's enabled in config. An additional field has been added to the feature negotiation functionality to communicate support from content nodes to distributors.
* Don't add tombstone in dummy persistence when a newer entry already existsTor Brede Vekterli2022-08-172-5/+52
| | | | | | | This better mirrors how Proton actually works, since it's not a multi version store. Since only the highest timestamped entry for a document is the one that is ever considered on a node, there's no point in storing an explicit tombstone that cannot be referenced.
* Add wrapper for <doc id, timestamp> tuple and update APIs to use thisTor Brede Vekterli2022-07-079-12/+69
| | | | | Feels more intuitive to have a tuple that implies "document foo at timestamp bar" rather than the current inverse of "timestamp bar with document foo".
* Collapse persistencetypes into persistenceHenning Baldersheim2022-05-184-1/+121
|
* - Move persitence/spi/types.h under to persitence/spi/types.hHenning Baldersheim2022-05-183-3/+3
| | | | - Cut dependency to persistencetypes for searchlib.
* GC unused code and dependenciesHenning Baldersheim2022-05-141-5/+0
|
* Don't attempt to actually execute document moves from a cancelled bucket moverTor Brede Vekterli2022-05-122-3/+63
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This prevents the following race condition where the bucket mover logic fails to notify the content layer that the bucket sub DB status has changed for a particular bucket: 1. Bucket state is changed over SPI, a mover is created and registered and a BucketTask is scheduled onto the persistence queues to actually do the document reads and finalize the move. 2. Before the bucket task is executed, bucket state is changed again over the SPI. A new mover is created, the old one is cancelled (tagging mover as not consistent) and another BucketTask is scheduled onto the persistence queues. Note: the old task still remains. 3. Old bucket task is executed and performs the actual document moving despite being cancelled. No notification is done towards the content layer since the mover was tagged as not being consistent. 4. New bucket task is executed and tries to move the same document set as the old mover. Since the documents are no longer present in the source document DB, the moves fail. This tags the mover as inconsistent and no notification is done. Bucket is automatically rechecked, but since all docs are already moved away there is nothing more to do and no subsequent mover is created. This means the "should notify?" edge is not triggered and the content layer remains blissfully unaware of any sub DB changes. This commit simply changes cancellation to actually inhibit document moves from taking place. This lets the preempting mover successfully complete its moves, thus triggering the notify-edge as expected.
* GC unused Context parameterHenning Baldersheim2022-03-3110-323/+259
|
* Remove copy constructors.Henning Baldersheim2022-03-281-108/+46
|
* Use both lvalue and rvalue specifier to avoid explicit std::move()Henning Baldersheim2022-03-281-3/+1
|
* Avoid the need for clone by using unique_ptr.Henning Baldersheim2022-03-281-2/+1
|
* Avoid need to copy/clone FieldUpdateHenning Baldersheim2022-03-271-49/+16
|
* Move BucketIdListResultHenning Baldersheim2022-03-094-20/+25
|
* Reduce visibility of document::DocumentHenning Baldersheim2022-03-071-0/+1
|
* Reduce use of Identifiable for document::DatatTypeHenning Baldersheim2022-03-032-0/+5
|
* Since we schedule the last chunk for commit in triggerSyncNow, we can assert ↵Henning Baldersheim2022-03-021-2/+2
| | | | | | that we will be fully synced on the next pull when it happens in the singleCommitter thread. That allows for further simplification.
* Revert "Revert "Balder/refactor docentry""Henning Baldersheim2022-01-0711-230/+320
|
* Revert "Balder/refactor docentry"Arnstein Ressem2022-01-0711-320/+230
|
* - Flags -> Enum.Henning Baldersheim2022-01-067-71/+65
| | | | - Consistently use DocEntryList as type for std::vector<spi::DocEntry::UP>
* Only care about size of payload. Also add payload containing only doctype ↵Henning Baldersheim2022-01-066-30/+74
| | | | and gid
* Use enum class for the flags.Henning Baldersheim2022-01-066-47/+44
|
* Simplify by avoid both DocumentSize and PersistedDocumentSize. That is the same.Henning Baldersheim2022-01-067-164/+153
|
* Simplify DocEntry to get a clean interface with multiple implementations, ↵Henning Baldersheim2022-01-0610-90/+156
| | | | | | instead of an mutant. Also add tests for the different variations a DocEntry can have.
* Declare noexcept move constructor and assignment for storage::spi::Result.Tor Egge2021-12-112-0/+4
|
* more descriptive name for header fileArne H Juul2021-12-021-1/+1
|
* track namespace move in documenttypes.defArne H Juul2021-12-022-4/+2
| | | | | | | * For C++ code this introduces a "document::config" namespace, which will sometimes conflict with the global "config" namespace. * Move all forward-declarations of the types DocumenttypesConfig and DocumenttypesConfigBuilder to a common header file.
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-243-25/+29
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* Continue serving search queries when in Maintenance node stateTor Brede Vekterli2021-11-243-10/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, entering maintenance state would implicitly deactivate all buckets on the searchnode and cause empty responses to be returned for searches. However, container query dispatch uses async health pings to decide which nodes to route queries to, so it would be possible for a node to still be used for queries for a few seconds until the ping discovered that the node should not be used. In the case of multiple groups without multiple ready replicas within the group, this would cause transient coverage loss since the dispatcher would not realize it should route queries to other groups instead. With this commit, maintenance edge behavior is changed as follows: - Buckets are _not_ deactivated when going from an available state to the maintenance state. However, they _are_ deactivate when going from maintenance state to an available state in order to avoid transient query duplicates immediately after the change. - Searches are executed as normal instead of returning empty replies when the node is in maintenance state. The following behavior is intentionally _not_ changed: - The search interface is still marked as offline when in maintenance state, as this signals that the node should be taken out of rotation. In particular, it's critical that the RPC health ping response is explicitly tagged as having zero active docs when the search interface is offline, even though many buckets may now actually be active. Otherwise, queries would not be gracefully drained from the node.
* Revert "Continue serving search queries when in Maintenance node state ↵Henning Baldersheim2021-11-233-58/+18
| | | | [run-systemtest]"
* Handle case where bucket spaces have differing maintenance state for a nodeTor Brede Vekterli2021-11-233-25/+29
| | | | | | | | | | | Only skip deactivating buckets if the entire _node_ is marked as maintenance state, i.e. the node has maintenance state across all bucket spaces provided in the bundle. Otherwise treat the state transition as if the node goes down, deactivating all buckets. Also ensure that the bucket deactivation logic above the SPI is identical to that within Proton. This avoids bucket DBs getting out of sync between the two.
* Continue serving search queries when in Maintenance node stateTor Brede Vekterli2021-11-223-10/+46
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, entering maintenance state would implicitly deactivate all buckets on the searchnode and cause empty responses to be returned for searches. However, container query dispatch uses async health pings to decide which nodes to route queries to, so it would be possible for a node to still be used for queries for a few seconds until the ping discovered that the node should not be used. In the case of multiple groups without multiple ready replicas within the group, this would cause transient coverage loss since the dispatcher would not realize it should route queries to other groups instead. With this commit, maintenance edge behavior is changed as follows: - Buckets are _not_ deactivated when going from an available state to the maintenance state. However, they _are_ deactivate when going from maintenance state to an available state in order to avoid transient query duplicates immediately after the change. - Searches are executed as normal instead of returning empty replies when the node is in maintenance state. The following behavior is intentionally _not_ changed: - The search interface is still marked as offline when in maintenance state, as this signals that the node should be taken out of rotation. In particular, it's critical that the RPC health ping response is explicitly tagged as having zero active docs when the search interface is offline, even though many buckets may now actually be active. Otherwise, queries would not be gracefully drained from the node.
* Let removeAsync handle list of documents.Henning Baldersheim2021-11-187-28/+68
|