aboutsummaryrefslogtreecommitdiffstats
path: root/storage/src/tests
Commit message (Collapse)AuthorAgeFilesLines
* Add async update and followup on PR comments.Henning Baldersheim2020-05-053-9/+8
|
* Implement async putHenning Baldersheim2020-05-047-41/+78
| | | | Implement async remove.
* Merge branch 'master' into ↵Henning Baldersheim2020-05-047-71/+36
|\ | | | | | | | | | | | | vekterli/remove-deprecated-bucket-disk-move-functionality Conflicts: storage/src/tests/persistence/diskmoveoperationhandlertest.cpp
| * Revert "- Implement async put"Harald Musum2020-05-048-72/+37
| |
* | Remove deprecated bucket cross-disk move functionalityTor Brede Vekterli2020-05-049-323/+12
|/ | | | | The notion of multiple disks hasn't been supported since we removed VDS, and likely won't be in the future either.
* - Implement async putHenning Baldersheim2020-05-048-37/+72
| | | | | | - Move result processing to MessageTracker - Wire putAsync through provider error wrapper too. - Handle both sync and async replies in tests.
* Remove deprecated BucketIntegrityCheckerTor Brede Vekterli2020-04-303-310/+0
| | | | Not in use after VDS was removed.
* - Add async interface to putHenning Baldersheim2020-04-2910-136/+147
| | | | | - Use MessageTracker for keeping context. - implement putAsync, but still use it synchronously.
* Use rvalue qualifierHenning Baldersheim2020-04-282-6/+6
|
* getReplySP => stealReplySPHenning Baldersheim2020-04-282-6/+6
|
* Implement hasReply avoid copying the shared_ptr just to peak at the result.Henning Baldersheim2020-04-284-23/+19
|
* Merge pull request #13084 from ↵Tor Brede Vekterli2020-04-281-36/+94
|\ | | | | | | | | vespa-engine/vekterli/optimize-btree-find-parents-with-fix Optimize B-tree bucket DB lookup with used-bits aggregation
| * Optimize B-tree bucket DB lookup with used-bits aggregationTor Brede Vekterli2020-04-271-36/+94
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By tracking the minimum used bits count across all buckets in the database we can immediately start seeking at that implicit level in the tree, as we know no parent buckets can exist above that level. Local synthetic benchmarking shows the following results with a DB size of 917504 buckets and performing getParents for all buckets in sequence: Before optimization: - B-tree DB: 0.593321 seconds - Legacy DB: 0.227947 seconds After optimization: - B-tree DB: 0.191971 seconds - Legacy DB: (unchanged)
* | Remove flush from provider interface.Henning Baldersheim2020-04-274-25/+4
| |
* | Remove batching of messages that has no effect in favor of making async ↵Henning Baldersheim2020-04-272-85/+0
| | | | | | | | operations easier to implement.
* | Prepare for making persistence layer async.Henning Baldersheim2020-04-262-19/+19
|/ | | | Avoid state in the thread.
* Revert "Optimize B-tree bucket DB lookup with used-bits aggregation"Tor Brede Vekterli2020-04-251-33/+0
|
* Optimize B-tree bucket DB lookup with used-bits aggregationTor Brede Vekterli2020-04-241-0/+33
| | | | | | | | | | | | | | | | | | | By tracking the minimum used bits count across all buckets in the database we can immediately start seeking at that implicit level in the tree, as we know no parent buckets can exist above that level. Local synthetic benchmarking shows the following results with a DB size of 917504 buckets and performing getParents for all buckets in sequence: Before optimization: - B-tree DB: 0.593321 seconds - Legacy DB: 0.227947 seconds After optimization: - B-tree DB: 0.213738 seconds - Legacy DB: (unchanged)
* Some libraries print "0x0" for a null void ptr,Tor Egge2020-04-221-1/+1
|
* Allow temporarily inhibiting maintenance ops when under loadTor Brede Vekterli2020-04-171-0/+8
| | | | | | | | If requests or responses from external sources are being constantly processed as part of the distributor tick, allow for up to N ticks to skip maintenance scanning, where N is a configurable number. This reduces the amount of CPU time spent on maintenance operations when the node has a lot of incoming data to deal with.
* Remove redundant bucket DB lookup in persistence reply handlingTor Brede Vekterli2020-04-161-5/+10
| | | | | | | | | | | | | | | | | | | Bucket DB updating happened unconditionally anyway; this was only used for failing operations in an overly pessimistic way. Removing this lookup has two benefits: - Less CPU spent in DB - Less impact expected during feeding during node state transitions since fewer operations will have to be needlessly retried by the client. Rationale: an operation towards a given bucket completes (i.e. is ACKed by all its replica nodes) at time t and the bucket is removed from the DB at time T. There is no fundamental change in correctness or behavior from the client's perspective if the order of events is tT or Tt. Both are equally valid, as the state transition edge happens independently of any reply processing.
* Only update bucket DB memory statistics at certain intervalsTor Brede Vekterli2020-04-071-0/+46
| | | | | B-tree/datastore stats can be expensive to sample, so don't do this after every full DB iteration. For now, wait at least 30s.
* Revert "Bypass communicationmanager Q"Henning Baldersheim2020-04-051-2/+2
|
* Merge pull request #12810 from ↵Tor Brede Vekterli2020-04-031-0/+35
|\ | | | | | | | | vespa-engine/vekterli/add-distributor-bucket-db-memory-usage-metrics Add distributor bucket db memory usage metrics
| * Add memory usage metrics for distributor bucket databasesTor Brede Vekterli2020-04-021-0/+35
| |
* | Bypass communicationmanager QHenning Baldersheim2020-04-021-2/+2
|/
* Reduce code duplication in test code.Tor Egge2020-03-302-14/+4
|
* Handle newer gtest versions where the legacy API is deprecated.Tor Egge2020-03-292-0/+10
|
* Track metrics for new inconsistent update phasesTor Brede Vekterli2020-03-241-0/+23
| | | | | | Reuses the old update-get metric for the single full Get sent after the initial metadata-only phase. Adds new metric set for the initial metadata Gets.
* Add comments and some extra safety handling of single Get commandTor Brede Vekterli2020-03-171-1/+16
|
* Add initial metadata-only phase to inconsistent update handlingTor Brede Vekterli2020-03-163-351/+604
| | | | | | | | | | | | | | | | | | | | | | If bucket replicas are inconsistent, the common case is that only a small subset of documents contained in the buckets are actually mutually out of sync. The added metadata phase optimizes for such a case by initially sending Get requests to all divergent replicas that only ask for the timestamps (and no fields). This is a very cheap and fast operation. If all returned timestamps are in sync, the update can be restarted in the fast path. Otherwise, a full Get will _only_ be sent to the newest replica, and its result will be used for performing the update on the distributor itself, before pushing out the result as Puts. This is in contrast to today's behavior where full Gets are sent to all replicas. For users with large documents this can be very expensive. In addition, the metadata Get operations are sent with weak internal read consistency (as they do not need to read any previously written, possibly in-flight fields). This lets them bypass the main commit queue entirely.
* Add count metric for number of documents garbage collectedTor Brede Vekterli2020-02-242-8/+31
| | | | | | | | | | | | | | | New distributor metric available as: ``` vds.idealstate.garbage_collection.documents_removed ``` Add documents removed statistics to `RemoveLocation` responses, which is what GC is currently built around. Could technically have been implemented as a diff of before/after BucketInfo, but GC is very low priority so many other mutating ops may have changed the bucket document set in the time span between sending the GC ops and receiving the replies. This relates to issue #12139
* extend crypto engine apiHåvard Pettersen2020-02-131-1/+1
| | | | | send spec for client connections to enable SNI as well as server name verification
* Add include statements needed by newer build environments.Tor Egge2020-01-261-0/+1
|
* Followup on code comments.Henning Baldersheim2020-01-231-1/+1
|
* Use a single chunkHenning Baldersheim2020-01-231-27/+26
|
* Merge pull request #11822 from vespa-engine/balder/reduce-bytebuffer-exposureHenning Baldersheim2020-01-213-8/+5
|\ | | | | Balder/reduce bytebuffer exposure
| * Add stream method and use memcpy over casting.Henning Baldersheim2020-01-213-3/+3
| |
| * Make it known that getting serialized size will always be expensive.Henning Baldersheim2020-01-201-1/+2
| |
| * GC a load of unused code. ByteBuffer towards read only.Henning Baldersheim2020-01-202-8/+4
| |
| * Unify towards nbostreamHenning Baldersheim2020-01-161-1/+1
| |
* | Add configurable support for weakly consistent client GetsTor Brede Vekterli2020-01-174-2/+86
|/ | | | | | | | | | | | | | | | | | | | | | If configured, Get operations initiated by the client are flagged with weak internal consistency. This allows the backend to bypass certain internal synchronization mechanisms, which minimizes latency at the cost of possibly not observing a consistent view of the document. This config should only be used in a very restricted set of cases where the document set is effectively read-only, or cross- field consistency or freshness does not matter. To enable the weak consistency, use an explicit config override: ``` <config name="vespa.config.content.core.stor-distributormanager"> <use_weak_internal_read_consistency_for_client_gets> true </use_weak_internal_read_consistency_for_client_gets> </config> ``` This closes #11811
* Merge pull request #11782 from ↵Henning Baldersheim2020-01-161-2/+2
|\ | | | | | | | | vespa-engine/balder/bring-you-backing-buffer-along Balder/bring your backing buffer along
| * Just use the stream method.Henning Baldersheim2020-01-161-2/+2
| |
* | Avoid inconsistent auto-created document versions taking precedenceTor Brede Vekterli2020-01-131-5/+65
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | Create-if-missing updates have a rather finicky behavior in the backend, wherein they'll set the timestamp of the previous document to that of the _new_ document timestamp if the update ended up creating a document from scratch. This particular behavior confuses the "after the fact" timestamp consistency checks, since it will seem like the document that was created from scratch is a better candidate to force convergence towards rather than the ones that actually updated an existing document. With this change we therefore detect this case specially and treat the received timestamps as if the document updated had a timestamp of zero. This matches the behavior of regular (non auto-create) updates. Note that another venue for solving this would be to alter the returned timestamp in the backend to be zero instead, but this would cause issues during rolling upgrades since some of the content nodes would be returning zero timestamps while others would be returning non-zero. This would in turn trigger false positives for the inconsistency sanity checks. Also note that this is a fallback path that should not be hit unless the a-priori inconsistency checks in the two-phase update operation somehow fails to recognize that the document versions may be out of sync. This relates to issue #11686
* Merge pull request #11704 from ↵Tor Brede Vekterli2020-01-092-0/+32
|\ | | | | | | | | vespa-engine/vekterli/support-config-disabling-of-merges Support config disabling of merges
| * Add distributor configuration for disabling merges for testingTor Brede Vekterli2020-01-082-0/+32
| | | | | | | | | | | | | | | | If config is set, all merges will be completely inhibited. This is useful for letting system tests operate against a bucket replica state that is deterministically out of sync. This relates to issue #11686
* | Use default constructor for time point when duration since epoch is zero.Tor Egge2020-01-081-1/+1
| |
* | system_time and steady_time might have different duration types.Tor Egge2020-01-081-1/+1
|/
* Ensure missing documents on replicas are not erroneously considered consistentTor Brede Vekterli2019-12-202-0/+31
| | | | | | | | | | | | | | | | | | Introducing a new member in the category "stupid bugs that I should have added explicit tests for when adding the feature initially". There was ambiguity in the GetOperation code where a timestamp sentinel value of zero was used to denote not having received any replies yet, but where a timestamp of zero also means "document not found on replica". This means that if the first reply was from a replica _without_ a document and the second reply was from a replica _with_ a document, the code would act as if the first reply effectively did not exist. Consequently the Get operation would be tagged as consistent. This had very bad consequences for the two-phase update operation logic that relied on this information to be correct. This change ensures there is no ambiguity between not having received a reply and having a received a reply with a missing document.