| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
| |
Adding this to see if it triggers on any of our internal tests
|
| |
|
|
| |
Move test config helpers out of cppunit submodule.
|
| | |
|
| |
|
|
|
|
| |
Move base message sender stub out to common test module to
avoid artificial dependency from persistence tests to the
distributor tests.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
Still builds shared non-test library.
|
| | |
|
| |
|
|
|
| |
Remove convoluted thread stress test which didn't actually _verify_
any kind of correctness (aside from the test not outright crashing).
|
| | |
|
| | |
|
| |
|
|
|
| |
Still some residual vdstestlib CppUnit traces that will need
cleaning up later.
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Scanning all buckets is expensive and blocks the main distributor thread
during the entire process. Instead, use the standard "recovery mode"
functionality that is triggered for all subsequent state transitions.
Recovery mode scans allow client operations to be scheduled alongside
the scanning, but still tries to scan the DB as quickly as possible.
There shouldn't be anything that special with the first cluster state
that implies a full scan is inherently required.
|
| |\
| |
| |
| |
| | |
vespa-engine/vekterli/more-efficient-bucket-db-bulk-apis
Add new DB merging API to distributor BucketDatabase
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Abstracts away how an ordered merge may be performed with the database
and an arbitrary sorted bucket sequence, with any number of buckets
skipped, updated or inserted as part of the merge.
Such an API is required to allow efficient bulk updates of a B-tree
backed database, as it is suboptimal to require constant tree mutations.
Other changes:
- Removed legacy mutable iteration API. Not needed with new merge API.
- Const-iteration of bucket database now uses an explicit const reference
entry type to avoid needing to construct a temporary entry when we can
instead just point directly into the backing ArrayStore.
- Micro-optimizations of node remover pass to avoid going via cluster
state's node state std::map for each bucket replica entry. Now uses
a precomputed bit vector. Also avoid BucketId bit reversing operations
as much as possible by using raw bucket keys in more places.
- Changed wording and contents of log message that triggers when buckets
are removed from the DB due to no remaining nodes containing replicas
for the bucket. Now more obvious what the message actually means.
- Added several benchmark tests (disabled by default)
|
| |/
|
|
| |
This makes it possible to run storage tests in parallel.
|
| |
|
|
| |
Not needed now that B-tree code has been moved to vespalib.
|
| |
|
|
|
|
|
|
|
|
| |
Namespace is still `search` and not `vespalib` due to the massive
amount of code that would need to be modified for such a change.
Other changes:
- Move `BufferWriter` from searchlib to vespalib
- Move assertion and rand48 utilities from staging_vespalib to vespalib
- Move gtest utility code from staging_vespalib to vespalib
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Avoids scheduling Put operations towards nodes that are available in the
current cluster state, but not in the one that is being async processed
in the background. Not only are operations to such nodes highly likely to
be doomed in the first place, we also risk triggering assertion failures
by code that does not expect bucket DB inserts to said nodes to ever
be rejected.
Since we're recently added explicit rejection of inserts to nodes that
are unavailable in the pending state, these assertions have the potential
for triggering in certain edge case scenarios.
|
| |
|
|
|
| |
Made side effects of replica resurrection bug more likely to be observed,
but not in itself a believed root cause of any issues.
|
| |
|
|
|
|
|
|
|
|
|
| |
We previously only checked for node availability in the _active_ state
without looking at the pending state. This opened up for a race condition
where a reply for a previously DB-pruned node could bring a replica back
in the DB iff received during a pending state window.
Consider Maintenance as unavailable for this case, not just Down.
Also move all `PutOperation` tests to GTest.
|
| |
|
|
|
|
|
|
|
|
| |
Buckets are sorted in increasing key order before being merged into
the database. Instead of repeatedly calling `toKey()` on entries, just
store the key verbatim.
Add a (by default disabled) benchmarking test for this case.
With the bucket key change, running this locally brings total merge time
for 917K buckets down from 2 seconds to 1.3 seconds.
|
| |
|
|
|
|
| |
There may be a currently unknown edge case where some valid edges do not
trigger pruning as they should, so disable optimization entirely until
we know for sure.
|
| |\
| |
| |
| |
| | |
vespa-engine/vekterli/add-distributor-btree-bucket-database-foundations
Add initial B+tree distributor bucket database
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Still uses legacy `BucketDatabase` API, which is not optimized for
bulk loading or updating. Focus for this iteration is functional
correctness rather than API redesign. Legacy DB is still the one
wired in for all production logic.
Unit tests have been expanded to cover discovered edge cases that
were not properly tested for.
Also move distributor bucket DB tests to GTest. Use value-
parameterized test fixture instead of ad-hoc CppUnit approach.
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Only do a (potentially expensive) pruning pass over the bucket DB
when the cluster state transition indicates that one or more nodes
are not in the same availability-state as the currently active state.
For instance, if a cluster state change is for a content node going
from Maintenance to Down, no pruning action is required since that
shall already have taken place during the processing of the initial
Down edge (and no buckets shall have been created for it in the meantime).
We do not currently attempt to elide distribution config changes, as
these happen much more rarely than cluster state changes.
|
| |
|
|
|
|
|
|
| |
Distributor bucket ownership is assigned on a per superbucket basis,
so all buckets with the same superbucket can use the same decision.
The bucket DB is explicitly ordered in such a way that all buckets belonging
to the same superbucket are ordered after each other, so we need only
maintain O(1) extra state for this.
|
| | |
|
| |
|
|
|
|
| |
Avoids starving other operations when there is a lot of merge activity
taking place.
For now, 1/2 of the total persistence thread pool may process merges.
|
| | |
|
| |\
| |
| |
| |
| | |
vespa-engine/vekterli/add-read-only-support-during-cluster-state-transitions
Add read-only support during cluster state transitions
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| | |
Otherwise we'd miss activation commands sent for a cluster state in
which our own node is marked down.
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Bonus: no more spurious "we have removed buckets" log messages caused
by ownership changes.
Also ensure that we BUSY-bounce operations in `ExternalOperationHandler`
when there is no actual state to send back in a `WrongDistributionReply`.
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
| | | |
|
