| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
| |
Implement async remove.
|
|\
| |
| |
| |
| |
| |
| | |
vekterli/remove-deprecated-bucket-disk-move-functionality
Conflicts:
storage/src/tests/persistence/diskmoveoperationhandlertest.cpp
|
| | |
|
|/
|
|
|
| |
The notion of multiple disks hasn't been supported since we
removed VDS, and likely won't be in the future either.
|
|
|
|
|
|
| |
- Move result processing to MessageTracker
- Wire putAsync through provider error wrapper too.
- Handle both sync and async replies in tests.
|
|
|
|
| |
Not in use after VDS was removed.
|
|
|
|
|
| |
- Use MessageTracker for keeping context.
- implement putAsync, but still use it synchronously.
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| | |
vespa-engine/vekterli/optimize-btree-find-parents-with-fix
Optimize B-tree bucket DB lookup with used-bits aggregation
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
By tracking the minimum used bits count across all buckets in
the database we can immediately start seeking at that implicit
level in the tree, as we know no parent buckets can exist above
that level.
Local synthetic benchmarking shows the following results with a
DB size of 917504 buckets and performing getParents for all
buckets in sequence:
Before optimization:
- B-tree DB: 0.593321 seconds
- Legacy DB: 0.227947 seconds
After optimization:
- B-tree DB: 0.191971 seconds
- Legacy DB: (unchanged)
|
| | |
|
| |
| |
| |
| | |
operations easier to implement.
|
|/
|
|
| |
Avoid state in the thread.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
By tracking the minimum used bits count across all buckets in
the database we can immediately start seeking at that implicit
level in the tree, as we know no parent buckets can exist above
that level.
Local synthetic benchmarking shows the following results with a
DB size of 917504 buckets and performing getParents for all
buckets in sequence:
Before optimization:
- B-tree DB: 0.593321 seconds
- Legacy DB: 0.227947 seconds
After optimization:
- B-tree DB: 0.213738 seconds
- Legacy DB: (unchanged)
|
| |
|
|
|
|
|
|
|
|
| |
If requests or responses from external sources are being constantly
processed as part of the distributor tick, allow for up to N ticks
to skip maintenance scanning, where N is a configurable number.
This reduces the amount of CPU time spent on maintenance operations
when the node has a lot of incoming data to deal with.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Bucket DB updating happened unconditionally anyway; this was
only used for failing operations in an overly pessimistic way.
Removing this lookup has two benefits:
- Less CPU spent in DB
- Less impact expected during feeding during node state
transitions since fewer operations will have to be needlessly
retried by the client.
Rationale: an operation towards a given bucket completes (i.e.
is ACKed by all its replica nodes) at time t and the bucket is
removed from the DB at time T. There is no fundamental change
in correctness or behavior from the client's perspective if
the order of events is tT or Tt. Both are equally valid, as
the state transition edge happens independently of any reply
processing.
|
|
|
|
|
| |
B-tree/datastore stats can be expensive to sample, so don't do
this after every full DB iteration. For now, wait at least 30s.
|
| |
|
|\
| |
| |
| |
| | |
vespa-engine/vekterli/add-distributor-bucket-db-memory-usage-metrics
Add distributor bucket db memory usage metrics
|
| | |
|
|/ |
|
| |
|
| |
|
|
|
|
|
|
| |
Reuses the old update-get metric for the single full Get sent
after the initial metadata-only phase. Adds new metric set for
the initial metadata Gets.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If bucket replicas are inconsistent, the common case is that only
a small subset of documents contained in the buckets are actually
mutually out of sync. The added metadata phase optimizes for
such a case by initially sending Get requests to all divergent
replicas that only ask for the timestamps (and no fields). This
is a very cheap and fast operation. If all returned timestamps
are in sync, the update can be restarted in the fast path.
Otherwise, a full Get will _only_ be sent to the newest replica,
and its result will be used for performing the update on the
distributor itself, before pushing out the result as Puts.
This is in contrast to today's behavior where full Gets are
sent to all replicas. For users with large documents this
can be very expensive.
In addition, the metadata Get operations are sent with weak
internal read consistency (as they do not need to read any
previously written, possibly in-flight fields). This lets them
bypass the main commit queue entirely.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
New distributor metric available as:
```
vds.idealstate.garbage_collection.documents_removed
```
Add documents removed statistics to `RemoveLocation` responses,
which is what GC is currently built around. Could technically have
been implemented as a diff of before/after BucketInfo, but GC is
very low priority so many other mutating ops may have changed the
bucket document set in the time span between sending the GC ops
and receiving the replies.
This relates to issue #12139
|
|
|
|
|
| |
send spec for client connections to enable SNI as well as server name
verification
|
| |
|
| |
|
| |
|
|\
| |
| | |
Balder/reduce bytebuffer exposure
|
| | |
|
| | |
|
| | |
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If configured, Get operations initiated by the client are flagged
with weak internal consistency. This allows the backend to bypass
certain internal synchronization mechanisms, which minimizes latency
at the cost of possibly not observing a consistent view of the
document. This config should only be used in a very restricted set
of cases where the document set is effectively read-only, or cross-
field consistency or freshness does not matter.
To enable the weak consistency, use an explicit config override:
```
<config name="vespa.config.content.core.stor-distributormanager">
<use_weak_internal_read_consistency_for_client_gets>
true
</use_weak_internal_read_consistency_for_client_gets>
</config>
```
This closes #11811
|
|\
| |
| |
| |
| | |
vespa-engine/balder/bring-you-backing-buffer-along
Balder/bring your backing buffer along
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Create-if-missing updates have a rather finicky behavior in the backend,
wherein they'll set the timestamp of the previous document to that of the
_new_ document timestamp if the update ended up creating a document from
scratch. This particular behavior confuses the "after the fact" timestamp
consistency checks, since it will seem like the document that was created
from scratch is a better candidate to force convergence towards rather
than the ones that actually updated an existing document.
With this change we therefore detect this case specially and treat the
received timestamps as if the document updated had a timestamp of zero.
This matches the behavior of regular (non auto-create) updates.
Note that another venue for solving this would be to alter the returned
timestamp in the backend to be zero instead, but this would cause issues
during rolling upgrades since some of the content nodes would be returning
zero timestamps while others would be returning non-zero. This would in
turn trigger false positives for the inconsistency sanity checks.
Also note that this is a fallback path that should not be hit unless
the a-priori inconsistency checks in the two-phase update operation
somehow fails to recognize that the document versions may be out of
sync.
This relates to issue #11686
|
|\
| |
| |
| |
| | |
vespa-engine/vekterli/support-config-disabling-of-merges
Support config disabling of merges
|
| |
| |
| |
| |
| |
| |
| |
| | |
If config is set, all merges will be completely inhibited. This is
useful for letting system tests operate against a bucket replica state
that is deterministically out of sync.
This relates to issue #11686
|
| | |
|
|/ |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introducing a new member in the category "stupid bugs that I should have
added explicit tests for when adding the feature initially".
There was ambiguity in the GetOperation code where a timestamp sentinel
value of zero was used to denote not having received any replies yet,
but where a timestamp of zero also means "document not found on replica".
This means that if the first reply was from a replica _without_ a document
and the second reply was from a replica _with_ a document, the code
would act as if the first reply effectively did not exist. Consequently
the Get operation would be tagged as consistent. This had very bad
consequences for the two-phase update operation logic that relied on
this information to be correct.
This change ensures there is no ambiguity between not having received
a reply and having a received a reply with a missing document.
|