| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
| |
Opaque config Slime representation is encoded as part of the
existing bundle object.
New cluster state version and broadcast is triggered if the
distribution config changes.
Including distribution config is controlled by a dedicated
config parameter, which currently defaults to `false` (i.e.
disabled).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is a grace period (currently hard-coded to 30s) between the
time a new state version is published and before nodes are counted
as not converging. This state is not sticky for a given node, so
if the state keeps changing constantly it's possible that we under-
count the number of non-converging nodes. But this can be improved
over time; for now it's primarily a source from which we can generate
alerts, as failures to converge can manifest itself as other failures
in the content cluster.
Also move out of sync ratio computations to the periodic metric
update functionality and reset it if the current controller is not
the believed cluster leader.
Start moving some CC timer code to use `Instant` instead `long`s with
implicit units.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With these changes the cluster controller continuously maintains
a global aggregate across all content nodes that represents the
number of pending and total buckets per bucket space. This aggregate
can be sampled in O(1) time.
An explicit metric `cluster-buckets-out-of-sync-ratio` has been
added, and the value is also emitted as part of the cluster state
REST API. Note: only emitted when statistics have been received
from _all_ distributors for a particular cluster state, as it would
otherwise potentially represent a state somewhere arbitrary between
two or more distinct states.
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
node down edge
The stored entry count encompasses both visible documents and
tombstones. Using this count rather than bucket count avoids any
issues where a node only containing empty buckets (i.e. no actual
data) is prohibited from being marked as permanently down.
Entry count is cross-checked with the visible document count;
if the former is zero, the latter should always be zero as well.
Since entry/doc counts were only recently introduced as part of
the HostInfo payload, we have to handle the case where these do
not exist. If entry count is not present, the decision to allow
or disallow the transition falls back to the bucket count check.
|
|
|
|
|
|
| |
The fake impl acts "as if" a single-node ZK quorum is present, so it
cannot be directly used with most multi-node tests that require
multiple nodes to actually participate in leader elections.
|
| |
|
|
|
|
|
|
| |
of the ObjectMapper.
Unless special options are used, use a common instance, or create via factory metod.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Messages now prefixed with content cluster name to help disambiguate
which cluster is exceeding its limits in multi-cluster deployments.
Example message:
```
in content cluster 'my-cool-cluster': disk on node 1 [my-node-1.example.com] is 81.0% full
(the configured limit is 80.0%). See https://docs.vespa.ai/en/operations/feed-block.html
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Messages are generated centrally by the cluster controller and pushed
to content nodes as part of a cluster state bundle; the distributors
nodes merely repeat back what they have been told. This changes the
cluster controller feed block error message code to be less ambiguous
and to include a URL to our public documentation about feed blocks.
Example of _old_ message:
```
disk on node 1 [storage.1.local] (0.510 > 0.500)
```
Same feed block with _new_ message:
```
disk on node 1 [storage.1.local] is 51.0% full (the configured limit is 50.0%).
See https://docs.vespa.ai/en/operations/feed-block.html
```
|
|
|
|
|
| |
This means we will also check distributors that are on same node as
a retired storage node
|
| |
|
|
|
|
|
|
|
| |
When we allow several groups to go down for maintenance we should check
nodes in the groups that are up if they have the required redundancy. They
might be up but have not yet synced all buckets after coming up. We want
to wait with allowing more nodes to be taken down until that is done.
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
Not used, reintroduce using junit TestInfo class if needed
|
|
|
|
|
|
|
| |
The timer used in fleetcontroller in these tests (except one in a
disabled test) are instances of RealTimer, whereas the one used
in this call is a FakeTimer from the superclass, so this call
makes no sense IMHO.
|
|
|
|
| |
GC dead code, optimize imports, fix unnecessary throws statements
|
| |
|
| |
|
|
|
|
| |
Also add group indexes for disallow messages where relevant
|
| |
|
| |
|
|\ |
|
| | |
|
| | |
|