aboutsummaryrefslogtreecommitdiffstats
path: root/clustercontroller-core/src
Commit message (Collapse)AuthorAgeFilesLines
* Include distribution config as part of cluster state bundleTor Brede Vekterli22 hours13-41/+332
| | | | | | | | | | | | Opaque config Slime representation is encoded as part of the existing bundle object. New cluster state version and broadcast is triggered if the distribution config changes. Including distribution config is controlled by a dedicated config parameter, which currently defaults to `false` (i.e. disabled).
* Add metric for number of nodes not converged to the latest cluster state versionTor Brede Vekterli2024-06-069-25/+147
| | | | | | | | | | | | | | | | | | There is a grace period (currently hard-coded to 30s) between the time a new state version is published and before nodes are counted as not converging. This state is not sticky for a given node, so if the state keeps changing constantly it's possible that we under- count the number of non-converging nodes. But this can be improved over time; for now it's primarily a source from which we can generate alerts, as failures to converge can manifest itself as other failures in the content cluster. Also move out of sync ratio computations to the periodic metric update functionality and reset it if the current controller is not the believed cluster leader. Start moving some CC timer code to use `Instant` instead `long`s with implicit units.
* Add cluster bucket sync status summary to CC status pageTor Brede Vekterli2024-05-281-5/+50
| | | | | | Uses the same "out of sync ratio" value as is currently exposed as a metric and through the State V2 REST API, but rendered as a percentage to be more human-friendly.
* Don't make elevated "events last week" column render as error if highTor Brede Vekterli2024-05-271-3/+1
| | | | | | The cluster controller has no notion of the nature of these events (could just be a benign upgrade cycle), so don't paint it with a scary red color that implies something is wrong in the cluster.
* Add rationale for why we cap pending buckets to total countTor Brede Vekterli2024-05-081-0/+6
|
* Emit single metric for how out of sync the cluster data isTor Brede Vekterli2024-05-0614-9/+224
| | | | | | | | | | | | | | With these changes the cluster controller continuously maintains a global aggregate across all content nodes that represents the number of pending and total buckets per bucket space. This aggregate can be sampled in O(1) time. An explicit metric `cluster-buckets-out-of-sync-ratio` has been added, and the value is also emitted as part of the cluster state REST API. Note: only emitted when statistics have been received from _all_ distributors for a particular cluster state, as it would otherwise potentially represent a state somewhere arbitrary between two or more distinct states.
* Replace all usages of Arrays.asList with List.of where possible.Henning Baldersheim2024-04-123-6/+7
|
* Unify on Set.ofHenning Baldersheim2024-04-113-12/+9
|
* Unify on Map.ofHenning Baldersheim2024-04-117-31/+22
|
* Use sentinel value for missing buckets rather than throwingTor Brede Vekterli2024-01-291-1/+1
| | | | Bucket count should have been pre-verified as present by the caller.
* Explicitly report docs, tombstones and buckets in disallow-messageTor Brede Vekterli2024-01-292-26/+38
|
* Use stored entry count rather than bucket count for (dis-)allowing permanent ↵Tor Brede Vekterli2024-01-262-112/+182
| | | | | | | | | | | | | | | | | node down edge The stored entry count encompasses both visible documents and tombstones. Using this count rather than bucket count avoids any issues where a node only containing empty buckets (i.e. no actual data) is prohibited from being marked as permanently down. Entry count is cross-checked with the visible document count; if the former is zero, the latter should always be zero as well. Since entry/doc counts were only recently introduced as part of the HostInfo payload, we have to handle the case where these do not exist. If entry count is not present, the decision to allow or disallow the transition falls back to the bucket count check.
* Use fake ZooKeeper database implementation for subset of CC testsTor Brede Vekterli2023-12-047-15/+184
| | | | | | The fake impl acts "as if" a single-node ZK quorum is present, so it cannot be directly used with most multi-node tests that require multiple nodes to actually participate in leader elections.
* Move Jackson util from vespajlib to container-core.Henning Baldersheim2023-11-242-2/+2
|
* jackson 2.16 changes some of its default settings so we consolidate our use ↵Henning Baldersheim2023-11-236-18/+19
| | | | | | of the ObjectMapper. Unless special options are used, use a common instance, or create via factory metod.
* Update copyrightJon Bratseth2023-10-09200-199/+201
|
* GC unused count and averageHenning Baldersheim2023-08-301-8/+0
|
* Consolidate hamcrest usage to 2.x and remove cthul-matchersBjørn Christian Seime2023-08-299-18/+1
|
* Revert "More logging at higher level, need more info"Harald Musum2023-08-251-5/+3
|
* Add content cluster name to generated feed block messageTor Brede Vekterli2023-07-264-34/+37
| | | | | | | | | | | Messages now prefixed with content cluster name to help disambiguate which cluster is exceeding its limits in multi-cluster deployments. Example message: ``` in content cluster 'my-cool-cluster': disk on node 1 [my-node-1.example.com] is 81.0% full (the configured limit is 80.0%). See https://docs.vespa.ai/en/operations/feed-block.html ```
* Make generated automatic feed block error messages more user-friendlyTor Brede Vekterli2023-07-265-38/+60
| | | | | | | | | | | | | | | | | | Messages are generated centrally by the cluster controller and pushed to content nodes as part of a cluster state bundle; the distributors nodes merely repeat back what they have been told. This changes the cluster controller feed block error message code to be less ambiguous and to include a URL to our public documentation about feed blocks. Example of _old_ message: ``` disk on node 1 [storage.1.local] (0.510 > 0.500) ``` Same feed block with _new_ message: ``` disk on node 1 [storage.1.local] is 51.0% full (the configured limit is 50.0%). See https://docs.vespa.ai/en/operations/feed-block.html ```
* More logging at higher level, need more infoHarald Musum2023-07-201-3/+5
|
* Use cluster state to check if disttributors are UPHarald Musum2023-07-181-5/+4
|
* Check min replication seen from all distributors that are UPHarald Musum2023-07-172-29/+17
| | | | | This means we will also check distributors that are on same node as a retired storage node
* Fix minor issues after code reviewsHarald Musum2023-07-172-3/+1
|
* Avoid code duplicationHarald Musum2023-07-161-9/+7
|
* Check redundancy also for groups that are upHarald Musum2023-07-162-18/+106
| | | | | | | When we allow several groups to go down for maintenance we should check nodes in the groups that are up if they have the required redundancy. They might be up but have not yet synced all buckets after coming up. We want to wait with allowing more nodes to be taken down until that is done.
* Split out method for finding min replication per distributorHarald Musum2023-07-141-15/+20
|
* Separate code at a higher level based on groupes setup or notHarald Musum2023-07-111-71/+66
|
* More minor changesHarald Musum2023-07-091-22/+29
|
* Split out method to avoid repeating codeHarald Musum2023-07-091-69/+68
|
* Renames and minor refactorings, no funcational changesHarald Musum2023-07-095-258/+226
|
* SimplifyHarald Musum2023-07-092-35/+17
|
* SimplifyHarald Musum2023-07-055-16/+9
|
* Code cleanup, no functional changesHarald Musum2023-07-051-5/+4
|
* Minor refactoring and start of some new testHarald Musum2023-07-052-25/+46
|
* Replace metrics strings with the corresponding enum constants.gjoranv2023-06-222-4/+6
| | | | - Add 'metrics' as provided dep for clustercontroller-core
* Revert "Replace metrics strings with the corresponding enum constants."Harald Musum2023-06-222-6/+4
|
* Replace metrics strings with the corresponding enum constants.gjoranv2023-06-212-4/+6
| | | | - Add 'metrics' as provided dep for clustercontroller-core
* Move fetchStatusPageHarald Musum2023-06-193-42/+79
|
* ModernizeHarald Musum2023-06-073-23/+23
|
* Require non-null MetricUpdaterHarald Musum2023-06-053-22/+11
|
* Add back some testing of getMaster()Harald Musum2023-06-051-0/+60
|
* Remove support for RPC method getMaster, only used in testsHarald Musum2023-06-054-176/+2
|
* Inline and simplifyHarald Musum2023-06-051-15/+2
|
* Set nextInLineCount to 0 initially instead of checking fleetcontroller countHarald Musum2023-06-022-2/+3
|
* Simplify and remove some test methodsHarald Musum2023-06-015-25/+16
|
* ZooKeeper is always used, simplifyHarald Musum2023-06-017-96/+28
|
* Require non-null zooKeeperServerAddress in FleetControllerOptionsHarald Musum2023-06-015-126/+135
|
* Move into method that does reconfigurationHarald Musum2023-06-012-1/+1
|