| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
| |
The fake impl acts "as if" a single-node ZK quorum is present, so it
cannot be directly used with most multi-node tests that require
multiple nodes to actually participate in leader elections.
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
Not used, reintroduce using junit TestInfo class if needed
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
loaded
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
initProgress and capacity. Also gc unused 'reliability' member.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ZooKeeper
This avoids a subtle edge case where the underlying ZK integration code
may fail silently a write, leaving the core controller logic to think that
it had actually durably persisted a particular state version.
In case of reelections racing with broadcasts, it would be possible for
leader-edge readbacks from ZK to retrieve a _lower_ version than one
that had already been published. This would cause the cluster controller
to get very confused about which cluster states nodes had already observed.
If a newly produced state version overlapped with a previously broadcast
state, the controller would not push the updated state to the nodes, as it
would (with good reason) assume the node had already observed it, seeing
that it had already ACKed the particular version number.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
The flag controlled config read by the Cluster Controller. Therefore, I have
left the ModelContextImpl.Properties method and implementation (now always
returning true), but the model has stopped using that method internally, and
the config is no longer used in the CC.
The field in the fleetcontroller.def is left unchanged and documented as
deprecated.
|
|
|
|
|
|
|
|
|
|
|
|
| |
This makes the Cluster Controller use the
vds.datastored.bucket_space.buckets_total, dimension bucketSpace=default, to
determine whether a content node manages zero buckets, and if so, will allow
the node to go permanently down. This is used when a node is retiring, and it
is to be removed from the application.
The change is guarded by the use-bucket-space-metric, default true. If the new
metric doesn't work as expected, we can revert to using the current/old metric
by flipping the flag. The flag can be controlled per application.
|
|
|
|
|
| |
Makes it easier for an external observer to understand what set of nodes
is causing the cluster state to not converge.
|
|
|
|
|
|
|
|
| |
Ensures that event is only emitted when we're actually publishing a
state containing the state transition. Emitting events in the timer
code is fragile when it does not modify any state, risking emitting
the same event an unbounded amount of times if the condition keeps
holding for each timer cycle.
|
| |
|
|
|
|
|
|
|
|
| |
Version mismatches in backend do not return explicit RPC errors, so
actual vs. desired versions must be checked in order to avoid potentially
spurious activation of other versions.
Also do some minor code cleanup.
|
|
|
|
| |
Mirrors the default values in the actual underlying config definitions.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Store synchronously upon each new versioned state, load whenever controller
is elected master. Effectively carries over visible node states from one
controller's lifetime to the next. This removes the edge case where default
bucket space content nodes are marked as in Maintainence until their global
merge status is known.
To avoid controller tripping over its own feet, state bundles are now _not_
versioned at all until the initial send time period has passed. This prevents
overwriting the state persisted from a previous controller with a transient
state where all nodes are down due to not having Slobrok contact yet.
A new cluster state recompute+send edge has been added when the master passes
its initial state send time period.
|
|
|
|
|
|
| |
Prevents an unstable cluster from potentially holding up all
container request processing threads indefinitely.
Deadline errors are translated into HTTP 504 errors to REST API clients.
|
|
|
|
|
|
| |
We check both for master status and task failure, as we otherwise place
a potentially dangerous silent dependency on the task always failing itself
if the controller is not a master.
|
|
|
|
|
|
| |
Avoids edge case where set-node-state requests sent to followers would
have their response delayed indefinitely due to controller not
publishing any versions that the task's ACK barrier could be released by.
|
| |
|
|
|
|
|
|
|
|
| |
Removes dependency on having to invoke broadcastNewState before being
able to observe that all distributors are in sync. Invocations of
broadcastNewState are gated by a grace period between each time,
so unless this is done we get artificial delays before a synchronous
task can be considered complete.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
Used to enable synchronous operation for set-node-state calls, which
ensure that side-effects of the call are visible when the response
returns.
If controller leadership is lost before state is published, tasks will
be failed back to the client.
|