Documentation
¶
Overview ¶
Package reconcile holds the diff and converge logic at the heart of the agent. It compares the desired state (from the control plane) against the actual state (containers on the host carrying our label prefix) and converges: start missing, stop extra, restart unhealthy or stale. Every operation is idempotent and safely retryable, so a converge that is interrupted and re-run reaches the same end state.
The logic here depends only on the docker.Client interface, so it is fully unit testable with a fake client (see reconcile_test.go).
Crash-loop restart limiting. A cracked Egg is restarted in place on every converge (see needsRestart), which is right for a transient fault but wrong for an Egg that can never come up: it bounces forever, and each bounce takes its neighbours' shared resources (and, for a database Egg, risks an unclean shutdown every cycle). So we cap the number of crash-restarts and then leave the Egg stopped (cracked) until an operator or a fresh deploy intervenes. A redeploy rolls the workload to a new container id, which the limiter has never seen, so the new build gets a full budget: the cap never wedges a good deploy.
The limiter is stateful and lives on the agent (one per process). It is deliberately kept out of the pure Diff so that stays trivially testable; ConvergeGoverned threads it in. British English. No em dashes.
Index ¶
- Constants
- func Apply(ctx context.Context, cli docker.Client, desired contract.DesiredState, ...) error
- type Action
- type CrashLoopLimiter
- func (l *CrashLoopLimiter) AllowRestart(c docker.Container) bool
- func (l *CrashLoopLimiter) Attempts(containerID string) int
- func (l *CrashLoopLimiter) Limited(containerID string) bool
- func (l *CrashLoopLimiter) Observe(actual []docker.Container)
- func (l *CrashLoopLimiter) TakeNewlyLimited() []docker.Container
- type Plan
- type RestartLimiter
- type Step
Constants ¶
const DefaultMaxCrashRestarts = 10
DefaultMaxCrashRestarts is how many times a crashed container is restarted before the limiter gives up and leaves it stopped. Ten covers a genuinely transient fault (a dependency that was briefly unreachable) whilst bounding a hard crash-loop to a couple of minutes at the ~10-15s poll cadence.
Variables ¶
This section is empty.
Functions ¶
func Apply ¶
Apply executes a plan against the docker client. It is idempotent: a step whose effect is already in place is a cheap no-op, and a failure on one step does not abort the rest, so the next poll retries what did not land. The set of errors encountered is returned joined, so the caller can log and carry on rather than crash.
Types ¶
type Action ¶
type Action string
Action names a single convergence step, for structured logging.
const ( // ActionStart starts a fresh container for a replica slot: a plain first start, // or the start half of a zero-downtime roll (the old container keeps serving). ActionStart Action = "start" // ActionStop stops and removes a container immediately: a scale-down, an extra, // a cold Egg or a duplicate. ActionStop Action = "stop" // ActionRestart restarts an up-to-date but crashed container in place. ActionRestart Action = "restart" // ActionRoll drains a superseded (stale or legacy) container gracefully, once a // healthy replacement is already serving: the stop half of a zero-downtime roll. ActionRoll Action = "roll" )
type CrashLoopLimiter ¶ added in v0.1.55
type CrashLoopLimiter struct {
// contains filtered or unexported fields
}
CrashLoopLimiter is the default RestartLimiter: an in-memory per-container attempt count with a hard ceiling. Safe for concurrent use (the shell and egress goroutines never touch it, but the interface is small enough to keep honest). A container that reaches the ceiling is "given up on" and stays so until Observe sees it healthy (recovered) or gone (rolled).
func NewCrashLoopLimiter ¶ added in v0.1.55
func NewCrashLoopLimiter(max int) *CrashLoopLimiter
NewCrashLoopLimiter returns a limiter capping crash-restarts at max. A max <= 0 disables the cap (AllowRestart always allows), which keeps the old unbounded behaviour available.
func (*CrashLoopLimiter) AllowRestart ¶ added in v0.1.55
func (l *CrashLoopLimiter) AllowRestart(c docker.Container) bool
AllowRestart reports whether this crashed container may be restarted again, counting the attempt. Once the ceiling is passed it returns false for good (until Observe clears the container), and records the container the first time so the give-up can be logged once.
func (*CrashLoopLimiter) Attempts ¶ added in v0.1.55
func (l *CrashLoopLimiter) Attempts(containerID string) int
Attempts is the number of crash-restarts counted for a container this episode, for the status report. Zero for a container the limiter has never restarted.
func (*CrashLoopLimiter) Limited ¶ added in v0.1.55
func (l *CrashLoopLimiter) Limited(containerID string) bool
Limited reports whether the limiter has given up on a container (ceiling reached), for the status report so the control plane can tell the customer the Egg is stopped, not looping.
func (*CrashLoopLimiter) Observe ¶ added in v0.1.55
func (l *CrashLoopLimiter) Observe(actual []docker.Container)
Observe resets the budget of any container that has recovered (running and healthy) and forgets any container that no longer exists, so the map cannot grow without bound and a redeployed or hand-restarted Egg starts fresh.
func (*CrashLoopLimiter) TakeNewlyLimited ¶ added in v0.1.55
func (l *CrashLoopLimiter) TakeNewlyLimited() []docker.Container
TakeNewlyLimited returns and clears the containers that have hit the ceiling since the last call, so the caller can log each give-up exactly once.
type Plan ¶
type Plan struct {
Steps []Step
}
Plan is the set of steps computed by Diff. A rolling deploy spans more than one plan: an early plan starts the new container (leaving the old serving), and a later plan drains the old once the new is healthy. No single plan ever stops a slot's only healthy backend.
func Converge ¶
Converge is the full diff-then-apply cycle: list actual containers, compute the plan and apply it. It returns the plan (for logging) and any converge errors. It is ConvergeGoverned with no crash-loop cap; the agent uses the governed form so a hard crash-loop is eventually left stopped.
func ConvergeGoverned ¶ added in v0.1.55
func ConvergeGoverned(ctx context.Context, cli docker.Client, desired contract.DesiredState, lim RestartLimiter) (Plan, error)
ConvergeGoverned is Converge with a crash-loop limiter. It lists actual state, computes the plan, drops any in-place restart the limiter refuses (leaving that container stopped), and applies the rest. A nil limiter is exactly Converge (no cap).
func Diff ¶
func Diff(desired contract.DesiredState, actual []docker.Container) Plan
Diff computes the convergence plan for a desired state against the observed containers. It is pure: it performs no I/O and mutates nothing, which is what makes it unit testable.
Rules, per desired workload with Replicas = N, one replica slot at a time (indices 0..N-1), converging towards one healthy, up-to-date container per slot WITHOUT ever taking the slot's only healthy backend away (zero-downtime rolls):
- No up-to-date container in the slot: START a new one. Any stale (old image or version) or legacy container in the slot is LEFT RUNNING so it keeps serving whilst the new one boots. This is the start half of a start-new-before-stop-old roll, and also the plain first start when the slot is empty.
- An up-to-date container exists and is HEALTHY: the slot is served by new code, so DRAIN the old and legacy containers now (the stop half of the roll), and stop any duplicate up-to-date containers.
- An up-to-date container exists but is still booting: WAIT. The old container (if any) keeps serving; nothing is stopped.
- An up-to-date container has crashed: RESTART it in place.
- A container at an index >= N (a scale-down) is surplus and stopped.
- Replicas == 0 stops every container for the workload (the Egg goes cold).
- A container with no matching desired workload is stopped (extra).
type RestartLimiter ¶ added in v0.1.55
type RestartLimiter interface {
Observe(actual []docker.Container)
AllowRestart(c docker.Container) bool
}
RestartLimiter decides whether a crashed container may be restarted again. Observe is called once per converge with the full container list so the limiter can reset a recovered container's budget and forget ones that no longer exist; AllowRestart is called for each in-place restart the plan wants to perform and counts the attempt.
type Step ¶
type Step struct {
Action Action
WorkloadID string
// BlobID is the internal Blob identity of the Egg the step concerns, if
// known. Never rendered to customers.
BlobID string
// ContainerID is the target container, where one exists.
ContainerID string
// ReplicaIndex is the 0-based replica slot this step concerns. ActionStart uses
// it to name and label the new container; stop, roll and restart carry it for
// logging and idempotency.
ReplicaIndex int
}
Step is one planned or performed convergence action, kept for logging and for assertion in tests.
Source Files