Help us shape the future of Go! Take the 2026 Go Developer Survey

liveness

package
v0.17.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: AGPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package liveness decides, from what the cluster's hosts can reach, when a host must stop its own apps and when the leader may take a host for lost (R-421, docs/design/notes-cluster.md "Deciding a host is gone, and fencing").

The decisions are pure and live in Tracker (one host's own view) and Ledger (the leader's collection of every host's view). Monitor, Watch and Service run them: probing, stopping, recording and starting standbys.

The rule, on both sides:

  • A host stops its own apps only when it has reached fewer than half the cluster's hosts, itself included, for Period.
  • The leader takes a host for lost only when more than half the hosts have not reached it for three times Period, so the lost host has stopped its apps before its standbys start.
  • With fewer than three hosts neither happens by itself.

Everything here leans one way. A contact anywhere in a window counts the peer as reached; a gap in this host's own probing (a suspended machine, a clock that jumped) restarts the window rather than counting as silence; a report the leader has not heard lately counts for nothing. Each of those can only delay a fence or a failover, never cause one: a host that is not cut off must never stop its apps.

Durations are measured on each host's own monotonic clock. No host compares its clock with another's: what crosses the wire is how long, never when.

Index

Constants

View Source
const (
	SkipNoStandby  = "no_standby"
	SkipFailed     = "failed"
	SkipGone       = "gone"
	SkipNotWired   = "standbys_unavailable"
	SkipPromoteErr = "promote_failed"
	// SkipAuditErr is a failover whose audit event could not be written:
	// it is not started without one (R-421, R-228), and the leader tries
	// again each round until it is.
	SkipAuditErr = "audit_unavailable"
)

Why an app on a lost host was not failed over.

View Source
const DefaultFailoverConcurrency = 8

DefaultFailoverConcurrency is how many of a lost host's queues of apps start at once [P].

View Source
const DefaultPeriod = 30 * time.Second

DefaultPeriod is T [P: 30 s] (R-421), and cluster.liveness_period's default (config.DefaultLivenessPeriod).

View Source
const MinHosts = 3

MinHosts is the smallest cluster in which anything stops or fails over by itself (R-421). Two hosts cannot tell a dead peer from a cut link.

Variables

View Source
var ErrContactOnly = errors.New("the host's machine answered without its view")

ErrContactOnly is wrapped by a probe's error when the peer's machine answered but gave no view: it turned the connection away, failed the handshake, or refused the ping. That is contact (R-421): the machine is up and on the network, though its Pando process may be down, restarting or waiting for its database, and its apps may be running. Only silence — a probe that timed out or found no route — is not contact.

Functions

func Elapsed

func Elapsed(c clock.Clock) func() time.Duration

Elapsed returns a reader of c's monotonic time since the call: a duration that never goes backward.

clock.System's Now is UTC, and UTC drops Go's monotonic reading, so for the system clock this reads time.Since on a monotonic start instead: a wall clock stepped by NTP or by hand moves nothing here. For any other clock — a test's Fake — it reads Now, and a reading earlier than the last is held at the last, so a clock set backward stops time rather than reversing it.

Types

type Actor

type Actor struct {
	Kind       audit.PrincipalKind
	ID         string
	OnBehalfOf string
	RequestID  string
}

Actor is who said a host is lost.

type Apps

type Apps interface {
	ByID(ctx context.Context, appID string) (state.App, bool, error)
}

Apps reads an app, for its state: a failed app is never failed over (R-151).

type Auditor

type Auditor interface {
	Write(ctx context.Context, e audit.Event) error
}

Auditor writes the audit log.

type Config

type Config struct {
	// Period is T: how long a host must have reached fewer than half the
	// cluster before it stops its apps. DefaultPeriod when zero.
	Period time.Duration

	// Interval is how often a host probes every other. Period/6 when zero,
	// and never more than that.
	Interval time.Duration

	// Timeout bounds one probe. Interval/2 when zero, and never more than
	// Interval.
	Timeout time.Duration

	// PeerWait is how long a round waits for the list of peers before it
	// probes the last one known. The list is read beside the round, never
	// in it, so a database that hangs never keeps a host from probing, or
	// from fencing. The smaller of Timeout and 2 s when zero.
	PeerWait time.Duration
}

Config holds the timings. Zero values take the defaults.

func (Config) Fresh

func (c Config) Fresh() time.Duration

Fresh is how recently the leader must have heard a host's report for it to count. A host the leader has not heard from lately has said nothing.

func (Config) LostAfter

func (c Config) LostAfter() time.Duration

LostAfter is how long a majority must not have reached a host before the leader takes it for lost: three times Period (R-421).

func (Config) MaxGap

func (c Config) MaxGap() time.Duration

MaxGap is the longest pause between two of a host's own probe rounds that still counts as probing without a break. A longer one — the machine was suspended, the process stalled, the clock jumped — starts the host's window again.

func (Config) Normalize

func (c Config) Normalize() Config

Normalize fills in the defaults and holds the timings to each other.

type Failover

type Failover interface {
	PromoteStandby(ctx context.Context, appID string) error
}

Failover starts an app's standby on its standby host, which becomes the app's home (R-419). The standby slice implements it; Service calls it once per app homed on a lost host that has a standby and is not failed.

type Fencer

type Fencer interface {
	Fence(ctx context.Context) (int, error)
}

Fencer stops this host's app containers: hostagent.DockerFencer. It returns how many it stopped. It is never asked to start one: a host that has fenced never restarts its apps by itself (R-421).

type HomedApp

type HomedApp struct {
	AppID      string
	HasStandby bool
	// Queue is which apps start one after another: those sharing a queue
	// start in the order listed, each once the one before has ended, and
	// different queues start side by side (R-422: standbys sharing one
	// host past the reservation take its room in priority order).
	Queue string
}

HomedApp is an app whose home is a host, and whether it has a standby.

type Homes

type Homes interface {
	AppsHomedOn(ctx context.Context, hostID string) ([]HomedApp, error)
}

Homes lists the apps homed on a host. The standby slice implements it, with Failover.

type HostLoss

type HostLoss struct {
	HostID      string       `json:"host_id"`
	Name        string       `json:"name"`
	AlreadyLost bool         `json:"already_lost"`
	Started     []string     `json:"started"`
	Skipped     []SkippedApp `json:"skipped"`
}

HostLoss is what taking a host for lost did.

type Ledger

type Ledger struct {
	// contains filtered or unexported fields
}

Ledger is the leader's collection of what every host reports it has not reached, and the one place a host is taken for lost (R-421). Times given to it are readings of the leader's own monotonic clock; the durations in a report are the observer's own.

It is kept in memory by whichever replica leads, not in the database. A decision needs only the last three Periods of reports, and a new leader that starts empty has only to wait those three Periods again: slower, never wrong. Keeping it in Postgres would put the database on the path of a decision that must not depend on it.

A Ledger is not safe for concurrent use; Watch holds it under one goroutine.

func NewLedger

func NewLedger(cfg Config) *Ledger

NewLedger returns an empty Ledger.

func (*Ledger) Forget

func (l *Ledger) Forget(observer string)

Forget drops what observer reported, as when it leaves the cluster.

func (*Ledger) Lost

func (l *Ledger) Lost(now time.Duration, hosts []string, x string) bool

Lost reports whether x is lost (R-421): the cluster, x included, has at least three hosts, and more than half of them have not reached x for three Periods. x reaches itself, so it is never one of them.

func (*Ledger) Record

func (l *Ledger) Record(now time.Duration, observer string, unreached map[string]time.Duration)

Record keeps observer's report, received at now: for each peer, how long the observer has watched it without contact.

func (*Ledger) Silent

func (l *Ledger) Silent(now time.Duration, hosts []string, x string) int

Silent is how many of hosts, x excepted, report having watched x without contact for at least three Periods, in a report the leader heard lately.

type Monitor

type Monitor struct {
	Self string

	// Peers lists the cluster's hosts. An error keeps the last list.
	Peers func(ctx context.Context) ([]Peer, error)

	// Probe pings a peer and returns the View it answered with, as JSON. An
	// error wrapping ErrContactOnly is contact without a view; any other
	// error is silence.
	Probe func(ctx context.Context, p Peer) ([]byte, error)

	Fencer Fencer
	Config Config
	Clock  clock.Clock
	Logger *zap.Logger
	// contains filtered or unexported fields
}

Monitor is a host's half of R-421, run by the host's agent: it probes every other host, answers their probes with its own view, and stops this host's apps when it is the one cut off.

It needs nothing from the control plane but the list of peers, and keeps the last list it was given when the control plane cannot be asked: Postgres or every control host being down never stops an app. What stops one is this host failing to reach most of the cluster's hosts, the control hosts counted as hosts like any other, by their agents. The list is read beside the round, and the fence runs beside it too: neither a database that hangs nor a runtime slow to stop containers keeps the host from probing.

func (*Monitor) Answer

func (m *Monitor) Answer(from string) []byte

Answer records contact from the peer that sent a probe, and returns this host's View as JSON. from is "" for a probe from the leader's collector, which is not a host's monitor.

func (*Monitor) Fenced

func (m *Monitor) Fenced() bool

Fenced reports whether this host has fenced and is still cut off.

func (*Monitor) Fencing

func (m *Monitor) Fencing() bool

Fencing reports whether a fence is decided and not lifted: from the moment this host decides it is cut off, through the stopping of its containers, until it reaches the cluster again (R-421). While it is, nothing may start a container here: one started while the others stop would run on beside the standby that replaces it.

func (*Monitor) Round

func (m *Monitor) Round(ctx context.Context)

Round probes every peer once, records who answered, and fences if this host must: when it has reached fewer than half the cluster for Period, or when more than half the cluster answers that it had not reached this host for as long as the leader takes a host for lost.

func (*Monitor) Run

func (m *Monitor) Run(ctx context.Context)

Run probes every Interval until ctx ends, then waits for a fence under way.

func (*Monitor) Settle

func (m *Monitor) Settle()

Settle waits for a fence under way to end.

func (*Monitor) UnreachedFor

func (m *Monitor) UnreachedFor(peer string) (time.Duration, bool)

UnreachedFor is how long this host has watched peer without contact, and whether it watches peer at all: what a person marking peer lost is checked against (Witness).

type Peer

type Peer struct {
	ID      string
	Name    string
	Address string

	// Lost is that the host is marked lost (R-421). A lost host is left out
	// of the cluster's size on every side — the leader's and each host's
	// alike, so they agree on it — and is not probed by the hosts; the
	// leader still probes it, to see it come back (R-419).
	Lost bool
}

Peer is another host of the cluster, as a probe reaches it.

type Returns

type Returns interface {
	AppsLeftOn(ctx context.Context, hostID string) ([]string, error)
	RemoveStale(ctx context.Context, appID string) error
}

Returns is what a lost host coming back needs (R-419): the apps failed over away from it, whose copies it still holds, and removing each copy, never restarting it (R-148 does not apply to a copy that was failed over). The standby slice implements it, with Failover.

type Service

type Service struct {
	Store    Store
	Apps     Apps
	Audit    Auditor
	Logger   *zap.Logger
	Homes    Homes
	Failover Failover
	// Returns removes a returned host's copies before its lost mark is
	// cleared. Nil leaves a lost host marked: without it nothing would
	// remove the copies, and a host still marked lost starts nothing.
	Returns Returns

	// Witness says whether a host has been heard from lately. A host that
	// has is not gone, and is not marked lost by hand: its apps may be
	// running, and a standby started beside them would be a second copy
	// (R-153). Nil skips the check.
	Witness interface {
		Silent(ctx context.Context, h state.Host) error
	}

	// FailoverConcurrency is how many queues of a lost host's apps start
	// at once; zero is DefaultFailoverConcurrency.
	FailoverConcurrency int
	// contains filtered or unexported fields
}

Service takes hosts for lost and starts their apps' standbys (R-419, R-421): automatically when Watch, on the leader, finds a majority has not reached a host for three Periods, and by hand through MarkLost. It also audits the fences hosts report.

func (*Service) MarkLost

func (s *Service) MarkLost(ctx context.Context, hostID string, by Actor) (HostLoss, error)

MarkLost is a person saying a host is gone (R-421): the host is marked lost and its apps' standbys started. It is how a cluster under three hosts fails over at all, and is allowed in any cluster. Authorization is the caller's: install.cluster.manage. Marking a host already lost again starts the standbys of apps still homed on it, which is how a failover that failed is retried.

func (*Service) RecordFence

func (s *Service) RecordFence(ctx context.Context, hostID string, v View)

RecordFence audits a fence a host reported, once per fence however often the leader hears of it (R-421).

func (*Service) Settle

func (s *Service) Settle()

Settle waits for the failovers under way to end.

type SkippedApp

type SkippedApp struct {
	AppID  string `json:"app_id"`
	Reason string `json:"reason"`
	Detail string `json:"detail,omitempty"`
}

SkippedApp is an app on a lost host that was not started elsewhere.

type Store

type Store interface {
	Host(ctx context.Context, hostID string) (state.Host, error)
	MarkHostLost(ctx context.Context, hostID, by string) (bool, error)
	ClearHostLost(ctx context.Context, hostID string) error
	LostHosts(ctx context.Context) (map[string]bool, error)
	RecordHostFence(ctx context.Context, hostID, fenceID string) (bool, error)
}

Store is the slice of state.Cluster the service uses.

type Tracker

type Tracker struct {
	// contains filtered or unexported fields
}

Tracker is one host's own view: when it last had contact with each other host, and how long it has been probing without a break. Every time it is given is a reading of this host's monotonic clock (Elapsed).

Contact is either direction: this host's probe of a peer was answered, or a peer's probe of this host arrived. Counting both is what keeps the two sides of a link in agreement, so that a host the others have not reached is also a host that has not reached them (R-421).

A Tracker is not safe for concurrent use; Monitor holds it under a lock.

func NewTracker

func NewTracker(cfg Config) *Tracker

NewTracker returns a Tracker with no peers.

func (*Tracker) Contact

func (t *Tracker) Contact(now time.Duration, peer string)

Contact records contact with peer at now: one of this host's probes was answered, or the peer's probe arrived. A peer outside the set is ignored.

func (*Tracker) CutOff

func (t *Tracker) CutOff(now time.Duration) bool

CutOff reports whether this host reaches fewer than half the cluster's hosts in the last Period. It is not yet a reason to stop anything: MustFence also needs the whole Period to have been watched, and a peer that joined less than Period ago to have been reached or to have stayed silent that long. A fence already decided ends only on contact, so a peer newly added never ends one.

func (*Tracker) Has

func (t *Tracker) Has(peer string) bool

Has reports whether peer is one of the cluster's other hosts here.

func (*Tracker) Hosts

func (t *Tracker) Hosts() int

Hosts is the cluster's size as this host knows it, itself included.

func (*Tracker) MustFence

func (t *Tracker) MustFence(now time.Duration) bool

MustFence reports whether this host must stop its own apps (R-421): the cluster has at least three hosts, this host has probed without a break for at least Period, and in all of that Period it had contact with fewer than half the cluster's hosts, itself included. A peer known for less than Period and not yet reached counts as reached: a host joining is never a reason for another to fence.

func (*Tracker) Reached

func (t *Tracker) Reached(now time.Duration) int

Reached is how many of the cluster's hosts this host has had contact with in the last Period, itself included.

func (*Tracker) Round

func (t *Tracker) Round(now time.Duration, answered []string)

Round records one round of this host's probes, finished at now, and the peers that answered.

func (*Tracker) SetPeers

func (t *Tracker) SetPeers(now time.Duration, ids []string)

SetPeers makes ids the other hosts of the cluster. A peer new to the set is taken as last reached when it was added, so a host that has just joined is not counted as silent for longer than it has been known.

func (*Tracker) Unreached

func (t *Tracker) Unreached(now time.Duration) map[string]time.Duration

Unreached is UnreachedFor for every peer: what this host reports to the leader.

func (*Tracker) UnreachedFor

func (t *Tracker) UnreachedFor(now time.Duration, peer string) time.Duration

UnreachedFor is how long this host has watched peer without contact, up to now: never more than it has been probing without a break, and never more than the peer has been known. Zero for a peer outside the set.

func (*Tracker) Watched

func (t *Tracker) Watched(now time.Duration) time.Duration

Watched is how long, up to now, this host has been probing without a break. Zero when it is not probing now.

type View

type View struct {
	Host string `json:"host"`

	// UnreachedMS is, for each peer, how many milliseconds this host has
	// watched it without contact (Tracker.UnreachedFor). A duration on this
	// host's own clock, never a time.
	UnreachedMS map[string]int64 `json:"unreached_ms"`

	// SilentForMS is how long this host had watched the host that sent the
	// probe without contact, until the probe arrived: what the majority
	// would say of the sender. A host that was suspended learns from it, at
	// its first round after it wakes, that it was taken for lost while its
	// own clock stood still (Monitor.Round). Zero for the leader's collector.
	SilentForMS int64 `json:"silent_for_ms,omitempty"`

	// Fenced is whether this host is cut off and has stopped its apps.
	Fenced bool `json:"fenced"`

	// FenceID names this host's latest fence, so the leader audits each one
	// once (cluster.host.fence) however often it hears of it. Kept after the
	// host is reached again, until it fences next.
	FenceID string `json:"fence_id,omitempty"`

	// Stopped is how many app containers the latest fence stopped.
	Stopped int `json:"stopped,omitempty"`
}

View is what a host answers a probe with: its own view, so that the leader learns every host's view from the probes it makes anyway.

func (View) Unreached

func (v View) Unreached() map[string]time.Duration

Unreached is UnreachedMS as durations.

type Watch

type Watch struct {
	// Hosts lists every host of the cluster, lost ones included: a lost
	// host is still probed, to see it come back, but no longer counts
	// toward the cluster's size.
	Hosts func(ctx context.Context) ([]Peer, error)

	// Probe asks a host's agent for its View, as JSON; as Monitor.Probe.
	Probe func(ctx context.Context, p Peer) ([]byte, error)

	Service *Service
	Config  Config
	Clock   clock.Clock
	Logger  *zap.Logger
	// contains filtered or unexported fields
}

Watch is the leader's half of R-421, run as a leader job: every Interval it probes every host's agent, keeps the View each one answers with in a Ledger, audits the fences they report, and takes for lost a host more than half the cluster has not reached for three Periods.

It is the leader's because only one replica may start a standby. It reads the host list from the database, and keeps the last one when it cannot; it cannot mark a host lost without the database, so with Postgres away nothing fails over.

func (*Watch) Round

func (w *Watch) Round(ctx context.Context)

Round probes every host once and acts on what the Ledger then says. A host marked lost counts no more toward the cluster's size (R-421), as on every host's side (Monitor); with fewer than three hosts left, nobody else is taken for lost. A host whose machine answered the leader's own probe this round is never taken for lost, whatever the others report: its apps may be running.

func (*Watch) Run

func (w *Watch) Run(ctx context.Context)

Run watches until ctx ends. A new leader starts with an empty Ledger, so it takes nobody for lost before three Periods of its own.

type Witness

type Witness struct {
	// Hosts lists the cluster's hosts, as Watch.Hosts.
	Hosts func(ctx context.Context) ([]Peer, error)

	// Probe is Watch.Probe.
	Probe func(ctx context.Context, p Peer) ([]byte, error)

	// Local is this control host's Monitor's UnreachedFor; nil when it has
	// none yet.
	Local func(hostID string) (time.Duration, bool)

	Config Config
}

Witness is what a person marking a host lost is checked against (R-421, R-153). Marking a host lost starts its apps' standbys, so it is refused unless the host has been silent, to everyone who can say, for at least Period:

  • this control host's own Monitor has watched it without contact for Period (a host already marked lost is no longer watched, and is not asked this);
  • its machine does not answer a probe now, not even to turn it away;
  • no other host that answers says it had contact with it within Period.

A host whose Pando process is down while its machine runs answers the probe by refusing the connection, so it is not marked lost by hand while its apps may be running.

func (*Witness) Silent

func (w *Witness) Silent(ctx context.Context, h state.Host) error

Silent returns nil when h may be marked lost, and otherwise why not.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL