Documentation
¶
Overview ¶
Package liveness decides, from what the cluster's hosts can reach, when a host must stop its own apps and when the leader may take a host for lost (R-421, docs/design/notes-cluster.md "Deciding a host is gone, and fencing").
The decisions are pure and live in Tracker (one host's own view) and Ledger (the leader's collection of every host's view). Monitor, Watch and Service run them: probing, stopping, recording and starting standbys.
The rule, on both sides:
- A host stops its own apps only when it has reached fewer than half the cluster's hosts, itself included, for Period.
- The leader takes a host for lost only when more than half the hosts have not reached it for three times Period, so the lost host has stopped its apps before its standbys start.
- With fewer than three hosts neither happens by itself.
Everything here leans one way. A contact anywhere in a window counts the peer as reached; a gap in this host's own probing (a suspended machine, a clock that jumped) restarts the window rather than counting as silence; a report the leader has not heard lately counts for nothing. Each of those can only delay a fence or a failover, never cause one: a host that is not cut off must never stop its apps.
Durations are measured on each host's own monotonic clock. No host compares its clock with another's: what crosses the wire is how long, never when.
Index ¶
- Constants
- Variables
- func Elapsed(c clock.Clock) func() time.Duration
- type Actor
- type Apps
- type Auditor
- type Config
- type Failover
- type Fencer
- type HomedApp
- type Homes
- type HostLoss
- type Ledger
- type Monitor
- type Peer
- type Returns
- type Service
- type SkippedApp
- type Store
- type Tracker
- func (t *Tracker) Contact(now time.Duration, peer string)
- func (t *Tracker) CutOff(now time.Duration) bool
- func (t *Tracker) Has(peer string) bool
- func (t *Tracker) Hosts() int
- func (t *Tracker) MustFence(now time.Duration) bool
- func (t *Tracker) Reached(now time.Duration) int
- func (t *Tracker) Round(now time.Duration, answered []string)
- func (t *Tracker) SetPeers(now time.Duration, ids []string)
- func (t *Tracker) Unreached(now time.Duration) map[string]time.Duration
- func (t *Tracker) UnreachedFor(now time.Duration, peer string) time.Duration
- func (t *Tracker) Watched(now time.Duration) time.Duration
- type View
- type Watch
- type Witness
Constants ¶
const ( SkipNoStandby = "no_standby" SkipFailed = "failed" SkipGone = "gone" SkipNotWired = "standbys_unavailable" SkipPromoteErr = "promote_failed" // SkipAuditErr is a failover whose audit event could not be written: // it is not started without one (R-421, R-228), and the leader tries // again each round until it is. SkipAuditErr = "audit_unavailable" )
Why an app on a lost host was not failed over.
const DefaultFailoverConcurrency = 8
DefaultFailoverConcurrency is how many of a lost host's queues of apps start at once [P].
const DefaultPeriod = 30 * time.Second
DefaultPeriod is T [P: 30 s] (R-421), and cluster.liveness_period's default (config.DefaultLivenessPeriod).
const MinHosts = 3
MinHosts is the smallest cluster in which anything stops or fails over by itself (R-421). Two hosts cannot tell a dead peer from a cut link.
Variables ¶
var ErrContactOnly = errors.New("the host's machine answered without its view")
ErrContactOnly is wrapped by a probe's error when the peer's machine answered but gave no view: it turned the connection away, failed the handshake, or refused the ping. That is contact (R-421): the machine is up and on the network, though its Pando process may be down, restarting or waiting for its database, and its apps may be running. Only silence — a probe that timed out or found no route — is not contact.
Functions ¶
func Elapsed ¶
Elapsed returns a reader of c's monotonic time since the call: a duration that never goes backward.
clock.System's Now is UTC, and UTC drops Go's monotonic reading, so for the system clock this reads time.Since on a monotonic start instead: a wall clock stepped by NTP or by hand moves nothing here. For any other clock — a test's Fake — it reads Now, and a reading earlier than the last is held at the last, so a clock set backward stops time rather than reversing it.
Types ¶
type Actor ¶
type Actor struct {
Kind audit.PrincipalKind
ID string
OnBehalfOf string
RequestID string
}
Actor is who said a host is lost.
type Config ¶
type Config struct {
// Period is T: how long a host must have reached fewer than half the
// cluster before it stops its apps. DefaultPeriod when zero.
Period time.Duration
// Interval is how often a host probes every other. Period/6 when zero,
// and never more than that.
Interval time.Duration
// Timeout bounds one probe. Interval/2 when zero, and never more than
// Interval.
Timeout time.Duration
// PeerWait is how long a round waits for the list of peers before it
// probes the last one known. The list is read beside the round, never
// in it, so a database that hangs never keeps a host from probing, or
// from fencing. The smaller of Timeout and 2 s when zero.
PeerWait time.Duration
}
Config holds the timings. Zero values take the defaults.
func (Config) Fresh ¶
Fresh is how recently the leader must have heard a host's report for it to count. A host the leader has not heard from lately has said nothing.
func (Config) LostAfter ¶
LostAfter is how long a majority must not have reached a host before the leader takes it for lost: three times Period (R-421).
type Failover ¶
Failover starts an app's standby on its standby host, which becomes the app's home (R-419). The standby slice implements it; Service calls it once per app homed on a lost host that has a standby and is not failed.
type Fencer ¶
Fencer stops this host's app containers: hostagent.DockerFencer. It returns how many it stopped. It is never asked to start one: a host that has fenced never restarts its apps by itself (R-421).
type HomedApp ¶
type HomedApp struct {
AppID string
HasStandby bool
// Queue is which apps start one after another: those sharing a queue
// start in the order listed, each once the one before has ended, and
// different queues start side by side (R-422: standbys sharing one
// host past the reservation take its room in priority order).
Queue string
}
HomedApp is an app whose home is a host, and whether it has a standby.
type HostLoss ¶
type HostLoss struct {
HostID string `json:"host_id"`
Name string `json:"name"`
AlreadyLost bool `json:"already_lost"`
Started []string `json:"started"`
Skipped []SkippedApp `json:"skipped"`
}
HostLoss is what taking a host for lost did.
type Ledger ¶
type Ledger struct {
// contains filtered or unexported fields
}
Ledger is the leader's collection of what every host reports it has not reached, and the one place a host is taken for lost (R-421). Times given to it are readings of the leader's own monotonic clock; the durations in a report are the observer's own.
It is kept in memory by whichever replica leads, not in the database. A decision needs only the last three Periods of reports, and a new leader that starts empty has only to wait those three Periods again: slower, never wrong. Keeping it in Postgres would put the database on the path of a decision that must not depend on it.
A Ledger is not safe for concurrent use; Watch holds it under one goroutine.
func (*Ledger) Lost ¶
Lost reports whether x is lost (R-421): the cluster, x included, has at least three hosts, and more than half of them have not reached x for three Periods. x reaches itself, so it is never one of them.
type Monitor ¶
type Monitor struct {
Self string
// Peers lists the cluster's hosts. An error keeps the last list.
Peers func(ctx context.Context) ([]Peer, error)
// Probe pings a peer and returns the View it answered with, as JSON. An
// error wrapping ErrContactOnly is contact without a view; any other
// error is silence.
Probe func(ctx context.Context, p Peer) ([]byte, error)
Fencer Fencer
Config Config
Clock clock.Clock
Logger *zap.Logger
// contains filtered or unexported fields
}
Monitor is a host's half of R-421, run by the host's agent: it probes every other host, answers their probes with its own view, and stops this host's apps when it is the one cut off.
It needs nothing from the control plane but the list of peers, and keeps the last list it was given when the control plane cannot be asked: Postgres or every control host being down never stops an app. What stops one is this host failing to reach most of the cluster's hosts, the control hosts counted as hosts like any other, by their agents. The list is read beside the round, and the fence runs beside it too: neither a database that hangs nor a runtime slow to stop containers keeps the host from probing.
func (*Monitor) Answer ¶
Answer records contact from the peer that sent a probe, and returns this host's View as JSON. from is "" for a probe from the leader's collector, which is not a host's monitor.
func (*Monitor) Fencing ¶
Fencing reports whether a fence is decided and not lifted: from the moment this host decides it is cut off, through the stopping of its containers, until it reaches the cluster again (R-421). While it is, nothing may start a container here: one started while the others stop would run on beside the standby that replaces it.
func (*Monitor) Round ¶
Round probes every peer once, records who answered, and fences if this host must: when it has reached fewer than half the cluster for Period, or when more than half the cluster answers that it had not reached this host for as long as the leader takes a host for lost.
type Peer ¶
type Peer struct {
ID string
Name string
Address string
// Lost is that the host is marked lost (R-421). A lost host is left out
// of the cluster's size on every side — the leader's and each host's
// alike, so they agree on it — and is not probed by the hosts; the
// leader still probes it, to see it come back (R-419).
Lost bool
}
Peer is another host of the cluster, as a probe reaches it.
type Returns ¶
type Returns interface {
AppsLeftOn(ctx context.Context, hostID string) ([]string, error)
RemoveStale(ctx context.Context, appID string) error
}
Returns is what a lost host coming back needs (R-419): the apps failed over away from it, whose copies it still holds, and removing each copy, never restarting it (R-148 does not apply to a copy that was failed over). The standby slice implements it, with Failover.
type Service ¶
type Service struct {
Store Store
Apps Apps
Audit Auditor
Logger *zap.Logger
Homes Homes
Failover Failover
// Returns removes a returned host's copies before its lost mark is
// cleared. Nil leaves a lost host marked: without it nothing would
// remove the copies, and a host still marked lost starts nothing.
Returns Returns
// Witness says whether a host has been heard from lately. A host that
// has is not gone, and is not marked lost by hand: its apps may be
// running, and a standby started beside them would be a second copy
// (R-153). Nil skips the check.
Witness interface {
Silent(ctx context.Context, h state.Host) error
}
// FailoverConcurrency is how many queues of a lost host's apps start
// at once; zero is DefaultFailoverConcurrency.
FailoverConcurrency int
// contains filtered or unexported fields
}
Service takes hosts for lost and starts their apps' standbys (R-419, R-421): automatically when Watch, on the leader, finds a majority has not reached a host for three Periods, and by hand through MarkLost. It also audits the fences hosts report.
func (*Service) MarkLost ¶
MarkLost is a person saying a host is gone (R-421): the host is marked lost and its apps' standbys started. It is how a cluster under three hosts fails over at all, and is allowed in any cluster. Authorization is the caller's: install.cluster.manage. Marking a host already lost again starts the standbys of apps still homed on it, which is how a failover that failed is retried.
func (*Service) RecordFence ¶
RecordFence audits a fence a host reported, once per fence however often the leader hears of it (R-421).
type SkippedApp ¶
type SkippedApp struct {
AppID string `json:"app_id"`
Reason string `json:"reason"`
Detail string `json:"detail,omitempty"`
}
SkippedApp is an app on a lost host that was not started elsewhere.
type Store ¶
type Store interface {
Host(ctx context.Context, hostID string) (state.Host, error)
MarkHostLost(ctx context.Context, hostID, by string) (bool, error)
ClearHostLost(ctx context.Context, hostID string) error
LostHosts(ctx context.Context) (map[string]bool, error)
RecordHostFence(ctx context.Context, hostID, fenceID string) (bool, error)
}
Store is the slice of state.Cluster the service uses.
type Tracker ¶
type Tracker struct {
// contains filtered or unexported fields
}
Tracker is one host's own view: when it last had contact with each other host, and how long it has been probing without a break. Every time it is given is a reading of this host's monotonic clock (Elapsed).
Contact is either direction: this host's probe of a peer was answered, or a peer's probe of this host arrived. Counting both is what keeps the two sides of a link in agreement, so that a host the others have not reached is also a host that has not reached them (R-421).
A Tracker is not safe for concurrent use; Monitor holds it under a lock.
func (*Tracker) Contact ¶
Contact records contact with peer at now: one of this host's probes was answered, or the peer's probe arrived. A peer outside the set is ignored.
func (*Tracker) CutOff ¶
CutOff reports whether this host reaches fewer than half the cluster's hosts in the last Period. It is not yet a reason to stop anything: MustFence also needs the whole Period to have been watched, and a peer that joined less than Period ago to have been reached or to have stayed silent that long. A fence already decided ends only on contact, so a peer newly added never ends one.
func (*Tracker) MustFence ¶
MustFence reports whether this host must stop its own apps (R-421): the cluster has at least three hosts, this host has probed without a break for at least Period, and in all of that Period it had contact with fewer than half the cluster's hosts, itself included. A peer known for less than Period and not yet reached counts as reached: a host joining is never a reason for another to fence.
func (*Tracker) Reached ¶
Reached is how many of the cluster's hosts this host has had contact with in the last Period, itself included.
func (*Tracker) Round ¶
Round records one round of this host's probes, finished at now, and the peers that answered.
func (*Tracker) SetPeers ¶
SetPeers makes ids the other hosts of the cluster. A peer new to the set is taken as last reached when it was added, so a host that has just joined is not counted as silent for longer than it has been known.
func (*Tracker) Unreached ¶
Unreached is UnreachedFor for every peer: what this host reports to the leader.
func (*Tracker) UnreachedFor ¶
UnreachedFor is how long this host has watched peer without contact, up to now: never more than it has been probing without a break, and never more than the peer has been known. Zero for a peer outside the set.
type View ¶
type View struct {
Host string `json:"host"`
// UnreachedMS is, for each peer, how many milliseconds this host has
// watched it without contact (Tracker.UnreachedFor). A duration on this
// host's own clock, never a time.
UnreachedMS map[string]int64 `json:"unreached_ms"`
// SilentForMS is how long this host had watched the host that sent the
// probe without contact, until the probe arrived: what the majority
// would say of the sender. A host that was suspended learns from it, at
// its first round after it wakes, that it was taken for lost while its
// own clock stood still (Monitor.Round). Zero for the leader's collector.
SilentForMS int64 `json:"silent_for_ms,omitempty"`
// Fenced is whether this host is cut off and has stopped its apps.
Fenced bool `json:"fenced"`
// FenceID names this host's latest fence, so the leader audits each one
// once (cluster.host.fence) however often it hears of it. Kept after the
// host is reached again, until it fences next.
FenceID string `json:"fence_id,omitempty"`
// Stopped is how many app containers the latest fence stopped.
Stopped int `json:"stopped,omitempty"`
}
View is what a host answers a probe with: its own view, so that the leader learns every host's view from the probes it makes anyway.
type Watch ¶
type Watch struct {
// Hosts lists every host of the cluster, lost ones included: a lost
// host is still probed, to see it come back, but no longer counts
// toward the cluster's size.
Hosts func(ctx context.Context) ([]Peer, error)
// Probe asks a host's agent for its View, as JSON; as Monitor.Probe.
Probe func(ctx context.Context, p Peer) ([]byte, error)
Service *Service
Config Config
Clock clock.Clock
Logger *zap.Logger
// contains filtered or unexported fields
}
Watch is the leader's half of R-421, run as a leader job: every Interval it probes every host's agent, keeps the View each one answers with in a Ledger, audits the fences they report, and takes for lost a host more than half the cluster has not reached for three Periods.
It is the leader's because only one replica may start a standby. It reads the host list from the database, and keeps the last one when it cannot; it cannot mark a host lost without the database, so with Postgres away nothing fails over.
func (*Watch) Round ¶
Round probes every host once and acts on what the Ledger then says. A host marked lost counts no more toward the cluster's size (R-421), as on every host's side (Monitor); with fewer than three hosts left, nobody else is taken for lost. A host whose machine answered the leader's own probe this round is never taken for lost, whatever the others report: its apps may be running.
type Witness ¶
type Witness struct {
// Hosts lists the cluster's hosts, as Watch.Hosts.
Hosts func(ctx context.Context) ([]Peer, error)
// Probe is Watch.Probe.
Probe func(ctx context.Context, p Peer) ([]byte, error)
// Local is this control host's Monitor's UnreachedFor; nil when it has
// none yet.
Local func(hostID string) (time.Duration, bool)
Config Config
}
Witness is what a person marking a host lost is checked against (R-421, R-153). Marking a host lost starts its apps' standbys, so it is refused unless the host has been silent, to everyone who can say, for at least Period:
- this control host's own Monitor has watched it without contact for Period (a host already marked lost is no longer watched, and is not asked this);
- its machine does not answer a probe now, not even to turn it away;
- no other host that answers says it had contact with it within Period.
A host whose Pando process is down while its machine runs answers the probe by refusing the connection, so it is not marked lost by hand while its apps may be running.