Documentation
¶
Overview ¶
Package standby keeps each app's warm standby (R-419, R-420, notes-cluster.md "Warm standby").
The pass runs on the leader. For each app on a runtime that can keep a standby, and that host policy or its own spec says should have one, it has the runtime make or refresh the stopped standby when the app's pinned spec changed, and ships the app's volumes to it when a copy is due [P: every 5 minutes]. An app whose standby is turned off, or that is deleted, has its standby removed; a deleted app's copy a failover left on a lost host is kept on record until the host returns and it is removed (RemoveStale). Nothing here decides a host is lost: that is the liveness pass's (R-421), which calls Promote.
A failed app is not failed over (R-151): Promote refuses it, and the pass leaves its standby as it was rather than refreshing it.
One app's standby has one writer at a time (R-153). In this process the pass and a promotion take the app's lock (Service.lock); across processes every write is conditional on the row being as it was read (SaveIf, on the row's version), so a pass working from a reading taken before a promotion never puts the old home back. The pass leaves alone an app whose home is lost or whose promotion is under way: that standby is the failover's.
Index ¶
- Constants
- type AppStates
- type Auditor
- type Failover
- func (f *Failover) AppsHomedOn(ctx context.Context, hostID string) ([]liveness.HomedApp, error)
- func (f *Failover) AppsLeftOn(ctx context.Context, hostID string) ([]string, error)
- func (f *Failover) PromoteStandby(ctx context.Context, appID string) error
- func (f *Failover) RemoveStale(ctx context.Context, appID string) error
- type Homes
- type Hosts
- type Plans
- type PolicyLoader
- type Runtimes
- type Service
- func (s *Service) Once(ctx context.Context)
- func (s *Service) Promote(ctx context.Context, appID, by string) (api.PromoteResult, error)
- func (s *Service) RemoveStale(ctx context.Context, appID, by string) error
- func (s *Service) Run(ctx context.Context)
- func (s *Service) Standby(ctx context.Context, appID string) (notice.StandbyStanding, error)
- type Services
- type Store
Constants ¶
const ( DefaultEvery = time.Minute DefaultShipEvery = 5 * time.Minute )
DefaultEvery is how often the pass looks at every app, and DefaultShipEvery how old an app's newest copy may get before another is shipped (R-420 [P]): cluster.standby_every's and cluster.ship_every's defaults.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Failover ¶
Failover is the standby side of the liveness pass (R-419, R-421): which apps a lost host held, starting their standbys, and, once the host is back, removing the copies it still holds. The liveness pass decides a host is lost or back; this never does. Hosts are the cluster's by ID there and the runtime's by name here: the runtime's host list names each joined host as cluster_hosts does (hosts.Runtime.Sync).
func (*Failover) AppsHomedOn ¶
AppsHomedOn is every app whose home is the host, highest failover priority first, then by app: the order a lost host's apps are started in, so that past the reservation (allow_failover_oversubscription) the ones that matter most take the room, and one that no longer fits is skipped for the next (R-422, multidocker failoverOrder). An app has a standby when one is placed on another host.
The priority is read with the rows, from each app's pinned spec (Standbys.HomedOn), not from a plan built per app, so a big host's failover starts its first standby at once (R-421).
The liveness pass starts apps in different queues side by side. Past the reservation the standbys on one host share a queue, so they start in this order, each into what the one before left; under it every standby fits, and each app is a queue of its own. Host policy that cannot be read is taken as past it.
func (*Failover) AppsLeftOn ¶
AppsLeftOn is every app failed over away from the host, whose stale copy the host still holds.
func (*Failover) PromoteStandby ¶
PromoteStandby starts the app's standby, by the liveness pass. A failed app is refused there too (R-151).
type Homes ¶
type Homes interface {
HomedOn(ctx context.Context, host string) ([]state.Standby, error)
LeftOn(ctx context.Context, host string) ([]state.Standby, error)
}
Homes is the standby state the liveness pass reads. *state.Standbys is one.
type Plans ¶
type Plans interface {
Shape(ctx context.Context, appID string) (api.BundlePlan, *spec.AppSpec, error)
Complete(ctx context.Context, plan *api.BundlePlan, s *spec.AppSpec) error
}
Plans builds what an app is started with. *reconciler.Reconciler is one. Shape is cheap and read on every pass; Complete resolves the pull credential and the environment, and is asked only before something is created from the plan.
type PolicyLoader ¶
PolicyLoader reads host policy.
type Runtimes ¶
type Runtimes interface {
Runtime(ref string) (api.RuntimeAdapter, bool)
}
Runtimes resolves runtime adapters by reference.
type Service ¶
type Service struct {
Store Store
Plans Plans
Runtimes Runtimes
Policy PolicyLoader
Apps AppStates
// Services and Auditor are optional. Without Services every provisioned
// service counts as copied with the volumes.
Services Services
Auditor Auditor
// RuntimeRefs lists the runtime adapters. When none of them can keep a
// standby, the pass builds no app's plan: a single-host install does no
// standby work at all. Nil asks each app's runtime.
RuntimeRefs func() []string
// LocalFenced reports whether this host is fenced (R-421): cut off from
// most of the cluster, so nothing here is placed, shipped or started.
// Nil is never.
LocalFenced func() bool
Clock clock.Clock
Logger *zap.Logger
// Every and ShipEvery override the defaults; Keep is the copies kept
// (zero is api.DefaultStandbyCopies), HealthTimeout what a promoted copy
// is given to come up (zero is the runtime's default).
Every time.Duration
ShipEvery time.Duration
Keep int
HealthTimeout time.Duration
// contains filtered or unexported fields
}
Service is the standby pass, and the actions on one app's standby.
func (*Service) Promote ¶
Promote starts an app's standby as the app, on its standby's host, once the app's host is lost (R-419). Whether it is lost is the caller's to say: the liveness pass when a majority of hosts agree (R-421), or a person with install.cluster.manage who says the host is gone. by is that person's ID, empty for the liveness pass.
A failed app is refused (R-151). From here the app's home is the standby's host, the host it left holds a stale copy (RemoveStale), and the next pass gives the app a new standby.
The promotion is written ahead (R-153): before the runtime starts anything the row says the app's home is the standby's host and a promotion is under way, so a promotion cut short — an error, a lost lease, a restart — never leaves the app running on the standby's host while Pando records it on the lost one. Only a refusal before anything was started (the runtime names no host) puts the row back.
func (*Service) RemoveStale ¶
RemoveStale removes the copy an app left on the host it was failed over from, once that host is back (R-419): it is removed, never started. That host then becomes where the app's next standby goes, unless the app has one elsewhere already. A deleted app's copy is removed too, keeping its volumes when the delete kept the app's storage (R-204), and its row is then forgotten. Whether the host is back is the caller's to say. While the app's promotion is under way nothing is removed and the caller is told to try again, so the host stays marked lost until it is done.
func (*Service) Standby ¶
Standby is where the app's standby stands, for its notice (R-422): when the last try to place it found no host with room, the runtime's message, which names the largest room any host has, and its remedy. Otherwise empty. The app deployed either way: no room is never a blocker. An app whose standby could not be refreshed still has the one it had, so it is not told it runs without one.
type Store ¶
type Store interface {
Candidates(ctx context.Context) ([]state.StandbyApp, error)
Get(ctx context.Context, appID string) (state.Standby, bool, error)
// SaveIf writes sb only if the row is still prev (nil: no row), and
// reports whether it did.
SaveIf(ctx context.Context, sb state.Standby, prev *state.Standby) (bool, error)
// SaveIfLive is SaveIf that also writes only while the app is live and
// not failed: a promotion's write-ahead (R-151).
SaveIfLive(ctx context.Context, sb state.Standby, prev *state.Standby) (bool, error)
// Delete forgets the app's standby, unless a promotion of it is under
// way.
Delete(ctx context.Context, appID string) error
// Fate reads whether the app is deleted, and what its delete said of
// its storage (R-204).
Fate(ctx context.Context, appID string) (state.AppFate, error)
}
Store is the standby state. *state.Standbys is one.
Source Files