Documentation
¶
Overview ¶
Package cluster is Ether's multi-node runtime: node identity, liveness heartbeats, leader election and a small cross-node event bus.
A standalone server (SQLite) uses Solo: it is always the leader, its bus is in-process and it is the only member. A clustered node (PostgreSQL) uses Postgres: heartbeats in the cluster_nodes table, leadership through a PostgreSQL session advisory lock held on a dedicated connection, and an event bus over LISTEN/NOTIFY.
Singleton work (catalogue sync, plex.tv publication, scheduled backups…) runs through RunAsLeader so exactly one node performs it at a time.
Index ¶
- Constants
- Variables
- func PeerAddrs(nodes []Node, selfID string, now time.Time, grace time.Duration) []string
- type Caps
- type Cluster
- type Health
- type Identity
- type LeaseHolder
- type Local
- type Node
- type Options
- type Postgres
- func (p *Postgres) AllNodes(ctx context.Context) ([]Node, error)
- func (p *Postgres) Beat()
- func (p *Postgres) BusConnected() bool
- func (p *Postgres) Channel() string
- func (p *Postgres) Close() error
- func (p *Postgres) Epoch() int64
- func (p *Postgres) Err() error
- func (p *Postgres) Health() Health
- func (p *Postgres) IsLeader() bool
- func (p *Postgres) Mode() string
- func (p *Postgres) Nodes(ctx context.Context) ([]Node, error)
- func (p *Postgres) OnLeaderChange(fn func(bool)) func()
- func (p *Postgres) Publish(ctx context.Context, topic string, payload []byte) error
- func (p *Postgres) RunAsLeader(ctx context.Context, name string, fn func(context.Context))
- func (p *Postgres) Self() Node
- func (p *Postgres) Start(ctx context.Context) error
- func (p *Postgres) StepDown(ctx context.Context)
- func (p *Postgres) Subscribe(topic string, fn func([]byte)) func()
- type Solo
- func (s *Solo) AllNodes(context.Context) ([]Node, error)
- func (s *Solo) Close() error
- func (s *Solo) Epoch() int64
- func (s *Solo) IsLeader() bool
- func (s *Solo) Mode() string
- func (s *Solo) Nodes(context.Context) ([]Node, error)
- func (s *Solo) OnLeaderChange(fn func(bool)) func()
- func (s *Solo) Publish(_ context.Context, topic string, payload []byte) error
- func (s *Solo) RunAsLeader(ctx context.Context, _ string, fn func(context.Context))
- func (s *Solo) Self() Node
- func (s *Solo) Start(context.Context) error
- func (s *Solo) StepDown(context.Context)
- func (s *Solo) Subscribe(topic string, fn func([]byte)) func()
- type UpdateLease
Constants ¶
const ( ModeSolo = "solo" ModePostgres = "postgres" )
Modes reported by Cluster.Mode.
const ( // TopicMembers is published when a node joins, leaves or changes its // addresses. The payload is the node ID. TopicMembers = "cluster:members" // TopicResync is delivered locally (never sent to other nodes) after the // event bus starts listening and after every reconnect: notifications may // have been missed, so caches that rely on bus invalidations should drop // everything. TopicResync = "bus:resync" )
Well-known topics.
const DefaultChannel = "ether_cluster"
DefaultChannel is the LISTEN/NOTIFY channel for the public schema; other schemas get a channel derived from their name (NOTIFY is database-wide).
const DefaultLockKey int32 = 0x45746843
DefaultLockKey is the first half of the leader advisory lock ("EthC"); the second half is hashtext(current_schema()), so nodes sharing a schema contend for one lock while unrelated schemas (test runs) never collide.
const DefaultUpdateLeaseTTL = 20 * time.Minute
DefaultUpdateLeaseTTL bounds how long a node may hold the update lease without renewing it: long enough to drain, install, restart and confirm a release, short enough that a node that died mid-update does not block the rest of the cluster for long.
const MaxPayload = 7 * 1024
MaxPayload is the largest event payload Publish accepts. PostgreSQL limits a NOTIFY payload to 8000 bytes; the envelope (topic, sender) must fit as well.
const NodeFile = "node.json"
NodeFile holds this node's identity in the data directory. It is node-local: never copy it to another node. Two live processes with one ID would overwrite each other's heartbeat, so the PostgreSQL runtime refuses to join (or leaves) when it sees another process heartbeating under its ID (ErrDuplicateNode).
Variables ¶
var ( // ErrPayloadTooLarge is returned by Publish for payloads over MaxPayload // (or whose encoded envelope would exceed PostgreSQL's NOTIFY limit). ErrPayloadTooLarge = errors.New("cluster: event payload too large (max 7 KB; PostgreSQL NOTIFY is limited to 8000 bytes)") // ErrBadTopic is returned by Publish for an empty, overlong or binary // topic. ErrBadTopic = errors.New("cluster: invalid event topic") )
var ErrDuplicateNode = errors.New("cluster: another running node uses this node's identity")
ErrDuplicateNode means another running process uses this node's ID: its node.json was copied from another node.
var ErrVersionFailed = errors.New("cluster: this release was rolled back on another node")
ErrVersionFailed is returned by UpdateLease.Acquire for a release that was rolled back on some node of the cluster; the other nodes skip it.
Functions ¶
Types ¶
type Caps ¶
type Caps struct {
// Transcode is true when ffmpeg is available on the node.
Transcode bool `json:"transcode"`
// HWAccel names the hardware transcoder ("videotoolbox", "nvenc", …);
// empty or "none" for software only.
HWAccel string `json:"hwAccel,omitempty"`
// Weight is the node's share weight for account-wide budgets such as
// Usenet connections (config server.clusterWeight; 0 means 1).
Weight int `json:"weight,omitempty"`
// EgressIP is the public address the node reaches the internet from,
// when known (providers may flag one account used from several).
EgressIP string `json:"egressIp,omitempty"`
}
Caps are a node's capabilities.
type Cluster ¶
type Cluster interface {
// Mode is ModeSolo or ModePostgres.
Mode() string
// Self describes this node.
Self() Node
// IsLeader reports whether this node currently holds leadership.
IsLeader() bool
// Epoch is the leadership epoch this node acquired (0 while not
// leading; always 1 for Solo). Every acquisition anywhere in the
// cluster increments it, so a singleton writer can fence a write with
// "... WHERE (SELECT epoch FROM cluster_leader WHERE id=1) = ?".
Epoch() int64
// OnLeaderChange calls fn (synchronously, from the election goroutine;
// keep it quick) whenever leadership is gained or lost.
OnLeaderChange(fn func(leader bool)) (unsubscribe func())
// RunAsLeader runs fn while this node leads, cancelling its context as
// soon as leadership is lost and starting it again when regained. It
// blocks until ctx ends and the last fn has returned. A fn that returns
// on its own is not restarted until leadership is lost and regained.
RunAsLeader(ctx context.Context, name string, fn func(ctx context.Context))
// Publish sends payload (≤ MaxPayload) to every subscriber of topic on
// every node, this one included (local subscribers synchronously).
Publish(ctx context.Context, topic string, payload []byte) error
// Subscribe registers fn for topic. fn runs on the bus goroutine (or
// the publisher's, for local events); it must not block and must treat
// payload as read-only.
Subscribe(topic string, fn func(payload []byte)) (unsubscribe func())
// Nodes lists live members, ordered by name.
Nodes(ctx context.Context) ([]Node, error)
// AllNodes lists every recorded member, dead or alive, ordered by name.
AllNodes(ctx context.Context) ([]Node, error)
// Start begins heartbeats, election and the bus (no-op for Solo).
Start(ctx context.Context) error
// Close leaves the cluster: stops background work, releases
// leadership and removes this node's heartbeat row.
Close() error
}
Cluster is the runtime the rest of the app uses.
type Health ¶
type Health struct {
Channel string `json:"channel"`
BusConnected bool `json:"busConnected"`
LastHeartbeat time.Time `json:"lastHeartbeat,omitempty"`
// ClockSkew is this node's clock minus the database server's, measured
// at Start.
ClockSkew time.Duration `json:"clockSkewNs"`
}
Health is the runtime's own state for diagnostics.
type Identity ¶
type Identity struct {
// NodeID is generated on first run and persisted in NodeFile.
NodeID string `json:"nodeId"`
// Name is the display name: NodeFile's "name" when the owner set one,
// else the hostname.
Name string `json:"name,omitempty"`
}
Identity names this node within a cluster.
func LoadIdentity ¶
LoadIdentity reads dataDir/node.json, creating it with a fresh node ID on first run. A corrupt file is an error rather than silently replaced, as a new ID would make the node a different cluster member.
type LeaseHolder ¶
type LeaseHolder struct {
NodeID string `json:"nodeId"`
Version string `json:"version"`
AcquiredAt time.Time `json:"acquiredAt"`
ExpiresAt time.Time `json:"expiresAt"`
}
LeaseHolder describes the current lease.
type Local ¶
type Local struct {
Identity Identity
Version string
Addrs func(ctx context.Context) []string
Caps func() Caps
Load func() int
// Draining reports whether the node is draining for an update; a
// draining node also hands leadership to another live node.
Draining func() bool
// Migrations are the schema migrations this binary knows
// (store.KnownMigrations), heartbeated for the schema compatibility
// check.
Migrations map[string]int
}
Local describes this node to the runtime. Addrs, Caps and Load are optional and read on every heartbeat.
type Node ¶
type Node struct {
ID string `json:"id"`
Name string `json:"name"`
Version string `json:"version"`
// Addrs are the connection URIs this node would publish to plex.tv
// (LAN, WAN and custom URLs, as the registrar builds them).
Addrs []string `json:"addrs"`
Caps Caps `json:"caps"`
Load int `json:"load"` // active playback sessions
StartedAt time.Time `json:"startedAt"`
LastSeen time.Time `json:"lastSeen"`
Leader bool `json:"leader"`
// Live is true when the node heartbeated recently (always true from
// Nodes, may be false from AllNodes).
Live bool `json:"live"`
// Draining is true while the node prepares to restart for an update:
// it serves its existing sessions but admits no new ones, and its
// addresses are left out of the published connections.
Draining bool `json:"draining,omitempty"`
// Migrations are the schema migrations the node's binary knows
// (component -> highest version); nil when the node predates this
// report.
Migrations map[string]int `json:"migrations,omitempty"`
}
Node is one cluster member.
type Options ¶
type Options struct {
// DB is the node's pool on the shared database (store "ether-pgx").
DB *sql.DB
Local Local
// ConnConfig is used for the dedicated leader-lock and LISTEN
// connections; nil copies the configuration of a pool connection, so
// they reach the same server, database and search_path.
ConnConfig *pgx.ConnConfig
Logger *slog.Logger
Now func() time.Time
HeartbeatInterval time.Duration // heartbeat upsert (5s)
LiveAfter time.Duration // a node is live if seen within this (20s)
PruneAfter time.Duration // rows dead this long are deleted (24h)
ElectionInterval time.Duration // lock attempt / leader probe (2s)
ProbeTimeout time.Duration // bound on each probe query (3s)
// StepDownHold keeps a node that just lost leadership out of the
// election for a while so a healthy node takes over (5s).
StepDownHold time.Duration
// IdleSessionTimeout is set as idle_session_timeout on the lock
// connection: if this node freezes or is partitioned from the database,
// the server ends the session and releases the lock after this long
// rather than waiting for TCP keepalives (15s; must exceed
// ElectionInterval).
IdleSessionTimeout time.Duration
ListenPing time.Duration // ping an idle LISTEN connection (15s)
MinBackoff time.Duration // bus reconnect backoff (500ms)
MaxBackoff time.Duration // (30s)
AddrTimeout time.Duration // bound on Local.Addrs (30s)
// IdentityPath names this node's node.json in duplicate-node errors.
IdentityPath string
LockKey int32 // DefaultLockKey
Channel string // derived from the schema when empty
// Availability, when set, receives every heartbeat's outcome: the
// node's shared view of database reachability, which lets deferrable
// writes queue at once during an outage and replay as soon as a
// heartbeat succeeds again.
Availability *store.Availability
// OnFatal is called once, from a runtime goroutine, when this node
// must stop serving as a cluster member: another live process
// heartbeats under the same node ID (a cloned machine or copied data
// directory). The runtime has already stopped heartbeating, released
// leadership and left the bus when it is called. The error wraps
// ErrDuplicateNode.
OnFatal func(error)
}
Options configure the PostgreSQL runtime. Zero durations take the defaults in parentheses.
type Postgres ¶
type Postgres struct {
// contains filtered or unexported fields
}
Postgres is the clustered runtime.
Heartbeats: every node upserts its cluster_nodes row every HeartbeatInterval with last_seen = its own clock in Unix milliseconds; a node is live while now-last_seen <= LiveAfter on the reader's clock. Timestamps are bound Go values (portable SQL, no server now()), so the scheme tolerates clock skew up to LiveAfter-HeartbeatInterval (15s by default); Start warns when this node's clock is more than a quarter of LiveAfter away from the database server's. Nodes should run NTP.
Leadership: a dedicated connection holds pg_try_advisory_lock; the holder probes it with SELECT 1 every ElectionInterval and steps down the moment a probe fails (subscribers and RunAsLeader contexts are cancelled immediately). Each acquisition increments cluster_leader.epoch: a leader that was frozen past IdleSessionTimeout may believe it leads for up to one ElectionInterval after waking, so singleton writes that must never overlap should be fenced on Epoch.
Bus: a dedicated connection LISTENs on one channel; Publish delivers to local subscribers immediately and sends pg_notify through the pool. The sender's own echo is ignored. After the LISTEN connection is re-established TopicResync is delivered locally, because notifications sent while it was down are lost; it is also delivered after the first LISTEN, covering events sent before this node joined the bus.
func NewPostgres ¶
NewPostgres validates options; call Start to join the cluster.
func (*Postgres) Beat ¶
func (p *Postgres) Beat()
Beat heartbeats now instead of at the next interval (after this node's draining state changed, so the leader republishes promptly).
func (*Postgres) BusConnected ¶
BusConnected reports whether the LISTEN connection is up.
func (*Postgres) OnLeaderChange ¶
func (*Postgres) Publish ¶
Publish delivers to local subscribers, then notifies the other nodes. Before Start the event is only delivered locally.
func (*Postgres) RunAsLeader ¶
func (*Postgres) Start ¶
Start joins the cluster: first heartbeat, then the heartbeat, address, election and bus goroutines. It waits briefly for the bus to listen so events published right after Start are received.
func (*Postgres) StepDown ¶
StepDown gives up leadership for good (until the process restarts) and releases the leader lock at once, so another node takes over within an election interval instead of waiting for this node to leave the cluster. Call it at the start of a graceful shutdown. It waits (bounded by ctx) for the lock to be released.
type Solo ¶
type Solo struct {
// contains filtered or unexported fields
}
Solo is the standalone runtime: the only member, always the leader, with an in-process bus. It starts no goroutines.
func (*Solo) OnLeaderChange ¶
OnLeaderChange never fires: a standalone server is always the leader.
func (*Solo) Publish ¶
Publish delivers to local subscribers after the same validation the clustered bus applies, so oversized events fail in standalone tests too.
func (*Solo) RunAsLeader ¶
RunAsLeader runs fn(ctx) directly.
type UpdateLease ¶
type UpdateLease struct {
DB *sql.DB
NodeID string
// TTL is the lease lifetime from each acquisition or renewal
// (DefaultUpdateLeaseTTL).
TTL time.Duration
Now func() time.Time
}
UpdateLease serialises self-updates across a cluster: a node installs a release only while it holds the lease, so nodes restart one at a time. The lease is one row (cluster_update_lease) taken with a conditional upsert — free, expired or already ours — and renewed by the holder while it drains, installs, restarts and confirms the new version. Times are Unix milliseconds from the caller's clock (see Postgres for the clock policy).
func (*UpdateLease) Acquire ¶
Acquire takes the lease for installing version, or renews it when this node already holds it. It reports false while another node holds an unexpired lease, and ErrVersionFailed when version rolled back on some node.
func (*UpdateLease) Holder ¶
func (l *UpdateLease) Holder(ctx context.Context) (*LeaseHolder, error)
Holder returns the current unexpired lease, or nil when the lease is free.
func (*UpdateLease) Release ¶
func (l *UpdateLease) Release(ctx context.Context) error
Release gives the lease back if this node holds it.
func (*UpdateLease) ReportFailed ¶
func (l *UpdateLease) ReportFailed(ctx context.Context, version string) error
ReportFailed records that version rolled back on this node — the other nodes will not install it — and releases the lease if this node holds it.