Help us shape the future of Go! Take the 2026 Go Developer Survey

cluster

package
v0.2.19 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: GPL-3.0 Imports: 24 Imported by: 0

Documentation

Overview

Package cluster is Ether's multi-node runtime: node identity, liveness heartbeats, leader election and a small cross-node event bus.

A standalone server (SQLite) uses Solo: it is always the leader, its bus is in-process and it is the only member. A clustered node (PostgreSQL) uses Postgres: heartbeats in the cluster_nodes table, leadership through a PostgreSQL session advisory lock held on a dedicated connection, and an event bus over LISTEN/NOTIFY.

Singleton work (catalogue sync, plex.tv publication, scheduled backups…) runs through RunAsLeader so exactly one node performs it at a time.

Index

Constants

View Source
const (
	ModeSolo     = "solo"
	ModePostgres = "postgres"
)

Modes reported by Cluster.Mode.

View Source
const (
	// TopicMembers is published when a node joins, leaves or changes its
	// addresses. The payload is the node ID.
	TopicMembers = "cluster:members"
	// TopicResync is delivered locally (never sent to other nodes) after the
	// event bus starts listening and after every reconnect: notifications may
	// have been missed, so caches that rely on bus invalidations should drop
	// everything.
	TopicResync = "bus:resync"
)

Well-known topics.

View Source
const DefaultChannel = "ether_cluster"

DefaultChannel is the LISTEN/NOTIFY channel for the public schema; other schemas get a channel derived from their name (NOTIFY is database-wide).

View Source
const DefaultLockKey int32 = 0x45746843

DefaultLockKey is the first half of the leader advisory lock ("EthC"); the second half is hashtext(current_schema()), so nodes sharing a schema contend for one lock while unrelated schemas (test runs) never collide.

View Source
const DefaultUpdateLeaseTTL = 20 * time.Minute

DefaultUpdateLeaseTTL bounds how long a node may hold the update lease without renewing it: long enough to drain, install, restart and confirm a release, short enough that a node that died mid-update does not block the rest of the cluster for long.

View Source
const MaxPayload = 7 * 1024

MaxPayload is the largest event payload Publish accepts. PostgreSQL limits a NOTIFY payload to 8000 bytes; the envelope (topic, sender) must fit as well.

View Source
const NodeFile = "node.json"

NodeFile holds this node's identity in the data directory. It is node-local: never copy it to another node. Two live processes with one ID would overwrite each other's heartbeat, so the PostgreSQL runtime refuses to join (or leaves) when it sees another process heartbeating under its ID (ErrDuplicateNode).

Variables

View Source
var (
	// ErrPayloadTooLarge is returned by Publish for payloads over MaxPayload
	// (or whose encoded envelope would exceed PostgreSQL's NOTIFY limit).
	ErrPayloadTooLarge = errors.New("cluster: event payload too large (max 7 KB; PostgreSQL NOTIFY is limited to 8000 bytes)")
	// ErrBadTopic is returned by Publish for an empty, overlong or binary
	// topic.
	ErrBadTopic = errors.New("cluster: invalid event topic")
)
View Source
var ErrDuplicateNode = errors.New("cluster: another running node uses this node's identity")

ErrDuplicateNode means another running process uses this node's ID: its node.json was copied from another node.

View Source
var ErrVersionFailed = errors.New("cluster: this release was rolled back on another node")

ErrVersionFailed is returned by UpdateLease.Acquire for a release that was rolled back on some node of the cluster; the other nodes skip it.

Functions

func PeerAddrs

func PeerAddrs(nodes []Node, selfID string, now time.Time, grace time.Duration) []string

PeerAddrs returns the connection URIs of every node except selfID that heartbeated within grace of now and is not draining, ordered by node name (then ID) and de-duplicated. The registrar appends them to the leader's own URIs.

Types

type Caps

type Caps struct {
	// Transcode is true when ffmpeg is available on the node.
	Transcode bool `json:"transcode"`
	// HWAccel names the hardware transcoder ("videotoolbox", "nvenc", …);
	// empty or "none" for software only.
	HWAccel string `json:"hwAccel,omitempty"`
	// Weight is the node's share weight for account-wide budgets such as
	// Usenet connections (config server.clusterWeight; 0 means 1).
	Weight int `json:"weight,omitempty"`
	// EgressIP is the public address the node reaches the internet from,
	// when known (providers may flag one account used from several).
	EgressIP string `json:"egressIp,omitempty"`
}

Caps are a node's capabilities.

type Cluster

type Cluster interface {
	// Mode is ModeSolo or ModePostgres.
	Mode() string
	// Self describes this node.
	Self() Node
	// IsLeader reports whether this node currently holds leadership.
	IsLeader() bool
	// Epoch is the leadership epoch this node acquired (0 while not
	// leading; always 1 for Solo). Every acquisition anywhere in the
	// cluster increments it, so a singleton writer can fence a write with
	// "... WHERE (SELECT epoch FROM cluster_leader WHERE id=1) = ?".
	Epoch() int64
	// OnLeaderChange calls fn (synchronously, from the election goroutine;
	// keep it quick) whenever leadership is gained or lost.
	OnLeaderChange(fn func(leader bool)) (unsubscribe func())
	// RunAsLeader runs fn while this node leads, cancelling its context as
	// soon as leadership is lost and starting it again when regained. It
	// blocks until ctx ends and the last fn has returned. A fn that returns
	// on its own is not restarted until leadership is lost and regained.
	RunAsLeader(ctx context.Context, name string, fn func(ctx context.Context))
	// Publish sends payload (≤ MaxPayload) to every subscriber of topic on
	// every node, this one included (local subscribers synchronously).
	Publish(ctx context.Context, topic string, payload []byte) error
	// Subscribe registers fn for topic. fn runs on the bus goroutine (or
	// the publisher's, for local events); it must not block and must treat
	// payload as read-only.
	Subscribe(topic string, fn func(payload []byte)) (unsubscribe func())
	// Nodes lists live members, ordered by name.
	Nodes(ctx context.Context) ([]Node, error)
	// AllNodes lists every recorded member, dead or alive, ordered by name.
	AllNodes(ctx context.Context) ([]Node, error)
	// Start begins heartbeats, election and the bus (no-op for Solo).
	Start(ctx context.Context) error
	// Close leaves the cluster: stops background work, releases
	// leadership and removes this node's heartbeat row.
	Close() error
}

Cluster is the runtime the rest of the app uses.

type Health

type Health struct {
	Channel       string    `json:"channel"`
	BusConnected  bool      `json:"busConnected"`
	LastHeartbeat time.Time `json:"lastHeartbeat,omitempty"`
	// ClockSkew is this node's clock minus the database server's, measured
	// at Start.
	ClockSkew time.Duration `json:"clockSkewNs"`
}

Health is the runtime's own state for diagnostics.

type Identity

type Identity struct {
	// NodeID is generated on first run and persisted in NodeFile.
	NodeID string `json:"nodeId"`
	// Name is the display name: NodeFile's "name" when the owner set one,
	// else the hostname.
	Name string `json:"name,omitempty"`
}

Identity names this node within a cluster.

func LoadIdentity

func LoadIdentity(dataDir string) (Identity, error)

LoadIdentity reads dataDir/node.json, creating it with a fresh node ID on first run. A corrupt file is an error rather than silently replaced, as a new ID would make the node a different cluster member.

type LeaseHolder

type LeaseHolder struct {
	NodeID     string    `json:"nodeId"`
	Version    string    `json:"version"`
	AcquiredAt time.Time `json:"acquiredAt"`
	ExpiresAt  time.Time `json:"expiresAt"`
}

LeaseHolder describes the current lease.

type Local

type Local struct {
	Identity Identity
	Version  string
	Addrs    func(ctx context.Context) []string
	Caps     func() Caps
	Load     func() int
	// Draining reports whether the node is draining for an update; a
	// draining node also hands leadership to another live node.
	Draining func() bool
	// Migrations are the schema migrations this binary knows
	// (store.KnownMigrations), heartbeated for the schema compatibility
	// check.
	Migrations map[string]int
}

Local describes this node to the runtime. Addrs, Caps and Load are optional and read on every heartbeat.

type Node

type Node struct {
	ID      string `json:"id"`
	Name    string `json:"name"`
	Version string `json:"version"`
	// Addrs are the connection URIs this node would publish to plex.tv
	// (LAN, WAN and custom URLs, as the registrar builds them).
	Addrs     []string  `json:"addrs"`
	Caps      Caps      `json:"caps"`
	Load      int       `json:"load"` // active playback sessions
	StartedAt time.Time `json:"startedAt"`
	LastSeen  time.Time `json:"lastSeen"`
	Leader    bool      `json:"leader"`
	// Live is true when the node heartbeated recently (always true from
	// Nodes, may be false from AllNodes).
	Live bool `json:"live"`
	// Draining is true while the node prepares to restart for an update:
	// it serves its existing sessions but admits no new ones, and its
	// addresses are left out of the published connections.
	Draining bool `json:"draining,omitempty"`
	// Migrations are the schema migrations the node's binary knows
	// (component -> highest version); nil when the node predates this
	// report.
	Migrations map[string]int `json:"migrations,omitempty"`
}

Node is one cluster member.

type Options

type Options struct {
	// DB is the node's pool on the shared database (store "ether-pgx").
	DB    *sql.DB
	Local Local
	// ConnConfig is used for the dedicated leader-lock and LISTEN
	// connections; nil copies the configuration of a pool connection, so
	// they reach the same server, database and search_path.
	ConnConfig *pgx.ConnConfig
	Logger     *slog.Logger
	Now        func() time.Time

	HeartbeatInterval time.Duration // heartbeat upsert (5s)
	LiveAfter         time.Duration // a node is live if seen within this (20s)
	PruneAfter        time.Duration // rows dead this long are deleted (24h)
	ElectionInterval  time.Duration // lock attempt / leader probe (2s)
	ProbeTimeout      time.Duration // bound on each probe query (3s)
	// StepDownHold keeps a node that just lost leadership out of the
	// election for a while so a healthy node takes over (5s).
	StepDownHold time.Duration
	// IdleSessionTimeout is set as idle_session_timeout on the lock
	// connection: if this node freezes or is partitioned from the database,
	// the server ends the session and releases the lock after this long
	// rather than waiting for TCP keepalives (15s; must exceed
	// ElectionInterval).
	IdleSessionTimeout time.Duration
	ListenPing         time.Duration // ping an idle LISTEN connection (15s)
	MinBackoff         time.Duration // bus reconnect backoff (500ms)
	MaxBackoff         time.Duration // (30s)
	AddrTimeout        time.Duration // bound on Local.Addrs (30s)
	// IdentityPath names this node's node.json in duplicate-node errors.
	IdentityPath string

	LockKey int32  // DefaultLockKey
	Channel string // derived from the schema when empty

	// Availability, when set, receives every heartbeat's outcome: the
	// node's shared view of database reachability, which lets deferrable
	// writes queue at once during an outage and replay as soon as a
	// heartbeat succeeds again.
	Availability *store.Availability

	// OnFatal is called once, from a runtime goroutine, when this node
	// must stop serving as a cluster member: another live process
	// heartbeats under the same node ID (a cloned machine or copied data
	// directory). The runtime has already stopped heartbeating, released
	// leadership and left the bus when it is called. The error wraps
	// ErrDuplicateNode.
	OnFatal func(error)
}

Options configure the PostgreSQL runtime. Zero durations take the defaults in parentheses.

type Postgres

type Postgres struct {
	// contains filtered or unexported fields
}

Postgres is the clustered runtime.

Heartbeats: every node upserts its cluster_nodes row every HeartbeatInterval with last_seen = its own clock in Unix milliseconds; a node is live while now-last_seen <= LiveAfter on the reader's clock. Timestamps are bound Go values (portable SQL, no server now()), so the scheme tolerates clock skew up to LiveAfter-HeartbeatInterval (15s by default); Start warns when this node's clock is more than a quarter of LiveAfter away from the database server's. Nodes should run NTP.

Leadership: a dedicated connection holds pg_try_advisory_lock; the holder probes it with SELECT 1 every ElectionInterval and steps down the moment a probe fails (subscribers and RunAsLeader contexts are cancelled immediately). Each acquisition increments cluster_leader.epoch: a leader that was frozen past IdleSessionTimeout may believe it leads for up to one ElectionInterval after waking, so singleton writes that must never overlap should be fenced on Epoch.

Bus: a dedicated connection LISTENs on one channel; Publish delivers to local subscribers immediately and sends pg_notify through the pool. The sender's own echo is ignored. After the LISTEN connection is re-established TopicResync is delivered locally, because notifications sent while it was down are lost; it is also delivered after the first LISTEN, covering events sent before this node joined the bus.

func NewPostgres

func NewPostgres(o Options) (*Postgres, error)

NewPostgres validates options; call Start to join the cluster.

func (*Postgres) AllNodes

func (p *Postgres) AllNodes(ctx context.Context) ([]Node, error)

AllNodes lists every recorded member with its liveness.

func (*Postgres) Beat

func (p *Postgres) Beat()

Beat heartbeats now instead of at the next interval (after this node's draining state changed, so the leader republishes promptly).

func (*Postgres) BusConnected

func (p *Postgres) BusConnected() bool

BusConnected reports whether the LISTEN connection is up.

func (*Postgres) Channel

func (p *Postgres) Channel() string

Channel is the NOTIFY channel (empty before Start).

func (*Postgres) Close

func (p *Postgres) Close() error

Close leaves the cluster.

func (*Postgres) Epoch

func (p *Postgres) Epoch() int64

func (*Postgres) Err

func (p *Postgres) Err() error

Err is the fatal error that ended this node's membership, if any.

func (*Postgres) Health

func (p *Postgres) Health() Health

Health reports bus, heartbeat and clock state.

func (*Postgres) IsLeader

func (p *Postgres) IsLeader() bool

func (*Postgres) Mode

func (p *Postgres) Mode() string

func (*Postgres) Nodes

func (p *Postgres) Nodes(ctx context.Context) ([]Node, error)

Nodes lists live members.

func (*Postgres) OnLeaderChange

func (p *Postgres) OnLeaderChange(fn func(bool)) func()

func (*Postgres) Publish

func (p *Postgres) Publish(ctx context.Context, topic string, payload []byte) error

Publish delivers to local subscribers, then notifies the other nodes. Before Start the event is only delivered locally.

func (*Postgres) RunAsLeader

func (p *Postgres) RunAsLeader(ctx context.Context, name string, fn func(context.Context))

func (*Postgres) Self

func (p *Postgres) Self() Node

Self is this node as of its last heartbeat.

func (*Postgres) Start

func (p *Postgres) Start(ctx context.Context) error

Start joins the cluster: first heartbeat, then the heartbeat, address, election and bus goroutines. It waits briefly for the bus to listen so events published right after Start are received.

func (*Postgres) StepDown

func (p *Postgres) StepDown(ctx context.Context)

StepDown gives up leadership for good (until the process restarts) and releases the leader lock at once, so another node takes over within an election interval instead of waiting for this node to leave the cluster. Call it at the start of a graceful shutdown. It waits (bounded by ctx) for the lock to be released.

func (*Postgres) Subscribe

func (p *Postgres) Subscribe(topic string, fn func([]byte)) func()

type Solo

type Solo struct {
	// contains filtered or unexported fields
}

Solo is the standalone runtime: the only member, always the leader, with an in-process bus. It starts no goroutines.

func NewSolo

func NewSolo(local Local, log *slog.Logger) *Solo

NewSolo returns the standalone runtime for local.

func (*Solo) AllNodes

func (s *Solo) AllNodes(context.Context) ([]Node, error)

func (*Solo) Close

func (s *Solo) Close() error

func (*Solo) Epoch

func (s *Solo) Epoch() int64

func (*Solo) IsLeader

func (s *Solo) IsLeader() bool

func (*Solo) Mode

func (s *Solo) Mode() string

func (*Solo) Nodes

func (s *Solo) Nodes(context.Context) ([]Node, error)

func (*Solo) OnLeaderChange

func (s *Solo) OnLeaderChange(fn func(bool)) func()

OnLeaderChange never fires: a standalone server is always the leader.

func (*Solo) Publish

func (s *Solo) Publish(_ context.Context, topic string, payload []byte) error

Publish delivers to local subscribers after the same validation the clustered bus applies, so oversized events fail in standalone tests too.

func (*Solo) RunAsLeader

func (s *Solo) RunAsLeader(ctx context.Context, _ string, fn func(context.Context))

RunAsLeader runs fn(ctx) directly.

func (*Solo) Self

func (s *Solo) Self() Node

Self describes this node; Addrs is read with a short timeout.

func (*Solo) Start

func (s *Solo) Start(context.Context) error

func (*Solo) StepDown

func (s *Solo) StepDown(context.Context)

StepDown is a no-op: a standalone server always leads.

func (*Solo) Subscribe

func (s *Solo) Subscribe(topic string, fn func([]byte)) func()

type UpdateLease

type UpdateLease struct {
	DB     *sql.DB
	NodeID string
	// TTL is the lease lifetime from each acquisition or renewal
	// (DefaultUpdateLeaseTTL).
	TTL time.Duration
	Now func() time.Time
}

UpdateLease serialises self-updates across a cluster: a node installs a release only while it holds the lease, so nodes restart one at a time. The lease is one row (cluster_update_lease) taken with a conditional upsert — free, expired or already ours — and renewed by the holder while it drains, installs, restarts and confirms the new version. Times are Unix milliseconds from the caller's clock (see Postgres for the clock policy).

func (*UpdateLease) Acquire

func (l *UpdateLease) Acquire(ctx context.Context, version string) (bool, error)

Acquire takes the lease for installing version, or renews it when this node already holds it. It reports false while another node holds an unexpired lease, and ErrVersionFailed when version rolled back on some node.

func (*UpdateLease) Holder

func (l *UpdateLease) Holder(ctx context.Context) (*LeaseHolder, error)

Holder returns the current unexpired lease, or nil when the lease is free.

func (*UpdateLease) Release

func (l *UpdateLease) Release(ctx context.Context) error

Release gives the lease back if this node holds it.

func (*UpdateLease) ReportFailed

func (l *UpdateLease) ReportFailed(ctx context.Context, version string) error

ReportFailed records that version rolled back on this node — the other nodes will not install it — and releases the lease if this node holds it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL