cluster

package
v0.1.23 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package cluster lets several Perfuse servers share one PostgreSQL store and decide, between them, which runs what.

The problem it solves

One server is one point of failure: when it stops, every interface stops. Two servers on one database solve that only if they never both run the same channel, because two MLLP listeners on one feed take turns with the sender, two queue workers deliver the same message twice, and two DICOM pollers each report the same study.

How

A lease is a row naming who holds something, with a counter its holder moves on at every renewal, every few seconds. No clock is compared with another: a holder that cannot renew (crashed, cut off from the database) stops what the lease covers once its own clock says the lease is nearly done, and another server takes it only after its own clock has watched the counter stand still for the whole lease. Each measures an interval on its own clock, so the servers' clocks, and the database's, need not agree; they need only run at about the same rate. A clock that jumps (a virtual machine resynchronising, an administrator setting the time) changes nothing: the first version used the database's clock for expiry, and a test found a database whose clock jumped thirteen seconds hand every lease to a new server while the old ones still ran them.

Between the holder stopping and the next one starting there is a short gap with nobody running, which costs a sender a few seconds of refused connections, and never a moment with two.

Two modes:

  • active-passive: one lease, "leader". Its holder runs every channel and every job that must run once (subscription notifications, alerts). The others wait, and the first to see it lapse takes over.
  • active-active: the leader still runs the jobs, and every channel has a lease of its own, spread evenly over the live servers. A server that leaves gives its channels to the rest; one that joins takes its share as they come free.

Each lease carries an epoch, counted up whenever its holder changes, which names a holding in the history.

Index

Constants

View Source 
const LeaderLease = "leader"

LeaderLease is the lease whose holder runs what must run once.

View Source 
const MaxFileSize = 8 << 20

MaxFileSize bounds one replicated file: these are configuration, and a bigger one is a mistake (a database copied into the channels directory) that should not be shipped to every server.

Variables

View Source 
var ErrNeedsPostgres = errors.New("cluster: servers can share only a PostgreSQL store; give -db a postgres:// URL")

ErrNeedsPostgres is a cluster asked for on a SQLite store, which only one server can open.

Functions

func ChannelLease 

func ChannelLease(tenant, channel string) string

ChannelLease is the lease name of one channel.

func NodeID 

func NodeID(dir string) (string, error)

NodeID reads this server's id from dir, or makes one and writes it there.

Types

type Area 

type Area struct {
	Name string
	Dir  string
	// Keep chooses the files, by slash-separated path within Dir. Nil keeps every regular file.
	Keep func(rel string) bool
}

Area is a directory, or some files in one, kept the same everywhere.

type Config 

type Config struct {
	Mode Mode
	// NodeID is stable for a server: it is kept in its data directory (NodeID reads or makes it).
	NodeID string
	// Name is what people see. Default: the node id.
	Name string
	// Address is the URL of this server's console and API, as the others and a load balancer reach it.
	Address string
	// Version is the build, shown beside each node: a rolling upgrade has two for a while.
	Version string
	// TTL is how long a lease lasts without renewal: how long a crashed holder blocks its work. Default 15s.
	TTL time.Duration
	// Renew is how often leases are renewed and taken. Default TTL/5.
	Renew time.Duration
}

Config describes this server in the cluster.

type Coordinator 

type Coordinator struct {
	// contains filtered or unexported fields
}

Coordinator holds and renews this server's leases.

func New 

func New(ctx context.Context, db *sql.DB, cfg Config, hooks Hooks, log *slog.Logger) (*Coordinator, error)

New prepares the cluster tables and a coordinator for this server.

func (*Coordinator) Config 

func (c *Coordinator) Config() Config

Config is this server's configuration, defaults filled in.

func (*Coordinator) Desired 

func (c *Coordinator) Desired(ctx context.Context) (map[string]bool, error)

Desired reads every recorded start or stop: tenant/channel to whether it should run. A channel with no record runs if its file enables it.

func (*Coordinator) Healthy 

func (c *Coordinator) Healthy() error

Healthy reports whether the last renewal reached the store.

func (*Coordinator) Holder 

func (c *Coordinator) Holder(ctx context.Context, lease string) (id, name string)

Holder names the server holding a lease now (its id and name), or "" when none does: as this server sees it, a lease whose counter has not moved for a whole lease has no holder.

func (*Coordinator) Holds 

func (c *Coordinator) Holds(lease string) bool

Holds reports whether this server holds a lease now, by its own clock.

func (*Coordinator) IsLeader 

func (c *Coordinator) IsLeader() bool

IsLeader reports whether this server is the leader now.

func (*Coordinator) Move 

func (c *Coordinator) Move(ctx context.Context, lease, toNode, by string) error

Move asks for a channel to run on another server. Its holder stops it and lets it go at its next renewal, and the server asked for takes it.

func (*Coordinator) Prune 

func (c *Coordinator) Prune(ctx context.Context, keep time.Duration) error

Prune forgets events older than keep.

func (*Coordinator) Run 

func (c *Coordinator) Run(ctx context.Context)

Run renews and takes leases until ctx ends, then gives up what it holds so another server takes over at once.

func (*Coordinator) Runs 

func (c *Coordinator) Runs(tenant, channel string) bool

Runs reports whether this server is the one to run a channel now.

func (*Coordinator) SetDesired 

func (c *Coordinator) SetDesired(ctx context.Context, tenant, channel string, running bool, by string) error

SetDesired records whether a channel should run, wherever it runs: a person's start or stop on any server.

func (*Coordinator) Status 

func (c *Coordinator) Status(ctx context.Context) (Status, error)

Status reads the whole cluster.

func (*Coordinator) StepDown 

func (c *Coordinator) StepDown(ctx context.Context, for_ time.Duration, by string) error

StepDown gives up leadership and keeps away from it for a while, so another server takes over: for maintenance on this one without stopping it.

type Event 

type Event struct {
	At     time.Time `json:"at"`
	Node   string    `json:"node"`
	Kind   string    `json:"kind"`
	Lease  string    `json:"lease,omitempty"`
	Detail string    `json:"detail,omitempty"`
}

Event is a lease gained, lost, moved or given up.

type Files 

type Files struct {

	// Changed is told which files another server changed here (area, slash-separated path, deleted).
	Changed func(area, rel string, deleted bool)
	// contains filtered or unexported fields
}

Files keeps directories the same on every server of a cluster, through the shared store: the channel files, the settings and alert files, sign-in configuration. Each server keeps its own copy on its own disk, so a channel loads from a file as it always has, and a server that loses the store keeps what it had.

A change made on any server (through the console, by hand, by a git pull) is noticed at the next scan and sent to the store; every other server writes it into its own copy at its next look and is told which file changed, so a running channel restarts with it. A deletion travels the same way. When the same file changes on two servers between looks, the one that reached the store first wins, and the other's version is kept beside it as NAME.conflict-NODE for a person to compare.

func NewFiles 

func NewFiles(ctx context.Context, db *sql.DB, node string, areas []Area, log *slog.Logger) (*Files, error)

NewFiles prepares the table.

func (*Files) Run 

func (f *Files) Run(ctx context.Context, every time.Duration)

Run syncs every interval until ctx ends.

func (*Files) Sync 

func (f *Files) Sync(ctx context.Context) error

Sync takes what others changed, then sends what changed here.

type Hooks 

type Hooks struct {
	Gain func(lease string)
	Lose func(lease string)
	// Channels lists the channel leases there should be (active-active), from the channels configured.
	Channels func() []string
}

Hooks are what the coordinator calls when this server gains or loses a lease. They run on the coordinator's goroutine, one at a time; Lose must return only once the work has stopped.

type Lease 

type Lease struct {
	Name     string    `json:"name"`
	Holder   string    `json:"holder"`
	Epoch    int64     `json:"epoch"`
	Expires  time.Time `json:"expiresAt"`
	Acquired time.Time `json:"acquiredAt"`
	WantedBy string    `json:"wantedBy,omitempty"`
	Lapsed   bool      `json:"lapsed"`
}

Lease is one lease as the store has it.

type Mode 

type Mode string

Mode is how the servers divide the work. 

const (
	// ActivePassive runs everything on the leader.
	ActivePassive Mode = "active-passive"
	// ActiveActive spreads the channels over every live server.
	ActiveActive Mode = "active-active"
)

type Node 

type Node struct {
	ID       string    `json:"id"`
	Name     string    `json:"name"`
	Address  string    `json:"address"`
	Version  string    `json:"version"`
	Mode     string    `json:"mode"`
	Started  time.Time `json:"startedAt"`
	Seen     time.Time `json:"seenAt"`
	Live     bool      `json:"live"`
	Leaving  bool      `json:"leaving"`
	Leader   bool      `json:"leader"`
	Channels []string  `json:"channels,omitempty"`
	This     bool      `json:"this"`
}

Node is one server of the cluster.

type Status 

type Status struct {
	Mode   Mode    `json:"mode"`
	This   string  `json:"this"`
	Leader string  `json:"leader,omitempty"`
	TTL    float64 `json:"leaseSeconds"`
	Nodes  []Node  `json:"nodes"`
	Leases []Lease `json:"leases"`
	Events []Event `json:"events"`
	Error  string  `json:"error,omitempty"`
	// Members lists, per channel lease, the channels it covers when more than one: channels that route to one another
	// run on one server and move together.
	Members map[string][]string `json:"members,omitempty"`
}

Status is the cluster as a console shows it.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.