cluster

package
v1.29.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 10, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package cluster provides a generic substrate for consistent routing, exclusive ownership, and stream-interest tracking of namespaced keys across a fleet of servers.

It is built from four decoupled layers:

  1. Membership: each process registers a member record with a fresh instance ID and keeps it alive with a heartbeat counter. Peers poll the registry and judge liveness by observing counter movement against their own local clock (never by comparing wall clocks across machines).

  2. Routing: a pure function over the live member set. Rendezvous hashing (HRW) deterministically elects the preferred owner for a key; overrides and per-namespace placement hooks filter candidacy first.

  3. Ownership: a claim record per (namespace, key), acquired lazily on first demand and validated against the owner's liveness. Routing decides who should own; the claim record is the sole arbiter of who does. Non-owners receive a redirect and forward.

  4. Subscriptions: a non-exclusive interest registry per (namespace, key) topic — which servers currently host live streams for the topic, one row per interested server no matter how many local streams ride it. There is no arbitration and no fence; a row simply stops counting the moment its member stops being live.

Ownership is an accelerator, not an availability gate: consumers must keep a store-serialized fallback path so that losing an owner degrades to contention, never to unavailability or corruption. Subscription-driven delivery is a hint, not a contract: consumers must keep a pull backstop (e.g. sequence-log delta sync) so a missed delivery is healed, never lost.

Index

Constants

View Source
const OverrideInstanceLabel = "instance"

OverrideInstanceLabel is the reserved selector key matching Member.InstanceID.

Variables

View Source
var (
	// ErrMemberNotFound indicates the member record does not exist (e.g. a
	// heartbeat after the record was deleted or garbage-collected).
	ErrMemberNotFound = errors.New("cluster member not found")

	// ErrClaimNotFound indicates no active claim exists for the key.
	ErrClaimNotFound = errors.New("claim not found")

	// ErrClaimHeld indicates the claim is actively held by another member (or
	// a takeover's evidence no longer held at commit time). AcquireClaim
	// returns the current holder alongside this error so the caller can
	// redirect without a second read.
	ErrClaimHeld = errors.New("claim held by another member")

	// ErrNoMembers indicates routing found no eligible members for a key.
	ErrNoMembers = errors.New("no eligible cluster members")
)
View Source
var ErrObserverMembership = errors.New("observer membership cannot register interest")

ErrObserverMembership is returned by Subscribe on a runtime backed by an observer membership. An observer has no member record, so a row it wrote would never count (a row's validity is its member's liveness) and would eventually be swept as a corpse. Observers resolve and forward; only registered members host streams.

View Source
var ErrOwnershipLost = errors.New("cluster ownership force-released with work in flight")

ErrOwnershipLost is the cancellation cause delivered to fn's context when its claim is force-released with the call still in flight (a drain deadline expiring, or a liveness-session purge). It means a successor may now legitimately own the key: fn should stop and not trust its claim further. Retrieve it with context.Cause.

View Source
var ErrSubscriptionsDraining = errors.New("cluster subscriptions are draining")

ErrSubscriptionsDraining is returned by Subscribe once Drain has begun: a registration accepted mid-drain would write a row nothing will clean up.

Functions

This section is empty.

Types

type Claim

type Claim struct {
	Namespace       string
	Key             []byte
	OwnerInstanceID string
	OwnerAddress    string
	Fence           uint64
}

Claim is exclusive ownership of a key within a namespace. Fence increases monotonically per key on every change of owner (never on an owner's re-acquire), so a displaced owner's writes can be rejected by any store that checks it. Fencing is opt-in per namespace: consumers whose data store already serializes writes (e.g. via conditional writes) do not need it.

func (*Claim) Clone

func (c *Claim) Clone() *Claim

Clone returns a deep copy.

type ClaimStore

type ClaimStore interface {
	// AcquireClaim atomically acquires (namespace, key) for self. It succeeds
	// if the claim is absent, released, or already held by self (an owner's
	// re-acquire returns the existing claim with the fence unchanged).
	//
	// If takeover is non-nil, the claim may additionally be displaced from
	// takeover.InstanceID, but only if — atomically at commit time — that
	// member's registry record is absent or its heartbeat counter still equals
	// takeover.HeartbeatCounter. The fence increments on every change of
	// owner, including a fresh acquire after a release.
	//
	// When the claim is (still) held by another member, the current holder is
	// returned alongside ErrClaimHeld.
	AcquireClaim(ctx context.Context, namespace string, key []byte, self *Member, takeover *TakeoverTarget) (*Claim, error)

	// GetClaim returns the active claim for (namespace, key), or
	// ErrClaimNotFound if the key is unclaimed or released.
	GetClaim(ctx context.Context, namespace string, key []byte) (*Claim, error)

	// ReleaseClaim releases the claim if it is currently held by instanceID; a
	// release by anyone else is a no-op. The fence survives release, so a
	// later acquire of the same key still increments monotonically.
	ReleaseClaim(ctx context.Context, namespace string, key []byte, instanceID string) error
}

ClaimStore persists ownership claims. It is the sole arbiter of ownership: routing only makes contention rare.

type Member

type Member struct {
	InstanceID string
	Address    string
	Labels     map[string]string
	Draining   bool
}

Member is a single server process in the cluster. InstanceID is unique per process incarnation — a restarted server registers a new ID, so claims held by a previous incarnation are never confused with the current one.

func (*Member) Clone

func (m *Member) Clone() *Member

Clone returns a deep copy.

type MemberRecord

type MemberRecord struct {
	Member
	HeartbeatCounter uint64
}

MemberRecord is a member's registry row, including the heartbeat counter whose movement (not value) is the liveness signal.

type Membership

type Membership struct {
	// contains filtered or unexported fields
}

Membership registers this process in the cluster and maintains an observed view of every other member. Liveness is judged KCL-style: a peer is live while its heartbeat counter keeps moving across this process's own observation timeline — wall clocks are never compared across machines, so clock skew cannot produce false takeovers.

A Membership constructed with NewObserver observes without registering: it maintains the same live view but writes no member record (see NewObserver for the exact semantics of the self-referential methods).

func NewMembership

func NewMembership(log *zap.Logger, store RegistryStore, self *Member, cfg MembershipConfig) *Membership

NewMembership creates the membership runtime for self. Call Start to register and begin heartbeating.

func NewObserver

func NewObserver(log *zap.Logger, store RegistryStore, cfg MembershipConfig) *Membership

NewObserver creates a read-only membership runtime: it polls the registry and maintains the observed live view exactly like a member, but never registers a record of its own — it cannot be routed to, holds no claims or subscriptions, and leaves nothing behind on exit. For processes that consume the cluster without serving it, e.g. an event-streaming-only client that resolves owners and forwards but hosts nothing.

On an observer: Self returns nil; SetDraining and Deregister are vacuous no-ops (there is no record to flip or delete); OnSessionLost never fires (there is no liveness session to lose); and SelfHealthy reports whether registry polls are landing — a stale view, not a stale heartbeat, is the failure mode an observer must surface.

func (*Membership) Deregister

func (m *Membership) Deregister(ctx context.Context) error

Deregister removes self from the registry and stops the loops. Call only after all owned claims are released: a deregistered member's claims are instantly displaceable. On an observer it only stops the poll loop — there is no record to remove.

func (*Membership) HeartbeatInterval

func (m *Membership) HeartbeatInterval() time.Duration

HeartbeatInterval returns the member's heartbeat cadence.

func (*Membership) Live

func (m *Membership) Live() []*Member

Live returns the members currently considered live, draining members included (they hold valid claims); routing filters draining out of candidacy separately.

func (*Membership) LivenessInfo

func (m *Membership) LivenessInfo(instanceID string) (counter uint64, staleFor time.Duration, ok bool)

LivenessInfo returns this process's observation of a member: the last heartbeat counter seen and how long ago (on our clock) it last changed. ok is false if the member has never been observed. The pair (counter, staleness) is the evidence a takeover presents to the claim store.

func (*Membership) LivenessWindow

func (m *Membership) LivenessWindow() time.Duration

LivenessWindow returns the configured window after which an unmoving heartbeat means dead.

func (*Membership) OnSessionLost

func (m *Membership) OnSessionLost(fn func())

OnSessionLost registers fn to be called after this member's liveness session was interrupted and re-established: its registry record was deleted (GC'd by a peer, so every held claim became displaceable) or its heartbeats gapped past SelfUnhealthyAfter (so peers may have displaced claims via takeover). Local ownership state can no longer be trusted; subscribers should shed it and let demand re-acquire. Called from the heartbeat loop's goroutine. Never fires on an observer: with no registration there is no session.

func (*Membership) PollInterval

func (m *Membership) PollInterval() time.Duration

PollInterval returns the roster poll cadence.

func (*Membership) Refresh

func (m *Membership) Refresh(ctx context.Context) error

Refresh re-reads the registry and rebuilds the live set immediately. The poll loop calls this on its interval; tests call it to converge deterministically.

func (*Membership) Self

func (m *Membership) Self() *Member

Self returns this process's member identity (with its current draining flag), or nil on an observer — which has no identity to route to.

func (*Membership) SelfHealthy

func (m *Membership) SelfHealthy() bool

SelfHealthy reports whether this process's own heartbeats are landing. Once they have failed for SelfUnhealthyAfter, a suspicious peer may already be able to displace this member's claims, so it must stop trusting them.

An observer has no heartbeat; for it this reports whether registry polls are landing (last successful refresh within SelfUnhealthyAfter), since a stale view is the observer analogue of a stale heartbeat.

func (*Membership) SetDraining

func (m *Membership) SetDraining(ctx context.Context, draining bool) error

SetDraining flips this member's draining flag in the registry and locally. Draining removes the member from routing candidacy while its heartbeat keeps held claims valid — the first step of graceful shutdown. On an observer it is a vacuous no-op: an observer was never a routing candidate.

func (*Membership) Start

func (m *Membership) Start(ctx context.Context) error

Start registers self (observers skip registration), performs an initial refresh, and launches the heartbeat and poll loops (observers run only the poll loop). The loops run until Stop (or Deregister).

func (*Membership) Stop

func (m *Membership) Stop()

Stop halts the heartbeat and poll loops without deregistering: peers will observe the heartbeat go stale and expire this member the hard way. Prefer Deregister for a graceful exit.

func (*Membership) Subscribe

func (m *Membership) Subscribe(fn func())

Subscribe registers fn to be called whenever the live set changes (join, leave, death, or a draining flip). fn is invoked from the poll loop and must not block; do real work on another goroutine.

type MembershipConfig

type MembershipConfig struct {
	// HeartbeatInterval is how often the member's own heartbeat counter is
	// advanced. Default 5s.
	HeartbeatInterval time.Duration

	// PollInterval is how often the registry is re-read to refresh the live
	// set. Must be well under LivenessWindow. Default 2s.
	PollInterval time.Duration

	// LivenessWindow is how long a peer's heartbeat counter may sit unchanged
	// (in this process's own observations) before the peer is considered dead.
	// Default 15s.
	LivenessWindow time.Duration

	// SelfUnhealthyAfter is how long this process's own heartbeat writes may
	// fail before it must assume the rest of the cluster considers it dead and
	// stop serving owned keys. Defaulted to the session-gap floor so Do starts
	// refusing owned keys no later than a suspicious peer could displace them;
	// raising it toward LivenessWindow trades a wider potential dual-serving
	// window for fewer fallback-path detours during store blips. Default:
	// min(SessionGapThreshold, LivenessWindow).
	SelfUnhealthyAfter time.Duration

	// SessionGapThreshold is the gap between successful heartbeat writes at
	// which the member treats its own liveness session as interrupted (firing
	// OnSessionLost, which sheds local ownership) — whether the intervening
	// beats were merely delayed or failing outright. Aligned by default with
	// the suspicion floor (HeartbeatInterval + PollInterval, capped at an
	// explicitly set SelfUnhealthyAfter): a heartbeat gap past that floor is
	// exactly the window in which a peer holding a failed-forward report may
	// have displaced this member's claims while it still looked healthy to
	// itself — shedding then bounds any such dual ownership at the gap's
	// length instead of IdleTTL. Default: HeartbeatInterval + PollInterval,
	// capped at SelfUnhealthyAfter when that is set.
	SessionGapThreshold time.Duration

	// MemberGCAfter is how long a member's heartbeat may sit unchanged before
	// any observer actively deletes its registry record. Store-level TTLs are
	// lazy (DynamoDB's can lag days), and a lingering corpse record misleads
	// freshly started observers — a first-sighted record is presumed live for
	// a full LivenessWindow. Must be much larger than LivenessWindow; a
	// wrongly deleted (merely wedged) member re-registers on its next
	// heartbeat. Default: 10 × LivenessWindow.
	MemberGCAfter time.Duration
}

MembershipConfig tunes the membership runtime. Zero values take defaults.

type NamespaceHooks

type NamespaceHooks struct {
	OnAcquired func(ctx context.Context, key []byte)
	OnReleased func(ctx context.Context, key []byte)
}

NamespaceHooks are a consumer's ownership lifecycle callbacks. OnAcquired runs after a claim is won and before any work is served under it (warm state); OnReleased runs after the key's in-flight work has quiesced and before the claim is released (flush state). Hooks must not assume they run exactly once per key: a key can be acquired, idle-released, and re-acquired indefinitely.

OnReleased's context is detached from whatever triggered the release and carries a DrainDeadline timeout, so the flush is never aborted by a drain budget expiring or the runtime stopping. Hooks must honor their context: the runtime cannot interrupt a hook that ignores it, and a hung OnReleased blocks its release wave (and therefore Drain) indefinitely.

Hooks should not panic. A panicking OnAcquired retires the key's ownership (the half-warmed entry is dropped and the claim handed back for demand to re-acquire) and the panic propagates to Do's caller. A panicking OnReleased propagates through whichever goroutine is releasing — for the background reaper and rescan loops that is fatal to the process.

type NotOwnerError

type NotOwnerError struct {
	Redirect *Member
}

NotOwnerError is returned by Ownership.Do when this server does not (and should not) own the key. A non-nil Redirect identifies the member believed to hold or deserve the claim: forward the request there. A nil Redirect means no healthy owner is known (e.g. this server's own heartbeats are failing, or the claim is mid-handoff): fall back to the store-serialized path or retry.

func (*NotOwnerError) Error

func (e *NotOwnerError) Error() string

type Overrides

type Overrides interface {
	Get(ctx context.Context, namespace string, key []byte) (selector map[string]string, ok bool, err error)
}

Overrides pins individual keys to a class of members, consulted before hashing. It is the operational escape hatch for moving a hot key onto dedicated capacity (and the seam where a load-aware assigner could plug in later) — normally empty.

A pin is a label selector, not an instance ID: instance IDs die with each process incarnation, so an instance-level pin would silently expire on the pinned server's first deploy. A selector names deploy-stable intent (e.g. {"pool": "xl"}): candidates matching every selector entry form the pool, and hashing elects deterministically within it — so a multi-member pool needs no further configuration and a deploy inside the pool hands off to another pool member. The reserved key "instance" matches a member's InstanceID, for the rare pin that truly means one incarnation. A selector matching no eligible candidate is ignored, failing open to hashing over the full candidate set.

type Ownership

type Ownership struct {
	// contains filtered or unexported fields
}

Ownership acquires and serves exclusive key ownership on top of routing and claims. Keys are acquired lazily on first demand, held stickily until idle or rerouted, and drained gracefully on shutdown. It never blocks availability: any path that cannot own returns NotOwnerError so the caller can forward to the holder or fall back to its store-serialized path.

func NewOwnership

func NewOwnership(log *zap.Logger, membership *Membership, router *Router, claims ClaimStore, cfg OwnershipConfig) *Ownership

NewOwnership creates the ownership runtime. Call Start to begin the idle reaper and membership rescans.

Ownership requires a registered membership: a claim's validity is its holder's liveness, which an observer (NewObserver) does not have. Panics on an observer membership — construction-time misuse, not a runtime condition.

func (*Ownership) Do

func (o *Ownership) Do(ctx context.Context, namespace string, key []byte, fn func(ctx context.Context, claim *Claim) error) error

Do runs fn under ownership of (namespace, key), acquiring the claim if this server is the routed owner and the key is unclaimed (or its holder is provably dead). It returns NotOwnerError when the key belongs elsewhere: a non-nil Redirect is the member to forward to; a nil Redirect means no healthy owner is known and the caller should use its store-serialized fallback. Any other error is fn's own.

fn's context derives from ctx and is additionally cancelled — with ErrOwnershipLost as its cause — if the claim is force-released while fn is still running (a drain deadline expiring under it, or a liveness-session purge). That cancellation means a successor may already own the key; fn should stop, and any write it must still land needs its own protection (Claim.Fence, or the consumer's store-serialized path).

func (*Ownership) Drain

func (o *Ownership) Drain(ctx context.Context) error

Drain gracefully hands off everything this server owns: mark draining in the registry (leaving routing candidacy while heartbeats keep held claims valid), then quiesce, flush, and release every owned key. Call on shutdown before Membership.Deregister. Respects ctx as an overall deadline; keys not drained in time are force-released.

func (*Ownership) NoteUnreachable

func (o *Ownership) NoteUnreachable(instanceID string)

NoteUnreachable records that a forward to the member failed. Combined with a heartbeat stale for at least SuspicionWindow, this makes the member's claims takeover-eligible ahead of the full liveness window — corroborated suspicion, so one network blip alone never displaces anyone.

func (*Ownership) OwnedKeys

func (o *Ownership) OwnedKeys(namespace string) [][]byte

OwnedKeys returns the keys currently owned in the namespace.

func (*Ownership) RegisterNamespace

func (o *Ownership) RegisterNamespace(namespace string, hooks NamespaceHooks)

RegisterNamespace installs a namespace's lifecycle hooks. Register before serving traffic for the namespace.

func (*Ownership) Resume

func (o *Ownership) Resume(ctx context.Context) error

Resume returns a drained (or drain-aborted) instance to service: new ownership may be acquired again, and the member re-enters routing candidacy as peers observe the cleared flag. Only for callers that deliberately abort a shutdown after Drain — never on the way to exit, where accepting new keys again would strand them.

func (*Ownership) Start

func (o *Ownership) Start(ctx context.Context)

Start launches the idle reaper and subscribes to membership changes for rebalance drains.

func (*Ownership) Stop

func (o *Ownership) Stop()

Stop halts the background loops without draining. Use Drain first for a graceful shutdown.

type OwnershipConfig

type OwnershipConfig struct {
	// IdleTTL is how long an owned key may go unused before its claim is
	// released. Deliberately long: claims are sticky, and only liveness is
	// fresh — a short TTL converts the coordination plane's cheapest property
	// (claims touched only at activity boundaries) into churn. Default 15m.
	IdleTTL time.Duration

	// ReapInterval is how often idle claims are scanned for release.
	// Default 1m.
	ReapInterval time.Duration

	// DrainDeadline bounds how long a key's in-flight work may delay its
	// release during handoff or shutdown; past it the release proceeds
	// regardless (the consumer's store-serialized fallback makes a forced
	// release safe). It is also the timeout on the detached contexts handed
	// to OnReleased flushes and claim-release writes — a context-honoring
	// hook therefore bounds each release at roughly 3× DrainDeadline
	// (quiesce, then flush). Default 5s.
	DrainDeadline time.Duration

	// SuspicionWindow is the minimum heartbeat staleness at which a holder
	// reported unreachable (via NoteUnreachable) becomes takeover-eligible
	// ahead of the full LivenessWindow. Default 5s; floored at construction
	// to the membership's HeartbeatInterval + PollInterval — anything lower
	// leaves zero tolerance for ordinary heartbeat jitter, displacing owners
	// whose beat merely landed late.
	SuspicionWindow time.Duration

	// RedirectCacheTTL bounds how long a non-owner reuses a resolved redirect
	// for a key without re-reading the claim. Keeps the forward path from
	// paying a store read per request on hot keys; a stale redirect
	// self-corrects (the target answers NotOwner, or a failed forward
	// invalidates via NoteUnreachable). Default 1s.
	RedirectCacheTTL time.Duration
}

OwnershipConfig tunes the ownership runtime. Zero values take defaults.

type Placement

type Placement func(ctx context.Context, key []byte, candidates []*Member) ([]*Member, error)

Placement filters or reorders the candidate members eligible to own a key in a namespace, before hashing. It is the seam for role- and region-aware routing: e.g. restricting a namespace to members labeled with an owner role, or to the key's home region. Returning the input unchanged means "any live member". Placement must be deterministic given the same inputs — every server runs it independently and must reach the same answer.

type RegistryStore

type RegistryStore interface {
	// PutMember creates (or replaces) the member's record with the given
	// heartbeat counter. Instance IDs are unique per process incarnation, so a
	// replace only ever overwrites this process's own registration.
	//
	// The counter must never repeat within an instance ID's lifetime: takeover
	// evidence is "counter X was observed stale", checked as equality at
	// commit time, so a re-registration after record deletion must resume
	// strictly above the last value it wrote — resetting would let a peer's
	// stale prior-epoch evidence displace a live owner.
	PutMember(ctx context.Context, member *Member, heartbeatCounter uint64) error

	// Heartbeat advances the member's heartbeat counter and returns the new
	// value. Returns ErrMemberNotFound if the record no longer exists.
	Heartbeat(ctx context.Context, instanceID string) (uint64, error)

	// SetDraining updates the member's draining flag. A draining member is
	// excluded from routing candidacy but remains live, so claims it still
	// holds stay valid while it hands them off. Returns ErrMemberNotFound if
	// the record no longer exists.
	SetDraining(ctx context.Context, instanceID string, draining bool) error

	// DeleteMember removes the member's record. Idempotent.
	DeleteMember(ctx context.Context, instanceID string) error

	// GetMembers returns all registered member records.
	GetMembers(ctx context.Context) ([]*MemberRecord, error)
}

RegistryStore persists cluster membership. Liveness is never derived inside the store: records carry a heartbeat counter, and observers judge liveness by watching the counter move against their own clocks. Store-level TTLs, where an implementation has them, are garbage collection only — never correctness.

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router deterministically elects the member that should own a key: live, non-draining members are filtered by the namespace's Placement, overrides are consulted, and rendezvous hashing decides among what remains. It is a pure function over the membership snapshot — no I/O on the lookup path — and only says who *should* own a key; the claim record decides who does.

func NewRouter

func NewRouter(membership *Membership, overrides Overrides) *Router

NewRouter creates a router over the membership's live set. A nil overrides installs the empty implementation.

func (*Router) Owner

func (r *Router) Owner(ctx context.Context, namespace string, key []byte) (*Member, error)

Owner returns the member that should own the key, or ErrNoMembers if no eligible candidate exists.

func (*Router) OwnerExcluding

func (r *Router) OwnerExcluding(ctx context.Context, namespace string, key []byte, exclude map[string]bool) (*Member, error)

OwnerExcluding is Owner with additional members removed from candidacy — used to route around a member the caller has evidence is dead or unreachable, without waiting for the live set to converge.

func (*Router) RegisterPlacement

func (r *Router) RegisterPlacement(namespace string, placement Placement)

RegisterPlacement installs the namespace's placement hook.

type Store

type Store interface {
	RegistryStore
	ClaimStore
	SubscriptionStore
}

Store combines the registry, claim, and subscription stores. Implementations back all three from the same technology so the takeover condition can span registry and claims atomically.

type Subscription

type Subscription struct {
	Namespace  string
	Key        []byte
	InstanceID string
	// Address is the member's dialable endpoint, resolved by the Subscriptions
	// runtime from the membership view at resolution time — rows don't store
	// it. It is set on every Subscription a live resolution returns, and empty
	// on raw store reads.
	Address string
}

Subscription is one member's registered interest in a (namespace, key) topic: "this server hosts at least one live stream for this topic — deliver the topic's events here". Unlike a Claim it is non-exclusive (a topic holds one row per interested server) and carries no fence: there is nothing to serialize. A subscription counts only while its member is live, judged by the same heartbeat observation as claims — so steady state needs no row refreshes, and a dead member's rows are ignored immediately regardless of when they get swept.

func (*Subscription) Clone

func (s *Subscription) Clone() *Subscription

Clone returns a deep copy.

type SubscriptionHandle

type SubscriptionHandle struct {
	// contains filtered or unexported fields
}

SubscriptionHandle is one local stream's registration against a topic. Close it when the stream ends; the topic's registry row is removed when the last local handle closes. Close is idempotent.

func (*SubscriptionHandle) Close

func (h *SubscriptionHandle) Close(ctx context.Context) error

Close releases this handle's registration.

type SubscriptionStore

type SubscriptionStore interface {
	// PutSubscription registers (or reasserts) the instance's interest row for
	// the topic. Idempotent upsert. Rows carry identity only — consumers
	// resolve dial addresses through the membership view at resolution time.
	PutSubscription(ctx context.Context, namespace string, key []byte, instanceID string) error

	// PutSubscriptions registers the instance's interest rows for every listed
	// topic, as PutSubscription does for one, in as few store round trips as
	// the backend allows. Duplicate topics collapse. It is not atomic: on
	// error an arbitrary subset of the rows may have been written — acceptable
	// because a subscription row is interest, not ownership, and the runtime
	// sweeps its own refcount-less rows at resolution time.
	PutSubscriptions(ctx context.Context, topics []SubscriptionTopic, instanceID string) error

	// DeleteSubscription removes the member's interest row for the topic.
	// Idempotent.
	DeleteSubscription(ctx context.Context, namespace string, key []byte, instanceID string) error

	// DeleteSubscriptions removes the member's interest rows for every listed
	// topic, as DeleteSubscription does for one, in as few store round trips
	// as the backend allows. Duplicate topics collapse; idempotent. Not
	// atomic: on error an arbitrary subset of the rows may have been deleted
	// — the survivors stop counting once the member stops heartbeating, and
	// are swept as corpse rows.
	DeleteSubscriptions(ctx context.Context, topics []SubscriptionTopic, instanceID string) error

	// GetSubscribers returns every interest row for the topic, dead members'
	// rows included — liveness filtering is the caller's job.
	GetSubscribers(ctx context.Context, namespace string, key []byte) ([]*Subscription, error)
}

SubscriptionStore persists stream-interest registrations: one row per (namespace, key, instance) meaning "this member hosts live streams for this topic". Rows are non-exclusive and unfenced — there is nothing to arbitrate.

A row counts only while its member is live (the same heartbeat-observation rule as claims), so implementations must not attach row-level TTLs: nothing refreshes a held row (steady state is deliberately write-free), and a row expiring under a live subscriber would silently stop delivery to it. Cleanup is explicit instead — drains delete their own rows, and observers sweep crashed instances' rows at resolution time.

type SubscriptionTopic

type SubscriptionTopic struct {
	Namespace string
	Key       []byte
}

SubscriptionTopic names one (namespace, key) topic, for batch registration.

type Subscriptions

type Subscriptions struct {
	// contains filtered or unexported fields
}

Subscriptions tracks which servers host live streams for each (namespace, key) topic — the non-exclusive sibling of Ownership. A subscription is interest, not ownership: any number of members may subscribe to a topic, there is no fence and no takeover, and a dead member's rows simply stop counting (validity is the member's liveness, exactly as for claims).

The registry names servers, not streams: the first local stream for a topic writes one row, later streams ride it for free, and the last close removes it — so a topic's row count (and a publisher's per-event fan-out) is bounded by fleet size no matter how many clients subscribe. Steady state is write-free; the process heartbeat is what keeps rows valid.

Delivery is the consumer's job: resolve Subscribers, deliver locally when subscribed itself, forward to the rest. Delivery is best-effort by contract — a cached resolution may briefly miss a just-opened stream or include a just-closed one, and the consumer's pull backstop (delta sync against its sequenced store) is what makes that safe.

func NewSubscriptions

func NewSubscriptions(log *zap.Logger, membership *Membership, store SubscriptionStore, cfg SubscriptionsConfig) *Subscriptions

NewSubscriptions creates the subscriptions runtime. It has no background loops; cleanup work rides resolution calls and membership callbacks.

An observer membership (NewObserver) is a valid backing for the read side: Subscribers resolves (and sweeps corpse rows) exactly as on a member, so an event-forwarding-only process can publish toward the fleet. Subscribe is refused with ErrObserverMembership — an observer cannot host streams.

func (*Subscriptions) CloseAll

func (s *Subscriptions) CloseAll(ctx context.Context, handles []*SubscriptionHandle) error

CloseAll releases every handle in one pass, batching the registry deletes for the topics whose last local handle is among them — the teardown twin of SubscribeAll. Each handle is released exactly once across any mix of CloseAll and Close calls; handles already closed (or nil) are skipped, and a handle from another runtime falls back to its own single-row Close.

The batch delete's error is returned from CloseAll rather than attributed to any handle — the registrations are gone locally regardless, and as with Drain, a row that fails to delete stops counting once this member stops heartbeating and is eventually swept.

func (*Subscriptions) Drain

func (s *Subscriptions) Drain(ctx context.Context) error

Drain removes every registry row this instance holds and refuses new subscriptions; outstanding handles' Closes become no-ops. Call on shutdown once streams are closing. Best-effort by design: a row that fails to delete stops counting anyway once this member's heartbeats stop, and is swept as a corpse row after RowGCAfter.

func (*Subscriptions) Resume

func (s *Subscriptions) Resume()

Resume lifts the drain latch for a caller that deliberately aborts a shutdown after Drain. Drained registrations are gone — consumers re-register as their streams reopen.

func (*Subscriptions) Self

func (s *Subscriptions) Self() *Member

Self returns the local member whose interest this runtime registers — the identity carried by every subscriber row this process writes. Consumers use it to recognize their own rows in Subscribers results exactly (by instance ID) instead of comparing separately-configured addresses. Nil on an observer, which writes no rows.

func (*Subscriptions) Subscribe

func (s *Subscriptions) Subscribe(ctx context.Context, namespace string, key []byte) (*SubscriptionHandle, error)

Subscribe registers a local stream's interest in the topic. The first local handle writes the topic's registry row; later handles share it. Returns ErrSubscriptionsDraining once Drain has begun, and ErrObserverMembership on a runtime backed by an observer membership.

func (*Subscriptions) SubscribeAll

func (s *Subscriptions) SubscribeAll(ctx context.Context, topics []SubscriptionTopic) ([]*SubscriptionHandle, error)

SubscribeAll registers a local stream's interest in every listed topic, as Subscribe does for one, writing the missing registry rows in a single store batch instead of one round trip per topic. It returns one handle per entry, in input order; a duplicated topic gets distinct handles against the same registration, exactly as two Subscribe calls would.

All or nothing at the registration level: on error no handles exist and no refcounts moved. A failed batch may still have landed some rows — they are refcount-less self rows, indistinguishable from a failed unsubscribe delete, and the next local resolution of the topic sweeps them (see Subscribers).

func (*Subscriptions) Subscribers

func (s *Subscriptions) Subscribers(ctx context.Context, namespace string, key []byte) ([]*Subscription, error)

Subscribers resolves the servers currently interested in the topic: registry rows filtered by member liveness, with this process served from its local refcounts rather than the store. Results are cached for CacheTTL (empty results included), so hot topics cost one registry read per TTL regardless of event rate.

type SubscriptionsConfig

type SubscriptionsConfig struct {
	// CacheTTL bounds how long a resolved subscriber set is reused without
	// re-reading the registry, so the publish path pays at most one registry
	// read per topic per TTL no matter the event rate. Empty results are
	// cached too — publishes toward offline users must not read per event.
	// The staleness is covered by the consumer's pull backstop: a just-opened
	// stream misses at most one TTL of events before its delta sync heals it.
	// Default 250ms — hot-topic read load at 4 reads/s per publisher is still
	// orders of magnitude under DynamoDB's per-partition ceiling, so the
	// freshness is nearly free; shrinking it much further buys latency the
	// delta sync already covers.
	CacheTTL time.Duration

	// RowGCAfter is how long a row's member may sit continuously outside this
	// observer's live view before the row is treated as a crashed instance's
	// leftover and deleted at resolution time. Graceful drains remove their
	// own rows; this sweeps what crashes leave behind on topics still being
	// published to. Floored at construction well above the liveness window so
	// a merely-slow member is never swept — and a wrongly swept row is
	// re-asserted by its owner on its next liveness session recovery. Default:
	// 10 × the membership's LivenessWindow, resolved at construction.
	RowGCAfter time.Duration
}

SubscriptionsConfig tunes the subscriptions runtime. Zero values take defaults.

type TakeoverTarget

type TakeoverTarget struct {
	InstanceID       string
	HeartbeatCounter uint64
}

TakeoverTarget is the evidence justifying displacement of a claim holder: the holder's identity and the heartbeat counter value the caller observed to be stale. The store honors the takeover only if, atomically at commit time, the holder's registry record is absent or its counter still equals HeartbeatCounter — so a holder that was merely slow (and has heartbeated since the observation) cannot be displaced.

The evidence is produced by the membership layer and passed through opaquely; alternative store implementations may not need it and are free to ignore it.

Directories

Path Synopsis
Package internalrpc is the server-to-server RPC plumbing for the fleet: a per-peer connection pool, API-key authentication for internal-only endpoints, and helpers for forwarding a request to the member that owns its key.
Package internalrpc is the server-to-server RPC plumbing for the fleet: a per-peer connection pool, API-key authentication for internal-only endpoints, and helpers for forwarding a request to the member that owns its key.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL