Documentation
¶
Overview ¶
Package etcd backs the L0 cluster ring with etcd: a node registers itself under a lease and watches the member set, so membership is live and self-healing — a crashed node's lease expires and it drops out of every other node's ring within the TTL, with no manual deregistration. The ring (cluster/ring) is rebuilt locally from the watched member set, so placement stays coordinator-free on the hot path; etcd only distributes membership.
Index ¶
- Constants
- type Member
- type Membership
- func (m *Membership) AddrOf(id string) string
- func (m *Membership) Close(ctx context.Context) error
- func (m *Membership) FenceDeadline() time.Time
- func (m *Membership) Fenced() bool
- func (m *Membership) LeaseID() clientv3.LeaseID
- func (m *Membership) Members() []Member
- func (m *Membership) OnRejoin(fn func(clientv3.LeaseID))
- func (m *Membership) Rejoins() int64
- func (m *Membership) Ring() *ring.Ring
- func (m *Membership) SelfAbsent() bool
- func (m *Membership) SetClock(now func() time.Time)
- func (m *Membership) SetLogger(l *zap.Logger)
- type Ownership
- func (o *Ownership) Acquire(ctx context.Context, shard string) (term uint64, ok bool, err error)
- func (o *Ownership) Claims(ctx context.Context) ([]string, error)
- func (o *Ownership) LastPlan() []rebalance.Reassignment
- func (o *Ownership) Owned() []string
- func (o *Ownership) Reconcile(ctx context.Context, r *ring.Ring, shards []string) ([]string, error)
- func (o *Ownership) Release(ctx context.Context, shard string) error
- func (o *Ownership) SetFence(fenced func() bool)
- func (o *Ownership) SetLease(id clientv3.LeaseID)
- func (o *Ownership) SetPlanRF(rfOf func(shard string) int)
- func (o *Ownership) Term(shard string) (uint64, bool)
Constants ¶
const DefaultTTL = 30 * time.Second
DefaultTTL is the lease TTL: a node absent for this long (no keepalive) is evicted from the ring. It bounds failure-detection latency, and it is equally the blast radius of a hiccup — a GC pause, a CPU-starved node or a brief etcd stall longer than the TTL costs the node its lease. 30s trades slower failure detection for not evicting healthy nodes: the ring only has to be right within a rebalance, while a spurious eviction moves ownership for nothing. Re-registration is what makes an eviction survivable at all; the TTL only sets how often one happens. Join takes a per-cluster override.
const FenceMargin = 5 * time.Second
FenceMargin is how far before a lease's nominal expiry a node stops trusting it. It has to cover the clock error between this node and etcd plus the delay between the deadline passing and the node noticing, and it comes out of the useful window, so it is a few seconds against DefaultTTL's thirty. A TTL too small for it uses a third of the TTL instead, so a short-TTL cluster is fenced late in its window rather than permanently.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Member ¶
Member is a cluster node's advertised identity: its ring ID, failure-domain location, and the Addr its peers reach it on. Zone is the single-level failure domain (a rack); Domains is the hierarchical form (coarsest first, e.g. {rack, server}) that supersedes Zone when set — erasure-coded shards balance across it so a rack/server/disk failure loses the fewest shards.
type Membership ¶
type Membership struct {
// contains filtered or unexported fields
}
Membership is a live, etcd-backed view of the cluster. It keeps this node registered (under a keep-alive'd lease) and watches the member set, exposing the current ring.Ring. Safe for concurrent use; Membership.Ring and Membership.Members are lock-free / cheap.
func Join ¶
func Join(ctx context.Context, client *clientv3.Client, root string, self Member, ttl time.Duration) (*Membership, error)
Join registers self in the cluster rooted at root (an etcd key prefix) under a lease of ttl (≤ 0 ⇒ DefaultTTL), snapshots the current members, and starts watching for changes. The returned Membership must be closed to deregister.
func Watch ¶ added in v0.38.0
Watch returns a read-only Membership: it snapshots the member set and follows it, exposing the same live ring Join does, but registers nothing. The caller takes no lease, appears in no other node's ring, and is therefore never placed as an owner — which is what a stateless tier (a query or ingest process) needs to route by the ring without holding data for it.
The returned value is otherwise an ordinary Membership: Membership.Ring, Membership.AddrOf and Membership.Members behave identically. Membership.LeaseID is zero, and Membership.Close stops the watch without revoking anything.
func (*Membership) AddrOf ¶
func (m *Membership) AddrOf(id string) string
AddrOf returns the network address of the member with the given ring node ID, or "" if the member is unknown. It is the resolver the cluster write path uses to turn ring owners into transport targets.
func (*Membership) Close ¶
func (m *Membership) Close(ctx context.Context) error
Close stops watching, revokes this node's lease (so its peers drop it immediately rather than after the TTL), and waits for the background goroutines to exit. The revoke is bounded by a derived timeout. An observer (Watch) holds no lease, so it only stops watching.
func (*Membership) FenceDeadline ¶ added in v0.40.0
func (m *Membership) FenceDeadline() time.Time
FenceDeadline is the instant after which this node can no longer prove it still holds its membership lease: the last keep-alive etcd answered, plus the TTL, less FenceMargin. Past it another node's Ownership.Acquire may already have succeeded, so everything the lease backs — every compaction claim, and with it the right to act as a shard's primary — must be treated as lost.
The zero time means "already fenced": the node holds no lease, or knows its registration is gone and has not yet re-registered.
func (*Membership) Fenced ¶ added in v0.40.0
func (m *Membership) Fenced() bool
Fenced reports whether this node is past Membership.FenceDeadline and so can prove nothing about what it owns. It is deliberately not "can I reach etcd": a node unable to reach etcd but still inside a live lease is not wrong to serve as primary, and a node whose lease has lapsed is wrong whether or not etcd is reachable.
func (*Membership) LeaseID ¶
func (m *Membership) LeaseID() clientv3.LeaseID
LeaseID is this node's membership lease. Ownership claims bind to it so they auto-release when the node dies (the basis for the rebalance handoff).
func (*Membership) Members ¶
func (m *Membership) Members() []Member
Members returns the current members, sorted by ID.
func (*Membership) OnRejoin ¶ added in v0.40.0
func (m *Membership) OnRejoin(fn func(clientv3.LeaseID))
OnRejoin registers a hook invoked after this node re-registers under a fresh lease. Anything bound to the old lease died with it — compaction claims above all (see Ownership.SetLease) — so the hook is how a dependent rebinds. It runs on the maintainer goroutine and must not block. A nil hook clears it.
func (*Membership) Rejoins ¶ added in v0.40.0
func (m *Membership) Rejoins() int64
Rejoins counts the times this node re-registered after losing its registration. A value that keeps climbing means the lease TTL is too tight for the environment.
func (*Membership) Ring ¶
func (m *Membership) Ring() *ring.Ring
Ring returns the current ring (lock-free). It is replaced atomically as membership changes.
func (*Membership) SelfAbsent ¶ added in v0.40.0
func (m *Membership) SelfAbsent() bool
SelfAbsent reports that this node believes it is missing from the cluster member set: its lease was lost (or its key deleted) and re-registration has not yet succeeded. It is a contradiction — the node is running — and while it holds, the node is invisible to every peer's ring, so it takes no writes and owns no shards. It clears on the next successful registration.
func (*Membership) SetClock ¶ added in v0.40.0
func (m *Membership) SetClock(now func() time.Time)
SetClock overrides the clock the fence deadline is compared against (Membership.Fenced). nil restores time.Now. It exists because the deadline is the only wall-clock decision in this package, and a lease-fencing test that cannot move the clock has to sleep out a TTL.
func (*Membership) SetLogger ¶ added in v0.5.0
func (m *Membership) SetLogger(l *zap.Logger)
SetLogger attaches a logger that records member joins and leaves (Info on each change). It must be called before Join starts the watch loop in practice it is set immediately after Join. nil disables logging. Safe only before the watch observes its first event.
type Ownership ¶
type Ownership struct {
// contains filtered or unexported fields
}
Ownership coordinates exclusive **compaction ownership** of shards across the cluster via etcd, so a shard (a tenant) is flushed/merged by exactly one node at a time — the rebalance executor. A node claims a shard with a CAS write keyed by the shard and bound to its membership lease; the claim auto-releases if the node dies, so a new primary can take over without manual handoff. Placement still comes from the ring; etcd only arbitrates the claim during the brief windows where nodes disagree on the ring (watch-propagation lag) or a node has failed.
Reconcile is **event-driven and minimal-move**: it tracks the claims this node currently holds and, on each pass, only issues etcd writes for the shards whose ring-primary actually changed since the last pass (plus retrying any wanted-but-uncontended claim). In steady state — an unchanged ring with no new tenants — it makes no etcd round-trips at all, instead of one acquire/release per shard every tick. When the ring does change it records the rebalance.Plan it enacted (see Ownership.LastPlan) for observability/preview.
func NewOwnership ¶
NewOwnership returns an ownership coordinator for node id, claiming under root with the node's membership lease (see Membership.LeaseID).
func (*Ownership) Acquire ¶
Acquire tries to claim shard for this node. It reports whether the claim is now held by this node (newly acquired or already ours), and the claim's term. The claim is a CAS: create the key only if absent; otherwise it belongs to whoever already created it.
The term is the etcd revision the claim key was created at, which costs nothing — it is in the response either way. etcd revisions are cluster-wide monotonic, so a term orders every ownership tenure of a shard against every other, and it is *stable* for the life of one tenure: reacquiring a claim this node already holds reports the revision it was created at, not the current one, so repeated Ownership.Reconcile passes do not keep moving it.
That is what the bucket index's commit generation needs to survive a node restored from an old snapshot: reacquiring the shard puts its writes above everything its replicas hold, and a node that lost the shard keeps the lower term of a tenure that has ended.
func (*Ownership) Claims ¶ added in v0.33.0
Claims returns every currently-claimed shard across the cluster (sorted), from one etcd range read. It is the cluster-wide tenant/shard discovery a node needs when it is promoted into a shard's owner set without ever having held the shard locally (a spare): its own engine maps do not know the tenant, but any live shard has a compaction owner whose claim names it. A shard whose every owner died has no claim and is not discoverable here — its data is only recoverable through the backend (shared store) or peer listings.
func (*Ownership) LastPlan ¶ added in v0.10.0
func (o *Ownership) LastPlan() []rebalance.Reassignment
LastPlan returns the owner-set handoffs enacted at the most recent ring change (empty if the ring has not changed since open). It is informational — a preview of what the last rebalance moved — for an operator dashboard. See Ownership.SetPlanRF for the owner-set breadth.
func (*Ownership) Owned ¶ added in v0.10.0
Owned returns a sorted snapshot of the shards this node currently holds a compaction claim on.
func (*Ownership) Reconcile ¶
Reconcile makes this node's claims match the ring: it acquires every shard this node is the ring-primary of and releases the rest. It returns the shards this node now owns — the set it should flush and compact. Idempotent, so it is safe to call on every membership change and on a timer.
The work is minimal: ring-primary lookups are pure in-memory HRW hashing (no etcd), and an etcd write is issued only when a claim must change — a wanted shard not yet held is acquired (retried every pass, which is what lets a stale claim's release converge even under an unchanged ring), and a held shard no longer wanted is released. Steady state issues no etcd writes. On a ring change the enacted primary handoffs are recorded in Ownership.LastPlan.
func (*Ownership) Release ¶
Release relinquishes shard, but only if this node still holds the claim (a guarded delete, so it never deletes another node's claim).
func (*Ownership) SetFence ¶ added in v0.40.0
SetFence installs the predicate that reports this node's claims unprovable — wire it to Membership.Fenced. While it reports true this node holds nothing as far as every caller is concerned: Ownership.Term disclaims, Ownership.Owned is empty, and Ownership.Reconcile is a no-op, so the node neither flushes nor stamps an index for a shard whose tenure it can no longer prove. The held set itself is kept, so a lease confirmed again — the same lease, without a rejoin — resumes the claims rather than re-acquiring them. nil clears the fence.
func (*Ownership) SetLease ¶ added in v0.40.0
SetLease rebinds claims to a new membership lease, after the node lost its old one and re-registered (wire it with Membership.OnRejoin). Every claim written under the old lease went with it, so the held set is dropped too: it would otherwise record ownership this node no longer has, and Reconcile only writes for shards it does not already believe it holds. The next Reconcile re-acquires under the new lease.
func (*Ownership) SetPlanRF ¶ added in v0.32.0
SetPlanRF sets the per-shard replication factor used when recording Ownership.LastPlan (e.g. the tenant durability policy's RF), so the recorded plan reflects each shard's full owner-set diff rather than only the primary handoff. It does not affect claim reconciliation — compaction ownership always tracks the primary alone. Call before the first Reconcile; a nil resolver (the default) records primary-only (rf=1) plans.
Source Files