Documentation
¶
Overview ¶
Package hublease fences a cloop control plane to one process (Task 20214).
The problem it solves is a silent one. Two `cloop ui` processes can open the same .cloop/state.db and neither will ever see an error: SQLite's WAL keeps the file consistent under concurrent writers, which is exactly why the bug hides. What diverges is everything the hub keeps *beside* the database — pkg/ui holds its project-status cache, run registry, chat histories, live-log rooms and WebSocket client set in process memory, and broadcasts reach only the clients of the process that produced the event. A browser attached to the second hub therefore reads a snapshot that quietly stopped advancing, while both processes run the same background sweeps against shared rows and can issue two stop signals for one run.
deploy/helm/cloop-hub pins replicaCount to 1 for this reason, but a values comment cannot enforce anything. This package makes the constraint the code's: a hub takes a lease before it touches the control plane, renews it on a ticker, and gives it up on a clean shutdown.
The design rests on three decisions worth stating, because each one is the answer to a way leases usually fail:
A lease is liveness, not ownership. It is held for a TTL and renewed, so the holder crashing costs a bounded wait rather than a permanent wedge. "Treat a stale lease as free" is what stops a SIGKILLed hub from locking its own successor out forever. Renewal is fenced. Every write is conditioned on our instance id, so a hub that was paused past the TTL — swapped out, stopped, or stuck in a long GC — discovers at its next beat that the row is no longer its own, and stands down. Without this, "stale means free" would be a licence for two live hubs, which is worse than the bug being fixed. A crash on the same host does not cost the TTL. A holder that recorded its pid under this machine's boot id can be probed directly; if the process is gone, the lease is dead now rather than in a minute. That matters because the common case is systemd restarting a killed hub within seconds, and a restart that refuses for a minute reads as an outage.
Policy lives here; the SQL is pkg/statedb's (migrations/0026_hub_instances.sql).
Index ¶
Constants ¶
const ( // DefaultTTL is how long a lease survives without a renewal. Past it the // lease is free and another hub may take it. // // Sixty seconds is a compromise between two costs that pull opposite ways. // Too short and an ordinary stall — a slow fsync, a paused container, a // host under load — hands the control plane to a second process while the // first is still alive and about to notice. Too long and a hub killed // without a chance to release locks its own restart out for that long, // which on a supervised deployment means a visible outage. Three missed // beats is the usual ratio; the same-host liveness probe below is what // keeps the common crash from paying this cost at all. DefaultTTL = 60 * time.Second // DefaultInterval is how often the holder renews. TTL/3, so two beats can // be lost to transient database contention without the lease lapsing. DefaultInterval = 20 * time.Second )
Variables ¶
var ErrLeaseLive = errors.New("hublease: the lease is live")
ErrLeaseLive is returned by Clear when the lease is still held.
Functions ¶
func Clear ¶
func Clear(opts Options) (statedb.HubLeaseRow, error)
Clear releases a lapsed lease on an operator's behalf.
It refuses while the lease is live, and that gate is the reason the command is safe to document: the failure mode of a "force" escape hatch is an operator evicting a hub that was working, which produces exactly the two-hub state the fence exists to prevent — except now with the second hub believing it is sole owner. A lease that really is abandoned lapses on its own, so there is nothing a force flag would unblock that waiting does not.
The release is conditioned on the row still carrying the heartbeat Clear judged, so a holder that comes back between the check and the write keeps its lease rather than being evicted by a decision that is no longer true.
Types ¶
type ConflictError ¶
type ConflictError struct {
DBPath string
Holder statedb.HubLeaseRow
TTL time.Duration
Age time.Duration
// Raced is true when the holder won a simultaneous start rather than
// already holding the lease when we looked.
Raced bool
}
ConflictError reports that another instance holds the lease.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
type Identity ¶
type Identity struct {
Hostname string
PID int
// BootID identifies this boot of this kernel. A pid recorded before a
// reboot names an unrelated process after one, so the liveness probe
// refuses to trust a pid unless the boot ids match and are non-empty.
BootID string
}
Identity is who the holder claims to be.
type Lease ¶
type Lease struct {
// contains filtered or unexported fields
}
Lease is a held fence. It is safe to use from multiple goroutines.
func Acquire ¶
Acquire takes the lease, or explains who has it.
A refusal is a *ConflictError carrying the holder, so callers that want to react to the distinction (rather than print it) can use errors.As. Anything else is an ordinary failure to reach the database.
The returned Lease is not yet renewing; call Start to begin heartbeats. The split exists so a caller can acquire early — before anything else touches the control plane — and only start the ticker once it has a context to tie it to.
func (*Lease) InstanceID ¶
InstanceID is this holder's identity, recorded in every row it writes.
func (*Lease) Lost ¶
func (l *Lease) Lost() <-chan struct{}
Lost is closed when this instance no longer holds the lease and did not release it: another hub took over, or renewals failed for longer than the TTL so another hub is now entitled to. A hub must stop serving when it fires — past that point it has no claim to the control plane and would be the second writer this package exists to prevent.
A nil Lease returns nil, which blocks forever in a select. That is deliberate: it lets callers that did not take a lease use the same select as callers that did.
type Options ¶
type Options struct {
// DBPath is the control-plane database to fence. Ignored when Store is set.
DBPath string
// Store overrides DBPath. When supplied the caller owns its lifetime;
// Release will not close it.
Store Store
// Scope names the fence. Empty means statedb.HubScope.
Scope string
// Address is the listen address this hub will serve, recorded so a refusal
// can name the port the holder is on rather than only its pid.
Address string
// Version is this build's identifier, recorded for forensics.
Version string
// TTL and Interval default to DefaultTTL and DefaultInterval.
TTL time.Duration
Interval time.Duration
// Now and ProcessAlive are injection points for tests. Production leaves
// them nil.
Now func() time.Time
ProcessAlive func(pid int) bool
// Identity overrides the recorded hostname/pid/boot id. Tests use it to
// impersonate another host; production leaves it zero.
Identity Identity
}
Options configures Acquire. Only DBPath (or Store) is required.
type Status ¶
type Status struct {
Present bool
Row statedb.HubLeaseRow
// Live is true when the lease is held and may not be taken.
Live bool
// Age is how long since the last heartbeat.
Age time.Duration
// Expires is how long until a live lease lapses; zero when it is not live.
Expires time.Duration
// Reason explains a non-live lease.
Reason string
TTL time.Duration
}
Status is a read-only view of the fence, for `cloop hub lease status`.
type Store ¶
type Store interface {
GetHubLease(scope string) (statedb.HubLeaseRow, error)
AcquireHubLease(row, expect statedb.HubLeaseRow) (bool, error)
RenewHubLease(scope, instanceID string, at time.Time) (bool, error)
ReleaseHubLease(scope, instanceID string, at time.Time) (bool, error)
ClearHubLease(scope string, expect statedb.HubLeaseRow, at time.Time) (bool, error)
}
Store is the persistence hublease needs. *statedb.DB satisfies it; tests substitute their own to drive races that are not reproducible against a real database (a holder that renews between another hub's read and its write).
Source Files