Documentation
¶
Overview ¶
Package lease is a named, process-level singleton lock over Postgres advisory locks: at most one replica at a time runs a given job, every other replica waits as a hot standby and takes over when the holder exits, crashes or loses its database connection.
What it is for ¶
A background job that is NOT safe to run on two replicas at once (a sweeper that reads a value and writes a derived one, a reconciler that talks to a billing provider) cannot be scaled by raising the replica count. Without a lease the choices are one replica and an outage on every rollout, or two replicas and a race. Jobs that already tolerate concurrency (queue consumers on SELECT ... FOR UPDATE SKIP LOCKED, durable JetStream consumers, Temporal workers, idempotent set-based sweeps) need no lease and should not take one.
Why advisory locks ¶
Every forge service already has a database, so the lease needs no new infrastructure, no schema and no Kubernetes RBAC, and behaves identically under `forge env up`, docker compose and a cluster. A SESSION-level advisory lock is released by Postgres the instant the holder's connection ends, so a crashed replica needs no TTL to expire.
The connection must be session-capable ¶
The lock lives on one dedicated connection, so the *sql.DB given to New must hand back the same server session for the life of a connection. That excludes a transaction-pooling PgBouncer (the lock would be taken on a backend the client stops talking to, and silently never released). Point it at the direct database or a session-mode pooler entry, and give it its own pool: one open connection per Locker is all it uses.
Overlap is bounded, not impossible ¶
Postgres will not grant a held lock to a second session, so two replicas are never leaders while the first holder's connection is alive. If the holder's connection is lost the server drops the lock, a standby can take it at once, and the old holder only learns of it at its next heartbeat. The jobs run under a lease therefore see at most Options.HeartbeatInterval plus Options.QueryTimeout of overlap after a partition (the next check, and the time it may take to give up on a silent network), the same kind of guarantee as a Kubernetes Lease, and must stay correct (if inefficient) under that window.
A slow reply is not a lost session ¶
Every query on the session runs under Options.QueryTimeout, not the heartbeat interval. A query that outlives its deadline cannot be abandoned politely: the driver closes the connection, the server drops the session and every lock on it, so a timeout does not merely report a lost lease, it causes one. A deadline as short as the heartbeat interval turned local scheduling delay (a CPU-starved process, a stalled pod, a slow round trip) into dropped leases on a perfectly healthy session. How OFTEN to check and how LONG to wait for an answer are separate questions with separate knobs.
Key space ¶
A lease name is hashed to the 64-bit advisory-lock key (see Key). Other code sharing the database may use advisory locks of its own; the hash is namespaced so a collision needs a 64-bit coincidence.
Index ¶
Constants ¶
const ( // DefaultRetryInterval is how often a standby polls for a lease it does // not hold. It bounds failover after a graceful release. DefaultRetryInterval = 5 * time.Second // DefaultHeartbeatInterval is how often the holder re-checks that its // session still owns its locks. DefaultHeartbeatInterval = 5 * time.Second // DefaultQueryTimeout is how long one query on the lease session may take // before the session is declared lost. It is deliberately generous: a // query that times out closes the connection and drops every lease on it, // so it must only fire on a network or server that has stopped answering, // never on a slow reply. DefaultQueryTimeout = 10 * time.Second )
Variables ¶
var ErrAlreadyRunning = errors.New("lease: this lease name is already running on this Locker")
ErrAlreadyRunning is returned by Locker.Run when this Locker is already running the same lease name. Postgres advisory locks are re-entrant within a session, so without this check two in-process callers would both believe they lead.
Functions ¶
Types ¶
type Locker ¶
type Locker struct {
// contains filtered or unexported fields
}
Locker contends for leases on one dedicated database session. Create one per process and share it between every job that needs a lease: all of them ride a single connection, so the cost is one session however many jobs.
func New ¶
New returns a Locker over db, which must be session-capable (see the package doc). The Locker does not own db; Close releases only its leases.
func (*Locker) Close ¶
Close drops the session, which releases every lease it holds, and makes every Run (running or later) return nil. It is for tests and orderly teardown; a normal process just cancels the contexts it passed to Run.
func (*Locker) Run ¶
Run contends for the lease name and, while it holds it, runs fn.
fn receives a context that is cancelled when the lease is lost or ctx ends; it must return promptly then. Run's behaviour on fn's return:
- ctx is done: Run returns nil.
- the lease was lost (heartbeat failure) and fn returned: Run goes back to contending and calls fn again when it next wins. fn must therefore be restartable.
- fn returned an error: Run releases the lease and returns the error.
- fn returned nil with the lease still held: the job is finished (for example a worker that disables itself). Run releases the lease and returns nil rather than spin re-acquiring it.
The lease is always released (pg_advisory_unlock) after fn has returned, never before, so a graceful handover never overlaps the outgoing leader.
type Options ¶
type Options struct {
// RetryInterval is the standby poll interval (default
// [DefaultRetryInterval]). Each wait is jittered by up to 20% so replicas
// that start together do not poll in lockstep.
RetryInterval time.Duration
// HeartbeatInterval is the holder's session check interval (default
// [DefaultHeartbeatInterval]). A check that fails loses every lease held
// on the session.
HeartbeatInterval time.Duration
// QueryTimeout bounds every query on the session, the heartbeat's
// included (default [DefaultQueryTimeout]). It is independent of
// HeartbeatInterval: a short interval checks often, it does not demand a
// faster answer. A query that times out loses every lease on the session
// (see the package doc), so keep it well above the worst round trip a
// healthy database and a busy process can produce.
QueryTimeout time.Duration
// Logger receives acquire / lose / session events (default slog.Default()).
Logger *slog.Logger
}
Options tunes a Locker. The zero value is production-ready.
Source Files