Documentation
¶
Overview ¶
Package distlock defines the provider-neutral distributed-lock contract for the GoCell runtime layer. Concrete backend implementations live in adapters/ (currently only adapters/redis).
Design rationale ¶
GoCell's layering rule prohibits runtime/ from importing adapters/, so the Locker / Driver interfaces must live here rather than in adapters/redis. The shape follows PR#177's runtime/outbox.Store precedent exactly.
Lock-as-Resource contract (see Lock type godoc) ¶
Locker.Acquire returns a *Lock, NOT a context.Context. Caller-ctx is consumed for the acquire RPC only; once held, the lock lifecycle is independent of caller ctx and ends only via Release(), Orphan(), renewal failure, or TTL expiry after Orphan(). This matches the prevailing industry convention (bsm/redislock, go-redsync, etcd, consul, Curator) and prevents the misuse class identified in GH #20.
Three per-lock terminal signals (Lock.Cause()):
- ErrLockReleased — Release() was called: renewal stopped, backend key deleted.
- ErrLockOrphaned — Orphan() was called: renewal stopped, backend key NOT deleted (expires on its lease TTL — ~1×TTL from the last successful renewal; see Lock.Orphan godoc for the best-effort bound). Use for graceful shutdown/handoff. ref: etcd-io/etcd client/v3/concurrency/session.go Session.Orphan
- ErrLockLost — renewal failed or ownership taken by another holder.
Resource model ¶
Each call to New() creates one Manager. The Manager's resource footprint per active lock set is:
- 1 manager goroutine: owns the renewal min-heap and dispatches every Driver I/O call (Renew, Release) to a short-lived background goroutine so the loop never blocks on a slow or unreachable backend — Orphan() / Stop() stay responsive even while a Renew/Release RPC is in flight.
- 0 persistent per-lock goroutines: *Lock is a value handle; lock-end is delivered by the manager via markCause (closes Done() channel, sets Cause()). Transient goroutines exist only for the duration of an individual Renew/Release RPC.
N active locks = 1 manager goroutine + O(N) heap + transient per-RPC goroutines. One persistent goroutine for N locks.
Non-goals ¶
This is an efficiency lock, NOT a correctness lock. It is suitable for avoiding duplicate work (e.g., "only one pod runs a scheduled job"). For correctness-critical paths use application-level conditional writes (e.g., Postgres optimistic locking with row versions). This matches the Redlock paper's own scoping: Redsync / redis/v9 make the same disclaimer.
References ¶
- ref: github.com/go-redsync/redsync mutex.go — caller-ctx scoped to acquire
- ref: github.com/etcd-io/etcd client/v3/concurrency/session.go — session-scoped keepalive
- ref: github.com/hashicorp/consul/api lock.go — explicit Unlock contract
- ref: github.com/bsm/redislock — refresh as application concern
- ref: PR#177 runtime/outbox.Store — identical layering rationale
- ref: ADR docs/architecture/202605200000-adr-distlock-lock-as-resource.md
Index ¶
Constants ¶
const ErrLockTimeout = errcode.ErrDistlockTimeout
ErrLockTimeout is a package-level alias for errcode.ErrDistlockTimeout. It is returned by Acquire when the key is already held by another holder. The canonical definition lives in pkg/errcode for cross-package matching at HTTP handler boundaries; this alias keeps call sites within distlock concise.
const MinTTL = time.Millisecond
MinTTL is the smallest lock TTL Acquire accepts. Redis SetNX/PEXPIRE take TTL in integer milliseconds; sub-millisecond values truncate to 0, which go-redis v9 documents as "no expiration" — a permanent lock that survives process death. Callers that derive a distlock TTL from their own config (e.g. runtime/saga.Coordinator's LeaseDuration) reference this const to fail fast at construction instead of at the first Acquire.
Variables ¶
var ( // ErrLockLost is returned by Lock.Cause() when the manager fails to renew // the lock or the backend reports ownership has been taken by another holder. // KindConflict (HTTP 409) is the natural mapping if it ever surfaces to a // caller: another holder owns the resource. ErrLockLost = errcode.New(errcode.KindConflict, errcode.ErrDistlockLockLost, "distlock: lock lost") // ErrLockReleased is returned by Lock.Cause() when Lock.Release() is called // by the application (normal end-of-critical-section). KindInternal (HTTP // 500) is deliberate: a normal release is NOT a conflict, and if this // sentinel ever reaches an HTTP handler that is a server-side programming // bug — 500 surfaces it as such rather than misleading the client with 409. ErrLockReleased = errcode.New(errcode.KindInternal, errcode.ErrDistlockLockReleased, "distlock: lock released") // ErrLockOrphaned is returned by Lock.Cause() when Lock.Orphan() is called // by the application. Renewal is stopped and the backend key is NOT deleted // — it expires on its lease TTL (~1×TTL from the last successful renewal; // best-effort, see Lock.Orphan godoc), handing the lock to a competitor // WITHOUT a Release round-trip that could hang or fail at shutdown. // KindInternal (HTTP 500) matches ErrLockReleased: an orphaned // lock surfacing to an HTTP handler is a server-side programming bug — 500 // is preferable to a misleading 409 (external conflict). Orphan is a // deliberate local action; if it reaches an HTTP boundary that is a // caller error, not a backend conflict. // // ref: etcd-io/etcd client/v3/concurrency/session.go Session.Orphan ErrLockOrphaned = errcode.New(errcode.KindInternal, errcode.ErrDistlockLockOrphaned, "distlock: lock orphaned by caller (renewal stopped; key expires after TTL)") )
Sentinel errors returned by Lock.Cause() when the lock ends. Callers distinguish them via errors.Is(lock.Cause(), ErrLockLost) or direct == comparison (markCause stores the exact package-level pointer). *errcode.Error has no custom Is(target error) bool method; errors.Is matches by package-level pointer identity. Callers that wrap with fmt.Errorf("%w", ErrLockLost) still work via Unwrap chain traversal. To match by Code regardless of pointer identity, use:
var ec *errcode.Error
if errors.As(err, &ec) && ec.Code == errcode.ErrDistlockLockLost { ... }
HTTP Kind tagging is fail-closed: these sentinels are internal Lock.Cause() signals and are NOT intended to reach an HTTP handler via httputil.WriteError. If one accidentally does, the Kind below produces the least-misleading response — never a confidently-wrong status.
Functions ¶
This section is empty.
Types ¶
type Driver ¶
type Driver interface {
// SetNX attempts to acquire the lock for key with the given token and TTL.
// Returns (true, nil) on success, (false, nil) when another holder owns the
// key (not an error — caller interprets false as "busy"), and (false, err)
// on I/O failure.
SetNX(ctx context.Context, key, token string, ttl time.Duration) (acquired bool, err error)
// Renew extends the TTL of an existing lock only if token still matches.
// Returns (true, nil) on success, (false, nil) when the token no longer
// matches (ownership lost — not an I/O error), and (false, err) on I/O failure.
Renew(ctx context.Context, key, token string, ttl time.Duration) (held bool, err error)
// Release deletes the lock key only if token still matches.
// Returns nil on success or when the key is already gone (idempotent).
// Returns a non-nil error only on I/O failure.
Release(ctx context.Context, key, token string) error
}
Driver is the storage-backend contract for distributed locks. Implementations live in adapters/ (e.g. adapters/redis).
Three semantic primitives encapsulate all backend-specific logic (e.g. Lua scripts for Redis), keeping the runtime layer backend-agnostic.
ref: kubernetes/client-go tools/leaderelection/resourcelock/interface.go
— storage primitives shape adopted; GoCell collapses to 3 methods vs k8s 5 because Get/Update/Create/Delete lifecycle is not needed here.
ref: go-redsync/redsync redsync.go — SetNX/Renew/Release semantics
type Lock ¶
type Lock struct {
// contains filtered or unexported fields
}
Lock is a held distributed lock returned by Locker.Acquire.
Lock-as-Resource design ¶
Lock is intentionally NOT a context.Context. The caller-supplied context passed to Acquire is consumed only for the acquire RPC (SetNX); once the lock is held, its lifecycle is independent of the caller ctx. This matches the prevailing industry convention (bsm/redislock, go-redsync, etcd client/v3/concurrency, HashiCorp consul, Apache Curator): caller-ctx cancellation does NOT release a held lock; only explicit Release(), Orphan(), or renewal failure (ErrLockLost) will end it. (Explicit Locker.Shutdown is deferred to a future iteration — see ADR docs/architecture/202605200000-adr-distlock-lock-as-resource.md §"Out of scope".) Per-lock Orphan() now exists for graceful shutdown/handoff (bounded-TTL takeover, no release RPC). DISTLOCK-LOCKER-SHUTDOWN-01 tracks the deferred locker-level Shutdown/Close path.
Rationale: caller-ctx cancellation expresses "I no longer care about the outcome of THIS request" — it does NOT express "release the resource I acquired". Conflating the two (the previous Lock-as-Context design) caused GH #20: a forgotten caller could leave a held lock being renewed indefinitely, while a busy caller whose ctx happened to time out could suddenly lose its lock mid-critical-section.
Caller responsibility ¶
Acquire returns *Lock. Caller must:
lock, err := locker.Acquire(ctx, key, ttl)
if err != nil { return err }
defer lock.Release()
Process crash falls back to Redis-side TTL expiry. A living caller that forgets Release leaks the lock until process exit.
Why not context.Context ¶
Acquire could have returned a context.Context whose Done()/Cause() reflect lock-end events, but doing so invites the misuse pattern that GH #20 exposed: caller passes the lock ctx to db.QueryContext / http.NewRequest downstream, assuming caller-ctx semantics. By returning *Lock (which does not implement context.Context), the compiler rejects such misuse outright.
Values from the caller's ctx are still exposed via Lock.Value(key) — the underlying context.WithoutCancel(callerCtx).Value lookup preserves trace IDs and auth claims while shielding the lock from caller-ctx cancellation/deadline.
Callers should avoid stashing large objects, raw secrets, or request-scoped resources whose lifetime should not extend past the request as values on callerCtx before calling Acquire: every value reachable from callerCtx remains pinned through the captured valueLookup closure for the full held-lock duration (including any auto-renewal cycles) and is not eligible for GC until lock.Release() returns or the lock is lost. Trace IDs / auth claims / span contexts (small immutable objects) are the intended use; tokens and PII should be parameterized explicitly instead.
ref: GH #20 ; ADR docs/architecture/202605200000-adr-distlock-lock-as-resource.md
func (*Lock) Cause ¶
Cause returns the reason the lock ended, or nil if the lock is still held.
Possible non-nil values:
- ErrLockReleased: Release() was called by the application.
- ErrLockLost: renewal failed or backend reports ownership taken by another holder.
- ErrLockOrphaned: Orphan() was called by the application (renewal stopped; backend key expires naturally on its lease TTL — see Orphan for the bound).
Cause never returns context.Cause(callerCtx) — caller-ctx cancellation does not end the lock under the Lock-as-Resource contract.
func (*Lock) Done ¶
func (l *Lock) Done() <-chan struct{}
Done returns a channel that closes when the lock ends. Read it to learn that the critical section must be aborted; pair with Cause() to learn why.
select {
case <-lock.Done():
return fmt.Errorf("lock ended: %w", lock.Cause())
case result := <-doWork(...):
return result
}
func (*Lock) Orphan ¶
func (l *Lock) Orphan()
Orphan stops this lock's lease renewal WITHOUT deleting the backend key. No Release I/O is performed, so Orphan never blocks on or fails due to backend reachability. After Orphan, Done() is closed and Cause() reports ErrLockOrphaned.
Takeover bound (best-effort, not a hard cap from the Orphan call): Orphan stops scheduling future renewals immediately and cancels any in-flight renewal, but the cancel is not atomic with the backend — a renewal whose write already reached the backend at Orphan time may extend the lease one more TTL window from its commit. The competitor can therefore acquire after ~1×TTL measured from the last successful renewal (≈one TTL window from the Orphan call, plus a renewal RPC latency in the worst case). This mirrors etcd Session.Orphan: keepalive stops, the lease lapses on its own TTL.
Orphan and Release are mutually exclusive and idempotent: the first call of either wins; later calls of either are no-ops. Use Orphan for graceful shutdown/handoff (bounded-TTL takeover, no release RPC); use Release for immediate end-of-critical-section.
ref: etcd-io/etcd client/v3/concurrency/session.go Session.Orphan — stop keepalive, lease expires after TTL (vs Close which revokes immediately).
func (*Lock) Release ¶
Release ends the lock. Idempotent: safe to call multiple times; only the first call performs Driver.Release I/O and Cause() will be ErrLockReleased afterwards. Subsequent calls return the first call's result.
Release blocks until Driver.Release completes (bounded by WithReleaseTimeout, default 5s).
func (*Lock) Value ¶
Value returns the value associated with the caller-supplied context's key. Caller-ctx cancellation and deadline do NOT propagate to Lock; only values do. Use this to retrieve trace IDs / auth claims that were on the caller ctx without re-plumbing them through Acquire's signature.
type Locker ¶
type Locker interface {
// Acquire blocks until the lock is granted or ctx is canceled.
//
// On success it returns a *Lock; caller MUST eventually end the lock via
// lock.Release() (delete the backend key now) or lock.Orphan() (stop
// renewal; the key expires on its lease TTL — ~1×TTL from the last
// successful renewal, best-effort). Both are terminal and mutually
// exclusive.
//
// Lock-end signals (lock.Done() closed; lock.Cause() reports):
// - ErrLockReleased — Release() was called (normal end-of-critical-section)
// - ErrLockLost — renewal failed or backend reports ownership taken
// - ErrLockOrphaned — Orphan() was called (renewal stopped; key expires on its lease TTL — see Lock.Orphan)
//
// Notably absent: caller-ctx cancellation does NOT end the lock. If the
// caller wants the lock to end when its ctx is canceled, the caller must
// explicitly arrange a goroutine that does so.
//
// Per-lock Orphan() now exists for graceful shutdown/handoff (bounded-TTL
// takeover, no release RPC). Locker.Shutdown/Close remains deferred
// (DISTLOCK-LOCKER-SHUTDOWN-01).
//
// Idiomatic patterns for combining lock-end with caller-ctx:
//
// // Pattern A — caller wants ctx cancel to also release the lock:
// lock, err := locker.Acquire(ctx, key, ttl)
// if err != nil { return err }
// defer lock.Release()
// go func() {
// // Wait for whichever ends first; the second case prevents the
// // goroutine from blocking on ctx forever after a normal Release
// // or a renewal-failure ends the lock.
// select {
// case <-ctx.Done():
// _ = lock.Release() // best-effort; idempotent.
// case <-lock.Done():
// // lock ended (Release / lost) — nothing more to do.
// }
// }()
//
// // Pattern B — caller wants the *first* of (ctx-cancel | lock-lost) to abort:
// select {
// case <-ctx.Done():
// _ = lock.Release()
// return ctx.Err()
// case <-lock.Done():
// return fmt.Errorf("lock ended: %w", lock.Cause())
// }
//
// TTL is the only ceiling on a held-but-forgotten lock. Choose ttl
// commensurate with the critical-section worst case (typically seconds
// to minutes); avoid hour-scale TTLs unless the workload genuinely
// runs that long, since a caller that aborts without Release leaves
// peers blocked for the full ttl window. The fallback after process
// crash is Redis-side TTL expiry.
//
// On failure it returns (nil, err) where err carries ErrLockTimeout when
// another holder owns the key, or ctx.Err() (wrapped) if the parent was canceled.
//
// The lock is auto-renewed by a single shared manager goroutine (not per-lock)
// until lock.Release() is called or renewal fails.
// N active locks = 1 manager goroutine + O(N) heap. Zero per-lock goroutines.
//
// Driver.Release uses context.Background() with WithReleaseTimeout (default
// 5s). lock.Release() blocks until the I/O completes and returns nil on
// success or a wrapped error on I/O failure. lock.Release() is idempotent —
// a second call returns the first call's result without contacting the backend.
Acquire(ctx context.Context, key string, ttl time.Duration) (*Lock, error)
// Stats reports observable state of the Locker for health checks and metrics.
Stats() Stats
}
Locker acquires named distributed locks.
Lock-as-Resource design ¶
Acquire returns a *Lock — intentionally NOT a context.Context. Caller-ctx cancellation does NOT release a held lock; only explicit Release or renewal failure will end it. This matches the prevailing industry convention (bsm/redislock, go-redsync, etcd client/v3/concurrency, HashiCorp consul, Apache Curator) and prevents the misuse pattern that GH #20 exposed under the previous Lock-as-Context design. (Explicit Locker.Shutdown is deferred — see ADR docs/architecture/202605200000-adr-distlock-lock-as-resource.md §"Out of scope".)
Caller responsibility:
lock, err := locker.Acquire(ctx, key, ttl)
if err != nil { return err }
defer lock.Release()
ctx is consumed only for the acquire RPC (SetNX). Once the lock is held, caller-ctx cancellation is decoupled. Values from ctx (trace IDs, auth claims) are still exposed via Lock.Value via context.WithoutCancel.
ref: GH #20 ; ADR docs/architecture/202605200000-adr-distlock-lock-as-resource.md ref: go-redsync/redsync — caller-ctx scoped to acquire only ref: etcd client/v3/concurrency — session-scoped keepalive, decoupled from per-op ctx
func New ¶
New creates a Locker backed by the given Driver.
Returns an error if driver is nil or if any configuration parameter is out of range:
- renewFraction must be in (0, 1)
- driftFactor must be in [0, 1)
- releaseTimeout must be > 0
The returned Locker uses a single shared manager goroutine for all locks. Resource shape:
- 1 manager goroutine (owns the renewal heap and all Driver calls)
- 0 per-lock goroutines — *Lock is a signal/value handle; lock-end is delivered by the manager goroutine via Lock.markCause (closes Done() channel, sets Cause()) without spawning watchers.
N active locks = 1 manager goroutine + O(N) heap.
ref: plan "共享 manager goroutine" section
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager runs a single shared goroutine that owns the renewal heap and calls Driver.Renew for all active locks.
The manager goroutine is the SOLE writer of the heap and locks map, which eliminates data races. External callers communicate exclusively through the events channel.
Lifecycle:
- lazy-started on first Acquire (via lockerImpl.add)
- manager exits when the last lock is removed
- started is closed once the manager enters its main select loop
- drained is closed once the manager exits after the last lock removal
ref: golang.org/x/tools/internal/event — single goroutine dispatch pattern
func (*Manager) Drained ¶
func (m *Manager) Drained() <-chan struct{}
Drained returns a channel that is closed once the manager goroutine exits after the last lock has been dispatched through eventRemove. Background Driver.Release I/O goroutines spawned by handleRemove may still be in flight when Drained closes — Release blocks the *caller* on the I/O result via the eventRemove resultCh, but the manager does not wait for those goroutines before exiting. Drained therefore signals "no more renewal activity will occur" rather than "all backend keys have been released".
func (*Manager) RenewNotify ¶
func (m *Manager) RenewNotify() <-chan struct{}
RenewNotify returns a read-only channel that receives a signal after each successful Driver.Renew call. Intended for test synchronization only.
func (*Manager) Snapshot ¶
func (m *Manager) Snapshot() ManagerSnapshot
Snapshot returns a read-only view of current manager state.
type ManagerSnapshot ¶
type ManagerSnapshot struct {
// Locks is the number of active locks currently tracked.
Locks int
}
ManagerSnapshot is a read-only view of the manager's current state. Exported for testing only.
type Option ¶
type Option func(*config)
Option is a functional option for configuring a Locker.
func WithDriftFactor ¶
WithDriftFactor sets the renewal I/O timeout safety margin. The Renew RPC context deadline is set to clock.Now() + ttl × (1 − driftFactor), so the manager gives up on a slow Driver.Renew before the backend key would expire. Does NOT alter the TTL written to the backend.
Recommended range: 0.01–0.05. Higher values make Renew calls fail more often under transient I/O slowness; lower values risk the call outliving the backend TTL on slow networks. Must be in [0, 1). Default: 0.01.
ref: go-redsync/redsync redsync.go driftFactor=0.01
func WithMaxRenewAttempts ¶
WithMaxRenewAttempts sets the maximum number of Driver.Renew attempts per renewal tick before the lock is declared lost. Must be ≥ 1. Default: 3.
Only transient I/O errors (err != nil) are retried; permanent ownership-lost responses (held=false, err=nil) immediately declare the lock lost regardless of this setting.
All retry attempts share the same renewTimeout window derived from the lock TTL and drift factor. New() returns a validation error if the final value is < 1.
func WithReleaseTimeout ¶
WithReleaseTimeout sets the context deadline applied to each background Driver.Release call issued by the fire-and-forget release path. If Redis (or another backend) hangs, the Release goroutine will be unblocked after this duration rather than leaking indefinitely.
Default: 5s (conservative; tune down for low-latency backends or up for high-latency ones). Must be > 0; New() returns a validation error if the final value is ≤ 0.
func WithRenewFraction ¶
WithRenewFraction sets the fraction of TTL at which the shared manager schedules the next renewal. Must be in (0, 1). Default: 0.5.