Documentation
¶
Overview ¶
Package outbox provides the runtime-layer Store interface and relay worker for transactional outbox delivery. SQL dialect is intentionally not part of this package's surface — adapter packages implement Store against concrete backends.
Index ¶
- Constants
- func SanitizeError(errMsg string, maxLen int) string
- func TruncateError(msg string, maxLen int) string
- type ClaimedEntry
- type FailureBudget
- type PendingDepthObserver
- type Relay
- func (r *Relay) Probes() []healthz.Probe
- func (r *Relay) Ready() <-chan struct{}
- func (r *Relay) Start(ctx context.Context) error
- func (r *Relay) Stop(ctx context.Context) error
- func (r *Relay) WithCommandDispatch(reg *command.Registry, ...) *Relay
- func (r *Relay) WithPendingDepthObserver(o PendingDepthObserver) *Relay
- func (r *Relay) Worker() kworker.Worker
- type RelayConfig
- type Store
Constants ¶
const ( ProbePoll healthz.ProbeName = "outbox_relay_poll" ProbeReclaim healthz.ProbeName = "outbox_relay_reclaim" ProbeCleanup healthz.ProbeName = "outbox_relay_cleanup" )
ProbePoll / ProbeReclaim / ProbeCleanup are the typed healthz.ProbeName consts for the three relay failure-budget probes. They participate in the PROBENAME-SEALED-FUNNEL-01 typed funnel so the composition root no longer translates bare strings from ManagedResource — the names flow as typed const all the way from Relay.Probes() to /readyz registration. Wire values are preserved (no "_ready" suffix; relay budgets are not adapter dependency-availability probes — see adapterSanctionedPkgs convention).
const ( // DefaultRelayReclaimInterval is how often the relay reclaims stale claimed // entries (and, when a PendingDepthObserver is wired, samples the pending // depth metric). Documents the cadence anchor that // runtime/observability/metrics godoc, docs/ops/alerting-rules.md, and // related cmd/corebundle tests reference by name. DefaultRelayReclaimInterval = 30 * time.Second )
Variables ¶
This section is empty.
Functions ¶
func SanitizeError ¶
SanitizeError redacts sensitive substrings then truncates to maxLen runes before storing in a last_error column.
func TruncateError ¶
TruncateError truncates an error message to maxLen runes, preserving valid UTF-8. Negative or zero maxLen is a no-op.
Types ¶
type ClaimedEntry ¶
ClaimedEntry extends kernel/outbox.Entry with the Attempts counter that is relay-runtime state (not a domain concept) and the LeaseID fencing token that the calling worker must echo to MarkPublished / MarkRetry / MarkDead.
LeaseID is a UUID generated by ClaimPending; it is the only authority over who owns a claiming row. After a stale-claim ReclaimStale, a worker holding the previous LeaseID will fail every CAS — its publish/fail completion becomes a no-op rather than overwriting the new owner's outcome.
type FailureBudget ¶
type FailureBudget struct {
// contains filtered or unexported fields
}
FailureBudget tracks consecutive failures for a named operation and exposes a health.Checker compatible probe. When the consecutive failure count reaches the threshold the budget is "tripped" and Checker() returns a non-nil error. A single success resets the counter and clears the tripped state.
Design: absolute count (no sliding window), success clears zero — mirrors K8s workqueue ItemExponentialFailureRateLimiter.Forget(item) semantics.
ref: k8s.io/client-go/util/workqueue/default_rate_limiters.go — absolute count + Forget(item) clears to zero; no decay window. ref: controller-runtime/pkg/healthz — AddReadyzCheck(name, Checker) aggregation.
func NewFailureBudget ¶
func NewFailureBudget(name string, threshold int) *FailureBudget
NewFailureBudget creates a FailureBudget with the given name and threshold. threshold=0 disables the budget (Checker always returns nil, Tripped always false). Uses slog.Default() as the logger.
func (*FailureBudget) Checker ¶
func (b *FailureBudget) Checker() func(context.Context) error
Checker returns a func(context.Context) error suitable for use as a health.Checker. When threshold is 0 (disabled), returns nil. When the budget is not tripped, the returned func returns nil. When tripped, the returned func returns an error containing the budget name and threshold.
The context parameter is accepted for interface compatibility but is not used: the budget check is a pure atomic-read operation with no I/O.
func (*FailureBudget) ConsecutiveFailures ¶
func (b *FailureBudget) ConsecutiveFailures() int64
ConsecutiveFailures returns the current consecutive failure count.
func (*FailureBudget) Record ¶
func (b *FailureBudget) Record(err error)
Record records the outcome of one operation. err!=nil increments the consecutive failure counter and trips the budget when the threshold is reached. err==nil resets the counter and clears the tripped state.
Thread-safe: uses atomic operations throughout.
func (*FailureBudget) Reset ¶
func (b *FailureBudget) Reset()
Reset clears the consecutive failure counter and tripped state. Called on Relay.Start to avoid stale state from a previous run. Does NOT log; silent state reset.
func (*FailureBudget) Tripped ¶
func (b *FailureBudget) Tripped() bool
Tripped returns true when the consecutive failure count has reached the threshold and has not yet been reset by a successful Record(nil) call.
type PendingDepthObserver ¶
PendingDepthObserver receives the current pending-entry count once per reclaim cycle. The production implementation is runtime/observability/metrics.OutboxPendingDepthCollector; tests may use a simple func adapter. A nil observer is silently ignored (no-op).
Intentionally defined here (not imported from runtime/observability/metrics) to avoid coupling runtime/outbox to its sibling package.
type Relay ¶
type Relay struct {
// contains filtered or unexported fields
}
Relay polls unpublished outbox entries via a Store interface and publishes them via the provided outbox.Publisher using a three-phase approach:
Phase 1 (claim): Store.ClaimPending — short tx in the Store impl Phase 2 (publish): outside tx — publish each entry to broker Phase 3 (writeBack): Store.MarkPublished / MarkRetry / MarkDead — short tx each
Consistency level: L2 (OutboxFact)
Outbox entry state machine:
pending ──claim──→ claiming ──publish ok──→ published ──retention──→ (deleted)
↑ │
│ (fail, attempts < max)
└──────────────────┘
│ (fail, attempts >= max)
↓
dead ──dead retention──→ (deleted)
ReclaimStale: claiming entries past ClaimTTL are recovered with attempts++. If attempts reaches MaxAttempts during reclaim, the entry is marked dead.
ref: Watermill router.go — goroutine-per-handler lifecycle pattern
func NewRelay ¶
NewRelay creates a Relay that polls from store and publishes via pub. Zero or negative cfg values are replaced with defaults via cfg.WithDefaults(). A nil Metrics is replaced with NoopRelayCollector; the collector is then wrapped in safeRelayCollector so panics cannot crash relay goroutines.
func (*Relay) Probes ¶
Probes returns the typed health probes contributed by the Relay, one per enabled failure budget. Each Probe carries a healthz.ProbeName-typed const (ProbePoll / ProbeReclaim / ProbeCleanup) and a Check function with the healthz contract: nil return = healthy; non-nil = unhealthy.
Only budgets with a positive threshold are included; threshold=0 (disabled) budgets are excluded from the slice so callers can iterate and register every returned probe unconditionally.
Consumed by runtime/bootstrap.relayAdapter to satisfy ManagedResource on behalf of *Relay (see ADR docs/architecture/202605201400-adr-relay-managedresource-isolation.md).
ref: controller-runtime/pkg/healthz AddReadyzCheck — named-checker aggregation.
func (*Relay) Ready ¶
func (r *Relay) Ready() <-chan struct{}
Ready returns the channel that is closed when Start() has transitioned the relay to the running state. Callers can use this to synchronize without polling:
select {
case <-relay.Ready():
// relay is running
case <-time.After(deadline):
// timeout
}
Ready() never returns nil. Before Start() completes (or after Stop()), the returned channel is open and will not close until the next Start() runs to completion. Callers that want to detect the "not yet started" state should still guard the select with a timeout.
func (*Relay) Start ¶
Start begins the relay polling loop, cleanup goroutine, and reclaim loop. It blocks until ctx is canceled or Stop is called.
func (*Relay) Stop ¶
Stop signals the relay to shut down gracefully and waits for goroutines. It respects the caller's context deadline: if ctx expires before goroutines finish, Stop returns an error instead of blocking indefinitely. Stop is fully idempotent: calling it multiple times (e.g. via both ManagedResource.Close and WorkerGroup.Stop) is safe and returns nil on every call after the relay is already stopping or stopped.
func (*Relay) WithCommandDispatch ¶
func (r *Relay) WithCommandDispatch( reg *command.Registry, dispatch map[command.CommandID]command.AsyncDispatchFunc, ) *Relay
WithCommandDispatch wires the in-process async command bus into the relay (#1667 / ADR docs/architecture/202606040550-1044-adr-command-bus-dispatch-funnel.md §5 ④). dispatch maps each command id to its generated DispatchAsync; reg is the shared command.Registry the handlers were registered into. When a claimed entry's RoutingTopic equals a key in dispatch, publishBatch routes it to the mapped AsyncDispatchFunc in-process (decode payload → LookupHandler → handler) instead of marshaling + publishing to the broker. Events (and any topic not in the map) are unaffected.
Composition root usage (the dispatch values MUST be generated DispatchAsync symbols — archtest COMMAND-ASYNC-DISPATCH-CALLER-01 locks this):
reg := command.NewRegistry()
_ = enqueue.Register(reg, handler)
relay.WithCommandDispatch(reg, map[command.CommandID]command.AsyncDispatchFunc{
enqueue.DispatchID: enqueue.DispatchAsync,
})
This is a replace-semantics builder option, NOT accumulating: each call sets (does not merge) cmdRegistry + cmdDispatch, so a second call REPLACES the whole table (last-write-wins) — pass all commands in one call. A nil reg or an empty/all-nil dispatch map is a silent no-op (the relay keeps operating event-only, leaving any prior table untouched), and nil entries within the map are dropped. reg and the map values are concrete (pointer / func) types, so plain == nil is the correct nil check here (validation.IsNilInterface guards interface-typed values). Must be called before Start().
func (*Relay) WithPendingDepthObserver ¶
func (r *Relay) WithPendingDepthObserver(o PendingDepthObserver) *Relay
WithPendingDepthObserver wires a PendingDepthObserver that receives the pending-entry count once per reclaim cycle. Both bare-nil and typed-nil inputs are silently ignored (no observer stored); the relay operates without observation when no observer is set. Must be called before Start().
func (*Relay) Worker ¶
Worker returns the Relay itself as the background worker. Relay implements kernel/worker.Worker (Start/Stop), so the bootstrap relay adapter can manage its goroutine lifecycle.
Consumed by runtime/bootstrap.relayAdapter to satisfy ManagedResource on behalf of *Relay.
ref: uber-go/fx internal/lifecycle/lifecycle.go — resource self-reports hook.
type RelayConfig ¶
type RelayConfig struct {
// PollInterval is how often the relay polls for pending entries.
PollInterval time.Duration
// BatchSize is the maximum number of entries fetched per poll cycle.
BatchSize int
// RetentionPeriod is how long published entries are kept before cleanup.
RetentionPeriod time.Duration
// MaxAttempts is the maximum number of publish attempts before an entry
// is marked as dead-lettered. Default 5.
MaxAttempts int
// BaseRetryDelay is the base delay for exponential backoff. Default 5s.
// Actual delay = cappedDelay(BaseRetryDelay * 2^attempts) + jitter.
BaseRetryDelay time.Duration
// ClaimTTL is how long a claiming entry is held before ReclaimStale
// recovers it back to pending. Default 60s.
ClaimTTL time.Duration
// MaxRetryDelay caps the exponential backoff delay to prevent
// unbounded retry intervals at high attempt counts. Default 5m.
MaxRetryDelay time.Duration
// ReclaimInterval controls the independent ReclaimStale goroutine
// frequency, decoupled from cleanup interval. Default 30s.
ReclaimInterval time.Duration
// ReclaimBatchSize caps a single ReclaimStale UPDATE; the relay's reclaim
// loop drains residual on its own tick when the stale row count exceeds
// this cap. Default 1000 (see defaultRelayReclaimBatchSize for the
// industry-comparison rationale).
ReclaimBatchSize int
// DeadRetentionPeriod is how long dead-lettered entries are kept before
// cleanup. Separate from RetentionPeriod to give operators more time
// to investigate and manually retry failed entries. Default 30 days.
DeadRetentionPeriod time.Duration
// Metrics is the relay metrics collector for Prometheus integration.
// If nil, a NoopRelayCollector is used (zero overhead).
// ref: Temporal client.Options{MetricsHandler} — inject-at-construction pattern
Metrics kout.RelayCollector
// PollFailureBudget is the consecutive poll-loop failure count that trips
// /readyz unhealthy. 0 disables the checker. Default 5.
// ref: K8s workqueue ItemExponentialFailureRateLimiter — absolute count + Forget.
PollFailureBudget int
// ReclaimFailureBudget is the consecutive reclaim-loop failure count that
// trips /readyz unhealthy. 0 disables. Default 5.
ReclaimFailureBudget int
// CleanupFailureBudget is the consecutive cleanup-loop failure count that
// trips /readyz unhealthy. 0 disables. Default 5.
CleanupFailureBudget int
// CleanupWaitFloor is the minimum sleep between cleanup passes.
// Exported so tests can lower it to 1ms without touching the global
// constant. <= 0 uses the package default (5s).
// Tests can set it directly via the RelayConfig literal.
CleanupWaitFloor time.Duration
}
RelayConfig configures the outbox relay behavior. Extracted from adapters/postgres/outbox_relay.go to live at the runtime layer so future relay implementations (non-PG) can share the same config surface.
func DefaultRelayConfig ¶
func DefaultRelayConfig() RelayConfig
DefaultRelayConfig returns a RelayConfig with sensible defaults. Field values are identical to adapters/postgres DefaultRelayConfig to ensure zero behavior change during Phase C migration.
func (RelayConfig) WithDefaults ¶
func (c RelayConfig) WithDefaults() RelayConfig
WithDefaults fills zero/negative fields with values from DefaultRelayConfig. It does NOT set Metrics (handled by adapter constructors which wrap in safeRelayCollector). Returns the filled config.
type Store ¶
type Store interface {
// ClaimPending atomically transitions up to batchSize rows from pending
// to claiming status, skipping rows locked by other relay instances.
// Each entry returned carries the LeaseID generated for this claim batch;
// callers MUST echo LeaseID back through Mark* CAS to retain ownership.
// Returns an empty slice + nil error when there is nothing to claim.
ClaimPending(ctx context.Context, batchSize int) ([]ClaimedEntry, error)
// MarkPublished optimistically transitions an entry from claiming to
// published. The leaseID must match the value returned from ClaimPending;
// updated=false means the lease was lost (entry reclaimed by ReclaimStale
// or claimed by another worker) — not an error.
MarkPublished(ctx context.Context, id, leaseID string) (updated bool, err error)
// MarkRetry transitions a failing entry back to pending with attempts
// incremented and the supplied nextRetryAt. Relay is responsible for
// computing nextRetryAt (Go-side backoff). updated=false same as above.
MarkRetry(ctx context.Context, id, leaseID string, attempts int, nextRetryAt time.Time, lastError string) (updated bool, err error)
// MarkDead transitions a failing entry to dead (exceeded max attempts).
// updated=false same as above.
MarkDead(ctx context.Context, id, leaseID string, attempts int, lastError string) (updated bool, err error)
// ReclaimStale transitions up to batchSize claiming rows whose claimed_at
// is older than claimTTL back to pending (with attempts+1 and
// next_retry_at = backoff) or to dead (when attempts+1 >= maxAttempts).
// Returns count of rows recovered across both destinations. Callers MUST
// loop until count < batchSize so a queue larger than one sweep drains
// promptly without producing a multi-second UPDATE that blocks
// VACUUM/replication.
ReclaimStale(
ctx context.Context,
claimTTL time.Duration,
maxAttempts int,
baseDelay, maxDelay time.Duration,
batchSize int,
) (count int, err error)
// CleanupPublished deletes a batch of published rows older than cutoff.
// Caller is responsible for looping until deleted < batchSize.
CleanupPublished(ctx context.Context, cutoff time.Time, batchSize int) (deleted int, err error)
// CleanupDead deletes a batch of dead rows older than cutoff.
CleanupDead(ctx context.Context, cutoff time.Time, batchSize int) (deleted int, err error)
// CountPending returns the number of pending entries eligible for ClaimPending:
// status=pending AND (next_retry_at IS NULL OR next_retry_at <= now()).
// Rows that are pending but still in backoff (next_retry_at > now()) are
// intentionally excluded so the count reflects actual work available to relay
// workers — consistent with the ClaimPending eligibility predicate.
//
// May be approximate under high concurrency — multiple concurrent ClaimPending
// calls can race between the COUNT and the next claim. Callers should treat
// any error as transient and skip the metric update without panicking.
//
// ref: Eventuate transactional-outbox events_pending;
// Debezium MilliSecondsBehindSource — non-blocking depth probe on reclaim cadence.
CountPending(ctx context.Context) (int64, error)
// OldestEligibleAt returns the oldest published_at (when status=kout.StatePublished)
// or dead_at (when status=kout.StateDead) in the table. The relay uses this to
// schedule data-driven cleanup wake-ups: sleep until oldest+retention instead of
// polling on a fixed interval. Returns ok=false when no rows of the given status
// exist.
//
// status MUST be kout.StatePublished or kout.StateDead. Implementations must
// reject other values to keep the contract narrow.
OldestEligibleAt(ctx context.Context, status kout.State) (at time.Time, ok bool, err error)
}
Store is the contract between the relay worker and its backing storage. Implementations MUST be SQL-dialect-neutral at the interface boundary (no *sql.DB / *pgx.Tx in method signatures). Each method opens its own short transaction; methods do not compose into a larger transaction.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package outboxtest provides a public in-memory Store implementation and a Store conformance test suite for use in unit tests.
|
Package outboxtest provides a public in-memory Store implementation and a Store conformance test suite for use in unit tests. |