Documentation
¶
Overview ¶
Package outbox implements a transactional outbox for PostgreSQL.
Messages are written to the outbox table inside the caller's business transaction; a background relay later claims them with FOR UPDATE SKIP LOCKED (leased to an externally supplied instance id) and hands them to a transport-agnostic Publisher. Delivery is at-least-once: consumers must deduplicate.
This file is the package's single public surface: it re-exports the types and constructors from the implementation subpackages (message, port, config, backoff, engine, migrations) and provides the Outbox facade. The database surface is the Executor interface (Exec, Query, QueryRow); plug your own. The logger is the standard library's *slog.Logger, injected via WithLogger.
Index ¶
- Constants
- Variables
- func Migrations() []string
- func MigrationsFS() embed.FS
- type AbortError
- type Backoff
- type BatchError
- type Codec
- type Config
- type DiscardError
- type DiscardHooks
- type Executor
- type Hooks
- type Instrumentation
- type Message
- type NoopHooks
- type NoopInstrumentation
- type Option
- type Outbox
- func (o *Outbox) Enqueue(ctx context.Context, m Message) error
- func (o *Outbox) EnqueueBatch(ctx context.Context, ms []Message) error
- func (o *Outbox) EnqueueValue(ctx context.Context, topic, partitionKey, msgType string, v any) error
- func (o *Outbox) Published(ctx context.Context, ids ...string) (bool, error)
- func (o *Outbox) Requeue(ctx context.Context, id string) (bool, error)
- func (o *Outbox) Run(ctx context.Context) error
- func (o *Outbox) WaitPublished(ctx context.Context, ids ...string) error
- type Outcome
- type PanicError
- type PermanentError
- type Publisher
- type RetryAfterError
- type Settings
- type Status
- type UndeliveredError
- type UnknownOutcomeError
Constants ¶
const ( StatusPending = message.StatusPending StatusProcessing = message.StatusProcessing StatusPublished = message.StatusPublished StatusDead = message.StatusDead StatusDiscarded = message.StatusDiscarded )
Status values.
const TableName = message.TableName
TableName is the fixed outbox table name (only the schema is configurable).
Variables ¶
var ( WithConfig = config.WithConfig WithInstanceID = config.WithInstanceID WithSchema = config.WithSchema WithPollInterval = config.WithPollInterval WithBatchSize = config.WithBatchSize WithLeaseDuration = config.WithLeaseDuration WithConcurrency = config.WithConcurrency WithMaxAttempts = config.WithMaxAttempts WithRetryBackoff = config.WithRetryBackoff WithRetention = config.WithRetention WithDeadRetention = config.WithDeadRetention WithDiscardRetention = config.WithDiscardRetention WithClearPayloadOnComplete = config.WithClearPayloadOnComplete WithPublishTimeout = config.WithPublishTimeout WithCleanupInterval = config.WithCleanupInterval WithOrdered = config.WithOrdered WithoutNotify = config.WithoutNotify WithWaitRecheck = config.WithWaitRecheck WithSettleLinger = config.WithSettleLinger WithListenPool = config.WithListenPool WithFatal = config.WithFatal WithHooks = config.WithHooks WithInstrumentation = config.WithInstrumentation WithErrorRenderer = config.WithErrorRenderer WithTombstoneRetention = config.WithTombstoneRetention WithCodec = config.WithCodec WithLogger = config.WithLogger )
var ( Permanent = port.Permanent RetryAfter = port.RetryAfter Discard = port.Discard Abort = port.Abort PerMessage = port.PerMessage ErrLeaseLost = port.ErrLeaseLost )
Publisher outcome controls apply to the whole batch, or per message through PerMessage, and preserve wrapped errors.
var ( // ErrNotSettling: Settle outside a relay Publish call or after it returned. ErrNotSettling = port.ErrNotSettling // ErrUnknownMessage: Settle for a message not in the batch. ErrUnknownMessage = port.ErrUnknownMessage // ErrAlreadySettled: a second Settle for the same message. ErrAlreadySettled = port.ErrAlreadySettled )
var ( // ErrEmptyTopic is returned when a message has no topic. ErrEmptyTopic = message.ErrEmptyTopic // ErrNilPayload is returned when a message payload is nil. ErrNilPayload = message.ErrNilPayload // ErrInvalidSchema is returned by New when the schema is not a valid identifier. ErrInvalidSchema = config.ErrInvalidSchema // ErrNoCodec is returned by EnqueueValue when no codec was configured. ErrNoCodec = errors.New("outbox: no codec configured (use WithCodec)") // ErrNoExecutor is returned by New when no relay Executor was supplied. ErrNoExecutor = errors.New("outbox: a relay executor is required") // ErrInvalidID is returned by Published / WaitPublished for a non-uuid id. ErrInvalidID = engine.ErrInvalidID // ErrUndelivered matches an *UndeliveredError through errors.Is. ErrUndelivered = engine.ErrUndelivered // ErrOutcomeUnknown matches an *UnknownOutcomeError through errors.Is. ErrOutcomeUnknown = engine.ErrOutcomeUnknown )
var DefaultBackoff = backoff.Default
DefaultBackoff is exponential 100ms -> 30s with full jitter.
var ExpBackoff = backoff.Exp
ExpBackoff returns an exponential backoff (base doubled per attempt, capped).
var MessageContext = port.MessageContext
MessageContext returns the context the Instrumentation opened for m's delivery, or ctx when there is none. Publishers that send messages one by one publish m with it.
var Settle = port.Settle
Settle reports one message's outcome from inside Publish as soon as it is known, so the row stops holding its partition; see port.Settle.
Functions ¶
func Migrations ¶
func Migrations() []string
Migrations returns the contents of every forward-only *.sql file, ordered by filename, ready to pass to Exec for callers applying migrations without a tool.
func MigrationsFS ¶
MigrationsFS returns the embedded migration files for golang-migrate (iofs source), goose, atlas, etc. The SQL is schema-unqualified: set search_path on the migration connection to install into a non-default schema.
Types ¶
type AbortError ¶ added in v1.1.1
type AbortError = port.AbortError
type BatchError ¶ added in v1.2.0
type BatchError = port.BatchError
type DiscardError ¶ added in v1.1.0
type DiscardError = port.DiscardError
type DiscardHooks ¶ added in v1.1.0
type DiscardHooks = port.DiscardHooks
type Instrumentation ¶ added in v1.4.0
type Instrumentation = port.Instrumentation
Instrumentation plugs telemetry into Enqueue and delivery.
type NoopInstrumentation ¶ added in v1.4.0
type NoopInstrumentation = port.NoopInstrumentation
type Outbox ¶
type Outbox struct {
// contains filtered or unexported fields
}
Outbox is the public facade: enqueue messages (transactionally) and run the background relay + cleaner.
func New ¶
New builds an Outbox.
- exec is the relay/cleaner Executor: it runs the background claim, mark and cleanup queries and must NOT be bound to a business transaction (pass a *pgxpool.Pool, or your own Executor implementation).
- enq is the enqueue Executor: pass one that resolves the caller's transaction from the context (e.g. pgtx's tx.DB) so inserts join the business transaction.
- pub is the transport publisher.
LISTEN/NOTIFY wake-ups require a *pgxpool.Pool: supply one with WithListenPool, or pass a *pgxpool.Pool as exec and it is detected automatically. Without one the relay falls back to polling.
func (*Outbox) EnqueueBatch ¶
EnqueueBatch writes multiple messages inside the caller's current transaction.
func (*Outbox) EnqueueValue ¶
func (o *Outbox) EnqueueValue(ctx context.Context, topic, partitionKey, msgType string, v any) error
EnqueueValue marshals v with the configured codec and enqueues it. Requires WithCodec.
func (*Outbox) Published ¶ added in v1.3.0
Published reports whether every listed message is delivered. A row still pending or being published makes it false. A row that ended dead or discarded returns an error matching ErrUndelivered (an *UndeliveredError naming the rows): it will not be delivered unless requeued. ids are the Message.IDs given to Enqueue; a non-uuid id returns ErrInvalidID. It reads only those rows, by primary key, through the relay executor (never a caller transaction), so other rows and partitions do not affect it.
A removed dead or discarded row leaves a tombstone for WithTombstoneRetention, so its refusal is still reported. An id with neither row nor tombstone is delivered when the cleaner removes no refused rows (only a published row can be gone), or when it is a UUIDv7 created within the horizon (shortest refusal retention + tombstone retention − 1h): a refusal that recent would still have its tombstone. Any other such id returns an error matching ErrOutcomeUnknown (an *UnknownOutcomeError naming the rows). Configure the Outbox that asks with the same retentions as the one that cleans. An id never enqueued reads as gone as well.
func (*Outbox) Requeue ¶ added in v1.1.0
Requeue atomically returns a dead row to pending and resets its attempts. It uses the enqueue executor, so authorization and application checks can be performed in the same caller transaction. False means no dead row matched. IDs and creation time are preserved; the next claim gets a fresh lease token.
func (*Outbox) Run ¶
Run starts the relay and cleaner and blocks until ctx is cancelled, then returns ctx.Err(). A fatal error (an Abort outcome, a recovered panic, or an error the WithFatal classifier accepts) stops both and is returned instead; the batch it hit stays claimed and is re-claimed after its lease expires. The lease owner id was resolved at New time (WithInstanceID, or auto-generated).
func (*Outbox) WaitPublished ¶ added in v1.3.0
WaitPublished blocks until every listed message is delivered (as Published reports it), returning nil; or until one ends dead or discarded, returning an error matching ErrUndelivered; or until ctx ends, returning an error that wraps ctx.Err() (and the last failed state read, if any).
It learns outcomes from the relays' commit notifications (DoneChannel), whichever process published the rows, through the LISTEN connection of this Outbox's Run: no query per wait beyond the first check. Without that listener (Run not running here, no LISTEN pool, WithoutNotify) or for a missed notification, one shared re-read for all waiting calls every WithWaitRecheck interval settles it.
type PanicError ¶ added in v1.1.1
type PanicError = port.PanicError
type PermanentError ¶ added in v1.1.0
type PermanentError = port.PermanentError
type RetryAfterError ¶ added in v1.1.0
type RetryAfterError = port.RetryAfterError
type UndeliveredError ¶ added in v1.3.0
type UndeliveredError = engine.UndeliveredError
UndeliveredError names the rows Published / WaitPublished found dead or discarded; they will not be delivered unless requeued.
type UnknownOutcomeError ¶ added in v1.5.1
type UnknownOutcomeError = engine.UnknownOutcomeError
UnknownOutcomeError names the rows Published / WaitPublished found gone while dead or discarded rows are collected too, so their outcome is unknown.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package backoff provides retry backoff strategies for the outbox relay.
|
Package backoff provides retry backoff strategies for the outbox relay. |
|
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment.
|
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment. |
|
contrib
|
|
|
libs/xtrace
module
|
|
|
publishers/kafka
module
|
|
|
publishers/nats
module
|
|
|
publishers/valkey
module
|
|
|
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers.
|
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers. |
|
Package message defines the outbox data model.
|
Package message defines the outbox data model. |
|
Package migrations exposes the embedded outbox SQL migrations.
|
Package migrations exposes the embedded outbox SQL migrations. |
|
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor.
|
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor. |