Documentation
¶
Overview ¶
Package outbox implements a transactional outbox for PostgreSQL.
Messages are written to the outbox table inside the caller's business transaction; a background relay later claims them with FOR UPDATE SKIP LOCKED (leased to an externally supplied instance id) and hands them to a transport-agnostic Publisher. Delivery is at-least-once: consumers must deduplicate.
This file is the package's single public surface: it re-exports the types and constructors from the implementation subpackages (message, port, config, backoff, engine, migrations) and provides the Outbox facade. The database surface is the Executor interface (Exec, Query, QueryRow); plug your own. The logger is the standard library's *slog.Logger, injected via WithLogger.
Index ¶
- Constants
- Variables
- func Migrations() []string
- func MigrationsFS() embed.FS
- type AbortError
- type Backoff
- type BatchError
- type Codec
- type Config
- type DiscardError
- type DiscardHooks
- type Executor
- type Hooks
- type Instrumentation
- type Message
- type NoopHooks
- type NoopInstrumentation
- type Option
- type Outbox
- func (o *Outbox) Enqueue(ctx context.Context, m Message) error
- func (o *Outbox) EnqueueBatch(ctx context.Context, ms []Message) error
- func (o *Outbox) EnqueueValue(ctx context.Context, topic, partitionKey, msgType string, v any) error
- func (o *Outbox) Published(ctx context.Context, ids ...string) (bool, error)
- func (o *Outbox) Requeue(ctx context.Context, id string) (bool, error)
- func (o *Outbox) Run(ctx context.Context) error
- func (o *Outbox) WaitPublished(ctx context.Context, ids ...string) error
- type Outcome
- type PanicError
- type PermanentError
- type Publisher
- type RetryAfterError
- type Settings
- type Status
- type UndeliveredError
Constants ¶
const ( StatusPending = message.StatusPending StatusProcessing = message.StatusProcessing StatusPublished = message.StatusPublished StatusDead = message.StatusDead StatusDiscarded = message.StatusDiscarded )
Status values.
const TableName = message.TableName
TableName is the fixed outbox table name (only the schema is configurable).
Variables ¶
var ( WithConfig = config.WithConfig WithInstanceID = config.WithInstanceID WithSchema = config.WithSchema WithPollInterval = config.WithPollInterval WithBatchSize = config.WithBatchSize WithLeaseDuration = config.WithLeaseDuration WithConcurrency = config.WithConcurrency WithMaxAttempts = config.WithMaxAttempts WithRetryBackoff = config.WithRetryBackoff WithRetention = config.WithRetention WithDeadRetention = config.WithDeadRetention WithDiscardRetention = config.WithDiscardRetention WithClearPayloadOnComplete = config.WithClearPayloadOnComplete WithPublishTimeout = config.WithPublishTimeout WithCleanupInterval = config.WithCleanupInterval WithOrdered = config.WithOrdered WithoutNotify = config.WithoutNotify WithWaitRecheck = config.WithWaitRecheck WithListenPool = config.WithListenPool WithFatal = config.WithFatal WithHooks = config.WithHooks WithInstrumentation = config.WithInstrumentation WithCodec = config.WithCodec WithLogger = config.WithLogger )
var ( Permanent = port.Permanent RetryAfter = port.RetryAfter Discard = port.Discard Abort = port.Abort PerMessage = port.PerMessage ErrLeaseLost = port.ErrLeaseLost )
Publisher outcome controls apply to the whole batch, or per message through PerMessage, and preserve wrapped errors.
var ( // ErrEmptyTopic is returned when a message has no topic. ErrEmptyTopic = message.ErrEmptyTopic // ErrNilPayload is returned when a message payload is nil. ErrNilPayload = message.ErrNilPayload // ErrInvalidSchema is returned by New when the schema is not a valid identifier. ErrInvalidSchema = config.ErrInvalidSchema // ErrNoCodec is returned by EnqueueValue when no codec was configured. ErrNoCodec = errors.New("outbox: no codec configured (use WithCodec)") // ErrNoExecutor is returned by New when no relay Executor was supplied. ErrNoExecutor = errors.New("outbox: a relay executor is required") // ErrInvalidID is returned by Published / WaitPublished for a non-uuid id. ErrInvalidID = engine.ErrInvalidID // ErrUndelivered matches an *UndeliveredError through errors.Is. ErrUndelivered = engine.ErrUndelivered )
var DefaultBackoff = backoff.Default
DefaultBackoff is exponential 100ms -> 30s with full jitter.
var ExpBackoff = backoff.Exp
ExpBackoff returns an exponential backoff (base doubled per attempt, capped).
var MessageContext = port.MessageContext
MessageContext returns the context the Instrumentation opened for m's delivery, or ctx when there is none. Publishers that send messages one by one publish m with it.
Functions ¶
func Migrations ¶
func Migrations() []string
Migrations returns the contents of every forward-only *.sql file, ordered by filename, ready to pass to Exec for callers applying migrations without a tool.
func MigrationsFS ¶
MigrationsFS returns the embedded migration files for golang-migrate (iofs source), goose, atlas, etc. The SQL is schema-unqualified: set search_path on the migration connection to install into a non-default schema.
Types ¶
type AbortError ¶ added in v1.1.1
type AbortError = port.AbortError
type BatchError ¶ added in v1.2.0
type BatchError = port.BatchError
type DiscardError ¶ added in v1.1.0
type DiscardError = port.DiscardError
type DiscardHooks ¶ added in v1.1.0
type DiscardHooks = port.DiscardHooks
type Instrumentation ¶ added in v1.4.0
type Instrumentation = port.Instrumentation
Instrumentation plugs telemetry into Enqueue and delivery.
type NoopInstrumentation ¶ added in v1.4.0
type NoopInstrumentation = port.NoopInstrumentation
type Outbox ¶
type Outbox struct {
// contains filtered or unexported fields
}
Outbox is the public facade: enqueue messages (transactionally) and run the background relay + cleaner.
func New ¶
New builds an Outbox.
- exec is the relay/cleaner Executor: it runs the background claim, mark and cleanup queries and must NOT be bound to a business transaction (pass a *pgxpool.Pool, or your own Executor implementation).
- enq is the enqueue Executor: pass one that resolves the caller's transaction from the context (e.g. pgtx's tx.DB) so inserts join the business transaction.
- pub is the transport publisher.
LISTEN/NOTIFY wake-ups require a *pgxpool.Pool: supply one with WithListenPool, or pass a *pgxpool.Pool as exec and it is detected automatically. Without one the relay falls back to polling.
func (*Outbox) EnqueueBatch ¶
EnqueueBatch writes multiple messages inside the caller's current transaction.
func (*Outbox) EnqueueValue ¶
func (o *Outbox) EnqueueValue(ctx context.Context, topic, partitionKey, msgType string, v any) error
EnqueueValue marshals v with the configured codec and enqueues it. Requires WithCodec.
func (*Outbox) Published ¶ added in v1.3.0
Published reports whether every listed message is delivered: published, or already removed by the cleaner. A row still pending or being published makes it false. A row that ended dead or discarded returns an error matching ErrUndelivered (an *UndeliveredError naming the rows): it will not be delivered unless requeued. ids are the Message.IDs given to Enqueue; a non-uuid id returns ErrInvalidID. It reads only those rows, by primary key, through the relay executor (never a caller transaction), so other rows and partitions do not affect it.
An id with no row reads as delivered: that covers rows the cleaner removed, but also an id never enqueued and a dead row removed by WithDeadRetention.
func (*Outbox) Requeue ¶ added in v1.1.0
Requeue atomically returns a dead row to pending and resets its attempts. It uses the enqueue executor, so authorization and application checks can be performed in the same caller transaction. False means no dead row matched. IDs and creation time are preserved; the next claim gets a fresh lease token.
func (*Outbox) Run ¶
Run starts the relay and cleaner and blocks until ctx is cancelled, then returns ctx.Err(). A fatal error (an Abort outcome, a recovered panic, or an error the WithFatal classifier accepts) stops both and is returned instead; the batch it hit stays claimed and is re-claimed after its lease expires. The lease owner id was resolved at New time (WithInstanceID, or auto-generated).
func (*Outbox) WaitPublished ¶ added in v1.3.0
WaitPublished blocks until every listed message is delivered (as Published reports it), returning nil; or until one ends dead or discarded, returning an error matching ErrUndelivered; or until ctx ends, returning an error that wraps ctx.Err() (and the last failed state read, if any).
It learns outcomes from the relays' commit notifications (DoneChannel), whichever process published the rows, through the LISTEN connection of this Outbox's Run: no query per wait beyond the first check. Without that listener (Run not running here, no LISTEN pool, WithoutNotify) or for a missed notification, one shared re-read for all waiting calls every WithWaitRecheck interval settles it.
type PanicError ¶ added in v1.1.1
type PanicError = port.PanicError
type PermanentError ¶ added in v1.1.0
type PermanentError = port.PermanentError
type RetryAfterError ¶ added in v1.1.0
type RetryAfterError = port.RetryAfterError
type UndeliveredError ¶ added in v1.3.0
type UndeliveredError = engine.UndeliveredError
UndeliveredError names the rows Published / WaitPublished found dead or discarded; they will not be delivered unless requeued.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package backoff provides retry backoff strategies for the outbox relay.
|
Package backoff provides retry backoff strategies for the outbox relay. |
|
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment.
|
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment. |
|
contrib
|
|
|
libs/xtrace
module
|
|
|
publishers/kafka
module
|
|
|
publishers/nats
module
|
|
|
publishers/valkey
module
|
|
|
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers.
|
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers. |
|
Package message defines the outbox data model.
|
Package message defines the outbox data model. |
|
Package migrations exposes the embedded outbox SQL migrations.
|
Package migrations exposes the embedded outbox SQL migrations. |
|
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor.
|
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor. |