outbox

package module
v1.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: MIT Imports: 10 Imported by: 4

README

pg-outbox

Transactional outbox for PostgreSQL. Transport-agnostic, horizontally scalable.

The database surface is the Executor interface (Exec / Query / QueryRow). *pgxpool.Pool, pgx.Tx, *pgx.Conn and pgtx's tx.DB all satisfy it, so you can plug your own pooling / routing / instrumentation layer.

Install

go get github.com/gopherex/pg-outbox

Migrate

The SQL is schema-unqualified. Apply it with your tool of choice via outbox.MigrationsFS() (an embed.FS), or apply the raw statements from outbox.Migrations():

for _, stmt := range outbox.Migrations() {
    if _, err := pool.Exec(ctx, stmt); err != nil { log.Fatal(err) }
}

To install into a non-default schema, set search_path on the migration connection; then pass the same schema to WithSchema.

Use

// exec drives the background relay (must NOT be tx-bound);
// enq is pgtx's tx.DB (so Enqueue joins your business transaction).
ob, err := outbox.New(pool, enq, myPublisher,
    outbox.WithInstanceID(os.Getenv("POD_NAME")), // unique, stable per process
    outbox.WithSchema("public"),
    outbox.WithConcurrency(4),
    outbox.WithRetention(72*time.Hour),
    outbox.WithCleanupInterval(time.Hour),
)
if err != nil { log.Fatal(err) }

// Inside a pgtx transaction, atomically with your business writes:
err = tx.DoSerializable(ctx, mgr, func(ctx context.Context) error {
    if err := repo.Save(ctx, order); err != nil { return err }
    return ob.Enqueue(ctx, outbox.Message{
        Topic:        "orders",
        PartitionKey: order.ID,
        Payload:      data,
        Headers:      map[string]string{"trace-id": traceID},
    })
})

// Under the service lifecycle: blocks until ctx is cancelled (returns ctx.Err())
// or until a fatal error stops the relay (returns that error).
err = ob.Run(ctx)

Use a contrib publisher:

import outboxnats "github.com/gopherex/pg-outbox/contrib/publishers/nats"

pub := outboxnats.New(natsConn)
ob, err := outbox.New(pool, enq, pub)

Available publisher contrib modules:

  • github.com/gopherex/pg-outbox/contrib/publishers/nats
  • github.com/gopherex/pg-outbox/contrib/publishers/kafka
  • github.com/gopherex/pg-outbox/contrib/publishers/valkey

Or implement outbox.Publisher for your own transport:

type Publisher interface {
    Publish(ctx context.Context, msgs []outbox.Message) error
}

Plug your own database layer by implementing outbox.Executor:

type Executor interface {
    Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
    Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
    QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
}

LISTEN/NOTIFY wake-ups need a *pgxpool.Pool: supply one with WithListenPool, or pass a *pgxpool.Pool as the relay executor and it is detected automatically. Without one the relay falls back to polling.

Configuration

Two interchangeable styles, usable together (functional options win):

// declarative — mapstructure/validate/default tags, load from file or env:
var cfg outbox.Config // see config.Config fields
ob, _ := outbox.New(pool, enq, pub,
    outbox.WithConfig(cfg),
    outbox.WithLogger(log), // logger injected separately, not config data
)

// or pure options:
ob, _ := outbox.New(pool, enq, pub, outbox.WithInstanceID("pod-1"), ...)

Logging

The logger is the standard library's *slog.Logger, injected via WithLogger. Default is a no-op (slog.DiscardHandler), so nothing is logged until you supply one — no third-party logging dependency:

ob, _ := outbox.New(pool, enq, pub, outbox.WithLogger(slog.Default()))

Retry backoff

Default is exponential 100ms→30s with jitter. Override with WithRetryBackoff. The Backoff type is func(attempt int) time.Duration (stateless, indexed by the message's persisted attempt count — survives restarts). Adapters bridge popular libraries without adding them as dependencies:

// github.com/cenkalti/backoff
outbox.WithRetryBackoff(backoff.FromNextBackOff(func() backoff.NextBackOffer {
    return cbackoff.NewExponentialBackOff()
}))

// github.com/sethvargo/go-retry
outbox.WithRetryBackoff(backoff.FromNexter(func() backoff.Nexter { ... }))

Delivery control

Existing publishers remain compatible: nil means published; an ordinary error retries the entire batch with the configured backoff and attempt limit. A publisher can return these wrapped outcomes (recognized through errors.As):

return outbox.Permanent(err)                  // dead immediately
return outbox.RetryAfter(err, 2*time.Minute)   // minimum delay; attempt limit still applies
return outbox.Discard(reason)                 // terminal, without publication or dead-letter
return outbox.Abort(err)                      // stop the relay; batch left claimed

These outcomes apply to every message in the batch. Use WithBatchSize(1) for independent outcomes and WithConcurrency for parallelism. RetryAfter uses the greater of its delay and the configured backoff; zero/negative delays never reduce backoff. Wrappers preserve errors.Is / errors.As. Do not mix outcome markers; if nested/joined, Abort takes precedence over Permanent, then Discard, then RetryAfter. Nil causes still produce the requested outcome.

Fatal stop and panics

A fatal error stops the relay and the cleaner, and Run returns it to the caller's lifecycle. The batch it hit is not completed, retried or dead-lettered: its rows stay processing under their lease, and another instance (or the restarted one) re-claims them once the lease expires. Other workers are cancelled exactly as on a normal shutdown. Fatal errors are:

  • outbox.Abort(err) returned by the publisher (or by a custom Executor);
  • a panic anywhere in the relay or cleaner (publisher, hooks, executor, lease renewal): it is recovered as *outbox.PanicError with the panic Value and Stack, never crashes the process, and is always fatal. A panicked error value is reachable through errors.Is / errors.As;
  • any error the WithFatal classifier accepts. It is consulted on every Publish error and on every internal error (claim, renew, completion, cleanup) while the context is live; errors caused by cancellation are never classified. Default: nothing else is fatal.
outbox.WithFatal(func(err error) bool { return errors.Is(err, errBrokerGone) })

The library treats payloads as opaque bytes. Application policy, deadlines, provider error classification, recipient preferences, and authorization belong in the caller/publisher. Store only sanitized errors: error text is persisted in last_error and passed to observability hooks.

Claims, cancellation, and completion

Every claim gets a fresh Message.LeaseToken. Completion and renewal require that token and an unexpired processing lease. A stale worker cannot finish, retry, or dead-letter a later attempt, even with the same instance ID or after an operator resets attempts. ErrLeaseLost is observable through the relay's error logger; stale completions do not emit outcome hooks.

While Publish runs, the relay renews the batch every third of LeaseDuration. A renewal failure cancels the publisher context and leaves recovery to lease expiry. Publishers must honor cancellation and configure transport timeouts; WithPublishTimeout optionally bounds the context for each Publish call (disabled by default). Cancellation cannot undo an external side effect, and an adapter that ignores context cannot be forcibly stopped. Delivery remains at-least-once, including when publication succeeds but recording it fails.

Hooks run after database transitions and receive the resulting row snapshots. They are best-effort observations, not a transaction or a durable event stream. An optional DiscardHooks interface adds OnDiscarded(ctx, msgs, reason); existing Hooks implementations do not need to change. Discards never call OnPublished or OnDead. Process termination between commit and a hook can lose that observation.

Terminal data and replay
outbox.WithClearPayloadOnComplete(true) // empty payload + headers on published/discarded
outbox.WithRetention(24*time.Hour)       // delete published rows
outbox.WithDeadRetention(7*24*time.Hour) // delete dead rows
outbox.WithDiscardRetention(time.Hour)  // delete discarded rows
outbox.WithCleanupInterval(time.Minute)

All data-clearing/deletion policies are opt-in. Payload and headers are cleared atomically with the successful/discarded transition; dead rows retain them for investigation and replay. Hooks see the cleared fields when enabled. Other columns, including topic, partition key, and last_error, are not scrubbed. SQL deletion/clearing does not erase historical WAL, backups, or caller logs.

Dead/discarded retention starts at finished_at, published retention at published_at. Cleanup locks selected rows and skips locks, preventing it from deleting a concurrently requeued job. Pending/processing rows are never removed by the cleaner.

ob.Requeue(ctx, id) returns (changed bool, err error). Only dead rows qualify; it resets attempts and terminal/lease state and emits a transactional wake-up. It uses the enqueue executor, so the caller can validate application policy and requeue in the same transaction. It preserves ID, creation time, and data; the next attempt receives a new lease token. Replaying a job can repeat an external side effect. The caller owns permission and eligibility checks.

Upgrade from v1.0

Apply 0002_delivery_control.sql before using the new relay. It adds claim tokens, terminal timestamps, and the generic discarded state. Existing data is preserved; historical dead rows begin their retention window at migration time because their original terminal time is unknown. New retention settings remain disabled unless configured.

Stop/drain old relays before starting new ones. Old binaries do not fence their completion writes; do not mix old and new relays against the same queue. The Publisher, Hooks, and Executor interfaces remain source-compatible.

Package layout

The root package outbox is the core public surface (facade + re-exports). The implementation lives in subpackages: message, port (Publisher/Codec/Hooks/ Executor), config, backoff, migrations, engine (store/relay/cleaner). Concrete publishers live in nested contrib/publishers/* modules so Kafka, NATS and Valkey dependencies never enter the core module's go.mod.

Semantics

  • At-least-once. A failed batch is retried in full; consumers must deduplicate (e.g. on Message.ID).
  • Scaling. Run one relay per instance with a unique WithInstanceID. Claims use FOR UPDATE SKIP LOCKED + a lease (locked_until); a crashed instance's rows are re-claimed once its lease expires.
  • Retries / dead-letter. On failure attempts increments and the row is rescheduled with backoff; after WithMaxAttempts it becomes dead and stays in the table for inspection.
  • Ordering. Best-effort by created_at. WithOrdered(true) enforces strict per-PartitionKey ordering (transaction time, then insertion order, so rows written in one transaction keep their order); a dead row blocks its key until resolved or removed. Published/discarded rows do not block it.
  • Latency. A trigger fires NOTIFY; the relay wakes near-instantly and falls back to polling every WithPollInterval. Once awake it keeps claiming until a pass finds nothing due, so a backlog (including a long run of one key in ordered mode) drains without waiting for the next poll.
  • Scheduling. None: an enqueued row is due immediately; NextAttemptAt is set only by retry backoff.

Notes

  • attempts increments at claim time, so a crash mid-publish counts as an attempt (poison-message protection).
  • The table name is fixed (outbox_messages); only the schema is configurable.
  • Codecs (JSON/proto) are out of scope — bring your own.

Tests

make test                                 # root + contrib module tests, no DB
cd test/integration && go test ./...      # black-box integration (testcontainers Postgres, Docker required)

Contrib publishers and integration tests live in separate nested modules so their dependency trees never enter the core module's go.mod. The core library stays a single pgx dependency.

Documentation

Overview

Package outbox implements a transactional outbox for PostgreSQL.

Messages are written to the outbox table inside the caller's business transaction; a background relay later claims them with FOR UPDATE SKIP LOCKED (leased to an externally supplied instance id) and hands them to a transport-agnostic Publisher. Delivery is at-least-once: consumers must deduplicate.

This file is the package's single public surface: it re-exports the types and constructors from the implementation subpackages (message, port, config, backoff, engine, migrations) and provides the Outbox facade. The database surface is the Executor interface (Exec, Query, QueryRow); plug your own. The logger is the standard library's *slog.Logger, injected via WithLogger.

Index

Constants

View Source
const (
	StatusPending    = message.StatusPending
	StatusProcessing = message.StatusProcessing
	StatusPublished  = message.StatusPublished
	StatusDead       = message.StatusDead
	StatusDiscarded  = message.StatusDiscarded
)

Status values.

View Source
const TableName = message.TableName

TableName is the fixed outbox table name (only the schema is configurable).

Variables

View Source
var (
	WithConfig                 = config.WithConfig
	WithInstanceID             = config.WithInstanceID
	WithSchema                 = config.WithSchema
	WithPollInterval           = config.WithPollInterval
	WithBatchSize              = config.WithBatchSize
	WithLeaseDuration          = config.WithLeaseDuration
	WithConcurrency            = config.WithConcurrency
	WithMaxAttempts            = config.WithMaxAttempts
	WithRetryBackoff           = config.WithRetryBackoff
	WithRetention              = config.WithRetention
	WithDeadRetention          = config.WithDeadRetention
	WithDiscardRetention       = config.WithDiscardRetention
	WithClearPayloadOnComplete = config.WithClearPayloadOnComplete
	WithPublishTimeout         = config.WithPublishTimeout
	WithCleanupInterval        = config.WithCleanupInterval
	WithOrdered                = config.WithOrdered
	WithoutNotify              = config.WithoutNotify
	WithListenPool             = config.WithListenPool
	WithFatal                  = config.WithFatal
	WithHooks                  = config.WithHooks
	WithCodec                  = config.WithCodec
	WithLogger                 = config.WithLogger
)
View Source
var (
	Permanent    = port.Permanent
	RetryAfter   = port.RetryAfter
	Discard      = port.Discard
	Abort        = port.Abort
	ErrLeaseLost = port.ErrLeaseLost
)

Publisher outcome controls apply to the whole batch and preserve wrapped errors.

View Source
var (
	// ErrEmptyTopic is returned when a message has no topic.
	ErrEmptyTopic = message.ErrEmptyTopic
	// ErrNilPayload is returned when a message payload is nil.
	ErrNilPayload = message.ErrNilPayload
	// ErrInvalidSchema is returned by New when the schema is not a valid identifier.
	ErrInvalidSchema = config.ErrInvalidSchema
	// ErrNoCodec is returned by EnqueueValue when no codec was configured.
	ErrNoCodec = errors.New("outbox: no codec configured (use WithCodec)")
	// ErrNoExecutor is returned by New when no relay Executor was supplied.
	ErrNoExecutor = errors.New("outbox: a relay executor is required")
)
View Source
var DefaultBackoff = backoff.Default

DefaultBackoff is exponential 100ms -> 30s with full jitter.

View Source
var ExpBackoff = backoff.Exp

ExpBackoff returns an exponential backoff (base doubled per attempt, capped).

Functions

func Migrations

func Migrations() []string

Migrations returns the contents of every forward-only *.sql file, ordered by filename, ready to pass to Exec for callers applying migrations without a tool.

func MigrationsFS

func MigrationsFS() embed.FS

MigrationsFS returns the embedded migration files for golang-migrate (iofs source), goose, atlas, etc. The SQL is schema-unqualified: set search_path on the migration connection to install into a non-default schema.

Types

type AbortError added in v1.1.1

type AbortError = port.AbortError

type Backoff

type Backoff = backoff.Backoff

Backoff returns the delay before the retry after a given attempt.

type Codec

type Codec = port.Codec

Codec marshals a value to bytes for storage and back.

type Config

type Config = config.Config

Config is the declarative configuration (mapstructure/validate/default).

type DiscardError added in v1.1.0

type DiscardError = port.DiscardError

type DiscardHooks added in v1.1.0

type DiscardHooks = port.DiscardHooks

type Executor

type Executor = port.Executor

Executor is the pluggable pgx query surface (Exec, Query, QueryRow).

type Hooks

type Hooks = port.Hooks

Hooks observes relay activity.

type Message

type Message = message.Message

Message is a single outbox row.

type NoopHooks

type NoopHooks = port.NoopHooks

NoopHooks is the default no-op Hooks implementation.

type Option

type Option = config.Option

Option configures the outbox.

type Outbox

type Outbox struct {
	// contains filtered or unexported fields
}

Outbox is the public facade: enqueue messages (transactionally) and run the background relay + cleaner.

func New

func New(exec Executor, enq Executor, pub Publisher, opts ...Option) (*Outbox, error)

New builds an Outbox.

  • exec is the relay/cleaner Executor: it runs the background claim, mark and cleanup queries and must NOT be bound to a business transaction (pass a *pgxpool.Pool, or your own Executor implementation).
  • enq is the enqueue Executor: pass one that resolves the caller's transaction from the context (e.g. pgtx's tx.DB) so inserts join the business transaction.
  • pub is the transport publisher.

LISTEN/NOTIFY wake-ups require a *pgxpool.Pool: supply one with WithListenPool, or pass a *pgxpool.Pool as exec and it is detected automatically. Without one the relay falls back to polling.

func (*Outbox) Enqueue

func (o *Outbox) Enqueue(ctx context.Context, m Message) error

Enqueue writes one message inside the caller's current transaction.

func (*Outbox) EnqueueBatch

func (o *Outbox) EnqueueBatch(ctx context.Context, ms []Message) error

EnqueueBatch writes multiple messages inside the caller's current transaction.

func (*Outbox) EnqueueValue

func (o *Outbox) EnqueueValue(ctx context.Context, topic, partitionKey, msgType string, v any) error

EnqueueValue marshals v with the configured codec and enqueues it. Requires WithCodec.

func (*Outbox) Requeue added in v1.1.0

func (o *Outbox) Requeue(ctx context.Context, id string) (bool, error)

Requeue atomically returns a dead row to pending and resets its attempts. It uses the enqueue executor, so authorization and application checks can be performed in the same caller transaction. False means no dead row matched. IDs and creation time are preserved; the next claim gets a fresh lease token.

func (*Outbox) Run

func (o *Outbox) Run(ctx context.Context) error

Run starts the relay and cleaner and blocks until ctx is cancelled, then returns ctx.Err(). A fatal error (an Abort outcome, a recovered panic, or an error the WithFatal classifier accepts) stops both and is returned instead; the batch it hit stays claimed and is re-claimed after its lease expires. The lease owner id was resolved at New time (WithInstanceID, or auto-generated).

type PanicError added in v1.1.1

type PanicError = port.PanicError

type PermanentError added in v1.1.0

type PermanentError = port.PermanentError

type Publisher

type Publisher = port.Publisher

Publisher delivers a batch of claimed messages to a transport.

type RetryAfterError added in v1.1.0

type RetryAfterError = port.RetryAfterError

type Settings

type Settings = config.Settings

Settings is the resolved runtime configuration.

type Status

type Status = message.Status

Status is the lifecycle state of an outbox message.

Directories

Path Synopsis
Package backoff provides retry backoff strategies for the outbox relay.
Package backoff provides retry backoff strategies for the outbox relay.
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment.
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment.
contrib
libs/xtrace module
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers.
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers.
Package message defines the outbox data model.
Package message defines the outbox data model.
Package migrations exposes the embedded outbox SQL migrations.
Package migrations exposes the embedded outbox SQL migrations.
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor.
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL