outbox

package module
v1.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT Imports: 11 Imported by: 4

README

pg-outbox

Transactional outbox for PostgreSQL. Transport-agnostic, horizontally scalable.

The database surface is the Executor interface (Exec / Query / QueryRow). *pgxpool.Pool, pgx.Tx, *pgx.Conn and pgtx's tx.DB all satisfy it, so you can plug your own pooling / routing / instrumentation layer.

Install

go get github.com/gopherex/pg-outbox

Migrate

The SQL is schema-unqualified. Apply it with your tool of choice via outbox.MigrationsFS() (an embed.FS), or apply the raw statements from outbox.Migrations():

for _, stmt := range outbox.Migrations() {
    if _, err := pool.Exec(ctx, stmt); err != nil { log.Fatal(err) }
}

To install into a non-default schema, set search_path on the migration connection; then pass the same schema to WithSchema.

Use

// exec drives the background relay (must NOT be tx-bound);
// enq is pgtx's tx.DB (so Enqueue joins your business transaction).
ob, err := outbox.New(pool, enq, myPublisher,
    outbox.WithInstanceID(os.Getenv("POD_NAME")), // unique, stable per process
    outbox.WithSchema("public"),
    outbox.WithConcurrency(4),
    outbox.WithRetention(72*time.Hour),
    outbox.WithCleanupInterval(time.Hour),
)
if err != nil { log.Fatal(err) }

// Inside a pgtx transaction, atomically with your business writes:
err = tx.DoSerializable(ctx, mgr, func(ctx context.Context) error {
    if err := repo.Save(ctx, order); err != nil { return err }
    return ob.Enqueue(ctx, outbox.Message{
        Topic:        "orders",
        PartitionKey: order.ID,
        Payload:      data,
        Headers:      map[string]string{"trace-id": traceID},
    })
})

// Under the service lifecycle: blocks until ctx is cancelled (returns ctx.Err())
// or until a fatal error stops the relay (returns that error).
err = ob.Run(ctx)

Use a contrib publisher:

import outboxnats "github.com/gopherex/pg-outbox/contrib/publishers/nats"

pub := outboxnats.New(natsConn)
ob, err := outbox.New(pool, enq, pub)

Available publisher contrib modules:

  • github.com/gopherex/pg-outbox/contrib/publishers/nats
  • github.com/gopherex/pg-outbox/contrib/publishers/kafka
  • github.com/gopherex/pg-outbox/contrib/publishers/valkey

Or implement outbox.Publisher for your own transport:

type Publisher interface {
    Publish(ctx context.Context, msgs []outbox.Message) error
}

Plug your own database layer by implementing outbox.Executor:

type Executor interface {
    Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
    Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
    QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
}

LISTEN/NOTIFY wake-ups need a *pgxpool.Pool: supply one with WithListenPool, or pass a *pgxpool.Pool as the relay executor and it is detected automatically. Without one the relay falls back to polling.

Configuration

Two interchangeable styles, usable together (functional options win):

// declarative — mapstructure/validate/default tags, load from file or env:
var cfg outbox.Config // see config.Config fields
ob, _ := outbox.New(pool, enq, pub,
    outbox.WithConfig(cfg),
    outbox.WithLogger(log), // logger injected separately, not config data
)

// or pure options:
ob, _ := outbox.New(pool, enq, pub, outbox.WithInstanceID("pod-1"), ...)

Logging

The logger is the standard library's *slog.Logger, injected via WithLogger. Default is a no-op (slog.DiscardHandler), so nothing is logged until you supply one — no third-party logging dependency:

ob, _ := outbox.New(pool, enq, pub, outbox.WithLogger(slog.Default()))

Retry backoff

Default is exponential 100ms→30s with jitter. Override with WithRetryBackoff. The Backoff type is func(attempt int) time.Duration (stateless, indexed by the message's persisted attempt count — survives restarts). Adapters bridge popular libraries without adding them as dependencies:

// github.com/cenkalti/backoff
outbox.WithRetryBackoff(backoff.FromNextBackOff(func() backoff.NextBackOffer {
    return cbackoff.NewExponentialBackOff()
}))

// github.com/sethvargo/go-retry
outbox.WithRetryBackoff(backoff.FromNexter(func() backoff.Nexter { ... }))

Delivery control

Existing publishers remain compatible: nil means published; an ordinary error retries the entire batch with the configured backoff and attempt limit. A publisher can return these wrapped outcomes (recognized through errors.As):

return outbox.Permanent(err)                  // dead immediately
return outbox.RetryAfter(err, 2*time.Minute)   // minimum delay; attempt limit still applies
return outbox.Discard(reason)                 // terminal, without publication or dead-letter
return outbox.Abort(err)                      // stop the relay; batch left claimed

These outcomes apply to every message in the batch. RetryAfter uses the greater of its delay and the configured backoff; zero/negative delays never reduce backoff. Wrappers preserve errors.Is / errors.As. Do not mix outcome markers; if nested/joined, Abort takes precedence over PerMessage, then Permanent, then Discard, then RetryAfter. Nil causes still produce the requested outcome.

For a batch of unrelated messages, give each its own outcome, keyed by Message.ID:

return outbox.PerMessage(map[string]error{
    a.ID: outbox.Permanent(err),           // only a is dead
    c.ID: outbox.RetryAfter(err, time.Minute),
}, nil)                                    // every other message: published

A nil entry means published; a non-nil entry is settled exactly like the whole-batch outcomes above (a plain error retries with backoff). Messages without an entry take the second argument as their outcome. IDs outside the batch are ignored. An Abort in any entry stops the relay with the whole batch left claimed, and a WithFatal classifier sees the whole *BatchError (errors.Is / errors.As reach every entry). In ordered mode each failed head holds only its own key, exactly as with a whole-batch failure.

Settling messages early

Publish returns once per batch, so by default every row of a batch stays processing until the slowest one is done, holding the next row of its partition, even rows of other partitions already delivered. A publisher that learns outcomes one by one (a JetStream ack per message) reports each as soon as it knows it:

func (p *jsPublisher) Publish(ctx context.Context, msgs []outbox.Message) error {
    for _, m := range msgs {
        go func() {
            _, err := p.js.PublishMsg(outbox.MessageContext(ctx, m), toNATS(m))
            _ = outbox.Settle(ctx, m, err) // nil = published; outcomes as in PerMessage
        }()
    }
    // ... wait for all acks
    return nil
}

The relay commits early outcomes right away, group-committed within WithSettleLinger (default 1ms) or until Publish returns, so a row stops holding its partition and another worker (WithConcurrency > 1) claims the partition's next row while the batch's slow rows still publish. Ordering per PartitionKey is unchanged. Rows a publisher does not settle take what Publish returns. Leases of the rows still in flight keep being renewed.

  • Settle outcomes are read like PerMessage entries; a fatal one (Abort, or one WithFatal accepts) leaves that row claimed and stops the relay when Publish returns, while rows settled before it stay committed.
  • Misuse is reported: ErrNotSettling outside a relay Publish call (or after it returned), ErrUnknownMessage for a message not in the batch, ErrAlreadySettled for a second report.
  • A publisher that settles every message costs about as many statements as a batch return.
Fatal stop and panics

A fatal error stops the relay and the cleaner, and Run returns it to the caller's lifecycle. The batch it hit is not completed, retried or dead-lettered: its rows stay processing under their lease, and another instance (or the restarted one) re-claims them once the lease expires. Other workers are cancelled exactly as on a normal shutdown. Fatal errors are:

  • outbox.Abort(err) returned by the publisher (or by a custom Executor);
  • a panic anywhere in the relay or cleaner (publisher, hooks, executor, lease renewal): it is recovered as *outbox.PanicError with the panic Value and Stack, never crashes the process, and is always fatal. A panicked error value is reachable through errors.Is / errors.As;
  • any error the WithFatal classifier accepts. It is consulted on every Publish error and on every internal error (claim, renew, completion, cleanup) while the context is live; errors caused by cancellation are never classified. Default: nothing else is fatal.
outbox.WithFatal(func(err error) bool { return errors.Is(err, errBrokerGone) })

The library treats payloads as opaque bytes. Application policy, deadlines, provider error classification, recipient preferences, and authorization belong in the caller/publisher. Store only sanitized errors: error text is persisted in last_error and passed to observability hooks.

Claims, cancellation, and completion

Every claim gets a fresh Message.LeaseToken. Completion and renewal require that token and an unexpired processing lease. A stale worker cannot finish, retry, or dead-letter a later attempt, even with the same instance ID or after an operator resets attempts. ErrLeaseLost is observable through the relay's error logger; stale completions do not emit outcome hooks.

While Publish runs, the relay renews the batch every third of LeaseDuration. A renewal failure cancels the publisher context and leaves recovery to lease expiry. Publishers must honor cancellation and configure transport timeouts; WithPublishTimeout optionally bounds the context for each Publish call (disabled by default). Cancellation cannot undo an external side effect, and an adapter that ignores context cannot be forcibly stopped. Delivery remains at-least-once, including when publication succeeds but recording it fails.

Hooks run after database transitions and receive the resulting row snapshots. They are best-effort observations, not a transaction or a durable event stream. An optional DiscardHooks interface adds OnDiscarded(ctx, msgs, reason); existing Hooks implementations do not need to change. Discards never call OnPublished or OnDead. Process termination between commit and a hook can lose that observation.

Terminal data and replay
outbox.WithClearPayloadOnComplete(true) // empty payload + headers on published/discarded
outbox.WithRetention(24*time.Hour)       // delete published rows
outbox.WithDeadRetention(7*24*time.Hour) // delete dead rows
outbox.WithDiscardRetention(time.Hour)  // delete discarded rows
outbox.WithCleanupInterval(time.Minute)

All data-clearing/deletion policies are opt-in. Payload and headers are cleared atomically with the successful/discarded transition; dead rows retain them for investigation and replay. Hooks see the cleared fields when enabled. Other columns, including topic, partition key, and last_error, are not scrubbed. SQL deletion/clearing does not erase historical WAL, backups, or caller logs.

Dead/discarded retention starts at finished_at, published retention at published_at. Cleanup locks selected rows and skips locks, preventing it from deleting a concurrently requeued job. Pending/processing rows are never removed by the cleaner.

ob.Requeue(ctx, id) returns (changed bool, err error). Only dead rows qualify; it resets attempts and terminal/lease state and emits a transactional wake-up. It uses the enqueue executor, so the caller can validate application policy and requeue in the same transaction. It preserves ID, creation time, and data; the next attempt receives a new lease token. Replaying a job can repeat an external side effect. The caller owns permission and eligibility checks.

Instrumentation

outbox.WithInstrumentation(i) plugs telemetry into both ends of the outbox (default NoopInstrumentation). The core has no telemetry dependency; ready implementations live in contrib/libs/* modules: contrib/libs/xtrace traces each delivery as a new trace linked to the span that enqueued it.

type Instrumentation interface {
    Capture(ctx context.Context, headers map[string]string)
    Deliver(ctx context.Context, m Message) (context.Context, func(Outcome))
}
  • Capture runs at Enqueue, in the caller's context, before the insert. What it writes into the fresh headers map is merged into the message headers (stored in the row, sent to the transport); the caller's own entries win and the caller's map is never written.
  • Deliver runs in the relay for every claimed message before Publish. The context it returns is derived from the publish context (deadline and cancellation apply); a Publisher that sends messages one by one gets it with outbox.MessageContext(ctx, m). The built-in Valkey publishers do.
  • The messages handed to Publish also carry Capture of their delivery context in Headers (over the stored entries; the row is not changed), so every publisher that maps Message.Headers to the transport — NATS headers, Kafka headers, Valkey stream fields — carries the delivery on to consumers with no context API of its own.
  • The returned end func is called exactly once per message with its Outcome: the committed status (published, pending for a scheduled retry, dead, discarded, per message under PerMessage), or processing with the cause when the attempt left the row claimed (abort, panic, lost lease, failed completion).
Waiting for delivery

A caller that must not answer before its own messages left the outbox (for example, a command whose effect other services must see first) asks by the Message.IDs it enqueued:

done, err := ob.Published(ctx, ids...) // one read: true = all delivered
err = ob.WaitPublished(ctx, ids...)    // blocks until delivered, undeliverable, or ctx ends
  • Delivered means published, or no row at all while the cleaner removes no refused rows (no WithDeadRetention / WithDiscardRetention): then only a published row can be gone. With either retention on, a gone row may have been dead or discarded too, so it returns an error matching outbox.ErrOutcomeUnknown (an *outbox.UnknownOutcomeError naming the rows) instead of a false success. Give the Outbox that asks the same retention options as the one that cleans. An id never enqueued reads as gone.
  • A row that ended dead or discarded returns an error matching outbox.ErrUndelivered (an *outbox.UndeliveredError naming the rows): it will not be delivered unless requeued. A non-uuid id returns ErrInvalidID.
  • WaitPublished returns an error wrapping ctx.Err() (and the last failed state read, if any) when ctx ends first.
  • Only the listed rows are read, by primary key, through the relay executor; other rows and partitions never hold a wait.

How WaitPublished learns the outcome:

  1. Every relay (in any process) announces rows reaching a terminal status on the outbox_messages_done channel, from the statement that completes them, so a listener receives it exactly at commit. The payload is <schema>:<status>:<id>,<id>,..., at most 200 ids per notification.
  2. The LISTEN connection of this Outbox's Run (the one that also wakes the relay) feeds those notifications to the waiting calls: no query per wait beyond its first check, made after it registered so no notification is missed. Notifications for another schema are ignored.
  3. One shared re-read of all waiting ids every WithWaitRecheck interval (default 1s, at most one query per interval however many calls wait) settles what notifications did not: a listener reconnecting, a process that does not Run this Outbox or has no LISTEN pool, or WithoutNotify.

So wait on the Outbox whose Run is active in the same process for commit-time latency; elsewhere waits still finish, at the recheck pace. A relay running with WithoutNotify announces nothing, so waits on its rows also fall back to the recheck.

Upgrade from v1.0

Apply 0002_delivery_control.sql before using the new relay. It adds claim tokens, terminal timestamps, and the generic discarded state. Existing data is preserved; historical dead rows begin their retention window at migration time because their original terminal time is unknown. New retention settings remain disabled unless configured.

Stop/drain old relays before starting new ones. Old binaries do not fence their completion writes; do not mix old and new relays against the same queue. The Publisher, Hooks, and Executor interfaces remain source-compatible.

Package layout

The root package outbox is the core public surface (facade + re-exports). The implementation lives in subpackages: message, port (Publisher/Codec/Hooks/ Instrumentation/Executor), config, backoff, migrations, engine (store/relay/cleaner/waiters). Concrete publishers live in nested contrib/publishers/* modules and integrations with other libraries in contrib/libs/* (xtrace), so Kafka, NATS, Valkey and OpenTelemetry dependencies never enter the core module's go.mod.

Semantics

  • At-least-once. A failed batch is retried in full; consumers must deduplicate (e.g. on Message.ID).
  • Idempotent enqueue. Message.ID is the dedupe key while its row exists: enqueuing an existing ID (in the same or another transaction, or twice in one batch) is a no-op that keeps the first row and does not abort the caller's transaction. Derive the ID from the change to dedupe repeated triggers. Once the cleaner deletes a finished row, the same ID can be enqueued again, so receivers still deduplicate.
  • Scaling. Run one relay per instance with a unique WithInstanceID. Claims use FOR UPDATE SKIP LOCKED + a lease (locked_until); a crashed instance's rows are re-claimed once its lease expires.
  • Retries / dead-letter. On failure attempts increments and the row is rescheduled with backoff; after WithMaxAttempts it becomes dead and stays in the table for inspection.
  • Ordering. Best-effort by created_at. WithOrdered(true) enforces strict per-PartitionKey ordering (transaction time, then insertion order, so rows written in one transaction keep their order); a dead row blocks its key until resolved or removed. Published/discarded rows do not block it.
  • Latency. A trigger fires NOTIFY; the relay wakes near-instantly and falls back to polling every WithPollInterval. Once awake it keeps claiming until a pass finds nothing due, so a backlog (including a long run of one key in ordered mode) drains without waiting for the next poll.
  • Scheduling. None: an enqueued row is due immediately; NextAttemptAt is set only by retry backoff.

Notes

  • attempts increments at claim time, so a crash mid-publish counts as an attempt (poison-message protection).
  • The table name is fixed (outbox_messages); only the schema is configurable.
  • Codecs (JSON/proto) are out of scope — bring your own.

Tests

make test                                 # root + contrib module tests, no DB
cd test/integration && go test ./...      # black-box integration (testcontainers Postgres, Docker required)

Contrib publishers and integration tests live in separate nested modules so their dependency trees never enter the core module's go.mod. The core library stays a single pgx dependency.

Documentation

Overview

Package outbox implements a transactional outbox for PostgreSQL.

Messages are written to the outbox table inside the caller's business transaction; a background relay later claims them with FOR UPDATE SKIP LOCKED (leased to an externally supplied instance id) and hands them to a transport-agnostic Publisher. Delivery is at-least-once: consumers must deduplicate.

This file is the package's single public surface: it re-exports the types and constructors from the implementation subpackages (message, port, config, backoff, engine, migrations) and provides the Outbox facade. The database surface is the Executor interface (Exec, Query, QueryRow); plug your own. The logger is the standard library's *slog.Logger, injected via WithLogger.

Index

Constants

View Source
const (
	StatusPending    = message.StatusPending
	StatusProcessing = message.StatusProcessing
	StatusPublished  = message.StatusPublished
	StatusDead       = message.StatusDead
	StatusDiscarded  = message.StatusDiscarded
)

Status values.

View Source
const TableName = message.TableName

TableName is the fixed outbox table name (only the schema is configurable).

Variables

View Source
var (
	WithConfig                 = config.WithConfig
	WithInstanceID             = config.WithInstanceID
	WithSchema                 = config.WithSchema
	WithPollInterval           = config.WithPollInterval
	WithBatchSize              = config.WithBatchSize
	WithLeaseDuration          = config.WithLeaseDuration
	WithConcurrency            = config.WithConcurrency
	WithMaxAttempts            = config.WithMaxAttempts
	WithRetryBackoff           = config.WithRetryBackoff
	WithRetention              = config.WithRetention
	WithDeadRetention          = config.WithDeadRetention
	WithDiscardRetention       = config.WithDiscardRetention
	WithClearPayloadOnComplete = config.WithClearPayloadOnComplete
	WithPublishTimeout         = config.WithPublishTimeout
	WithCleanupInterval        = config.WithCleanupInterval
	WithOrdered                = config.WithOrdered
	WithoutNotify              = config.WithoutNotify
	WithWaitRecheck            = config.WithWaitRecheck
	WithSettleLinger           = config.WithSettleLinger
	WithListenPool             = config.WithListenPool
	WithFatal                  = config.WithFatal
	WithHooks                  = config.WithHooks
	WithInstrumentation        = config.WithInstrumentation
	WithCodec                  = config.WithCodec
	WithLogger                 = config.WithLogger
)
View Source
var (
	Permanent    = port.Permanent
	RetryAfter   = port.RetryAfter
	Discard      = port.Discard
	Abort        = port.Abort
	PerMessage   = port.PerMessage
	ErrLeaseLost = port.ErrLeaseLost
)

Publisher outcome controls apply to the whole batch, or per message through PerMessage, and preserve wrapped errors.

View Source
var (
	// ErrNotSettling: Settle outside a relay Publish call or after it returned.
	ErrNotSettling = port.ErrNotSettling
	// ErrUnknownMessage: Settle for a message not in the batch.
	ErrUnknownMessage = port.ErrUnknownMessage
	// ErrAlreadySettled: a second Settle for the same message.
	ErrAlreadySettled = port.ErrAlreadySettled
)
View Source
var (
	// ErrEmptyTopic is returned when a message has no topic.
	ErrEmptyTopic = message.ErrEmptyTopic
	// ErrNilPayload is returned when a message payload is nil.
	ErrNilPayload = message.ErrNilPayload
	// ErrInvalidSchema is returned by New when the schema is not a valid identifier.
	ErrInvalidSchema = config.ErrInvalidSchema
	// ErrNoCodec is returned by EnqueueValue when no codec was configured.
	ErrNoCodec = errors.New("outbox: no codec configured (use WithCodec)")
	// ErrNoExecutor is returned by New when no relay Executor was supplied.
	ErrNoExecutor = errors.New("outbox: a relay executor is required")
	// ErrInvalidID is returned by Published / WaitPublished for a non-uuid id.
	ErrInvalidID = engine.ErrInvalidID
	// ErrUndelivered matches an *UndeliveredError through errors.Is.
	ErrUndelivered = engine.ErrUndelivered
	// ErrOutcomeUnknown matches an *UnknownOutcomeError through errors.Is.
	ErrOutcomeUnknown = engine.ErrOutcomeUnknown
)
View Source
var DefaultBackoff = backoff.Default

DefaultBackoff is exponential 100ms -> 30s with full jitter.

View Source
var ExpBackoff = backoff.Exp

ExpBackoff returns an exponential backoff (base doubled per attempt, capped).

View Source
var MessageContext = port.MessageContext

MessageContext returns the context the Instrumentation opened for m's delivery, or ctx when there is none. Publishers that send messages one by one publish m with it.

View Source
var Settle = port.Settle

Settle reports one message's outcome from inside Publish as soon as it is known, so the row stops holding its partition; see port.Settle.

Functions

func Migrations

func Migrations() []string

Migrations returns the contents of every forward-only *.sql file, ordered by filename, ready to pass to Exec for callers applying migrations without a tool.

func MigrationsFS

func MigrationsFS() embed.FS

MigrationsFS returns the embedded migration files for golang-migrate (iofs source), goose, atlas, etc. The SQL is schema-unqualified: set search_path on the migration connection to install into a non-default schema.

Types

type AbortError added in v1.1.1

type AbortError = port.AbortError

type Backoff

type Backoff = backoff.Backoff

Backoff returns the delay before the retry after a given attempt.

type BatchError added in v1.2.0

type BatchError = port.BatchError

type Codec

type Codec = port.Codec

Codec marshals a value to bytes for storage and back.

type Config

type Config = config.Config

Config is the declarative configuration (mapstructure/validate/default).

type DiscardError added in v1.1.0

type DiscardError = port.DiscardError

type DiscardHooks added in v1.1.0

type DiscardHooks = port.DiscardHooks

type Executor

type Executor = port.Executor

Executor is the pluggable pgx query surface (Exec, Query, QueryRow).

type Hooks

type Hooks = port.Hooks

Hooks observes relay activity.

type Instrumentation added in v1.4.0

type Instrumentation = port.Instrumentation

Instrumentation plugs telemetry into Enqueue and delivery.

type Message

type Message = message.Message

Message is a single outbox row.

type NoopHooks

type NoopHooks = port.NoopHooks

NoopHooks is the default no-op Hooks implementation.

type NoopInstrumentation added in v1.4.0

type NoopInstrumentation = port.NoopInstrumentation

type Option

type Option = config.Option

Option configures the outbox.

type Outbox

type Outbox struct {
	// contains filtered or unexported fields
}

Outbox is the public facade: enqueue messages (transactionally) and run the background relay + cleaner.

func New

func New(exec Executor, enq Executor, pub Publisher, opts ...Option) (*Outbox, error)

New builds an Outbox.

  • exec is the relay/cleaner Executor: it runs the background claim, mark and cleanup queries and must NOT be bound to a business transaction (pass a *pgxpool.Pool, or your own Executor implementation).
  • enq is the enqueue Executor: pass one that resolves the caller's transaction from the context (e.g. pgtx's tx.DB) so inserts join the business transaction.
  • pub is the transport publisher.

LISTEN/NOTIFY wake-ups require a *pgxpool.Pool: supply one with WithListenPool, or pass a *pgxpool.Pool as exec and it is detected automatically. Without one the relay falls back to polling.

func (*Outbox) Enqueue

func (o *Outbox) Enqueue(ctx context.Context, m Message) error

Enqueue writes one message inside the caller's current transaction.

func (*Outbox) EnqueueBatch

func (o *Outbox) EnqueueBatch(ctx context.Context, ms []Message) error

EnqueueBatch writes multiple messages inside the caller's current transaction.

func (*Outbox) EnqueueValue

func (o *Outbox) EnqueueValue(ctx context.Context, topic, partitionKey, msgType string, v any) error

EnqueueValue marshals v with the configured codec and enqueues it. Requires WithCodec.

func (*Outbox) Published added in v1.3.0

func (o *Outbox) Published(ctx context.Context, ids ...string) (bool, error)

Published reports whether every listed message is delivered. A row still pending or being published makes it false. A row that ended dead or discarded returns an error matching ErrUndelivered (an *UndeliveredError naming the rows): it will not be delivered unless requeued. ids are the Message.IDs given to Enqueue; a non-uuid id returns ErrInvalidID. It reads only those rows, by primary key, through the relay executor (never a caller transaction), so other rows and partitions do not affect it.

A row that is gone is delivered only while the cleaner removes no refused rows (no WithDeadRetention / WithDiscardRetention): then only a published row can have been removed. With either retention on, a gone row may have been dead or discarded too, and the answer is an error matching ErrOutcomeUnknown (an *UnknownOutcomeError naming the rows). Configure the Outbox that asks with the same retention as the one that cleans. An id never enqueued reads as gone as well.

func (*Outbox) Requeue added in v1.1.0

func (o *Outbox) Requeue(ctx context.Context, id string) (bool, error)

Requeue atomically returns a dead row to pending and resets its attempts. It uses the enqueue executor, so authorization and application checks can be performed in the same caller transaction. False means no dead row matched. IDs and creation time are preserved; the next claim gets a fresh lease token.

func (*Outbox) Run

func (o *Outbox) Run(ctx context.Context) error

Run starts the relay and cleaner and blocks until ctx is cancelled, then returns ctx.Err(). A fatal error (an Abort outcome, a recovered panic, or an error the WithFatal classifier accepts) stops both and is returned instead; the batch it hit stays claimed and is re-claimed after its lease expires. The lease owner id was resolved at New time (WithInstanceID, or auto-generated).

func (*Outbox) WaitPublished added in v1.3.0

func (o *Outbox) WaitPublished(ctx context.Context, ids ...string) error

WaitPublished blocks until every listed message is delivered (as Published reports it), returning nil; or until one ends dead or discarded, returning an error matching ErrUndelivered; or until ctx ends, returning an error that wraps ctx.Err() (and the last failed state read, if any).

It learns outcomes from the relays' commit notifications (DoneChannel), whichever process published the rows, through the LISTEN connection of this Outbox's Run: no query per wait beyond the first check. Without that listener (Run not running here, no LISTEN pool, WithoutNotify) or for a missed notification, one shared re-read for all waiting calls every WithWaitRecheck interval settles it.

type Outcome added in v1.4.0

type Outcome = port.Outcome

type PanicError added in v1.1.1

type PanicError = port.PanicError

type PermanentError added in v1.1.0

type PermanentError = port.PermanentError

type Publisher

type Publisher = port.Publisher

Publisher delivers a batch of claimed messages to a transport.

type RetryAfterError added in v1.1.0

type RetryAfterError = port.RetryAfterError

type Settings

type Settings = config.Settings

Settings is the resolved runtime configuration.

type Status

type Status = message.Status

Status is the lifecycle state of an outbox message.

type UndeliveredError added in v1.3.0

type UndeliveredError = engine.UndeliveredError

UndeliveredError names the rows Published / WaitPublished found dead or discarded; they will not be delivered unless requeued.

type UnknownOutcomeError added in v1.5.1

type UnknownOutcomeError = engine.UnknownOutcomeError

UnknownOutcomeError names the rows Published / WaitPublished found gone while dead or discarded rows are collected too, so their outcome is unknown.

Directories

Path Synopsis
Package backoff provides retry backoff strategies for the outbox relay.
Package backoff provides retry backoff strategies for the outbox relay.
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment.
Package config holds the outbox runtime settings, the functional options that build them, and the declarative Config struct (mapstructure/validate/default tags) for loading from files or environment.
contrib
libs/xtrace module
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers.
Package engine implements the outbox background machinery: the store (enqueue/claim/mark/cleanup queries) and the relay and cleaner workers.
Package message defines the outbox data model.
Package message defines the outbox data model.
Package migrations exposes the embedded outbox SQL migrations.
Package migrations exposes the embedded outbox SQL migrations.
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor.
Package port defines the pluggable integration interfaces of the outbox: the transport Publisher, the value Codec, observability Hooks, and the database Executor.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL