outbox

package
v0.64.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 10, 2026 License: MIT Imports: 29 Imported by: 0

Documentation

Overview

Package outbox provides a transactional outbox pattern for reliable event publishing.

The transactional outbox solves the dual-write problem in microservices: events are written to an outbox table in the SAME database transaction as business data, then reliably delivered to the message broker by a background relay.

This guarantees at-least-once delivery: events are never lost even if the broker is temporarily unavailable. Consumers MUST be idempotent — dedup on the outbox.HeaderEventID ("x-outbox-event-id") header via messaging.Metadata.DedupKey and inbox.ProcessOnce.

Usage:

func (m *Module) Init(deps *app.ModuleDeps) error {
    m.outbox = deps.Outbox
    return nil
}

func (s *Service) CreateOrder(ctx context.Context, order Order) error {
    tx, err := db.Begin(ctx)
    if err != nil { return err }
    defer tx.Rollback(ctx)

    tx.Exec(ctx, "INSERT INTO orders ...", args...)
    s.outbox.Publish(ctx, tx, &app.OutboxEvent{
        EventType:   "order.created",
        AggregateID: "order-123",
        Payload:     payload,
        Exchange:    "order.events",
    })
    return tx.Commit(ctx)
}

Index

Constants

View Source
const (
	// HeaderEventID carries the unique outbox event id (a UUID) used for
	// consumer-side deduplication. The literal lives in messaging so
	// Metadata.DedupKey can read it; this is the same constant.
	HeaderEventID = messaging.HeaderEventID

	// HeaderEventType carries the event type of the published outbox record.
	HeaderEventType = "x-outbox-event-type"
)

AMQP delivery header names stamped by the relay for consumer idempotency. The relay references these when publishing, so consumers can dedupe without re-declaring the literals.

View Source
const (
	LaneAMQP   = "amqp"
	LaneStream = "stream"
)

Lane constants name the transport a row is dispatched on.

View Source
const (
	StatusPending   = "pending"
	StatusPublished = "published"
	StatusFailed    = "failed"
)

Event status constants.

View Source
const DefaultTableName = "gobricks_outbox"

DefaultTableName is the default outbox table name.

Variables

View Source
var (
	// ErrStreamTargetRequiresTenant is returned when an event targets a super stream
	// but the context carries no tenant to take the partition key from.
	ErrStreamTargetRequiresTenant = errors.New("outbox: a stream target takes its partition key from the context tenant, and the context carries none")

	// ErrConflictingTargets is returned when an event names both a stream and an
	// exchange or routing key.
	ErrConflictingTargets = errors.New("outbox: an event targets either an exchange or a stream; a stream target takes no exchange or routing key")

	// ErrStreamNotAnOutboxTarget is returned when an event names a stream the relay
	// was not configured to publish to.
	ErrStreamNotAnOutboxTarget = errors.New("outbox: stream is not listed in outbox.superstreams")

	// ErrReservedHeaderPrefix is returned when a caller's event headers claim the
	// x-gobricks- prefix, the framework's own namespace inside a persisted row's
	// headers. The framework is that namespace's only writer — it stamps the
	// payload's encoding there and the relay reads it back off the row — so a
	// caller header under it is refused rather than silently dropped: a drop hides
	// both the mistake and the attempt, and the caller never learns its header
	// went nowhere. Rename the header out of the prefix.
	ErrReservedHeaderPrefix = errors.New("outbox: the x-gobricks- header prefix is reserved for the framework's own stamps")

	// ErrNotLeader is returned by Store.Lead when another relay instance holds the
	// ledger's leader row.
	ErrNotLeader = errors.New("outbox: another relay instance leads this ledger")
)
View Source
var ErrSealedPayloadNeedsBytes = errors.New("outbox: payload type carries seal tags; seal it first with Publisher[T].Seal and publish the returned bytes")

ErrSealedPayloadNeedsBytes: a struct (or pointer) payload whose type carries `seal` tags reached Publish as plaintext. The outbox persists what it is given, so the sealed form must be produced first — Publisher[T].Seal — and handed over as []byte. Only the struct door is guarded; a hand-marshaled plaintext []byte is the documented residual.

Functions

func EventIDFromHeaders added in v0.40.0

func EventIDFromHeaders(h amqp.Table) (string, bool)

EventIDFromHeaders extracts the outbox event id from AMQP delivery headers, returning ok=false when the header is absent, empty, or not a string/[]byte. AMQP header values can arrive as either string or []byte depending on the broker and client, so both are normalized. The value is extracted, not validated: inbox.ProcessOnce refuses an id outside the ledger grammar at the ledger door, and messaging.Metadata.DedupKey validates as it extracts.

Types

type Cleanup

type Cleanup struct {
	// contains filtered or unexported fields
}

Cleanup is a scheduler.Executor that removes published events older than the configured retention period.

Runs daily at 04:00 by default (registered via scheduler.DailyAt). Like the relay, it resolves the database through the tenant-aware getDB resolver and fans out across the configured tenants in multi-tenant mode (shared with the inbox cleanup via multitenant.FanOutRetentionCleanup).

func (*Cleanup) Execute

func (c *Cleanup) Execute(jobCtx scheduler.JobContext) error

type Leadership added in v0.61.0

type Leadership interface {
	// Probe fails once the claim is gone (statement timeout, recycled connection,
	// partition). The caller must stop draining on the first failed probe.
	Probe(ctx context.Context) error

	// Release gives up the claim. It is safe to defer.
	Release(ctx context.Context) error
}

Leadership is a held claim on a ledger's leader row. Probe reports whether the claim still stands; Release gives it up.

type Module

type Module struct {
	// contains filtered or unexported fields
}

Module implements the GoBricks Module interface for transactional outbox. It provides reliable event publishing by writing events to a database table within the caller's transaction, then publishing them to the message broker via a background relay job.

The module is registered like any other GoBricks module:

for _, m := range []app.Module{
    scheduler.NewModule(), // Required: relay runs as a scheduled job
    outbox.NewModule(),    // Outbox module
    &myapp.OrderModule{},
} {
    if err := fw.RegisterModule(m); err != nil {
        log.Fatal(err)
    }
}

func NewModule

func NewModule() *Module

NewModule creates a new Module instance.

func (*Module) DeclareStreams added in v0.61.0

func (m *Module) DeclareStreams(decls *streams.Declarations)

DeclareStreams registers one publisher per configured super stream, satisfying streams.StreamDeclarer. It is a no-op for a disabled outbox or one with no targets, so a deployment that never mentions superstreams declares nothing.

func (*Module) Init

func (m *Module) Init(deps *app.ModuleDeps) error

Init implements app.Module. Stores dependencies and initializes the publisher; the vendor-specific store is created lazily on first use (see ensureStoreInitialized).

func (*Module) Name

func (m *Module) Name() string

Name implements app.Module.

func (*Module) OutboxPublisher

func (m *Module) OutboxPublisher() app.OutboxPublisher

OutboxPublisher implements app.OutboxProvider — returns the Publisher for ModuleDeps wiring.

func (*Module) RegisterJobs

func (m *Module) RegisterJobs(registrar app.JobRegistrar) error

RegisterJobs implements app.JobProvider. Registers the relay and cleanup jobs with the scheduler.

func (*Module) SetSharedResolvers added in v0.54.0

func (m *Module) SetSharedResolvers(
	db func(context.Context) (dbtypes.Interface, error),
	msg func(context.Context) (messaging.AMQPClient, error),
)

SetSharedResolvers injects the control-plane ("" key) resolvers. Called by app.RegisterModule; used only when outbox.tenancy=shared.

func (*Module) Shutdown

func (m *Module) Shutdown() error

Shutdown implements app.Module.

type Record

type Record struct {
	ID           string     // UUID, generated on insert
	EventType    string     // Event type for routing
	AggregateID  string     // Aggregate identifier for correlation
	Payload      []byte     // Event payload; JSON-encoded unless the caller supplied []byte, which is stored as-is
	Headers      []byte     // JSON-encoded AMQP headers (nullable)
	Exchange     string     // Target AMQP exchange
	RoutingKey   string     // AMQP routing key
	Lane         string     // LaneAMQP or LaneStream; the store fills an empty lane with LaneAMQP
	Stream       string     // Stream-lane target super stream (empty on the AMQP lane)
	PartitionKey string     // Stream-lane partition key: the row's tenant stamp
	Seq          int64      // Per-ledger sequence assigned by the database at insert; zero before insert, never written by Insert
	Status       string     // "pending", "published", or "failed"
	RetryCount   int        // Number of failed publish attempts so far (not incremented on eventual success)
	Error        string     // Last recorded failure message; NOT cleared on a later successful publish
	CreatedAt    time.Time  // When the event was created
	PublishedAt  *time.Time // When the event was successfully published (nil if pending)
}

Record represents a single row in the outbox table. Records are created by Publisher.Publish() and consumed by the relay job.

type Relay

type Relay struct {
	// contains filtered or unexported fields
}

Relay is a scheduler.Executor that polls for pending outbox events and hands them to the shipper for their lane.

The relay runs as a scheduled job (registered via scheduler.FixedRate), getting overlapping prevention, panic recovery, and OTel metrics for free.

Resources are resolved through the tenant-aware getDB resolver and each shipper's own (the module's deps.DB/deps.Messaging) rather than the scheduler JobContext, because the scheduler builds the JobContext from a tenant-less context — so in multi-tenant mode the relay must inject each tenant into the context itself before resolving that tenant's database and broker.

func (*Relay) Execute

func (r *Relay) Execute(jobCtx scheduler.JobContext) error

Execute runs one relay cycle per configured tenant. In single-tenant mode this is a single pass with no tenant in context; in static multi-tenant mode it fans out across the configured tenants, resolving each tenant's database and broker independently. Per-tenant failures are collected so one unhealthy tenant does not block the others.

type Store

type Store interface {
	// Insert writes an event row to the outbox table within the given transaction.
	Insert(ctx context.Context, tx dbtypes.Tx, record *Record) error

	// FetchPending retrieves up to batchSize pending events in ledger sequence order.
	// Selection is status-gated only: parking is driven by the "failed" status
	// (set by MarkDeadLettered), NOT by retry_count, so an outage-inflated count can
	// never freeze a healthy pending event.
	FetchPending(ctx context.Context, db dbtypes.Interface, batchSize int) ([]Record, error)

	// MarkPublished updates the event status to published with a timestamp.
	MarkPublished(ctx context.Context, db dbtypes.Interface, eventID string) error

	// MarkFailed increments retry count and records the error, leaving the event
	// "pending" so the relay retries it on a later cycle.
	MarkFailed(ctx context.Context, db dbtypes.Interface, eventID, errMsg string) error

	// MarkDeadLettered increments retry count, records the error, and sets the event
	// status to "failed" — a terminal state the relay stops retrying. Used ONLY for
	// poison events — the message-intrinsic classes enumerated at Relay.deadLetterPoison —
	// that exhaust MaxRetries. Connectivity failures (broker down, NACK, confirmation
	// timeout) must never call this — they advance retry_count via MarkFailed and keep
	// retrying indefinitely.
	MarkDeadLettered(ctx context.Context, db dbtypes.Interface, eventID, errMsg string) error

	// DeletePublished removes events that were published before the given time.
	// Returns the number of rows deleted.
	DeletePublished(ctx context.Context, db dbtypes.Interface, before time.Time) (int64, error)

	// Lead takes the ledger's leader row FOR UPDATE NOWAIT in a transaction it holds
	// until Release. ErrNotLeader when another instance holds it. Probe fails once the
	// transaction is gone (timeout, recycled connection, partition), and the caller
	// must stop draining on the first failed probe.
	Lead(ctx context.Context, db dbtypes.Interface) (Leadership, error)

	// CreateTable creates the outbox table, its indexes, and the companion leader
	// table with its single row, if they do not exist.
	// Used for auto-migration when outbox.autocreatetable is true.
	CreateTable(ctx context.Context, db dbtypes.Interface) error
}

Store abstracts outbox table operations for vendor-agnostic SQL. Implementations exist for PostgreSQL and Oracle with vendor-specific placeholder styles and DDL.

func NewOracleStore

func NewOracleStore(tableName string) (Store, error)

NewOracleStore creates a new Oracle outbox store. Returns an error if the table name contains invalid identifier characters.

func NewPostgresStore

func NewPostgresStore(tableName string) (Store, error)

NewPostgresStore creates a new PostgreSQL outbox store. Returns an error if the table name contains invalid identifier characters.

Directories

Path Synopsis
Package testing provides test utilities for the transactional outbox pattern.
Package testing provides test utilities for the transactional outbox pattern.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL