txoutbox

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2025 License: MIT Imports: 8 Imported by: 0

README

txoutbox

Golang utilities for the Transactional Outbox pattern. Write messages inside your business transaction, then drain them asynchronously with a relay that handles retries, leasing, and backoff.

Features

  • Store / Sender separation: queue messages with any SQL database, dispatch through pluggable senders ( Kafka/SQS/Webhook/etc.).
  • Relay with leasing: avoids duplicate deliveries via Claim + LeaseTTL, retries with configurable backoff and attempt limits.
  • DB-specific packages: root module exposes the interfaces, while stores/postgres_store / stores/mysql_store / stores/sqlite_store (and future backends) bring their own SQL.
  • Observability-ready hooks: leveled Logger interface, context propagation, and overridable clock (Options.Now) for deterministic tests, plus pluggable Hooks so you can emit metrics/traces for claims, retries, and failures.
  • Docker playground: compose.yaml runs PostgreSQL + LocalStack SQS so you can try the flow locally.

Quick Start

  1. Install

    go get github.com/mickamy/txoutbox
    
  2. Create the table – an example PostgreSQL schema exists under postgres/init.sql:

    CREATE TABLE txoutbox (
      id            BIGSERIAL PRIMARY KEY,
      topic         TEXT        NOT NULL,
      key           TEXT,
      payload       JSONB       NOT NULL,
      status        TEXT        NOT NULL DEFAULT 'pending',
      retry_count   INT         NOT NULL DEFAULT 0,
      next_retry_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
      claimed_by    TEXT,
      claimed_at    TIMESTAMPTZ,
      created_at    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
      sent_at       TIMESTAMPTZ
    );
    
  3. Run the richer example (example/ module)
    After docker compose up, use the commands below in separate terminals:

    cd example
    go run ./cmd/webhook                   # receives POSTs on :8081/events and logs them
    go run ./cmd/enqueue                   # seeds an order row and queues an outbox message
    go run ./cmd/relay                     # leases messages and POSTs them to the webhook
    SENDER=sqs \
      QUEUE_URL=http://localhost:4566/000000000000/worker-queue \
      go run ./cmd/relay                   # alternative: send to LocalStack SQS
    SENDER=sqs \
      QUEUE_URL=http://localhost:4566/000000000000/worker-queue \
      go run ./cmd/consumer                # optional: drain SQS messages and log them
    
    • POSTGRES_DSN (default postgres://postgres:password@localhost:5432/txoutbox?sslmode=disable) controls database access.
    • SENDER chooses the dispatcher (webhook default, sqs for LocalStack). Use WEBHOOK_URL or SQS_ENDPOINT/ QUEUE_URL to point at your infra.
    • example/cmd/relay also exposes expvar metrics at http://localhost:2112/debug/vars so you can inspect the new Hooks counters.
    • cmd/enqueue creates an orders table (if needed), writes a fake order, and calls stores.NewPostgresStore(...).Add inside a transaction.
    • cmd/relay runs the shared relay with either the HTTP sender or the SQS sender so you can observe leasing/retries interacting with downstream processes (cmd/webhook or LocalStack SQS).

License

MIT

Documentation

Overview

Package txoutbox provides primitives for the Transactional Outbox pattern.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Backoff

type Backoff func(attempt int) time.Duration

Backoff returns the wait duration before the given attempt.

func Exponential

func Exponential(base time.Duration, factor float64, max time.Duration) Backoff

Exponential creates a capped exponential backoff function.

type Envelope

type Envelope struct {
	// ID is the primary key of the outbox row.
	ID int64
	// Topic is copied from the original Message for routing/logging.
	Topic string
	// Key is optional metadata used by senders for partitioning/idempotency.
	Key *string
	// Payload is the raw JSON message stored in the outbox.
	Payload json.RawMessage
	// RetryCount tracks how many attempts have been made (before this lease).
	RetryCount int
	// CreatedAt records when the row was inserted.
	CreatedAt time.Time
}

Envelope represents a row leased by the relay for delivery.

func (Envelope) Decode

func (e Envelope) Decode(dest any) error

Decode unmarshals the payload into the provided destination.

type Executor

type Executor interface {
	ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error)
}

Executor is the minimal surface needed from *sql.Tx or *sql.DB.

type Hooks

type Hooks interface {
	// OnClaim fires after each Claim with the requested batch vs actual rows.
	OnClaim(ctx context.Context, batchSize int, claimed int)
	// OnSendSuccess fires for every Envelope delivered successfully.
	OnSendSuccess(ctx context.Context, env Envelope)
	// OnSendFailure fires when Sender returns an error before retry/fail handling.
	OnSendFailure(ctx context.Context, env Envelope, err error)
	// OnRetry fires when a message is rescheduled for another attempt.
	OnRetry(ctx context.Context, env Envelope, nextAttempt int, delay time.Duration)
	// OnFail fires when a message is permanently failed.
	OnFail(ctx context.Context, env Envelope, attempts int, err error)
	// OnStoreError fires when a Store call returns an error.
	OnStoreError(ctx context.Context, op string, id int64, err error)
	// OnCycle fires once per processOnce iteration with the elapsed duration.
	OnCycle(ctx context.Context, duration time.Duration)
}

Hooks lets callers observe relay activity for metrics/tracing/logs.

type Logger

type Logger interface {
	Info(ctx context.Context, format string, v ...any)
	Warn(ctx context.Context, format string, v ...any)
	Error(ctx context.Context, format string, v ...any)
}

Logger captures Relay logs; implementors can wrap slog/zap/etc.

type Message

type Message struct {
	// Topic identifies the logical event or routing destination (e.g. "order.created").
	Topic string
	// Key optionally provides a partition/idempotency key; leave empty if unused.
	Key string
	// Body is the user payload that will be marshaled to JSON.
	Body any
}

Message represents an application-level event queued inside a DB transaction.

func (Message) MarshalPayload

func (m Message) MarshalPayload() ([]byte, error)

MarshalPayload turns the body into JSON for storage.

type Options

type Options struct {
	// BatchSize controls how many records the relay claims per iteration.
	BatchSize int
	// LeaseTTL defines how long a claimed message stays owned before expiring.
	LeaseTTL time.Duration
	// MaxAttempts is the number of total send tries before marking as failed.
	MaxAttempts int
	// PollInterval is the sleep duration between claim cycles when no work exists.
	PollInterval time.Duration
	// Backoff computes the retry delay based on attempt count.
	Backoff Backoff
	// Logger emits structured logs for relay activity.
	Logger Logger
	// Hooks let callers plug metrics/tracing/etc. into relay events.
	Hooks Hooks
	// WorkerID identifies this relay instance in the database.
	WorkerID string
	// Now supplies the current time; override for tests or custom time sources.
	Now func() time.Time
}

Options configure Relay behaviour and tuning knobs for workers.

type Relay

type Relay struct {
	// contains filtered or unexported fields
}

Relay coordinates pulling messages from the store and sending them via a Sender.

func NewRelay

func NewRelay(store Store, sender Sender, opts Options) *Relay

NewRelay wires a Store and Sender with the provided options.

func (*Relay) Run

func (r *Relay) Run(ctx context.Context) error

Run processes messages until the context is cancelled.

type Sender

type Sender interface {
	Send(ctx context.Context, msg Envelope) error
}

Sender dispatches an outbox message to the actual transport.

type Store

type Store interface {
	// Add enqueues a message using the provided transaction/executor (typically *sql.Tx).
	Add(ctx context.Context, exec Executor, msg Message) error
	// Claim selects pending messages and leases them to a worker, returning envelopes to process.
	Claim(ctx context.Context, workerID string, limit int, leaseTTL time.Duration) ([]Envelope, error)
	// Send marks a message as successfully delivered.
	Send(ctx context.Context, id int64, sendAt time.Time) error
	// Retry releases the message to be retried later after incrementing the attempt counter.
	Retry(ctx context.Context, id int64, retryCount int, nextRetry time.Time) error
	// Fail flags the message as permanently failed so operators can inspect the row.
	Fail(ctx context.Context, id int64, retryCount int) error
}

Store encapsulates DB operations used by the relay.

Directories

Path Synopsis
example module
internal
test

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL