consumer

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 18, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

Package consumer runs Consumers: named, Cursor-tracked readers of the event log.

A Consumer reads the events matching its event-type filter in Cursor order and hands each one to its Handler, usually a Router built from typed handlers:

r := consumer.NewRouter(codec)
consumer.On(r, func(ctx context.Context, env eventstore.Envelope, e TaskOpened) error {
	return q.InsertTask(ctx, postgres.Conn(ctx, pool), env.StreamID, e.Title)
})
tasks, err := consumer.New(rt, consumer.Config{
	Name:    "task_list",
	Mode:    consumer.Transactional,
	Handler: r,
})

In Transactional mode the handler's writes and the Cursor commit in one Cursor transaction, so every event takes effect exactly once. In OwnTransaction mode the Cursor advances after each event, so delivery is at least once.

A failing event is retried with backoff and then the Consumer goes Stopped; it never skips an event. Permanent errors, panics and decode errors stop it at once. A Stopped Consumer stays at its Cursor until Runtime.Resume.

The filter is stored on the Cursor and checked when a release changes it (ADR-0006). Removed types are always accepted. An OwnTransaction Consumer handles added types from its Cursor onwards. A Transactional Consumer accepts added types only when no event of them lies at or below its Cursor; otherwise it goes Stopped, and the change needs a new Projection version. Resume does not skip this check.

The Runtime keeps the Cursors. NewPgRuntime runs on an eventstore.PgStore with Cursors in the consumer_cursors table; handlers write through postgres.Conn, which joins the Cursor transaction. Each Consumer runs under a lease, so only one process at a time processes it, and wakes up on NOTIFY when a Listener is given. NewInMemoryRuntime runs on an eventstore.InMemoryStore; its handlers register writes with postgres.Stage so they roll back with the Cursor transaction.

Consumers only see events below pg_snapshot_xmin (or the in-memory equivalent), so an event of a transaction that commits late is never skipped. Any open transaction that holds a transaction ID, in any database of the cluster, delays visibility until it ends.

Run all Consumers of a process with a Group:

err := consumer.Group{tasks, mailer}.Run(ctx)

Tests and one-shot jobs call Drain, which processes synchronously up to the head of the log at the time of the call.

Observability

Each batch runs in a CONSUMER span "process {consumer}". It is a new root, linked to the send spans that appended up to 128 of its events. WithEventSpans adds an INTERNAL span "handle {event type}" per event. The histograms eventsourcing.consumer.batch.duration and eventsourcing.consumer.process.duration carry only the Consumer's name. While Run holds the lease, the gauges eventsourcing.consumer.pending, eventsourcing.consumer.lag and eventsourcing.consumer.stopped are polled from Status at each metric collection, so only the lease owner reports them; Drain does not. A PgRuntime reports eventsourcing.xmin.age.

Index

Constants

View Source
const PendingCap = 10_000

PendingCap caps Status.Pending: a Consumer with more pending events reports PendingCap+1.

Variables

View Source
var (
	// ErrInvalidConfig reports a Config or Option that New rejects.
	ErrInvalidConfig = errors.New("consumer: invalid config")
	// ErrDuplicate reports a second Consumer with the same name on one
	// Runtime.
	ErrDuplicate = errors.New("consumer: duplicate name")
	// ErrRunning reports Run on a Consumer that already runs, or Drain while
	// the Consumer runs or drains.
	ErrRunning = errors.New("consumer: already running")
	// ErrStopped reports a Consumer that is Stopped. It stays Stopped at its
	// Cursor until Runtime.Resume clears it.
	ErrStopped = errors.New("consumer: stopped")
	// ErrNotFound reports a Consumer name the Runtime does not know.
	ErrNotFound = errors.New("consumer: not found")
)

Functions

func On

func On[E aggregate.Event](r *Router, fn func(ctx context.Context, env eventstore.Envelope, e E) error)

On routes events of type E to fn, which receives the Envelope and the decoded, upcast event. It panics on a nil fn, when E is not registered on the Router's codec, or when E is already routed. It is meant to be called at startup.

func Permanent

func Permanent(err error) error

Permanent marks err as hopeless: the Consumer goes Stopped at once instead of retrying. Permanent(nil) returns nil.

Types

type Config

type Config struct {
	// Name identifies the Consumer and its Cursor; unique per Runtime.
	Name string
	// EventTypes is the filter. When empty, a Handler with an EventTypes
	// method (a Router) provides it; otherwise every event is delivered.
	EventTypes []string
	Mode       Mode
	Handler    Handler
}

Config configures a Consumer.

type Consumer

type Consumer struct {
	// contains filtered or unexported fields
}

Consumer is a named, Cursor-tracked reader of the event log. It reads the events matching its filter in Cursor order and hands each to its Handler. A failing event is retried and then stops the Consumer; it is never skipped.

func New

func New(rt Runtime, cfg Config, opts ...Option) (*Consumer, error)

New creates a Consumer on rt. It fails with ErrInvalidConfig for an empty Name, a nil Handler, an unknown Mode or invalid options, and with ErrDuplicate when rt already has a Consumer with that Name.

func (*Consumer) Await

func (c *Consumer) Await(ctx context.Context, pos eventstore.Cursor) Freshness

Await waits until the Consumer has caught up to pos, typically a command's CommandResult.Position, so an API can serve read-your-writes or mark its response stale. The budget is the ctx deadline, else 2s.

The Consumer has caught up when its Cursor is at or after pos, or when no matching event lies between its Cursor and pos and every transaction up to pos has ended. So a filtered Consumer whose Cursor sits at its last matching event is Fresh. A zero pos is Fresh at once. Await returns StaleStopped as soon as it finds the Consumer Stopped short of pos, and Stale when the budget runs out. All Awaits on a Consumer share one poller.

func (*Consumer) Drain

func (c *Consumer) Drain(ctx context.Context) error

Drain processes events synchronously up to the head of the log at the time of the call. It fails with ErrRunning while the Consumer runs or drains, and with a wrapped ErrStopped when the Consumer is or goes Stopped.

func (*Consumer) Name

func (c *Consumer) Name() string

Name returns the Consumer's name.

func (*Consumer) Run

func (c *Consumer) Run(ctx context.Context) error

Run processes events until ctx ends, then returns nil. A Stopped Consumer keeps running and waits for Runtime.Resume. Run fails with ErrRunning when the Consumer already runs.

func (*Consumer) Status

func (c *Consumer) Status(ctx context.Context) (Status, error)

Status returns the Consumer's Cursor, its pending events and its Stopped state.

type Freshness

type Freshness int

Freshness tells whether a Consumer has caught up to a Cursor.

const (
	// Fresh means the Consumer has processed every matching event up to the
	// Cursor.
	Fresh Freshness = iota + 1
	// Stale means the Consumer did not catch up within the budget.
	Stale
	// StaleStopped means the Consumer has not caught up and is Stopped, so
	// it will not catch up without Resume.
	StaleStopped
)

func (Freshness) String

func (f Freshness) String() string

String returns the name of f.

type Group

type Group []*Consumer

Group runs Consumers together.

func (Group) Run

func (g Group) Run(ctx context.Context) error

Run runs each Consumer in its own goroutine until ctx ends, then returns nil once all have returned. When one fails, the others are cancelled and its error is returned.

type Handler

type Handler interface {
	Handle(ctx context.Context, env eventstore.Envelope) error
}

Handler handles one event. It runs with the Envelope's Tenant and Correlation in ctx. A returned error is retried, unless it is Permanent.

type HandlerFunc

type HandlerFunc func(ctx context.Context, env eventstore.Envelope) error

HandlerFunc adapts a function to a Handler.

func (HandlerFunc) Handle

func (f HandlerFunc) Handle(ctx context.Context, env eventstore.Envelope) error

Handle calls f.

type InMemoryRuntime

type InMemoryRuntime struct {
	// contains filtered or unexported fields
}

InMemoryRuntime runs Consumers on an eventstore.InMemoryStore, for prototypes and tests. Cursors live in memory. A Cursor transaction is a postgres.InMemoryUnitOfWork transaction, so handler writes registered with postgres.Stage commit or roll back together with the Cursor.

func NewInMemoryRuntime

func NewInMemoryRuntime(cfg InMemoryRuntimeConfig) *InMemoryRuntime

NewInMemoryRuntime creates an InMemoryRuntime over cfg.Store. It panics on a nil Store.

func (*InMemoryRuntime) Resume

func (rt *InMemoryRuntime) Resume(_ context.Context, name string) error

Resume clears the Stopped state of the named Consumer.

type InMemoryRuntimeConfig

type InMemoryRuntimeConfig struct {
	// Store is the event log the Consumers read.
	Store *eventstore.InMemoryStore
}

InMemoryRuntimeConfig configures an InMemoryRuntime.

type Mode

type Mode int

Mode selects the delivery guarantee of a Consumer.

const (
	// Transactional runs the handlers of a batch and the Cursor update in one
	// Cursor transaction, so every event takes effect exactly once. The
	// handler writes through the transaction in ctx (postgres.Conn, or
	// postgres.Stage in memory).
	Transactional Mode = iota + 1
	// OwnTransaction runs each handler on its own and advances the Cursor
	// after each event, so delivery is at least once.
	OwnTransaction
)

type Option

type Option func(*options)

Option configures a Consumer.

func WithBatchSize

func WithBatchSize(n int) Option

WithBatchSize sets how many events one batch reads. The default is 100.

func WithEventSpans

func WithEventSpans(on bool) Option

WithEventSpans turns the per-event "handle {event type}" spans on or off. They are off by default.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger. The default is slog.Default().

func WithMeterProvider

func WithMeterProvider(mp metric.MeterProvider) Option

WithMeterProvider sets the meter provider of the histograms and gauges. The default is the OTel global.

func WithPollInterval

func WithPollInterval(d time.Duration) Option

WithPollInterval sets how long a running Consumer waits for new events before it polls again. The wait is jittered down by up to a fifth. The default is 1s.

func WithRetry

func WithRetry(r Retry) Option

WithRetry sets the retry schedule. The default is 5 attempts with delays from 1s up to 30s.

func WithTracerProvider

func WithTracerProvider(tp trace.TracerProvider) Option

WithTracerProvider sets the tracer provider. The default is the OTel global.

type PgRuntime

type PgRuntime struct {
	// contains filtered or unexported fields
}

PgRuntime runs Consumers on an eventstore.PgStore. Cursors live in the consumer_cursors table of docs/postgres/schema.sql. A Consumer reads only the events below pg_snapshot_xmin, so it never skips an event of a transaction that commits later with a smaller Cursor. A filtered Consumer reads one index range per event type.

Each Consumer runs under a lease from Claims, also while Stopped. A Cursor transaction is a Postgres transaction in ctx: handler writes made through postgres.Conn commit or roll back together with the Cursor.

func NewPgRuntime

func NewPgRuntime(cfg PgRuntimeConfig) (*PgRuntime, error)

NewPgRuntime creates a PgRuntime. It fails with ErrInvalidConfig when Pool, Store or Claims is missing.

func (*PgRuntime) Close

func (rt *PgRuntime) Close()

Close stops reporting eventsourcing.xmin.age for this Runtime; the gauge keeps reporting while another PgRuntime of the process with the same MeterProvider is open. Call it once the Runtime's Consumers have stopped. Close is idempotent.

func (*PgRuntime) Resume

func (rt *PgRuntime) Resume(ctx context.Context, name string) error

Resume clears the Stopped state of the named Consumer. It also works for a Consumer this Runtime does not run, as long as its Cursor exists.

type PgRuntimeConfig

type PgRuntimeConfig struct {
	// Pool runs the reads and Cursor transactions. Required.
	Pool *pgxpool.Pool
	// Store is the event log the Consumers read; its Schema qualifies the
	// consumer tables, the wake-up channel <schema>.events and the lease
	// namespace <schema>.consumer. Required.
	Store *eventstore.PgStore
	// Listener wakes Consumers on new events. Optional: without it the
	// Consumers only poll.
	Listener *notify.Listener
	// Claims grants each Consumer its lease. Required, and Claims.Run must
	// be running.
	Claims *lease.Claims
	// MeterProvider records the eventsourcing.xmin.age gauge, one series
	// per MeterProvider until Close. Optional: the default is the OTel
	// global.
	MeterProvider metric.MeterProvider
	// SetLocalTenant makes a Cursor transaction set app.tenant_id to the
	// Tenant of each event before its handler runs, like
	// postgres.SetLocalTenant does for a UnitOfWork, so row-level security
	// on read models sees the Tenant. Off by default.
	SetLocalTenant bool
}

PgRuntimeConfig configures a PgRuntime.

type Retry

type Retry struct {
	MaxAttempts int
	Base        time.Duration
	Max         time.Duration
}

Retry is the retry schedule of a failing event: up to MaxAttempts attempts in total, with jittered exponential delays from Base up to Max.

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router is a Handler built from typed handlers registered with On. Its EventTypes are the Consumer's filter unless Config.EventTypes is set. Events of unrouted types are ignored.

func NewRouter

func NewRouter(codec *eventstore.Codec) *Router

NewRouter returns an empty Router for events registered on codec. It panics on a nil codec.

func (*Router) EventTypes

func (r *Router) EventTypes() []string

EventTypes returns the routed Event types, sorted.

func (*Router) Handle

func (r *Router) Handle(ctx context.Context, env eventstore.Envelope) error

Handle calls the route for env's Event type and returns nil for an unrouted type.

type Runtime

type Runtime interface {
	// Resume clears the Stopped state of the named Consumer, which then
	// continues from its Cursor. It fails with ErrNotFound for an unknown
	// name and is a no-op when the Consumer is not Stopped.
	Resume(ctx context.Context, name string) error
	// contains filtered or unexported methods
}

Runtime runs Consumers on one event log and keeps their Cursors. It is implemented by PgRuntime and InMemoryRuntime.

type Status

type Status struct {
	// Cursor is the last processed event.
	Cursor eventstore.Cursor
	// Pending counts the visible matching events after Cursor, up to
	// PendingCap+1.
	Pending int
	// Age is how long the oldest pending event has been in the log; zero
	// when nothing is pending.
	Age time.Duration
	// Stopped is set while the Consumer is Stopped.
	Stopped *Stop
}

Status is a Consumer's progress.

type Stop

type Stop struct {
	At    time.Time
	Cause string
}

Stop describes why and when a Consumer went Stopped.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL