sqlrecorder

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package sqlrecorder is a recorder.Store on Postgres, MySQL or SQLite, so a run history the app owns lives in its own database, next to any driver (a Hatchet engine included), and survives the driver's own retention.

It is built on database/sql alone and imports no database driver: the app opens the *sql.DB with the driver it already uses and names the engine:

db, _ := sql.Open("pgx", dsn)                  // the app's own driver
store, err := sqlrecorder.New(db, sqlrecorder.Postgres,
	sqlrecorder.WithRetention(recorder.Retention{
		RetentionRule: recorder.RetentionRule{Steps: 7 * 24 * time.Hour, Runs: 30 * 24 * time.Hour},
	}))
err = store.Migrate(ctx)                        // idempotent, run on every start
rec, err := recorder.New(store)
c, err := workflow.Use(client, rec.Plugin())

Tables (prefix "wf_rec_" by default, WithTablePrefix): runs (one row per run), steps (one row per step attempt), events (one row per step event), labels (one row per run label, for indexed label filters) and schema (the applied SchemaVersion). Steps are rows of their own rather than an array on the run row, so a busy run never rewrites one large row per event.

Every write is one transaction that locks the run's row first and is retried on contention; reads answer workflow.RunQuerier and the recorder's richer Records and Run. recordertest.RunStore holds it to the Store contract on all three engines.

Index

Constants

View Source
const (
	// DefaultTablePrefix starts every table name: wf_rec_runs,
	// wf_rec_steps, wf_rec_events, wf_rec_labels and wf_rec_schema.
	DefaultTablePrefix = "wf_rec_"

	// DefaultTxAttempts is how many times a write transaction runs before
	// a contention error (see DefaultRetryable) is returned.
	DefaultTxAttempts = 8
)

Defaults of a Store.

View Source
const SchemaVersion = 1

SchemaVersion is the version of the tables this release writes. Migrate records it in the <prefix>schema table, so a later release knows which steps to apply.

Variables

This section is empty.

Functions

func DefaultRetryable

func DefaultRetryable(err error) bool

DefaultRetryable reports whether err looks like a contention error worth retrying the transaction for (see WithRetryable).

Types

type Dialect

type Dialect string

Dialect is the SQL engine the app's *sql.DB talks to. The store writes plain SQL on database/sql for each, so this module imports no database driver: the app opens the database with the driver it already uses.

const (
	// Postgres: PostgreSQL 13 or later.
	Postgres Dialect = "postgres"
	// MySQL: MySQL 8.0 or later (InnoDB).
	MySQL Dialect = "mysql"
	// SQLite: SQLite 3.35 or later.
	SQLite Dialect = "sqlite"
)

The supported engines, the same three as the sqlstore driver.

func Dialects

func Dialects() []Dialect

Dialects returns every supported dialect, as a fresh slice.

func (Dialect) Valid

func (d Dialect) Valid() bool

Valid reports whether d is a supported dialect.

type Option

type Option func(*Store) error

Option configures a Store.

func WithClock

func WithClock(c clock.Clock) Option

WithClock sets the clock Prune measures ages with (clock.Real by default).

func WithRetention

func WithRetention(r recorder.Retention) Option

WithRetention sets the retention Prune applies (see recorder.Retention); none by default, so Prune does nothing.

func WithRetryable

func WithRetryable(fn func(error) bool) Option

WithRetryable replaces DefaultRetryable, the test of whether a failed transaction is retried, for a driver whose contention errors read differently.

func WithTablePrefix

func WithTablePrefix(p string) Option

WithTablePrefix sets the prefix of every table name (DefaultTablePrefix by default): lower-case letters, digits and underscores, starting with a letter, at most 32 characters. Two stores with different prefixes share one database without meeting.

func WithTxAttempts

func WithTxAttempts(n int) Option

WithTxAttempts sets how many times a write transaction runs on a contention error (DefaultTxAttempts by default, at least 1).

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a recorder.Store on a SQL database (Postgres, MySQL or SQLite) through database/sql. Each run is one row in <prefix>runs, each step attempt one row in <prefix>steps, each event one row in <prefix>events, and the run's labels one row per key in <prefix>labels (so label filters are indexed lookups, not JSON scans). A busy run therefore never rewrites one large row per event.

Every write is one transaction that locks the run's row first, so the writes of one run are serialized across processes; a contention error (two first writes of a run racing, a deadlock, a busy SQLite file) is retried (WithTxAttempts, WithRetryable). Times are stored as Unix nanoseconds in UTC, zero for an unset time, so ordering and range filters behave the same on every engine.

It implements recorder.Store, recorder.Pruner and workflow.RunQuerier.

func New

func New(db *sql.DB, d Dialect, opts ...Option) (*Store, error)

New returns a Store over db, which the app opened with its own driver for dialect d. It does not touch the database: call Migrate before the first write. It fails on a nil db, an unknown dialect or an invalid option, with workflow.ErrInvalid.

func (*Store) AddEvent

func (s *Store) AddEvent(ctx context.Context, e recorder.EventWrite) error

AddEvent implements recorder.Store. The event's Seq is one more than the attempt's last, read under the run's row lock, so concurrent events of one run never share a Seq.

func (*Store) Dialect

func (s *Store) Dialect() Dialect

Dialect returns the store's dialect.

func (*Store) EndStep

func (s *Store) EndStep(ctx context.Context, e recorder.StepEnd) error

EndStep implements recorder.Store.

func (*Store) FinishRun

func (s *Store) FinishRun(ctx context.Context, f recorder.Finish) error

FinishRun implements recorder.Store.

func (*Store) MarkChecked

func (s *Store) MarkChecked(ctx context.Context, c recorder.Check) error

MarkChecked implements recorder.Store. A check of a run the store does not hold writes nothing.

func (*Store) Migrate

func (s *Store) Migrate(ctx context.Context) error

Migrate creates the store's tables and indexes when they are missing and records SchemaVersion. It is idempotent and safe to run on every start; two processes running it at once both succeed. It refuses a database whose recorded version is newer than this release knows (an older binary against a newer schema), with an error naming both versions.

func (*Store) MigrationScripts

func (s *Store) MigrationScripts() []string

MigrationScripts returns the statements Migrate runs, for an app that applies schema changes with its own migration tool instead. After running them, the app inserts SchemaVersion into the <prefix>schema table, or calls Migrate once, which then only records the version.

func (*Store) Pending

func (s *Store) Pending(ctx context.Context, limit int) ([]recorder.RunRecord, error)

Pending implements recorder.Store.

func (*Store) Prune

func (s *Store) Prune(ctx context.Context) (recorder.PruneResult, error)

Prune implements recorder.Pruner with the store's Retention (WithRetention), in batches of Retention.BatchSize so a backlog never holds one long transaction:

  • Stage two first: finished runs past their Runs age are read in full (steps and events included), handed to Archive when set, and deleted with their steps, events and labels. An Archive error stops the prune with that batch untouched.
  • Then stage one: finished runs past their Steps age lose their step and event rows; the run row stays.

Each rule is applied to its workflows: a PerWorkflow entry to its name, the default rule to every workflow without one. A zero age keeps that stage. Unfinished runs are never touched.

func (*Store) Records

func (s *Store) Records(ctx context.Context, q recorder.Query) (recorder.RecordPage, error)

Records implements recorder.Store: newest enqueue first, ties by key, descending, in pages of q.PageLimit with a recorder.EncodeCursor cursor.

func (*Store) Run

Run implements recorder.Store: the run with its steps and their events.

func (*Store) Runs

Runs implements recorder.Store and workflow.RunQuerier.

func (*Store) SetOutcome

func (s *Store) SetOutcome(ctx context.Context, o recorder.OutcomeWrite) error

SetOutcome implements recorder.Store.

func (*Store) StartStep

func (s *Store) StartStep(ctx context.Context, st recorder.StepStart) error

StartStep implements recorder.Store.

func (*Store) TablePrefix

func (s *Store) TablePrefix() string

TablePrefix returns the store's table prefix.

func (*Store) UpsertRun

func (s *Store) UpsertRun(ctx context.Context, w recorder.RunWrite) error

UpsertRun implements recorder.Store.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL