Documentation
¶
Overview ¶
Package sqlrecorder is a recorder.Store on Postgres, MySQL or SQLite, so a run history the app owns lives in its own database, next to any driver (a Hatchet engine included), and survives the driver's own retention.
It is built on database/sql alone and imports no database driver: the app opens the *sql.DB with the driver it already uses and names the engine:
db, _ := sql.Open("pgx", dsn) // the app's own driver
store, err := sqlrecorder.New(db, sqlrecorder.Postgres,
sqlrecorder.WithRetention(recorder.Retention{
RetentionRule: recorder.RetentionRule{Steps: 7 * 24 * time.Hour, Runs: 30 * 24 * time.Hour},
}))
err = store.Migrate(ctx) // idempotent, run on every start
rec, err := recorder.New(store)
c, err := workflow.Use(client, rec.Plugin())
Tables (prefix "wf_rec_" by default, WithTablePrefix): runs (one row per run), steps (one row per step attempt), events (one row per step event), labels (one row per run label, for indexed label filters) and schema (the applied SchemaVersion). Steps are rows of their own rather than an array on the run row, so a busy run never rewrites one large row per event.
Every write is one transaction that locks the run's row first and is retried on contention; reads answer workflow.RunQuerier and the recorder's richer Records and Run. recordertest.RunStore holds it to the Store contract on all three engines.
Index ¶
- Constants
- func DefaultRetryable(err error) bool
- type Dialect
- type Option
- type Store
- func (s *Store) AddEvent(ctx context.Context, e recorder.EventWrite) error
- func (s *Store) Dialect() Dialect
- func (s *Store) EndStep(ctx context.Context, e recorder.StepEnd) error
- func (s *Store) FinishRun(ctx context.Context, f recorder.Finish) error
- func (s *Store) MarkChecked(ctx context.Context, c recorder.Check) error
- func (s *Store) Migrate(ctx context.Context) error
- func (s *Store) MigrationScripts() []string
- func (s *Store) Pending(ctx context.Context, limit int) ([]recorder.RunRecord, error)
- func (s *Store) Prune(ctx context.Context) (recorder.PruneResult, error)
- func (s *Store) Records(ctx context.Context, q recorder.Query) (recorder.RecordPage, error)
- func (s *Store) Run(ctx context.Context, ref workflow.RunRef) (recorder.RunRecord, error)
- func (s *Store) Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (s *Store) SetOutcome(ctx context.Context, o recorder.OutcomeWrite) error
- func (s *Store) StartStep(ctx context.Context, st recorder.StepStart) error
- func (s *Store) TablePrefix() string
- func (s *Store) UpsertRun(ctx context.Context, w recorder.RunWrite) error
Constants ¶
const ( // DefaultTablePrefix starts every table name: wf_rec_runs, // wf_rec_steps, wf_rec_events, wf_rec_labels and wf_rec_schema. DefaultTablePrefix = "wf_rec_" // DefaultTxAttempts is how many times a write transaction runs before // a contention error (see DefaultRetryable) is returned. DefaultTxAttempts = 8 )
Defaults of a Store.
const SchemaVersion = 1
SchemaVersion is the version of the tables this release writes. Migrate records it in the <prefix>schema table, so a later release knows which steps to apply.
Variables ¶
This section is empty.
Functions ¶
func DefaultRetryable ¶
DefaultRetryable reports whether err looks like a contention error worth retrying the transaction for (see WithRetryable).
Types ¶
type Dialect ¶
type Dialect string
Dialect is the SQL engine the app's *sql.DB talks to. The store writes plain SQL on database/sql for each, so this module imports no database driver: the app opens the database with the driver it already uses.
const ( // Postgres: PostgreSQL 13 or later. Postgres Dialect = "postgres" // MySQL: MySQL 8.0 or later (InnoDB). MySQL Dialect = "mysql" // SQLite: SQLite 3.35 or later. SQLite Dialect = "sqlite" )
The supported engines, the same three as the sqlstore driver.
type Option ¶
Option configures a Store.
func WithRetention ¶
WithRetention sets the retention Prune applies (see recorder.Retention); none by default, so Prune does nothing.
func WithRetryable ¶
WithRetryable replaces DefaultRetryable, the test of whether a failed transaction is retried, for a driver whose contention errors read differently.
func WithTablePrefix ¶
WithTablePrefix sets the prefix of every table name (DefaultTablePrefix by default): lower-case letters, digits and underscores, starting with a letter, at most 32 characters. Two stores with different prefixes share one database without meeting.
func WithTxAttempts ¶
WithTxAttempts sets how many times a write transaction runs on a contention error (DefaultTxAttempts by default, at least 1).
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a recorder.Store on a SQL database (Postgres, MySQL or SQLite) through database/sql. Each run is one row in <prefix>runs, each step attempt one row in <prefix>steps, each event one row in <prefix>events, and the run's labels one row per key in <prefix>labels (so label filters are indexed lookups, not JSON scans). A busy run therefore never rewrites one large row per event.
Every write is one transaction that locks the run's row first, so the writes of one run are serialized across processes; a contention error (two first writes of a run racing, a deadlock, a busy SQLite file) is retried (WithTxAttempts, WithRetryable). Times are stored as Unix nanoseconds in UTC, zero for an unset time, so ordering and range filters behave the same on every engine.
It implements recorder.Store, recorder.Pruner and workflow.RunQuerier.
func New ¶
New returns a Store over db, which the app opened with its own driver for dialect d. It does not touch the database: call Migrate before the first write. It fails on a nil db, an unknown dialect or an invalid option, with workflow.ErrInvalid.
func (*Store) AddEvent ¶
AddEvent implements recorder.Store. The event's Seq is one more than the attempt's last, read under the run's row lock, so concurrent events of one run never share a Seq.
func (*Store) MarkChecked ¶
MarkChecked implements recorder.Store. A check of a run the store does not hold writes nothing.
func (*Store) Migrate ¶
Migrate creates the store's tables and indexes when they are missing and records SchemaVersion. It is idempotent and safe to run on every start; two processes running it at once both succeed. It refuses a database whose recorded version is newer than this release knows (an older binary against a newer schema), with an error naming both versions.
func (*Store) MigrationScripts ¶
MigrationScripts returns the statements Migrate runs, for an app that applies schema changes with its own migration tool instead. After running them, the app inserts SchemaVersion into the <prefix>schema table, or calls Migrate once, which then only records the version.
func (*Store) Prune ¶
Prune implements recorder.Pruner with the store's Retention (WithRetention), in batches of Retention.BatchSize so a backlog never holds one long transaction:
- Stage two first: finished runs past their Runs age are read in full (steps and events included), handed to Archive when set, and deleted with their steps, events and labels. An Archive error stops the prune with that batch untouched.
- Then stage one: finished runs past their Steps age lose their step and event rows; the run row stays.
Each rule is applied to its workflows: a PerWorkflow entry to its name, the default rule to every workflow without one. A zero age keeps that stage. Unfinished runs are never touched.
func (*Store) Records ¶
Records implements recorder.Store: newest enqueue first, ties by key, descending, in pages of q.PageLimit with a recorder.EncodeCursor cursor.
func (*Store) SetOutcome ¶
SetOutcome implements recorder.Store.
func (*Store) TablePrefix ¶
TablePrefix returns the store's table prefix.