recorder

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package recorder keeps a run history that the app owns, separate from the driver's own store: one record per run, whatever started it, with its trigger source, input, every step attempt with its input, output, error and timeline events, the result, the outcome and the timings.

Why it exists: a driver's run history is the driver's store. On an engine driver it ends where the engine's retention ends, and no driver keeps a step's timeline or says who started a run. A recorder fixes that for every driver at once, because it is built only from the seams every driver already runs: enqueue middleware (a run is accepted), step middleware (an attempt starts and ends, events, outcome) and the run-finished hook (a run ends). No driver changes, and an app that never imports this module pays nothing.

Wiring it in:

store := recorder.NewMemoryStore()           // or sqlrecorder.New(db, dialect)
rec, err := recorder.New(store)
c, err := workflow.Use(memory.New(), rec.Plugin())
// register and start through c; then read with store.Runs / store.Run

The pieces:

  • Store is the contract a run store keeps: idempotent upserts keyed by RunRef, reads that answer workflow.RunQuerier plus richer Records and Run. MemoryStore is the in-process one; recorder/sqlrecorder is a SQL one in its own module; an app can bring its own and prove it with recordertest.RunStore. The Apply* functions hold the merge rules, so every store merges a write the same way.
  • Recorder turns the three seams into Store writes. A failed write never fails a run: it goes to the OnError hook.
  • Redactor scrubs what is stored (JSONPaths, PerWorkflow, Chain), and WithMaxPayload bounds it; a runs table is read by more people than the queue, so it is where a token in a payload leaks.
  • Reconciler closes the gaps the seams leave (a worker that died before its step ended, a cancel from an engine UI, runs started from an engine's CLI): it re-checks unfinished records against the client, flags stale ones (never cancelling anything) and backfills runs it never saw with SourceExternal.

Delivery: every write is an upsert, so at-least-once delivery from any seam is safe. Run ends are guaranteed only on a driver with workflow.CapRunFinished; elsewhere the Reconciler is the fallback.

Index

Constants

View Source
const (
	// DefaultReconcileInterval is the pause between two passes of Run.
	DefaultReconcileInterval = 30 * time.Second

	// DefaultStaleAfter is how long an unfinished run may go without any
	// recorded activity before it is flagged stale.
	DefaultStaleAfter = 10 * time.Minute

	// DefaultBackfillWindow is how far back each pass lists the client's
	// runs looking for ones the recorder never saw.
	DefaultBackfillWindow = time.Hour

	// DefaultReconcileBatch is how many unfinished records one pass checks,
	// and the page size of the backfill listing.
	DefaultReconcileBatch = 100

	// DefaultBackfillPages bounds the pages one backfill pass reads, so a
	// huge window cannot make one pass unbounded.
	DefaultBackfillPages = 10
)

Defaults of a Reconciler.

View Source
const (
	// DefaultMaxPayload is the largest body (input, step input and
	// output, result) a Recorder stores; a larger one is recorded by size
	// only.
	DefaultMaxPayload = 64 << 10

	// DefaultWriteTimeout bounds one store write, so a slow store delays a
	// step by at most this much.
	DefaultWriteTimeout = 5 * time.Second

	// DefaultPluginName is the name Plugin reports to workflow.Use.
	DefaultPluginName = "recorder"
)

Defaults of a Recorder.

View Source
const DefaultPruneBatch = 500

DefaultPruneBatch is Retention.Batch when zero.

View Source
const MaxPruneBatch = 10000

MaxPruneBatch bounds Retention.Batch.

View Source
const MaxSearchLen = 256

MaxSearchLen is the longest Query.Search, in bytes.

QueryCaps is what a Store's Runs and Records honour of a workflow.RunQuery: label prefixes, tenant and due-time filters on top of the basic ones. Stores pass it to workflow.CheckRunQuery (through ValidateQuery), and a caller that wants to know before it asks, an admin view, can read it.

View Source
const Redacted = "[REDACTED]"

Redacted is the value a JSONPaths redactor writes in place of a sensitive value (as a JSON string in a body, as is in an event attribute).

Variables

This section is empty.

Functions

func ApplyCheck

func ApplyCheck(r *RunRecord, c Check)

ApplyCheck records a Reconciler check. It moves CheckedAt and the Stale flag, and the status when the client reported a different unfinished one; it never touches UpdatedAt, which measures the run's own activity. A record that became terminal meanwhile keeps its status.

func ApplyEvent

func ApplyEvent(r *RunRecord, e EventWrite)

ApplyEvent merges the run-level part of an event: activity.

func ApplyFinish

func ApplyFinish(r *RunRecord, f Finish, latestStart time.Time) bool

ApplyFinish merges f into r and reports whether it did. A finish is ignored when an attempt started after it (latestStart, the newest StartedAt among the run's step records): it is a late duplicate of an end the run has since been retried out of. Otherwise the status, error, end time and result are f's (a repeated finish rewrites the same values), and the outcome defaults from the status unless a step set one.

func ApplyOutcome

func ApplyOutcome(r *RunRecord, o OutcomeWrite)

ApplyOutcome records a set outcome; the last one set wins.

func ApplyRunWrite

func ApplyRunWrite(r *RunRecord, isNew bool, w RunWrite)

ApplyRunWrite merges w into r. isNew says r came from NewRecord, so its status is set from w.

func ApplyStepEnd

func ApplyStepEnd(r *RunRecord, e StepEnd)

ApplyStepEnd merges the run-level part of e into r: a failed attempt joins the error history (bounded by e.ErrorHistory) and, while the run is unfinished, becomes its current Error, as RunInfo.Error documents for a run waiting on its retry. A snooze is not a failure and adds nothing.

func ApplyStepEndToStep

func ApplyStepEndToStep(sr *StepRecord, e StepEnd)

ApplyStepEndToStep applies e to the attempt's own record. An end for an attempt the store never saw start creates it from e (sr is then the zero StepRecord with Step and Attempt set by the store).

func ApplyStepStart

func ApplyStepStart(r *RunRecord, isNewAttempt bool, s StepStart)

ApplyStepStart merges the run-level part of s into r: the run is running (a terminal run re-opens, because only a retry starts an attempt after the end), its first start and attempt count move, and the node joins AttemptedBy. The step record itself is NewStepRecord(s). isNewAttempt says the store had no record of this (step, attempt) yet, so a repeated start (at-least-once, or a snoozed attempt's next try) does not count twice.

func CheckRef

func CheckRef(op string, ref workflow.RunRef) error

CheckRef refuses, with workflow.ErrInvalid, a write that names no run. The bundled stores call it first; an app's own store should too (recordertest.RunStore expects the refusal).

func CheckStep

func CheckStep(op string, ref workflow.RunRef, step string, attempt int) error

CheckStep refuses, with workflow.ErrInvalid, a step write without a RunID, a step name or an attempt from 1.

func EncodeCursor

func EncodeCursor(c Cursor) string

EncodeCursor makes the opaque cursor string the bundled stores hand out as a page's Next.

func InfoPage

func InfoPage(p RecordPage) workflow.RunPage

InfoPage converts a page of records to the RunPage a RunQuerier returns.

func KeyOf

func KeyOf(ref workflow.RunRef) string

KeyOf is the Store identity of ref (see RunRecord.Key). The prefixes keep a RunID and a ScheduledID with the same text apart.

func PruneEvery

func PruneEvery(ctx context.Context, p Pruner, interval time.Duration, onError func(error)) error

PruneEvery calls p.Prune every interval until ctx is cancelled, handing each error to onError (nil ignores them), and returns nil when ctx ends. It is the loop an app runs in a goroutine next to its client; a store's Prune is also safe to call from a scheduled job instead.

func ResetStepRecord

func ResetStepRecord(sr *StepRecord, s StepStart)

ResetStepRecord applies a repeated StepStart to an existing attempt record: running again, its end cleared, its events kept.

func SortPending

func SortPending(rs []RunRecord)

SortPending orders records the way Pending returns them: never checked first, then least recently checked, ties by Key.

func SortRecords

func SortRecords(rs []RunRecord)

SortRecords orders records the way Runs and Records list them: newest enqueue first, ties broken by Key descending.

func SortSteps

func SortSteps(steps []StepRecord)

SortSteps orders step records by start time, then step name, then attempt, the order RunRecord.Steps documents.

func StartsNewAttempt

func StartsNewAttempt(existing *StepRecord) bool

StartsNewAttempt reports whether a StepStart counts as a new attempt, given the store's record of that (step, attempt), nil when it has none. A record with a zero StartedAt is a placeholder an event created when it raced ahead of its attempt's start, so the start still counts; a record that started before is a repeat (at-least-once, or a snoozed attempt's next try) and does not.

func ValidateQuery

func ValidateQuery(q Query) error

ValidateQuery checks q: workflow.ValidateRunQuery and workflow.CheckRunQuery against QueryCaps for the embedded query, known sources and outcomes, and Search within MaxSearchLen. Errors wrap workflow.ErrInvalid or workflow.ErrUnsupported.

func ValidateRetention

func ValidateRetention(r Retention) error

ValidateRetention refuses negative ages and a Batch outside [0, MaxPruneBatch]. Errors wrap workflow.ErrInvalid.

Types

type Check

type Check struct {
	Ref workflow.RunRef

	// Status is the non-terminal status the client reported, or "" to
	// leave the record's status (the client did not know the run).
	Status workflow.Status

	// Stale sets or clears the record's Stale flag.
	Stale bool

	// At is when the check was made; it becomes the record's CheckedAt.
	At time.Time
}

Check is the Reconciler's verdict on an unfinished record.

type Cursor

type Cursor struct {
	EnqueuedAt time.Time
	Key        string
}

Cursor is the position after one record in the listing order (newest enqueue first, Key descending): the next page starts after it.

func DecodeCursor

func DecodeCursor(s string) (Cursor, error)

DecodeCursor reads a cursor EncodeCursor made. Anything else is an error wrapping workflow.ErrInvalid, as RunQuerier documents for a cursor the store did not issue.

func (Cursor) After

func (c Cursor) After(r RunRecord) bool

After reports whether r sorts after c in the listing order, so it belongs on the page that c continues.

type ErrorFunc

type ErrorFunc func(ctx context.Context, op WriteOp, ref workflow.RunRef, err error)

ErrorFunc is told about a failed write. The run goes on regardless: a recorder never fails a run. It runs on the goroutine that made the write (a step's, an enqueuer's, the driver's run-finished goroutine), so it must return quickly.

type EventRecord

type EventRecord struct {
	// Seq orders the events of one attempt; the store assigns it (1, 2,
	// ...).
	Seq int

	// Message, Level and Attrs are workflow.StepEvent's, after redaction.
	Message string
	Level   slog.Level
	Attrs   []workflow.EventAttr

	// At is when it was recorded, on the recorder's clock.
	At time.Time
}

EventRecord is one stored step event.

func NewEventRecord

func NewEventRecord(e EventWrite, seq int) EventRecord

NewEventRecord is the stored form of e; the store sets Seq to one more than the attempt's last event.

func (EventRecord) Clone

func (e EventRecord) Clone() EventRecord

Clone returns a deep copy.

type EventWrite

type EventWrite struct {
	Ref     workflow.RunRef
	Step    string
	Attempt int

	Message string
	Level   slog.Level
	Attrs   []workflow.EventAttr

	// At is when the event was recorded.
	At time.Time
}

EventWrite is one step event to append.

type Finish

type Finish struct {
	// Ref carries both IDs when the run was delayed and the driver knows
	// both (workflow.RunEvent.Ref does).
	Ref workflow.RunRef

	Workflow  string
	Tenant    string
	Labels    map[string]string
	Source    workflow.Source
	SourceRef string

	// Status is terminal.
	Status workflow.Status

	// Error is why the run failed or was cancelled; FailedStep the step
	// whose failure ended it.
	Error      string
	FailedStep string

	// Output is the run's result after redaction when the writer read it,
	// else nil; OutputSize is its size before that.
	Output     []byte
	OutputSize int

	// FinishedAt is when the run ended, on the driver's clock.
	FinishedAt time.Time

	// At is when the write was made.
	At time.Time
}

Finish is a run reaching a terminal status (the run-finished seam, or the Reconciler finding the client's answer).

type MemoryOption

type MemoryOption func(*MemoryStore)

MemoryOption configures a MemoryStore.

func WithMemoryClock

func WithMemoryClock(c clock.Clock) MemoryOption

WithMemoryClock sets the clock Prune measures ages with; clock.Real by default.

func WithMemoryRetention

func WithMemoryRetention(r Retention) MemoryOption

WithMemoryRetention sets the retention Prune applies (see Retention); none by default, so Prune does nothing.

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is the in-process Store: records live in a map guarded by a mutex, so it is for tests, single-process apps and as the reference the store conformance suite (recordertest.RunStore) is written against. It keeps nothing across a restart. It implements Pruner.

func NewMemoryStore

func NewMemoryStore(opts ...MemoryOption) (*MemoryStore, error)

NewMemoryStore returns an empty MemoryStore. It fails only on an invalid retention (ValidateRetention).

func (*MemoryStore) AddEvent

func (s *MemoryStore) AddEvent(_ context.Context, e EventWrite) error

AddEvent implements Store.

func (*MemoryStore) EndStep

func (s *MemoryStore) EndStep(_ context.Context, e StepEnd) error

EndStep implements Store.

func (*MemoryStore) FinishRun

func (s *MemoryStore) FinishRun(_ context.Context, f Finish) error

FinishRun implements Store.

func (*MemoryStore) MarkChecked

func (s *MemoryStore) MarkChecked(_ context.Context, c Check) error

MarkChecked implements Store. A check of a run the store does not hold does nothing: the Reconciler only checks records it read.

func (*MemoryStore) Pending

func (s *MemoryStore) Pending(_ context.Context, limit int) ([]RunRecord, error)

Pending implements Store.

func (*MemoryStore) Prune

func (s *MemoryStore) Prune(ctx context.Context) (PruneResult, error)

Prune implements Pruner: stage one drops the steps of runs finished longer ago than the rule's Steps age, stage two archives and deletes runs finished longer ago than its Runs age.

func (*MemoryStore) Records

func (s *MemoryStore) Records(_ context.Context, q Query) (RecordPage, error)

Records implements Store.

func (*MemoryStore) Run

Run implements Store.

func (*MemoryStore) Runs

Runs implements Store and workflow.RunQuerier.

func (*MemoryStore) SetOutcome

func (s *MemoryStore) SetOutcome(_ context.Context, o OutcomeWrite) error

SetOutcome implements Store.

func (*MemoryStore) StartStep

func (s *MemoryStore) StartStep(_ context.Context, st StepStart) error

StartStep implements Store.

func (*MemoryStore) UpsertRun

func (s *MemoryStore) UpsertRun(_ context.Context, w RunWrite) error

UpsertRun implements Store.

type Option

type Option func(*Recorder) error

Option configures a Recorder.

func WithClock

func WithClock(c clock.Clock) Option

WithClock sets the clock the recorder stamps writes with; clock.Real by default. Use the driver's clock so records and the driver agree.

func WithDefaultSource

func WithDefaultSource(s workflow.Source) Option

WithDefaultSource is the source recorded for a run whose labels say nothing (workflow.SourceOf returns ""). Empty by default, which records no source. Must pass workflow.ValidateSource when set.

func WithErrorHistory

func WithErrorHistory(n int) Option

WithErrorHistory bounds the attempt errors kept per run (workflow.DefaultErrorHistory by default); zero keeps none.

func WithMaxPayload

func WithMaxPayload(n int) Option

WithMaxPayload sets the largest body stored, in bytes (DefaultMaxPayload by default). A larger body is recorded by its size only. Zero stores no bodies at all, only sizes, for apps whose payloads must never be copied; a negative value is invalid.

func WithName

func WithName(name string) Option

WithName sets the plugin name (DefaultPluginName by default), so two recorders, say one per store, can be applied to one client.

func WithOnError

func WithOnError(fn ErrorFunc) Option

WithOnError is told about every failed write. By default a failure is logged with slog.Default at warn level.

func WithRedactor

func WithRedactor(red Redactor) Option

WithRedactor scrubs every body and event attribute before it is stored (see Redactor, JSONPaths). None by default.

func WithResultReader

func WithResultReader(rr workflow.ResultReader) Option

WithResultReader reads a succeeded run's result from rr (usually the client, on a driver with workflow.CapResult) when the run ends, so the record holds it. Without it the record holds the step outputs only.

func WithStepBodies

func WithStepBodies(keep bool) Option

WithStepBodies says whether step inputs and outputs are stored (true by default). The run's own input and result follow WithMaxPayload only.

func WithWriteTimeout

func WithWriteTimeout(d time.Duration) Option

WithWriteTimeout bounds each store write (DefaultWriteTimeout by default). Writes run on a context detached from the caller's cancellation, so a step whose ctx was cancelled still records its end.

type OutcomeWrite

type OutcomeWrite struct {
	Ref     workflow.RunRef
	Outcome workflow.Outcome
	Note    string
	At      time.Time
}

OutcomeWrite is the outcome a step set.

type PayloadInfo

type PayloadInfo struct {
	Kind     PayloadKind
	Workflow string
	// Step is empty for PayloadInput and PayloadResult.
	Step string
}

PayloadInfo says what is about to be stored, so one Redactor can treat workflows, steps and kinds differently.

type PayloadKind

type PayloadKind string

PayloadKind is which body or field a Redactor is asked about.

const (
	// PayloadInput is the run's input (Job.Payload) at enqueue.
	PayloadInput PayloadKind = "input"
	// PayloadStepInput is a step attempt's input.
	PayloadStepInput PayloadKind = "step-input"
	// PayloadStepOutput is a step attempt's output.
	PayloadStepOutput PayloadKind = "step-output"
	// PayloadResult is the run's result (WithResultReader).
	PayloadResult PayloadKind = "result"
	// PayloadEvent is a step event's attribute.
	PayloadEvent PayloadKind = "event"
)

The kinds of data a Recorder stores and so redacts.

type PruneResult

type PruneResult struct {
	// StepsCleared counts runs whose steps stage one dropped.
	StepsCleared int

	// RunsDeleted counts runs stage two deleted.
	RunsDeleted int

	// Archived counts runs handed to Archive (equal to RunsDeleted when
	// Archive is set).
	Archived int
}

PruneResult is what one Prune did.

type Pruner

type Pruner interface {
	Prune(ctx context.Context) (PruneResult, error)
}

Pruner is the optional retention of a Store: MemoryStore and sqlrecorder.Store implement it, configured with a Retention.

type Query

type Query struct {
	workflow.RunQuery

	// Sources matches runs whose Source is any of these; empty matches
	// any.
	Sources []workflow.Source

	// Outcomes matches runs whose Outcome is any of these; empty matches
	// any. An unfinished run with no outcome set matches none.
	Outcomes []workflow.Outcome

	// Search matches runs where it is a case-insensitive substring of the
	// RunID, the ScheduledID, the workflow name or the SourceRef; empty
	// matches any. It is a lookup box for an operator, not a full-text
	// index: a store may scan to answer it.
	Search string

	// OnlyStale matches only runs the Reconciler flagged stale.
	OnlyStale bool
}

Query is a workflow.RunQuery plus the filters only a recorder can answer, because only it knows a run's source and outcome.

func (Query) Match

func (q Query) Match(r RunRecord) bool

Match reports whether r passes q's filters (everything but paging). The in-memory store filters with it; a SQL store builds the same filters as a WHERE clause.

type ReconcileOption

type ReconcileOption func(*Reconciler) error

ReconcileOption configures a Reconciler.

func WithBackfill

func WithBackfill(on bool) ReconcileOption

WithBackfill turns the backfill on or off (on by default).

func WithBackfillPages

func WithBackfillPages(n int) ReconcileOption

WithBackfillPages bounds the pages one backfill pass reads.

func WithBackfillQuery

func WithBackfillQuery(q workflow.RunQuery) ReconcileOption

WithBackfillQuery scopes the backfill listing to the runs q matches (a workflow, a tenant, labels), for a client whose engine other apps share. Its From, Limit and Cursor are the Reconciler's own and are ignored; the rest must pass workflow.ValidateRunQuery.

func WithBackfillWindow

func WithBackfillWindow(d time.Duration) ReconcileOption

WithBackfillWindow sets how far back the backfill lists runs.

func WithBatch

func WithBatch(n int) ReconcileOption

WithBatch sets how many records one pass checks and the backfill page size, from 1 to workflow.MaxRunsLimit.

func WithInterval

func WithInterval(d time.Duration) ReconcileOption

WithInterval sets the pause between passes of Run.

func WithReconcileClock

func WithReconcileClock(c clock.Clock) ReconcileOption

WithReconcileClock sets the clock the Reconciler measures with (clock.Real by default); use the recorder's.

func WithReconcileOnError

func WithReconcileOnError(fn func(ctx context.Context, err error)) ReconcileOption

WithReconcileOnError is told about every error of a pass that did not stop it (one run's check or write); Run also hands it a pass's error. By default errors are dropped by Pass and logged by Run with slog.Default.

func WithStaleAfter

func WithStaleAfter(d time.Duration) ReconcileOption

WithStaleAfter sets how long an unfinished run may be silent before it is flagged stale.

type ReconcileReport

type ReconcileReport struct {
	// Checked counts unfinished records compared with the client.
	Checked int
	// Finished counts records the pass finished because the client
	// reported them terminal.
	Finished int
	// Stale counts records flagged stale by this pass.
	Stale int
	// Missing counts records the client did not know (also flagged stale).
	Missing int
	// Backfilled counts runs the backfill upserted (new or filled).
	Backfilled int
	// BackfillSkipped is true when the client is not a RunQuerier, so no
	// backfill ran.
	BackfillSkipped bool
	// Errors counts per-run errors that did not stop the pass.
	Errors int
}

ReconcileReport is what one Pass did.

type Reconciler

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler closes the gaps the recorder's seams leave:

  • Check: every unfinished record (oldest check first, a batch per pass) is compared with the client's Status. A run the client reports terminal is finished on the record, with its error and end time when the client is a workflow.RunQuerier. A run the client still reports unfinished takes the client's status, and is flagged Stale when nothing was recorded for it for StaleAfter. A run the client does not know (pruned, or a lost delayed start) is flagged Stale. Nothing is ever cancelled: Stale is for an operator to look at.
  • Backfill: when the client is a workflow.RunQuerier, the runs it lists enqueued within the backfill window are upserted, so a run started past the recorder (an engine UI or CLI, a crash between accept and record, a workflow registered on the bare driver) gets a record, with SourceExternal when its labels name no source. Upserts only fill, so a run the recorder already holds is left as recorded (its ScheduledID and RunID are joined when the listing shows both).

A Reconciler works on any client; backfill and error details need the client to be a workflow.RunQuerier and are skipped otherwise (ReconcileReport.BackfillSkipped). Run several Reconcilers over one store safely: every write is an upsert.

func NewReconciler

func NewReconciler(c workflow.Client, s Store, opts ...ReconcileOption) (*Reconciler, error)

NewReconciler returns a Reconciler of s against c. It fails on a nil client or store, or an invalid option, with workflow.ErrInvalid.

func (*Reconciler) Pass

Pass runs one check and one backfill. It returns an error only when it could not read the store's pending records or the client's run listing; a single run's failure is counted, handed to the error hook, and the pass goes on.

func (*Reconciler) Run

func (r *Reconciler) Run(ctx context.Context) error

Run calls Pass every interval until ctx is cancelled and then returns nil. A pass's error goes to the error hook (logged by default) and the loop goes on.

type RecordPage

type RecordPage struct {
	Runs []RunRecord
	Next string
}

RecordPage is one page of Records.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder turns the three seams every driver runs (enqueue middleware, step middleware, run-finished observers) into Store writes. Build one with New and apply it with workflow.Use(client, rec.Plugin()); a Recorder is safe for concurrent use and keeps no state of its own besides its configuration, so one Recorder may serve several clients.

func New

func New(s Store, opts ...Option) (*Recorder, error)

New returns a Recorder writing to s. It fails on a nil store or an invalid option, with an error wrapping workflow.ErrInvalid.

func (*Recorder) EnqueueMiddleware

func (r *Recorder) EnqueueMiddleware() workflow.EnqueueMiddleware

EnqueueMiddleware records every run the client accepts. A delayed run is recorded under its ScheduledID with StatusScheduled; others under their RunID with StatusQueued. A failed enqueue records nothing, and a failed write never fails the enqueue. Chained jobs are recorded when their first step starts, since the driver enqueues them past the middleware.

func (*Recorder) EnqueuePlugin

func (r *Recorder) EnqueuePlugin() workflow.Plugin

EnqueuePlugin returns a workflow.Plugin with the recorder's enqueue middleware only, for a client that enqueues but runs nothing: a web or API node built in a driver's enqueue-only mode (workflow.CapEnqueueOnly), which has no run-finished hook for Plugin to register on. It records each accepted run with its source and input; the worker processes, wrapped with Plugin over the same store, record the steps and the end into the same record (every write is an upsert keyed by the run).

func (*Recorder) OnRunFinished

func (r *Recorder) OnRunFinished(ctx context.Context, ev workflow.RunEvent)

OnRunFinished records a run's end; it is the run-finished observer the plugin registers, exported for an app that wires the seams by hand. Delivery is at least once and the write is an upsert, so a repeat is harmless. With WithResultReader it reads a succeeded run's result first.

func (*Recorder) Plugin

func (r *Recorder) Plugin() workflow.Plugin

Plugin returns the workflow.Plugin that applies the recorder to a client: workflow.Use(client, rec.Plugin()). It contributes one enqueue middleware, one step middleware and one run-finished observer, so the client must implement workflow.RunNotifier (every bundled driver does, except in enqueue-only mode: use EnqueuePlugin there).

func (*Recorder) StepMiddleware

func (r *Recorder) StepMiddleware() workflow.Middleware

StepMiddleware records each step attempt's start and end, and binds on the step's ctx a workflow.StepEventSink (workflow.RecordEvent lands on the attempt) and a workflow.OutcomeSetter (workflow.SetOutcome lands on the run). A ctx without workflow.StepInfo (a handler called outside a driver) passes through unrecorded. The handler's output and error are returned unchanged.

func (*Recorder) Store

func (r *Recorder) Store() Store

Store returns the store the recorder writes to.

type Redactor

type Redactor interface {
	// RedactPayload returns body as it may be stored.
	RedactPayload(info PayloadInfo, body []byte) []byte

	// RedactAttr returns a as it may be stored.
	RedactAttr(info PayloadInfo, a workflow.EventAttr) workflow.EventAttr
}

Redactor scrubs data before a Recorder stores it. It runs on the step's goroutine for every body and event, so it must be quick and safe for concurrent use, and must not modify its input slice (return a new one).

Event messages are not redacted: a message is the step author's own text. Put values in attributes, which are.

func Chain

func Chain(rs ...Redactor) Redactor

Chain returns a Redactor that applies rs in order, each to what the one before returned; nil entries are skipped.

func JSONPaths

func JSONPaths(paths ...string) (Redactor, error)

JSONPaths returns a Redactor that replaces the values at the given dot paths of a JSON body with Redacted, and the value of an event attribute whose key matches a path. A segment "*" matches any key of an object or any element of an array; a number segment matches that array index.

recorder.JSONPaths("password", "card.number", "users.*.token")

A body that is not valid JSON is stored unchanged (the redactor cannot find a path in it); an app with non-JSON payloads writes its own Redactor, or sets WithMaxPayload(0) to keep no bodies at all. A body with nothing to redact is stored byte for byte; one with a match is re-encoded, so its object keys come out sorted. An empty path or segment is an error wrapping workflow.ErrInvalid.

func PerWorkflow

func PerWorkflow(byName map[string]Redactor, fallback Redactor) Redactor

PerWorkflow returns a Redactor that uses byName[info.Workflow] when the workflow has an entry and fallback otherwise; a nil fallback stores the data of other workflows unchanged. The map is copied.

type Retention

type Retention struct {
	RetentionRule

	// PerWorkflow overrides RetentionRule for the named workflows.
	PerWorkflow map[string]RetentionRule

	// Archive, when set, is handed every record before stage two deletes
	// it (with its steps, when stage one left any). An error stops the
	// prune and deletes nothing of that batch, so nothing is lost when an
	// archive is down; the next prune tries again.
	Archive func(ctx context.Context, runs []RunRecord) error

	// Batch is how many runs one prune step handles at a time, so a big
	// backlog never holds one long transaction. Zero means
	// DefaultPruneBatch.
	Batch int
}

Retention is how long a store keeps finished runs, in two stages, so the heavy part of a run goes first and the summary row a runs page lists stays longer:

  • Stage one, after Steps: the run's step records and their events are dropped; the run record (status, source, outcome, error, timings, input and result) stays.
  • Stage two, after Runs: the run record itself is deleted, after Archive (when set) accepted it.

Ages are measured from the run's FinishedAt; unfinished runs are never pruned. A zero duration keeps that stage forever. PerWorkflow overrides both ages for a workflow name.

func (Retention) BatchSize

func (r Retention) BatchSize() int

BatchSize returns Batch with its default applied.

func (Retention) IsZero

func (r Retention) IsZero() bool

IsZero reports whether r prunes nothing.

func (Retention) Rule

func (r Retention) Rule(wf string) RetentionRule

Rule returns the rule for workflow wf: its PerWorkflow entry, else the default.

type RetentionRule

type RetentionRule struct {
	// Steps is the age after which a finished run's steps and events are
	// dropped; zero keeps them.
	Steps time.Duration

	// Runs is the age after which a finished run is deleted; zero keeps
	// it.
	Runs time.Duration
}

RetentionRule is the two ages of a Retention.

type RunRecord

type RunRecord struct {
	workflow.RunInfo

	// Source is what started the run (workflow.SourceOf of its labels, or
	// the recorder's default source, or SourceExternal for a backfilled
	// run); SourceRef is the reference within it.
	Source    workflow.Source
	SourceRef string

	// Outcome is the last outcome a step set (workflow.SetOutcome), or
	// workflow.DefaultOutcome of the terminal status when none was set;
	// "" while the run is unfinished and nothing was set. OutcomeNote is
	// the note that came with a set outcome.
	Outcome     workflow.Outcome
	OutcomeNote string

	// OutcomeSet is true when a step set the outcome, so a later default
	// never overwrites it.
	OutcomeSet bool

	// PayloadSize and OutputSize are the sizes of the input and result as
	// the recorder saw them; larger than len(Payload) or len(Output) when
	// the body was not kept (WithMaxPayload).
	PayloadSize int
	OutputSize  int

	// Stale is set by the Reconciler when the client still reports the run
	// unfinished (or does not know it) and nothing was recorded for it for
	// longer than its StaleAfter. It is a flag for an operator; nothing is
	// cancelled. Any later write for the run clears it.
	Stale bool

	// UpdatedAt is the time of the last write that changed the run's life
	// (enqueue, step start or end, event, outcome, finish), on the
	// recorder's clock. The Reconciler measures staleness from it.
	UpdatedAt time.Time

	// CheckedAt is when the Reconciler last compared the record with the
	// client; zero when it never did. It orders Pending so every
	// unfinished record gets its turn.
	CheckedAt time.Time

	// Steps is every step attempt, ordered by start time then step name
	// then attempt, each with its events. Store.Run fills it; listings
	// (Records, Runs) leave it nil to stay cheap.
	Steps []StepRecord
}

RunRecord is one run as a Store keeps it: the RunInfo a RunQuerier answers with, plus what only a recorder knows.

The embedded RunInfo is filled from the recorder's own writes, never from the driver: Payload is the run's input as enqueued (after redaction, and nil when it was larger than the recorder's MaxPayload; PayloadSize says how big it was), Output is the run's result when the recorder could read one (WithResultReader) or nil, Errors is the attempt error history, and AttemptedBy the nodes that ran attempts. NextAttempt, NextAttemptAt and Progress stay zero: a recorder does not see them.

func MergeRecords

func MergeRecords(run, sched RunRecord) RunRecord

MergeRecords joins sched, a record kept under a ScheduledID, into run, the record of the same run kept under its RunID, when a write reveals both IDs. run's fields win; sched fills what run lacks, the enqueue time is the earliest, and the steps of both are kept.

func NewRecord

func NewRecord(ref workflow.RunRef) RunRecord

NewRecord is the empty record a store starts from for a run it has never seen.

func (RunRecord) Clone

func (r RunRecord) Clone() RunRecord

Clone returns a deep copy, so a store can hand a record out without sharing its state.

func (RunRecord) Key

func (r RunRecord) Key() string

Key is the record's identity within a Store: the RunID once known, else the ScheduledID. A delayed run is first recorded under its ScheduledID; the first write that carries both IDs moves it under the RunID.

type RunWrite

type RunWrite struct {
	// Ref is the run: only ScheduledID for a delayed run the enqueue seam
	// saw, both IDs when the Reconciler found it on the client.
	Ref workflow.RunRef

	Workflow string
	Queue    string
	Tenant   string
	Labels   map[string]string

	Source    workflow.Source
	SourceRef string

	// Status is the run's status when the writer saw it:
	// workflow.StatusScheduled or StatusQueued at enqueue, whatever the
	// client reported in a backfill. It sets the record's status only when
	// the record is new.
	Status workflow.Status

	// Payload is the input after redaction, nil when not kept;
	// PayloadSize is its size before that.
	Payload     []byte
	PayloadSize int

	Priority int

	// RunAt is the time a delayed run is due; zero to start now.
	RunAt time.Time

	// EnqueuedAt is when the run was accepted. The earliest known value
	// wins, because a record first seen at a step start carries that
	// start as its guess.
	EnqueuedAt time.Time

	// At is when the write was made, on the recorder's clock.
	At time.Time
}

RunWrite is a run accepted by the client (or found on it). Fields are what the writer knows; the zero value of each means "unknown".

type StepEnd

type StepEnd struct {
	Ref     workflow.RunRef
	Step    string
	Attempt int

	// Status is StepSucceeded, StepFailed or StepSnoozed.
	Status StepStatus

	// Output is the step's output after redaction (nil when not kept);
	// OutputSize is its size before that.
	Output     []byte
	OutputSize int

	// Error and Stack are the attempt's error (empty when it succeeded).
	Error string
	Stack string

	// Node ran the attempt.
	Node string

	// ErrorHistory bounds the run's error history (RunInfo.Errors) when
	// this end adds to it: the newest ErrorHistory entries are kept, zero
	// or less keeps none.
	ErrorHistory int

	// At is when the attempt ended.
	At time.Time
}

StepEnd is a step attempt ending.

type StepRecord

type StepRecord struct {
	// Step is the step name; Attempt is 1 for the first attempt. A snoozed
	// attempt that runs again keeps its number, so its record is updated
	// in place by the next try.
	Step    string
	Attempt int

	// Status is where the attempt is.
	Status StepStatus

	// Input and Output are the step's input and output after redaction;
	// nil when the recorder keeps no step bodies (WithStepBodies(false))
	// or the body was larger than MaxPayload (InputSize and OutputSize say
	// how big it was).
	Input      []byte
	Output     []byte
	InputSize  int
	OutputSize int

	// Error is the attempt's error message and Stack its stack trace
	// (workflow.StackOf), both empty unless Status is StepFailed or
	// StepSnoozed.
	Error string
	Stack string

	// Node ran the attempt (workflow.StepInfo.Node); empty on a driver
	// without CapNodeIdentity.
	Node string

	// StartedAt and FinishedAt are on the recorder's clock; FinishedAt is
	// zero while the attempt runs.
	StartedAt  time.Time
	FinishedAt time.Time

	// Events is the attempt's timeline (workflow.RecordEvent), in the
	// order recorded.
	Events []EventRecord
}

StepRecord is one step attempt.

func NewStepRecord

func NewStepRecord(s StepStart) StepRecord

NewStepRecord is the step record a StepStart creates. A store that already holds the (step, attempt) record (a snoozed attempt runs again) resets it with ResetStepRecord instead, keeping its events.

func (StepRecord) Clone

func (s StepRecord) Clone() StepRecord

Clone returns a deep copy.

func (StepRecord) Duration

func (s StepRecord) Duration() time.Duration

Duration is how long the attempt ran; zero while it runs.

type StepStart

type StepStart struct {
	// Ref is the run; a step always knows its RunID.
	Ref workflow.RunRef

	Workflow  string
	Queue     string
	Tenant    string
	Labels    map[string]string
	Source    workflow.Source
	SourceRef string

	Step    string
	Attempt int
	Node    string

	// Input is the step's input after redaction (nil when not kept);
	// InputSize is its size before that.
	Input     []byte
	InputSize int

	// At is when the attempt started.
	At time.Time
}

StepStart is a step attempt starting. It carries the run's own fields too, because the run may not have been seen enqueued (a schedule fire, a chained job, a child spawned on the driver's internal path).

type StepStatus

type StepStatus string

StepStatus is where one step attempt is.

Code that switches on it must keep a default branch: a later version may add statuses.

const (
	// StepRunning: the attempt started and has not ended (or its worker
	// died before it could say so).
	StepRunning StepStatus = "running"

	// StepSucceeded: the handler returned no error.
	StepSucceeded StepStatus = "succeeded"

	// StepFailed: the handler returned an error (the step may retry).
	StepFailed StepStatus = "failed"

	// StepSnoozed: the handler returned workflow.Snooze; the step runs
	// again later without spending an attempt.
	StepSnoozed StepStatus = "snoozed"
)

The step attempt statuses.

func StepStatuses

func StepStatuses() []StepStatus

StepStatuses returns every step status, as a fresh slice.

func (StepStatus) Valid

func (s StepStatus) Valid() bool

Valid reports whether s is one of the StepStatus constants.

type Store

type Store interface {
	// UpsertRun records that a run was accepted (the enqueue seam) or found
	// on the client (the Reconciler's backfill).
	UpsertRun(ctx context.Context, w RunWrite) error

	// StartStep records that a step attempt started.
	StartStep(ctx context.Context, s StepStart) error

	// EndStep records that a step attempt ended.
	EndStep(ctx context.Context, e StepEnd) error

	// AddEvent appends a timeline event to a step attempt.
	AddEvent(ctx context.Context, e EventWrite) error

	// SetOutcome records the outcome a step set.
	SetOutcome(ctx context.Context, o OutcomeWrite) error

	// FinishRun records that a run reached a terminal status.
	FinishRun(ctx context.Context, f Finish) error

	// MarkChecked records a Reconciler check of an unfinished run.
	MarkChecked(ctx context.Context, c Check) error

	// Runs answers workflow.RunQuerier from the store's records.
	Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)

	// Records lists records with the recorder-only filters (Query); its
	// records carry no Steps.
	Records(ctx context.Context, q Query) (RecordPage, error)

	// Run returns one record with its steps and their events.
	Run(ctx context.Context, ref workflow.RunRef) (RunRecord, error)

	// Pending returns up to limit unfinished records (status not
	// terminal), the least recently checked first (never checked before
	// any checked), for the Reconciler. They carry no Steps.
	Pending(ctx context.Context, limit int) ([]RunRecord, error)
}

Store is the contract a run store keeps. The Recorder writes through it, the Reconciler reads and writes through it, and apps read run history from it (it is a workflow.RunQuerier, so anything that lists runs, an admin view included, can read the recorder instead of the driver).

The rules every implementation keeps (recordertest.RunStore checks them):

  • Every write is an upsert keyed by the run (RunRef): delivery from the seams is at least once and in no guaranteed order across steps, so the same write twice, or a step write before the run's own, must land on one record. A write for a run the store has never seen creates its record from what the write carries.
  • A record is found by its RunID, else by its ScheduledID. A write whose Ref carries both IDs joins a record kept under the ScheduledID with the one under the RunID, so a delayed run ends as one record.
  • Merging follows the Apply* functions of this package, so every store merges a write the same way: ApplyRunWrite, ApplyStepStart, ApplyStepEnd, ApplyEvent, ApplyOutcome, ApplyFinish, ApplyCheck, MergeRecords.
  • Reads return copies; a caller can modify what it got.
  • Runs and Records order newest enqueue first, ties broken by Key descending, and page with an opaque cursor (EncodeCursor is the one the bundled stores use); they validate the query with ValidateQuery.
  • Run returns workflow.ErrNotFound for a run the store does not hold.

Methods may be called concurrently.

type WriteOp

type WriteOp string

WriteOp names the store write that failed, for an OnError hook.

const (
	OpUpsertRun  WriteOp = "upsert-run"
	OpStartStep  WriteOp = "start-step"
	OpEndStep    WriteOp = "end-step"
	OpAddEvent   WriteOp = "add-event"
	OpFinishRun  WriteOp = "finish-run"
	OpReadResult WriteOp = "read-result"
)

The writes a Recorder makes.

Directories

Path Synopsis
Package recordertest holds the two conformance suites of the recorder: RunStore holds any recorder.Store to the Store contract (the in-memory store, sqlrecorder on every database, an app's own store), and RunDriver runs a Recorder over any workflow driver through a workflowtest.Harness, so the recorder is proven on memory, SQL, Redis and Hatchet alike.
Package recordertest holds the two conformance suites of the recorder: RunStore holds any recorder.Store to the Store contract (the in-memory store, sqlrecorder on every database, an app's own store), and RunDriver runs a Recorder over any workflow driver through a workflowtest.Harness, so the recorder is proven on memory, SQL, Redis and Hatchet alike.
sqlrecorder module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL