Documentation
¶
Overview ¶
Package recorder keeps a run history that the app owns, separate from the driver's own store: one record per run, whatever started it, with its trigger source, input, every step attempt with its input, output, error and timeline events, the result, the outcome and the timings.
Why it exists: a driver's run history is the driver's store. On an engine driver it ends where the engine's retention ends, and no driver keeps a step's timeline or says who started a run. A recorder fixes that for every driver at once, because it is built only from the seams every driver already runs: enqueue middleware (a run is accepted), step middleware (an attempt starts and ends, events, outcome) and the run-finished hook (a run ends). No driver changes, and an app that never imports this module pays nothing.
Wiring it in:
store := recorder.NewMemoryStore() // or sqlrecorder.New(db, dialect) rec, err := recorder.New(store) c, err := workflow.Use(memory.New(), rec.Plugin()) // register and start through c; then read with store.Runs / store.Run
The pieces:
- Store is the contract a run store keeps: idempotent upserts keyed by RunRef, reads that answer workflow.RunQuerier plus richer Records and Run. MemoryStore is the in-process one; recorder/sqlrecorder is a SQL one in its own module; an app can bring its own and prove it with recordertest.RunStore. The Apply* functions hold the merge rules, so every store merges a write the same way.
- Recorder turns the three seams into Store writes. A failed write never fails a run: it goes to the OnError hook.
- Redactor scrubs what is stored (JSONPaths, PerWorkflow, Chain), and WithMaxPayload bounds it; a runs table is read by more people than the queue, so it is where a token in a payload leaks.
- Reconciler closes the gaps the seams leave (a worker that died before its step ended, a cancel from an engine UI, runs started from an engine's CLI): it re-checks unfinished records against the client, flags stale ones (never cancelling anything) and backfills runs it never saw with SourceExternal.
Delivery: every write is an upsert, so at-least-once delivery from any seam is safe. Run ends are guaranteed only on a driver with workflow.CapRunFinished; elsewhere the Reconciler is the fallback.
Index ¶
- Constants
- func ApplyCheck(r *RunRecord, c Check)
- func ApplyEvent(r *RunRecord, e EventWrite)
- func ApplyFinish(r *RunRecord, f Finish, latestStart time.Time) bool
- func ApplyOutcome(r *RunRecord, o OutcomeWrite)
- func ApplyRunWrite(r *RunRecord, isNew bool, w RunWrite)
- func ApplyStepEnd(r *RunRecord, e StepEnd)
- func ApplyStepEndToStep(sr *StepRecord, e StepEnd)
- func ApplyStepStart(r *RunRecord, isNewAttempt bool, s StepStart)
- func CheckRef(op string, ref workflow.RunRef) error
- func CheckStep(op string, ref workflow.RunRef, step string, attempt int) error
- func EncodeCursor(c Cursor) string
- func InfoPage(p RecordPage) workflow.RunPage
- func KeyOf(ref workflow.RunRef) string
- func PruneEvery(ctx context.Context, p Pruner, interval time.Duration, onError func(error)) error
- func ResetStepRecord(sr *StepRecord, s StepStart)
- func SortPending(rs []RunRecord)
- func SortRecords(rs []RunRecord)
- func SortSteps(steps []StepRecord)
- func StartsNewAttempt(existing *StepRecord) bool
- func ValidateQuery(q Query) error
- func ValidateRetention(r Retention) error
- type Check
- type Cursor
- type ErrorFunc
- type EventRecord
- type EventWrite
- type Finish
- type MemoryOption
- type MemoryStore
- func (s *MemoryStore) AddEvent(_ context.Context, e EventWrite) error
- func (s *MemoryStore) EndStep(_ context.Context, e StepEnd) error
- func (s *MemoryStore) FinishRun(_ context.Context, f Finish) error
- func (s *MemoryStore) MarkChecked(_ context.Context, c Check) error
- func (s *MemoryStore) Pending(_ context.Context, limit int) ([]RunRecord, error)
- func (s *MemoryStore) Prune(ctx context.Context) (PruneResult, error)
- func (s *MemoryStore) Records(_ context.Context, q Query) (RecordPage, error)
- func (s *MemoryStore) Run(_ context.Context, ref workflow.RunRef) (RunRecord, error)
- func (s *MemoryStore) Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
- func (s *MemoryStore) SetOutcome(_ context.Context, o OutcomeWrite) error
- func (s *MemoryStore) StartStep(_ context.Context, st StepStart) error
- func (s *MemoryStore) UpsertRun(_ context.Context, w RunWrite) error
- type Option
- func WithClock(c clock.Clock) Option
- func WithDefaultSource(s workflow.Source) Option
- func WithErrorHistory(n int) Option
- func WithMaxPayload(n int) Option
- func WithName(name string) Option
- func WithOnError(fn ErrorFunc) Option
- func WithRedactor(red Redactor) Option
- func WithResultReader(rr workflow.ResultReader) Option
- func WithStepBodies(keep bool) Option
- func WithWriteTimeout(d time.Duration) Option
- type OutcomeWrite
- type PayloadInfo
- type PayloadKind
- type PruneResult
- type Pruner
- type Query
- type ReconcileOption
- func WithBackfill(on bool) ReconcileOption
- func WithBackfillPages(n int) ReconcileOption
- func WithBackfillQuery(q workflow.RunQuery) ReconcileOption
- func WithBackfillWindow(d time.Duration) ReconcileOption
- func WithBatch(n int) ReconcileOption
- func WithInterval(d time.Duration) ReconcileOption
- func WithReconcileClock(c clock.Clock) ReconcileOption
- func WithReconcileOnError(fn func(ctx context.Context, err error)) ReconcileOption
- func WithStaleAfter(d time.Duration) ReconcileOption
- type ReconcileReport
- type Reconciler
- type RecordPage
- type Recorder
- func (r *Recorder) EnqueueMiddleware() workflow.EnqueueMiddleware
- func (r *Recorder) EnqueuePlugin() workflow.Plugin
- func (r *Recorder) OnRunFinished(ctx context.Context, ev workflow.RunEvent)
- func (r *Recorder) Plugin() workflow.Plugin
- func (r *Recorder) StepMiddleware() workflow.Middleware
- func (r *Recorder) Store() Store
- type Redactor
- type Retention
- type RetentionRule
- type RunRecord
- type RunWrite
- type StepEnd
- type StepRecord
- type StepStart
- type StepStatus
- type Store
- type WriteOp
Constants ¶
const ( // DefaultReconcileInterval is the pause between two passes of Run. DefaultReconcileInterval = 30 * time.Second // DefaultStaleAfter is how long an unfinished run may go without any // recorded activity before it is flagged stale. DefaultStaleAfter = 10 * time.Minute // DefaultBackfillWindow is how far back each pass lists the client's // runs looking for ones the recorder never saw. DefaultBackfillWindow = time.Hour // DefaultReconcileBatch is how many unfinished records one pass checks, // and the page size of the backfill listing. DefaultReconcileBatch = 100 // DefaultBackfillPages bounds the pages one backfill pass reads, so a // huge window cannot make one pass unbounded. DefaultBackfillPages = 10 )
Defaults of a Reconciler.
const ( // DefaultMaxPayload is the largest body (input, step input and // output, result) a Recorder stores; a larger one is recorded by size // only. DefaultMaxPayload = 64 << 10 // DefaultWriteTimeout bounds one store write, so a slow store delays a // step by at most this much. DefaultWriteTimeout = 5 * time.Second // DefaultPluginName is the name Plugin reports to workflow.Use. DefaultPluginName = "recorder" )
Defaults of a Recorder.
const DefaultPruneBatch = 500
DefaultPruneBatch is Retention.Batch when zero.
const MaxPruneBatch = 10000
MaxPruneBatch bounds Retention.Batch.
const MaxSearchLen = 256
MaxSearchLen is the longest Query.Search, in bytes.
const QueryCaps = workflow.CapRunQuery | workflow.CapLabelPrefix | workflow.CapTenant | workflow.CapDueQuery
QueryCaps is what a Store's Runs and Records honour of a workflow.RunQuery: label prefixes, tenant and due-time filters on top of the basic ones. Stores pass it to workflow.CheckRunQuery (through ValidateQuery), and a caller that wants to know before it asks, an admin view, can read it.
const Redacted = "[REDACTED]"
Redacted is the value a JSONPaths redactor writes in place of a sensitive value (as a JSON string in a body, as is in an event attribute).
Variables ¶
This section is empty.
Functions ¶
func ApplyCheck ¶
ApplyCheck records a Reconciler check. It moves CheckedAt and the Stale flag, and the status when the client reported a different unfinished one; it never touches UpdatedAt, which measures the run's own activity. A record that became terminal meanwhile keeps its status.
func ApplyEvent ¶
func ApplyEvent(r *RunRecord, e EventWrite)
ApplyEvent merges the run-level part of an event: activity.
func ApplyFinish ¶
ApplyFinish merges f into r and reports whether it did. A finish is ignored when an attempt started after it (latestStart, the newest StartedAt among the run's step records): it is a late duplicate of an end the run has since been retried out of. Otherwise the status, error, end time and result are f's (a repeated finish rewrites the same values), and the outcome defaults from the status unless a step set one.
func ApplyOutcome ¶
func ApplyOutcome(r *RunRecord, o OutcomeWrite)
ApplyOutcome records a set outcome; the last one set wins.
func ApplyRunWrite ¶
ApplyRunWrite merges w into r. isNew says r came from NewRecord, so its status is set from w.
func ApplyStepEnd ¶
ApplyStepEnd merges the run-level part of e into r: a failed attempt joins the error history (bounded by e.ErrorHistory) and, while the run is unfinished, becomes its current Error, as RunInfo.Error documents for a run waiting on its retry. A snooze is not a failure and adds nothing.
func ApplyStepEndToStep ¶
func ApplyStepEndToStep(sr *StepRecord, e StepEnd)
ApplyStepEndToStep applies e to the attempt's own record. An end for an attempt the store never saw start creates it from e (sr is then the zero StepRecord with Step and Attempt set by the store).
func ApplyStepStart ¶
ApplyStepStart merges the run-level part of s into r: the run is running (a terminal run re-opens, because only a retry starts an attempt after the end), its first start and attempt count move, and the node joins AttemptedBy. The step record itself is NewStepRecord(s). isNewAttempt says the store had no record of this (step, attempt) yet, so a repeated start (at-least-once, or a snoozed attempt's next try) does not count twice.
func CheckRef ¶
CheckRef refuses, with workflow.ErrInvalid, a write that names no run. The bundled stores call it first; an app's own store should too (recordertest.RunStore expects the refusal).
func CheckStep ¶
CheckStep refuses, with workflow.ErrInvalid, a step write without a RunID, a step name or an attempt from 1.
func EncodeCursor ¶
EncodeCursor makes the opaque cursor string the bundled stores hand out as a page's Next.
func InfoPage ¶
func InfoPage(p RecordPage) workflow.RunPage
InfoPage converts a page of records to the RunPage a RunQuerier returns.
func KeyOf ¶
KeyOf is the Store identity of ref (see RunRecord.Key). The prefixes keep a RunID and a ScheduledID with the same text apart.
func PruneEvery ¶
PruneEvery calls p.Prune every interval until ctx is cancelled, handing each error to onError (nil ignores them), and returns nil when ctx ends. It is the loop an app runs in a goroutine next to its client; a store's Prune is also safe to call from a scheduled job instead.
func ResetStepRecord ¶
func ResetStepRecord(sr *StepRecord, s StepStart)
ResetStepRecord applies a repeated StepStart to an existing attempt record: running again, its end cleared, its events kept.
func SortPending ¶
func SortPending(rs []RunRecord)
SortPending orders records the way Pending returns them: never checked first, then least recently checked, ties by Key.
func SortRecords ¶
func SortRecords(rs []RunRecord)
SortRecords orders records the way Runs and Records list them: newest enqueue first, ties broken by Key descending.
func SortSteps ¶
func SortSteps(steps []StepRecord)
SortSteps orders step records by start time, then step name, then attempt, the order RunRecord.Steps documents.
func StartsNewAttempt ¶
func StartsNewAttempt(existing *StepRecord) bool
StartsNewAttempt reports whether a StepStart counts as a new attempt, given the store's record of that (step, attempt), nil when it has none. A record with a zero StartedAt is a placeholder an event created when it raced ahead of its attempt's start, so the start still counts; a record that started before is a repeat (at-least-once, or a snoozed attempt's next try) and does not.
func ValidateQuery ¶
ValidateQuery checks q: workflow.ValidateRunQuery and workflow.CheckRunQuery against QueryCaps for the embedded query, known sources and outcomes, and Search within MaxSearchLen. Errors wrap workflow.ErrInvalid or workflow.ErrUnsupported.
func ValidateRetention ¶
ValidateRetention refuses negative ages and a Batch outside [0, MaxPruneBatch]. Errors wrap workflow.ErrInvalid.
Types ¶
type Check ¶
type Check struct {
Ref workflow.RunRef
// Status is the non-terminal status the client reported, or "" to
// leave the record's status (the client did not know the run).
Status workflow.Status
// Stale sets or clears the record's Stale flag.
Stale bool
// At is when the check was made; it becomes the record's CheckedAt.
At time.Time
}
Check is the Reconciler's verdict on an unfinished record.
type Cursor ¶
Cursor is the position after one record in the listing order (newest enqueue first, Key descending): the next page starts after it.
func DecodeCursor ¶
DecodeCursor reads a cursor EncodeCursor made. Anything else is an error wrapping workflow.ErrInvalid, as RunQuerier documents for a cursor the store did not issue.
type ErrorFunc ¶
ErrorFunc is told about a failed write. The run goes on regardless: a recorder never fails a run. It runs on the goroutine that made the write (a step's, an enqueuer's, the driver's run-finished goroutine), so it must return quickly.
type EventRecord ¶
type EventRecord struct {
// Seq orders the events of one attempt; the store assigns it (1, 2,
// ...).
Seq int
// Message, Level and Attrs are workflow.StepEvent's, after redaction.
Message string
Level slog.Level
Attrs []workflow.EventAttr
// At is when it was recorded, on the recorder's clock.
At time.Time
}
EventRecord is one stored step event.
func NewEventRecord ¶
func NewEventRecord(e EventWrite, seq int) EventRecord
NewEventRecord is the stored form of e; the store sets Seq to one more than the attempt's last event.
type EventWrite ¶
type EventWrite struct {
Ref workflow.RunRef
Step string
Attempt int
Message string
Level slog.Level
Attrs []workflow.EventAttr
// At is when the event was recorded.
At time.Time
}
EventWrite is one step event to append.
type Finish ¶
type Finish struct {
// Ref carries both IDs when the run was delayed and the driver knows
// both (workflow.RunEvent.Ref does).
Ref workflow.RunRef
Workflow string
Tenant string
Labels map[string]string
Source workflow.Source
SourceRef string
// Status is terminal.
Status workflow.Status
// Error is why the run failed or was cancelled; FailedStep the step
// whose failure ended it.
Error string
FailedStep string
// Output is the run's result after redaction when the writer read it,
// else nil; OutputSize is its size before that.
Output []byte
OutputSize int
// FinishedAt is when the run ended, on the driver's clock.
FinishedAt time.Time
// At is when the write was made.
At time.Time
}
Finish is a run reaching a terminal status (the run-finished seam, or the Reconciler finding the client's answer).
type MemoryOption ¶
type MemoryOption func(*MemoryStore)
MemoryOption configures a MemoryStore.
func WithMemoryClock ¶
func WithMemoryClock(c clock.Clock) MemoryOption
WithMemoryClock sets the clock Prune measures ages with; clock.Real by default.
func WithMemoryRetention ¶
func WithMemoryRetention(r Retention) MemoryOption
WithMemoryRetention sets the retention Prune applies (see Retention); none by default, so Prune does nothing.
type MemoryStore ¶
type MemoryStore struct {
// contains filtered or unexported fields
}
MemoryStore is the in-process Store: records live in a map guarded by a mutex, so it is for tests, single-process apps and as the reference the store conformance suite (recordertest.RunStore) is written against. It keeps nothing across a restart. It implements Pruner.
func NewMemoryStore ¶
func NewMemoryStore(opts ...MemoryOption) (*MemoryStore, error)
NewMemoryStore returns an empty MemoryStore. It fails only on an invalid retention (ValidateRetention).
func (*MemoryStore) AddEvent ¶
func (s *MemoryStore) AddEvent(_ context.Context, e EventWrite) error
AddEvent implements Store.
func (*MemoryStore) EndStep ¶
func (s *MemoryStore) EndStep(_ context.Context, e StepEnd) error
EndStep implements Store.
func (*MemoryStore) FinishRun ¶
func (s *MemoryStore) FinishRun(_ context.Context, f Finish) error
FinishRun implements Store.
func (*MemoryStore) MarkChecked ¶
func (s *MemoryStore) MarkChecked(_ context.Context, c Check) error
MarkChecked implements Store. A check of a run the store does not hold does nothing: the Reconciler only checks records it read.
func (*MemoryStore) Prune ¶
func (s *MemoryStore) Prune(ctx context.Context) (PruneResult, error)
Prune implements Pruner: stage one drops the steps of runs finished longer ago than the rule's Steps age, stage two archives and deletes runs finished longer ago than its Runs age.
func (*MemoryStore) Records ¶
func (s *MemoryStore) Records(_ context.Context, q Query) (RecordPage, error)
Records implements Store.
func (*MemoryStore) SetOutcome ¶
func (s *MemoryStore) SetOutcome(_ context.Context, o OutcomeWrite) error
SetOutcome implements Store.
type Option ¶
Option configures a Recorder.
func WithClock ¶
WithClock sets the clock the recorder stamps writes with; clock.Real by default. Use the driver's clock so records and the driver agree.
func WithDefaultSource ¶
WithDefaultSource is the source recorded for a run whose labels say nothing (workflow.SourceOf returns ""). Empty by default, which records no source. Must pass workflow.ValidateSource when set.
func WithErrorHistory ¶
WithErrorHistory bounds the attempt errors kept per run (workflow.DefaultErrorHistory by default); zero keeps none.
func WithMaxPayload ¶
WithMaxPayload sets the largest body stored, in bytes (DefaultMaxPayload by default). A larger body is recorded by its size only. Zero stores no bodies at all, only sizes, for apps whose payloads must never be copied; a negative value is invalid.
func WithName ¶
WithName sets the plugin name (DefaultPluginName by default), so two recorders, say one per store, can be applied to one client.
func WithOnError ¶
WithOnError is told about every failed write. By default a failure is logged with slog.Default at warn level.
func WithRedactor ¶
WithRedactor scrubs every body and event attribute before it is stored (see Redactor, JSONPaths). None by default.
func WithResultReader ¶
func WithResultReader(rr workflow.ResultReader) Option
WithResultReader reads a succeeded run's result from rr (usually the client, on a driver with workflow.CapResult) when the run ends, so the record holds it. Without it the record holds the step outputs only.
func WithStepBodies ¶
WithStepBodies says whether step inputs and outputs are stored (true by default). The run's own input and result follow WithMaxPayload only.
func WithWriteTimeout ¶
WithWriteTimeout bounds each store write (DefaultWriteTimeout by default). Writes run on a context detached from the caller's cancellation, so a step whose ctx was cancelled still records its end.
type OutcomeWrite ¶
OutcomeWrite is the outcome a step set.
type PayloadInfo ¶
type PayloadInfo struct {
Kind PayloadKind
Workflow string
// Step is empty for PayloadInput and PayloadResult.
Step string
}
PayloadInfo says what is about to be stored, so one Redactor can treat workflows, steps and kinds differently.
type PayloadKind ¶
type PayloadKind string
PayloadKind is which body or field a Redactor is asked about.
const ( // PayloadInput is the run's input (Job.Payload) at enqueue. PayloadInput PayloadKind = "input" // PayloadStepInput is a step attempt's input. PayloadStepInput PayloadKind = "step-input" // PayloadStepOutput is a step attempt's output. PayloadStepOutput PayloadKind = "step-output" // PayloadResult is the run's result (WithResultReader). PayloadResult PayloadKind = "result" // PayloadEvent is a step event's attribute. PayloadEvent PayloadKind = "event" )
The kinds of data a Recorder stores and so redacts.
type PruneResult ¶
type PruneResult struct {
// StepsCleared counts runs whose steps stage one dropped.
StepsCleared int
// RunsDeleted counts runs stage two deleted.
RunsDeleted int
// Archived counts runs handed to Archive (equal to RunsDeleted when
// Archive is set).
Archived int
}
PruneResult is what one Prune did.
type Pruner ¶
type Pruner interface {
Prune(ctx context.Context) (PruneResult, error)
}
Pruner is the optional retention of a Store: MemoryStore and sqlrecorder.Store implement it, configured with a Retention.
type Query ¶
type Query struct {
workflow.RunQuery
// Sources matches runs whose Source is any of these; empty matches
// any.
Sources []workflow.Source
// Outcomes matches runs whose Outcome is any of these; empty matches
// any. An unfinished run with no outcome set matches none.
Outcomes []workflow.Outcome
// Search matches runs where it is a case-insensitive substring of the
// RunID, the ScheduledID, the workflow name or the SourceRef; empty
// matches any. It is a lookup box for an operator, not a full-text
// index: a store may scan to answer it.
Search string
// OnlyStale matches only runs the Reconciler flagged stale.
OnlyStale bool
}
Query is a workflow.RunQuery plus the filters only a recorder can answer, because only it knows a run's source and outcome.
type ReconcileOption ¶
type ReconcileOption func(*Reconciler) error
ReconcileOption configures a Reconciler.
func WithBackfill ¶
func WithBackfill(on bool) ReconcileOption
WithBackfill turns the backfill on or off (on by default).
func WithBackfillPages ¶
func WithBackfillPages(n int) ReconcileOption
WithBackfillPages bounds the pages one backfill pass reads.
func WithBackfillQuery ¶
func WithBackfillQuery(q workflow.RunQuery) ReconcileOption
WithBackfillQuery scopes the backfill listing to the runs q matches (a workflow, a tenant, labels), for a client whose engine other apps share. Its From, Limit and Cursor are the Reconciler's own and are ignored; the rest must pass workflow.ValidateRunQuery.
func WithBackfillWindow ¶
func WithBackfillWindow(d time.Duration) ReconcileOption
WithBackfillWindow sets how far back the backfill lists runs.
func WithBatch ¶
func WithBatch(n int) ReconcileOption
WithBatch sets how many records one pass checks and the backfill page size, from 1 to workflow.MaxRunsLimit.
func WithInterval ¶
func WithInterval(d time.Duration) ReconcileOption
WithInterval sets the pause between passes of Run.
func WithReconcileClock ¶
func WithReconcileClock(c clock.Clock) ReconcileOption
WithReconcileClock sets the clock the Reconciler measures with (clock.Real by default); use the recorder's.
func WithReconcileOnError ¶
func WithReconcileOnError(fn func(ctx context.Context, err error)) ReconcileOption
WithReconcileOnError is told about every error of a pass that did not stop it (one run's check or write); Run also hands it a pass's error. By default errors are dropped by Pass and logged by Run with slog.Default.
func WithStaleAfter ¶
func WithStaleAfter(d time.Duration) ReconcileOption
WithStaleAfter sets how long an unfinished run may be silent before it is flagged stale.
type ReconcileReport ¶
type ReconcileReport struct {
// Checked counts unfinished records compared with the client.
Checked int
// Finished counts records the pass finished because the client
// reported them terminal.
Finished int
// Stale counts records flagged stale by this pass.
Stale int
// Missing counts records the client did not know (also flagged stale).
Missing int
// Backfilled counts runs the backfill upserted (new or filled).
Backfilled int
// BackfillSkipped is true when the client is not a RunQuerier, so no
// backfill ran.
BackfillSkipped bool
// Errors counts per-run errors that did not stop the pass.
Errors int
}
ReconcileReport is what one Pass did.
type Reconciler ¶
type Reconciler struct {
// contains filtered or unexported fields
}
Reconciler closes the gaps the recorder's seams leave:
- Check: every unfinished record (oldest check first, a batch per pass) is compared with the client's Status. A run the client reports terminal is finished on the record, with its error and end time when the client is a workflow.RunQuerier. A run the client still reports unfinished takes the client's status, and is flagged Stale when nothing was recorded for it for StaleAfter. A run the client does not know (pruned, or a lost delayed start) is flagged Stale. Nothing is ever cancelled: Stale is for an operator to look at.
- Backfill: when the client is a workflow.RunQuerier, the runs it lists enqueued within the backfill window are upserted, so a run started past the recorder (an engine UI or CLI, a crash between accept and record, a workflow registered on the bare driver) gets a record, with SourceExternal when its labels name no source. Upserts only fill, so a run the recorder already holds is left as recorded (its ScheduledID and RunID are joined when the listing shows both).
A Reconciler works on any client; backfill and error details need the client to be a workflow.RunQuerier and are skipped otherwise (ReconcileReport.BackfillSkipped). Run several Reconcilers over one store safely: every write is an upsert.
func NewReconciler ¶
func NewReconciler(c workflow.Client, s Store, opts ...ReconcileOption) (*Reconciler, error)
NewReconciler returns a Reconciler of s against c. It fails on a nil client or store, or an invalid option, with workflow.ErrInvalid.
func (*Reconciler) Pass ¶
func (r *Reconciler) Pass(ctx context.Context) (ReconcileReport, error)
Pass runs one check and one backfill. It returns an error only when it could not read the store's pending records or the client's run listing; a single run's failure is counted, handed to the error hook, and the pass goes on.
type RecordPage ¶
RecordPage is one page of Records.
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder turns the three seams every driver runs (enqueue middleware, step middleware, run-finished observers) into Store writes. Build one with New and apply it with workflow.Use(client, rec.Plugin()); a Recorder is safe for concurrent use and keeps no state of its own besides its configuration, so one Recorder may serve several clients.
func New ¶
New returns a Recorder writing to s. It fails on a nil store or an invalid option, with an error wrapping workflow.ErrInvalid.
func (*Recorder) EnqueueMiddleware ¶
func (r *Recorder) EnqueueMiddleware() workflow.EnqueueMiddleware
EnqueueMiddleware records every run the client accepts. A delayed run is recorded under its ScheduledID with StatusScheduled; others under their RunID with StatusQueued. A failed enqueue records nothing, and a failed write never fails the enqueue. Chained jobs are recorded when their first step starts, since the driver enqueues them past the middleware.
func (*Recorder) EnqueuePlugin ¶
EnqueuePlugin returns a workflow.Plugin with the recorder's enqueue middleware only, for a client that enqueues but runs nothing: a web or API node built in a driver's enqueue-only mode (workflow.CapEnqueueOnly), which has no run-finished hook for Plugin to register on. It records each accepted run with its source and input; the worker processes, wrapped with Plugin over the same store, record the steps and the end into the same record (every write is an upsert keyed by the run).
func (*Recorder) OnRunFinished ¶
OnRunFinished records a run's end; it is the run-finished observer the plugin registers, exported for an app that wires the seams by hand. Delivery is at least once and the write is an upsert, so a repeat is harmless. With WithResultReader it reads a succeeded run's result first.
func (*Recorder) Plugin ¶
Plugin returns the workflow.Plugin that applies the recorder to a client: workflow.Use(client, rec.Plugin()). It contributes one enqueue middleware, one step middleware and one run-finished observer, so the client must implement workflow.RunNotifier (every bundled driver does, except in enqueue-only mode: use EnqueuePlugin there).
func (*Recorder) StepMiddleware ¶
func (r *Recorder) StepMiddleware() workflow.Middleware
StepMiddleware records each step attempt's start and end, and binds on the step's ctx a workflow.StepEventSink (workflow.RecordEvent lands on the attempt) and a workflow.OutcomeSetter (workflow.SetOutcome lands on the run). A ctx without workflow.StepInfo (a handler called outside a driver) passes through unrecorded. The handler's output and error are returned unchanged.
type Redactor ¶
type Redactor interface {
// RedactPayload returns body as it may be stored.
RedactPayload(info PayloadInfo, body []byte) []byte
// RedactAttr returns a as it may be stored.
RedactAttr(info PayloadInfo, a workflow.EventAttr) workflow.EventAttr
}
Redactor scrubs data before a Recorder stores it. It runs on the step's goroutine for every body and event, so it must be quick and safe for concurrent use, and must not modify its input slice (return a new one).
Event messages are not redacted: a message is the step author's own text. Put values in attributes, which are.
func Chain ¶
Chain returns a Redactor that applies rs in order, each to what the one before returned; nil entries are skipped.
func JSONPaths ¶
JSONPaths returns a Redactor that replaces the values at the given dot paths of a JSON body with Redacted, and the value of an event attribute whose key matches a path. A segment "*" matches any key of an object or any element of an array; a number segment matches that array index.
recorder.JSONPaths("password", "card.number", "users.*.token")
A body that is not valid JSON is stored unchanged (the redactor cannot find a path in it); an app with non-JSON payloads writes its own Redactor, or sets WithMaxPayload(0) to keep no bodies at all. A body with nothing to redact is stored byte for byte; one with a match is re-encoded, so its object keys come out sorted. An empty path or segment is an error wrapping workflow.ErrInvalid.
type Retention ¶
type Retention struct {
RetentionRule
// PerWorkflow overrides RetentionRule for the named workflows.
PerWorkflow map[string]RetentionRule
// Archive, when set, is handed every record before stage two deletes
// it (with its steps, when stage one left any). An error stops the
// prune and deletes nothing of that batch, so nothing is lost when an
// archive is down; the next prune tries again.
Archive func(ctx context.Context, runs []RunRecord) error
// Batch is how many runs one prune step handles at a time, so a big
// backlog never holds one long transaction. Zero means
// DefaultPruneBatch.
Batch int
}
Retention is how long a store keeps finished runs, in two stages, so the heavy part of a run goes first and the summary row a runs page lists stays longer:
- Stage one, after Steps: the run's step records and their events are dropped; the run record (status, source, outcome, error, timings, input and result) stays.
- Stage two, after Runs: the run record itself is deleted, after Archive (when set) accepted it.
Ages are measured from the run's FinishedAt; unfinished runs are never pruned. A zero duration keeps that stage forever. PerWorkflow overrides both ages for a workflow name.
func (Retention) Rule ¶
func (r Retention) Rule(wf string) RetentionRule
Rule returns the rule for workflow wf: its PerWorkflow entry, else the default.
type RetentionRule ¶
type RetentionRule struct {
// Steps is the age after which a finished run's steps and events are
// dropped; zero keeps them.
Steps time.Duration
// Runs is the age after which a finished run is deleted; zero keeps
// it.
Runs time.Duration
}
RetentionRule is the two ages of a Retention.
type RunRecord ¶
type RunRecord struct {
workflow.RunInfo
// Source is what started the run (workflow.SourceOf of its labels, or
// the recorder's default source, or SourceExternal for a backfilled
// run); SourceRef is the reference within it.
Source workflow.Source
SourceRef string
// Outcome is the last outcome a step set (workflow.SetOutcome), or
// workflow.DefaultOutcome of the terminal status when none was set;
// "" while the run is unfinished and nothing was set. OutcomeNote is
// the note that came with a set outcome.
Outcome workflow.Outcome
OutcomeNote string
// OutcomeSet is true when a step set the outcome, so a later default
// never overwrites it.
OutcomeSet bool
// PayloadSize and OutputSize are the sizes of the input and result as
// the recorder saw them; larger than len(Payload) or len(Output) when
// the body was not kept (WithMaxPayload).
PayloadSize int
OutputSize int
// Stale is set by the Reconciler when the client still reports the run
// unfinished (or does not know it) and nothing was recorded for it for
// longer than its StaleAfter. It is a flag for an operator; nothing is
// cancelled. Any later write for the run clears it.
Stale bool
// UpdatedAt is the time of the last write that changed the run's life
// (enqueue, step start or end, event, outcome, finish), on the
// recorder's clock. The Reconciler measures staleness from it.
UpdatedAt time.Time
// CheckedAt is when the Reconciler last compared the record with the
// client; zero when it never did. It orders Pending so every
// unfinished record gets its turn.
CheckedAt time.Time
// Steps is every step attempt, ordered by start time then step name
// then attempt, each with its events. Store.Run fills it; listings
// (Records, Runs) leave it nil to stay cheap.
Steps []StepRecord
}
RunRecord is one run as a Store keeps it: the RunInfo a RunQuerier answers with, plus what only a recorder knows.
The embedded RunInfo is filled from the recorder's own writes, never from the driver: Payload is the run's input as enqueued (after redaction, and nil when it was larger than the recorder's MaxPayload; PayloadSize says how big it was), Output is the run's result when the recorder could read one (WithResultReader) or nil, Errors is the attempt error history, and AttemptedBy the nodes that ran attempts. NextAttempt, NextAttemptAt and Progress stay zero: a recorder does not see them.
func MergeRecords ¶
MergeRecords joins sched, a record kept under a ScheduledID, into run, the record of the same run kept under its RunID, when a write reveals both IDs. run's fields win; sched fills what run lacks, the enqueue time is the earliest, and the steps of both are kept.
type RunWrite ¶
type RunWrite struct {
// Ref is the run: only ScheduledID for a delayed run the enqueue seam
// saw, both IDs when the Reconciler found it on the client.
Ref workflow.RunRef
Workflow string
Queue string
Tenant string
Labels map[string]string
Source workflow.Source
SourceRef string
// Status is the run's status when the writer saw it:
// workflow.StatusScheduled or StatusQueued at enqueue, whatever the
// client reported in a backfill. It sets the record's status only when
// the record is new.
Status workflow.Status
// Payload is the input after redaction, nil when not kept;
// PayloadSize is its size before that.
Payload []byte
PayloadSize int
Priority int
// RunAt is the time a delayed run is due; zero to start now.
RunAt time.Time
// EnqueuedAt is when the run was accepted. The earliest known value
// wins, because a record first seen at a step start carries that
// start as its guess.
EnqueuedAt time.Time
// At is when the write was made, on the recorder's clock.
At time.Time
}
RunWrite is a run accepted by the client (or found on it). Fields are what the writer knows; the zero value of each means "unknown".
type StepEnd ¶
type StepEnd struct {
Ref workflow.RunRef
Step string
Attempt int
// Status is StepSucceeded, StepFailed or StepSnoozed.
Status StepStatus
// Output is the step's output after redaction (nil when not kept);
// OutputSize is its size before that.
Output []byte
OutputSize int
// Error and Stack are the attempt's error (empty when it succeeded).
Error string
Stack string
// Node ran the attempt.
Node string
// ErrorHistory bounds the run's error history (RunInfo.Errors) when
// this end adds to it: the newest ErrorHistory entries are kept, zero
// or less keeps none.
ErrorHistory int
// At is when the attempt ended.
At time.Time
}
StepEnd is a step attempt ending.
type StepRecord ¶
type StepRecord struct {
// Step is the step name; Attempt is 1 for the first attempt. A snoozed
// attempt that runs again keeps its number, so its record is updated
// in place by the next try.
Step string
Attempt int
// Status is where the attempt is.
Status StepStatus
// Input and Output are the step's input and output after redaction;
// nil when the recorder keeps no step bodies (WithStepBodies(false))
// or the body was larger than MaxPayload (InputSize and OutputSize say
// how big it was).
Input []byte
Output []byte
InputSize int
OutputSize int
// Error is the attempt's error message and Stack its stack trace
// (workflow.StackOf), both empty unless Status is StepFailed or
// StepSnoozed.
Error string
Stack string
// Node ran the attempt (workflow.StepInfo.Node); empty on a driver
// without CapNodeIdentity.
Node string
// StartedAt and FinishedAt are on the recorder's clock; FinishedAt is
// zero while the attempt runs.
StartedAt time.Time
FinishedAt time.Time
// Events is the attempt's timeline (workflow.RecordEvent), in the
// order recorded.
Events []EventRecord
}
StepRecord is one step attempt.
func NewStepRecord ¶
func NewStepRecord(s StepStart) StepRecord
NewStepRecord is the step record a StepStart creates. A store that already holds the (step, attempt) record (a snoozed attempt runs again) resets it with ResetStepRecord instead, keeping its events.
func (StepRecord) Duration ¶
func (s StepRecord) Duration() time.Duration
Duration is how long the attempt ran; zero while it runs.
type StepStart ¶
type StepStart struct {
// Ref is the run; a step always knows its RunID.
Ref workflow.RunRef
Workflow string
Queue string
Tenant string
Labels map[string]string
Source workflow.Source
SourceRef string
Step string
Attempt int
Node string
// Input is the step's input after redaction (nil when not kept);
// InputSize is its size before that.
Input []byte
InputSize int
// At is when the attempt started.
At time.Time
}
StepStart is a step attempt starting. It carries the run's own fields too, because the run may not have been seen enqueued (a schedule fire, a chained job, a child spawned on the driver's internal path).
type StepStatus ¶
type StepStatus string
StepStatus is where one step attempt is.
Code that switches on it must keep a default branch: a later version may add statuses.
const ( // StepRunning: the attempt started and has not ended (or its worker // died before it could say so). StepRunning StepStatus = "running" // StepSucceeded: the handler returned no error. StepSucceeded StepStatus = "succeeded" // StepFailed: the handler returned an error (the step may retry). StepFailed StepStatus = "failed" // StepSnoozed: the handler returned workflow.Snooze; the step runs // again later without spending an attempt. StepSnoozed StepStatus = "snoozed" )
The step attempt statuses.
func StepStatuses ¶
func StepStatuses() []StepStatus
StepStatuses returns every step status, as a fresh slice.
func (StepStatus) Valid ¶
func (s StepStatus) Valid() bool
Valid reports whether s is one of the StepStatus constants.
type Store ¶
type Store interface {
// UpsertRun records that a run was accepted (the enqueue seam) or found
// on the client (the Reconciler's backfill).
UpsertRun(ctx context.Context, w RunWrite) error
// StartStep records that a step attempt started.
StartStep(ctx context.Context, s StepStart) error
// EndStep records that a step attempt ended.
EndStep(ctx context.Context, e StepEnd) error
// AddEvent appends a timeline event to a step attempt.
AddEvent(ctx context.Context, e EventWrite) error
// SetOutcome records the outcome a step set.
SetOutcome(ctx context.Context, o OutcomeWrite) error
// FinishRun records that a run reached a terminal status.
FinishRun(ctx context.Context, f Finish) error
// MarkChecked records a Reconciler check of an unfinished run.
MarkChecked(ctx context.Context, c Check) error
// Runs answers workflow.RunQuerier from the store's records.
Runs(ctx context.Context, q workflow.RunQuery) (workflow.RunPage, error)
// Records lists records with the recorder-only filters (Query); its
// records carry no Steps.
Records(ctx context.Context, q Query) (RecordPage, error)
// Run returns one record with its steps and their events.
Run(ctx context.Context, ref workflow.RunRef) (RunRecord, error)
// Pending returns up to limit unfinished records (status not
// terminal), the least recently checked first (never checked before
// any checked), for the Reconciler. They carry no Steps.
Pending(ctx context.Context, limit int) ([]RunRecord, error)
}
Store is the contract a run store keeps. The Recorder writes through it, the Reconciler reads and writes through it, and apps read run history from it (it is a workflow.RunQuerier, so anything that lists runs, an admin view included, can read the recorder instead of the driver).
The rules every implementation keeps (recordertest.RunStore checks them):
- Every write is an upsert keyed by the run (RunRef): delivery from the seams is at least once and in no guaranteed order across steps, so the same write twice, or a step write before the run's own, must land on one record. A write for a run the store has never seen creates its record from what the write carries.
- A record is found by its RunID, else by its ScheduledID. A write whose Ref carries both IDs joins a record kept under the ScheduledID with the one under the RunID, so a delayed run ends as one record.
- Merging follows the Apply* functions of this package, so every store merges a write the same way: ApplyRunWrite, ApplyStepStart, ApplyStepEnd, ApplyEvent, ApplyOutcome, ApplyFinish, ApplyCheck, MergeRecords.
- Reads return copies; a caller can modify what it got.
- Runs and Records order newest enqueue first, ties broken by Key descending, and page with an opaque cursor (EncodeCursor is the one the bundled stores use); they validate the query with ValidateQuery.
- Run returns workflow.ErrNotFound for a run the store does not hold.
Methods may be called concurrently.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package recordertest holds the two conformance suites of the recorder: RunStore holds any recorder.Store to the Store contract (the in-memory store, sqlrecorder on every database, an app's own store), and RunDriver runs a Recorder over any workflow driver through a workflowtest.Harness, so the recorder is proven on memory, SQL, Redis and Hatchet alike.
|
Package recordertest holds the two conformance suites of the recorder: RunStore holds any recorder.Store to the Store contract (the in-memory store, sqlrecorder on every database, an app's own store), and RunDriver runs a Recorder over any workflow driver through a workflowtest.Harness, so the recorder is proven on memory, SQL, Redis and Hatchet alike. |
|
sqlrecorder
module
|