flowtest

package
v19.4.0-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 21, 2026 License: MIT Imports: 25 Imported by: 0

README

flowtest

Generative, invariant-checked testing for AutoFlow and autocore. flowtest generates workflows, drives them the way GitLab Rails would, records everything it does in a ledger, and afterwards verifies that the engine kept its promises.

The concept

Every test is a Unit: pure data describing one workflow. How to start it, which channel sends and cancels to perform at which offsets (the drive script), and which outcomes are legal. Sources produce Units and the Driver executes each Unit as a Session: start the workflow, run the drive script, record every intent and outcome in the ledger before acting. The oracle then judges the run from the ledger and the engine's own tables: did every started workflow reach an allowed terminal state, did autocore's documented invariants hold, did the run exercise anything at all.

Generated Units are built outcome-first: the generator derives the script, the drive script, and the expectation from one plan, so every generated flow is self-validating by construction.

Glossary

Term Meaning
Unit One self-contained test definition; pure data, built via the builder.
Source Produces Units: flowgen (generated AutoFlow), corpus (pre-defined AutoFlow), scenarios (pre-defined Go workflows), adversarial (sends the engine must refuse). Mixed by config weight.
Driver Executes Units at a paced rate and plays Rails: start, sends, cancel.
Session One execution of a Unit; what the ledger records and the oracle verifies.
Ledger The driver's own database: expectations, per-op intent/outcome, observed terminal state. An op with no outcome is indeterminate, never guessed.
Oracle The invariant registry plus verify; renders the generated catalog in doc/flowtest-invariants.md.
Scenario A Go workflow shape in scenarios/ with generators for its input, drive script, and expectation. Covers the Go-only surface (panics, replay mismatch, inline activities).

Tiers

Tier Where What
T1 internal/it/flowtest The whole loop in containers on every MR, minutes.
T2 CI loadtest env Bounded run, light chaos, verify after the drain.
T3 northstar Continuous, full chaos, periodic verification.

Same code at every tier. A generated Unit is a pure function of its recorded seed, generator version and generator config, so a red T2/T3 session re-renders locally, script, drive script and expectation alike: go run ./scripts/loadtests/autoflow/cmd --mode=render --ledger-dsn <dsn> --render-session <uuid>.

Adding a test for your feature

Reachable from AutoFlow? Add a corpus flow: write the .star file, register it in corpus/ with the builder, and let the builder validate the drive script against the declared channel args:

import pkg_autoflow "gitlab.com/gitlab-org/cluster-integration/gitlab-agent/pkg/autoflow/v19"

func myFeature() *flowtest.Unit {
	return mustBuild(flowtest.NewUnit("my-feature").
		Description("What this flow exercises").
		AutoFlow(myFeatureStar, "input").             // channels must be declared args
		Send("input", pkg_autoflow.String("go"), 2*time.Second).
		Cancel(5 * time.Second).                      // optional: race cancel vs completion
		ExpectTerminal(flowtest.TerminalCompleted, flowtest.TerminalCanceled))
}

The unit must be added to Units() in corpus/ to run.

Something a generated flow should exercise continuously? Extend the flowgen grammar with a block instead.

Go-only autocore surface (panics, retry policies, inline activities)? Add a Scenario in scenarios/: Register for the workflow, NewInput, and Expect, plus NewDrive if it needs external sends or a cancel. The source instantiates it into Units with generated inputs.

Flows record side effects through the testkit module (echo, delay, record, counter): record writes a session-unique token the oracle later checks for exactly-once delivery -- declare the suffixes with ExpectTokens and the kwarg carrying the session UUID with SessionIDKwarg. counter increments a session-scoped ledger counter and returns its new value on every attempt, which is what a generated poll block waits on.

Generated flows also call the real built-in modules, served from the in-process stubbackend: gitlab.call_api against an HTTP server that answers any request deterministically, event.emit into a counting publisher. Neither a GitLab nor an events platform is needed, and a non-2xx status is data rather than a failure, so a flow's outcome stays known.

Ledger and verification

The ledger is the client-side truth: a session's intents are committed before any op is issued, so a workflow the engine lost is distinguishable from one the driver never started. An observer resolves terminal states back into the ledger. verify (a loadtest binary mode) then evaluates every registered invariant — ledger checks (expectations met, nothing lost), history checks (autocore's own tables are consistent), metrics checks (conservation and coverage via Prometheus) — and exits non-zero on violation. CI runs it after every loadtest drain.

To add an invariant: register it in oracle/ and run make regenerate-flowtest-docs; the catalog cannot drift from the registry.

Operations

Deployment, workload config, chaos levels, and analysis tooling live in scripts/loadtests/autoflow/.

Documentation

Overview

Package flowtest provides the workload abstraction for AutoFlow and autocore testing: a Unit describes one workflow test (how to start it, how to drive it from the outside, and what outcome to expect), a Source produces Units, and the loadtest driver executes them.

Index

Constants

View Source
const (
	// DefaultObserveInterval is the default pause between observer ticks.
	DefaultObserveInterval = 2 * time.Second
	// DefaultObserveBatchSize is the default cap on sessions resolved per tick.
	DefaultObserveBatchSize = 4096
	// DefaultObserveMinAge is the default lower session age bound: younger
	// sessions likely still run their drive scripts, so polling them wastes
	// round trips on workflows that cannot be terminal yet.
	DefaultObserveMinAge = 5 * time.Second
	// DefaultObserveMaxAge is the default upper session age bound: sessions
	// older than this stop being polled, and deciding their fate is the
	// verifier's business.
	DefaultObserveMaxAge = time.Hour
)
View Source
const DefaultMaxInFlightSessions = 8192

DefaultMaxInFlightSessions comfortably covers normal session lifetimes at northstar arrival rates while bounding goroutine growth when the target cannot keep up.

View Source
const DefaultRateSeed uint64 = 0x2545F4914F6CDD1D

DefaultRateSeed seeds the value-noise random walk. It is a fixed constant keyed together with wall-clock time, so every replica computes the same macro rate curve at the same instant -- keeping the fleet in phase so swings are visible in aggregate -- while the level sequence never repeats.

View Source
const DefaultSessionDeadline = 15 * time.Minute

DefaultSessionDeadline gives a workflow ample time to reach a terminal state after its drive script ends before the verifier treats it as overdue.

Variables

This section is empty.

Functions

This section is empty.

Types

type AutoFlowStart

type AutoFlowStart struct {
	Script []byte
	// ChannelArgs names the channel-typed arguments passed with the start
	// request. Only these channels are externally sendable: channel tokens
	// are minted for channels the caller passes in, never for channels the
	// flow creates itself.
	ChannelArgs []string
	// SessionIDKwarg, when non-empty, names a string kwarg the target injects
	// into the start request carrying the session UUID. AutoFlow scripts reject
	// undeclared kwargs, so injection is opt-in per unit; the script uses the
	// kwarg to mint session-unique side-effect tokens.
	SessionIDKwarg string
}

AutoFlowStart starts a workflow through the AutoFlow engine.

type AutocoreStart

type AutocoreStart struct {
	Workflow string
	Input    proto.Message
}

AutocoreStart starts a registered autocore workflow directly.

type AutocoreTarget

type AutocoreTarget struct {
	Client              autocore.Client
	ExternalNamespaceID autocore.ExternalNamespaceID
}

AutocoreTarget drives autocore Units against an autocore client.

func (*AutocoreTarget) Start

func (t *AutocoreTarget) Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)

type CancelOp

type CancelOp struct{}

CancelOp requests cancellation of the workflow.

type CompositeTarget

type CompositeTarget struct {
	Autocore Target
	AutoFlow Target
}

CompositeTarget dispatches each Unit to the arm that supports its start kind. A nil arm rejects the Units it would serve.

func (*CompositeTarget) Start

func (t *CompositeTarget) Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)

type DriveOp

type DriveOp struct {
	At     time.Duration
	Send   *SendOp
	Cancel *CancelOp
	// ExpectError turns the op into an assertion about the engine's error
	// surface: the op must fail with an error whose text contains this
	// substring, and only then counts as behaved. Send ops only; empty means
	// the op is expected to succeed.
	//
	// Only sound for a refusal the engine decides without I/O, such as a
	// channel token it cannot parse or a payload it cannot marshal. A refusal
	// that could instead race an infrastructure fault would let chaos turn
	// environment noise into a safety violation, because a timeout or a
	// dropped connection is indistinguishable from the engine declining. The
	// assertion also cannot tell a harness-side failure from an engine one:
	// the target's own error for a channel it has no token for reads exactly
	// like a refusal, so the substring must name something only the engine says.
	ExpectError string
}

DriveOp is one timed external operation against the running workflow, as an external client (e.g. Rails) would perform it. Exactly one of Send and Cancel is set. At is the offset from the start acknowledgment.

type Driver

type Driver struct {
	// contains filtered or unexported fields
}

Driver executes Units: it paces session starts per the rate model, starts each Unit's workflow, performs its drive script at the configured offsets, and plays the role of the external client (e.g. Rails).

func NewDriver

func NewDriver(cfg *DriverConfig) (*Driver, error)

func (*Driver) Run

func (d *Driver) Run(ctx context.Context)

Run produces and executes sessions until ctx is canceled or the configured injection duration elapses, then waits for in-flight sessions to finish their drive scripts.

type DriverConfig

type DriverConfig struct {
	Log     *slog.Logger
	Target  Target
	Sources []WeightedSource
	Rate    RateModel
	// RNG drives source picking and unit generation. Seeded per replica so
	// replicas produce independent unit streams.
	RNG *rand.Rand
	// MaxInFlightSessions bounds concurrently running sessions. New arrivals
	// are skipped at the bound: the driver stays open-loop, so a saturated
	// target surfaces as skipped sessions rather than as silently stretched
	// arrival gaps. 0 means DefaultMaxInFlightSessions.
	MaxInFlightSessions int
	// InjectionDuration bounds how long Run produces new sessions. Zero means
	// unbounded. In-flight sessions finish their drive scripts before Run
	// returns; external cancellation aborts them.
	InjectionDuration time.Duration
	// Ledger records session intents and op outcomes. Nil disables recording.
	Ledger SessionLedger
	// Meter instruments the driver. Required when Ledger is set so skipped
	// sessions stay visible; a noop meter is used when nil otherwise.
	Meter otelmetric.Meter
	// SessionDeadline is the verification deadline margin recorded per session:
	// the session's ledger deadline is its start time plus its last drive
	// offset plus this. 0 means DefaultSessionDeadline.
	SessionDeadline time.Duration
}

DriverConfig configures a Driver.

type Expectation

type Expectation struct {
	// TerminalStates are the allowed outcomes. A set rather than a single
	// state, to absorb legitimate races such as cancel vs. completion.
	TerminalStates []TerminalState
	// Tokens are session-agnostic suffixes of the side-effect tokens the
	// workflow must record via the testkit module, exactly once each: the
	// driver prefixes each with "<session-uuid>:" per session, and the script
	// builds the full token the same way from its session-id kwarg.
	// Order-irrelevant; empty means the unit expects none.
	Tokens []string
}

Expectation is the declarative verdict spec of a Unit.

type FlowcoreEngine

type FlowcoreEngine interface {
	RunWorkflow(ctx context.Context, namespaceID int64, idempotencyKey string, opts *flowcore.RunWorkflowOptions) (*flowcore.RunWorkflowResult, error)
	CancelWorkflow(ctx context.Context, workflowKey, workflowToken string) error
	SendToWorkflowChannel(ctx context.Context, idempotencyKey, channelToken, workflowToken string, value *pkg_autoflow.Value) error
}

FlowcoreEngine is the subset of *flowcore.Engine the target drives.

type FlowcoreTarget

type FlowcoreTarget struct {
	Engine      FlowcoreEngine
	NamespaceID int64
}

FlowcoreTarget drives AutoFlow Units against a flowcore engine. Each of the Unit's channel args is passed as a channel-valued kwarg of the same name, the session UUID is passed as a string kwarg when the Unit opts in via SessionIDKwarg, and sends are addressed with the per-session channel tokens the start returns.

func (*FlowcoreTarget) Start

func (t *FlowcoreTarget) Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)

type Observer

type Observer struct {
	// contains filtered or unexported fields
}

Observer resolves the terminal states of recorded sessions: it polls the ledger's unobserved window, asks the workflow engine for each workflow's state, and writes the observations back in one batch per tick. Running out of process from the driver's session goroutines decouples workflow lifetime from drive script lifetime.

func NewObserver

func NewObserver(cfg *ObserverConfig) (*Observer, error)

func (*Observer) Run

func (o *Observer) Run(ctx context.Context)

Run ticks until ctx is canceled.

func (*Observer) Tick

func (o *Observer) Tick(ctx context.Context)

Tick performs one polling pass: list unobserved sessions in the observation window, resolve each workflow's state, and write all observations in one batch. Exported so tests can drive the observer without its timer.

type ObserverConfig

type ObserverConfig struct {
	Log    *slog.Logger
	Store  *ledger.Store
	Client WorkflowInfoClient
	// Meter instruments the observer. Optional; nil means no metrics.
	Meter otelmetric.Meter
	// Interval is the pause between ticks. 0 means DefaultObserveInterval.
	Interval time.Duration
	// BatchSize caps sessions resolved per tick. 0 means DefaultObserveBatchSize.
	BatchSize int
	// MinAge excludes sessions younger than this. 0 means DefaultObserveMinAge.
	MinAge time.Duration
	// MaxAge excludes sessions older than this. 0 means DefaultObserveMaxAge.
	MaxAge time.Duration
}

ObserverConfig configures an Observer.

type Provenance

type Provenance struct {
	Seed             *int64
	GeneratorVersion string
	// GeneratorConfig is the generator's own canonical form of the effective
	// config it drew under: the input a re-render needs beyond the seed, recorded
	// per unit so it survives a retuned or swarmed generator.
	GeneratorConfig string
}

Provenance records how a generated Unit was rendered, so any session's unit can be re-rendered from (seed, generator version, generator config). Zero for hand-written units.

type RateModel

type RateModel struct {
	Min    int
	Max    int
	Period time.Duration
	Seed   uint64
}

RateModel yields the target submission rate (units/sec) at a wall-clock instant. With Max <= Min the rate is constant. Otherwise it is a bounded random walk realized as value noise: an independent random level in [Min, Max] is drawn for each Period segment and smoothly interpolated (smoothstep) between consecutive segments. Keyed by wall-clock time + a shared seed, so the curve is identical across replicas (the fleet stays in phase), organic, non-repeating, and bounded -- no fixed sine rhythm.

func (RateModel) At

func (m RateModel) At(now time.Time) int

At returns the target rate at now, always at least 1.

type SendOp

type SendOp struct {
	Channel string
	// RawToken addresses the send with this literal channel token instead of
	// the token the start minted for a channel, which is how a unit addresses
	// a channel the engine never handed it. AutoFlow units only.
	RawToken string
	Payload  proto.Message
}

SendOp sends a value on one of the workflow's channels. Exactly one of Channel and RawToken addresses it.

type SessionHandle

type SessionHandle interface {
	// WorkflowKey identifies the started workflow in its engine's format.
	WorkflowKey() string
	Send(ctx context.Context, idempotencyKey string, op *SendOp) error
	Cancel(ctx context.Context) error
}

SessionHandle drives one started workflow. Sends and cancellation are addressed the way the workflow's engine addresses them, which is what the handle encapsulates.

type SessionInfo

type SessionInfo struct {
	ID             uuid.UUID
	IdempotencyKey string
}

SessionInfo identifies one driver session at start: ID is the session UUID (also the ledger's session_id) and IdempotencyKey deduplicates the start request.

type SessionLedger

type SessionLedger interface {
	SessionStarted(ctx context.Context, rec *ledger.SessionRecord) error
	WorkflowKeyKnown(sessionID uuid.UUID, key string)
	OpDone(sessionID uuid.UUID, opIndex int32, err error)
}

SessionLedger is the subset of *ledger.Writer the Driver records sessions to.

type Source

type Source interface {
	Name() string
	// NextUnit returns the next Unit. All randomness must come from rng so a
	// run is reproducible from its seed.
	NextUnit(rng *rand.Rand) (*Unit, error)
}

Source produces Units. Implementations are the generation strategies: pre-defined corpora pick units, generative sources synthesize them.

type StartSpec

type StartSpec struct {
	AutoFlow *AutoFlowStart
	Autocore *AutocoreStart
}

StartSpec describes how a Unit's workflow starts. Exactly one arm is set.

type Target

type Target interface {
	Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)
}

Target starts a Unit's workflow and returns the handle used to drive it. A target that does not support the Unit's start kind returns an error.

type TerminalState

type TerminalState int

TerminalState is a workflow outcome a Unit's Expectation can allow.

const (
	TerminalCompleted TerminalState = iota + 1
	TerminalFailed
	TerminalCanceled
	TerminalTimedOut
	TerminalSystemFailed
)

func (TerminalState) String

func (s TerminalState) String() string

type Unit

type Unit struct {
	Name        string
	Description string
	Start       StartSpec
	// Drive holds the timed external operations performed against the
	// running workflow, ordered by offset.
	Drive      []DriveOp
	Expect     Expectation
	Provenance Provenance
}

Unit is a complete, self-contained test definition. It is pure data with all randomness already spent: a generated Unit carries chosen values and offsets, not distributions. Construct Units with NewUnit; the driver executes each Unit as one session.

type UnitBuilder

type UnitBuilder struct {
	// contains filtered or unexported fields
}

UnitBuilder assembles a Unit. Build validates structural rules that would otherwise only fail at execution time, e.g. sending on a channel the driver holds no token for.

func NewUnit

func NewUnit(name string) *UnitBuilder

NewUnit starts building a Unit with the given name.

func (*UnitBuilder) AutoFlow

func (b *UnitBuilder) AutoFlow(script []byte, channelArgs ...string) *UnitBuilder

AutoFlow makes the Unit start an AutoFlow workflow from the given Starlark script. channelArgs declares the channel-typed arguments passed with the start request; only these channels can be targeted by Send.

func (*UnitBuilder) Autocore

func (b *UnitBuilder) Autocore(workflow string, input proto.Message) *UnitBuilder

Autocore makes the Unit start the given registered autocore workflow.

func (*UnitBuilder) Build

func (b *UnitBuilder) Build() (*Unit, error)

Build validates the Unit and returns it. Drive ops are ordered by offset, preserving insertion order for equal offsets.

func (*UnitBuilder) Cancel

func (b *UnitBuilder) Cancel(at time.Duration) *UnitBuilder

Cancel schedules a cancellation request at the given offset from the start acknowledgment. At most one cancel per Unit.

func (*UnitBuilder) Description

func (b *UnitBuilder) Description(d string) *UnitBuilder

func (*UnitBuilder) ExpectTerminal

func (b *UnitBuilder) ExpectTerminal(states ...TerminalState) *UnitBuilder

ExpectTerminal declares the set of allowed terminal states.

func (*UnitBuilder) ExpectTokens

func (b *UnitBuilder) ExpectTokens(tokens ...string) *UnitBuilder

ExpectTokens declares the session-agnostic suffixes of the side-effect tokens the workflow must record via the testkit module, exactly once each; the driver prefixes each with "<session-uuid>:" per session. Order-irrelevant.

func (*UnitBuilder) Provenance

func (b *UnitBuilder) Provenance(seed int64, version, config string) *UnitBuilder

Provenance records the generator seed, version and canonical config the Unit was rendered from, so it can be re-rendered. At most one call.

func (*UnitBuilder) Send

func (b *UnitBuilder) Send(channel string, payload proto.Message, at time.Duration) *UnitBuilder

Send schedules a value send on the named channel at the given offset from the start acknowledgment. For AutoFlow units the channel must be a declared channel arg and the payload a *autoflow.Value; for autocore units the name is engine-generated and only checked at execution.

func (*UnitBuilder) SendExpectingError

func (b *UnitBuilder) SendExpectingError(channel string, payload proto.Message, at time.Duration, errSubstring string) *UnitBuilder

SendExpectingError schedules a send the engine must refuse: the op counts as behaved only when it fails with an error whose text contains errSubstring, and as violating when the engine accepts it or refuses it differently. Only sound when the engine decides the refusal without I/O, and errSubstring names something only the engine says; see DriveOp.ExpectError for why both matter.

func (*UnitBuilder) SendRawTokenExpectingError

func (b *UnitBuilder) SendRawTokenExpectingError(token string, payload proto.Message, at time.Duration, errSubstring string) *UnitBuilder

SendRawTokenExpectingError schedules a send addressed with a literal channel token rather than one the start minted, which the engine must refuse with an error whose text contains errSubstring. Requires an AutoFlow start: the token format is the flowcore engine's. Only sound when the engine decides the refusal without I/O, and errSubstring names something only the engine says; see DriveOp.ExpectError for why both matter.

func (*UnitBuilder) SessionIDKwarg

func (b *UnitBuilder) SessionIDKwarg(name string) *UnitBuilder

SessionIDKwarg names the string kwarg the target injects into the AutoFlow start request carrying the session UUID, so a static script can mint session-unique side-effect tokens. Requires an AutoFlow start; the name must not collide with a declared channel arg.

type WeightedSource

type WeightedSource struct {
	Source Source
	Weight int
}

WeightedSource is a Source with its share in the Driver's unit mix.

type WorkflowInfoClient

type WorkflowInfoClient interface {
	GetWorkflowInfo(ctx context.Context, key autocore.WorkflowKey) (*autocore.WorkflowInfo, error)
}

WorkflowInfoClient is the subset of autocore.Client the Observer resolves workflow states through. Both autocore and AutoFlow units share the same workflow key format, so one client serves all sessions.

Directories

Path Synopsis
Package corpus holds the pre-defined AutoFlow Units of the flowtest workload.
Package corpus holds the pre-defined AutoFlow Units of the flowtest workload.
Package flowgen generates flowtest Units outcome-first: a typed plan is drawn from a seeded, weighted block grammar, and the Starlark script, the drive ops and the expectation all derive from that one plan, so every generated flow is self-validating by construction.
Package flowgen generates flowtest Units outcome-first: a typed plan is drawn from a seeded, weighted block grammar, and the Starlark script, the drive ops and the expectation all derive from that one plan, so every generated flow is self-validating by construction.
Package ledger is the loadtest driver's durable record of executed sessions: the authoritative client-side observation a later verifier consumes.
Package ledger is the loadtest driver's durable record of executed sessions: the authoritative client-side observation a later verifier consumes.
Package loadtestmodule provides a compiled-in AutoFlow module for testing.
Package loadtestmodule provides a compiled-in AutoFlow module for testing.
Package oracle verifies invariants over the observation channels a flowtest run leaves behind.
Package oracle verifies invariants over the observation channels a flowtest run leaves behind.
cmd/docgen command
Command docgen writes the flowtest invariant catalog rendered from the oracle registry.
Command docgen writes the flowtest invariant catalog rendered from the oracle registry.
Package stubbackend stands in for the external systems the real built-in AutoFlow modules talk to, inside the process that runs them: an HTTP server in place of the GitLab REST API and a counting sink in place of the events platform.
Package stubbackend stands in for the external systems the real built-in AutoFlow modules talk to, inside the process that runs them: an HTTP server in place of the GitLab REST API and a counting sink in place of the events platform.
Package testkitmodule provides the compiled-in "testkit" AutoFlow module for generated test flows: deterministic actions whose side effects the flowtest oracle can verify against the session ledger.
Package testkitmodule provides the compiled-in "testkit" AutoFlow module for generated test flows: deterministic actions whose side effects the flowtest oracle can verify against the session ledger.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL