Documentation
¶
Overview ¶
Package flowtest provides the workload abstraction for AutoFlow and autocore testing: a Unit describes one workflow test (how to start it, how to drive it from the outside, and what outcome to expect), a Source produces Units, and the loadtest driver executes them.
Index ¶
- Constants
- type AutoFlowStart
- type AutocoreStart
- type AutocoreTarget
- type CancelOp
- type CompositeTarget
- type DriveOp
- type Driver
- type DriverConfig
- type Expectation
- type FlowcoreEngine
- type FlowcoreTarget
- type Observer
- type ObserverConfig
- type Provenance
- type RateModel
- type SendOp
- type SessionHandle
- type SessionInfo
- type SessionLedger
- type Source
- type StartSpec
- type Target
- type TerminalState
- type Unit
- type UnitBuilder
- func (b *UnitBuilder) AutoFlow(script []byte, channelArgs ...string) *UnitBuilder
- func (b *UnitBuilder) Autocore(workflow string, input proto.Message) *UnitBuilder
- func (b *UnitBuilder) Build() (*Unit, error)
- func (b *UnitBuilder) Cancel(at time.Duration) *UnitBuilder
- func (b *UnitBuilder) Description(d string) *UnitBuilder
- func (b *UnitBuilder) ExpectTerminal(states ...TerminalState) *UnitBuilder
- func (b *UnitBuilder) ExpectTokens(tokens ...string) *UnitBuilder
- func (b *UnitBuilder) Provenance(seed int64, version, config string) *UnitBuilder
- func (b *UnitBuilder) Send(channel string, payload proto.Message, at time.Duration) *UnitBuilder
- func (b *UnitBuilder) SendExpectingError(channel string, payload proto.Message, at time.Duration, errSubstring string) *UnitBuilder
- func (b *UnitBuilder) SendRawTokenExpectingError(token string, payload proto.Message, at time.Duration, errSubstring string) *UnitBuilder
- func (b *UnitBuilder) SessionIDKwarg(name string) *UnitBuilder
- type WeightedSource
- type WorkflowInfoClient
Constants ¶
const ( // DefaultObserveInterval is the default pause between observer ticks. DefaultObserveInterval = 2 * time.Second // DefaultObserveBatchSize is the default cap on sessions resolved per tick. DefaultObserveBatchSize = 4096 // DefaultObserveMinAge is the default lower session age bound: younger // sessions likely still run their drive scripts, so polling them wastes // round trips on workflows that cannot be terminal yet. DefaultObserveMinAge = 5 * time.Second // DefaultObserveMaxAge is the default upper session age bound: sessions // older than this stop being polled, and deciding their fate is the // verifier's business. DefaultObserveMaxAge = time.Hour )
const DefaultMaxInFlightSessions = 8192
DefaultMaxInFlightSessions comfortably covers normal session lifetimes at northstar arrival rates while bounding goroutine growth when the target cannot keep up.
const DefaultRateSeed uint64 = 0x2545F4914F6CDD1D
DefaultRateSeed seeds the value-noise random walk. It is a fixed constant keyed together with wall-clock time, so every replica computes the same macro rate curve at the same instant -- keeping the fleet in phase so swings are visible in aggregate -- while the level sequence never repeats.
const DefaultSessionDeadline = 15 * time.Minute
DefaultSessionDeadline gives a workflow ample time to reach a terminal state after its drive script ends before the verifier treats it as overdue.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AutoFlowStart ¶
type AutoFlowStart struct {
Script []byte
// ChannelArgs names the channel-typed arguments passed with the start
// request. Only these channels are externally sendable: channel tokens
// are minted for channels the caller passes in, never for channels the
// flow creates itself.
ChannelArgs []string
// SessionIDKwarg, when non-empty, names a string kwarg the target injects
// into the start request carrying the session UUID. AutoFlow scripts reject
// undeclared kwargs, so injection is opt-in per unit; the script uses the
// kwarg to mint session-unique side-effect tokens.
SessionIDKwarg string
}
AutoFlowStart starts a workflow through the AutoFlow engine.
type AutocoreStart ¶
AutocoreStart starts a registered autocore workflow directly.
type AutocoreTarget ¶
type AutocoreTarget struct {
Client autocore.Client
ExternalNamespaceID autocore.ExternalNamespaceID
}
AutocoreTarget drives autocore Units against an autocore client.
func (*AutocoreTarget) Start ¶
func (t *AutocoreTarget) Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)
type CompositeTarget ¶
CompositeTarget dispatches each Unit to the arm that supports its start kind. A nil arm rejects the Units it would serve.
func (*CompositeTarget) Start ¶
func (t *CompositeTarget) Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)
type DriveOp ¶
type DriveOp struct {
At time.Duration
Send *SendOp
Cancel *CancelOp
// ExpectError turns the op into an assertion about the engine's error
// surface: the op must fail with an error whose text contains this
// substring, and only then counts as behaved. Send ops only; empty means
// the op is expected to succeed.
//
// Only sound for a refusal the engine decides without I/O, such as a
// channel token it cannot parse or a payload it cannot marshal. A refusal
// that could instead race an infrastructure fault would let chaos turn
// environment noise into a safety violation, because a timeout or a
// dropped connection is indistinguishable from the engine declining. The
// assertion also cannot tell a harness-side failure from an engine one:
// the target's own error for a channel it has no token for reads exactly
// like a refusal, so the substring must name something only the engine says.
ExpectError string
}
DriveOp is one timed external operation against the running workflow, as an external client (e.g. Rails) would perform it. Exactly one of Send and Cancel is set. At is the offset from the start acknowledgment.
type Driver ¶
type Driver struct {
// contains filtered or unexported fields
}
Driver executes Units: it paces session starts per the rate model, starts each Unit's workflow, performs its drive script at the configured offsets, and plays the role of the external client (e.g. Rails).
func NewDriver ¶
func NewDriver(cfg *DriverConfig) (*Driver, error)
type DriverConfig ¶
type DriverConfig struct {
Log *slog.Logger
Target Target
Sources []WeightedSource
Rate RateModel
// RNG drives source picking and unit generation. Seeded per replica so
// replicas produce independent unit streams.
RNG *rand.Rand
// MaxInFlightSessions bounds concurrently running sessions. New arrivals
// are skipped at the bound: the driver stays open-loop, so a saturated
// target surfaces as skipped sessions rather than as silently stretched
// arrival gaps. 0 means DefaultMaxInFlightSessions.
MaxInFlightSessions int
// InjectionDuration bounds how long Run produces new sessions. Zero means
// unbounded. In-flight sessions finish their drive scripts before Run
// returns; external cancellation aborts them.
InjectionDuration time.Duration
// Ledger records session intents and op outcomes. Nil disables recording.
Ledger SessionLedger
// Meter instruments the driver. Required when Ledger is set so skipped
// sessions stay visible; a noop meter is used when nil otherwise.
Meter otelmetric.Meter
// SessionDeadline is the verification deadline margin recorded per session:
// the session's ledger deadline is its start time plus its last drive
// offset plus this. 0 means DefaultSessionDeadline.
SessionDeadline time.Duration
}
DriverConfig configures a Driver.
type Expectation ¶
type Expectation struct {
// TerminalStates are the allowed outcomes. A set rather than a single
// state, to absorb legitimate races such as cancel vs. completion.
TerminalStates []TerminalState
// Tokens are session-agnostic suffixes of the side-effect tokens the
// workflow must record via the testkit module, exactly once each: the
// driver prefixes each with "<session-uuid>:" per session, and the script
// builds the full token the same way from its session-id kwarg.
// Order-irrelevant; empty means the unit expects none.
Tokens []string
}
Expectation is the declarative verdict spec of a Unit.
type FlowcoreEngine ¶
type FlowcoreEngine interface {
RunWorkflow(ctx context.Context, namespaceID int64, idempotencyKey string, opts *flowcore.RunWorkflowOptions) (*flowcore.RunWorkflowResult, error)
CancelWorkflow(ctx context.Context, workflowKey, workflowToken string) error
SendToWorkflowChannel(ctx context.Context, idempotencyKey, channelToken, workflowToken string, value *pkg_autoflow.Value) error
}
FlowcoreEngine is the subset of *flowcore.Engine the target drives.
type FlowcoreTarget ¶
type FlowcoreTarget struct {
Engine FlowcoreEngine
NamespaceID int64
}
FlowcoreTarget drives AutoFlow Units against a flowcore engine. Each of the Unit's channel args is passed as a channel-valued kwarg of the same name, the session UUID is passed as a string kwarg when the Unit opts in via SessionIDKwarg, and sends are addressed with the per-session channel tokens the start returns.
func (*FlowcoreTarget) Start ¶
func (t *FlowcoreTarget) Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)
type Observer ¶
type Observer struct {
// contains filtered or unexported fields
}
Observer resolves the terminal states of recorded sessions: it polls the ledger's unobserved window, asks the workflow engine for each workflow's state, and writes the observations back in one batch per tick. Running out of process from the driver's session goroutines decouples workflow lifetime from drive script lifetime.
func NewObserver ¶
func NewObserver(cfg *ObserverConfig) (*Observer, error)
type ObserverConfig ¶
type ObserverConfig struct {
Log *slog.Logger
Store *ledger.Store
Client WorkflowInfoClient
// Meter instruments the observer. Optional; nil means no metrics.
Meter otelmetric.Meter
// Interval is the pause between ticks. 0 means DefaultObserveInterval.
Interval time.Duration
// BatchSize caps sessions resolved per tick. 0 means DefaultObserveBatchSize.
BatchSize int
// MinAge excludes sessions younger than this. 0 means DefaultObserveMinAge.
MinAge time.Duration
// MaxAge excludes sessions older than this. 0 means DefaultObserveMaxAge.
MaxAge time.Duration
}
ObserverConfig configures an Observer.
type Provenance ¶
type Provenance struct {
Seed *int64
GeneratorVersion string
// GeneratorConfig is the generator's own canonical form of the effective
// config it drew under: the input a re-render needs beyond the seed, recorded
// per unit so it survives a retuned or swarmed generator.
GeneratorConfig string
}
Provenance records how a generated Unit was rendered, so any session's unit can be re-rendered from (seed, generator version, generator config). Zero for hand-written units.
type RateModel ¶
RateModel yields the target submission rate (units/sec) at a wall-clock instant. With Max <= Min the rate is constant. Otherwise it is a bounded random walk realized as value noise: an independent random level in [Min, Max] is drawn for each Period segment and smoothly interpolated (smoothstep) between consecutive segments. Keyed by wall-clock time + a shared seed, so the curve is identical across replicas (the fleet stays in phase), organic, non-repeating, and bounded -- no fixed sine rhythm.
type SendOp ¶
type SendOp struct {
Channel string
// RawToken addresses the send with this literal channel token instead of
// the token the start minted for a channel, which is how a unit addresses
// a channel the engine never handed it. AutoFlow units only.
RawToken string
Payload proto.Message
}
SendOp sends a value on one of the workflow's channels. Exactly one of Channel and RawToken addresses it.
type SessionHandle ¶
type SessionHandle interface {
// WorkflowKey identifies the started workflow in its engine's format.
WorkflowKey() string
Send(ctx context.Context, idempotencyKey string, op *SendOp) error
Cancel(ctx context.Context) error
}
SessionHandle drives one started workflow. Sends and cancellation are addressed the way the workflow's engine addresses them, which is what the handle encapsulates.
type SessionInfo ¶
SessionInfo identifies one driver session at start: ID is the session UUID (also the ledger's session_id) and IdempotencyKey deduplicates the start request.
type SessionLedger ¶
type SessionLedger interface {
SessionStarted(ctx context.Context, rec *ledger.SessionRecord) error
WorkflowKeyKnown(sessionID uuid.UUID, key string)
OpDone(sessionID uuid.UUID, opIndex int32, err error)
}
SessionLedger is the subset of *ledger.Writer the Driver records sessions to.
type Source ¶
type Source interface {
Name() string
// NextUnit returns the next Unit. All randomness must come from rng so a
// run is reproducible from its seed.
NextUnit(rng *rand.Rand) (*Unit, error)
}
Source produces Units. Implementations are the generation strategies: pre-defined corpora pick units, generative sources synthesize them.
type StartSpec ¶
type StartSpec struct {
AutoFlow *AutoFlowStart
Autocore *AutocoreStart
}
StartSpec describes how a Unit's workflow starts. Exactly one arm is set.
type Target ¶
type Target interface {
Start(ctx context.Context, session SessionInfo, start StartSpec) (SessionHandle, error)
}
Target starts a Unit's workflow and returns the handle used to drive it. A target that does not support the Unit's start kind returns an error.
type TerminalState ¶
type TerminalState int
TerminalState is a workflow outcome a Unit's Expectation can allow.
const ( TerminalCompleted TerminalState = iota + 1 TerminalFailed TerminalCanceled TerminalTimedOut TerminalSystemFailed )
func (TerminalState) String ¶
func (s TerminalState) String() string
type Unit ¶
type Unit struct {
Name string
Description string
Start StartSpec
// Drive holds the timed external operations performed against the
// running workflow, ordered by offset.
Drive []DriveOp
Expect Expectation
Provenance Provenance
}
Unit is a complete, self-contained test definition. It is pure data with all randomness already spent: a generated Unit carries chosen values and offsets, not distributions. Construct Units with NewUnit; the driver executes each Unit as one session.
type UnitBuilder ¶
type UnitBuilder struct {
// contains filtered or unexported fields
}
UnitBuilder assembles a Unit. Build validates structural rules that would otherwise only fail at execution time, e.g. sending on a channel the driver holds no token for.
func NewUnit ¶
func NewUnit(name string) *UnitBuilder
NewUnit starts building a Unit with the given name.
func (*UnitBuilder) AutoFlow ¶
func (b *UnitBuilder) AutoFlow(script []byte, channelArgs ...string) *UnitBuilder
AutoFlow makes the Unit start an AutoFlow workflow from the given Starlark script. channelArgs declares the channel-typed arguments passed with the start request; only these channels can be targeted by Send.
func (*UnitBuilder) Autocore ¶
func (b *UnitBuilder) Autocore(workflow string, input proto.Message) *UnitBuilder
Autocore makes the Unit start the given registered autocore workflow.
func (*UnitBuilder) Build ¶
func (b *UnitBuilder) Build() (*Unit, error)
Build validates the Unit and returns it. Drive ops are ordered by offset, preserving insertion order for equal offsets.
func (*UnitBuilder) Cancel ¶
func (b *UnitBuilder) Cancel(at time.Duration) *UnitBuilder
Cancel schedules a cancellation request at the given offset from the start acknowledgment. At most one cancel per Unit.
func (*UnitBuilder) Description ¶
func (b *UnitBuilder) Description(d string) *UnitBuilder
func (*UnitBuilder) ExpectTerminal ¶
func (b *UnitBuilder) ExpectTerminal(states ...TerminalState) *UnitBuilder
ExpectTerminal declares the set of allowed terminal states.
func (*UnitBuilder) ExpectTokens ¶
func (b *UnitBuilder) ExpectTokens(tokens ...string) *UnitBuilder
ExpectTokens declares the session-agnostic suffixes of the side-effect tokens the workflow must record via the testkit module, exactly once each; the driver prefixes each with "<session-uuid>:" per session. Order-irrelevant.
func (*UnitBuilder) Provenance ¶
func (b *UnitBuilder) Provenance(seed int64, version, config string) *UnitBuilder
Provenance records the generator seed, version and canonical config the Unit was rendered from, so it can be re-rendered. At most one call.
func (*UnitBuilder) Send ¶
func (b *UnitBuilder) Send(channel string, payload proto.Message, at time.Duration) *UnitBuilder
Send schedules a value send on the named channel at the given offset from the start acknowledgment. For AutoFlow units the channel must be a declared channel arg and the payload a *autoflow.Value; for autocore units the name is engine-generated and only checked at execution.
func (*UnitBuilder) SendExpectingError ¶
func (b *UnitBuilder) SendExpectingError(channel string, payload proto.Message, at time.Duration, errSubstring string) *UnitBuilder
SendExpectingError schedules a send the engine must refuse: the op counts as behaved only when it fails with an error whose text contains errSubstring, and as violating when the engine accepts it or refuses it differently. Only sound when the engine decides the refusal without I/O, and errSubstring names something only the engine says; see DriveOp.ExpectError for why both matter.
func (*UnitBuilder) SendRawTokenExpectingError ¶
func (b *UnitBuilder) SendRawTokenExpectingError(token string, payload proto.Message, at time.Duration, errSubstring string) *UnitBuilder
SendRawTokenExpectingError schedules a send addressed with a literal channel token rather than one the start minted, which the engine must refuse with an error whose text contains errSubstring. Requires an AutoFlow start: the token format is the flowcore engine's. Only sound when the engine decides the refusal without I/O, and errSubstring names something only the engine says; see DriveOp.ExpectError for why both matter.
func (*UnitBuilder) SessionIDKwarg ¶
func (b *UnitBuilder) SessionIDKwarg(name string) *UnitBuilder
SessionIDKwarg names the string kwarg the target injects into the AutoFlow start request carrying the session UUID, so a static script can mint session-unique side-effect tokens. Requires an AutoFlow start; the name must not collide with a declared channel arg.
type WeightedSource ¶
WeightedSource is a Source with its share in the Driver's unit mix.
type WorkflowInfoClient ¶
type WorkflowInfoClient interface {
GetWorkflowInfo(ctx context.Context, key autocore.WorkflowKey) (*autocore.WorkflowInfo, error)
}
WorkflowInfoClient is the subset of autocore.Client the Observer resolves workflow states through. Both autocore and AutoFlow units share the same workflow key format, so one client serves all sessions.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package corpus holds the pre-defined AutoFlow Units of the flowtest workload.
|
Package corpus holds the pre-defined AutoFlow Units of the flowtest workload. |
|
Package flowgen generates flowtest Units outcome-first: a typed plan is drawn from a seeded, weighted block grammar, and the Starlark script, the drive ops and the expectation all derive from that one plan, so every generated flow is self-validating by construction.
|
Package flowgen generates flowtest Units outcome-first: a typed plan is drawn from a seeded, weighted block grammar, and the Starlark script, the drive ops and the expectation all derive from that one plan, so every generated flow is self-validating by construction. |
|
Package ledger is the loadtest driver's durable record of executed sessions: the authoritative client-side observation a later verifier consumes.
|
Package ledger is the loadtest driver's durable record of executed sessions: the authoritative client-side observation a later verifier consumes. |
|
Package loadtestmodule provides a compiled-in AutoFlow module for testing.
|
Package loadtestmodule provides a compiled-in AutoFlow module for testing. |
|
Package oracle verifies invariants over the observation channels a flowtest run leaves behind.
|
Package oracle verifies invariants over the observation channels a flowtest run leaves behind. |
|
cmd/docgen
command
Command docgen writes the flowtest invariant catalog rendered from the oracle registry.
|
Command docgen writes the flowtest invariant catalog rendered from the oracle registry. |
|
Package stubbackend stands in for the external systems the real built-in AutoFlow modules talk to, inside the process that runs them: an HTTP server in place of the GitLab REST API and a counting sink in place of the events platform.
|
Package stubbackend stands in for the external systems the real built-in AutoFlow modules talk to, inside the process that runs them: an HTTP server in place of the GitLab REST API and a counting sink in place of the events platform. |
|
Package testkitmodule provides the compiled-in "testkit" AutoFlow module for generated test flows: deterministic actions whose side effects the flowtest oracle can verify against the session ledger.
|
Package testkitmodule provides the compiled-in "testkit" AutoFlow module for generated test flows: deterministic actions whose side effects the flowtest oracle can verify against the session ledger. |