Documentation
¶
Overview ¶
Package soltesting provides types and utilities for functional testing of solutions. It defines the test case specification, assertion types, and result structures used by the `scafctl test functional` and `scafctl test list` commands.
Test definitions live under `spec.testing` in solution YAML and support compose-based splitting into separate files.
Index ¶
- Constants
- func BuildAssertionContext(cmdOutput *CommandOutput) map[string]any
- func BuiltinName(shortName string) string
- func CompareSnapshot(actual, snapshotPath, sandboxPath string) (bool, string, error)
- func DeriveTestName(command, args []string) string
- func DiagnoseExpression(ctx context.Context, expr string, celCtx map[string]any) string
- func ExtendsChainString(tests map[string]*TestCase, name string) string
- func GenerateToYAML(result *GenerateResult) ([]byte, error)
- func IsBuiltin(name string) bool
- func Normalize(input, sandboxPath string) string
- func ReportList(solutions []SolutionTests, opts *kvx.OutputOptions, includeBuiltins bool) error
- func ReportResults(results []TestResult, opts *kvx.OutputOptions, verbose bool, ...) error
- func ResolveExtends(tests map[string]*TestCase) error
- func ScaffoldToYAML(result *ScaffoldResult) ([]byte, error)
- func SortedTestNames(st SolutionTests) []string
- func TemplateNamePattern() *regexp.Regexp
- func TestNamePattern() *regexp.Regexp
- func UpdateSnapshot(actual, snapshotPath, sandboxPath string) error
- func WriteJUnitReport(results []TestResult, path string) error
- type Assertion
- type AssertionResult
- type CommandBuilder
- type CommandOutput
- type FileInfo
- type FilterOptions
- type GenerateInput
- type GenerateResult
- type InitStep
- type ResultSummary
- type Runner
- type Sandbox
- type ScaffoldInput
- type ScaffoldResult
- type ServiceConfig
- type SkipBuiltinsValue
- type SkipValue
- func (s SkipValue) HasExpression() bool
- func (s SkipValue) IsZero() bool
- func (s SkipValue) MarshalJSON() ([]byte, error)
- func (s SkipValue) MarshalYAML() (any, error)
- func (s SkipValue) ShouldSkip() bool
- func (s SkipValue) String() string
- func (s *SkipValue) UnmarshalJSON(data []byte) error
- func (s *SkipValue) UnmarshalYAML(value *yaml.Node) error
- type SolutionInfo
- type SolutionTests
- type Status
- type TestCase
- type TestConfig
- type TestProgressCallback
- type TestResult
- type TestSuite
- type WatchOptions
- type WatchResult
- type Watcher
Constants ¶
const ( BuiltinParse = "parse" BuiltinLint = "lint" BuiltinResolveDefault = "resolve-defaults" BuiltinRenderDefault = "render-defaults" )
Builtin test names (without prefix, for skipBuiltins matching).
const ( // MaxFileSize is the maximum file size (10MB) before content is replaced with a placeholder. MaxFileSize = 10 * 1024 * 1024 // FileTooLargePlaceholder is the placeholder for files exceeding MaxFileSize. FileTooLargePlaceholder = "<file too large>" // BinaryFilePlaceholder is the placeholder for non-UTF-8 (binary) files. BinaryFilePlaceholder = "<binary file>" )
const ( TimestampPlaceholder = "<TIMESTAMP>" UUIDPlaceholder = "<UUID>" SandboxPlaceholder = "<SANDBOX>" )
Snapshot normalization placeholders.
const ( // MaxAssertionsPerTest is the maximum number of assertions allowed per test case. MaxAssertionsPerTest = 100 // MaxFilesPerTest is the maximum number of file entries allowed per test case. MaxFilesPerTest = 50 // MaxTagsPerTest is the maximum number of tags allowed per test case. MaxTagsPerTest = 20 // MaxExtendsDepth is the maximum depth of extends inheritance chains. MaxExtendsDepth = 10 // MaxTestsPerSolution is the maximum number of tests per solution. MaxTestsPerSolution = 500 // MaxRetries is the maximum number of retry attempts for a failing test. MaxRetries = 10 )
Max limits enforced by Validate().
const ( // DefaultDebounceDuration is the default debounce interval for file changes. // Rapid successive writes (e.g. editor save-all) are collapsed into a single re-run. DefaultDebounceDuration = 300 * time.Millisecond )
const MaxGeneratedAssertions = 20
MaxGeneratedAssertions caps the number of assertions produced by the generator. The user is expected to curate the generated list before committing it.
Variables ¶
This section is empty.
Functions ¶
func BuildAssertionContext ¶
func BuildAssertionContext(cmdOutput *CommandOutput) map[string]any
BuildAssertionContext creates the CEL variable map from command output. The returned map contains top-level variables that can be referenced directly in CEL expressions: __stdout, __stderr, __exitCode, __output, __files. The __ prefix follows the convention used elsewhere in scafctl CEL contexts (e.g. __self, __item, __execution) and avoids collisions with user-defined names.
When output is nil (stdout was not valid JSON or -o json was not specified), the "__output" variable is set to nil. CEL expressions referencing it will be caught during evaluation and produce a StatusError result with an appropriate diagnostic.
func BuiltinName ¶
BuiltinName returns the full builtin name with the "builtin:" prefix.
func CompareSnapshot ¶
CompareSnapshot normalizes actual, reads the golden file at snapshotPath, and compares them. Returns (match, unifiedDiff, error).
func DeriveTestName ¶ added in v0.5.0
DeriveTestName produces a kebab-case test name from a command path and its args.
The algorithm:
- Starts with the command words (e.g. "render", "solution").
- For each flag that takes a value (-r env=prod), splits "key=value" pairs and appends both parts; plain values are appended as-is.
- Positional args are appended directly.
- The result is lowercased, non-alphanumeric chars replaced with dashes, consecutive dashes collapsed, and leading/trailing dashes stripped.
Examples:
DeriveTestName(["render","solution"], ["-r","env=prod"]) → "render-solution-env-prod" DeriveTestName(["run","resolver"], ["db"]) → "run-resolver-db"
func DiagnoseExpression ¶
DiagnoseExpression inspects a failing CEL expression and returns a diagnostic string showing sub-expression values. For comparison expressions (==, !=, <, >, >=, <=), it evaluates both sides independently and shows the mismatch. For non-comparison expressions, it returns the expression and its result. Falls back to "expected true, got false" for expressions too complex to decompose.
func ExtendsChainString ¶
ExtendsChainString returns a human-readable representation of the extends chain for diagnostic purposes.
func GenerateToYAML ¶ added in v0.5.0
func GenerateToYAML(result *GenerateResult) ([]byte, error)
GenerateToYAML marshals a GenerateResult to YAML ready for pasting into the spec.testing.cases section of a solution file. The outer key is the test name.
func Normalize ¶
Normalize applies the fixed normalization pipeline to the input string:
- Sort JSON map keys deterministically (if valid JSON).
- Replace ISO-8601 timestamps with <TIMESTAMP>.
- Replace UUIDs with <UUID>.
- Replace sandbox absolute paths with <SANDBOX>.
func ReportList ¶
func ReportList(solutions []SolutionTests, opts *kvx.OutputOptions, includeBuiltins bool) error
ReportList formats and writes test discovery results.
func ReportResults ¶
func ReportResults(results []TestResult, opts *kvx.OutputOptions, verbose bool, elapsed time.Duration) error
ReportResults formats and writes test results using the given output options. For table format it writes a human-readable table with summary. For JSON/YAML it delegates to kvx.OutputOptions.Write. For quiet format it writes nothing. The elapsed parameter, when > 0, overrides the summed individual durations in the summary line with the actual wall-clock time.
func ResolveExtends ¶
ResolveExtends resolves all extends chains in the tests map in-place. For each test with extends, fields are merged from the referenced templates left-to-right using the following merge strategy:
- command: child wins if set
- args: appended (base first, then child)
- assertions: appended (base first, then child)
- files: appended, deduplicated
- init: base steps prepended before child steps
- cleanup: base steps appended after child steps
- tags: appended, deduplicated
- env: merged map, child wins on key conflict
- Scalar fields: child wins if set
Returns an error if a circular dependency is detected, an extends reference does not exist, or the inheritance depth exceeds MaxExtendsDepth.
func ScaffoldToYAML ¶
func ScaffoldToYAML(result *ScaffoldResult) ([]byte, error)
ScaffoldToYAML marshals the scaffold result to YAML suitable for embedding in a solution's spec.testing.cases section.
func SortedTestNames ¶
func SortedTestNames(st SolutionTests) []string
SortedTestNames returns the test names from a SolutionTests in sorted order. Builtin tests (prefixed with "builtin:") sort first, then alphabetical.
func TemplateNamePattern ¶
TemplateNamePattern returns the compiled regex for valid template names.
func TestNamePattern ¶
TestNamePattern returns the compiled regex for valid test names.
func UpdateSnapshot ¶
UpdateSnapshot normalizes the actual output and writes it to the snapshotPath.
func WriteJUnitReport ¶
func WriteJUnitReport(results []TestResult, path string) error
WriteJUnitReport generates a JUnit XML report from test results and writes it to path.
Types ¶
type Assertion ¶
type Assertion struct {
// Expression is a CEL expression evaluating to bool.
Expression celexp.Expression `json:"expression,omitempty" yaml:"expression,omitempty" doc:"CEL expression evaluating to bool"`
// Regex is a regex pattern that must match somewhere in the target text.
Regex string `json:"regex,omitempty" yaml:"regex,omitempty" doc:"Regex pattern that must match" maxLength:"1000"`
// Contains is a substring that must appear in the target text.
Contains string `json:"contains,omitempty" yaml:"contains,omitempty" doc:"Substring that must appear" maxLength:"5000"`
// NotRegex is a regex pattern that must NOT match anywhere in the target text.
NotRegex string `json:"notRegex,omitempty" yaml:"notRegex,omitempty" doc:"Regex pattern that must NOT match" maxLength:"1000"`
// NotContains is a substring that must NOT appear in the target text.
NotContains string `json:"notContains,omitempty" yaml:"notContains,omitempty" doc:"Substring that must NOT appear" maxLength:"5000"`
// Target specifies which output stream to match against: stdout (default), stderr, or combined.
// Only applies to regex, contains, notRegex, notContains.
// CEL expressions access both via context variables.
Target string `` /* 227-byte string literal not displayed */
// Message is a custom failure message (optional).
Message string `json:"message,omitempty" yaml:"message,omitempty" doc:"Custom failure message" maxLength:"1000"`
}
Assertion validates command output. Exactly one of Expression, Regex, Contains, NotRegex, or NotContains must be set.
type AssertionResult ¶
type AssertionResult struct {
// Type is the assertion type (expression, regex, contains, notRegex, notContains).
Type string `json:"type"`
// Input is the assertion input (expression string, regex pattern, or substring).
Input string `json:"input"`
// Passed indicates whether the assertion passed.
Passed bool `json:"passed"`
// Message is the failure diagnostic message.
Message string `json:"message,omitempty"`
// Actual is the actual value encountered (for diagnostics).
Actual any `json:"actual,omitempty"`
}
AssertionResult captures the outcome of a single assertion.
func EvaluateAssertions ¶
func EvaluateAssertions(ctx context.Context, assertions []Assertion, cmdOutput *CommandOutput) []AssertionResult
EvaluateAssertions evaluates all assertions against the command output. It never short-circuits — all assertions are evaluated even if some fail.
type CommandBuilder ¶
CommandBuilder is a function that creates a cobra.Command for in-process execution. It receives IOStreams and an exit function and returns a configured root command. This indirection avoids an import cycle between soltesting and cmd/scafctl.
type CommandOutput ¶
type CommandOutput struct {
// Stdout is the raw stdout text.
Stdout string `json:"stdout"`
// Stderr is the raw stderr text.
Stderr string `json:"stderr"`
// ExitCode is the process exit code.
ExitCode int `json:"exitCode"`
// Output is the parsed JSON output. nil when the command doesn't support -o json.
Output map[string]any `json:"output,omitempty"`
// Files contains files created or modified in the sandbox during command execution.
Files map[string]FileInfo `json:"files"`
}
CommandOutput is the assertion context passed to CEL expressions.
type FileInfo ¶
type FileInfo struct {
// Exists is always true for entries in the map (present for consistency).
Exists bool `json:"exists"`
// Content is the full file content as a string.
Content string `json:"content"`
}
FileInfo represents a file created or modified in the sandbox.
type FilterOptions ¶
type FilterOptions struct {
// NamePatterns are glob patterns matched against test names.
// If a pattern contains "/", it is split into solution-glob/test-glob.
NamePatterns []string
// Tags filters tests that have at least one matching tag (any-match).
Tags []string
// SolutionPatterns are glob patterns matched against solution names.
SolutionPatterns []string
}
FilterOptions specifies how to filter discovered tests.
type GenerateInput ¶ added in v0.5.0
type GenerateInput struct {
// Command is the scafctl subcommand path, e.g. ["render", "solution"].
Command []string `json:"command" yaml:"command" doc:"scafctl subcommand path" maxItems:"10"`
// Args are the command arguments excluding the -f/--file and -o test flags,
// e.g. ["-r", "env=prod"]. The generator appends "-o", "json" automatically
// when no -o flag is already present, so the generated test will use structured
// output and populate __output for CEL assertions.
Args []string `json:"args,omitempty" yaml:"args,omitempty" doc:"Command arguments (without -f and -o)" maxItems:"50"`
// TestName is the desired test name. When empty, one is derived from Command
// and Args via DeriveTestName. Must match ^[a-zA-Z0-9][a-zA-Z0-9_-]*$.
TestName string `json:"testName,omitempty" yaml:"testName,omitempty" doc:"Test name override" maxLength:"100"`
// SnapshotDir is the directory where testdata/<name>.json is written.
// Defaults to "testdata" relative to the working directory.
SnapshotDir string `json:"snapshotDir,omitempty" yaml:"snapshotDir,omitempty" doc:"Directory for snapshot files" maxLength:"500"`
// Data is the command output as a parsed Go value used for assertion derivation.
// Typically produced by json.Unmarshal on the command's JSON output.
// When nil, no CEL assertions are derived.
Data any `json:"-" yaml:"-"`
// RawJSON is the raw JSON bytes written as the snapshot golden file.
// When empty, no snapshot file is written and the test case omits a snapshot field.
RawJSON []byte `json:"-" yaml:"-"`
}
GenerateInput holds the inputs for test case generation.
type GenerateResult ¶ added in v0.5.0
type GenerateResult struct {
// TestName is the (possibly derived) name of the test.
TestName string `json:"testName" yaml:"testName"`
// TestCase is the generated test case, ready to paste into spec.testing.cases.
TestCase *TestCase `json:"testCase" yaml:"testCase"`
// SnapshotPath is the relative (or absolute) path where the snapshot file was
// written. Empty when no snapshot was written.
SnapshotPath string `json:"snapshotPath,omitempty" yaml:"snapshotPath,omitempty"`
// SnapshotWritten is true when the snapshot file was created or updated on disk.
SnapshotWritten bool `json:"snapshotWritten" yaml:"snapshotWritten"`
}
GenerateResult holds the generated test case and snapshot metadata.
func Generate ¶ added in v0.5.0
func Generate(input *GenerateInput) (*GenerateResult, error)
Generate produces a TestCase definition from command output.
It:
- Derives a test name from Command + Args (unless TestName is set).
- Walks Data up to depth 2 and generates CEL assertions.
- Writes a normalized snapshot golden file to SnapshotDir/<name>.json.
- Returns a GenerateResult with the test case and snapshot metadata.
type InitStep ¶
type InitStep struct {
// Command is the command to execute. Supports POSIX shell syntax (pipes, redirections, variables).
Command string `json:"command" yaml:"command" doc:"Command to execute" maxLength:"1000"`
// Args contains additional arguments, automatically shell-quoted.
Args []string `json:"args,omitempty" yaml:"args,omitempty" doc:"Additional arguments" maxItems:"50"`
// Stdin provides standard input to the command.
Stdin string `json:"stdin,omitempty" yaml:"stdin,omitempty" doc:"Standard input to provide to the command" maxLength:"10000"`
// WorkingDir is the working directory relative to sandbox root.
WorkingDir string `json:"workingDir,omitempty" yaml:"workingDir,omitempty" doc:"Working directory (relative to sandbox root)" maxLength:"500"`
// Env contains environment variables merged with the parent process.
Env map[string]string `json:"env,omitempty" yaml:"env,omitempty" doc:"Environment variables merged with the parent process"`
// Timeout is the timeout in seconds (default: 30).
Timeout int `json:"timeout,omitempty" yaml:"timeout,omitempty" doc:"Timeout in seconds" maximum:"3600"`
// Shell specifies the shell interpreter: auto (default), sh, bash, pwsh, cmd.
Shell string `` /* 168-byte string literal not displayed */
}
InitStep is a setup/cleanup command. Uses the same input schema as the exec provider.
type ResultSummary ¶
type ResultSummary struct {
Passed int `json:"passed"`
Failed int `json:"failed"`
Errors int `json:"errors"`
Skipped int `json:"skipped"`
Total int `json:"total"`
Duration time.Duration `json:"duration"`
WallDuration time.Duration `json:"wallDuration,omitempty"`
}
ResultSummary holds aggregated counts for reporting.
func Summarize ¶
func Summarize(results []TestResult) ResultSummary
Summarize computes a ResultSummary from a slice of TestResults.
func (ResultSummary) ElapsedDuration ¶
func (s ResultSummary) ElapsedDuration() time.Duration
ElapsedDuration returns WallDuration if set, otherwise falls back to the summed individual Duration. Use this for summary display so that parallel runs show wall-clock time instead of cumulative CPU time.
type Runner ¶
type Runner struct {
// BinaryPath is the absolute path to the scafctl binary for subprocess
// execution. Each test's CLI command runs as an isolated child process,
// giving true parallelism and process-level env/state isolation.
// When empty, falls back to NewCommand (in-process, for unit tests only).
BinaryPath string
// Concurrency is the number of tests to run in parallel. 0 or 1 = sequential.
Concurrency int
// FailFast stops remaining tests for a solution on first failure.
FailFast bool
// UpdateSnapshots writes actual output to golden files instead of comparing.
UpdateSnapshots bool
// Verbose enables extra output (assertion counts, etc.).
Verbose bool
// KeepSandbox prevents cleanup of sandbox directories after tests.
KeepSandbox bool
// TestTimeout is the default timeout per test.
TestTimeout time.Duration
// GlobalTimeout is the overall timeout for all tests.
GlobalTimeout time.Duration
// DryRun validates tests without executing commands.
DryRun bool
// IOStreams provides input/output streams.
IOStreams *terminal.IOStreams
// Filter contains filter options to apply.
Filter FilterOptions
// NewCommand builds a root cobra.Command for in-process execution.
// Used only as a fallback when BinaryPath is empty (unit tests).
// Production code should always set BinaryPath instead.
NewCommand CommandBuilder
// Progress receives notifications as tests execute.
// When nil, no progress output is emitted.
Progress TestProgressCallback
}
Runner is the main test execution engine for functional tests.
func (*Runner) Run ¶
func (r *Runner) Run(ctx context.Context, solutions []SolutionTests) ([]TestResult, error)
Run orchestrates functional test execution across all solutions. It returns the results and a non-nil error only for infrastructure failures (not test failures — those are reflected in the results).
type Sandbox ¶
type Sandbox struct {
// contains filtered or unexported fields
}
Sandbox provides an isolated temporary directory for test execution. It supports file copying, pre/post snapshots for diff detection, and cleanup of temporary resources.
func NewBaseSandbox ¶
NewBaseSandbox creates a base sandbox for suite-level setup. The base sandbox can be copied per-test via CopyForTest.
func NewSandbox ¶
NewSandbox creates a new sandbox by copying the solution file, bundle files, and test-specific files into an isolated temporary directory. All paths are relative to solutionDir (the directory containing the solution file). Symlinks and path traversal above the solution root are rejected.
func (*Sandbox) Cleanup ¶
func (s *Sandbox) Cleanup()
Cleanup removes the sandbox temporary directory.
func (*Sandbox) CopyForTest ¶
CopyForTest creates a deep copy of the sandbox into a new temp directory and adds test-specific files from the original solution directory.
func (*Sandbox) PostSnapshot ¶
PostSnapshot diffs the current sandbox against the pre-snapshot and returns only new or modified files. Applies the 10MB size guard and binary file guard.
func (*Sandbox) PreSnapshot ¶
PreSnapshot records all file paths and modification times in the sandbox. Call this before executing the test command.
func (*Sandbox) SolutionPath ¶
SolutionPath returns the path to the solution file within the sandbox.
type ScaffoldInput ¶
type ScaffoldInput struct {
// Resolvers is the map of resolver definitions from the solution spec.
Resolvers map[string]*resolver.Resolver
// Workflow is the action workflow from the solution spec (may be nil).
Workflow *action.Workflow
// FileDependencies is a list of file paths (relative to the solution dir)
// discovered through static analysis of provider inputs.
// These are automatically populated onto generated test cases.
FileDependencies []string
}
ScaffoldInput provides the solution data needed for scaffold generation, avoiding a direct dependency on the solution package (which imports soltesting).
type ScaffoldResult ¶
type ScaffoldResult struct {
// Cases is a map of generated test cases keyed by name.
Cases map[string]*TestCase `json:"cases" yaml:"cases"`
}
ScaffoldResult holds the generated test scaffold for a solution.
func Scaffold ¶
func Scaffold(input *ScaffoldInput) *ScaffoldResult
Scaffold generates a skeleton test suite from the provided solution data. It performs structural analysis only — no commands are executed.
type ServiceConfig ¶ added in v0.5.0
type ServiceConfig struct {
// Name is a unique identifier for the service within this solution's tests.
Name string `json:"name" yaml:"name" doc:"Unique service identifier" maxLength:"100" pattern:"^[a-zA-Z0-9][a-zA-Z0-9_-]*$"`
// Type is the service type. Supported: "http" (mock HTTP server), "exec" (mock shell commands).
Type string `json:"type" yaml:"type" doc:"Service type" pattern:"^(http|exec)$" patternDescription:"Must be: http or exec"`
// PortEnv is the environment variable name where the assigned port is exposed.
// Tests can reference this via testConfig.env or in resolver inputs (e.g., $MOCK_HTTP_PORT).
// Only used when Type is "http".
PortEnv string `` /* 142-byte string literal not displayed */
// BaseURLEnv is the environment variable name where the base URL is exposed (e.g., http://127.0.0.1:PORT).
// Optional — if empty, only PortEnv is set. Only used when Type is "http".
BaseURLEnv string `json:"baseUrlEnv,omitempty" yaml:"baseUrlEnv,omitempty" doc:"Env var name for base URL (http only)" maxLength:"100"`
// Routes defines the mock HTTP routes (only used when Type is "http").
Routes []mockserver.Route `json:"routes,omitempty" yaml:"routes,omitempty" doc:"Mock HTTP routes" maxItems:"100"`
// ExecRules defines mock command rules (only used when Type is "exec").
// Rules are matched in order — the first matching rule wins.
ExecRules []mockexec.Rule `json:"execRules,omitempty" yaml:"execRules,omitempty" doc:"Mock command rules" maxItems:"100"`
// Passthrough allows unmatched exec commands to run normally (only used when Type is "exec").
// When false (default), unmatched commands fail with an error.
Passthrough bool `json:"passthrough,omitempty" yaml:"passthrough,omitempty" doc:"Allow unmatched exec commands to run (exec only)"`
}
ServiceConfig defines a background service for test infrastructure.
type SkipBuiltinsValue ¶
type SkipBuiltinsValue struct {
// All when true means skip all builtins.
All bool `json:"-" yaml:"-"`
// Names lists specific builtin names to skip.
Names []string `json:"-" yaml:"-"`
}
SkipBuiltinsValue supports both bool and []string via custom UnmarshalYAML. When bool: true skips all builtins, false skips none. When []string: skips only the named builtins (without "builtin:" prefix). Both UnmarshalYAML and MarshalYAML are required to survive the deepCopySolution YAML round-trip used in compose.
func (SkipBuiltinsValue) IsSkipped ¶
func (s SkipBuiltinsValue) IsSkipped() bool
IsSkipped returns true if the SkipBuiltinsValue indicates all builtins are skipped or if the value has specific builtin names listed.
func (SkipBuiltinsValue) MarshalJSON ¶
func (s SkipBuiltinsValue) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler for SkipBuiltinsValue.
func (SkipBuiltinsValue) MarshalYAML ¶
func (s SkipBuiltinsValue) MarshalYAML() (any, error)
MarshalYAML implements yaml.Marshaler for SkipBuiltinsValue.
func (*SkipBuiltinsValue) UnmarshalJSON ¶
func (s *SkipBuiltinsValue) UnmarshalJSON(data []byte) error
UnmarshalJSON implements json.Unmarshaler for SkipBuiltinsValue.
func (*SkipBuiltinsValue) UnmarshalYAML ¶
func (s *SkipBuiltinsValue) UnmarshalYAML(value *yaml.Node) error
UnmarshalYAML implements yaml.Unmarshaler for SkipBuiltinsValue.
type SkipValue ¶ added in v0.6.0
type SkipValue struct {
// Static when true means unconditionally skip.
Static bool `json:"-" yaml:"-"`
// Expression is a CEL expression for conditional skipping.
Expression celexp.Expression `json:"-" yaml:"-"`
}
SkipValue supports both bool and CEL expression string via custom UnmarshalYAML. When bool: true skips unconditionally, false means no skip. When string: the string is evaluated as a CEL expression at discovery time. YAML usage:
skip: true # unconditional skip skip: 'os == "windows"' # conditional skip via CEL
func (SkipValue) HasExpression ¶ added in v0.6.0
HasExpression returns true if the SkipValue has a CEL expression.
func (SkipValue) IsZero ¶ added in v0.6.0
IsZero returns true if the SkipValue is the zero value (no skip configured).
func (SkipValue) MarshalJSON ¶ added in v0.6.0
MarshalJSON implements json.Marshaler for SkipValue.
func (SkipValue) MarshalYAML ¶ added in v0.6.0
MarshalYAML implements yaml.Marshaler for SkipValue.
func (SkipValue) ShouldSkip ¶ added in v0.6.0
ShouldSkip returns true if the SkipValue indicates unconditional skip. For expression-based skip, use the runner's evaluation logic.
func (*SkipValue) UnmarshalJSON ¶ added in v0.6.0
UnmarshalJSON implements json.Unmarshaler for SkipValue.
type SolutionInfo ¶
type SolutionInfo struct {
// SolutionPath is the absolute path to the main solution file.
SolutionPath string
// ComposeFiles are the absolute paths of the compose files referenced by
// the solution's compose field.
ComposeFiles []string
}
SolutionInfo tracks the files associated with a single solution.
type SolutionTests ¶
type SolutionTests struct {
// SolutionName is the metadata.name from the solution.
SolutionName string `json:"solutionName"`
// Cases contains the test definitions keyed by test name.
Cases map[string]*TestCase `json:"cases"`
// Config holds the solution-level test configuration.
Config *TestConfig `json:"config,omitempty"`
// FilePath is the absolute path to the solution file.
FilePath string `json:"filePath"`
// DetectedFiles contains file patterns auto-detected from resolver specs
// (e.g., directory provider paths). Used to populate builtin test files.
DetectedFiles []string `json:"detectedFiles,omitempty"`
}
SolutionTests groups the tests extracted from a single solution file.
func DiscoverFromFile ¶
func DiscoverFromFile(filePath string) (*SolutionTests, error)
DiscoverFromFile parses a single solution file and extracts tests. Returns nil if the file has no tests defined.
func DiscoverSolutions ¶
func DiscoverSolutions(testsPath string) ([]SolutionTests, error)
DiscoverSolutions recursively finds solution files in the given path, parses their specs, and extracts tests. The path can be a file or directory.
func FilterTests ¶
func FilterTests(solutions []SolutionTests, opts FilterOptions) []SolutionTests
FilterTests applies the filter options to the discovered tests. All filter criteria are ANDed together. Returns a new slice with filtered results. Template tests (names starting with _) are excluded from results.
type Status ¶
type Status string
Status represents the outcome of a test.
const ( // StatusPass indicates the test passed. StatusPass Status = "pass" // StatusFail indicates the test failed (assertion failure). StatusFail Status = "fail" // StatusSkip indicates the test was skipped. StatusSkip Status = "skip" // StatusError indicates the test encountered an infrastructure/setup error. StatusError Status = "error" )
type TestCase ¶
type TestCase struct {
// Name is the test name (auto-set from map key).
Name string `` /* 245-byte string literal not displayed */
// Description is a human-readable test description.
Description string `json:"description" yaml:"description" doc:"Human-readable test description" maxLength:"500"`
// Command is the scafctl subcommand as an array (e.g., [render, solution]).
Command []string `json:"command,omitempty" yaml:"command,omitempty" doc:"scafctl subcommand as an array" maxItems:"10"`
// Args contains additional CLI flags appended after the command.
Args []string `json:"args,omitempty" yaml:"args,omitempty" doc:"Additional CLI flags appended after the command" maxItems:"50"`
// Extends lists names of test templates to inherit from. Applied left-to-right.
Extends []string `json:"extends,omitempty" yaml:"extends,omitempty" doc:"Names of test templates to inherit from" maxItems:"10"`
// Tags are labels for categorization and filtering.
Tags []string `json:"tags,omitempty" yaml:"tags,omitempty" doc:"Tags for categorization and filtering" maxItems:"20"`
// Env contains per-test environment variables.
Env map[string]string `json:"env,omitempty" yaml:"env,omitempty" doc:"Per-test environment variables"`
// Files lists relative paths, glob patterns, or directory paths for files required by this test.
// Globs (e.g., 'templates/**/*.yaml') are expanded using doublestar matching.
// Directories (e.g., 'templates/') are recursively expanded to include all files.
Files []string `` /* 207-byte string literal not displayed */
// Init contains setup steps executed sequentially before the command.
Init []InitStep `json:"init,omitempty" yaml:"init,omitempty" doc:"Setup steps executed sequentially before the command" maxItems:"50"`
// Cleanup contains teardown steps executed after the command, even on failure.
Cleanup []InitStep `` /* 128-byte string literal not displayed */
// Assertions validates command output. Required unless snapshot is set.
Assertions []Assertion `json:"assertions,omitempty" yaml:"assertions,omitempty" doc:"Output assertions" maxItems:"100"`
// Snapshot is a relative path to a golden file for normalized comparison.
Snapshot string `` /* 130-byte string literal not displayed */
// InjectFile controls whether the runner auto-injects -f <sandbox-solution-path>.
// Default is true. Set to false for commands that don't accept -f.
InjectFile *bool `json:"injectFile,omitempty" yaml:"injectFile,omitempty" doc:"When true, auto-inject -f <sandbox-solution-path>"`
// ExpectFailure when true, the test passes if the command exits non-zero.
ExpectFailure bool `json:"expectFailure,omitempty" yaml:"expectFailure,omitempty" doc:"When true, test passes if command exits non-zero"`
// ExitCode is the exact expected exit code. Mutually exclusive with ExpectFailure.
ExitCode *int `json:"exitCode,omitempty" yaml:"exitCode,omitempty" doc:"Exact expected exit code"`
// Timeout is the per-test timeout as a Go duration string.
Timeout *duration.Duration `json:"timeout,omitempty" yaml:"timeout,omitempty" doc:"Per-test timeout as a Go duration string"`
// Skip controls test skipping. Accepts true (unconditional) or a CEL expression string (conditional).
Skip SkipValue `` /* 137-byte string literal not displayed */
// SkipReason is a human-readable reason for skipping.
SkipReason string `json:"skipReason,omitempty" yaml:"skipReason,omitempty" doc:"Human-readable reason for skipping" maxLength:"500"`
// Retries is the number of retry attempts for a failing test.
Retries int `json:"retries,omitempty" yaml:"retries,omitempty" doc:"Number of retry attempts for failing tests" maximum:"10"`
}
TestCase defines a single functional test for a solution.
func BuiltinTests ¶
func BuiltinTests(testConfig *TestConfig) []*TestCase
BuiltinTests returns the builtin test cases, filtered by testConfig.skipBuiltins. If testConfig is nil, all builtins are returned.
func (*TestCase) GetInjectFile ¶
GetInjectFile returns the value of InjectFile, defaulting to true if not set.
func (*TestCase) IsTemplate ¶
IsTemplate returns true if this test is a template (name starts with _). Template tests are not executed directly but can be inherited via extends.
type TestConfig ¶
type TestConfig struct {
// SkipBuiltins disables builtin tests. true for all, or list of specific names.
SkipBuiltins SkipBuiltinsValue `json:"skipBuiltins,omitempty" yaml:"skipBuiltins,omitempty" doc:"Disable builtins: true for all, or list of specific names"`
// Env provides suite-level environment variables applied to all tests.
Env map[string]string `json:"env,omitempty" yaml:"env,omitempty" doc:"Suite-level environment variables applied to all tests"`
// Setup contains suite-level setup steps run once, then the resulting sandbox is copied per-test.
Setup []InitStep `` /* 129-byte string literal not displayed */
// Cleanup contains suite-level teardown steps run once after all tests complete, even on failure.
Cleanup []InitStep `` /* 148-byte string literal not displayed */
// Services defines background services started before tests and stopped after.
// Currently supports "http" type which starts a mock HTTP server.
Services []ServiceConfig `json:"services,omitempty" yaml:"services,omitempty" doc:"Background services started before tests" maxItems:"10"`
}
TestConfig holds solution-level test configuration.
type TestProgressCallback ¶
type TestProgressCallback interface {
// OnTestStart is called when an individual test begins execution.
// It is not called for tests that fail before execution (validation errors,
// dry runs, setup failures, etc.) — those only receive OnTestComplete.
OnTestStart(solution, test string)
// OnTestComplete is called when a test finishes with its result.
// This is called for every test, including those that were skipped or
// errored before execution began.
OnTestComplete(result TestResult)
// Wait blocks until all progress output has been flushed.
// Must be called after the last OnTestComplete before reading stdout output.
Wait()
}
TestProgressCallback receives notifications as test execution progresses. Implementations must be safe for concurrent use when runner concurrency > 1.
type TestResult ¶
type TestResult struct {
// Solution is the solution name.
Solution string `json:"solution"`
// Test is the test name.
Test string `json:"test"`
// Status is the test outcome.
Status Status `json:"status"`
// Duration is how long the test took.
Duration time.Duration `json:"duration"`
// Message provides details about skip, error, or failure.
Message string `json:"message,omitempty"`
// Assertions contains the results of each assertion evaluation.
Assertions []AssertionResult `json:"assertions,omitempty"`
// RetryAttempt indicates which retry attempt passed (0 = first attempt).
RetryAttempt int `json:"retryAttempt,omitempty"`
// SandboxPath is the sandbox directory path (when --keep-sandbox is set).
SandboxPath string `json:"sandboxPath,omitempty"`
}
TestResult captures the outcome of a single test.
type TestSuite ¶ added in v0.5.0
type TestSuite struct {
// Config holds solution-level test configuration.
Config *TestConfig `json:"config,omitempty" yaml:"config,omitempty" doc:"Test configuration"`
// Cases is a map of functional test definitions keyed by test name.
// Test names must be unique and must match ^[a-zA-Z0-9][a-zA-Z0-9_-]*$.
// Names starting with _ are templates that are not executed directly.
Cases map[string]*TestCase `json:"cases,omitempty" yaml:"cases,omitempty" doc:"Test case definitions keyed by name"`
}
TestSuite groups all test-related configuration under a single top-level property.
type WatchOptions ¶
type WatchOptions struct {
// DebounceDuration controls how long to wait after the last file change
// before triggering a re-run. Defaults to DefaultDebounceDuration.
DebounceDuration time.Duration
// OnRunStart is called just before each test re-run.
// The string argument identifies the triggering file (relative path when possible).
OnRunStart func(triggerFile string)
// OnRunComplete is called after each test re-run with the results.
OnRunComplete func(results []TestResult, elapsed time.Duration, err error)
}
WatchOptions configures the watch mode behaviour.
type WatchResult ¶
type WatchResult struct {
// TriggerFile is the file change that caused the re-run.
TriggerFile string
// Results contains the test outcomes.
Results []TestResult
// Elapsed is the wall-clock duration of the run.
Elapsed time.Duration
// Err is set when the run fails for infrastructure reasons.
Err error
}
WatchResult captures the outcome of a single watch-triggered test run.
type Watcher ¶
type Watcher struct {
// Runner is the test runner to use for re-runs.
Runner *Runner
// TestsPath is the path to the solution file or directory being tested.
TestsPath string
// Options configures watch behaviour.
Options WatchOptions
// contains filtered or unexported fields
}
Watcher monitors solution files for changes and re-runs affected tests.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package mockexec provides a configurable mock for shell command execution in functional tests.
|
Package mockexec provides a configurable mock for shell command execution in functional tests. |
|
Package mockserver provides a configurable HTTP mock server for functional testing.
|
Package mockserver provides a configurable HTTP mock server for functional testing. |