testrunner

package
v0.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2026 License: Apache-2.0 Imports: 34 Imported by: 0

README

pkg/controller/testrunner

Pure component that executes the embedded validation tests under spec.validationTests in a HAProxyTemplateConfig. No EventBus dependency — it's a library that CLI and webhook paths both call directly.

What It Does

For each test case:

  1. Build a fixture-driven render context (the test's fixtures are injected as a parallel resource store, no cluster calls).
  2. Render every template in the config.
  3. Evaluate each assertion in the test (haproxy_valid, contains, not_contains, match_count, equals, jsonpath, match_order, deterministic). The haproxy_valid assertion type runs haproxy -c against the rendered output using the supplied ValidationPaths; other assertion types do not invoke the HAProxy binary.
  4. Collect timing, rendered content, and assertion results into a TestResult.

The whole suite runs in a worker pool (Options.Workers, defaults to the CPU and memory budget). TestResults aggregates pass/fail/skip counts and optional summary timings.

Minimal Usage

import (
    "context"
    "path/filepath"

    "github.com/haproxy-haptic/HAPTIC/pkg/controller/testrunner"
    "github.com/haproxy-haptic/HAPTIC/pkg/core/config"
    "github.com/haproxy-haptic/HAPTIC/pkg/dataplane"
    "github.com/haproxy-haptic/HAPTIC/pkg/templating"
)

cfg, _, _ := conversion.ParseCRD(unstructuredHAProxyTemplateConfig)
config.SetDefaults(cfg)

engine, _ := templating.New(buildTemplates(cfg), nil)

paths := &dataplane.ValidationPaths{
    TempDir:           tempDir,                               // created + cleaned up by the caller
    MapsDir:           filepath.Join(tempDir, "maps"),
    SSLCertsDir:       filepath.Join(tempDir, "ssl"),
    CRTListDir:        filepath.Join(tempDir, "crt-list"),    // on HAProxy < 3.2 this may equal SSLCertsDir
    GeneralStorageDir: filepath.Join(tempDir, "general"),
    ConfigFile:        filepath.Join(tempDir, "haproxy.cfg"),
}

runner := testrunner.New(cfg, engine, paths, &testrunner.Options{
    Workers:         0,            // 0 → CPU and memory budget
    DebugFilters:    false,        // set for `--debug-filters`
    ProfileIncludes: false,        // set for `--profile-includes`
    Capabilities:    capabilities, // from dataplane.CapabilitiesFromVersion
    HAProxyVersion:  haproxyVer,
})

results, err := runner.RunTests(context.Background(), "") // "" = all tests

out, _ := testrunner.FormatResults(results, testrunner.OutputOptions{
    Format:       testrunner.OutputFormatSummary, // or OutputFormatJSON / OutputFormatYAML
    Verbose:      false,
    DumpRendered: false,
})
fmt.Println(out)

The HAProxy binary is looked up via exec.LookPath("haproxy") by pkg/dataplane — there's no field on ValidationPaths for it. If no binary is on $PATH, tests whose only assertion is haproxy_valid are skipped (not failed) and surface as SkipReason: "haproxy binary not found" in the results.

API Surface

func New(
    cfg *config.Config,
    engine templating.Engine,
    validationPaths *dataplane.ValidationPaths,
    options *Options,
) *Runner

func (r *Runner) RunTests(ctx context.Context, testName string) (*TestResults, error)

func FormatResults(results *TestResults, options OutputOptions) (string, error)

testName == "" runs every test; otherwise the runner filters by exact name and returns a TestResults containing just that one (errors if the name doesn't exist). Context cancellation aborts in-flight tests and returns whatever completed.

Options

type Options struct {
    TestName        string                 // filter; empty = all tests (alternative to RunTests' second arg)
    Logger          *slog.Logger           // defaults to slog.Default()
    Workers         int                    // 0 → CPU and memory budget; 1 → sequential
    DebugFilters    bool                   // wired to --debug-filters
    ProfileIncludes bool                   // wired to --profile-includes
    Capabilities    dataplane.Capabilities // value, not pointer; gates tests that need specific DP API features
    HAProxyVersion  *dataplane.Version
}

DebugFilters and ProfileIncludes control diagnostic output for each render. Workers share the compiled template engine; render context and paths remain separate.

Result Types

  • TestResults — suite-level: totals, duration, slice of TestResult.
  • TestResult — per-test: name, pass/fail, duration, skip reason, rendered content (populated for --dump-rendered), auxiliary files, assertion results.
  • AssertionResult — per-assertion: type, target (haproxy.cfg, map:<name>, file:<name>, cert:<name>, crt-list:<name>, rendering_error), pass/fail, human-readable error message, target size in bytes, and a 200-char preview of the target on failure.

See pkg/controller/testrunner/types.go for the full schema; FormatResults can render any of them as summary, json, or yaml.

Concurrency Model

The automatic worker count is the smaller of twice GOMAXPROCS and one worker per 128 MiB of Go's soft memory limit, with a minimum of one. This lets workers render while others wait for HAProxy checks; simultaneous HAProxy processes remain capped at GOMAXPROCS. Explicit Options.Workers or haptic validate --workers values override the worker count. Every selected test and assertion still runs.

haproxy -c subprocesses consume container memory outside Go's soft limit. In a 1 GiB container, 32 validation workers reached 29 concurrent HAProxy processes using 415 MiB alongside the Go process and were OOM-killed. Eight workers completed the same suite with a 578 MiB container peak. The 128 MiB allowance bounds both rendering and subprocess concurrency; it is not a hard limit on an individual test. Larger fixtures can still require more memory.

  • Workers share the compiled templating.Engine and pass filter state through each render context.
  • Fixtures are in-memory overlays built per-test; no shared mutable state between tests.
  • RunTests fans tests out over a channel, fans results back in, and preserves deterministic ordering in the result slice.
  • Safe to call RunTests multiple times on the same Runner — per-run state (rendered content, stores) is re-created each invocation.

Error Simplification

Rendering and validation errors pass through templating.RenderErrorMessage / dataplane.SimplifyValidationError before landing in TestResult.RenderError / TestResult.Error. This surfaces the user-relevant message (a template's fail() text) instead of the full Go error chain, matching what the admission webhook emits — so CLI and webhook failure output stay consistent.

See Also

License

Apache-2.0 — see root LICENSE.

Documentation

Overview

Package testrunner implements validation test execution for HAProxyTemplateConfig.

This package provides a test runner that executes embedded validation tests defined in HAProxyTemplateConfig CRDs. It can be used both by the CLI (controller validate command) and by the admission webhook for validation.

The test runner:

  • Creates resource stores from test fixtures
  • Renders templates with fixture context
  • Runs assertions against rendered output
  • Returns structured test results

This is a pure component with no EventBus dependency - it's called directly by the CLI and by the DryRunValidator component.

Index

Constants

This section is empty.

Variables

View Source 
var SupportedAssertionTypes = []string{
	"haproxy_valid",
	"contains",
	"not_contains",
	"equals",
	"jsonpath",
	"match_count",
	"match_order",
	"not_exists",
	"deterministic",
}

runAssertion executes a single assertion. SupportedAssertionTypes is every assertion type runAssertion dispatches. The CRD's ValidationAssertion.Type enum must list exactly these: a type the enum omits makes the apiserver refuse the whole library object at apply time, which no offline gate sees.

Functions

func AssertionExtraContext 

func AssertionExtraContext(cfg *config.Config, test *config.ValidationTest) map[string]any

AssertionExtraContext is the extraContext a test's assertions render with: the config's testExtraContext, then _global, then the test's own. The deployment's extraContext never reaches it, the same way live resources never reach a fixture store.

func CreateHTTPStoreFromFixtures 

func CreateHTTPStoreFromFixtures(fixtures []config.HTTPResourceFixture, logger *slog.Logger) *httpstore.HTTPStore

CreateHTTPStoreFromFixtures creates an HTTPStore pre-populated with fixture content.

Parameters:

  • fixtures: HTTP fixtures from test definition
  • logger: Logger for debug messages

Returns:

  • HTTPStore with fixtures loaded as accepted content

func DeploymentExtraContext 

func DeploymentExtraContext(cfg *config.Config, test *config.ValidationTest) map[string]any

DeploymentExtraContext is what the deployment check renders a test's fixtures with: the deployment's extraContext beneath _global and the test's own. _global stays on top because it binds names to fixtures (the default certificate) that the deployment's values point elsewhere.

func ExtraContextOptions 

func ExtraContextOptions(extraContext map[string]any) ([]rendercontext.Option, error)

ExtraContextOptions hands a test's extraContext to the context builder. A test simulates an admission with extraContext.renderMode and extraContext.admissionSubject; the builder owns both globals and would otherwise overwrite the promoted keys.

func FormatResults 

func FormatResults(results *TestResults, options OutputOptions) (string, error)

FormatResults formats test results according to the specified options.

func MergeFixtures 

func MergeFixtures(globalFixtures, testFixtures map[string][]any) map[string][]any

MergeFixtures deep merges global fixtures with test fixtures by resource identity.

Resource identity is defined by: apiVersion + kind + namespace + name

Merge strategy:

  • Global fixtures are added first
  • Test fixtures override global fixtures when same resource identity exists
  • Resource types not in global fixtures are taken from test fixtures

Parameters:

  • globalFixtures: Fixtures from validationTests._global
  • testFixtures: Fixtures from specific test

Returns:

  • Merged fixtures map (resource type → list of resources)

func MergeHTTPFixtures 

func MergeHTTPFixtures(globalFixtures, testFixtures []config.HTTPResourceFixture) []config.HTTPResourceFixture

MergeHTTPFixtures merges global and test-specific HTTP fixtures.

Test-specific fixtures override global fixtures for the same URL.

Parameters:

  • globalFixtures: HTTP fixtures from _global test
  • testFixtures: HTTP fixtures from specific test

Returns:

  • Merged HTTP fixtures list

func SingularizeResourceType 

func SingularizeResourceType(plural string) string

SingularizeResourceType converts a plural resource type to singular kind.

This is a simple heuristic that handles common English pluralization rules. For proper kind resolution, use RESTMapper.KindFor().

Examples:

  • "services" → "Service"
  • "pods" → "Pod"
  • "configmaps" → "ConfigMap"
  • "namespaces" → "Namespace"

Types

type AssertionResult 

type AssertionResult struct {
	// ConfigRejected distinguishes a checked invalid input from an unavailable checker.
	ConfigRejected bool `json:"configRejected,omitempty" yaml:"configRejected,omitempty"`
	// Type is the assertion type (haproxy_valid, contains, etc).
	Type string

	// Description is the assertion description.
	Description string

	// Passed is true if the assertion passed.
	Passed bool

	// Error contains the failure message if assertion failed.
	Error string

	// Target is the assertion target (e.g., "haproxy.cfg", "map:path-prefix.map").
	Target string `json:"target,omitempty" yaml:"target,omitempty"`

	// TargetSize is the size of the target content in bytes.
	TargetSize int `json:"targetSize,omitempty" yaml:"targetSize,omitempty"`

	// TargetPreview is a preview of the target content (first 200 chars, only for failed assertions).
	TargetPreview string `json:"targetPreview,omitempty" yaml:"targetPreview,omitempty"`
	// contains filtered or unexported fields
}

AssertionResult contains the result of running a single assertion.

type FixtureHTTPStoreWrapper 

type FixtureHTTPStoreWrapper struct {
	// contains filtered or unexported fields
}

FixtureHTTPStoreWrapper wraps an HTTPStore pre-populated with fixtures for test execution.

Unlike the production HTTPStoreWrapper, this wrapper:

  • Returns content only for URLs that have fixtures loaded
  • Returns an error for URLs without fixtures (ensuring all HTTP dependencies are mocked)
  • Does NOT make actual HTTP requests

Template usage: 

{{ http.Fetch("http://blocklist.example.com/list.txt") }}

If the URL is not in fixtures, the template fails with a clear error message.

func NewFixtureHTTPStoreWrapper 

func NewFixtureHTTPStoreWrapper(store *httpstore.HTTPStore, logger *slog.Logger) *FixtureHTTPStoreWrapper

NewFixtureHTTPStoreWrapper creates a new fixture-only HTTP wrapper.

Parameters:

  • store: HTTPStore pre-populated with fixtures via LoadFixture()
  • logger: Logger for debug messages

func (*FixtureHTTPStoreWrapper) Fetch 

func (w *FixtureHTTPStoreWrapper) Fetch(args ...any) (any, error)

Fetch returns fixture content for a URL.

Options and authentication are validated exactly as the production wrapper validates them, so an invalid declaration fails the test, but only the URL selects the fixture; nothing is sent anywhere.

func (*FixtureHTTPStoreWrapper) Pending 

func (w *FixtureHTTPStoreWrapper) Pending(url string) bool

Pending reports a fixture declared pending.

func (*FixtureHTTPStoreWrapper) WithPending 

func (w *FixtureHTTPStoreWrapper) WithPending(fixtures []config.HTTPResourceFixture) *FixtureHTTPStoreWrapper

WithPending marks the fixtures declared pending: Fetch returns empty content for them and Pending reports true.

type Options 

type Options struct {
	// TestName filters tests to run. If empty, all tests run.
	TestName string

	// Logger for structured logging. If nil, uses default logger.
	Logger *slog.Logger

	// Workers is the number of parallel workers for test execution.
	// Default (0): twice GOMAXPROCS, capped at one worker per 128 MiB of GOMEMLIMIT.
	// Set to 1 for sequential execution.
	Workers int

	// DebugFilters enables detailed filter operation logging.
	// When enabled, each sort comparison is logged with values and results.
	DebugFilters bool

	// ProfileIncludes enables include timing profiling.
	// When enabled, shows which included templates take the most time.
	ProfileIncludes bool

	// Capabilities defines which features are available for the local HAProxy version.
	// Used to determine path resolution (e.g., CRT-list paths fallback when not supported).
	Capabilities dataplane.Capabilities

	// HAProxyVersion is the detected local HAProxy version.
	// When set, tests with MinHAProxyVersion above this version are skipped.
	HAProxyVersion *dataplane.Version

	// CheckWithoutBinary answers `haproxy_valid` where there is neither a
	// writable filesystem nor the haproxy binary — the browser (WASM)
	// playground. Non-nil also means per-test temp directories are not created.
	// Whatever it checks is weaker than `haproxy -c`, so callers must label the
	// results as what their check actually covers.
	CheckWithoutBinary func(haproxyConfig string) error

	// TypedResourceTypes is the per-resource generated Go type the
	// engine was constructed with via typebootstrap. Same shape as
	// renderer.RenderServiceConfig.TypedResourceTypes — see
	// pkg/controller/renderer/typed_resources.go for the production
	// counterpart. Passed straight through to
	// rendercontext.WithTypedResources so the offline render context
	// includes the typed top-level globals the engine compiled
	// against. When nil/empty, the test runner behaves as before
	// (no typed globals injected — chart code keeps using
	// resources.<name>.List() through dig()).
	TypedResourceTypes map[string]reflect.Type
}

Options configures the test runner.

type OutputFormat 

type OutputFormat string

OutputFormat specifies the output format for test results. 

const (
	// OutputFormatSummary outputs a human-readable summary.
	OutputFormatSummary OutputFormat = "summary"

	// OutputFormatJSON outputs structured JSON.
	OutputFormatJSON OutputFormat = "json"

	// OutputFormatYAML outputs structured YAML.
	OutputFormatYAML OutputFormat = "yaml"
)

type OutputOptions 

type OutputOptions struct {
	// Format specifies the output format (summary, json, yaml).
	Format OutputFormat

	// Verbose enables showing rendered content previews for failed assertions.
	Verbose bool
}

OutputOptions configures output formatting.

type RenderDependencies 

type RenderDependencies struct {
	Engine          templating.Engine
	Stores          map[string]stores.Store
	ValidationPaths *dataplane.ValidationPaths
	HTTPStore       *FixtureHTTPStoreWrapper
	CurrentConfig   *renderplan.CurrentConfig
	CurrentFiles    map[string]string
	ExtraContext    map[string]any
}

RenderDependencies holds all dependencies needed to re-render templates. This is used by the deterministic assertion to render again and compare.

type RenderOutput 

type RenderOutput struct {
	HAProxyConfig  string
	AuxiliaryFiles *dataplane.AuxiliaryFiles
	K8sResources   map[string]string
	StatusPatches  map[string]string
	// Events is the newline-joined serialization of the Kubernetes Events the
	// templates recorded via recordEvent(), one per line, so validation tests
	// can assert on them with the `target: events` resolver.
	Events       string
	IncludeStats []templating.IncludeStats

	// Plan is what the templates declared about this render, built exactly as
	// the production renderer builds it.
	Plan *renderplan.Plan
}

RenderOutput bundles every artifact produced by a single test render so callers don't have to thread six positional return values.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes validation tests for HAProxyTemplateConfig.

It's a pure component with no EventBus dependency, designed to be called directly from the CLI or from the DryRunValidator.

func New 

func New(
	cfg *config.Config,
	engine templating.Engine,
	validationPaths *dataplane.ValidationPaths,
	options *Options,
) *Runner

New creates a new test runner.

Parameters:

  • cfg: The internal config containing templates and validation tests
  • engine: Pre-compiled template engine
  • validationPaths: Filesystem paths for HAProxy validation
  • options: Runner options (pointer to avoid hugeParam copy — the embedded Capabilities + new typed-resource-types map push the struct past 80 bytes). Pass nil to use defaults.

Returns:

  • A new Runner instance ready to execute tests

func (*Runner) CreateStoresFromFixtures 

func (r *Runner) CreateStoresFromFixtures(fixtures map[string][]any) (map[string]stores.Store, error)

CreateStoresFromFixtures creates resource stores from test fixtures.

This converts the test fixtures (map of resource type → list of resources) into resource stores that can be used for template rendering.

Implementation:

  • Phase 1: Create empty stores for ALL watched resources
  • Phase 2: Populate stores with fixture data where provided

This ensures templates can safely call .List() on any watched resource type, even if that resource type is not present in test fixtures.

Parameters:

  • fixtures: Map of resource type names to lists of Kubernetes resources

Returns:

  • Map of resource type names to resource stores
  • error if fixture processing fails

func (*Runner) Render 

func (r *Runner) Render(ctx context.Context, testName string) (RenderOutput, error)

Render renders one validation test and returns everything the render produced — including the plan the templates declared — without executing the test's assertions. It renders with the deployment's extraContext, so a diff shows what a config change does to the test's scenario. It renders into the runner's base validation paths, which nothing writes to while no assertion runs.

func (*Runner) RenderWithoutFixtures 

func (r *Runner) RenderWithoutFixtures(ctx context.Context) (RenderOutput, error)

RenderWithoutFixtures renders the configuration against empty stores: no watched resource exists, no HTTP fixture answers, nothing was deployed before. It is the configuration's own skeleton — what `haptic diff` compares when the operator names no validationTest, so the answer is about the configuration change rather than about one test's fixtures.

func (*Runner) RunNamedTests 

func (r *Runner) RunNamedTests(ctx context.Context, names []string) (*TestResults, error)

RunNamedTests runs every selected test once; an empty selection runs all tests.

func (*Runner) RunTests 

func (r *Runner) RunTests(ctx context.Context, testName string) (*TestResults, error)

RunTests runs one named test, or all tests when testName is empty.

type TestResult 

type TestResult struct {
	// TestName is the name of the test.
	TestName string

	// Description is the test description.
	Description string

	// Passed is true if all assertions passed.
	Passed bool

	// Skipped is true if the test was skipped (e.g., HAProxy version too low).
	Skipped bool

	// SkipReason explains why the test was skipped.
	SkipReason string

	// Duration is the time taken to run this test.
	Duration time.Duration

	// Assertions contains results for each assertion.
	Assertions []AssertionResult

	// RenderError is set if template rendering failed.
	RenderError string

	// RenderedConfig contains the rendered HAProxy configuration (for --dump-rendered).
	RenderedConfig string `json:"renderedConfig,omitempty" yaml:"renderedConfig,omitempty"`

	// RenderedMaps contains rendered map files (for --dump-rendered).
	RenderedMaps map[string]string `json:"renderedMaps,omitempty" yaml:"renderedMaps,omitempty"`

	// RenderedFiles contains rendered general files (for --dump-rendered).
	RenderedFiles map[string]string `json:"renderedFiles,omitempty" yaml:"renderedFiles,omitempty"`

	// RenderedCerts contains rendered SSL certificates (for --dump-rendered).
	RenderedCerts map[string]string `json:"renderedCerts,omitempty" yaml:"renderedCerts,omitempty"`

	// RenderedK8sResources contains rendered output of every
	// `spec.k8sResources` template (template name → rendered YAML).
	// Asserted via `target: k8s:<template-name>`.
	RenderedK8sResources map[string]string `json:"renderedK8sResources,omitempty" yaml:"renderedK8sResources,omitempty"`

	// RenderedStatusPatches contains the JSON-marshalled status payload
	// emitted by every `statusPatch()` call during the haproxy.cfg
	// render, keyed by `<namespace>/<name>:<phase>` (cluster-scoped
	// resources use `:<phase>` with empty namespace). Asserted via
	// `target: status:<namespace>/<name>:<phase>`.
	RenderedStatusPatches map[string]string `json:"renderedStatusPatches,omitempty" yaml:"renderedStatusPatches,omitempty"`

	// IncludeStats contains timing statistics for included templates (for --profile-includes).
	IncludeStats []templating.IncludeStats `json:"includeStats,omitempty" yaml:"includeStats,omitempty"`
}

TestResult contains the result of running a single validation test.

type TestResults 

type TestResults struct {
	// TotalTests is the total number of tests executed (excluding skipped).
	TotalTests int

	// PassedTests is the number of tests that passed all assertions.
	PassedTests int

	// FailedTests is the number of tests with at least one failed assertion.
	FailedTests int

	// SkippedTests is the number of tests skipped (e.g., due to HAProxy version requirements).
	SkippedTests int

	// TestResults contains detailed results for each test.
	TestResults []TestResult

	// Duration is the total time taken to run all tests.
	Duration time.Duration
}

TestResults contains the results of running validation tests.

func (*TestResults) AllPassed 

func (r *TestResults) AllPassed() bool

AllPassed returns true if all tests passed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.