testrunner

package
v0.2.0-alpha.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 5, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

README

pkg/controller/testrunner

Pure component that executes the embedded validation tests under spec.validationTests in a HAProxyTemplateConfig. No EventBus dependency — it's a library that CLI and webhook paths both call directly.

What It Does

For each test case:

  1. Build a fixture-driven render context (the test's fixtures are injected as a parallel resource store, no cluster calls).
  2. Render every template in the config.
  3. Evaluate each assertion in the test (haproxy_valid, contains, not_contains, match_count, equals, jsonpath, match_order, deterministic). The haproxy_valid assertion type runs haproxy -c against the rendered output using the supplied ValidationPaths; other assertion types do not invoke the HAProxy binary.
  4. Collect timing, rendered content, and assertion results into a TestResult.

The whole suite runs in a worker pool (Options.Workers, defaults to runtime.NumCPU). TestResults aggregates pass/fail/skip counts and optional summary timings.

Minimal Usage

import (
    "context"
    "path/filepath"

    "gitlab.com/haproxy-haptic/haptic/pkg/controller/testrunner"
    "gitlab.com/haproxy-haptic/haptic/pkg/core/config"
    "gitlab.com/haproxy-haptic/haptic/pkg/dataplane"
    "gitlab.com/haproxy-haptic/haptic/pkg/templating"
)

cfg, _ := config.LoadConfig(configYAML)
config.SetDefaults(cfg)

engine, _ := templating.New(buildTemplates(cfg), nil, nil, nil)

paths := &dataplane.ValidationPaths{
    TempDir:           tempDir,                               // created + cleaned up by the caller
    MapsDir:           filepath.Join(tempDir, "maps"),
    SSLCertsDir:       filepath.Join(tempDir, "ssl"),
    CRTListDir:        filepath.Join(tempDir, "crt-list"),    // on HAProxy < 3.2 this may equal SSLCertsDir
    GeneralStorageDir: filepath.Join(tempDir, "general"),
    ConfigFile:        filepath.Join(tempDir, "haproxy.cfg"),
}

runner := testrunner.New(cfg, engine, paths, testrunner.Options{
    Workers:         0,            // 0 → NumCPU
    DebugFilters:    false,        // set for `--debug-filters`
    ProfileIncludes: false,        // set for `--profile-includes`
    Capabilities:    capabilities, // from dataplane.CapabilitiesFromVersion
    HAProxyVersion:  haproxyVer,
})

results, err := runner.RunTests(context.Background(), "") // "" = all tests

out, _ := testrunner.FormatResults(results, testrunner.OutputOptions{
    Format:       testrunner.OutputFormatSummary, // or OutputFormatJSON / OutputFormatYAML
    Verbose:      false,
    DumpRendered: false,
})
fmt.Println(out)

The HAProxy binary is looked up via exec.LookPath("haproxy") by pkg/dataplane — there's no field on ValidationPaths for it. If no binary is on $PATH, tests whose only assertion is haproxy_valid are skipped (not failed) and surface as SkipReason: "haproxy binary not found" in the results.

API Surface

func New(
    cfg *config.Config,
    engine templating.Engine,
    validationPaths *dataplane.ValidationPaths,
    options Options,
) *Runner

func (r *Runner) RunTests(ctx context.Context, testName string) (*TestResults, error)

func FormatResults(results *TestResults, options OutputOptions) (string, error)

testName == "" runs every test; otherwise the runner filters by exact name and returns a TestResults containing just that one (errors if the name doesn't exist). Context cancellation aborts in-flight tests and returns whatever completed.

Options

type Options struct {
    TestName        string                 // filter; empty = all tests (alternative to RunTests' second arg)
    Logger          *slog.Logger           // defaults to slog.Default()
    Workers         int                    // 0 → NumCPU; 1 → sequential
    DebugFilters    bool                   // wired to --debug-filters
    ProfileIncludes bool                   // wired to --profile-includes
    Capabilities    dataplane.Capabilities // value, not pointer; gates tests that need specific DP API features
    HAProxyVersion  *dataplane.Version
}

DebugFilters and ProfileIncludes are passed through to per-worker engine configuration — every worker gets its own templating.Engine clone so tracing and debug output don't cross-contaminate between parallel tests.

Result Types

  • TestResults — suite-level: totals, duration, slice of TestResult.
  • TestResult — per-test: name, pass/fail, duration, skip reason, rendered content (populated for --dump-rendered), auxiliary files, assertion results.
  • AssertionResult — per-assertion: type, target (haproxy.cfg, map:<name>, file:<name>, cert:<name>, crt-list:<name>, rendering_error), pass/fail, human-readable error message, target size in bytes, and a 200-char preview of the target on failure.

See pkg/controller/testrunner/types.go for the full schema; FormatResults can render any of them as summary, json, or yaml.

Concurrency Model

  • Each worker holds its own cloned templating.Engine (sharing the parent compiled templates but with independent trace/debug state).
  • Fixtures are in-memory overlays built per-test; no shared mutable state between tests.
  • RunTests fans tests out over a channel, fans results back in, and preserves deterministic ordering in the result slice.
  • Safe to call RunTests multiple times on the same Runner — per-run state (rendered content, stores) is re-created each invocation.

Error Simplification

Rendering and validation errors pass through dataplane.SimplifyRenderingError / dataplane.SimplifyValidationError before landing in TestResult.RenderError / TestResult.Error. This surfaces the user-relevant message (Service 'api' not found in namespace 'default') instead of the full Go error chain, matching what the admission webhook emits — so CLI and webhook failure output stay consistent.

See Also

License

Apache-2.0 — see root LICENSE.

Documentation

Overview

Package testrunner implements validation test execution for HAProxyTemplateConfig.

This package provides a test runner that executes embedded validation tests defined in HAProxyTemplateConfig CRDs. It can be used both by the CLI (controller validate command) and by the admission webhook for validation.

The test runner:

  • Creates resource stores from test fixtures
  • Renders templates with fixture context
  • Runs assertions against rendered output
  • Returns structured test results

This is a pure component with no EventBus dependency - it's called directly by the CLI and by the DryRunValidator component.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateHTTPStoreFromFixtures 

func CreateHTTPStoreFromFixtures(fixtures []config.HTTPResourceFixture, logger *slog.Logger) *httpstore.HTTPStore

CreateHTTPStoreFromFixtures creates an HTTPStore pre-populated with fixture content.

Parameters:

  • fixtures: HTTP fixtures from test definition
  • logger: Logger for debug messages

Returns:

  • HTTPStore with fixtures loaded as accepted content

func FormatResults 

func FormatResults(results *TestResults, options OutputOptions) (string, error)

FormatResults formats test results according to the specified options.

func MergeFixtures 

func MergeFixtures(globalFixtures, testFixtures map[string][]any) map[string][]any

MergeFixtures deep merges global fixtures with test fixtures by resource identity.

Resource identity is defined by: apiVersion + kind + namespace + name

Merge strategy:

  • Global fixtures are added first
  • Test fixtures override global fixtures when same resource identity exists
  • Resource types not in global fixtures are taken from test fixtures

Parameters:

  • globalFixtures: Fixtures from validationTests._global
  • testFixtures: Fixtures from specific test

Returns:

  • Merged fixtures map (resource type → list of resources)

func MergeHTTPFixtures 

func MergeHTTPFixtures(globalFixtures, testFixtures []config.HTTPResourceFixture) []config.HTTPResourceFixture

MergeHTTPFixtures merges global and test-specific HTTP fixtures.

Test-specific fixtures override global fixtures for the same URL.

Parameters:

  • globalFixtures: HTTP fixtures from _global test
  • testFixtures: HTTP fixtures from specific test

Returns:

  • Merged HTTP fixtures list

func SingularizeResourceType 

func SingularizeResourceType(plural string) string

SingularizeResourceType converts a plural resource type to singular kind.

This is a simple heuristic that handles common English pluralization rules. For proper kind resolution, use RESTMapper.KindFor().

Examples:

  • "services" → "Service"
  • "pods" → "Pod"
  • "configmaps" → "ConfigMap"
  • "namespaces" → "Namespace"

Types

type AssertionResult 

type AssertionResult struct {
	// Type is the assertion type (haproxy_valid, contains, etc).
	Type string

	// Description is the assertion description.
	Description string

	// Passed is true if the assertion passed.
	Passed bool

	// Error contains the failure message if assertion failed.
	Error string

	// Target is the assertion target (e.g., "haproxy.cfg", "map:path-prefix.map").
	Target string `json:"target,omitempty" yaml:"target,omitempty"`

	// TargetSize is the size of the target content in bytes.
	TargetSize int `json:"targetSize,omitempty" yaml:"targetSize,omitempty"`

	// TargetPreview is a preview of the target content (first 200 chars, only for failed assertions).
	TargetPreview string `json:"targetPreview,omitempty" yaml:"targetPreview,omitempty"`
}

AssertionResult contains the result of running a single assertion.

type FixtureHTTPStoreWrapper 

type FixtureHTTPStoreWrapper struct {
	// contains filtered or unexported fields
}

FixtureHTTPStoreWrapper wraps an HTTPStore pre-populated with fixtures for test execution.

Unlike the production HTTPStoreWrapper, this wrapper:

  • Returns content only for URLs that have fixtures loaded
  • Returns an error for URLs without fixtures (ensuring all HTTP dependencies are mocked)
  • Does NOT make actual HTTP requests

Template usage: 

{{ http.Fetch("http://blocklist.example.com/list.txt") }}

If the URL is not in fixtures, the template fails with a clear error message.

func NewFixtureHTTPStoreWrapper 

func NewFixtureHTTPStoreWrapper(store *httpstore.HTTPStore, logger *slog.Logger) *FixtureHTTPStoreWrapper

NewFixtureHTTPStoreWrapper creates a new fixture-only HTTP wrapper.

Parameters:

  • store: HTTPStore pre-populated with fixtures via LoadFixture()
  • logger: Logger for debug messages

func (*FixtureHTTPStoreWrapper) Fetch 

func (w *FixtureHTTPStoreWrapper) Fetch(args ...any) (any, error)

Fetch returns fixture content for a URL.

Template usage (same as production wrapper): 

{{ http.Fetch("http://example.com/data.txt") }}
{{ http.Fetch("http://example.com/data.txt", {"delay": "5m"}) }}

In fixture mode:

  • Options (delay, timeout, etc.) are ignored
  • Authentication is ignored
  • Only the URL is used to look up fixture content
  • Returns error if URL is not in fixtures

Returns:

  • Content string if URL has fixture
  • Error if URL is not in fixtures

type Options 

type Options struct {
	// TestName filters tests to run. If empty, all tests run.
	TestName string

	// Logger for structured logging. If nil, uses default logger.
	Logger *slog.Logger

	// Workers is the number of parallel workers for test execution.
	// Default (0): runtime.NumCPU().
	// Set to 1 for sequential execution.
	Workers int

	// DebugFilters enables detailed filter operation logging.
	// When enabled, each sort comparison is logged with values and results.
	DebugFilters bool

	// ProfileIncludes enables include timing profiling.
	// When enabled, shows which included templates take the most time.
	ProfileIncludes bool

	// Capabilities defines which features are available for the local HAProxy version.
	// Used to determine path resolution (e.g., CRT-list paths fallback when not supported).
	Capabilities dataplane.Capabilities

	// HAProxyVersion is the detected local HAProxy version.
	// When set, tests with MinHAProxyVersion above this version are skipped.
	HAProxyVersion *dataplane.Version

	// TypedResourceTypes is the per-resource generated Go type the
	// engine was constructed with via typebootstrap. Same shape as
	// renderer.RenderServiceConfig.TypedResourceTypes — see
	// pkg/controller/renderer/typed_resources.go for the production
	// counterpart. Passed straight through to
	// rendercontext.WithTypedResources so the offline render context
	// includes the typed top-level globals the engine compiled
	// against. When nil/empty, the test runner behaves as before
	// (no typed globals injected — chart code keeps using
	// resources.<name>.List() through dig()).
	TypedResourceTypes map[string]reflect.Type
}

Options configures the test runner.

type OutputFormat 

type OutputFormat string

OutputFormat specifies the output format for test results. 

const (
	// OutputFormatSummary outputs a human-readable summary.
	OutputFormatSummary OutputFormat = "summary"

	// OutputFormatJSON outputs structured JSON.
	OutputFormatJSON OutputFormat = "json"

	// OutputFormatYAML outputs structured YAML.
	OutputFormatYAML OutputFormat = "yaml"
)

type OutputOptions 

type OutputOptions struct {
	// Format specifies the output format (summary, json, yaml).
	Format OutputFormat

	// Verbose enables showing rendered content previews for failed assertions.
	Verbose bool
}

OutputOptions configures output formatting.

type RenderDependencies 

type RenderDependencies struct {
	Engine          templating.Engine
	Stores          map[string]stores.Store
	ValidationPaths *dataplane.ValidationPaths
	HTTPStore       *FixtureHTTPStoreWrapper
	CurrentConfig   *parserconfig.StructuredConfig
	ExtraContext    map[string]any
}

RenderDependencies holds all dependencies needed to re-render templates. This is used by the deterministic assertion to render again and compare.

type RenderOutput 

type RenderOutput struct {
	HAProxyConfig  string
	AuxiliaryFiles *dataplane.AuxiliaryFiles
	K8sResources   map[string]string
	StatusPatches  map[string]string
	IncludeStats   []templating.IncludeStats
}

RenderOutput bundles every artifact produced by a single test render so callers don't have to thread six positional return values.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes validation tests for HAProxyTemplateConfig.

It's a pure component with no EventBus dependency, designed to be called directly from the CLI or from the DryRunValidator.

func New 

func New(
	cfg *config.Config,
	engine templating.Engine,
	validationPaths *dataplane.ValidationPaths,
	options *Options,
) *Runner

New creates a new test runner.

Parameters:

  • cfg: The internal config containing templates and validation tests
  • engine: Pre-compiled template engine
  • validationPaths: Filesystem paths for HAProxy validation
  • options: Runner options (pointer to avoid hugeParam copy — the embedded Capabilities + new typed-resource-types map push the struct past 80 bytes). Pass nil to use defaults.

Returns:

  • A new Runner instance ready to execute tests

func (*Runner) CreateStoresFromFixtures 

func (r *Runner) CreateStoresFromFixtures(fixtures map[string][]any) (map[string]stores.Store, error)

CreateStoresFromFixtures creates resource stores from test fixtures.

This converts the test fixtures (map of resource type → list of resources) into resource stores that can be used for template rendering.

Implementation:

  • Phase 1: Create empty stores for ALL watched resources
  • Phase 2: Populate stores with fixture data where provided

This ensures templates can safely call .List() on any watched resource type, even if that resource type is not present in test fixtures.

Parameters:

  • fixtures: Map of resource type names to lists of Kubernetes resources

Returns:

  • Map of resource type names to resource stores
  • error if fixture processing fails

func (*Runner) RunTests 

func (r *Runner) RunTests(ctx context.Context, testName string) (*TestResults, error)

RunTests executes all validation tests (or a specific test if filtered).

This method:

  1. Filters tests if a specific test name was requested
  2. For each test: - Creates resource stores from fixtures - Renders HAProxy configuration - Runs all assertions
  3. Aggregates and returns results

Parameters:

  • ctx: Context for cancellation and timeouts

Returns:

  • TestResults containing results for all executed tests
  • error if a fatal error occurred (not test failures)

type TestResult 

type TestResult struct {
	// TestName is the name of the test.
	TestName string

	// Description is the test description.
	Description string

	// Passed is true if all assertions passed.
	Passed bool

	// Skipped is true if the test was skipped (e.g., HAProxy version too low).
	Skipped bool

	// SkipReason explains why the test was skipped.
	SkipReason string

	// Duration is the time taken to run this test.
	Duration time.Duration

	// Assertions contains results for each assertion.
	Assertions []AssertionResult

	// RenderError is set if template rendering failed.
	RenderError string

	// RenderedConfig contains the rendered HAProxy configuration (for --dump-rendered).
	RenderedConfig string `json:"renderedConfig,omitempty" yaml:"renderedConfig,omitempty"`

	// RenderedMaps contains rendered map files (for --dump-rendered).
	RenderedMaps map[string]string `json:"renderedMaps,omitempty" yaml:"renderedMaps,omitempty"`

	// RenderedFiles contains rendered general files (for --dump-rendered).
	RenderedFiles map[string]string `json:"renderedFiles,omitempty" yaml:"renderedFiles,omitempty"`

	// RenderedCerts contains rendered SSL certificates (for --dump-rendered).
	RenderedCerts map[string]string `json:"renderedCerts,omitempty" yaml:"renderedCerts,omitempty"`

	// RenderedK8sResources contains rendered output of every
	// `spec.k8sResources` template (template name → rendered YAML).
	// Asserted via `target: k8s:<template-name>`.
	RenderedK8sResources map[string]string `json:"renderedK8sResources,omitempty" yaml:"renderedK8sResources,omitempty"`

	// RenderedStatusPatches contains the JSON-marshalled status payload
	// emitted by every `statusPatch()` call during the haproxy.cfg
	// render, keyed by `<namespace>/<name>:<phase>` (cluster-scoped
	// resources use `:<phase>` with empty namespace). Asserted via
	// `target: status:<namespace>/<name>:<phase>`.
	RenderedStatusPatches map[string]string `json:"renderedStatusPatches,omitempty" yaml:"renderedStatusPatches,omitempty"`

	// IncludeStats contains timing statistics for included templates (for --profile-includes).
	IncludeStats []templating.IncludeStats `json:"includeStats,omitempty" yaml:"includeStats,omitempty"`
}

TestResult contains the result of running a single validation test.

type TestResults 

type TestResults struct {
	// TotalTests is the total number of tests executed (excluding skipped).
	TotalTests int

	// PassedTests is the number of tests that passed all assertions.
	PassedTests int

	// FailedTests is the number of tests with at least one failed assertion.
	FailedTests int

	// SkippedTests is the number of tests skipped (e.g., due to HAProxy version requirements).
	SkippedTests int

	// TestResults contains detailed results for each test.
	TestResults []TestResult

	// Duration is the total time taken to run all tests.
	Duration time.Duration
}

TestResults contains the results of running validation tests.

func (*TestResults) AllPassed 

func (r *TestResults) AllPassed() bool

AllPassed returns true if all tests passed.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.