harness

package
v0.7.21 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 21, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package harness holds the scenario/capability model, the use-case registry, and the thin SDK client wrapper shared by every integration test.

The registry is the single source of truth the report generator joins against to decide, per scenario, whether a use case PASSED, FAILED, was SKIPPED (scenario capabilities don't satisfy it) or is PENDING (no test implemented yet). Keeping IDs + required capabilities here — not scattered across test files — is what lets report/gen.go produce the coverage matrix without re-deriving intent from test names.

Index

Constants

View Source
const (
	EnvBaseURL  = "AEROL_BASE_URL" // e.g. https://abc.itest.example.com or http://localhost:21212
	EnvPAT      = "AEROL_PAT"
	EnvScenario = "AEROL_SCENARIO"
	EnvCaps     = "AEROL_CAPS"   // path to the scenario caps.yml
	EnvDomain   = "AEROL_DOMAIN" // leased apex domain; empty in local-mode
	// EnvCustomDomain is a real hostname OUTSIDE the deployment base domain that
	// the operator has already provisioned for custom-domain tests (UC-35/36):
	// a `_aerol-verify.<host>` TXT record proving ownership and a CNAME pointing
	// at ingress. Only scenarios advertising CapExternalDNSZone export it.
	EnvCustomDomain = "AEROL_CUSTOM_DOMAIN"
)

Env var names the harness reads. run.sh exports these after provisioning.

View Source
const DefaultImage = "alpine:3.20"

DefaultImage is the small, always-available image used by lifecycle/network use cases that don't care what runs inside.

Variables

AllRuntimes is the five sandbox runtimes for "all" rows.

View Source
var CatalogueRegistry = buildCatalogueRegistry()

CatalogueRegistry is the 287-row master list (range IDs expanded).

View Source
var Registry = []UseCase{}/* 111 elements not displayed */

Registry is the full use-case catalogue. Order is the matrix row order.

Every use case now has a test (Phases 1-2 fanned out the full suite), so all are Implemented=true. A use case a given scenario can't satisfy SKIPs (reported not-applicable, not PENDING) via harness.Require; one that needs an out-of-band fixture (a staged wasm module, fault injection, registry creds) t.Skips with a reason. PENDING now only appears if a brand-new UC is added to this registry without a test — which is exactly the gap signal we want.

Functions

func CatalogueCategoryOrder

func CatalogueCategoryOrder() []string

CatalogueCategoryOrder returns stable category ordering for reports.

func DisruptiveAllowed

func DisruptiveAllowed() bool

DisruptiveAllowed reports whether fault-injection tests (node kill, etc.) may run. cluster-hetero enables this by default via integration-tests/run.sh; pass --no-disruptive or use make integration-cluster-hetero-safe to opt out.

func EC2InstanceState

func EC2InstanceState(t *testing.T, instanceID string) string

EC2InstanceState runs `aws ec2 describe-instances` for one instance id.

func FormatScenarios

func FormatScenarios(tags []ScenarioTag) string

FormatScenarios renders scenario tags for markdown tables.

func HostHasAEROLVMUserJump

func HostHasAEROLVMUserJump(t *testing.T, node IntegrationNode) bool

HostHasAEROLVMUserJump reports whether filter FORWARD jumps to AEROLVM-USER. Works for both iptables-nft and legacy (iptables -S).

func RestartSystemdUnit

func RestartSystemdUnit(t *testing.T, node IntegrationNode, unit string)

RestartSystemdUnit restarts a unit on the node and waits until it is active.

func RestartSystemdUnitOnAll

func RestartSystemdUnitOnAll(t *testing.T, targets *IntegrationTargets, unit string)

RestartSystemdUnitOnAll restarts unit on every SSH-reachable node.

func SSHRun

func SSHRun(t *testing.T, target, script string) (string, error)

SSHRun runs a remote shell command via SSH. Returns combined stdout/stderr.

func SSHTarget

func SSHTarget(n IntegrationNode) (string, bool)

SSHTarget builds user@host for a provisioned node. Prefers PublicIP.

func SSHUser

func SSHUser() string

SSHUser returns the OS login used for host-side integration SSH. Matches run.sh / common.sh (ubuntu@…).

func SetEC2InstanceRunning

func SetEC2InstanceRunning(t *testing.T, instanceID string, running bool)

SetEC2InstanceRunning stops or starts an EC2 instance via the AWS CLI. region/profile follow the ambient AWS_* env (same as terraform apply).

func SimIDForCatalogue

func SimIDForCatalogue(catalogueID string) string

SimIDForCatalogue maps catalogue IDs to simulation package IDs when a live sim exists (priority subset); empty means registry-only / stub.

func UniqueName

func UniqueName(sc *Scenario, t *testing.T) string

UniqueName returns a deployment-unique sandbox name for a test, so parallel or repeated runs never collide: "<scenario>-<test>-<unixnano>".

Types

type Capability

type Capability string

Capability is a property a deployment scenario either has or doesn't. A use case lists the capabilities it needs; a scenario that lacks one causes the use case to SKIP there (reported as not-applicable, not failed).

const (
	CapDocker      Capability = "docker"      // docker runtime available
	CapFirecracker Capability = "firecracker" // a firecracker-capable worker
	CapGvisor      Capability = "gvisor"      // a gvisor-capable worker
	CapWasm        Capability = "wasm"        // wasm runtime + staged module
	// CapIsolate gates the V8-isolate (workerd) use cases (UC-103..105). Like
	// CapGvisor it is advertisement-only: the runtime is enabled purely by
	// provisioning (default_with_isolate=true → install.sh --with-isolate writes
	// SB_ENABLE_ISOLATE=true + installs workerd), not by any run.sh config-overlay
	// flip. A scenario advertises it when its node was provisioned --with-isolate;
	// where the runtime is off, the isolate UCs skip (not-applicable) rather than
	// hard-fail. Unlike CapWasm it needs no node-side module staging — the UCs
	// upload JS bundles over POST /v1/js-bundles at runtime.
	CapIsolate Capability = "isolate" // V8-isolate (workerd) runtime available
	CapGPU     Capability = "gpu"     // a GPU worker
	CapDomain  Capability = "domain"  // public domain + TLS (not local-mode)
	CapCluster Capability = "cluster" // multi-node cluster (raft/forwarding)
	// CapMixedArchNegative gates UC-79: inject a foreign-arch snapshot ref and
	// assert the arm64 cluster refuses to resume it.
	CapMixedArchNegative Capability = "mixed-arch-negative"
	// CapCustomDomains gates the custom-domain attach path (UC-35/36). It's
	// distinct from CapDomain: a deployment can have a public domain + TLS yet
	// run with SB_ENABLE_CUSTOM_DOMAINS=false, in which case the API rejects
	// AddCustomDomain. Only scenarios that opt the feature on (and provision the
	// env) advertise this, so the custom-domain UCs skip instead of hard-failing
	// where the feature is off.
	CapCustomDomains Capability = "custom-domains"
	// CapExternalDNSZone gates the custom-domain verification path (UC-35/36).
	// A real custom domain must, by design, live OUTSIDE the deployment base
	// domain (the API rejects hosts under the wildcard zone) AND prove ownership
	// via a `_aerol-verify.<host>` TXT record before attach succeeds. Satisfying
	// that requires the harness to control a second DNS zone it can provision
	// verification (and CNAME) records in — the leased base zone is not enough.
	// Scenarios advertise this only when they wire up such a zone; otherwise the
	// custom-domain reachability UCs skip instead of failing against a gate they
	// structurally cannot pass.
	CapExternalDNSZone Capability = "external-dns-zone"
	// CapPlatformVolumes gates the platform-volume UCs (UC-81..UC-84). A
	// deployment advertises it only when the operator has enabled platform
	// volumes AND configured a shared backend (SB_PLATFORM_VOLUMES_ENABLED=true
	// + an S3 bucket or NFS export). The tests are backend-agnostic — they
	// attach by name, write, and read back — so the same UCs run whether the
	// scenario configured S3 or NFS. Where the feature is off, the UCs skip
	// (not-applicable) rather than fail against a 412.
	CapPlatformVolumes Capability = "platform-volumes"
	// CapBenchmark gates the create-benchmark UCs (UC-94/UC-95). It is
	// deliberately separate from the runtime capabilities: the benchmark is
	// slow and provisions many sandboxes (the density probe runs until the
	// fleet rejects on capacity), so it must be opt-in even on a scenario that
	// otherwise has every runtime. Only scenarios that explicitly advertise it
	// — currently cluster-hetero — run the benchmark; everywhere else the UCs
	// skip (not-applicable) instead of inflating cost on a normal pass.
	CapBenchmark Capability = "benchmark"
	// CapDockerPool is consumed by run.sh's config overlay, not by any UC's
	// Requires: it flips docker.pool.enabled in the scenario's cluster.yml so
	// UC-94 measures the warm-hit create path. It is deliberately separate
	// from CapDocker because each parked slot holds a default-shaped capacity
	// reservation, lowering the UC-95 density ceiling — scenarios opt in
	// (plans/docker-warm-pool.md §9 documents the adjusted gates).
	CapDockerPool Capability = "docker-pool"
	// CapDockerNetnsPool is likewise consumed by run.sh's config overlay: it
	// flips docker.netns_pool.enabled so cold docker creates adopt prepaid
	// pause-container network namespaces. Separate from CapDockerPool because
	// the two pools are independent (netns slots hold no capacity
	// reservations, so this one leaves the UC-95 density gate untouched).
	CapDockerNetnsPool Capability = "docker-netns-pool"
	// CapDockerEngine is consumed by run.sh's engine selection, not by any UC's
	// Requires. It is an OPT-OUT: containerd is the DEFAULT engine for every
	// scenario (run.sh writes SB_CONTAINER_ENGINE=containerd), and only a
	// scenario advertising this cap runs dockerd instead. Exactly two do — the
	// local dev install (local-mode) and the docker A/B benchmark baseline
	// (cluster-3-mixed-docker, kept on docker so it stays a valid comparison
	// against cluster-3-mixed-containerd). This encodes the target end-state of
	// the docker->containerd migration: "local install uses docker, every real
	// deployment uses containerd."
	CapDockerEngine Capability = "docker-engine"
	// CapContainerdEngine does NOT select the engine — that is now the default
	// (see CapDockerEngine). It gates the containerd-SPECIFIC coverage: the
	// Phase 5 soak/coexistence UCs (UC-99..102, plans/containerd-engine.md
	// §6/§8) and the distinctly-labeled `containerd` benchmark row + density.
	// Advertised only on the dedicated containerd validation scenarios
	// (single-node-containerd, cluster-3-mixed-containerd) so UC-102's
	// dockerd-coexistence restart stays off the runtime/metal scenarios where a
	// pure-containerd host makes it not-applicable.
	CapContainerdEngine Capability = "containerd-engine"
	// CapObservability gates UC-106/107 ("Grafana up", "Prometheus sees all
	// sandboxd nodes") and tells run.sh to pass -var deploy_obs=true so
	// Terraform/obs.tf provisions the dedicated obs EC2. Advertisement +
	// provisioning only — same shape as CapGvisor/CapIsolate.
	CapObservability Capability = "observability"
	// CapSimulations gates the suite/sims workload catalogue and UC-108
	// (per-sim pass/fail). Opt-in like CapBenchmark: slow, provisions long-
	// lived services, and needs AEROL_SIMS=1. UC-108 must never roll up to a
	// single "all green" — each sim records independently.
	CapSimulations Capability = "simulations"
)

type CatalogueRow

type CatalogueRow struct {
	ID          string
	Question    string
	Category    string
	Subcategory string
	Runtimes    []Runtime
	Scenarios   []ScenarioTag
	UCRef       string
	SignalDesc  string
}

CatalogueRow is one investor question from benchmark-usecase-catalogue.md. UCRef rows store only Question/Category/UCRef; runtime and signal may be derived from harness.Registry at report time (CQ-2).

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client wraps the Go SDK so every test shares one configured client and the resource-hygiene helpers (unique naming + guaranteed cleanup). Using the real SDK is deliberate: the suite doubles as Go-SDK integration coverage.

func NewClient

func NewClient(t *testing.T, sc *Scenario) *Client

NewClient builds a Client for the scenario. Fails the test immediately if the SDK can't be constructed — there's no point continuing without an API client.

func (*Client) BaseURL

func (c *Client) BaseURL() string

BaseURL exposes the scenario's API base URL for tests that build their own requests (e.g. the no-auth 401 check already does this directly via sc).

func (*Client) Delete

func (c *Client) Delete(ctx context.Context, path string) error

Delete issues an authenticated DELETE and returns an error on any non-2xx. Used by the isolate js-bundle catalogue UC, whose delete verb the Go SDK does not wrap. 204 No Content (the catalogue's success status) and any 2xx count as success; the body snippet on failure surfaces what the server said.

func (*Client) GetJSON

func (c *Client) GetJSON(ctx context.Context, path string, out any) error

GetJSON GETs path and decodes a 2xx JSON body into out. Any non-2xx is an error carrying the status and a snippet of the body, so a failing use case reports what the server actually said.

func (*Client) GetText

func (c *Client) GetText(ctx context.Context, path string) (string, error)

GetText GETs path and returns the raw 2xx body (for /v1/metrics, which is Prometheus text, not JSON).

func (*Client) LastCreateReadinessSource

func (c *Client) LastCreateReadinessSource() (string, bool)

LastCreateReadinessSource returns readiness;desc= from the latest create.

func (*Client) LastServerCreateMS

func (c *Client) LastServerCreateMS() (float64, bool)

LastServerCreateMS returns the server-reported create duration (ms) from the most recent create's Server-Timing header, then clears it. ok is false when the last create carried no such header (e.g. an older server). This lets a benchmark measure server-side create time, excluding client<->cluster network.

func (*Client) LastServerCreateStages

func (c *Client) LastServerCreateStages() (map[string]float64, bool)

LastServerCreateStages returns every <name>;dur= Server-Timing pair from the most recent create (create, runtime_wait, fc_verify, fc_driver, …), then clears it. ok is false when the last create carried no header. The bench folds these into the per-stage breakdown of the JSON artifact.

func (*Client) NewSandbox

func (c *Client) NewSandbox(t *testing.T, opts sdktypes.CreateSandboxOptions) *microvm.Sandbox

NewSandbox creates a sandbox with a unique, scenario-scoped name and registers a t.Cleanup that destroys it. This is the hygiene contract: no test leaks a sandbox into the shared deployment. Returns the running sandbox.

func (*Client) OwnerNodeID

func (c *Client) OwnerNodeID(ctx context.Context, sandboxID string) (string, error)

OwnerNodeID returns the cluster node that currently owns sandboxID, read from the FSM placement view (/v1/cluster/placements/{id} → owner.node_id). It is how a test establishes co-location, e.g. neighbor-isolation, which is a same-bridge (same-node) property. The error is reported faithfully so a caller can distinguish "not yet placed / transient" (retry) from a stable answer; a single-node Noop deployment (no placement endpoint) surfaces as an error, which the caller treats as "one node, one bridge" after retries.

func (*Client) PostJSON

func (c *Client) PostJSON(ctx context.Context, path string, body, out any) error

PostJSON POSTs an optional JSON body and decodes a 2xx JSON response into out (out may be nil to ignore the body). Non-2xx is an error with the status.

func (*Client) SDK

func (c *Client) SDK() *microvm.Client

SDK exposes the underlying client for use cases that need a method this wrapper doesn't surface yet.

type ExpandedCatalogueEntry

type ExpandedCatalogueEntry struct {
	ID          string
	Question    string
	Category    string
	Subcategory string
	Runtime     string
	Scenarios   []ScenarioTag
	UCRef       string
	SignalDesc  string
}

ExpandedCatalogueEntry is a runtime-resolved catalogue row for artifacts.

func ExpandedEntries

func ExpandedEntries(rows []CatalogueRow) []ExpandedCatalogueEntry

ExpandedEntries expands runtime crossings for JSON/gen output.

type IntegrationNode

type IntegrationNode struct {
	Name       string `json:"name"`
	Role       string `json:"role"`
	Seed       bool   `json:"seed"`
	PublicIP   string `json:"public_ip"`
	PrivateIP  string `json:"private_ip"`
	InstanceID string `json:"instance_id"`
	Spot       bool   `json:"spot"`
}

IntegrationNode is one row from Terraform's integration_targets.nodes output.

func LookupIntegrationNode

func LookupIntegrationNode(targets *IntegrationTargets, name string) (IntegrationNode, bool)

LookupIntegrationNode finds a provisioned node by terraform name (node_id).

func PickSSHNode

func PickSSHNode(targets *IntegrationTargets) (IntegrationNode, bool)

PickSSHNode returns a node we can SSH into from IntegrationTargets. Prefers the seed (local-mode tunnel / single-node), else first public IP.

type IntegrationTargets

type IntegrationTargets struct {
	BaseURL        string            `json:"base_url"`
	Domain         string            `json:"domain"`
	IngressIP      string            `json:"ingress_ip"`
	SeedIP         string            `json:"seed_ip"`
	Nodes          []IntegrationNode `json:"nodes"`
	GrafanaURL     string            `json:"grafana_url"`
	PushgatewayURL string            `json:"pushgateway_url"`
	ObsPublicIP    string            `json:"obs_public_ip"`
	ObsPrivateIP   string            `json:"obs_private_ip"`
}

IntegrationTargets mirrors Terraform output integration_targets.

func LoadIntegrationTargets

func LoadIntegrationTargets() *IntegrationTargets

LoadIntegrationTargets parses AEROL_INTEGRATION_TARGETS (JSON from run.sh). Returns nil when unset — disruptive infra helpers skip in that case.

type Runtime

type Runtime string

Runtime labels match plans/benchmark-usecase-catalogue.md legend.

const (
	RTContainerd  Runtime = "containerd"
	RTGvisor      Runtime = "gvisor"
	RTWasm        Runtime = "wasm"
	RTIsolate     Runtime = "isolate"
	RTFirecracker Runtime = "firecracker"
	RTAll         Runtime = "all"
)

type Scenario

type Scenario struct {
	Name    string
	BaseURL string
	PAT     string
	// Domain is the leased apex domain for domain-bearing scenarios (empty in
	// local-mode). Tests that build hostnames under the wildcard zone — e.g.
	// the custom-domain reachability use case — read it. The wildcard
	// *.<Domain> record already points at ingress, so any label resolves.
	Domain string
	// CustomDomain is a real hostname OUTSIDE the base domain, pre-provisioned
	// with a verification TXT record + CNAME to ingress. Populated only for
	// scenarios advertising CapExternalDNSZone; UC-35/36 read it.
	CustomDomain string
	// contains filtered or unexported fields
}

Scenario is the runtime description of the deployment under test: where to reach it, the PAT, and which capabilities it has. Capabilities come from the scenario's caps.yml (loaded by LoadScenario); BaseURL/PAT come from env.

func LoadScenario

func LoadScenario() (*Scenario, error)

LoadScenario builds a Scenario from the environment. It reads the caps file pointed at by AEROL_CAPS and the base URL + PAT from env. Returns an error (rather than skipping) when required env is missing so a misconfigured run fails loudly instead of silently testing nothing.

func (*Scenario) Has

func (s *Scenario) Has(c Capability) bool

Has reports whether the scenario advertises a capability.

func (*Scenario) MissingCaps

func (s *Scenario) MissingCaps(uc UseCase) []Capability

MissingCaps returns the capabilities a use case needs that the scenario lacks. Used for human-readable skip messages.

func (*Scenario) Satisfies

func (s *Scenario) Satisfies(uc UseCase) bool

Satisfies reports whether the scenario has every capability a use case needs. This is the load-bearing predicate behind every skip decision and the report classification, so it is kept pure (no I/O, no testing import) and is unit- tested offline in skip_test.go. A bug here silently turns an un-runnable use case green, which is why it gets its own test.

type ScenarioTag

type ScenarioTag string

Scenario tags from the catalogue doc.

const (
	ScnMixed  ScenarioTag = "M"
	ScnHetero ScenarioTag = "H"
	ScnBoth   ScenarioTag = "M+H"
)

type UseCase

type UseCase struct {
	ID    string
	Title string
	// Requires lists capabilities a scenario must have for this UC to run.
	Requires []Capability
	// Implemented marks whether a test function exists yet. False => the
	// report shows PENDING (a real gap) rather than a green/skip. The full
	// suite is implemented, so this is true for every current entry; it stays
	// in the model so a newly-added UC without a test surfaces as PENDING.
	Implemented bool
}

UseCase is one row of the coverage matrix.

func Lookup

func Lookup(id string) (UseCase, bool)

Lookup returns the use case for an ID and whether it exists.

func Require

func Require(t *testing.T, sc *Scenario, ucID string, alsoCovers ...string) UseCase

Require skips the test unless the scenario satisfies the use case's capabilities. The skip message names the missing capabilities so the report (and a human) can see exactly why it didn't run.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL