Documentation
¶
Overview ¶
Package harness holds the scenario/capability model, the use-case registry, and the thin SDK client wrapper shared by every integration test.
The registry is the single source of truth the report generator joins against to decide, per scenario, whether a use case PASSED, FAILED, was SKIPPED (scenario capabilities don't satisfy it) or is PENDING (no test implemented yet). Keeping IDs + required capabilities here — not scattered across test files — is what lets report/gen.go produce the coverage matrix without re-deriving intent from test names.
Index ¶
- Constants
- Variables
- func CatalogueCategoryOrder() []string
- func DisruptiveAllowed() bool
- func EC2InstanceState(t *testing.T, instanceID string) string
- func FormatScenarios(tags []ScenarioTag) string
- func HostHasAEROLVMUserJump(t *testing.T, node IntegrationNode) bool
- func RestartSystemdUnit(t *testing.T, node IntegrationNode, unit string)
- func RestartSystemdUnitOnAll(t *testing.T, targets *IntegrationTargets, unit string)
- func SSHRun(t *testing.T, target, script string) (string, error)
- func SSHTarget(n IntegrationNode) (string, bool)
- func SSHUser() string
- func SetEC2InstanceRunning(t *testing.T, instanceID string, running bool)
- func SimIDForCatalogue(catalogueID string) string
- func UniqueName(sc *Scenario, t *testing.T) string
- type Capability
- type CatalogueRow
- type Client
- func (c *Client) BaseURL() string
- func (c *Client) Delete(ctx context.Context, path string) error
- func (c *Client) GetJSON(ctx context.Context, path string, out any) error
- func (c *Client) GetText(ctx context.Context, path string) (string, error)
- func (c *Client) LastCreateReadinessSource() (string, bool)
- func (c *Client) LastServerCreateMS() (float64, bool)
- func (c *Client) LastServerCreateStages() (map[string]float64, bool)
- func (c *Client) NewSandbox(t *testing.T, opts sdktypes.CreateSandboxOptions) *microvm.Sandbox
- func (c *Client) OwnerNodeID(ctx context.Context, sandboxID string) (string, error)
- func (c *Client) PostJSON(ctx context.Context, path string, body, out any) error
- func (c *Client) SDK() *microvm.Client
- type ExpandedCatalogueEntry
- type IntegrationNode
- type IntegrationTargets
- type Runtime
- type Scenario
- type ScenarioTag
- type UseCase
Constants ¶
const ( EnvBaseURL = "AEROL_BASE_URL" // e.g. https://abc.itest.example.com or http://localhost:21212 EnvPAT = "AEROL_PAT" EnvScenario = "AEROL_SCENARIO" EnvCaps = "AEROL_CAPS" // path to the scenario caps.yml EnvDomain = "AEROL_DOMAIN" // leased apex domain; empty in local-mode // EnvCustomDomain is a real hostname OUTSIDE the deployment base domain that // the operator has already provisioned for custom-domain tests (UC-35/36): // a `_aerol-verify.<host>` TXT record proving ownership and a CNAME pointing // at ingress. Only scenarios advertising CapExternalDNSZone export it. EnvCustomDomain = "AEROL_CUSTOM_DOMAIN" )
Env var names the harness reads. run.sh exports these after provisioning.
const DefaultImage = "alpine:3.20"
DefaultImage is the small, always-available image used by lifecycle/network use cases that don't care what runs inside.
Variables ¶
var AllRuntimes = []Runtime{RTContainerd, RTGvisor, RTWasm, RTIsolate, RTFirecracker}
AllRuntimes is the five sandbox runtimes for "all" rows.
var CatalogueRegistry = buildCatalogueRegistry()
CatalogueRegistry is the 287-row master list (range IDs expanded).
var Registry = []UseCase{}/* 111 elements not displayed */
Registry is the full use-case catalogue. Order is the matrix row order.
Every use case now has a test (Phases 1-2 fanned out the full suite), so all are Implemented=true. A use case a given scenario can't satisfy SKIPs (reported not-applicable, not PENDING) via harness.Require; one that needs an out-of-band fixture (a staged wasm module, fault injection, registry creds) t.Skips with a reason. PENDING now only appears if a brand-new UC is added to this registry without a test — which is exactly the gap signal we want.
Functions ¶
func CatalogueCategoryOrder ¶
func CatalogueCategoryOrder() []string
CatalogueCategoryOrder returns stable category ordering for reports.
func DisruptiveAllowed ¶
func DisruptiveAllowed() bool
DisruptiveAllowed reports whether fault-injection tests (node kill, etc.) may run. cluster-hetero enables this by default via integration-tests/run.sh; pass --no-disruptive or use make integration-cluster-hetero-safe to opt out.
func EC2InstanceState ¶
EC2InstanceState runs `aws ec2 describe-instances` for one instance id.
func FormatScenarios ¶
func FormatScenarios(tags []ScenarioTag) string
FormatScenarios renders scenario tags for markdown tables.
func HostHasAEROLVMUserJump ¶
func HostHasAEROLVMUserJump(t *testing.T, node IntegrationNode) bool
HostHasAEROLVMUserJump reports whether filter FORWARD jumps to AEROLVM-USER. Works for both iptables-nft and legacy (iptables -S).
func RestartSystemdUnit ¶
func RestartSystemdUnit(t *testing.T, node IntegrationNode, unit string)
RestartSystemdUnit restarts a unit on the node and waits until it is active.
func RestartSystemdUnitOnAll ¶
func RestartSystemdUnitOnAll(t *testing.T, targets *IntegrationTargets, unit string)
RestartSystemdUnitOnAll restarts unit on every SSH-reachable node.
func SSHTarget ¶
func SSHTarget(n IntegrationNode) (string, bool)
SSHTarget builds user@host for a provisioned node. Prefers PublicIP.
func SSHUser ¶
func SSHUser() string
SSHUser returns the OS login used for host-side integration SSH. Matches run.sh / common.sh (ubuntu@…).
func SetEC2InstanceRunning ¶
SetEC2InstanceRunning stops or starts an EC2 instance via the AWS CLI. region/profile follow the ambient AWS_* env (same as terraform apply).
func SimIDForCatalogue ¶
SimIDForCatalogue maps catalogue IDs to simulation package IDs when a live sim exists (priority subset); empty means registry-only / stub.
Types ¶
type Capability ¶
type Capability string
Capability is a property a deployment scenario either has or doesn't. A use case lists the capabilities it needs; a scenario that lacks one causes the use case to SKIP there (reported as not-applicable, not failed).
const ( CapDocker Capability = "docker" // docker runtime available CapFirecracker Capability = "firecracker" // a firecracker-capable worker CapGvisor Capability = "gvisor" // a gvisor-capable worker CapWasm Capability = "wasm" // wasm runtime + staged module // CapIsolate gates the V8-isolate (workerd) use cases (UC-103..105). Like // CapGvisor it is advertisement-only: the runtime is enabled purely by // provisioning (default_with_isolate=true → install.sh --with-isolate writes // SB_ENABLE_ISOLATE=true + installs workerd), not by any run.sh config-overlay // flip. A scenario advertises it when its node was provisioned --with-isolate; // where the runtime is off, the isolate UCs skip (not-applicable) rather than // hard-fail. Unlike CapWasm it needs no node-side module staging — the UCs // upload JS bundles over POST /v1/js-bundles at runtime. CapIsolate Capability = "isolate" // V8-isolate (workerd) runtime available CapGPU Capability = "gpu" // a GPU worker CapDomain Capability = "domain" // public domain + TLS (not local-mode) CapCluster Capability = "cluster" // multi-node cluster (raft/forwarding) // CapMixedArchNegative gates UC-79: inject a foreign-arch snapshot ref and // assert the arm64 cluster refuses to resume it. CapMixedArchNegative Capability = "mixed-arch-negative" // CapCustomDomains gates the custom-domain attach path (UC-35/36). It's // distinct from CapDomain: a deployment can have a public domain + TLS yet // run with SB_ENABLE_CUSTOM_DOMAINS=false, in which case the API rejects // AddCustomDomain. Only scenarios that opt the feature on (and provision the // env) advertise this, so the custom-domain UCs skip instead of hard-failing // where the feature is off. CapCustomDomains Capability = "custom-domains" // CapExternalDNSZone gates the custom-domain verification path (UC-35/36). // A real custom domain must, by design, live OUTSIDE the deployment base // domain (the API rejects hosts under the wildcard zone) AND prove ownership // via a `_aerol-verify.<host>` TXT record before attach succeeds. Satisfying // that requires the harness to control a second DNS zone it can provision // verification (and CNAME) records in — the leased base zone is not enough. // Scenarios advertise this only when they wire up such a zone; otherwise the // custom-domain reachability UCs skip instead of failing against a gate they // structurally cannot pass. CapExternalDNSZone Capability = "external-dns-zone" // CapPlatformVolumes gates the platform-volume UCs (UC-81..UC-84). A // deployment advertises it only when the operator has enabled platform // volumes AND configured a shared backend (SB_PLATFORM_VOLUMES_ENABLED=true // + an S3 bucket or NFS export). The tests are backend-agnostic — they // attach by name, write, and read back — so the same UCs run whether the // scenario configured S3 or NFS. Where the feature is off, the UCs skip // (not-applicable) rather than fail against a 412. CapPlatformVolumes Capability = "platform-volumes" // CapBenchmark gates the create-benchmark UCs (UC-94/UC-95). It is // deliberately separate from the runtime capabilities: the benchmark is // slow and provisions many sandboxes (the density probe runs until the // fleet rejects on capacity), so it must be opt-in even on a scenario that // otherwise has every runtime. Only scenarios that explicitly advertise it // — currently cluster-hetero — run the benchmark; everywhere else the UCs // skip (not-applicable) instead of inflating cost on a normal pass. CapBenchmark Capability = "benchmark" // CapDockerPool is consumed by run.sh's config overlay, not by any UC's // Requires: it flips docker.pool.enabled in the scenario's cluster.yml so // UC-94 measures the warm-hit create path. It is deliberately separate // from CapDocker because each parked slot holds a default-shaped capacity // reservation, lowering the UC-95 density ceiling — scenarios opt in // (plans/docker-warm-pool.md §9 documents the adjusted gates). CapDockerPool Capability = "docker-pool" // CapDockerNetnsPool is likewise consumed by run.sh's config overlay: it // flips docker.netns_pool.enabled so cold docker creates adopt prepaid // pause-container network namespaces. Separate from CapDockerPool because // the two pools are independent (netns slots hold no capacity // reservations, so this one leaves the UC-95 density gate untouched). CapDockerNetnsPool Capability = "docker-netns-pool" // CapDockerEngine is consumed by run.sh's engine selection, not by any UC's // Requires. It is an OPT-OUT: containerd is the DEFAULT engine for every // scenario (run.sh writes SB_CONTAINER_ENGINE=containerd), and only a // scenario advertising this cap runs dockerd instead. Exactly two do — the // local dev install (local-mode) and the docker A/B benchmark baseline // (cluster-3-mixed-docker, kept on docker so it stays a valid comparison // against cluster-3-mixed-containerd). This encodes the target end-state of // the docker->containerd migration: "local install uses docker, every real // deployment uses containerd." CapDockerEngine Capability = "docker-engine" // CapContainerdEngine does NOT select the engine — that is now the default // (see CapDockerEngine). It gates the containerd-SPECIFIC coverage: the // Phase 5 soak/coexistence UCs (UC-99..102, plans/containerd-engine.md // §6/§8) and the distinctly-labeled `containerd` benchmark row + density. // Advertised only on the dedicated containerd validation scenarios // (single-node-containerd, cluster-3-mixed-containerd) so UC-102's // dockerd-coexistence restart stays off the runtime/metal scenarios where a // pure-containerd host makes it not-applicable. CapContainerdEngine Capability = "containerd-engine" // CapObservability gates UC-106/107 ("Grafana up", "Prometheus sees all // sandboxd nodes") and tells run.sh to pass -var deploy_obs=true so // Terraform/obs.tf provisions the dedicated obs EC2. Advertisement + // provisioning only — same shape as CapGvisor/CapIsolate. CapObservability Capability = "observability" // CapSimulations gates the suite/sims workload catalogue and UC-108 // (per-sim pass/fail). Opt-in like CapBenchmark: slow, provisions long- // lived services, and needs AEROL_SIMS=1. UC-108 must never roll up to a // single "all green" — each sim records independently. CapSimulations Capability = "simulations" )
type CatalogueRow ¶
type CatalogueRow struct {
ID string
Question string
Category string
Subcategory string
Runtimes []Runtime
Scenarios []ScenarioTag
UCRef string
SignalDesc string
}
CatalogueRow is one investor question from benchmark-usecase-catalogue.md. UCRef rows store only Question/Category/UCRef; runtime and signal may be derived from harness.Registry at report time (CQ-2).
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client wraps the Go SDK so every test shares one configured client and the resource-hygiene helpers (unique naming + guaranteed cleanup). Using the real SDK is deliberate: the suite doubles as Go-SDK integration coverage.
func NewClient ¶
NewClient builds a Client for the scenario. Fails the test immediately if the SDK can't be constructed — there's no point continuing without an API client.
func (*Client) BaseURL ¶
BaseURL exposes the scenario's API base URL for tests that build their own requests (e.g. the no-auth 401 check already does this directly via sc).
func (*Client) Delete ¶
Delete issues an authenticated DELETE and returns an error on any non-2xx. Used by the isolate js-bundle catalogue UC, whose delete verb the Go SDK does not wrap. 204 No Content (the catalogue's success status) and any 2xx count as success; the body snippet on failure surfaces what the server said.
func (*Client) GetJSON ¶
GetJSON GETs path and decodes a 2xx JSON body into out. Any non-2xx is an error carrying the status and a snippet of the body, so a failing use case reports what the server actually said.
func (*Client) GetText ¶
GetText GETs path and returns the raw 2xx body (for /v1/metrics, which is Prometheus text, not JSON).
func (*Client) LastCreateReadinessSource ¶
LastCreateReadinessSource returns readiness;desc= from the latest create.
func (*Client) LastServerCreateMS ¶
LastServerCreateMS returns the server-reported create duration (ms) from the most recent create's Server-Timing header, then clears it. ok is false when the last create carried no such header (e.g. an older server). This lets a benchmark measure server-side create time, excluding client<->cluster network.
func (*Client) LastServerCreateStages ¶
LastServerCreateStages returns every <name>;dur= Server-Timing pair from the most recent create (create, runtime_wait, fc_verify, fc_driver, …), then clears it. ok is false when the last create carried no header. The bench folds these into the per-stage breakdown of the JSON artifact.
func (*Client) NewSandbox ¶
NewSandbox creates a sandbox with a unique, scenario-scoped name and registers a t.Cleanup that destroys it. This is the hygiene contract: no test leaks a sandbox into the shared deployment. Returns the running sandbox.
func (*Client) OwnerNodeID ¶
OwnerNodeID returns the cluster node that currently owns sandboxID, read from the FSM placement view (/v1/cluster/placements/{id} → owner.node_id). It is how a test establishes co-location, e.g. neighbor-isolation, which is a same-bridge (same-node) property. The error is reported faithfully so a caller can distinguish "not yet placed / transient" (retry) from a stable answer; a single-node Noop deployment (no placement endpoint) surfaces as an error, which the caller treats as "one node, one bridge" after retries.
type ExpandedCatalogueEntry ¶
type ExpandedCatalogueEntry struct {
ID string
Question string
Category string
Subcategory string
Runtime string
Scenarios []ScenarioTag
UCRef string
SignalDesc string
}
ExpandedCatalogueEntry is a runtime-resolved catalogue row for artifacts.
func ExpandedEntries ¶
func ExpandedEntries(rows []CatalogueRow) []ExpandedCatalogueEntry
ExpandedEntries expands runtime crossings for JSON/gen output.
type IntegrationNode ¶
type IntegrationNode struct {
Name string `json:"name"`
Role string `json:"role"`
Seed bool `json:"seed"`
PublicIP string `json:"public_ip"`
PrivateIP string `json:"private_ip"`
InstanceID string `json:"instance_id"`
Spot bool `json:"spot"`
}
IntegrationNode is one row from Terraform's integration_targets.nodes output.
func LookupIntegrationNode ¶
func LookupIntegrationNode(targets *IntegrationTargets, name string) (IntegrationNode, bool)
LookupIntegrationNode finds a provisioned node by terraform name (node_id).
func PickSSHNode ¶
func PickSSHNode(targets *IntegrationTargets) (IntegrationNode, bool)
PickSSHNode returns a node we can SSH into from IntegrationTargets. Prefers the seed (local-mode tunnel / single-node), else first public IP.
type IntegrationTargets ¶
type IntegrationTargets struct {
BaseURL string `json:"base_url"`
Domain string `json:"domain"`
IngressIP string `json:"ingress_ip"`
SeedIP string `json:"seed_ip"`
Nodes []IntegrationNode `json:"nodes"`
GrafanaURL string `json:"grafana_url"`
PushgatewayURL string `json:"pushgateway_url"`
ObsPublicIP string `json:"obs_public_ip"`
ObsPrivateIP string `json:"obs_private_ip"`
}
IntegrationTargets mirrors Terraform output integration_targets.
func LoadIntegrationTargets ¶
func LoadIntegrationTargets() *IntegrationTargets
LoadIntegrationTargets parses AEROL_INTEGRATION_TARGETS (JSON from run.sh). Returns nil when unset — disruptive infra helpers skip in that case.
type Runtime ¶
type Runtime string
Runtime labels match plans/benchmark-usecase-catalogue.md legend.
type Scenario ¶
type Scenario struct {
Name string
BaseURL string
PAT string
// Domain is the leased apex domain for domain-bearing scenarios (empty in
// local-mode). Tests that build hostnames under the wildcard zone — e.g.
// the custom-domain reachability use case — read it. The wildcard
// *.<Domain> record already points at ingress, so any label resolves.
Domain string
// CustomDomain is a real hostname OUTSIDE the base domain, pre-provisioned
// with a verification TXT record + CNAME to ingress. Populated only for
// scenarios advertising CapExternalDNSZone; UC-35/36 read it.
CustomDomain string
// contains filtered or unexported fields
}
Scenario is the runtime description of the deployment under test: where to reach it, the PAT, and which capabilities it has. Capabilities come from the scenario's caps.yml (loaded by LoadScenario); BaseURL/PAT come from env.
func LoadScenario ¶
LoadScenario builds a Scenario from the environment. It reads the caps file pointed at by AEROL_CAPS and the base URL + PAT from env. Returns an error (rather than skipping) when required env is missing so a misconfigured run fails loudly instead of silently testing nothing.
func (*Scenario) Has ¶
func (s *Scenario) Has(c Capability) bool
Has reports whether the scenario advertises a capability.
func (*Scenario) MissingCaps ¶
func (s *Scenario) MissingCaps(uc UseCase) []Capability
MissingCaps returns the capabilities a use case needs that the scenario lacks. Used for human-readable skip messages.
func (*Scenario) Satisfies ¶
Satisfies reports whether the scenario has every capability a use case needs. This is the load-bearing predicate behind every skip decision and the report classification, so it is kept pure (no I/O, no testing import) and is unit- tested offline in skip_test.go. A bug here silently turns an un-runnable use case green, which is why it gets its own test.
type ScenarioTag ¶
type ScenarioTag string
Scenario tags from the catalogue doc.
const ( ScnMixed ScenarioTag = "M" ScnHetero ScenarioTag = "H" ScnBoth ScenarioTag = "M+H" )
type UseCase ¶
type UseCase struct {
ID string
Title string
// Requires lists capabilities a scenario must have for this UC to run.
Requires []Capability
// Implemented marks whether a test function exists yet. False => the
// report shows PENDING (a real gap) rather than a green/skip. The full
// suite is implemented, so this is true for every current entry; it stays
// in the model so a newly-added UC without a test surfaces as PENDING.
Implemented bool
}
UseCase is one row of the coverage matrix.