conformance

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 5, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func WriteManifest

func WriteManifest(path string, m Manifest) error

WriteManifest writes m to path as YAML, sorted by script path for a stable, diffable file.

Types

type Expect

type Expect string

Expect is the semantic outcome the manifest records for a script: what TestConformance should see the next time it runs the suite.

const (
	// ExpectPass means the script must semantically pass (exit code and
	// stdout oracle, if any, both match); a regression here fails the
	// blocking-lane gate.
	ExpectPass Expect = "pass"
	// ExpectFail means the script is known to fail right now; it needs a
	// Reason and never gates the build, but a script that starts passing
	// is reported as newly passing so it can be promoted.
	ExpectFail Expect = "fail"
	// ExpectSkip means the script is expected to be skipped this run (an
	// unmet prerequisite, a disabled lane, or missing tooling); it needs a
	// Reason for the same reasons as ExpectFail.
	ExpectSkip Expect = "skip"
)

type Lane

type Lane string

Lane groups scripts that need the same test environment and the same gating policy.

const (
	// LaneBlocking scripts need only the plain HTTP server on :8000. This
	// is the lane whose pass rate matters for CI gating (Phase 5).
	LaneBlocking Lane = "blocking"
	// LaneExtended scripts additionally need TLS servers, the Unix-socket
	// server, an IPv6 listener, or the local proxy.
	LaneExtended Lane = "extended"
	// LaneNetwork scripts talk to hosts outside localhost/127.0.0.1/::1;
	// skipped unless SONDE_CONFORMANCE_NETWORK=1.
	LaneNetwork Lane = "network"
	// LaneTiming scripts are known to be timing-sensitive (sleeps,
	// retries, rate limits) and flaky under load; run, but reported
	// separately from LaneBlocking.
	LaneTiming Lane = "timing"
	// LaneUnsupported scripts need tooling the harness does not provide
	// (e.g. the upstream formatter binary). Not run.
	LaneUnsupported Lane = "unsupported"
)

type Manifest

type Manifest map[string]ManifestEntry

Manifest maps a script path (relative to testdata/conformance/hurl, forward-slash separated, e.g. "tests_ok/hello/hello.sh") to its recorded expectation. It is committed at internal/conformance/manifest.yaml — never inside the vendored testdata/conformance/hurl tree.

func LoadManifest

func LoadManifest(path string) (Manifest, error)

LoadManifest reads and validates the manifest at path. A missing file is treated as an empty manifest (every script implicitly unclassified) rather than an error, so a fresh checkout can bootstrap one with `make conformance-update`.

func (Manifest) Validate

func (m Manifest) Validate() error

Validate checks manifest invariants: Expect must be one of pass/fail/skip, and every fail/skip entry must carry a non-empty Reason.

type ManifestEntry

type ManifestEntry struct {
	Lane   Lane   `yaml:"lane"`
	Expect Expect `yaml:"expect"`
	// Reason explains why the script is not expected to pass (required for
	// ExpectFail and ExpectSkip; e.g. "reports: Phase 5 in progress",
	// "digest unsupported", "network"). Left empty for ExpectPass.
	Reason string `yaml:"reason,omitempty"`
}

ManifestEntry is the recorded expectation for one script.

type Script

type Script struct {
	// Path is relative to the vendored hurl root (testdata/conformance/hurl),
	// forward-slash separated, e.g. "tests_ok/add_header/add_header.sh".
	Path string
	Lane Lane
	// Reason explains the lane, mainly useful for LaneUnsupported and
	// LaneNetwork.
	Reason string
	// NeedsProxy is set when the script's classification depends on the
	// local squid proxy (port 3128); the runner skips it at run time if
	// squid could not be started, without changing its static lane.
	NeedsProxy bool
}

Script is one discovered, classified conformance test entry point.

func DiscoverScripts

func DiscoverScripts(root string) ([]Script, error)

DiscoverScripts walks the vendored directories under root and classifies every *.sh entry point it finds.

type ScriptResult

type ScriptResult struct {
	Path         string        `json:"path"`
	Lane         Lane          `json:"lane"`
	Skipped      bool          `json:"skipped"`
	SkipReason   string        `json:"skip_reason,omitempty"`
	ExpectedExit int           `json:"expected_exit"`
	ActualExit   int           `json:"actual_exit"`
	SemanticPass bool          `json:"semantic_pass"`
	FullPass     bool          `json:"full_pass"`
	MismatchInfo string        `json:"mismatch,omitempty"`
	Duration     time.Duration `json:"duration_ns"`
}

ScriptResult is one script's outcome, serialized into results.json.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL