conformance

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source
const (
	TreeHurl    = "hurl"
	TreeHurlfmt = "hurlfmt"
)

Tree names: the first segment of every manifest key, and the directory under the conformance root that holds the vendored upstream tree.

Variables

This section is empty.

Functions

func WriteManifest

func WriteManifest(path string, m Manifest) error

WriteManifest writes m to path as YAML under the standard header, sorted by script path for a stable, diffable file.

Types

type Expect

type Expect string

Expect is the semantic outcome the manifest records for a script: what TestConformance should see the next time it runs the suite.

const (
	// ExpectPass means the script must semantically pass (exit code and
	// stdout oracle, if any, both match); a regression here fails the
	// blocking-lane gate.
	ExpectPass Expect = "pass"
	// ExpectFail means the script is known to fail right now; it needs a
	// Reason and never gates the build, but a script that starts passing
	// is reported as newly passing so it can be promoted.
	ExpectFail Expect = "fail"
	// ExpectSkip means the script is expected to be skipped this run (an
	// unmet prerequisite, a disabled lane, or missing tooling); it needs a
	// Reason for the same reasons as ExpectFail.
	ExpectSkip Expect = "skip"
)

type Lane

type Lane string

Lane groups scripts that need the same test environment and the same gating policy.

const (
	// LaneBlocking scripts need only the plain HTTP server on :8000. This
	// is the lane whose pass rate matters for CI gating (Phase 5).
	LaneBlocking Lane = "blocking"
	// LaneExtended scripts additionally need TLS servers, the Unix-socket
	// server, an IPv6 listener, or the local proxy.
	LaneExtended Lane = "extended"
	// LaneNetwork scripts talk to hosts outside localhost/127.0.0.1/::1;
	// skipped unless SONDE_CONFORMANCE_NETWORK=1.
	LaneNetwork Lane = "network"
	// LaneTiming scripts are known to be timing-sensitive (sleeps,
	// retries, rate limits) and flaky under load; run, but reported
	// separately from LaneBlocking.
	LaneTiming Lane = "timing"
	// LaneUnsupported scripts need tooling the harness does not provide.
	// Not run.
	LaneUnsupported Lane = "unsupported"
	// LaneHurlfmt holds the formatter tree: its scripts plus one synthetic
	// entry per export fixture comparison (lint/json/html). Gating.
	LaneHurlfmt Lane = "hurlfmt"
	// LanePTY scripts need a terminal on stdout and stderr; they run
	// through the upstream two-pty runner (integration/term.py). Gating.
	LanePTY Lane = "pty"
)

type Manifest

type Manifest map[string]ManifestEntry

Manifest maps a script key (the tree name, then the path inside that tree, forward-slash separated, e.g. "hurl/tests_ok/hello/hello.sh"; see Script.Path) to its recorded expectation. It is committed at internal/conformance/manifest.yaml (manifest-next.yaml for the pinned upstream snapshot) — never inside a vendored tree.

func LoadManifest

func LoadManifest(path string) (Manifest, error)

LoadManifest reads and validates the manifest at path. A missing file is treated as an empty manifest (every script implicitly unclassified) rather than an error, so a fresh checkout can bootstrap one with `make conformance-update`.

func (Manifest) Validate

func (m Manifest) Validate() error

Validate checks manifest invariants: Expect must be one of pass/fail/skip, and every fail/skip entry must carry a non-empty Reason.

type ManifestEntry

type ManifestEntry struct {
	Lane   Lane   `yaml:"lane"`
	Expect Expect `yaml:"expect"`
	// Reason explains why the script is not expected to pass (required for
	// ExpectFail and ExpectSkip; e.g. "reports: Phase 5 in progress",
	// "digest unsupported", "network"). Left empty for ExpectPass.
	Reason string `yaml:"reason,omitempty"`
}

ManifestEntry is the recorded expectation for one script.

type Script

type Script struct {
	// Path is the manifest key: the tree name, then the path inside that
	// tree, forward-slash separated, e.g.
	// "hurl/tests_ok/add_header/add_header.sh". An export fixture
	// comparison appends "#lint", "#json" or "#html" to its .hurl path.
	// DiscoverScripts alone (one tree) leaves the tree prefix off.
	Path string
	// Tree is TreeHurl or TreeHurlfmt; Rel is the script (or, for an export
	// comparison, the .hurl input) relative to that tree's root.
	Tree string
	Rel  string
	// Export is the hurlfmt --out format an export fixture comparison
	// checks ("hurl", "json" or "html"); empty for a script.
	Export string
	Lane   Lane
	// Reason explains the lane, mainly useful for LaneUnsupported and
	// LaneNetwork.
	Reason string
	// NeedsProxy is set when the script's classification depends on the
	// local squid proxy (port 3128); the runner skips it at run time if
	// squid could not be started, without changing its static lane.
	NeedsProxy bool
}

Script is one discovered, classified conformance test entry point.

func Discover added in v1.5.0

func Discover(confRoot string) ([]Script, error)

Discover finds every entry point under a conformance root (the directory holding the hurl/ and hurlfmt/ trees) and keys it by tree.

func DiscoverHurlfmt added in v1.5.0

func DiscoverHurlfmt(root string) ([]Script, error)

DiscoverHurlfmt lists the formatter tree's entry points, keyed with the "hurlfmt/" prefix: the tests_ok (recursive) and tests_failed scripts, and one comparison per tests_export fixture and format whose expected file exists, as integration/hurlfmt/integration.py does upstream.

func DiscoverScripts

func DiscoverScripts(root string) ([]Script, error)

DiscoverScripts walks the vendored directories under root (the hurl tree) and classifies every *.sh entry point it finds. Paths are relative to root, without the tree prefix Discover adds.

type ScriptResult

type ScriptResult struct {
	Path         string        `json:"path"`
	Lane         Lane          `json:"lane"`
	Skipped      bool          `json:"skipped"`
	SkipReason   string        `json:"skip_reason,omitempty"`
	ExpectedExit int           `json:"expected_exit"`
	ActualExit   int           `json:"actual_exit"`
	SemanticPass bool          `json:"semantic_pass"`
	FullPass     bool          `json:"full_pass"`
	MismatchInfo string        `json:"mismatch,omitempty"`
	Duration     time.Duration `json:"duration_ns"`
}

ScriptResult is one script's outcome, serialized into results.json.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL