apiconformance

package
v0.32.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 18, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package apiconformance is the conformance suite for the versioned backend contract in docs/api/openapi.yaml. Given a base URL it probes the live implementation, checks status codes and response shapes, and reports per family. It is a self-contained, dependency-light client: it parses the committed OpenAPI document, sends read-only HTTP requests, and validates response bodies against the schemas the document carries.

A Suite is configured with a base URL and a Spec (the parsed document) and carries a set of Probes — one per operation the suite wants to check. The default probe set (DefaultProbes) is a curated table of safe, read-only GET operations that return a usable response on a fresh, disposable workspace. Mutating operations are never probed by the safe suite: RunMutating only runs when an explicitly disposable workspace root has been set (WithDisposableWorkspace), which is the gate that keeps a conformance run from writing to a real workspace.

Schema validation is done with github.com/santhosh-tekuri/jsonschema/v6, chosen over Huma's own schema tooling on purpose. The generated document is a standalone OpenAPI file that is compiled directly and validated against an arbitrary response body — exactly what that library does (it compiles a JSON Schema and resolves $ref against a registered document). Huma's Registry and Validate are coupled to its registered-operation types and cannot validate a standalone document's schemas in isolation, which would force the suite to pull in the operation registry instead of the file. The santhosh-tekuri library is MIT-licensed, maintained, and already a repo dependency (see validate.go for the compile/resolve helper).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ValidateBody

func ValidateBody(schema *jsonschema.Schema, body []byte) (bool, string)

ValidateBody decodes the raw body as JSON and validates the value against schema. When schema is nil it only checks that the body is valid JSON (any value passes), which is the shape check for operations without a concrete 200 schema. It returns (ok, reason); reason is empty on success and a short human-readable explanation of the first failure otherwise.

Types

type FamilyResult

type FamilyResult struct {
	// Family is the family name this aggregate covers (redundant with the map
	// key so the value is self-describing when serialized).
	Family string
	// Probed is the number of probes sent in this family.
	Probed int
	// Passed is the number of probes that passed.
	Passed int
	// Failed is the number of probes that did not.
	Failed int
}

FamilyResult aggregates the per-probe outcomes for one family.

type Info

type Info struct {
	// Title is the document title (info.title).
	Title string
	// Version is the contract version (info.version), the value the version
	// negotiation reports.
	Version string
}

Info is the top-level document metadata.

type Operation

type Operation struct {
	// OperationID is the operationId (e.g. "get-config").
	OperationID string

	// Method is the HTTP verb in lowercase (e.g. "get").
	Method string

	// Path is the path template as written in the document (e.g.
	// "/api/sessions/{id}/export").
	Path string

	// Summary is the human-readable summary.
	Summary string

	// Tags is the operation's family tags, in document order. The first entry
	// is the primary family a probe is grouped under.
	Tags []string

	// OKStatusCode is the documented 2xx status (default 200 when the document
	// lists a 200 response).
	OKStatusCode int

	// HasOKSchema is true when the documented 200 response carries a concrete
	// response schema (not just a description). Only a handful of operations
	// have one; the rest are shape-checked as "valid JSON".
	HasOKSchema bool

	// OKSchemaRef is the $ref (if any) of the 200 response schema, e.g.
	// "#/components/schemas/ErrorModel". Empty when the 200 schema is inline.
	OKSchemaRef string

	// OKSchemaSchema is the inline 200 response schema (the schema object
	// under responses."200".content.<mime>.schema), or nil when the response
	// has no concrete schema or the schema is a bare $ref.
	OKSchemaSchema map[string]any
}

Operation is one HTTP operation (a single verb under a path) in the document. The fields the suite consumes are captured; the rest of the document stays in the generic tree.

type Option

type Option func(*Suite)

Option configures a Suite. Options are applied in order by New.

func WithClient

func WithClient(c *http.Client) Option

WithClient overrides the HTTP client used to send probes. A nil client is ignored (the default client with the suite's timeout is kept).

func WithDisposableWorkspace

func WithDisposableWorkspace(root string) Option

WithDisposableWorkspace marks a workspace root as disposable, enabling RunMutating. A conformance run only mutates state inside a workspace the caller has explicitly declared disposable.

func WithHTTPHeader

func WithHTTPHeader(name, value string) Option

WithHTTPHeader adds a single named header to every probe request. Multiple calls accumulate.

func WithProbes

func WithProbes(probes []Probe) Option

WithProbes replaces the default probe set. An explicitly empty set is honored (the suite then probes nothing); only when this option is not used at all does New fall back to DefaultProbes.

type Probe

type Probe struct {
	// OperationID is the operation's operationId; it is what a result is
	// reported by and how a suite maps a probe back to its spec Operation.
	OperationID string

	// Method is the HTTP verb to send.
	Method string

	// Path is the concrete request path. Path parameters (if any) are
	// substituted with a safe concrete value (or the path is left empty of the
	// parameter where that is the safe form). A probe is only added when the
	// resulting request is safe to fire at a fresh workspace.
	Path string

	// Query carries the request query string values. It is empty for the
	// majority of probes; a GET that needs a query parameter to return a 200
	// (e.g. search) supplies it here.
	Query map[string]string

	// ExpectedStatus is the status the probe expects. It is 200 for every probe;
	// the field exists so the table reads as data and a future probe family can
	// expect a different 2xx without a struct change.
	ExpectedStatus int

	// Family is the probe's primary family tag, the group a per-family report
	// aggregates under.
	Family string
}

Probe describes one operation the suite checks against a live base URL: the request to send (method, path, and any query) and the expectation (a 200 and a body shaped by the operation's 200 schema). A probe is deliberately conservative: it must be safe to run against a fresh, disposable workspace with no active chat and no running agent.

func DefaultProbes

func DefaultProbes() []Probe

DefaultProbes returns the curated set of safe, read-only GET probes for a fresh, disposable workspace. Each entry returns a usable 200 response with no active chat, running agent, or prior state required. The set deliberately excludes:

  • GETs that need a session, chat, or edit id to exist (editStatus, chatSessionWorktreeGet, the chat-sessions/{id} family, sessionExport) — a fresh workspace has none, and inventing an id only probes 404 paths;
  • GETs that depend on an external or optional subsystem being present (sshHosts, sshSessions, sshLaunchStatus, localLLMStatus, localLLMModels) — they 400/500 when the subsystem is absent, which is a property of the host, not the contract;
  • GETs whose happy path issues a live external call (providersModels: it 400s without a `provider` query parameter and then contacts the provider to discover its model list) — a conformance probe must not reach the network;
  • GETs that require a query parameter the table will not fabricate (searchQuery is the one exception: it needs ?query and is included with a literal query so a fresh workspace returns a valid, possibly-empty result set).

The default set is the high-signal baseline. A later suite consumer overrides or extends it through Suite.WithProbes; every probe here maps to a GET operation documented in openapi.yaml, and its Family is that operation's primary tag.

type Report

type Report struct {
	// PerFamily maps a family name to its aggregate. Only families that had at
	// least one probe appear; the map is populated from Results.
	PerFamily map[string]FamilyResult

	// Results is the per-probe outcome in probe order.
	Results []Result
}

Report is the aggregate of a conformance run: the per-probe results and a per-family roll-up. It is the value the safe and mutating runs return.

func NewReport

func NewReport(results []Result) *Report

NewReport builds a Report from a slice of per-probe results, computing the per-family aggregates. The results are not mutated; order is preserved.

func (*Report) AllPassed

func (r *Report) AllPassed() bool

AllPassed reports whether every probed operation passed. It is true (vacuously) when the report has no results.

func (*Report) Failed

func (r *Report) Failed() []Result

Failed returns the results that did not pass, in probe order. It is empty (but non-nil) when everything passed.

func (*Report) JSON

func (r *Report) JSON() ([]byte, error)

JSON renders the report as stable, deterministic JSON suitable for a CLI: families are emitted in sorted order and results in probe order, so the byte output is reproducible for a given run.

func (*Report) Render

func (r *Report) Render() string

Render produces a human-readable, per-family report: a one-line roll-up per family (name, probed count, passed/failed) followed by one line per failure (operationId, family, status, reason). Families are listed in a stable (sorted) order.

type Result

type Result struct {
	Probe

	// Status is the HTTP status the implementation returned. It is zero when
	// the request could not be made (transport error), in which case Passed is
	// false and Reason carries the error.
	Status int

	// Passed is true when the status was in the expected 2xx range and the body
	// shape check passed.
	Passed bool

	// Reason is empty on success and a short, single-line explanation of a
	// failure (the status mismatch or the first schema violation).
	Reason string
}

Result is the outcome of probing one operation: the probe that was sent, the status the implementation returned, whether it passed, and a short reason for a failure. It embeds Probe, so the family (Probe.Family) is promoted onto the result and is what per-family aggregation reads.

type Spec

type Spec struct {
	// Info holds the top-level document metadata (title and the contract
	// version).
	Info Info

	// Tags is the declared tag list (one entry per family). A family declared
	// but not yet used by any operation still appears here.
	Tags []string

	// Operations is every operation in the document, in file order.
	Operations []Operation

	// ComponentsSchemas is the raw components.schemas map. It is kept generic
	// (map[string]any) so a validator can hand it to a JSON-Schema compiler
	// and resolve $ref against the whole document.
	ComponentsSchemas map[string]any
}

Spec is the parsed in-memory form of the generated OpenAPI contract. It carries the metadata the suite needs (version and the family list), the flat list of operations, and the raw components section so a validator can resolve #/components/... references against the document.

func LoadSpec

func LoadSpec(path string) (*Spec, error)

LoadSpec parses the OpenAPI YAML at path into a Spec. The document is a generated OpenAPI 3.x file; only the fields the suite consumes are materialized as structs, the rest are kept in the generic tree.

func (*Spec) ContractVersion

func (s *Spec) ContractVersion() string

ContractVersion returns the contract version (info.version).

func (*Spec) Families

func (s *Spec) Families() []string

Families returns the distinct family names: the union of the declared tag list and every tag an operation actually uses, sorted for stable output. A family may be declared with no operations yet, so the declared list is the source of truth; operation tags are added only if they are new.

type Suite

type Suite struct {
	// contains filtered or unexported fields
}

Suite is a conformance run against a base URL. It carries the loaded Spec, the probe set to send, and the transport/shape-checking configuration. Build one with New and run it with RunSafe (read-only) or RunMutating (gated).

func New

func New(baseURL string, spec *Spec, opts ...Option) *Suite

New builds a Suite against baseURL from a loaded Spec, applying opts in order. New never fails on spec content: if the spec's schemas cannot be compiled into a validator, the error is surfaced by RunSafe/RunMutating rather than here, so the constructor stays a simple value factory. When opts does not set the probe set, it defaults to DefaultProbes.

func (*Suite) Probes

func (s *Suite) Probes() []Probe

Probes returns the probe set the suite will send (the configured set, or the default set when none was configured).

func (*Suite) RunMutating

func (s *Suite) RunMutating(ctx context.Context) (*Report, error)

RunMutating runs the mutating checks, but only when a disposable workspace was explicitly set (WithDisposableWorkspace); otherwise it returns errMutatingRequiresDisposable. The mutating battery is deliberately a documented no-op for now: the gate (not a full mutation set) is what this package delivers, and a later consumer supplies the mutating probes. When the gate is cleared with no mutating probes, the result is an empty, all-passed report.

func (*Suite) RunSafe

func (s *Suite) RunSafe(ctx context.Context) (*Report, error)

RunSafe sends every probe in the suite's set to the base URL and returns a per-family report. A probe passes when the response is a 2xx and its body shape-checks against the operation's 200 schema (or, when the operation has no concrete 200 schema, when the body is valid JSON). It returns an error only if the suite could not be initialized (a spec whose schemas do not compile).

func (*Suite) WithProbes

func (s *Suite) WithProbes(probes []Probe) *Suite

WithProbes replaces the suite's probe set and returns the suite for chaining. It is the imperative form of the WithProbes option, for callers that build a suite first and adjust it later (a later consumer overrides/extends the set this way).

type Validator

type Validator struct {
	// contains filtered or unexported fields
}

Validator compiles the response schemas of a Spec into JSON-Schema objects and validates decoded response bodies against them. It wraps the document's components.schemas in a single resource (so $ref resolves in-document) and adds a per-operation entry for every operation that carries a concrete 200 schema.

The choice of validator (santhosh-tekuri/jsonschema/v6 over Huma's own tooling) is recorded in the package doc; see spec.go's package comment.

func NewValidator

func NewValidator(spec *Spec) (*Validator, error)

NewValidator builds a Validator from a Spec. It registers a wrapper document (components + a per-operation schema map) as a single resource and pre-compiles each operation's 200 schema, the shared ErrorModel, and the inline schemas. Operations with no concrete 200 schema get no entry; their bodies are checked as "valid JSON" by the caller. A malformed document (one whose components do not form a valid JSON-Schema resource) is reported as an error.

func (*Validator) ErrModel

func (v *Validator) ErrModel() *jsonschema.Schema

ErrModel returns the compiled shared error model (nil when the document defines no components.schemas).

func (*Validator) SchemaFor

func (v *Validator) SchemaFor(operationID string) *jsonschema.Schema

SchemaFor returns the compiled 200 schema for the operation with the given id (nil when the operation has no concrete 200 schema, in which case a body is shape-checked as "valid JSON").

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL