conformance

package
v1.34.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 18, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package conformance is a tool-neutral format for reconcile test cases, and a runner that drives any declarative client of the Claude Developer Platform through them against the real service.

It exists so that `ant apply` and the Terraform provider — two tools that turn files into the same API calls — can share the cases that pin down what "converged" means: a create then an unchanged re-plan is empty, editing a sub-agent re-pins its coordinator, a field deleted from a file is cleared on the server, a server-normalised value is not drift. Nothing in this package imports either tool; each supplies an Adapter that renders desired state in its own language and reports its plan in the vocabulary below.

Case files

A case is one YAML file. Desired state is written as API create bodies keyed by a logical name, so neither tool's configuration language is privileged: 

schema: 1
name: editing a sub-agent re-pins its coordinator
resources:
  helper:
    kind: agent
    body: {name: "cf-{run}-helper", model: claude-sonnet-4-5, system: Help carefully.}
  lead:
    kind: agent
    body:
      name: "cf-{run}-lead"
      model: claude-sonnet-4-5
      multiagent: {type: coordinator, agents: [{$ref: helper}]}
steps:
  - expect:
      plan:   {helper: create, lead: create}
      remote: {lead: {"multiagent.agents[0].version": {$version: helper}}}
  - change: {helper: {body: {system: Help thoroughly.}}}
    expect:
      plan:   {helper: {update: [system]}, lead: {update: [multiagent]}}
  - remote: {helper: archive}
    expect: {error: "archived|no longer"}
  - flags: [force]
    expect:
      plan:   {helper: replace, lead: {update: [multiagent]}}

The rules:

  • {run} is replaced everywhere (bodies and file contents) with a short id unique to the run, and every resource's name must contain it: cases run end to end in a shared organization, in parallel, and some names (skill titles) are unique per organization.
  • A reference to another resource is written {$ref: name}. It stands for "however this tool points at that resource in this slot" — a relative path for ant, an attribute reference for Terraform — so cases never contain literal ids.
  • Skills carry their content as files: {SKILL.md: "...", notes.md: "..."}, paths relative to the skill's root directory.
  • The first step's desired state is `resources`. A later step's `change` is a JSON merge patch per resource (RFC 7396: objects merge, null deletes, anything else replaces); `name: null` removes the resource, and a new name with a `kind` adds one. Steps therefore read as what changed.
  • `remote` performs out-of-band edits before the step through the API directly, never through the tool under test: `archive`/`delete`, or {patch: {body...}, files: {...}} for a Console edit or a version published elsewhere.
  • `flags` is a closed list — force, prune — that each adapter maps to its own spelling or ignores if it has no equivalent.
  • `expect.plan` maps names to create | noop | update | {update: [top-level fields]} | replace | destroy. Names not listed must plan as noop, so an unexpected cascade fails the case. Omit `plan` to skip the check.
  • `expect.error` is a regular expression the plan (or, if the plan succeeds, the apply) must fail with. The step then checks nothing else: no `expect.plan`, no re-plan, no `expect.remote`.
  • `expect.remote` reads each named resource back from the API and checks dotted paths (`a.b[0].c`) against literals or matchers: {$id: name}, {$version: name}, {$set: true}, {$unset: true}, {$match: regex}.
  • After every step that applies, the runner re-plans and requires an empty plan. When the case ends, whether it passed or failed, the runner has the adapter destroy everything and confirms through the API that each resource is gone or archived.
  • `skip: {tool: reason}` and `only: [tool]` are the only per-tool escape hatches. A case that would need different input per tool is not a shared case; keep it in that tool's own suite.
  • A case that declares a kind the adapter does not manage is skipped for that adapter.

Running

Load reads a directory of cases; Run drives them through an Adapter, using a Remote for the out-of-band edits and read-backs. Both talk to the real API, so callers gate Run behind whatever opt-in their acceptance tests use.

Index

Constants

View Source 
const SchemaVersion = 1

SchemaVersion is the case format this package reads.

Variables

View Source 
var ErrNotFound = errors.New("not found")

ErrNotFound is what Remote.Get wraps for a resource that no longer exists.

Functions

func NewRunID 

func NewRunID() string

NewRunID returns the token substituted for {run}.

func Run 

func Run(t *testing.T, cases []*Case, tool Adapter, api Remote, opts Options)

Run drives each case through the adapter as a subtest named after the case. A case is skipped when its skip or only rules exclude the adapter, or when it declares a kind the adapter does not manage. A case that runs is always torn down, even when a step fails.

Types

type Action 

type Action string

Action is the tool-neutral vocabulary for what a plan does to a resource. 

const (
	ActionCreate Action = "create"
	ActionNoop   Action = "noop"
	ActionUpdate Action = "update"
	// ActionReplace is a new object created in place of one the tool can no
	// longer use.
	ActionReplace Action = "replace"
	ActionDestroy Action = "destroy"
)

type Adapter 

type Adapter interface {
	// Name is how cases refer to the tool in `skip` and `only`.
	Name() string
	// Supports reports whether the tool manages a kind at all.
	Supports(kind string) bool
	// Start begins one case in a fresh working directory and state.
	Start(t *testing.T) Session
}

Adapter is a tool under test.

type Case 

type Case struct {
	// Path is the file the case was read from.
	Path string
	Name string
	// Skip maps an Adapter's Name to the reason the case does not apply to it.
	Skip map[string]string
	// Only, when non-empty, names the only adapters the case runs against.
	Only      []string
	Resources Desired
	Steps     []Step
}

Case is one scenario: an initial desired state and a sequence of steps that change it, disturb the server, and say what a correct tool does next.

func Load 

func Load(dir, runID string) ([]*Case, error)

Load reads and validates every .yaml and .yml case under dir, recursively, substituting runID for {run}. Cases come back sorted by path; the first file that fails to load or validate stops the walk and is returned as the error.

func LoadFile 

func LoadFile(path, runID string) (*Case, error)

LoadFile reads and validates one case, substituting runID for {run}.

func (*Case) Kinds 

func (c *Case) Kinds() []string

Kinds lists every kind the case declares, across all steps, so a tool that does not manage one of them can skip the case whole.

type Desired 

type Desired map[string]Resource

Desired is a full desired state, keyed by logical name.

func (Desired) Names 

func (d Desired) Names() []string

Names lists the resources in a stable order.

type Expect 

type Expect struct {
	// Plan maps logical names to the action a correct plan takes. Nil skips
	// the check; otherwise unlisted names must be ActionNoop.
	Plan Plan
	// Error, when set, is a pattern the plan or apply must fail with.
	Error *regexp.Regexp
	// Remote maps logical names to dotted-path assertions on the object as
	// the API returns it after the step applies.
	Remote map[string]map[string]any
}

Expect is what a step asserts.

type Flags 

type Flags struct {
	Force bool
	Prune bool
}

Flags are the tool options a step runs with.

type Options 

type Options struct {
	// Parallel runs cases as parallel subtests. Every name carries {run}, and
	// by convention each case gives its names a distinct suffix, so rate
	// limits are the only reason not to.
	Parallel bool
	// KeepGoing reports a failed expectation and carries on, through the rest
	// of the step and the steps after it, instead of ending the case. An error
	// from the tool itself (render, plan, apply) still ends the case.
	KeepGoing bool
}

Options tune a run.

type Patch 

type Patch struct {
	// Kind is required when the name is new, and must match otherwise.
	Kind  string
	Body  map[string]any
	Files map[string]any // string content, or nil to delete the file
}

Patch is one resource's entry under `change`. A nil *Patch, written as null in the case file, removes the resource.

type Plan 

type Plan map[string]Planned

Plan is what a tool would do, keyed by logical name.

type Planned 

type Planned struct {
	Action Action
	// Fields are the top-level body fields an Update touches, sorted. Empty
	// in an expectation means "don't check which".
	Fields []string
}

Planned is one resource's line in a plan.

func (Planned) String 

func (p Planned) String() string

String renders p for a plan mismatch report: the bare action, or update[field ...] when Fields is set.

type Ref 

type Ref struct {
	Name string
}

Ref stands in a body for a reference to another resource in the case. An adapter renders it however its tool spells a reference in that slot.

type Remote 

type Remote interface {
	// Get returns the object as decoded JSON. Once the object is deleted, the
	// error wraps ErrNotFound. An archived object comes back with a non-nil
	// archived_at, which is how teardown tells archived from live.
	Get(ctx context.Context, kind, id string) (map[string]any, error)
	// Update edits the object in place: a merge-style body and, for kinds
	// whose content is files, a new set of them.
	Update(ctx context.Context, kind, id string, body map[string]any, files map[string]string) error
	// Destroy archives or deletes, whichever the kind supports.
	Destroy(ctx context.Context, kind, id string) error
}

Remote is the runner's own line to the API, for out-of-band edits and for reading back what a tool actually did. It must not share the tool's client, so a bug in that client cannot also hide itself from the checks.

type RemoteAction 

type RemoteAction struct {
	// Verb is "archive", "delete" or "patch". Body and Files are the patch's
	// merge-style update and new file set, and are unused otherwise.
	Verb  string
	Body  map[string]any
	Files map[string]string
}

RemoteAction is an out-of-band edit made through Remote.

type Resource 

type Resource struct {
	Kind string
	// Body is the create request body. Values may contain Ref.
	Body map[string]any
	// Files is a skill's content, keyed by path relative to its root.
	Files map[string]string
}

Resource is one API object as a case declares it.

type Session 

type Session interface {
	// Render writes desired as the tool's native files, replacing whatever
	// the previous step wrote. Ref values are the adapter's to translate.
	Render(desired Desired) error
	// Plan reports what the tool would do, by logical name. A plan the tool
	// refuses to apply (a blocked resource, a wrong-target guard) is an
	// error, not a Plan.
	Plan(ctx context.Context, flags Flags) (Plan, error)
	// Apply converges. It may re-plan internally.
	Apply(ctx context.Context, flags Flags) error
	// IDs maps every logical name the tool currently tracks to its remote id.
	IDs() map[string]string
	// Destroy removes everything the session created, however the tool
	// does that.
	Destroy(ctx context.Context) error
}

Session is one case's worth of a tool: a working directory, its state, and the operations the runner drives. Each step calls Render once, then Plan; unless the step expects an error, it then calls Apply and Plan again on the same files.

type Step 

type Step struct {
	// Change is a merge patch per resource against the previous step's
	// desired state. A nil patch removes the resource.
	Change map[string]*Patch
	// Remote lists out-of-band edits made through the API before planning.
	Remote map[string]RemoteAction
	Flags  Flags
	Expect Expect
}

Step is one round: edit the files, poke the server, plan, apply, check.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.