testharness

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package testharness is the first-class testing infrastructure for specd. It provides hermetic, deterministic building blocks that every specd test consumes: an isolated project root with env + working-directory isolation, a fluent SpecBuilder for authoring gate-valid specs, an in-process CommandRunner that captures stdout/stderr/exit-code exactly as the CLI would, a StateAsserter for state.json/filesystem/git assertions, and a FakeClock for reproducible timestamps.

Nothing here touches the network or the user's real git/working-directory state beyond t.TempDir, t.Setenv and a chdir restored on cleanup. Because the harness mutates process-global state (cwd, os.Stdout/os.Stderr during Run, and core.Clock), tests using it MUST NOT call t.Parallel.

Index

Constants

This section is empty.

Variables

View Source 
var Epoch = time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)

Epoch is the deterministic starting instant for every FakeClock: a fixed, human-recognisable UTC time so golden output is stable across runs and hosts.

Functions

func CaptureStderr 

func CaptureStderr(t *testing.T, fn func()) string

CaptureStderr runs fn with os.Stderr redirected to a pipe and returns everything fn wrote to it. It is the shared replacement for the per-file captureStderr helpers, so tests that assert diagnostic / R4 lines emitted to stderr do not each re-implement the redirect dance.

The original os.Stderr is always restored before CaptureStderr returns.

func CaptureStdout 

func CaptureStdout(t *testing.T, fn func()) string

CaptureStdout is the os.Stdout counterpart of CaptureStderr.

Types

type FakeClock 

type FakeClock struct {
	// contains filtered or unexported fields
}

FakeClock is a deterministic, goroutine-safe time source that replaces core.Clock for the duration of a test. By default each call to Now() returns the current instant and then advances by Step, guaranteeing that successive timestamps (e.g. createdAt vs updatedAt) are distinct yet fully reproducible. Call Freeze to make Now() constant.

func NewFakeClock 

func NewFakeClock() *FakeClock

NewFakeClock returns a clock starting at Epoch that auto-advances one second per read.

func (*FakeClock) Advance 

func (c *FakeClock) Advance(d time.Duration)

Advance moves the clock forward by d without consuming a read.

func (*FakeClock) Freeze 

func (c *FakeClock) Freeze()

Freeze stops auto-advance so every Now() returns the same instant — useful for byte-exact golden timestamp assertions.

func (*FakeClock) Now 

func (c *FakeClock) Now() time.Time

Now returns the current instant, then advances by the configured step.

func (*FakeClock) Set 

func (c *FakeClock) Set(t time.Time)

Set repositions the clock; the next Now() returns t.

func (*FakeClock) SetStep 

func (c *FakeClock) SetStep(d time.Duration)

Step sets the per-read auto-advance interval.

type FakeOrchestrationHost 

type FakeOrchestrationHost struct {
	H      *Harness
	Policy core.OrchestrationPolicy
	Cfg    core.OrchestrationCfg
}

FakeOrchestrationHost is a deterministic Brain/Pinky host loop for integration tests. It drives only public CLI/core contracts, never provider SDKs or network calls.

func NewFakeOrchestrationHost 

func NewFakeOrchestrationHost(h *Harness) *FakeOrchestrationHost

func (*FakeOrchestrationHost) ClaimAndVerify 

func (host *FakeOrchestrationHost) ClaimAndVerify(step core.OrchestrationStepResult) (core.PinkyMission, Result, *core.VerificationRecord)

func (*FakeOrchestrationHost) Complete 

func (host *FakeOrchestrationHost) Complete(step core.OrchestrationStepResult, summary string) core.PinkyEvidenceResult

func (*FakeOrchestrationHost) EnsureOrchestrated 

func (host *FakeOrchestrationHost) EnsureOrchestrated(slug string)

EnsureOrchestrated records executionMode=orchestrated on the spec so the Brain CLI gate (which refuses Base specs) lets the orchestration host drive it. The host models an already-capable, opted-in project, so it writes the recorded fact directly rather than going through the capability-gated `specd status --set-mode`.

func (*FakeOrchestrationHost) ExpireLeasesAndReclaim 

func (host *FakeOrchestrationHost) ExpireLeasesAndReclaim(sessionID string) int

func (*FakeOrchestrationHost) Mission 

func (host *FakeOrchestrationHost) Mission(step core.OrchestrationStepResult) core.PinkyMission

func (*FakeOrchestrationHost) PolicyArgs 

func (host *FakeOrchestrationHost) PolicyArgs(sessionID string) []string

func (*FakeOrchestrationHost) StartProgram 

func (host *FakeOrchestrationHost) StartProgram(sessionID string) core.ProgramStepResult

func (*FakeOrchestrationHost) StartSpec 

func (host *FakeOrchestrationHost) StartSpec(slug, sessionID string) core.OrchestrationStepResult

func (*FakeOrchestrationHost) StepProgram 

func (host *FakeOrchestrationHost) StepProgram(sessionID string) core.ProgramStepResult

func (*FakeOrchestrationHost) StepSpec 

func (host *FakeOrchestrationHost) StepSpec(slug, sessionID string) core.OrchestrationStepResult

type FakePinkyWorker 

type FakePinkyWorker struct {
	H        *Harness
	WorkerID string
	Cfg      core.OrchestrationCfg
}

FakePinkyWorker is a deterministic host worker for integration tests. It uses the public Pinky CLI for host-facing lease/report operations and core's public evidence reconciler for the Brain-side acceptance path. It performs no network calls and relies on the harness clock for stable timestamps.

func NewFakePinkyWorker 

func NewFakePinkyWorker(h *Harness, workerID string) *FakePinkyWorker

func (*FakePinkyWorker) AcknowledgeCancel 

func (w *FakePinkyWorker) AcknowledgeCancel(mission core.PinkyMission, reason string) (core.ACPEnvelope, error)

func (*FakePinkyWorker) Block 

func (w *FakePinkyWorker) Block(mission core.PinkyMission, reason string) Result

func (*FakePinkyWorker) Claim 

func (w *FakePinkyWorker) Claim(mission core.PinkyMission) core.PinkyClaim

func (*FakePinkyWorker) Heartbeat 

func (w *FakePinkyWorker) Heartbeat(mission core.PinkyMission) Result

func (*FakePinkyWorker) Progress 

func (w *FakePinkyWorker) Progress(mission core.PinkyMission, percent int, message string) Result

func (*FakePinkyWorker) Release 

func (w *FakePinkyWorker) Release(mission core.PinkyMission) Result

func (*FakePinkyWorker) ReportVerified 

func (w *FakePinkyWorker) ReportVerified(mission core.PinkyMission, rec *core.VerificationRecord, summary string) (core.PinkyEvidenceResult, Result, error)

func (*FakePinkyWorker) RunVerify 

func (w *FakePinkyWorker) RunVerify(mission core.PinkyMission) (Result, *core.VerificationRecord)

func (*FakePinkyWorker) SeedVerification 

func (w *FakePinkyWorker) SeedVerification(slug, taskID string, rec *core.VerificationRecord)

type Harness 

type Harness struct {
	T     *testing.T
	Root  string // absolute project root containing .specd/ (== process cwd)
	Clock *FakeClock
}

Harness is the root context for a single test: an isolated specd project plus the deterministic clock installed for its lifetime.

func New 

func New(t *testing.T) *Harness

New creates an isolated specd project: a fresh temp root with an empty .specd/specs/ tree, the working directory switched into it, hermetic environment defaults, and a FakeClock installed over core.Clock. All of it is torn down via t.Cleanup. The returned Harness.Root equals the process working directory, so it matches whatever the commands resolve via RequireSpecdRoot.

func (*Harness) AssertFileAbsent 

func (h *Harness) AssertFileAbsent(rel string)

AssertFileAbsent fails if the path exists.

func (*Harness) AssertFileContains 

func (h *Harness) AssertFileContains(rel, substr string)

AssertFileContains fails if the file is missing or lacks substr.

func (*Harness) AssertFileExists 

func (h *Harness) AssertFileExists(rel string)

AssertFileExists fails if the slash-relative path under the project root is missing.

func (*Harness) GitCommitAll 

func (h *Harness) GitCommitAll(msg string) string

GitCommitAll stages every change in the project root and commits it, returning the resulting HEAD hash.

func (*Harness) GitHead 

func (h *Harness) GitHead() string

GitHead returns the current HEAD commit hash.

func (*Harness) Init 

func (h *Harness) Init() Result

Init runs the real `specd init` command in the project root, scaffolding steering/, roles/, config.yml and AGENTS.md. Use it when a test needs role prompts, config, or the full project layout; New() alone only creates .specd/specs/.

func (*Harness) InitGit 

func (h *Harness) InitGit()

InitGit initialises a real but hermetic git repository at the project root and records an initial commit, giving commands that shell out to git (notably verify's gitHead capture) a HEAD to read. It uses a fixed in-repo identity and never touches the user's global git config. If git is not installed the test is skipped rather than failed.

This is the harness's substitute for a mocked git: a real repo confined to the temp root is both more faithful and simpler than reimplementing git plumbing.

func (*Harness) Path 

func (h *Harness) Path(rel string) string

Path returns an absolute path under the project root for a slash-relative path like ".specd/specs/foo/state.json".

func (*Harness) ReadFile 

func (h *Harness) ReadFile(rel string) string

ReadFile returns the contents of a slash-relative path, failing on error.

func (*Harness) Run 

func (h *Harness) Run(command string, args ...string) Result

Run executes a specd subcommand in-process — the same dispatch main.go performs — while capturing stdout, stderr and the integer exit code. Flags and positionals are parsed exactly like the real CLI via cli.ParseArgs, so `h.Run("task", "auth", "T1", "--status", "running")` mirrors the shell.

Run mutates os.Stdout/os.Stderr for the duration of the call; tests using it must not run in parallel.

func (*Harness) RunExpect 

func (h *Harness) RunExpect(want int, command string, args ...string) Result

RunExpect runs the command and fails the test if the exit code differs from want, surfacing the captured streams for diagnosis.

func (*Harness) Spec 

func (h *Harness) Spec(slug string) *SpecBuilder

Spec begins building a spec with the given slug. Title defaults to a title-cased slug and status to "requirements".

func (*Harness) SpecArtifact 

func (h *Harness) SpecArtifact(slug, name string) string

SpecArtifact reads a spec artifact (e.g. "tasks.md") for assertions.

func (*Harness) SpecPath 

func (h *Harness) SpecPath(slug, name string) string

SpecPath returns the absolute path of an artifact (e.g. "requirements.md") inside a spec.

func (*Harness) State 

func (h *Harness) State(slug string) *StateAsserter

State loads the spec's state.json and returns an asserter over it.

type Result 

type Result struct {
	Code   int    // process exit code
	Stdout string // everything written to os.Stdout during the command
	Stderr string // everything written to os.Stderr during the command
}

Result is the captured outcome of a single CLI invocation.

func (Result) OK 

func (r Result) OK() bool

OK reports whether the command exited 0.

func (Result) Out 

func (r Result) Out() string

Out is Stdout+Stderr joined, for substring assertions that don't care which stream a message landed on.

type SpecBuilder 

type SpecBuilder struct {
	// contains filtered or unexported fields
}

SpecBuilder is a fluent constructor for gate-valid specs. It writes the artifacts (requirements.md / design.md / tasks.md) and a consistent state.json directly to disk, bypassing the CLI gates so tests can start from any desired state. Defaults are chosen so the minimal spec still passes `specd check`.

func (*SpecBuilder) AddTask 

func (b *SpecBuilder) AddTask(t TaskSpec) *SpecBuilder

AddTask appends a task to the spec.

func (*SpecBuilder) Build 

func (b *SpecBuilder) Build() string

Build writes the spec to disk and returns its slug, failing the test on any error.

func (*SpecBuilder) DesignSection 

func (b *SpecBuilder) DesignSection(name, body string) *SpecBuilder

DesignSection sets or appends a single design section body.

func (*SpecBuilder) FullDesign 

func (b *SpecBuilder) FullDesign() *SpecBuilder

FullDesign fills all seven mandatory design sections with non-empty, TODO-free bodies so the design gate passes.

func (*SpecBuilder) Gate 

func (b *SpecBuilder) Gate(g core.Gate) *SpecBuilder

Gate sets the spec gate (e.g. core.GateAwaitingApproval).

func (*SpecBuilder) Orchestrated 

func (b *SpecBuilder) Orchestrated() *SpecBuilder

Orchestrated records executionMode=orchestrated (origin user) on the built spec, so Brain/Pinky entrypoints — which refuse Base specs — will drive it.

func (*SpecBuilder) Phase 

func (b *SpecBuilder) Phase(p core.Phase) *SpecBuilder

Phase overrides the spec phase.

func (*SpecBuilder) Req 

func (b *SpecBuilder) Req(name, story string, criteria ...string) *SpecBuilder

Req appends a requirement block. Each criterion must be a valid EARS sentence (e.g. "THE SYSTEM SHALL ..."); pass none to get a single default criterion.

func (*SpecBuilder) Status 

func (b *SpecBuilder) Status(s core.SpecStatus) *SpecBuilder

Status sets the spec status and derives a matching phase (override with Phase).

func (*SpecBuilder) Title 

func (b *SpecBuilder) Title(t string) *SpecBuilder

Title overrides the spec title.

func (*SpecBuilder) Turn 

func (b *SpecBuilder) Turn(n int) *SpecBuilder

Turn sets the turn counter.

type StateAsserter 

type StateAsserter struct {
	// contains filtered or unexported fields
}

StateAsserter provides chainable assertions over a spec's persisted state.json. Every failing assertion calls t.Errorf (non-fatal) so a single test can report several discrepancies at once; load failures are fatal.

func (*StateAsserter) AcceptanceStatus 

func (a *StateAsserter) AcceptanceStatus(key, want string) *StateAsserter

AcceptanceStatus asserts the recorded status ("pass"/"fail") of a criterion key like "1.2".

func (*StateAsserter) Gate 

func (a *StateAsserter) Gate(want core.Gate) *StateAsserter

func (*StateAsserter) HasBlocker 

func (a *StateAsserter) HasBlocker(taskID string) *StateAsserter

HasBlocker asserts a blocker is recorded for the given task id.

func (*StateAsserter) NoBlockers 

func (a *StateAsserter) NoBlockers() *StateAsserter

NoBlockers asserts the blocker list is empty.

func (*StateAsserter) Phase 

func (a *StateAsserter) Phase(want core.Phase) *StateAsserter

func (*StateAsserter) Raw 

func (a *StateAsserter) Raw() *core.State

Raw exposes the loaded state for assertions not covered by the helpers.

func (*StateAsserter) Status 

func (a *StateAsserter) Status(want core.SpecStatus) *StateAsserter

func (*StateAsserter) TaskEvidence 

func (a *StateAsserter) TaskEvidence(id, substr string) *StateAsserter

TaskEvidence asserts a task's recorded evidence contains substr.

func (*StateAsserter) TaskStatus 

func (a *StateAsserter) TaskStatus(id string, want core.TaskStatus) *StateAsserter

func (*StateAsserter) Turn 

func (a *StateAsserter) Turn(want int) *StateAsserter

type TaskSpec 

type TaskSpec struct {
	ID           string
	Title        string
	Role         string // default "craftsman"
	Wave         int    // default 1
	Depends      []string
	Requirements []int
	Verify       string // default "true" (a runnable, exit-0 command)
	Why          string
	Files        string
	Contract     string
	Acceptance   string
	Status       core.TaskStatus // default pending
}

TaskSpec describes one task to seed. Only ID is required; the rest default to values that satisfy the task-schema, DAG, evidence and sync gates.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.