testharness

package
v0.32.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package testharness provides integration test helpers for Nexus. It wraps engine setup, plugin registration, and event assertions for use in Go test files.

h := testharness.New(t, "configs/test-minimal.yaml")
h.Run()
h.AssertNoSystemOutput()
h.AssertEventEmitted("io.output")

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewEngine added in v0.31.0

func NewEngine(t *testing.T, configPath string, opts ...Option) *engine.Engine

NewEngine builds an engine from configPath with every plugin registered (the registry cmd/nexus uses) and core.sessions.root / core.storage.root redirected into a per-test temp dir removed when the test ends, exactly as New does. It neither boots nor stops the engine: it is for tests whose driver is not the scripted nexus.io.test lifecycle (an inbound HTTP request against a serve transport, say) and so cannot use Harness.Run, but must still keep their state out of ~/.nexus. A relative configPath resolves against the project root.

Only the options that shape engine construction take effect here (WithConfigRewrite, WithConfiguredRoots); the rest configure Harness.Run and are ignored.

Types

type AnthropicJudge

type AnthropicJudge struct {
	// contains filtered or unexported fields
}

AnthropicJudge uses Claude Haiku to evaluate output against criteria.

func NewAnthropicJudge

func NewAnthropicJudge(apiKey string) *AnthropicJudge

NewAnthropicJudge creates a judge using Claude Haiku.

func (*AnthropicJudge) Evaluate

func (j *AnthropicJudge) Evaluate(ctx context.Context, output string, criteria string) (JudgeResult, error)

Evaluate asks Haiku whether the output satisfies the criteria. Returns pass/fail with reasoning.

type Harness

type Harness struct {
	// contains filtered or unexported fields
}

Harness wraps a Nexus engine for integration testing.

func New

func New(t *testing.T, configPath string, opts ...Option) *Harness

New creates a test harness from a config file. The config should use nexus.io.test as its IO plugin.

Unless WithConfiguredRoots is passed, the harness overrides the config's core.sessions.root and core.storage.root with directories under a fresh per-test temp dir, removed when the test ends (or retained, and logged, when WithRetainSession is set and the test failed).

func (*Harness) AssertBooted

func (h *Harness) AssertBooted(pluginIDs ...string)

AssertBooted fails if any of the given plugin IDs were not initialized. Checks against the lifecycle manager's active plugin list.

func (*Harness) AssertEventCount

func (h *Harness) AssertEventCount(eventType string, min, max int)

AssertEventCount fails if the count of events matching eventType is outside [min, max].

func (*Harness) AssertEventEmitted

func (h *Harness) AssertEventEmitted(eventType string)

AssertEventEmitted fails if no event of the given type was collected.

func (*Harness) AssertEventNotEmitted

func (h *Harness) AssertEventNotEmitted(eventType string)

AssertEventNotEmitted fails if any event of the given type was collected.

func (*Harness) AssertNoSystemOutput

func (h *Harness) AssertNoSystemOutput()

AssertNoSystemOutput fails if any io.output event with role "system" was emitted. System-role outputs typically indicate gate vetoes or errors.

func (*Harness) AssertOutputContains

func (h *Harness) AssertOutputContains(substring string)

AssertOutputContains fails if no io.output event with role "assistant" contains the given substring.

func (*Harness) AssertOutputNotContains

func (h *Harness) AssertOutputNotContains(substring string)

AssertOutputNotContains fails if any assistant output contains the given substring.

func (*Harness) AssertOutputSemantic

func (h *Harness) AssertOutputSemantic(criteria string)

AssertOutputSemantic uses the configured LLM judge to evaluate whether assistant output satisfies the given criteria. Skips if no judge configured.

func (*Harness) AssertSessionArtifact

func (h *Harness) AssertSessionArtifact(relPath string)

AssertSessionArtifact fails if the given relative path does not exist in the session dir.

func (*Harness) AssertSystemOutputContains

func (h *Harness) AssertSystemOutputContains(substring string)

AssertSystemOutputContains fails if no io.output event with role "system" contains the given substring. Gates emit system-role output when they veto.

func (*Harness) AssertToolCalled

func (h *Harness) AssertToolCalled(toolName string)

AssertToolCalled fails if no tool.invoke event matches the given tool name.

func (*Harness) AssertToolNotCalled

func (h *Harness) AssertToolNotCalled(toolName string)

AssertToolNotCalled fails if any tool.invoke event matches the given tool name.

func (*Harness) Bus

func (h *Harness) Bus() engine.EventBus

Bus returns the engine's event bus. Use this before Run() to subscribe custom handlers for events the test IO plugin's wildcard collector does not observe — notably `before:*` vetoable events, which the bus dispatches only to explicitly-subscribed handlers.

func (*Harness) Events

func (h *Harness) Events() []testio.CollectedEvent

Events returns all collected events from the test run.

func (*Harness) Run

func (h *Harness) Run()

Run boots the engine, waits for the test IO plugin to signal completion (or timeout), then stops the engine. Call assertion methods after Run returns.

func (*Harness) SessionDir

func (h *Harness) SessionDir() string

SessionDir returns the absolute path of the session workspace created by the harness. Empty before Run; remains valid after Run when paired with WithKeepSession.

func (*Harness) TempRoot added in v0.31.0

func (h *Harness) TempRoot() string

TempRoot returns the per-test temp dir the harness redirected core.sessions.root (TempRoot()/sessions) and core.storage.root (TempRoot()/data) into. Empty under WithConfiguredRoots.

type Judge

type Judge interface {
	Evaluate(ctx context.Context, output string, criteria string) (JudgeResult, error)
}

Judge evaluates whether LLM output satisfies given criteria. Implementations may use an LLM, local model, or rule-based system.

type JudgeResult

type JudgeResult struct {
	Pass   bool
	Reason string
}

JudgeResult is the outcome of a semantic evaluation.

type Option

type Option func(*Harness)

Option configures the test harness.

func WithConfigRewrite added in v0.31.0

func WithConfigRewrite(fn func([]byte) []byte) Option

WithConfigRewrite registers fn to transform the raw config file bytes before the engine is constructed from them (and before the sessions and storage roots are redirected). Rewrites run in the order given. It lets a test keep a checked-in config runnable by hand — a fixed loopback port, say — while substituting a per-test value, without writing a temp copy.

func WithConfiguredRoots added in v0.31.0

func WithConfiguredRoots() Option

WithConfiguredRoots keeps core.sessions.root and core.storage.root exactly as the config file sets them. Without it the harness redirects both into a per-test temp dir so no harness-driven test writes under ~/.nexus or shares state with another run. Use it only for a test that deliberately points the engine at a root it prepared itself and must find again.

func WithFailFast

func WithFailFast() Option

WithFailFast disables the bus's panic-recovery wrapper so handler panics propagate with their original stack trace instead of being swallowed into a core.error event. Useful while iterating on a flaky test where the recovered emit obscures the real failure.

func WithJudge

func WithJudge(j Judge) Option

WithJudge sets a custom semantic judge for Tier 2 assertions.

func WithKeepSession

func WithKeepSession() Option

WithKeepSession keeps the session directory unconditionally so tests can read post-run artifacts (the journal, plugin data dirs, metadata) after Run returns. The directory still lives under the harness's per-test temp root, which is removed when the test ends.

func WithRetainSession

func WithRetainSession() Option

WithRetainSession keeps the session directory on test failure for debugging.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout overrides the default session timeout.

Directories

Path Synopsis
Package contract provides a lightweight unit-level test harness for asserting Nexus plugin event contracts (Subscriptions / Emissions).
Package contract provides a lightweight unit-level test harness for asserting Nexus plugin event contracts (Subscriptions / Emissions).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL