Documentation
¶
Overview ¶
Package testharness provides integration test helpers for Nexus. It wraps engine setup, plugin registration, and event assertions for use in Go test files.
h := testharness.New(t, "configs/test-minimal.yaml")
h.Run()
h.AssertNoSystemOutput()
h.AssertEventEmitted("io.output")
Index ¶
- func NewEngine(t *testing.T, configPath string, opts ...Option) *engine.Engine
- type AnthropicJudge
- type Harness
- func (h *Harness) AssertBooted(pluginIDs ...string)
- func (h *Harness) AssertEventCount(eventType string, min, max int)
- func (h *Harness) AssertEventEmitted(eventType string)
- func (h *Harness) AssertEventNotEmitted(eventType string)
- func (h *Harness) AssertNoSystemOutput()
- func (h *Harness) AssertOutputContains(substring string)
- func (h *Harness) AssertOutputNotContains(substring string)
- func (h *Harness) AssertOutputSemantic(criteria string)
- func (h *Harness) AssertSessionArtifact(relPath string)
- func (h *Harness) AssertSystemOutputContains(substring string)
- func (h *Harness) AssertToolCalled(toolName string)
- func (h *Harness) AssertToolNotCalled(toolName string)
- func (h *Harness) Bus() engine.EventBus
- func (h *Harness) Events() []testio.CollectedEvent
- func (h *Harness) Run()
- func (h *Harness) SessionDir() string
- func (h *Harness) TempRoot() string
- type Judge
- type JudgeResult
- type Option
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewEngine ¶ added in v0.31.0
NewEngine builds an engine from configPath with every plugin registered (the registry cmd/nexus uses) and core.sessions.root / core.storage.root redirected into a per-test temp dir removed when the test ends, exactly as New does. It neither boots nor stops the engine: it is for tests whose driver is not the scripted nexus.io.test lifecycle (an inbound HTTP request against a serve transport, say) and so cannot use Harness.Run, but must still keep their state out of ~/.nexus. A relative configPath resolves against the project root.
Only the options that shape engine construction take effect here (WithConfigRewrite, WithConfiguredRoots); the rest configure Harness.Run and are ignored.
Types ¶
type AnthropicJudge ¶
type AnthropicJudge struct {
// contains filtered or unexported fields
}
AnthropicJudge uses Claude Haiku to evaluate output against criteria.
func NewAnthropicJudge ¶
func NewAnthropicJudge(apiKey string) *AnthropicJudge
NewAnthropicJudge creates a judge using Claude Haiku.
func (*AnthropicJudge) Evaluate ¶
func (j *AnthropicJudge) Evaluate(ctx context.Context, output string, criteria string) (JudgeResult, error)
Evaluate asks Haiku whether the output satisfies the criteria. Returns pass/fail with reasoning.
type Harness ¶
type Harness struct {
// contains filtered or unexported fields
}
Harness wraps a Nexus engine for integration testing.
func New ¶
New creates a test harness from a config file. The config should use nexus.io.test as its IO plugin.
Unless WithConfiguredRoots is passed, the harness overrides the config's core.sessions.root and core.storage.root with directories under a fresh per-test temp dir, removed when the test ends (or retained, and logged, when WithRetainSession is set and the test failed).
func (*Harness) AssertBooted ¶
AssertBooted fails if any of the given plugin IDs were not initialized. Checks against the lifecycle manager's active plugin list.
func (*Harness) AssertEventCount ¶
AssertEventCount fails if the count of events matching eventType is outside [min, max].
func (*Harness) AssertEventEmitted ¶
AssertEventEmitted fails if no event of the given type was collected.
func (*Harness) AssertEventNotEmitted ¶
AssertEventNotEmitted fails if any event of the given type was collected.
func (*Harness) AssertNoSystemOutput ¶
func (h *Harness) AssertNoSystemOutput()
AssertNoSystemOutput fails if any io.output event with role "system" was emitted. System-role outputs typically indicate gate vetoes or errors.
func (*Harness) AssertOutputContains ¶
AssertOutputContains fails if no io.output event with role "assistant" contains the given substring.
func (*Harness) AssertOutputNotContains ¶
AssertOutputNotContains fails if any assistant output contains the given substring.
func (*Harness) AssertOutputSemantic ¶
AssertOutputSemantic uses the configured LLM judge to evaluate whether assistant output satisfies the given criteria. Skips if no judge configured.
func (*Harness) AssertSessionArtifact ¶
AssertSessionArtifact fails if the given relative path does not exist in the session dir.
func (*Harness) AssertSystemOutputContains ¶
AssertSystemOutputContains fails if no io.output event with role "system" contains the given substring. Gates emit system-role output when they veto.
func (*Harness) AssertToolCalled ¶
AssertToolCalled fails if no tool.invoke event matches the given tool name.
func (*Harness) AssertToolNotCalled ¶
AssertToolNotCalled fails if any tool.invoke event matches the given tool name.
func (*Harness) Bus ¶
Bus returns the engine's event bus. Use this before Run() to subscribe custom handlers for events the test IO plugin's wildcard collector does not observe — notably `before:*` vetoable events, which the bus dispatches only to explicitly-subscribed handlers.
func (*Harness) Events ¶
func (h *Harness) Events() []testio.CollectedEvent
Events returns all collected events from the test run.
func (*Harness) Run ¶
func (h *Harness) Run()
Run boots the engine, waits for the test IO plugin to signal completion (or timeout), then stops the engine. Call assertion methods after Run returns.
func (*Harness) SessionDir ¶
SessionDir returns the absolute path of the session workspace created by the harness. Empty before Run; remains valid after Run when paired with WithKeepSession.
type Judge ¶
type Judge interface {
Evaluate(ctx context.Context, output string, criteria string) (JudgeResult, error)
}
Judge evaluates whether LLM output satisfies given criteria. Implementations may use an LLM, local model, or rule-based system.
type JudgeResult ¶
JudgeResult is the outcome of a semantic evaluation.
type Option ¶
type Option func(*Harness)
Option configures the test harness.
func WithConfigRewrite ¶ added in v0.31.0
WithConfigRewrite registers fn to transform the raw config file bytes before the engine is constructed from them (and before the sessions and storage roots are redirected). Rewrites run in the order given. It lets a test keep a checked-in config runnable by hand — a fixed loopback port, say — while substituting a per-test value, without writing a temp copy.
func WithConfiguredRoots ¶ added in v0.31.0
func WithConfiguredRoots() Option
WithConfiguredRoots keeps core.sessions.root and core.storage.root exactly as the config file sets them. Without it the harness redirects both into a per-test temp dir so no harness-driven test writes under ~/.nexus or shares state with another run. Use it only for a test that deliberately points the engine at a root it prepared itself and must find again.
func WithFailFast ¶
func WithFailFast() Option
WithFailFast disables the bus's panic-recovery wrapper so handler panics propagate with their original stack trace instead of being swallowed into a core.error event. Useful while iterating on a flaky test where the recovered emit obscures the real failure.
func WithKeepSession ¶
func WithKeepSession() Option
WithKeepSession keeps the session directory unconditionally so tests can read post-run artifacts (the journal, plugin data dirs, metadata) after Run returns. The directory still lives under the harness's per-test temp root, which is removed when the test ends.
func WithRetainSession ¶
func WithRetainSession() Option
WithRetainSession keeps the session directory on test failure for debugging.
func WithTimeout ¶
WithTimeout overrides the default session timeout.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package contract provides a lightweight unit-level test harness for asserting Nexus plugin event contracts (Subscriptions / Emissions).
|
Package contract provides a lightweight unit-level test harness for asserting Nexus plugin event contracts (Subscriptions / Emissions). |