Documentation
¶
Overview ¶
Package testharness provides integration test helpers for Nexus. It wraps engine setup, plugin registration, and event assertions for use in Go test files.
h := testharness.New(t, "configs/test-minimal.yaml")
h.Run()
h.AssertNoSystemOutput()
h.AssertEventEmitted("io.output")
Index ¶
- type AnthropicJudge
- type Harness
- func (h *Harness) AssertBooted(pluginIDs ...string)
- func (h *Harness) AssertEventCount(eventType string, min, max int)
- func (h *Harness) AssertEventEmitted(eventType string)
- func (h *Harness) AssertEventNotEmitted(eventType string)
- func (h *Harness) AssertNoSystemOutput()
- func (h *Harness) AssertOutputContains(substring string)
- func (h *Harness) AssertOutputNotContains(substring string)
- func (h *Harness) AssertOutputSemantic(criteria string)
- func (h *Harness) AssertSessionArtifact(relPath string)
- func (h *Harness) AssertSystemOutputContains(substring string)
- func (h *Harness) AssertToolCalled(toolName string)
- func (h *Harness) AssertToolNotCalled(toolName string)
- func (h *Harness) Bus() engine.EventBus
- func (h *Harness) Events() []testio.CollectedEvent
- func (h *Harness) Run()
- func (h *Harness) SessionDir() string
- type Judge
- type JudgeResult
- type Option
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AnthropicJudge ¶
type AnthropicJudge struct {
// contains filtered or unexported fields
}
AnthropicJudge uses Claude Haiku to evaluate output against criteria.
func NewAnthropicJudge ¶
func NewAnthropicJudge(apiKey string) *AnthropicJudge
NewAnthropicJudge creates a judge using Claude Haiku.
func (*AnthropicJudge) Evaluate ¶
func (j *AnthropicJudge) Evaluate(ctx context.Context, output string, criteria string) (JudgeResult, error)
Evaluate asks Haiku whether the output satisfies the criteria. Returns pass/fail with reasoning.
type Harness ¶
type Harness struct {
// contains filtered or unexported fields
}
Harness wraps a Nexus engine for integration testing.
func New ¶
New creates a test harness from a config file. The config should use nexus.io.test as its IO plugin.
func (*Harness) AssertBooted ¶
AssertBooted fails if any of the given plugin IDs were not initialized. Checks against the lifecycle manager's active plugin list.
func (*Harness) AssertEventCount ¶
AssertEventCount fails if the count of events matching eventType is outside [min, max].
func (*Harness) AssertEventEmitted ¶
AssertEventEmitted fails if no event of the given type was collected.
func (*Harness) AssertEventNotEmitted ¶
AssertEventNotEmitted fails if any event of the given type was collected.
func (*Harness) AssertNoSystemOutput ¶
func (h *Harness) AssertNoSystemOutput()
AssertNoSystemOutput fails if any io.output event with role "system" was emitted. System-role outputs typically indicate gate vetoes or errors.
func (*Harness) AssertOutputContains ¶
AssertOutputContains fails if no io.output event with role "assistant" contains the given substring.
func (*Harness) AssertOutputNotContains ¶
AssertOutputNotContains fails if any assistant output contains the given substring.
func (*Harness) AssertOutputSemantic ¶
AssertOutputSemantic uses the configured LLM judge to evaluate whether assistant output satisfies the given criteria. Skips if no judge configured.
func (*Harness) AssertSessionArtifact ¶
AssertSessionArtifact fails if the given relative path does not exist in the session dir.
func (*Harness) AssertSystemOutputContains ¶
AssertSystemOutputContains fails if no io.output event with role "system" contains the given substring. Gates emit system-role output when they veto.
func (*Harness) AssertToolCalled ¶
AssertToolCalled fails if no tool.invoke event matches the given tool name.
func (*Harness) AssertToolNotCalled ¶
AssertToolNotCalled fails if any tool.invoke event matches the given tool name.
func (*Harness) Bus ¶
Bus returns the engine's event bus. Use this before Run() to subscribe custom handlers for events the test IO plugin's wildcard collector does not observe — notably `before:*` vetoable events, which the bus dispatches only to explicitly-subscribed handlers.
func (*Harness) Events ¶
func (h *Harness) Events() []testio.CollectedEvent
Events returns all collected events from the test run.
func (*Harness) Run ¶
func (h *Harness) Run()
Run boots the engine, waits for the test IO plugin to signal completion (or timeout), then stops the engine. Call assertion methods after Run returns.
func (*Harness) SessionDir ¶
SessionDir returns the absolute path of the session workspace created by the harness. Empty before Run; remains valid after Run when paired with WithKeepSession.
type Judge ¶
type Judge interface {
Evaluate(ctx context.Context, output string, criteria string) (JudgeResult, error)
}
Judge evaluates whether LLM output satisfies given criteria. Implementations may use an LLM, local model, or rule-based system.
type JudgeResult ¶
JudgeResult is the outcome of a semantic evaluation.
type Option ¶
type Option func(*Harness)
Option configures the test harness.
func WithFailFast ¶
func WithFailFast() Option
WithFailFast disables the bus's panic-recovery wrapper so handler panics propagate with their original stack trace instead of being swallowed into a core.error event. Useful while iterating on a flaky test where the recovered emit obscures the real failure.
func WithKeepSession ¶
func WithKeepSession() Option
WithKeepSession keeps the session directory unconditionally so tests can read post-run artifacts (the journal, plugin data dirs, metadata). The caller is responsible for cleanup; using t.TempDir() as the sessions root is the typical pattern.
func WithRetainSession ¶
func WithRetainSession() Option
WithRetainSession keeps the session directory on test failure for debugging.
func WithTimeout ¶
WithTimeout overrides the default session timeout.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package contract provides a lightweight unit-level test harness for asserting Nexus plugin event contracts (Subscriptions / Emissions).
|
Package contract provides a lightweight unit-level test harness for asserting Nexus plugin event contracts (Subscriptions / Emissions). |