Help us shape the future of Go! Take the 2026 Go Developer Survey

testkit

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 5, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package testkit is what a consumer uses to test its tools, policy, and agent against the runtime without a model: an interruption harness that crashes the loop at a chosen point and resumes it in a fresh Driver, a policy conformance table, schema fuzzing of tool arguments, and a render equivalence check. A consumer whose own loop drives a Gate runs the same harness over that loop, with Scenario.Loop. It is built on the public API alone and on the scripted package, which it re-exports nothing of.

A crash is a real one as far as the runtime can tell: the hook at the crash point parks the goroutine executing the run and the harness closes its Store, so the heartbeat stops and the lease is never released, which is what a process that died leaves behind. Resume in a fresh Driver on a fresh open of the same file is refused with ErrRunLeased until that lease expires, as a restarted process is, so the harness sets a short Config.LeaseTTL and waits it out once per crash. The parked goroutine is released when the test ends; its writes fail against the closed Store and nothing of its reaches the file.

Index

Examples

Constants

View Source
const Refused agentrt.PolicyOutcome = "refused"

Refused is the Want of a PolicyCase whose arguments the runtime refuses before policy: an unknown tool, arguments that fail the tool's schema, or JSON the boundary rejects. It is the kit's value, never a policy's.

Variables

This section is empty.

Functions

func CheckPolicy

func CheckPolicy(t testing.TB, p agentrt.Policy, tools []agentrt.Tool, cases []PolicyCase)

CheckPolicy evaluates every case against p and reports the ones that disagree as one table, with what the policy answered and why. Validation is the runtime's own, agentrt.CompileTool and its Check, and a case the runtime refuses never reaches p.

Example

A conformance table is decided through the runtime first, so a case the schema refuses is Refused before any policy sees it, and a disagreement prints as one table.

package main

import (
	"context"
	"fmt"
	"io"
	"os"
	"testing"
	"time"

	agentrt "github.com/joeylking/agent-runtime"
	"github.com/joeylking/agent-runtime/testkit"
)

// exampleTool records its calls; the examples are about what the runtime
// does around a call, not what the tool does.
type exampleTool struct {
	spec  agentrt.ToolSpec
	calls int
}

func (t *exampleTool) Spec() agentrt.ToolSpec { return t.spec }
func (t *exampleTool) Call(_ context.Context, c agentrt.ToolCall) (agentrt.ToolResult, error) {
	t.calls++
	return agentrt.ToolResult{Content: c.Args, Summary: "sent"}, nil
}

func newExampleTool(name string, se agentrt.SideEffect) *exampleTool {
	return &exampleTool{spec: agentrt.ToolSpec{Name: name, Description: name, SideEffect: se, Timeout: time.Second,
		InputSchema: []byte(`{"type":"object","properties":{"to":{"type":"string","minLength":1}},"required":["to"],"additionalProperties":false}`)}}
}

// The examples run the kit under a testing.TB they create, as a test
// would pass its own; nothing here is expected to fail.
func exampleTB() testing.TB {
	return &exampleT{}
}

type exampleT struct{ testing.TB }

func mustTempDir() string {
	dir, err := os.MkdirTemp("", "testkit-example")
	if err != nil {
		panic(err)
	}
	return dir
}

func (exampleT) Helper()                           {}
func (exampleT) TempDir() string                   { return mustTempDir() }
func (exampleT) Cleanup(func())                    {}
func (exampleT) Errorf(format string, args ...any) { fmt.Printf("FAIL: "+format+"\n", args...) }
func (exampleT) Fatalf(format string, args ...any) { panic(fmt.Sprintf(format, args...)) }
func (exampleT) Logf(string, ...any)               {}
func (exampleT) Fatal(args ...any)                 { panic(fmt.Sprint(args...)) }
func (exampleT) Error(args ...any)                 { fmt.Println("FAIL:", fmt.Sprint(args...)) }
func (exampleT) Name() string                      { return "example" }
func (exampleT) FailNow()                          { panic("FailNow") }
func (exampleT) Failed() bool                      { return false }
func (exampleT) Fail()                             {}
func (exampleT) Log(...any)                        {}
func (exampleT) Skip(...any)                       {}
func (exampleT) SkipNow()                          {}
func (exampleT) Skipf(string, ...any)              {}
func (exampleT) Skipped() bool                     { return false }
func (exampleT) Setenv(string, string)             {}
func (exampleT) Chdir(string)                      {}
func (exampleT) Context() context.Context          { return context.Background() }
func (exampleT) Attr(string, string)               {}
func (exampleT) Output() io.Writer                 { return io.Discard }

func main() {
	tools := []agentrt.Tool{newExampleTool("send", agentrt.RemoteMutation), newExampleTool("wipe", agentrt.Destructive)}
	testkit.CheckPolicy(exampleTB(), agentrt.DefaultPolicy(), tools, []testkit.PolicyCase{
		{Name: "send asks", Tool: "send", Args: `{"to":"ops"}`, Want: agentrt.RequireApproval, Kind: "remote_mutation"},
		{Name: "wipe denied", Tool: "wipe", Args: `{"to":"all"}`, Want: agentrt.Deny},
		{Name: "no recipient", Tool: "send", Args: `{}`, Want: testkit.Refused},
		{Name: "wrong expectation", Tool: "wipe", Args: `{"to":"all"}`, Want: agentrt.Allow},
	})
}
Output:
FAIL: testkit: 1 of 4 policy cases disagree:
case               request            want   got   detail
wrong expectation  wipe {"to":"all"}  allow  deny  side effect destructive is deny by policy

func Never

func Never(t testing.TB, p agentrt.Policy, tools []agentrt.Tool, class agentrt.SideEffect, outcome agentrt.PolicyOutcome, f Fuzz, views ...agentrt.RunView)

Never asserts that p never answers outcome for a tool whose side effect is class, over f.Count generated arguments per such tool that pass the runtime's validation, in each of views (the empty view when none are given). A class with no tool fails: the assertion would hold of nothing. A consumer that has no tool of the class, and asserts that its policy never allows one, adds StandIn(name, class) to tools: a tool the policy has never heard of, which it must answer for all the same.

func Rendered

Rendered is the model request agent makes for in: Decide is called with a ModelCaller that records the first request and answers it with an empty end_turn reply. It is the request a recording keys on, so what a replay depends on is exactly what it returns.

func SameRenders

func SameRenders(t testing.TB, fresh func() (Scenario, agentrt.Agent), crashes ...Crash)

SameRenders runs the Scenario fresh returns twice, with the same crashes, and fails unless the agent fresh returns renders a byte-identical model request from what the two runs handed their agents at every decision, the decisions after a pause or a crash included. fresh is called once per run: a Scenario whose tools keep state, or an agent that caches by run, must not be shared, and a deterministic Now and NewID in the Scenario make the two records identical in ids and times, which an agent that renders either depends on.

func StandIn added in v0.3.2

func StandIn(name string, class agentrt.SideEffect) agentrt.Tool

StandIn is a tool of side effect class for a tool set that has none, as Never needs: its schema is any JSON object, and a call returns the fixed result {"stand_in":name} without doing anything. In a Scenario its calls are recorded in Result.Calls, as every tool's are.

Types

type Call

type Call struct {
	Step int
	Tool string
	Args json.RawMessage
	// Returned is false for the call a Crash at AfterToolEffect stopped
	// after: the tool did its work and the runtime never recorded it.
	Returned bool
}

Call is one invocation a consumer tool received from the runtime.

type Crash

type Crash struct {
	Step int
	At   Point
}

Crash names a point at which the harness stops the loop. Step is the index of the step the event belongs to; it is ignored for a run-level Point. Every pause is a stop as well, with no Crash needed: the loop returns, the Operator decides, and a fresh Driver resumes.

func (Crash) String

func (c Crash) String() string

type Fuzz

type Fuzz struct {
	// Seed selects the sequence. Zero means 1.
	Seed uint64
	// Count is how many arguments Valid returns and at most how many
	// Invalid returns. Zero means 32.
	Count int
	// MaxItems bounds a generated array where the schema does not, and
	// MaxLength a generated string, in characters. Zero means 4 and 32.
	MaxItems  int
	MaxLength int
	// MaxDepth bounds nesting where the schema recurses: below it, optional
	// properties are left out and arrays are as short as allowed. Zero
	// means 4.
	MaxDepth int
}

Fuzz generates tool arguments from a tool's JSON Schema, with the same schema library the runtime validates with. Generation is deterministic for a Seed and bounded in size. It covers the subset the consumers' schemas use: type (one or several), enum, const, object properties with required, additionalProperties, minProperties and maxProperties, array items with minItems, maxItems, uniqueItems and prefixItems, string minLength and maxLength, numeric minimum and maximum (exclusive too), anyOf and oneOf by picking a branch, and $ref within the schema. A schema using pattern, multipleOf, allOf, not, if, patternProperties, propertyNames, dependencies, contains, or the unevaluated keywords is reported as one the kit cannot generate for.

Example

Valid arguments come from the schema with a fixed seed; the near misses each break it in one way, after the cases the runtime refuses before any schema.

package main

import (
	"fmt"

	"github.com/joeylking/agent-runtime/testkit"
)

func main() {
	f := testkit.Fuzz{Seed: 3, Count: 3, MaxLength: 6}
	schema := []byte(`{"type":"object","properties":{"to":{"type":"string","minLength":1,"maxLength":8},"urgent":{"type":"boolean"}},"required":["to"],"additionalProperties":false}`)
	valid, _ := f.Valid(schema)
	for _, v := range valid {
		fmt.Println("valid  ", string(v))
	}
	invalid, _ := f.Invalid(schema)
	fmt.Println("invalid", len(invalid), "cases; the first:", string(invalid[0][:12]), "...")
}
Output:
valid   {"to":"YSe4ß","urgent":true}
valid   {"to":"Z\"A1a","urgent":false}
valid   {"to":"Tzfr"}
invalid 6 cases; the first: {"_":[[[[[[[ ...

func (Fuzz) Check

func (f Fuzz) Check(t testing.TB, tools ...agentrt.Tool)

Check puts every tool's arguments through the runtime's acceptance of a tool call (agentrt.CompileTool and its Check): the valid ones must pass it, then reach the tool and return without a panic; the invalid ones must be refused, as the driver refuses them with an invalid_decision, and never reach the tool. A tool's own error on a valid argument is allowed, since a schema does not say what a tool accepts. Each call is bounded by the tool's Timeout, as the driver bounds it; a Terminal tool is called like any other.

func (Fuzz) Invalid

func (f Fuzz) Invalid(schema json.RawMessage) ([]json.RawMessage, error)

Invalid returns up to Count near misses: valid arguments with one change the schema refuses (a required property missing, an unexpected one present, a wrong type, an enum outsider, a length, bound, or item count just past its limit, a non-object root), each confirmed invalid against the compiled schema, after the cases the runtime refuses at its boundary before any schema: nesting past its depth cap, a duplicate object key, invalid UTF-8, an unpaired surrogate escape, truncated JSON, and two values. A schema with nothing to violate yields the boundary cases only.

func (Fuzz) Valid

func (f Fuzz) Valid(schema json.RawMessage) ([]json.RawMessage, error)

Valid returns Count arguments that satisfy schema. Each is checked against the compiled schema before it is returned.

type Point

type Point string

Point is where a Crash stops the loop: right after the named event of a step has been committed and before whatever follows it, or, for AfterToolEffect, after the tool returned and before its outcome is recorded. Each is a point a dying process could stop at.

const (
	// AfterStepStarted stops after step.started: the step exists, the
	// agent has not decided. Resume marks it interrupted while deciding
	// and continues.
	AfterStepStarted Point = agentrt.EventStepStarted
	// AfterDecided stops after step.decided: the decision is recorded,
	// policy has not seen it. Resume marks the step interrupted while
	// deciding; the request is never executed.
	AfterDecided Point = agentrt.EventStepDecided
	// AfterToolStarted stops after step.tool_started: the start is
	// recorded and the tool never ran. The record cannot tell this from
	// AfterToolEffect, so Resume treats both the same.
	AfterToolStarted Point = agentrt.EventStepToolStarted
	// AfterToolEffect stops once the tool has returned, its effect done,
	// and before the outcome is recorded: the step is left executing.
	AfterToolEffect Point = "tool.effect"
	// AfterToolFinished stops after step.tool_finished: the step is
	// complete and the next has not started.
	AfterToolFinished Point = agentrt.EventStepToolFinished
	// AfterResumed stops after run.resumed: on an approved resume that is
	// the transaction that also recorded the policy decision and the tool
	// start, so the approved request must not be executed again; on an
	// interrupted resume it is the move back to RUNNING before the loop.
	// Step is ignored.
	AfterResumed Point = agentrt.EventRunResumed
	// AfterAllowed stops once the policy has allowed a step's request and
	// before its tool start is written: a loop abandoned with an Allowed
	// verdict in hand, before Execute. The harness stops it as the policy
	// returns allow, which leaves the record exactly as that loop does,
	// because nothing about an allowed request but its decision is written
	// until Execute. Resume or Attach marks the step interrupted while
	// deciding; the request is never executed. A Driver reaches it too,
	// between its policy and the tool start.
	AfterAllowed Point = "policy.allowed"
	// AfterAttachAllowed stops after Attach returned Allowed for an
	// approved step, before Execute, and abandons the session: the run is
	// still WAITING with its grant approved and nothing written, so the
	// next Attach finds the grant and executes it once. Step is the
	// approved step's index. Only a Scenario with a Loop reaches it.
	AfterAttachAllowed Point = "attach.allowed"
	// AfterInterrupted stops after step.interrupted: a resume took the run
	// over and died before reconciling or pausing it. The next resume must
	// still know the step was executing.
	AfterInterrupted Point = agentrt.EventStepInterrupted
)

type PolicyCase

type PolicyCase struct {
	Name string
	// Tool is a registered tool's name and Args its JSON arguments; empty
	// Args mean {}. Both go through the runtime's validation first, as a
	// decision does, and the policy sees the ToolRequest the runtime built.
	Tool string
	Args string
	// View is the run state handed to the policy. Its Run.ID, when set,
	// is also the request's.
	View agentrt.RunView
	// Want is the outcome, or Refused when the runtime must refuse the
	// arguments before the policy sees them.
	Want agentrt.PolicyOutcome
	// Kind, when set, is the approval kind a require_approval must carry.
	// Reason, Capability, and Presentation, when set, are substrings the
	// decision's reason, capability JSON, and presentation JSON must
	// contain.
	Kind         string
	Reason       string
	Capability   string
	Presentation string
}

PolicyCase is one row of a conformance table: a request, the run state the policy sees, and the outcome it must give.

type Result

type Result struct {
	RunID     string
	Run       agentrt.Run
	Steps     []agentrt.Step
	Approvals []agentrt.Approval
	Events    []agentrt.Event
	// Calls is every tool invocation across every Driver, in order.
	Calls []Call
	// Inputs is what the Agent was handed at every decision across every
	// Driver, in order, for rendering checks.
	Inputs []agentrt.StepInput
	// Crashes are the crashes that fired, in order.
	Crashes []Crash
	// Resumes counts every Driver that ran the run after the first, which
	// started it: the resume after an operator's approval as well as the
	// resume after a crash. A resume refused while a crashed Driver's
	// lease is live, and the Driver that records the operator's decision,
	// are not counted. A run that pauses for one approval and crashes once
	// after it is resumed is started (Driver 1), resumed after the
	// approval (Driver 2, which crashes), and resumed after the crash
	// (Driver 3): Resumes is 3 - 1 = 2, with one Crash.
	Resumes int
}

Result is what a Scenario left behind, read from a fresh open of the file after the last Driver returned.

func Run

func Run(t testing.TB, sc Scenario, crashes ...Crash) Result

Run executes sc in a fresh file-backed store, crashing at each Crash in turn and resuming in a fresh Driver each time, or in a fresh Gate under the Scenario's Loop, until the run is terminal or an Operator leaves it waiting. It then asserts what the runtime promises across an interruption: a step never executes twice on its own (every execution past the first is under an approval of that step), an approved request executes at most once, every recorded tool start is followed by its finish or by the step's interruption, the run ends terminal or waiting with no step in flight, no run is left leased, and every Crash was reached. Failures are reported with t.Errorf, so a test sees all of them, and the Result is returned regardless.

Example

A side effect interrupted after it took effect and before the runtime recorded it is not run again by the runtime: the run waits for an operator, and approving runs the same step once more under that approval. The result carries every call the tool received.

package main

import (
	"context"
	"fmt"
	"io"
	"os"
	"testing"
	"time"

	agentrt "github.com/joeylking/agent-runtime"
	"github.com/joeylking/agent-runtime/scripted"
	"github.com/joeylking/agent-runtime/testkit"
)

// exampleTool records its calls; the examples are about what the runtime
// does around a call, not what the tool does.
type exampleTool struct {
	spec  agentrt.ToolSpec
	calls int
}

func (t *exampleTool) Spec() agentrt.ToolSpec { return t.spec }
func (t *exampleTool) Call(_ context.Context, c agentrt.ToolCall) (agentrt.ToolResult, error) {
	t.calls++
	return agentrt.ToolResult{Content: c.Args, Summary: "sent"}, nil
}

func newExampleTool(name string, se agentrt.SideEffect) *exampleTool {
	return &exampleTool{spec: agentrt.ToolSpec{Name: name, Description: name, SideEffect: se, Timeout: time.Second,
		InputSchema: []byte(`{"type":"object","properties":{"to":{"type":"string","minLength":1}},"required":["to"],"additionalProperties":false}`)}}
}

// The examples run the kit under a testing.TB they create, as a test
// would pass its own; nothing here is expected to fail.
func exampleTB() testing.TB {
	return &exampleT{}
}

type exampleT struct{ testing.TB }

func mustTempDir() string {
	dir, err := os.MkdirTemp("", "testkit-example")
	if err != nil {
		panic(err)
	}
	return dir
}

func (exampleT) Helper()                           {}
func (exampleT) TempDir() string                   { return mustTempDir() }
func (exampleT) Cleanup(func())                    {}
func (exampleT) Errorf(format string, args ...any) { fmt.Printf("FAIL: "+format+"\n", args...) }
func (exampleT) Fatalf(format string, args ...any) { panic(fmt.Sprintf(format, args...)) }
func (exampleT) Logf(string, ...any)               {}
func (exampleT) Fatal(args ...any)                 { panic(fmt.Sprint(args...)) }
func (exampleT) Error(args ...any)                 { fmt.Println("FAIL:", fmt.Sprint(args...)) }
func (exampleT) Name() string                      { return "example" }
func (exampleT) FailNow()                          { panic("FailNow") }
func (exampleT) Failed() bool                      { return false }
func (exampleT) Fail()                             {}
func (exampleT) Log(...any)                        {}
func (exampleT) Skip(...any)                       {}
func (exampleT) SkipNow()                          {}
func (exampleT) Skipf(string, ...any)              {}
func (exampleT) Skipped() bool                     { return false }
func (exampleT) Setenv(string, string)             {}
func (exampleT) Chdir(string)                      {}
func (exampleT) Context() context.Context          { return context.Background() }
func (exampleT) Attr(string, string)               {}
func (exampleT) Output() io.Writer                 { return io.Discard }

func main() {
	send := newExampleTool("send", agentrt.RemoteMutation)
	ids := 0
	res := testkit.Run(exampleTB(), testkit.Scenario{
		Tools:  []agentrt.Tool{send},
		Policy: agentrt.SideEffectPolicy{agentrt.RemoteMutation: agentrt.Allow},
		Agent:  &scripted.Agent{Decisions: []agentrt.Decision{scripted.ToolCall("send", `{"to":"ops"}`, ""), scripted.Complete(`{}`)}},
		NewID:  func() string { ids++; return fmt.Sprintf("id-%d", ids) },
	}, testkit.Crash{Step: 0, At: testkit.AfterToolEffect})

	fmt.Println(res.Run.Status, "after", res.Resumes, "resumes")
	for _, c := range res.Calls {
		fmt.Println("call step", c.Step, c.Tool, string(c.Args), "returned:", c.Returned)
	}
	fmt.Println("approval", res.Approvals[0].Kind, res.Approvals[0].Status)
}
Output:
COMPLETED after 2 resumes
call step 0 send {"to":"ops"} returned: false
call step 0 send {"to":"ops"} returned: true
approval interrupted_side_effect approved

func (Result) String

func (r Result) String() string

String summarizes the run and its steps as "index tool:status", for messages.

type Scenario

type Scenario struct {
	Tools  []agentrt.Tool
	Policy agentrt.Policy
	// Agent is usually a scripted.Agent or the consumer's own replayer. It
	// is handed to every Driver the harness builds, so it must decide from
	// the recorded steps alone, as the runtime requires anyway. A replayer
	// keyed by step index skips the decision of a step that was
	// interrupted, because the interrupted step keeps its index.
	Agent agentrt.Agent
	Goal  string
	// Limits for the run. A zero MaxSteps means agentrt.DefaultLimits.
	Limits agentrt.Limits
	// Reconcile is the consumer's Config.Reconcile; nil leaves the runtime
	// to pause an interrupted side effect for an operator.
	Reconcile func(context.Context, agentrt.RunView) (agentrt.Reconciliation, error)
	// Operator answers each pending approval. Nil approves every one.
	Operator func(agentrt.Approval) Verdict
	// Now and NewID are passed to every Driver, so a deterministic clock
	// and id sequence make two runs of one Scenario record identically.
	Now   func() time.Time
	NewID func() string
	// DB is the database file to run in, for a consumer whose own tables
	// share the runtime's file. Empty means a file of the harness's own.
	DB string
	// LeaseTTL is Config.LeaseTTL for every Driver. Zero means one second.
	// A crashed Driver never releases its lease, Resume refuses the run
	// until the lease expires, and the harness waits that long per crash,
	// as a restarted process would; a value too short for the machine
	// makes a live loop lose its own lease between heartbeats.
	LeaseTTL time.Duration
	// Loop, when set, is the consumer's own loop over a Gate, which the
	// harness runs in place of a Driver and its Agent. Each process opens a
	// fresh Gate on the scenario's tools, policy, reconciliation, clock,
	// ids, and lease, takes a session with Begin, or with Attach after a
	// pause or a crash, hands it to Loop, and closes it when Loop returns.
	// Loop drives the session until it is Done; when Current is not nil,
	// Attach left an approved step open for Execute. An error Loop returns
	// is reported. Result.Inputs is then empty, and SameRenders, which
	// renders what an Agent was handed, needs an Agent.
	Loop func(ctx context.Context, s *agentrt.Session) error
}

Scenario is one run's worth of consumer code and script: what a consumer hands NewDriver, plus the operator's answers.

type Verdict

type Verdict int

Verdict is an Operator's answer to a pending approval.

const (
	// Approve grants it and the harness resumes the run in a fresh Driver.
	Approve Verdict = iota
	// Reject refuses it; the runtime cancels the run.
	Reject
	// Leave decides nothing and the harness returns the waiting run.
	Leave
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL