Documentation
¶
Overview ¶
Package testkit is what a consumer uses to test its tools, policy, and agent against the runtime without a model: an interruption harness that crashes the loop at a chosen point and resumes it in a fresh Driver, a policy conformance table, schema fuzzing of tool arguments, and a render equivalence check. It is built on the public API alone and on the scripted package, which it re-exports nothing of.
A crash is a real one as far as the runtime can tell: the hook at the crash point parks the goroutine executing the run and the harness closes its Store, so the heartbeat stops and the lease is never released, which is what a process that died leaves behind. Resume in a fresh Driver on a fresh open of the same file is refused with ErrRunLeased until that lease expires, as a restarted process is, so the harness sets a short Config.LeaseTTL and waits it out once per crash. The parked goroutine is released when the test ends; its writes fail against the closed Store and nothing of its reaches the file.
Index ¶
- Constants
- func CheckPolicy(t testing.TB, p agentrt.Policy, tools []agentrt.Tool, cases []PolicyCase)
- func Never(t testing.TB, p agentrt.Policy, tools []agentrt.Tool, class agentrt.SideEffect, ...)
- func Rendered(t testing.TB, agent agentrt.Agent, in agentrt.StepInput) agentrt.ModelRequest
- func SameRenders(t testing.TB, fresh func() (Scenario, agentrt.Agent), crashes ...Crash)
- type Call
- type Crash
- type Fuzz
- type Point
- type PolicyCase
- type Result
- type Scenario
- type Verdict
Examples ¶
Constants ¶
const Refused agentrt.PolicyOutcome = "refused"
Refused is the Want of a PolicyCase whose arguments the runtime refuses before policy: an unknown tool, arguments that fail the tool's schema, or JSON the boundary rejects. It is the kit's value, never a policy's.
Variables ¶
This section is empty.
Functions ¶
func CheckPolicy ¶
CheckPolicy evaluates every case against p and reports the ones that disagree as one table, with what the policy answered and why. Validation is the runtime's own, agentrt.CompileTool and its Check, and a case the runtime refuses never reaches p.
Example ¶
A conformance table is decided through the runtime first, so a case the schema refuses is Refused before any policy sees it, and a disagreement prints as one table.
package main
import (
"context"
"fmt"
"io"
"os"
"testing"
"time"
agentrt "github.com/joeylking/agent-runtime"
"github.com/joeylking/agent-runtime/testkit"
)
// exampleTool records its calls; the examples are about what the runtime
// does around a call, not what the tool does.
type exampleTool struct {
spec agentrt.ToolSpec
calls int
}
func (t *exampleTool) Spec() agentrt.ToolSpec { return t.spec }
func (t *exampleTool) Call(_ context.Context, c agentrt.ToolCall) (agentrt.ToolResult, error) {
t.calls++
return agentrt.ToolResult{Content: c.Args, Summary: "sent"}, nil
}
func newExampleTool(name string, se agentrt.SideEffect) *exampleTool {
return &exampleTool{spec: agentrt.ToolSpec{Name: name, Description: name, SideEffect: se, Timeout: time.Second,
InputSchema: []byte(`{"type":"object","properties":{"to":{"type":"string","minLength":1}},"required":["to"],"additionalProperties":false}`)}}
}
// The examples run the kit under a testing.TB they create, as a test
// would pass its own; nothing here is expected to fail.
func exampleTB() testing.TB {
return &exampleT{}
}
type exampleT struct{ testing.TB }
func mustTempDir() string {
dir, err := os.MkdirTemp("", "testkit-example")
if err != nil {
panic(err)
}
return dir
}
func (exampleT) Helper() {}
func (exampleT) TempDir() string { return mustTempDir() }
func (exampleT) Cleanup(func()) {}
func (exampleT) Errorf(format string, args ...any) { fmt.Printf("FAIL: "+format+"\n", args...) }
func (exampleT) Fatalf(format string, args ...any) { panic(fmt.Sprintf(format, args...)) }
func (exampleT) Logf(string, ...any) {}
func (exampleT) Fatal(args ...any) { panic(fmt.Sprint(args...)) }
func (exampleT) Error(args ...any) { fmt.Println("FAIL:", fmt.Sprint(args...)) }
func (exampleT) Name() string { return "example" }
func (exampleT) FailNow() { panic("FailNow") }
func (exampleT) Failed() bool { return false }
func (exampleT) Fail() {}
func (exampleT) Log(...any) {}
func (exampleT) Skip(...any) {}
func (exampleT) SkipNow() {}
func (exampleT) Skipf(string, ...any) {}
func (exampleT) Skipped() bool { return false }
func (exampleT) Setenv(string, string) {}
func (exampleT) Chdir(string) {}
func (exampleT) Context() context.Context { return context.Background() }
func (exampleT) Attr(string, string) {}
func (exampleT) Output() io.Writer { return io.Discard }
func main() {
tools := []agentrt.Tool{newExampleTool("send", agentrt.RemoteMutation), newExampleTool("wipe", agentrt.Destructive)}
testkit.CheckPolicy(exampleTB(), agentrt.DefaultPolicy(), tools, []testkit.PolicyCase{
{Name: "send asks", Tool: "send", Args: `{"to":"ops"}`, Want: agentrt.RequireApproval, Kind: "remote_mutation"},
{Name: "wipe denied", Tool: "wipe", Args: `{"to":"all"}`, Want: agentrt.Deny},
{Name: "no recipient", Tool: "send", Args: `{}`, Want: testkit.Refused},
{Name: "wrong expectation", Tool: "wipe", Args: `{"to":"all"}`, Want: agentrt.Allow},
})
}
Output: FAIL: testkit: 1 of 4 policy cases disagree: case request want got detail wrong expectation wipe {"to":"all"} allow deny side effect destructive is deny by policy
func Never ¶
func Never(t testing.TB, p agentrt.Policy, tools []agentrt.Tool, class agentrt.SideEffect, outcome agentrt.PolicyOutcome, f Fuzz, views ...agentrt.RunView)
Never asserts that p never answers outcome for a tool whose side effect is class, over f.Count generated arguments per such tool that pass the runtime's validation, in each of views (the empty view when none are given). A class with no tool fails: the assertion would hold of nothing.
func Rendered ¶
Rendered is the model request agent makes for in: Decide is called with a ModelCaller that records the first request and answers it with an empty end_turn reply. It is the request a recording keys on, so what a replay depends on is exactly what it returns.
func SameRenders ¶
SameRenders runs the Scenario fresh returns twice, with the same crashes, and fails unless the agent fresh returns renders a byte-identical model request from what the two runs handed their agents at every decision, the decisions after a pause or a crash included. fresh is called once per run: a Scenario whose tools keep state, or an agent that caches by run, must not be shared, and a deterministic Now and NewID in the Scenario make the two records identical in ids and times, which an agent that renders either depends on.
Types ¶
type Call ¶
type Call struct {
Step int
Tool string
Args json.RawMessage
// Returned is false for the call a Crash at AfterToolEffect stopped
// after: the tool did its work and the runtime never recorded it.
Returned bool
}
Call is one invocation a consumer tool received from the runtime.
type Crash ¶
Crash names a point at which the harness stops the loop. Step is the index of the step the event belongs to; it is ignored for a run-level Point. Every pause is a stop as well, with no Crash needed: the loop returns, the Operator decides, and a fresh Driver resumes.
type Fuzz ¶
type Fuzz struct {
// Seed selects the sequence. Zero means 1.
Seed uint64
// Count is how many arguments Valid returns and at most how many
// Invalid returns. Zero means 32.
Count int
// MaxItems bounds a generated array where the schema does not, and
// MaxLength a generated string, in characters. Zero means 4 and 32.
MaxItems int
MaxLength int
// MaxDepth bounds nesting where the schema recurses: below it, optional
// properties are left out and arrays are as short as allowed. Zero
// means 4.
MaxDepth int
}
Fuzz generates tool arguments from a tool's JSON Schema, with the same schema library the runtime validates with. Generation is deterministic for a Seed and bounded in size. It covers the subset the consumers' schemas use: type (one or several), enum, const, object properties with required, additionalProperties, minProperties and maxProperties, array items with minItems, maxItems, uniqueItems and prefixItems, string minLength and maxLength, numeric minimum and maximum (exclusive too), anyOf and oneOf by picking a branch, and $ref within the schema. A schema using pattern, multipleOf, allOf, not, if, patternProperties, propertyNames, dependencies, contains, or the unevaluated keywords is reported as one the kit cannot generate for.
Example ¶
Valid arguments come from the schema with a fixed seed; the near misses each break it in one way, after the cases the runtime refuses before any schema.
package main
import (
"fmt"
"github.com/joeylking/agent-runtime/testkit"
)
func main() {
f := testkit.Fuzz{Seed: 3, Count: 3, MaxLength: 6}
schema := []byte(`{"type":"object","properties":{"to":{"type":"string","minLength":1,"maxLength":8},"urgent":{"type":"boolean"}},"required":["to"],"additionalProperties":false}`)
valid, _ := f.Valid(schema)
for _, v := range valid {
fmt.Println("valid ", string(v))
}
invalid, _ := f.Invalid(schema)
fmt.Println("invalid", len(invalid), "cases; the first:", string(invalid[0][:12]), "...")
}
Output: valid {"to":"YSe4ß","urgent":true} valid {"to":"Z\"A1a","urgent":false} valid {"to":"Tzfr"} invalid 6 cases; the first: {"_":[[[[[[[ ...
func (Fuzz) Check ¶
Check puts every tool's arguments through the runtime's acceptance of a tool call (agentrt.CompileTool and its Check): the valid ones must pass it, then reach the tool and return without a panic; the invalid ones must be refused, as the driver refuses them with an invalid_decision, and never reach the tool. A tool's own error on a valid argument is allowed, since a schema does not say what a tool accepts. Each call is bounded by the tool's Timeout, as the driver bounds it; a Terminal tool is called like any other.
func (Fuzz) Invalid ¶
func (f Fuzz) Invalid(schema json.RawMessage) ([]json.RawMessage, error)
Invalid returns up to Count near misses: valid arguments with one change the schema refuses (a required property missing, an unexpected one present, a wrong type, an enum outsider, a length, bound, or item count just past its limit, a non-object root), each confirmed invalid against the compiled schema, after the cases the runtime refuses at its boundary before any schema: nesting past its depth cap, a duplicate object key, invalid UTF-8, an unpaired surrogate escape, truncated JSON, and two values. A schema with nothing to violate yields the boundary cases only.
func (Fuzz) Valid ¶
func (f Fuzz) Valid(schema json.RawMessage) ([]json.RawMessage, error)
Valid returns Count arguments that satisfy schema. Each is checked against the compiled schema before it is returned.
type Point ¶
type Point string
Point is where a Crash stops the loop: right after the named event of a step has been committed and before whatever follows it, or, for AfterToolEffect, after the tool returned and before its outcome is recorded. Each is a point a dying process could stop at.
const ( // AfterStepStarted stops after step.started: the step exists, the // agent has not decided. Resume marks it interrupted while deciding // and continues. AfterStepStarted Point = agentrt.EventStepStarted // AfterDecided stops after step.decided: the decision is recorded, // policy has not seen it. Resume marks the step interrupted while // deciding; the request is never executed. AfterDecided Point = agentrt.EventStepDecided // AfterToolStarted stops after step.tool_started: the start is // recorded and the tool never ran. The record cannot tell this from // AfterToolEffect, so Resume treats both the same. AfterToolStarted Point = agentrt.EventStepToolStarted // AfterToolEffect stops once the tool has returned, its effect done, // and before the outcome is recorded: the step is left executing. AfterToolEffect Point = "tool.effect" // AfterToolFinished stops after step.tool_finished: the step is // complete and the next has not started. AfterToolFinished Point = agentrt.EventStepToolFinished // AfterResumed stops after run.resumed: on an approved resume that is // the transaction that also recorded the policy decision and the tool // start, so the approved request must not be executed again; on an // interrupted resume it is the move back to RUNNING before the loop. // Step is ignored. AfterResumed Point = agentrt.EventRunResumed // AfterInterrupted stops after step.interrupted: a resume took the run // over and died before reconciling or pausing it. The next resume must // still know the step was executing. AfterInterrupted Point = agentrt.EventStepInterrupted )
type PolicyCase ¶
type PolicyCase struct {
Name string
// Tool is a registered tool's name and Args its JSON arguments; empty
// Args mean {}. Both go through the runtime's validation first, as a
// decision does, and the policy sees the ToolRequest the runtime built.
Tool string
Args string
// View is the run state handed to the policy. Its Run.ID, when set,
// is also the request's.
View agentrt.RunView
// Want is the outcome, or Refused when the runtime must refuse the
// arguments before the policy sees them.
Want agentrt.PolicyOutcome
// Kind, when set, is the approval kind a require_approval must carry.
// Reason, Capability, and Presentation, when set, are substrings the
// decision's reason, capability JSON, and presentation JSON must
// contain.
Kind string
Reason string
Capability string
Presentation string
}
PolicyCase is one row of a conformance table: a request, the run state the policy sees, and the outcome it must give.
type Result ¶
type Result struct {
RunID string
Run agentrt.Run
Steps []agentrt.Step
Approvals []agentrt.Approval
Events []agentrt.Event
// Calls is every tool invocation across every Driver, in order.
Calls []Call
// Inputs is what the Agent was handed at every decision across every
// Driver, in order, for rendering checks.
Inputs []agentrt.StepInput
// Crashes are the crashes that fired, in order; Resumes counts the
// Drivers after the first.
Crashes []Crash
Resumes int
}
Result is what a Scenario left behind, read from a fresh open of the file after the last Driver returned.
func Run ¶
Run executes sc in a fresh file-backed store, crashing at each Crash in turn and resuming in a fresh Driver each time, until the run is terminal or an Operator leaves it waiting. It then asserts what the runtime promises across an interruption: a step never executes twice on its own (every execution past the first is under an approval of that step), an approved request executes at most once, every recorded tool start is followed by its finish or by the step's interruption, the run ends terminal or waiting with no step in flight, no run is left leased, and every Crash was reached. Failures are reported with t.Errorf, so a test sees all of them, and the Result is returned regardless.
Example ¶
A side effect interrupted after it took effect and before the runtime recorded it is not run again by the runtime: the run waits for an operator, and approving runs the same step once more under that approval. The result carries every call the tool received.
package main
import (
"context"
"fmt"
"io"
"os"
"testing"
"time"
agentrt "github.com/joeylking/agent-runtime"
"github.com/joeylking/agent-runtime/scripted"
"github.com/joeylking/agent-runtime/testkit"
)
// exampleTool records its calls; the examples are about what the runtime
// does around a call, not what the tool does.
type exampleTool struct {
spec agentrt.ToolSpec
calls int
}
func (t *exampleTool) Spec() agentrt.ToolSpec { return t.spec }
func (t *exampleTool) Call(_ context.Context, c agentrt.ToolCall) (agentrt.ToolResult, error) {
t.calls++
return agentrt.ToolResult{Content: c.Args, Summary: "sent"}, nil
}
func newExampleTool(name string, se agentrt.SideEffect) *exampleTool {
return &exampleTool{spec: agentrt.ToolSpec{Name: name, Description: name, SideEffect: se, Timeout: time.Second,
InputSchema: []byte(`{"type":"object","properties":{"to":{"type":"string","minLength":1}},"required":["to"],"additionalProperties":false}`)}}
}
// The examples run the kit under a testing.TB they create, as a test
// would pass its own; nothing here is expected to fail.
func exampleTB() testing.TB {
return &exampleT{}
}
type exampleT struct{ testing.TB }
func mustTempDir() string {
dir, err := os.MkdirTemp("", "testkit-example")
if err != nil {
panic(err)
}
return dir
}
func (exampleT) Helper() {}
func (exampleT) TempDir() string { return mustTempDir() }
func (exampleT) Cleanup(func()) {}
func (exampleT) Errorf(format string, args ...any) { fmt.Printf("FAIL: "+format+"\n", args...) }
func (exampleT) Fatalf(format string, args ...any) { panic(fmt.Sprintf(format, args...)) }
func (exampleT) Logf(string, ...any) {}
func (exampleT) Fatal(args ...any) { panic(fmt.Sprint(args...)) }
func (exampleT) Error(args ...any) { fmt.Println("FAIL:", fmt.Sprint(args...)) }
func (exampleT) Name() string { return "example" }
func (exampleT) FailNow() { panic("FailNow") }
func (exampleT) Failed() bool { return false }
func (exampleT) Fail() {}
func (exampleT) Log(...any) {}
func (exampleT) Skip(...any) {}
func (exampleT) SkipNow() {}
func (exampleT) Skipf(string, ...any) {}
func (exampleT) Skipped() bool { return false }
func (exampleT) Setenv(string, string) {}
func (exampleT) Chdir(string) {}
func (exampleT) Context() context.Context { return context.Background() }
func (exampleT) Attr(string, string) {}
func (exampleT) Output() io.Writer { return io.Discard }
func main() {
send := newExampleTool("send", agentrt.RemoteMutation)
ids := 0
res := testkit.Run(exampleTB(), testkit.Scenario{
Tools: []agentrt.Tool{send},
Policy: agentrt.SideEffectPolicy{agentrt.RemoteMutation: agentrt.Allow},
Agent: &scripted.Agent{Decisions: []agentrt.Decision{scripted.ToolCall("send", `{"to":"ops"}`, ""), scripted.Complete(`{}`)}},
NewID: func() string { ids++; return fmt.Sprintf("id-%d", ids) },
}, testkit.Crash{Step: 0, At: testkit.AfterToolEffect})
fmt.Println(res.Run.Status, "after", res.Resumes, "resumes")
for _, c := range res.Calls {
fmt.Println("call step", c.Step, c.Tool, string(c.Args), "returned:", c.Returned)
}
fmt.Println("approval", res.Approvals[0].Kind, res.Approvals[0].Status)
}
Output: COMPLETED after 2 resumes call step 0 send {"to":"ops"} returned: false call step 0 send {"to":"ops"} returned: true approval interrupted_side_effect approved
type Scenario ¶
type Scenario struct {
Tools []agentrt.Tool
Policy agentrt.Policy
// Agent is usually a scripted.Agent or the consumer's own replayer. It
// is handed to every Driver the harness builds, so it must decide from
// the recorded steps alone, as the runtime requires anyway. A replayer
// keyed by step index skips the decision of a step that was
// interrupted, because the interrupted step keeps its index.
Agent agentrt.Agent
Goal string
// Limits for the run. A zero MaxSteps means agentrt.DefaultLimits.
Limits agentrt.Limits
// Reconcile is the consumer's Config.Reconcile; nil leaves the runtime
// to pause an interrupted side effect for an operator.
Reconcile func(context.Context, agentrt.RunView) (agentrt.Reconciliation, error)
// Operator answers each pending approval. Nil approves every one.
Operator func(agentrt.Approval) Verdict
// Now and NewID are passed to every Driver, so a deterministic clock
// and id sequence make two runs of one Scenario record identically.
Now func() time.Time
NewID func() string
// DB is the database file to run in, for a consumer whose own tables
// share the runtime's file. Empty means a file of the harness's own.
DB string
// LeaseTTL is Config.LeaseTTL for every Driver. Zero means one second.
// A crashed Driver never releases its lease, Resume refuses the run
// until the lease expires, and the harness waits that long per crash,
// as a restarted process would; a value too short for the machine
// makes a live loop lose its own lease between heartbeats.
LeaseTTL time.Duration
}
Scenario is one run's worth of consumer code and script: what a consumer hands NewDriver, plus the operator's answers.
Source Files