testutil

package
v3.0.0-dev.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 21, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package testutil provides shared test helpers for Cogo, most notably FakeModel — a deterministic implementation of google.golang.org/adk/v2/model.LLM that lets us drive end-to-end agent tests without burning real tokens.

Index

Constants

This section is empty.

Variables

View Source
var ModelFacingBans = []struct {
	Phrase string
	Why    string
}{
	{"generously", "a frequency instruction the persona cannot see or countermand (#905)"},
	{"the codebase", "assumes a repository; many deployments have none"},
	{"shipping a feature", "assumes a code workload"},
	{"code review", "assumes a code workload"},
	{"code search", "assumes a code workload; say what the tool does — search file contents"},
	{"code investigation", "assumes a code workload; say what the tool does — read files, search them, list directories"},
	{"debugging session", "assumes a code workload"},
	{"source file", "assumes a code workload; a file is a file"},
	{"completion summary", "names a genre, not a content obligation"},
	{"status update", "names a genre, not a content obligation"},
	{"one-paragraph", "prescribes a document shape the caller's AGENTS.md should pick"},
	{"one-sentence detail", "names a genre and a length, not a content obligation"},
}

Tool descriptions and arg schemas are system-prompt-weight text that nothing reviews: AGENTS.md gets careful scrutiny, a `Description:` string in a Go file gets read as code. They outrank the persona at the point where the model decides what to call, and a recipe author can neither see nor override them (#909, #910).

core-agent began as an interactive coding agent and now also runs as a headless, long-lived daemon consuming machine signals, where "after shipping a feature" and "the codebase" describe nothing that happens. mark_task_done is the confirmed instance where that frame broke a live deployment: its "use this generously at natural task boundaries" beat a persona that explicitly forbade the behaviour, and its "one-paragraph completion summary" ARG SCHEMA — not the description — shaped what the model actually wrote (#905).

Two rules, in the order they matter:

  1. A string-typed argument on a model-facing tool is a WRITING PROMPT. It must name what the text has to CONTAIN (findings, evidence, the proposed change), never what GENRE of document it is. A genre name carries a rhetorical mode with it — "summary" implies retrospection, "report" implies an audience, "status" implies a state machine — and the model honours that mode over the persona, because satisfying the schema is a precondition for the call succeeding at all.

  2. Branch when the branch changes what is TRUE about this build (whenTool, gate.HasTool, sciontoolOnPath); delete when it only changes what is TYPICAL. There is no interactive/headless bit to switch on — the session that motivated #909 was a headless daemon with an operator attached over the TUI, i.e. both at once — so a mode-varying description would be computed from a fact that can change after it is baked. One neutral string per tool.

This list is the only thing that stops re-drift, because the review gate reads a changed description as a changed line of code. Precedent: markTaskDoneRepeatStatus is already a constant asserted by a test because the content IS the contract.

It lives here rather than beside any one catalog because the five packages that register model-facing tools — pkg/tools, pkg/tools/agentic, pkg/agent/background, pkg/agent/autonomous and pkg/agent — must not drift from each other. Two return tools disagreeing about whether a status line is acceptable is exactly the defect the audit found.

#909 shipped saying four, and pkg/agent was the missing one (#919) — which meant mark_task_done, the tool whose description started this whole thread, was the single tool with no re-drift guard. If you add a package that registers a tool a model can call, it gets a description_neutrality_test.go and this sentence gets a new name in it. The count is load-bearing: it is the only record of what "every registered tool" is being claimed over.

Matching is on lowercased text, so entries here must be lowercase.

Functions

func ApproverContext

func ApproverContext(ctx context.Context) context.Context

ApproverContext returns ctx carrying an approver context with a non-empty task, which is what a turn context gives the gate.

func ApproverToolContext

func ApproverToolContext() adkagent.Context

ApproverToolContext is ApproverContext as an adkagent.Context, for calling a tool's handler directly. Only the context.Context methods are backed; anything else panics on the nil embedded interface, which is the right failure for a handler that needs more.

func ModelFacingBanViolations

func ModelFacingBanViolations(text string) []string

ModelFacingBanViolations returns one message per banned phrase found in text. Empty means clean.

func ModelFacingText

func ModelFacingText(tl tool.Tool) (texts []string, scannedSchema bool)

ModelFacingText returns every string a tool puts in front of a model: its description plus its whole arg schema. The second return is false when the tool does not expose a declaration, i.e. its arg schema went unscanned and the caller should say so rather than pass quietly.

The schema goes in as marshalled JSON rather than a typed walk. functiontool populates ParametersJsonSchema (an `any` holding a jsonschema document), not the typed genai.Schema, so there is no struct to walk — and scanning the serialized form is what keeps this robust to a nested object or an array-of-objects arg growing a description later, which a hand-written walk would silently miss.

func PruneWalkDir

func PruneWalkDir(root, path, name string) bool

PruneWalkDir reports whether a directory met during a walk of this repo's source should be skipped entirely. path is the directory's full path, name its base; root is the walk root and is never pruned, however it happens to be named.

Every dot-directory is pruned, which is a rule about working state rather than a list of known offenders. `.git` is the obvious one; `.claude` is the one that actually broke a suite, because this repo's git worktrees live there and each is a **full second checkout**, so a walk that descends into it reports another branch's source as though it were this tree's. The 2026-09-04 analysis saw that as two test failures on a machine with 26 worktrees present, against a tree that passes `go test ./...` from a clean shallow clone (#964).

Pruning the class rather than the names is the point: `.claude` was added by name after it broke something, which left the next tool to put a checkout under a dot-directory to break it again. A suite that fails on the developer's untracked working state teaches people to ignore the suite, and that is a worse outcome than any single false negative it might now miss — a walk of this repo's own source has no business in a dot-directory in the first place, since nothing the module builds or ships lives in one.

Callers keep their own additional exclusions (an unswept top-level tree, a self-exemption) where those are about what the walk *means* rather than about what counts as this repo's source.

func RepoRoot

func RepoRoot(startFile string) (string, error)

RepoRoot walks up from startFile to the directory holding go.mod. Callers pass their own compiled-in path from runtime.Caller, so the answer is derived from the source tree rather than from the working directory, which `go test` sets to the package directory.

func UndeclaredArgRefs

func UndeclaredArgRefs(tl tool.Tool) (refs []string, checked bool)

UndeclaredArgRefs returns the argument names a tool's own description tells the model to populate that the tool does not actually declare.

This exists because #909's first draft rewrote the autonomous loop's default report_done description to say "put your actual findings in the result argument" — and that branch builds a coretools.NewLifecycleTool, whose arguments are {state, detail}. ADK validates function args with additionalProperties:false, so a model that obeyed would emit a hard validation error and the run would never receive its done signal. The two return tools in this repo take different argument names, and a sweep that only reads prose cannot tell that apart from a wording change.

The second return is false when the tool exposes no declaration or no arg schema, i.e. nothing was checked.

Types

type ApproverProbe

type ApproverProbe struct {
	// contains filtered or unexported fields
}

ApproverProbe is a permissions.Approver that records every request and denies it, so a tool under test reaches the approver and then does nothing: no command runs, no file is written, no request is sent. It exists to prove a call site hands the gate the call's full arguments (#1175 phase 2).

func AutoGate

func AutoGate(t testing.TB, eligible ...string) (*permissions.Gate, *ApproverProbe)

AutoGate returns a ModeAuto gate with no prompter whose approver is the returned probe and may decide calls matching eligible ("tool:pattern", as in permissions.allow). Pair it with ApproverToolContext: without an approver context every call escalates before the approver is asked.

func AutoGateInScope

func AutoGateInScope(t testing.TB, root string, eligible ...string) (*permissions.Gate, *ApproverProbe)

AutoGateInScope is AutoGate with root as the declared path scope (the project root), for file tools: a write outside the scope is a path-scope prompt, which never reaches the approver.

func (*ApproverProbe) Judge

Judge records req and denies it.

func (*ApproverProbe) Requests

func (p *ApproverProbe) Requests() []permissions.ApproverRequest

Requests returns what the approver was asked, in order.

func (*ApproverProbe) RequireArgs

func (p *ApproverProbe) RequireArgs(t testing.TB, toolName string, want ...string)

RequireArgs fails t unless the approver was asked exactly once, about toolName, with Args containing every one of want. A want that only the full arguments carry — write_file's content, a prompt past byte 200 — is what distinguishes a call site that passes its arguments from one that passes the summary alone.

type FakeModel

type FakeModel struct {
	ModelName string
	Script    []ScriptedResponse
	// contains filtered or unexported fields
}

FakeModel implements model.LLM with scripted output for tests.

Each call to GenerateContent advances the script by one entry. If the script is exhausted, an empty TurnComplete response is returned.

func (*FakeModel) Calls

func (f *FakeModel) Calls() int

Calls returns the number of GenerateContent invocations seen so far. Useful in tests that want to assert on call count.

func (*FakeModel) GenerateContent

func (f *FakeModel) GenerateContent(_ context.Context, _ *model.LLMRequest, stream bool) iter.Seq2[*model.LLMResponse, error]

GenerateContent yields the next scripted response. When stream is true, each TextChunk is emitted as its own Partial event; when false, only the final consolidated event is emitted.

func (*FakeModel) Name

func (f *FakeModel) Name() string

Name reports the configured model name (or "fake" if unset).

type ScriptedResponse

type ScriptedResponse struct {
	TextChunks   []string
	InputTokens  int
	OutputTokens int
}

ScriptedResponse describes one turn of FakeModel output.

TextChunks are emitted as individual streaming Partial events (one per chunk) when the caller asks for a stream. After the chunks, FakeModel emits a single TurnComplete=true event whose Content is the concatenation of all chunks. In non-streaming mode (stream=false) only the final TurnComplete event is emitted.

InputTokens / OutputTokens populate the final event's UsageMetadata when set, so usage-tracking tests don't need a real model.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL