Documentation
¶
Overview ¶
Package verify — default registry wiring + tool-input path extraction.
Two pieces glued here:
DefaultRegistry() returns a Registry pre-loaded with every built-in verifier. The agent boot path calls this once and hands the registry to the dispatcher.
ExtractWritePaths inspects a tool call's input JSON for keys that name a written file (path, file_path, target, etc.) so the dispatcher can run verifiers without each write tool opting in. New write tools work out of the box as long as they use one of the conventional key names.
Package verify — Go build verifier.
Runs `go build .` rooted at the modified file's package. Catches the failure modes that matter:
- hallucinated function names ("undefined: foo")
- hallucinated method signatures ("too many arguments")
- imports of packages that don't exist
- syntax errors from sloppy edits
We deliberately don't run tests here — that's Batch G2's follow-on (test-on-write), too slow for the inline-after-Edit case.
Package verify — reward-hacking detector (paper #48 design input).
Specific failure mode OpenAI's monitoring paper called out: agents editing test files to make them pass instead of fixing the bug. "All green" without the underlying code actually being fixed.
Detection heuristic: when a test file is modified but the corresponding source file isn't modified in the same turn, flag it. This is intentionally a WARNING, not a block — false positives happen (adding a new test for already-working code; refactoring a flaky assertion; renaming a test). The model sees the warning and either justifies the change or fixes the actual bug.
Heuristic IS heuristic — we treat it as a nudge, not a gate.
Package verify — syntax verifiers for declarative formats.
TOML/JSON/YAML files are easy to write code-completing into invalid states (unmatched braces, trailing commas, etc). Inline parse-only verification catches these before they propagate into "agent thinks the config is valid, downstream consumer crashes".
Each verifier reads the on-disk file (the write tool has already flushed) and parses. No shell-out, no temp files — pure in-process validation, sub-millisecond for typical config sizes.
Package verify — post-write verifiers (Batch G2 hallucination catching).
After every successful Write/Edit/Patch the agent runs the affected files through the matching verifier. A failure becomes a new tool message the model sees on its NEXT turn — "you wrote code that doesn't compile, here's the error" — closing the loop on hallucinated function calls / syntax errors / corrupt config files inside the same turn.
Design choices:
- Verifiers are per-file-extension. Picking the right verifier is a hash lookup; we don't sniff content because file extensions are reliable enough in agent workflows.
- Every verifier declares its own timeout. A slow build mustn't stall the agent's loop. Timeout = "skipped" (NOT failure) — "we couldn't check" is not the same as "we found a problem".
- Verifiers run sequentially per turn. A typical Edit touches 1–3 files; parallelizing would complicate timeout accounting for negligible win.
- The dispatcher returns a SINGLE consolidated message rather than firing one per file — fewer tokens per turn.
Index ¶
- Variables
- func ExtractWritePaths(toolName string, input json.RawMessage) []string
- func FormatRewardHackMessage(findings []RewardHackFinding) string
- func FormatToolMessage(results []VerifyResult) string
- func IsWriteCall(name string, input json.RawMessage) bool
- func IsWriteTool(toolName string) bool
- type GoVerifier
- type JSONVerifier
- type Registry
- type RewardHackFinding
- type TOMLVerifier
- type Verifier
- type VerifyResult
- type YAMLVerifier
Constants ¶
This section is empty.
Variables ¶
var WriteToolNames = map[string]bool{ "fs": true, "fs_write": true, "fs_edit": true, "edit": true, "write": true, "patch": true, "apply": true, "create": true, "slice": true, }
WriteToolNames is the set of tool names whose calls produce a post-write verification pass. Maintained as a denylist-of-everything- else rather than scraping the registry, because some non-write tools take a "path" argument (fs_read) and we'd otherwise verify reads too.
Functions ¶
func ExtractWritePaths ¶
func ExtractWritePaths(toolName string, input json.RawMessage) []string
ExtractWritePaths scans the tool input JSON for path-typed keys. Returns a deduplicated, cleaned list of paths the tool wrote to. Caller resolves them against cwd before passing to VerifyPaths.
Multi-file payloads (e.g. patch with `files: [{path: "..."}, ...]`) are handled by walking the JSON tree, not just top-level keys. Anything that LOOKS like a path under one of pathKeys gets collected.
func FormatRewardHackMessage ¶
func FormatRewardHackMessage(findings []RewardHackFinding) string
FormatRewardHackMessage renders findings as a single tool-message body the agent sees on its next turn. Empty when no findings — caller skips emission.
func FormatToolMessage ¶
func FormatToolMessage(results []VerifyResult) string
FormatToolMessage condenses verifier results into a single string the agent emits as a "tool"-role message on the next turn. Empty return ("") means "nothing to surface" — every file passed or was skipped. The dispatcher decides whether to emit.
Passing files are intentionally omitted from the output. The model doesn't need a "✓ everything fine" wall; it needs to know what broke.
func IsWriteCall ¶
func IsWriteCall(name string, input json.RawMessage) bool
IsWriteCall distinguishes mutating actions of multiplexed tools. Merely reading a JSON/Go file must not trigger build checks or write tracking.
func IsWriteTool ¶
IsWriteTool reports whether toolName can write. Multiplexed tools such as fs also require IsWriteCall to distinguish their read and write actions.
Types ¶
type GoVerifier ¶
type GoVerifier struct {
// MaxTime caps build duration. Default 20s — enough for most
// packages, short enough that an agent loop doesn't visibly
// stall. Builds that exceed return skipped=true (NOT failure).
MaxTime time.Duration
}
GoVerifier runs `go build` on the package containing the modified file. Build runs from the package directory so module resolution uses the right go.mod.
func NewGoVerifier ¶
func NewGoVerifier() *GoVerifier
NewGoVerifier returns a GoVerifier with sane defaults.
func (*GoVerifier) Name ¶
func (g *GoVerifier) Name() string
func (*GoVerifier) Timeout ¶
func (g *GoVerifier) Timeout() time.Duration
func (*GoVerifier) Verify ¶
func (g *GoVerifier) Verify(ctx context.Context, absPath string, content []byte) (bool, string, bool)
Verify runs `go build .` in the file's directory. ok=true when the build succeeds; detail captures stderr on failure.
Syntax checks use the supplied bytes; the build checks the on-disk package. The writing tool must flush before verification.
type JSONVerifier ¶
type JSONVerifier struct{}
JSONVerifier checks .json syntax. We unmarshal into any so the parser accepts both objects and arrays at the root.
func NewJSONVerifier ¶
func NewJSONVerifier() *JSONVerifier
func (*JSONVerifier) Name ¶
func (j *JSONVerifier) Name() string
func (*JSONVerifier) Timeout ¶
func (j *JSONVerifier) Timeout() time.Duration
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry maps file extensions (lowercase, leading dot) to a verifier. Lookups are case-insensitive. Last write wins so plugins can override built-ins.
func DefaultRegistry ¶
func DefaultRegistry() *Registry
DefaultRegistry returns a registry pre-loaded with built-in verifiers. Callers can Register more or override existing ones.
func NewRegistry ¶
func NewRegistry() *Registry
NewRegistry returns an empty registry. Callers wire built-ins via Register or via DefaultRegistry().
type RewardHackFinding ¶
type RewardHackFinding struct {
TestPath string
ExpectedCode string // what we'd expect to see modified alongside
Reason string
}
RewardHackFinding describes one test-without-code pair detected in a turn's written paths.
func AuditPaths ¶
func AuditPaths(paths []string) []RewardHackFinding
AuditPaths walks the list of paths written in a single agent turn and returns findings for each test file that lacks a corresponding code-file modification. paths should be the deduplicated absolute (or cwd-relative — both work, we just need a consistent set) list of every file the turn's tools touched.
Returns empty slice when nothing's suspicious — including the common case where the agent legitimately added a new test for existing code. We only flag "test changed but code did NOT" with strong-enough confidence.
type TOMLVerifier ¶
type TOMLVerifier struct{}
TOMLVerifier checks .toml syntax via the standard parser.
func NewTOMLVerifier ¶
func NewTOMLVerifier() *TOMLVerifier
func (*TOMLVerifier) Name ¶
func (t *TOMLVerifier) Name() string
func (*TOMLVerifier) Timeout ¶
func (t *TOMLVerifier) Timeout() time.Duration
type Verifier ¶
type Verifier interface {
Name() string
Timeout() time.Duration
// Verify the file at absPath. Most verifiers read from disk
// because tools like `go build` need to see the package as a
// whole, not just the modified bytes. content (the in-memory
// proposed bytes) is provided for verifiers that prefer to
// validate-before-write semantics; most ignore it.
Verify(ctx context.Context, absPath string, content []byte) (ok bool, detail string, skipped bool)
}
Verifier checks a single file. Returns ok=true when the content is well-formed for this file type; ok=false when something is wrong. Detail is the human-readable error (build stderr, parse error) the model reads on its next turn.
Verifiers run with a context bounded by Timeout(). ctx.Done firing during execution should bail and return (false, "<timeout>", true) — the third return value `skipped=true` signals "treat as pass". We don't fail a turn because the user's machine is slow.
type VerifyResult ¶
VerifyResult is one file's outcome. Skipped tracks "couldn't run" distinctly from "ran and passed" so a summary can report "3 ok, 1 skipped (no verifier)".
func VerifyPaths ¶
func VerifyPaths(ctx context.Context, reg *Registry, paths []string) []VerifyResult
VerifyPaths runs the matching verifier for each path. Returns one result per path (in input order). nil registry => empty results.
type YAMLVerifier ¶
type YAMLVerifier struct{}
YAMLVerifier checks .yaml/.yml syntax via yaml.v3.
func NewYAMLVerifier ¶
func NewYAMLVerifier() *YAMLVerifier
func (*YAMLVerifier) Name ¶
func (y *YAMLVerifier) Name() string
func (*YAMLVerifier) Timeout ¶
func (y *YAMLVerifier) Timeout() time.Duration
Source Files