eval/

directory
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0

Directories

Path Synopsis
Package agent defines the interface every coding-agent CLI is driven through.
Package agent defines the interface every coding-agent CLI is driven through.
claudecode
Package claudecode adapts the Claude Code CLI.
Package claudecode adapts the Claude Code CLI.
codex
Package codex reserves the Codex CLI adapter.
Package codex reserves the Codex CLI adapter.
cursor
Package cursor reserves the Cursor CLI adapter.
Package cursor reserves the Cursor CLI adapter.
opencode
Package opencode reserves the OpenCode CLI adapter.
Package opencode reserves the OpenCode CLI adapter.
Package bundlepath holds the one bundle-containment check every package that writes model-authored paths into a skill bundle shares.
Package bundlepath holds the one bundle-containment check every package that writes model-authored paths into a skill bundle shares.
Package contract compiles a spec.SkillSpec into a behavioural contract of deterministic matchers.
Package contract compiles a spec.SkillSpec into a behavioural contract of deterministic matchers.
Package drift measures how closely an agent's actual trajectory followed the behavioural contract compiled from a skill's specification.
Package drift measures how closely an agent's actual trajectory followed the behavioural contract compiled from a skill's specification.
Package gen writes a skill bundle — SKILL.md plus scripts/, references/, and assets/ — from an approved spec.SkillSpec.
Package gen writes a skill bundle — SKILL.md plus scripts/, references/, and assets/ — from an approved spec.SkillSpec.
Package lint validates a generated or imported skill bundle.
Package lint validates a generated or imported skill bundle.
llm
Package llm is the single seam through which every model call passes: interview, generation, suite drafting, judging, and repair.
Package llm is the single seam through which every model call passes: interview, generation, suite drafting, judging, and repair.
agentcli
Package agentcli implements the LLM gateway by shelling out to a coding-agent CLI in print mode, so every model call is billed to the operator's existing subscription rather than requiring an API key.
Package agentcli implements the LLM gateway by shelling out to a coding-agent CLI in print mode, so every model call is billed to the operator's existing subscription rather than requiring an API key.
api
Package api implements the LLM gateway against a direct Anthropic-compatible HTTP endpoint.
Package api implements the LLM gateway against a direct Anthropic-compatible HTTP endpoint.
fake
Package fake provides an in-memory Gateway for tests: these tests run with no LLM subscription, no API key, and no network, so the fake is how generation and suite behaviour get exercised.
Package fake provides an in-memory Gateway for tests: these tests run with no LLM subscription, no API key, and no network, so the fake is how generation and suite behaviour get exercised.
Package repair clusters evaluation failures into fixable defects and audits which assertions actually discriminate a skill's effect, so a repair loop spends its diff budget on real skill defects rather than on restating the same failure three times or rewording a broken verifier.
Package repair clusters evaluation failures into fixable defects and audits which assertions actually discriminate a skill's effect, so a repair loop spends its diff budget on real skill defects rather than on restating the same failure three times or rewording a broken verifier.
Package report defines the machine-readable result of an eval run: the schema a CI job reads, a later phase posts, and a future UI charts.
Package report defines the machine-readable result of an eval run: the schema a CI job reads, a later phase posts, and a future UI charts.
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Package sandbox is the seam every evaluation run executes through.
Package sandbox is the seam every evaluation run executes through.
docker
Package docker implements the sandbox driver on the Docker CLI.
Package docker implements the sandbox driver on the Docker CLI.
imagespec
Package imagespec renders everything an image needs as text: the base Dockerfile, a per-skill layer over it, the proxy configuration that enforces a network allowlist, and the digest that makes a layer cacheable.
Package imagespec renders everything an image needs as text: the base Dockerfile, a per-skill layer over it, the proxy configuration that enforces a network allowlist, and the digest that makes a layer cacheable.
sprites
Package sprites is the hosted, hardware-isolated sandbox driver: Firecracker microVMs with checkpoint/restore, which is what makes a 60-session tier one environment build instead of sixty, and what makes untrusted skills evaluable at all.
Package sprites is the hosted, hardware-isolated sandbox driver: Firecracker microVMs with checkpoint/restore, which is what makes a 60-session tier one environment build instead of sixty, and what makes untrusted skills evaluable at all.
Package score implements the LLM judge protocol: position-swapped pairwise comparisons and per-rule semantic scoring, both anchored on a worked rubric and both requiring a verbatim evidence quote before a verdict counts.
Package score implements the LLM judge protocol: position-swapped pairwise comparisons and per-rule semantic scoring, both anchored on a worked rubric and both requiring a verbatim evidence quote before a verdict counts.
Package spec holds the SkillSpec intermediate representation — the source of truth every other eval component derives from.
Package spec holds the SkillSpec intermediate representation — the source of truth every other eval component derives from.
Package store owns the .whetstone/ workspace: a SQLite index for versioned specs and the completion cache, plus the on-disk artifact layout.
Package store owns the .whetstone/ workspace: a SQLite index for versioned specs and the completion cache, plus the on-disk artifact layout.
Package suite drafts an evaluation suite from an approved spec.SkillSpec: a set of task packages, each with its own oracle and verifier, plus a trigger set used to measure whether a skill fires when it should (and stays silent when it shouldn't).
Package suite drafts an evaluation suite from an approved spec.SkillSpec: a set of task packages, each with its own oracle and verifier, plus a trigger set used to measure whether a skill fires when it should (and stays silent when it shouldn't).
assets
Package assets embeds the static suite assets.
Package assets embeds the static suite assets.
Package trajectory defines the normalized event schema every agent adapter maps its native stream onto.
Package trajectory defines the normalized event schema every agent adapter maps its native stream onto.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL