eval/

directory
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 8, 2026 License: Apache-2.0

Directories

Path Synopsis
Package agent defines the interface every coding-agent CLI is driven through.
Package agent defines the interface every coding-agent CLI is driven through.
Package derive builds an eval suite for a skill that has none, by recovering a spec from the bundle and drafting evals from it.
Package derive builds an eval suite for a skill that has none, by recovering a spec from the bundle and drafting evals from it.
Package lint validates a skill bundle across three layers: spec conformance, quality, and injection (security scan).
Package lint validates a skill bundle across three layers: spec conformance, quality, and injection (security scan).
llm
Package llm is the single seam through which every model call passes.
Package llm is the single seam through which every model call passes.
agentcli
Package agentcli implements the LLM gateway by shelling out to a coding-agent CLI in print mode, billing to the operator's existing subscription.
Package agentcli implements the LLM gateway by shelling out to a coding-agent CLI in print mode, billing to the operator's existing subscription.
api
Package api implements the LLM gateway against a direct Anthropic-compatible HTTP endpoint.
Package api implements the LLM gateway against a direct Anthropic-compatible HTTP endpoint.
fake
Package fake provides an in-memory Gateway for tests.
Package fake provides an in-memory Gateway for tests.
Package propose turns what a candidate missed in a contest into one change to the skill's prose.
Package propose turns what a candidate missed in a contest into one change to the skill's prose.
Package provider is the one place the environment is mapped onto an LLM backend.
Package provider is the one place the environment is mapped onto an LLM backend.
Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.
Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Package sandbox is the seam every evaluation run executes through.
Package sandbox is the seam every evaluation run executes through.
docker
Package docker implements the sandbox driver on the Docker CLI.
Package docker implements the sandbox driver on the Docker CLI.
imagespec
Package imagespec renders everything an image needs as text: the base Dockerfile, a per-skill layer over it, the proxy configuration that enforces a network allowlist, and the digest that makes a layer cacheable.
Package imagespec renders everything an image needs as text: the base Dockerfile, a per-skill layer over it, the proxy configuration that enforces a network allowlist, and the digest that makes a layer cacheable.
kubernetes
Package kubernetes implements the sandbox driver on the Kubernetes API.
Package kubernetes implements the sandbox driver on the Kubernetes API.
northflank
Package northflank implements the sandbox driver on Northflank's API.
Package northflank implements the sandbox driver on Northflank's API.
resolve
Package resolve is the one place the environment becomes a sandbox driver.
Package resolve is the one place the environment becomes a sandbox driver.
sandboxtest
Package sandboxtest holds the contract every sandbox.Driver must satisfy.
Package sandboxtest holds the contract every sandbox.Driver must satisfy.
Package spec holds the SkillSpec IR that generation, suite drafting, and scoring all derive from.
Package spec holds the SkillSpec IR that generation, suite drafting, and scoring all derive from.
Package store owns the .whetstone/ workspace: SQLite index, completion cache, and the on-disk artifact layout.
Package store owns the .whetstone/ workspace: SQLite index, completion cache, and the on-disk artifact layout.
Package suite is a skill's eval set: the evals/evals.json tasks it is measured against, the evals/triggers.json queries its description is measured against, and a shipped distractor pack that makes a trigger measurement discriminating.
Package suite is a skill's eval set: the evals/evals.json tasks it is measured against, the evals/triggers.json queries its description is measured against, and a shipped distractor pack that makes a trigger measurement discriminating.
assets
Package assets embeds the static suite assets.
Package assets embeds the static suite assets.
Package tune optimizes a skill's description for trigger accuracy.
Package tune optimizes a skill's description for trigger accuracy.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL