eval/

directory
v0.21.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0

Directories

Path Synopsis
Package agent defines the interface every coding-agent CLI is driven through.
Package agent defines the interface every coding-agent CLI is driven through.
Package derive builds an eval suite for a skill that has none, by recovering a spec from the bundle and drafting evals from it.
Package derive builds an eval suite for a skill that has none, by recovering a spec from the bundle and drafting evals from it.
Package lint validates a skill bundle across three layers: spec conformance, quality, and injection (security scan).
Package lint validates a skill bundle across three layers: spec conformance, quality, and injection (security scan).
llm
Package llm is the single seam through which every model call passes.
Package llm is the single seam through which every model call passes.
api
Package api implements the LLM gateway against an HTTP endpoint that speaks either Anthropic's Messages API or OpenAI's Chat Completions API.
Package api implements the LLM gateway against an HTTP endpoint that speaks either Anthropic's Messages API or OpenAI's Chat Completions API.
fake
Package fake provides an in-memory Gateway for tests.
Package fake provides an in-memory Gateway for tests.
Package propose turns what a candidate missed in a contest into one change to the skill's prose.
Package propose turns what a candidate missed in a contest into one change to the skill's prose.
Package provider resolves an LLM backend: the server's providers for a worker (Session, Judge), and one Anthropic-format endpoint from the environment for the local `whetstone` CLI (Resolve).
Package provider resolves an LLM backend: the server's providers for a worker (Session, Judge), and one Anthropic-format endpoint from the environment for the local `whetstone` CLI (Resolve).
Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.
Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Package runner plans and executes an evaluation: which sessions to run, in what environment, with what resumed from a previous attempt.
Package sandbox is the seam every evaluation run executes through.
Package sandbox is the seam every evaluation run executes through.
docker
Package docker implements the sandbox driver on the Docker CLI.
Package docker implements the sandbox driver on the Docker CLI.
imagespec
Package imagespec renders everything an image needs as text: the base Dockerfile, a per-skill layer over it, the proxy configuration that enforces a network allowlist, and the digest that makes a layer cacheable.
Package imagespec renders everything an image needs as text: the base Dockerfile, a per-skill layer over it, the proxy configuration that enforces a network allowlist, and the digest that makes a layer cacheable.
kubernetes
Package kubernetes implements the sandbox driver on the Kubernetes API.
Package kubernetes implements the sandbox driver on the Kubernetes API.
northflank
Package northflank implements the sandbox driver on Northflank's API.
Package northflank implements the sandbox driver on Northflank's API.
resolve
Package resolve is the one place the environment becomes a sandbox driver.
Package resolve is the one place the environment becomes a sandbox driver.
sandboxtest
Package sandboxtest holds the contract every sandbox.Driver must satisfy.
Package sandboxtest holds the contract every sandbox.Driver must satisfy.
Package spec holds the SkillSpec IR that generation, suite drafting, and scoring all derive from.
Package spec holds the SkillSpec IR that generation, suite drafting, and scoring all derive from.
Package store owns the .whetstone/ workspace: SQLite index, completion cache, and the on-disk artifact layout.
Package store owns the .whetstone/ workspace: SQLite index, completion cache, and the on-disk artifact layout.
Package suite is a skill's eval set: the evals/evals.json tasks it is measured against, the evals/triggers.json queries its description is measured against, and a shipped distractor pack that makes a trigger measurement discriminating.
Package suite is a skill's eval set: the evals/evals.json tasks it is measured against, the evals/triggers.json queries its description is measured against, and a shipped distractor pack that makes a trigger measurement discriminating.
assets
Package assets embeds the static suite assets.
Package assets embeds the static suite assets.
Package tune optimizes a skill's description for trigger accuracy.
Package tune optimizes a skill's description for trigger accuracy.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL