whetstone

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 47 Imported by: 0

Documentation

Overview

Package whetstone implements the whetstone CLI: the authoring front end for the skill specification, generation, lint, contract, and suite engine under internal/eval, plus the evaluation commands (eval, drift, repair, report) built on top of it.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Execute

func Execute()

Execute runs the root command. A command that runs sandboxed sessions (eval, suite check) passes cmd.Context() into docker.Driver.Run and suite.Check, both of which already clean up their containers and networks on context cancellation — but only if something actually cancels that context on SIGINT/SIGTERM. Without signal.NotifyContext here, Ctrl-C is process death: the docker client is killed with no chance to run its own cleanup, and every in-flight run leaks a whetstone-net-* network and a whetstone-proxy-* container.

func RunDrift

func RunDrift(_ context.Context, skill, ref string) (*report.Report, error)

RunDrift loads the report for skill's ref ("latest" or an eval id) and returns it, so a caller (the command above, or a test) gets the same typed document `whetstone eval` produced.

func RunEval

func RunEval(ctx context.Context, req EvalRequest) error

RunEval resolves EvalDeps from the environment — the workspace store, a docker sandbox driver, the LLM gateway `whetstone doctor` would choose, and the linked-in adapter registry — and runs the eval.

func RunEvalWith

func RunEvalWith(ctx context.Context, d EvalDeps, req EvalRequest) (*report.Report, error)

RunEvalWith runs one evaluation end to end: load the approved spec and suite, refuse an eval against unchecked or void tasks, plan and execute the panel, score the result, and persist and write the report. See the package doc for the phase-by-phase breakdown; this function follows it in order.

func RunGen

func RunGen(ctx context.Context, skill string) error

RunGen regenerates the bundle for a skill from its approved spec and lints the result.

func RunInit

func RunInit(root string) (string, error)

RunInit creates or opens the workspace under root, defaulting to the working directory when root is empty, and returns the workspace directory.

func RunLint

func RunLint(path string, strict bool) (int, error)

RunLint lints the bundle at path and returns the process exit code.

The error return is reserved for lint being unable to run at all (an unreadable bundle); findings are reported through the exit code, so a failing bundle is not also an error the caller has to special-case.

func RunNew

func RunNew(ctx context.Context, intent string, yes bool) error

RunNew runs the full authoring pipeline for a new skill against the workspace and gateway this machine is configured for.

func RunPack

func RunPack(bundleDir, outPath string) (string, error)

RunPack lints bundleDir and, if it has no errors, writes a tar.gz of it to outPath with the eval sidecar and the spec removed, and returns the archive's resolved path. When outPath is empty, the archive is written beside the bundle — in its parent directory, as "<bundle>.tar.gz" — rather than inside it: an archive inside the directory being packed is content the next pack has to exclude (lint.Excluded's root-tarball case) purely because it exists.

Linting first is not a convenience: an archive built from a bundle that fails lint installs and then fails at use time, a long way from the cause.

func RunRepair

func RunRepair(ctx context.Context, skill string, maxIter int, yes bool) error

RunRepair resolves EvalDeps and a suite from the environment and runs the repair loop against skill.

func RunReport

func RunReport(skill, ref string, open bool, opener func(path string) error) (string, error)

RunReport renders the HTML report for skill's ref ("latest" or an eval id) to <sidecar>/reports/<eval id>/report.html and returns that path. When open is true it hands the path to opener; opener defaults to defaultOpener (nil is not a caller error — the command above always passes it explicitly, but a test that only cares about the render path can pass nil too as long as open is false).

opener is injected rather than this function shelling out to a browser itself: a test that launches a browser is a test nobody runs twice.

func RunSpecApprove

func RunSpecApprove(skill string) error

RunSpecApprove marks the latest stored version of a spec approved.

func RunSpecEdit

func RunSpecEdit(skill string) error

RunSpecEdit opens the spec YAML in $EDITOR and stores the edited result as a new version. The new version is unapproved: approval is per version, so an edit that skipped the gate would otherwise inherit the previous approval.

func RunSpecShow

func RunSpecShow(skill string) error

RunSpecShow prints the latest stored spec for a skill.

func RunSuiteCheck

func RunSuiteCheck(ctx context.Context, skill string, allowVoid bool) error

RunSuiteCheck runs the oracle gate over a skill's written suite: for every task, does the oracle solve it, does the task's own verifier accept that solution, and does the verifier reject an untouched workspace. A task failing any of those three is void — excluded from a later eval rather than fatal to it, which is what --allow-void is for. Without it, any void task makes this command exit non-zero, which is what makes it usable as a CI gate: an author cannot otherwise tell a broken oracle from a broken verifier without reading source, so the reason each void task carries names the check that failed.

func RunSuiteGen

func RunSuiteGen(ctx context.Context, skill string) error

RunSuiteGen drafts an evaluation suite from a skill's approved spec and writes it into the skill's eval sidecar.

func RunSuitePush

func RunSuitePush(ctx context.Context, req SuitePushRequest) error

RunSuitePush uploads a skill's written suite, together with the oracle-gate results `whetstone suite check` last recorded for it, to a Skael server. It refuses when no check has been recorded for the current suite ref — an uploaded suite without check results is not something the server can distinguish from a passing one, so this is caught here rather than there.

func SetVersion

func SetVersion(version, commit, date string)

SetVersion is called by main to inject build-time version info.

Types

type DoctorReport

type DoctorReport struct {
	// AgentCLI is the resolved path of the agent CLI on PATH, empty when none
	// was found.
	AgentCLI string `json:"agent_cli"`
	// AgentCLIVersion is what that CLI reported for --version.
	AgentCLIVersion string `json:"agent_cli_version,omitempty"`
	// AgentCLIError explains why no CLI is usable, when none is.
	AgentCLIError string `json:"agent_cli_error,omitempty"`
	// Gateway is which backend would serve model calls: subscription, api, or
	// none.
	Gateway string `json:"gateway"`
	// GatewayDetail explains that choice, including why it is "none".
	GatewayDetail string `json:"gateway_detail"`
	// Docker reports whether a docker binary is on PATH. Required by the
	// commands that run a sandbox: eval, repair, and suite check.
	Docker bool `json:"docker"`
	// DockerPath is the resolved docker binary, empty when absent.
	DockerPath string `json:"docker_path,omitempty"`
	// Adapters names every registered agent adapter, so a missing blank
	// import shows up here rather than as an unexplained gap in a panel.
	Adapters []string `json:"adapters"`
	// Judge is the calibration result, present only when --judge was passed.
	Judge *score.CalResult `json:"judge,omitempty"`
	// JudgeError explains why calibration was not run, when --judge was
	// passed but no gateway is available. doctor diagnoses; it does not fail.
	JudgeError string `json:"judge_error,omitempty"`
}

DoctorReport is what `whetstone doctor` found. Every field is filled on every run, including the failures: doctor is run when something is already wrong, so it diagnoses rather than errors.

func RunDoctor

func RunDoctor(ctx context.Context, withJudge bool) (*DoctorReport, error)

RunDoctor collects the environment report. It returns an error only when the report itself could not be produced — a missing CLI, a missing docker, and an unusable gateway are all reported in the report, because the command exists to be run when something is already broken.

type EvalDeps

type EvalDeps struct {
	Store  *store.Store
	Driver sandbox.Driver
	// Gateway is nil when no LLM backend is available. A nil Gateway degrades
	// Uplift to the pass-rate fallback; it does not fail the eval — the
	// deterministic pillars (Reliability, TriggerF1, Efficiency) are still a
	// usable measurement without a judge.
	Gateway  llm.Gateway
	Adapters func(name string) (agent.Adapter, bool)
	// Now defaults to time.Now.
	Now func() time.Time
	// Sleep defaults to time.Sleep; the runner's rate-limit backoff uses it.
	Sleep         func(time.Duration)
	EngineVersion string
}

EvalDeps is everything RunEvalWith needs, injected so a test can supply a fake driver, adapter registry, and gateway with no Docker, no network, and no LLM subscription. RunEval (below) resolves these from the environment the same way `whetstone doctor` and `whetstone suite check` do.

type EvalRequest

type EvalRequest struct {
	Skill       string
	Tier        runner.Tier
	Agents      []string
	Models      []string
	Concurrency int
	Untrusted   bool
	AllowVoid   bool
	// Resume, when non-zero, reuses that eval id rather than starting a new
	// one — see RunEvalWith for the suite/panel mismatch guard.
	Resume int64
	// TaskFilter, when non-empty, restricts scoring to tasks whose ID
	// appears in it — the repair loop's way of re-running only the affected
	// dev tasks (or, at the end, only the holdout split) without touching
	// the suite on disk. Filtering happens after the suite is loaded, so
	// SuiteRef still identifies the full suite the tasks were drawn from.
	TaskFilter []string
}

EvalRequest is one `whetstone eval` invocation.

type SuitePushRequest

type SuitePushRequest struct {
	Store    *store.Store
	Skill    string
	Endpoint string
	APIKey   string
}

SuitePushRequest holds the inputs to RunSuitePush.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL