Documentation
¶
Overview ¶
Package whetstone implements the whetstone CLI: the authoring front end for the skill specification, generation, lint, contract, and suite engine under internal/eval, plus the evaluation commands (eval, drift, repair, report) built on top of it.
Index ¶
- func Execute()
- func RunDrift(_ context.Context, skill, ref string) (*report.Report, error)
- func RunEval(ctx context.Context, req EvalRequest) error
- func RunEvalWith(ctx context.Context, d EvalDeps, req EvalRequest) (*report.Report, error)
- func RunGen(ctx context.Context, skill string) error
- func RunInit(root string) (string, error)
- func RunLint(path string, strict bool) (int, error)
- func RunNew(ctx context.Context, intent string, yes bool) error
- func RunPack(bundleDir, outPath string) (string, error)
- func RunRepair(ctx context.Context, skill string, maxIter int, yes bool) error
- func RunReport(skill, ref string, open bool, opener func(path string) error) (string, error)
- func RunSpecApprove(skill string) error
- func RunSpecEdit(skill string) error
- func RunSpecShow(skill string) error
- func RunSuiteCheck(ctx context.Context, skill string, allowVoid bool) error
- func RunSuiteGen(ctx context.Context, skill string) error
- func RunSuitePush(ctx context.Context, req SuitePushRequest) error
- func SetVersion(version, commit, date string)
- type DoctorReport
- type EvalDeps
- type EvalRequest
- type SuitePushRequest
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Execute ¶
func Execute()
Execute runs the root command. A command that runs sandboxed sessions (eval, suite check) passes cmd.Context() into docker.Driver.Run and suite.Check, both of which already clean up their containers and networks on context cancellation — but only if something actually cancels that context on SIGINT/SIGTERM. Without signal.NotifyContext here, Ctrl-C is process death: the docker client is killed with no chance to run its own cleanup, and every in-flight run leaks a whetstone-net-* network and a whetstone-proxy-* container.
func RunDrift ¶
RunDrift loads the report for skill's ref ("latest" or an eval id) and returns it, so a caller (the command above, or a test) gets the same typed document `whetstone eval` produced.
func RunEval ¶
func RunEval(ctx context.Context, req EvalRequest) error
RunEval resolves EvalDeps from the environment — the workspace store, a docker sandbox driver, the LLM gateway `whetstone doctor` would choose, and the linked-in adapter registry — and runs the eval.
func RunEvalWith ¶
RunEvalWith runs one evaluation end to end: load the approved spec and suite, refuse an eval against unchecked or void tasks, plan and execute the panel, score the result, and persist and write the report. See the package doc for the phase-by-phase breakdown; this function follows it in order.
func RunGen ¶
RunGen regenerates the bundle for a skill from its approved spec and lints the result.
func RunInit ¶
RunInit creates or opens the workspace under root, defaulting to the working directory when root is empty, and returns the workspace directory.
func RunLint ¶
RunLint lints the bundle at path and returns the process exit code.
The error return is reserved for lint being unable to run at all (an unreadable bundle); findings are reported through the exit code, so a failing bundle is not also an error the caller has to special-case.
func RunNew ¶
RunNew runs the full authoring pipeline for a new skill against the workspace and gateway this machine is configured for.
func RunPack ¶
RunPack lints bundleDir and, if it has no errors, writes a tar.gz of it to outPath with the eval sidecar and the spec removed, and returns the archive's resolved path. When outPath is empty, the archive is written beside the bundle — in its parent directory, as "<bundle>.tar.gz" — rather than inside it: an archive inside the directory being packed is content the next pack has to exclude (lint.Excluded's root-tarball case) purely because it exists.
Linting first is not a convenience: an archive built from a bundle that fails lint installs and then fails at use time, a long way from the cause.
func RunRepair ¶
RunRepair resolves EvalDeps and a suite from the environment and runs the repair loop against skill.
func RunReport ¶
RunReport renders the HTML report for skill's ref ("latest" or an eval id) to <sidecar>/reports/<eval id>/report.html and returns that path. When open is true it hands the path to opener; opener defaults to defaultOpener (nil is not a caller error — the command above always passes it explicitly, but a test that only cares about the render path can pass nil too as long as open is false).
opener is injected rather than this function shelling out to a browser itself: a test that launches a browser is a test nobody runs twice.
func RunSpecApprove ¶
RunSpecApprove marks the latest stored version of a spec approved.
func RunSpecEdit ¶
RunSpecEdit opens the spec YAML in $EDITOR and stores the edited result as a new version. The new version is unapproved: approval is per version, so an edit that skipped the gate would otherwise inherit the previous approval.
func RunSpecShow ¶
RunSpecShow prints the latest stored spec for a skill.
func RunSuiteCheck ¶
RunSuiteCheck runs the oracle gate over a skill's written suite: for every task, does the oracle solve it, does the task's own verifier accept that solution, and does the verifier reject an untouched workspace. A task failing any of those three is void — excluded from a later eval rather than fatal to it, which is what --allow-void is for. Without it, any void task makes this command exit non-zero, which is what makes it usable as a CI gate: an author cannot otherwise tell a broken oracle from a broken verifier without reading source, so the reason each void task carries names the check that failed.
func RunSuiteGen ¶
RunSuiteGen drafts an evaluation suite from a skill's approved spec and writes it into the skill's eval sidecar.
func RunSuitePush ¶
func RunSuitePush(ctx context.Context, req SuitePushRequest) error
RunSuitePush uploads a skill's written suite, together with the oracle-gate results `whetstone suite check` last recorded for it, to a Skael server. It refuses when no check has been recorded for the current suite ref — an uploaded suite without check results is not something the server can distinguish from a passing one, so this is caught here rather than there.
func SetVersion ¶
func SetVersion(version, commit, date string)
SetVersion is called by main to inject build-time version info.
Types ¶
type DoctorReport ¶
type DoctorReport struct {
// AgentCLI is the resolved path of the agent CLI on PATH, empty when none
// was found.
AgentCLI string `json:"agent_cli"`
// AgentCLIVersion is what that CLI reported for --version.
AgentCLIVersion string `json:"agent_cli_version,omitempty"`
// AgentCLIError explains why no CLI is usable, when none is.
AgentCLIError string `json:"agent_cli_error,omitempty"`
// Gateway is which backend would serve model calls: subscription, api, or
// none.
Gateway string `json:"gateway"`
// GatewayDetail explains that choice, including why it is "none".
GatewayDetail string `json:"gateway_detail"`
// Docker reports whether a docker binary is on PATH. Required by the
// commands that run a sandbox: eval, repair, and suite check.
Docker bool `json:"docker"`
// DockerPath is the resolved docker binary, empty when absent.
DockerPath string `json:"docker_path,omitempty"`
// Adapters names every registered agent adapter, so a missing blank
// import shows up here rather than as an unexplained gap in a panel.
Adapters []string `json:"adapters"`
// Judge is the calibration result, present only when --judge was passed.
Judge *score.CalResult `json:"judge,omitempty"`
// JudgeError explains why calibration was not run, when --judge was
// passed but no gateway is available. doctor diagnoses; it does not fail.
JudgeError string `json:"judge_error,omitempty"`
}
DoctorReport is what `whetstone doctor` found. Every field is filled on every run, including the failures: doctor is run when something is already wrong, so it diagnoses rather than errors.
func RunDoctor ¶
func RunDoctor(ctx context.Context, withJudge bool) (*DoctorReport, error)
RunDoctor collects the environment report. It returns an error only when the report itself could not be produced — a missing CLI, a missing docker, and an unusable gateway are all reported in the report, because the command exists to be run when something is already broken.
type EvalDeps ¶
type EvalDeps struct {
Store *store.Store
Driver sandbox.Driver
// Gateway is nil when no LLM backend is available. A nil Gateway degrades
// Uplift to the pass-rate fallback; it does not fail the eval — the
// deterministic pillars (Reliability, TriggerF1, Efficiency) are still a
// usable measurement without a judge.
Gateway llm.Gateway
Adapters func(name string) (agent.Adapter, bool)
// Now defaults to time.Now.
Now func() time.Time
// Sleep defaults to time.Sleep; the runner's rate-limit backoff uses it.
Sleep func(time.Duration)
EngineVersion string
}
EvalDeps is everything RunEvalWith needs, injected so a test can supply a fake driver, adapter registry, and gateway with no Docker, no network, and no LLM subscription. RunEval (below) resolves these from the environment the same way `whetstone doctor` and `whetstone suite check` do.
type EvalRequest ¶
type EvalRequest struct {
Skill string
Tier runner.Tier
Agents []string
Models []string
Concurrency int
Untrusted bool
AllowVoid bool
// Resume, when non-zero, reuses that eval id rather than starting a new
// one — see RunEvalWith for the suite/panel mismatch guard.
Resume int64
// TaskFilter, when non-empty, restricts scoring to tasks whose ID
// appears in it — the repair loop's way of re-running only the affected
// dev tasks (or, at the end, only the holdout split) without touching
// the suite on disk. Filtering happens after the suite is loaded, so
// SuiteRef still identifies the full suite the tasks were drawn from.
TaskFilter []string
}
EvalRequest is one `whetstone eval` invocation.