whetstone

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2026 License: Apache-2.0 Imports: 43 Imported by: 0

Documentation

Overview

Package whetstone implements the whetstone CLI: the authoring front end for the skill specification, generation, lint, contract, and suite engine under internal/eval, plus the evaluation commands (eval, drift, repair, report) built on top of it.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Execute

func Execute()

Execute runs the root command. A command that runs sandboxed sessions (eval, suite check) passes cmd.Context() into docker.Driver.Run and suite.Check, both of which already clean up their containers and networks on context cancellation — but only if something actually cancels that context on SIGINT/SIGTERM. Without signal.NotifyContext here, Ctrl-C is process death: the docker client is killed with no chance to run its own cleanup, and every in-flight run leaks a whetstone-net-* network and a whetstone-proxy-* container.

func RenderEvalSummary added in v0.11.0

func RenderEvalSummary(rep *report.Report, evalID int64, skill string) string

RenderEvalSummary is the whole terminal result of an eval: the score, what it is made of, and every eval whose expectations did not all pass. Exported for its test — nothing outside this package calls it.

It leads with the failures because that is what the reader can act on.

func RunEval

func RunEval(ctx context.Context, req EvalRequest) error

RunEval resolves EvalDeps from the environment — the workspace store, the configured sandbox driver, the LLM gateway `whetstone doctor` would choose, and the linked-in adapter registry — and runs the eval.

func RunEvalWith

func RunEvalWith(ctx context.Context, d EvalDeps, req EvalRequest) (*report.Report, error)

RunEvalWith runs one evaluation end to end: load the approved spec and eval set, refuse an eval against void evals, plan and execute the panel, grade every session as it finishes, and persist and write the report.

func RunGen

func RunGen(ctx context.Context, skill string) error

RunGen regenerates the bundle for a skill from its approved spec and lints the result.

func RunInit

func RunInit(root string) (string, error)

RunInit creates or opens the workspace under root, defaulting to the working directory when root is empty, and returns the workspace directory.

func RunLint

func RunLint(path string, strict bool) (int, error)

RunLint lints the bundle at path and returns the process exit code.

The error return is reserved for lint being unable to run at all (an unreadable bundle); findings are reported through the exit code, so a failing bundle is not also an error the caller has to special-case.

func RunNew

func RunNew(ctx context.Context, intent string) error

RunNew runs the full authoring pipeline for a new skill against the workspace and gateway this machine is configured for.

func RunPack

func RunPack(bundleDir, outPath string) (string, error)

RunPack lints bundleDir and, if it has no errors, writes a tar.gz of it to outPath with the eval sidecar and the spec removed, and returns the archive's resolved path. When outPath is empty, the archive is written beside the bundle — in its parent directory, as "<bundle>.tar.gz" — rather than inside it: an archive inside the directory being packed is content the next pack has to exclude (lint.Excluded's root-tarball case) purely because it exists.

Linting first is not a convenience: an archive built from a bundle that fails lint installs and then fails at use time, a long way from the cause.

func RunReport

func RunReport(skill, ref string, open bool, opener func(path string) error) (string, error)

RunReport renders the HTML report for skill's ref ("latest" or an eval id) to <sidecar>/reports/<eval id>/report.html and returns that path. When open is true it hands the path to opener; opener defaults to defaultOpener (nil is not a caller error — the command above always passes it explicitly, but a test that only cares about the render path can pass nil too as long as open is false).

opener is injected rather than this function shelling out to a browser itself: a test that launches a browser is a test nobody runs twice.

func RunSpecApprove

func RunSpecApprove(skill string) error

RunSpecApprove marks the latest stored version of a spec approved.

func RunSpecEdit

func RunSpecEdit(skill string) error

RunSpecEdit opens the spec YAML in $EDITOR. It stores the edited result as a new, approved version. Approval is per version, so a prior version's approval never carries forward to one nobody has reviewed.

func RunSpecShow

func RunSpecShow(skill string) error

RunSpecShow prints the latest stored spec for a skill.

func RunSuiteCheck

func RunSuiteCheck(skill string, allowVoid bool) error

RunSuiteCheck validates a skill's eval set and reports the result.

It replaces an oracle gate that ran every task's reference solution in a container. With no verifier script left to prove correct, what remains is static: an eval with nothing to grade, or one naming an input file the set does not carry, cannot produce a measurement. Nothing is persisted: an eval run repeats the same pure check for itself, so a stored row would only be an older copy of an answer it already has.

A void eval makes this exit non-zero unless allowVoid is set, which is what makes the command usable as a CI gate.

func RunSuiteGen

func RunSuiteGen(ctx context.Context, skill string) error

RunSuiteGen drafts an eval set from a skill's approved spec and writes it into the skill's eval sidecar.

func RunSuitePush

func RunSuitePush(ctx context.Context, req SuitePushRequest) error

RunSuitePush uploads a skill's written suite to a Skael server, together with its own validation result as an audit trail. The check runs here rather than being read back from a stored row: suite.Validate is pure, so a push cannot go out with a stale answer and every later reader repeats it.

func RunTune added in v0.11.0

func RunTune(ctx context.Context, req TuneRequest) error

RunTune resolves the store and the gateway from the environment. It calls RunTuneWith with them.

func RunTuneWith added in v0.11.0

func RunTuneWith(ctx context.Context, st *store.Store, g llm.Gateway, req TuneRequest) (*tune.Result, error)

RunTuneWith tunes a skill's description against its trigger set.

func SetVersion

func SetVersion(version, commit, date string)

SetVersion is called by main to inject build-time version info.

Types

type DoctorReport

type DoctorReport struct {
	AgentCLI        string   `json:"agent_cli"`
	AgentCLIVersion string   `json:"agent_cli_version,omitempty"`
	AgentCLIError   string   `json:"agent_cli_error,omitempty"`
	Gateway         string   `json:"gateway"`
	GatewayDetail   string   `json:"gateway_detail"`
	Models          []string `json:"models,omitempty"`
	Warnings        []string `json:"warnings,omitempty"`
	LLMTimeout      string   `json:"llm_timeout"`
	Docker          bool     `json:"docker"`
	DockerPath      string   `json:"docker_path,omitempty"`
	Adapters        []string `json:"adapters"`
}

DoctorReport is the environment health check.

func RunDoctor

func RunDoctor(ctx context.Context) (*DoctorReport, error)

RunDoctor collects the environment report. Errors within it (missing CLI, missing docker) are folded into the report; only a malformed timeout is returned as an error.

type EvalDeps

type EvalDeps struct {
	Store  *store.Store
	Driver sandbox.Driver
	// Gateway grades every expectation. RunEvalWith refuses a nil one: the
	// score is an expectation pass rate, so there is no measurement without a
	// judge.
	Gateway  llm.Gateway
	Adapters func(name string) (agent.Adapter, bool)
	// Now defaults to time.Now.
	Now func() time.Time
	// Sleep defaults to time.Sleep; the runner's rate-limit backoff uses it.
	Sleep         func(time.Duration)
	EngineVersion string
	// WorkspaceRoot is passed through to runner.Options.WorkspaceRoot — see
	// there for why a containerized runner has to set it. Empty is correct
	// for the interactive CLI, which always shares a filesystem with the
	// daemon it starts sandboxes on.
	WorkspaceRoot string
	// PanelModels overrides the shipped panel's model ids, most capable
	// first. Already-resolved values rather than env lookups, so RunEvalWith
	// carries no policy — see provider.Config.PanelModels.
	PanelModels []string
	// PanelBaseURL is carried for diagnostics only, so an all-unhealthy panel
	// can name the endpoint that rejected its models.
	PanelBaseURL string
	// PanelExcludeEnv names credential variables the panel's sandboxes must
	// not be given — see provider.Config.PanelExcludeEnv and
	// runner.Options.PanelExcludeEnv. Resolved values rather than env
	// lookups, for the same reason PanelModels is.
	PanelExcludeEnv []string
}

EvalDeps is everything RunEvalWith needs, injected so a test can supply a fake driver, adapter registry, and gateway with no Docker, no network, and no LLM subscription. RunEval (below) resolves these from the environment the same way `whetstone doctor` does.

type EvalRequest

type EvalRequest struct {
	Skill       string
	Tier        runner.Tier
	Agents      []string
	Models      []string
	Concurrency int
	// GradeConcurrency bounds the judge calls, independently of the sandbox
	// sessions --concurrency bounds.
	GradeConcurrency int
	Untrusted        bool
	AllowVoid        bool
	// Resume, when non-zero, reuses that eval id rather than starting a new
	// one — see RunEvalWith for the suite/panel mismatch guard.
	Resume int64
	// FreshBaseline runs every baseline session rather than reusing a
	// matching one from an earlier eval.
	FreshBaseline bool
}

EvalRequest is one `whetstone eval` invocation.

type SuitePushRequest

type SuitePushRequest struct {
	Store    *store.Store
	Skill    string
	Endpoint string
	APIKey   string
}

SuitePushRequest holds the inputs to RunSuitePush.

type TuneRequest added in v0.11.0

type TuneRequest struct {
	Skill       string
	Queries     int
	Runs        int
	Iterations  int
	Concurrency int
	Threshold   float64
	Holdout     float64
	// Apply writes the winner to the spec and to SKILL.md. A false value
	// still measures the description. It still reports what it found. It
	// changes nothing.
	Apply bool
}

TuneRequest is one `whetstone tune` invocation.

Directories

Path Synopsis
Package gen writes a skill bundle from an approved spec.
Package gen writes a skill bundle from an approved spec.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL