Documentation
¶
Overview ¶
Package whetstone implements the whetstone CLI: the authoring front end for the skill specification, generation, lint, contract, and suite engine under internal/eval, plus the evaluation commands (eval, drift, repair, report) built on top of it.
Index ¶
- Constants
- func Execute()
- func RenderEvalSummary(rep *report.Report, evalID int64, skill string) string
- func RunEval(ctx context.Context, req EvalRequest) error
- func RunEvalWith(ctx context.Context, d EvalDeps, req EvalRequest) (*report.Report, error)
- func RunGen(ctx context.Context, skill string) error
- func RunInit(root string) (string, error)
- func RunLint(path string, strict bool) (int, error)
- func RunNew(ctx context.Context, intent string) error
- func RunPack(bundleDir, outPath string) (string, error)
- func RunPropose(ctx context.Context, req ProposeRequest) error
- func RunProposeWith(ctx context.Context, st *store.Store, g llm.Gateway, req ProposeRequest) (*propose.Result, error)
- func RunReport(skill, ref string, open bool, opener func(path string) error) (string, error)
- func RunSpecApprove(skill string) error
- func RunSpecEdit(skill string) error
- func RunSpecShow(skill string) error
- func RunSuiteCheck(skill string, allowVoid bool) error
- func RunSuiteGen(ctx context.Context, skill string) error
- func RunSuitePush(ctx context.Context, req SuitePushRequest) error
- func RunTune(ctx context.Context, req TuneRequest) error
- func RunTuneWith(ctx context.Context, st *store.Store, g llm.Gateway, req TuneRequest) (*tune.Result, error)
- func SetVersion(version, commit, date string)
- type DoctorReport
- type EvalDeps
- type EvalRequest
- type Progress
- type ProposeRequest
- type SuitePushRequest
- type TuneRequest
Constants ¶
const ( PhasePreparing = "preparing" PhaseHealth = "health check" PhaseSessions = "sessions" PhaseGrading = "grading" )
Phases of an evaluation, in order.
Variables ¶
This section is empty.
Functions ¶
func Execute ¶
func Execute()
Execute runs the root command. A command that runs sandboxed sessions (eval, suite check) passes cmd.Context() into docker.Driver.Run and suite.Check, both of which already clean up their containers and networks on context cancellation — but only if something actually cancels that context on SIGINT/SIGTERM. Without signal.NotifyContext here, Ctrl-C is process death: the docker client is killed with no chance to run its own cleanup, and every in-flight run leaks a whetstone-net-* network and a whetstone-proxy-* container.
func RenderEvalSummary ¶ added in v0.11.0
RenderEvalSummary is the whole terminal result of an eval: the score, what it is made of, and every eval whose expectations did not all pass. Exported for its test — nothing outside this package calls it.
It leads with the failures because that is what the reader can act on.
func RunEval ¶
func RunEval(ctx context.Context, req EvalRequest) error
RunEval resolves EvalDeps from the environment — the workspace store, the configured sandbox driver, the LLM gateway `whetstone doctor` would choose, and the linked-in adapter registry — and runs the eval.
func RunEvalWith ¶
RunEvalWith runs one evaluation end to end: load the approved spec and eval set, refuse an eval against void evals, plan and execute the panel, grade every session as it finishes, and persist and write the report.
func RunGen ¶
RunGen regenerates the bundle for a skill from its approved spec and lints the result.
func RunInit ¶
RunInit creates or opens the workspace under root, defaulting to the working directory when root is empty, and returns the workspace directory.
func RunLint ¶
RunLint lints the bundle at path and returns the process exit code.
The error return is reserved for lint being unable to run at all (an unreadable bundle); findings are reported through the exit code, so a failing bundle is not also an error the caller has to special-case.
func RunNew ¶
RunNew runs the full authoring pipeline for a new skill against the workspace and gateway this machine is configured for.
func RunPack ¶
RunPack lints bundleDir and, if it has no errors, writes a tar.gz of it to outPath with the eval sidecar and the spec removed, and returns the archive's resolved path. When outPath is empty, the archive is written beside the bundle — in its parent directory, as "<bundle>.tar.gz" — rather than inside it: an archive inside the directory being packed is content the next pack has to exclude (lint.Excluded's root-tarball case) purely because it exists.
Linting first is not a convenience: an archive built from a bundle that fails lint installs and then fails at use time, a long way from the cause.
func RunPropose ¶ added in v0.14.0
func RunPropose(ctx context.Context, req ProposeRequest) error
RunPropose opens the workspace and gateway, then proposes.
func RunProposeWith ¶ added in v0.14.0
func RunProposeWith(ctx context.Context, st *store.Store, g llm.Gateway, req ProposeRequest) (*propose.Result, error)
RunProposeWith reads what a candidate missed and writes one change.
func RunReport ¶
RunReport renders the HTML report for skill's ref ("latest" or an eval id) to <sidecar>/reports/<eval id>/report.html and returns that path. When open is true it hands the path to opener; opener defaults to defaultOpener (nil is not a caller error — the command above always passes it explicitly, but a test that only cares about the render path can pass nil too as long as open is false).
opener is injected rather than this function shelling out to a browser itself: a test that launches a browser is a test nobody runs twice.
func RunSpecApprove ¶
RunSpecApprove marks the latest stored version of a spec approved.
func RunSpecEdit ¶
RunSpecEdit opens the spec YAML in $EDITOR. It stores the edited result as a new, approved version. Approval is per version, so a prior version's approval never carries forward to one nobody has reviewed.
func RunSpecShow ¶
RunSpecShow prints the latest stored spec for a skill.
func RunSuiteCheck ¶
RunSuiteCheck validates a skill's eval set and reports the result.
It replaces an oracle gate that ran every task's reference solution in a container. With no verifier script left to prove correct, what remains is static: an eval with nothing to grade, or one naming an input file the set does not carry, cannot produce a measurement. Nothing is persisted: an eval run repeats the same pure check for itself, so a stored row would only be an older copy of an answer it already has.
A void eval makes this exit non-zero unless allowVoid is set, which is what makes the command usable as a CI gate.
func RunSuiteGen ¶
RunSuiteGen drafts an eval set from a skill's approved spec and writes it into the skill's eval sidecar.
func RunSuitePush ¶
func RunSuitePush(ctx context.Context, req SuitePushRequest) error
RunSuitePush uploads a skill's written suite to a Skael server, together with its own validation result as an audit trail. The check runs here rather than being read back from a stored row: suite.Validate is pure, so a push cannot go out with a stale answer and every later reader repeats it.
func RunTune ¶ added in v0.11.0
func RunTune(ctx context.Context, req TuneRequest) error
RunTune resolves the store and the gateway from the environment. It calls RunTuneWith with them.
func RunTuneWith ¶ added in v0.11.0
func RunTuneWith(ctx context.Context, st *store.Store, g llm.Gateway, req TuneRequest) (*tune.Result, error)
RunTuneWith tunes a skill's description against its trigger set.
func SetVersion ¶
func SetVersion(version, commit, date string)
SetVersion is called by main to inject build-time version info.
Types ¶
type DoctorReport ¶
type DoctorReport struct {
AgentCLI string `json:"agent_cli"`
AgentCLIVersion string `json:"agent_cli_version,omitempty"`
AgentCLIError string `json:"agent_cli_error,omitempty"`
Gateway string `json:"gateway"`
GatewayDetail string `json:"gateway_detail"`
Models []string `json:"models,omitempty"`
Warnings []string `json:"warnings,omitempty"`
LLMTimeout string `json:"llm_timeout"`
Docker bool `json:"docker"`
DockerPath string `json:"docker_path,omitempty"`
Adapters []string `json:"adapters"`
}
DoctorReport is the environment health check.
type EvalDeps ¶
type EvalDeps struct {
Store *store.Store
Driver sandbox.Driver
// Gateway grades every expectation. RunEvalWith refuses a nil one: the
// score is an expectation pass rate, so there is no measurement without a
// judge.
Gateway llm.Gateway
Adapters func(name string) (agent.Adapter, bool)
// Now defaults to time.Now.
Now func() time.Time
// Sleep defaults to time.Sleep; the runner's rate-limit backoff uses it.
Sleep func(time.Duration)
EngineVersion string
// WorkspaceRoot is passed through to runner.Options.WorkspaceRoot — see
// there for why a containerized runner has to set it. Empty is correct
// for the interactive CLI, which always shares a filesystem with the
// daemon it starts sandboxes on.
WorkspaceRoot string
// PanelModels overrides the shipped panel's model ids, most capable
// first. Already-resolved values rather than env lookups, so RunEvalWith
// carries no policy — see provider.Config.PanelModels.
PanelModels []string
// PanelBaseURL is carried for diagnostics only, so an all-unhealthy panel
// can name the endpoint that rejected its models.
PanelBaseURL string
// MemberAuth gives a member its own credentials; see
// runner.Options.MemberAuth. MemberProvider labels the member's backend
// in the report. Both are nil outside a server-configured worker.
MemberAuth func(agent, model string) ([]string, bool)
MemberProvider func(agent, model string) string
// Progress, when set, receives each phase change, session start and end,
// and grade, as it happens. The worker posts them as job events.
Progress func(Progress)
}
EvalDeps is everything RunEvalWith needs, injected so a test can supply a fake driver, adapter registry, and gateway with no Docker, no network, and no LLM account. RunEval (below) resolves these from the environment the same way `whetstone doctor` does.
type EvalRequest ¶
type EvalRequest struct {
Skill string
Tier runner.Tier
Agents []string
Models []string
Concurrency int
// GradeConcurrency bounds the judge calls, independently of the sandbox
// sessions --concurrency bounds.
GradeConcurrency int
Untrusted bool
AllowVoid bool
// Resume, when non-zero, reuses that eval id rather than starting a new
// one — see RunEvalWith for the suite/panel mismatch guard.
Resume int64
// FreshBaseline runs every baseline session rather than reusing a
// matching one from an earlier eval.
FreshBaseline bool
// Contest plans one contest candidate: Attempts per task with the skill,
// and no baseline or probe, because a verdict reads neither.
Contest bool
Attempts int
}
EvalRequest is one `whetstone eval` invocation.
type Progress ¶ added in v0.20.0
type Progress struct {
Kind string // phase, health, plan, session_start, session_end, grade
Phase string
Total int
Task string
Condition string
Model string
Attempt int
Status string
OK bool
Duration time.Duration
InputTok int64
OutputTok int64
CostUSD float64
Error string
}
Progress is one step of a running evaluation.
type ProposeRequest ¶ added in v0.14.0
type ProposeRequest struct {
ContestID string
// Candidate names whose losses to work from, as skill@version. Empty
// resolves to the only candidate whose skill is in the working tree.
Candidate string
Endpoint string
APIKey string
// Apply writes the change. False prints the diff and writes nothing.
Apply bool
}
ProposeRequest is one `whetstone propose` invocation.
type SuitePushRequest ¶
SuitePushRequest holds the inputs to RunSuitePush.
type TuneRequest ¶ added in v0.11.0
type TuneRequest struct {
Skill string
Queries int
Runs int
Iterations int
Concurrency int
Threshold float64
Holdout float64
// Apply writes the winner to the spec and to SKILL.md. A false value
// still measures the description. It still reports what it found. It
// changes nothing.
Apply bool
}
TuneRequest is one `whetstone tune` invocation.