Documentation
¶
Overview ¶
Package store owns the .whetstone/ workspace: SQLite index, completion cache, and the on-disk artifact layout. Uses modernc.org/sqlite (pure Go) because the release build is CGO_ENABLED=0.
Index ¶
- Constants
- type Condition
- type EvalRecord
- type ReportMeta
- type RunGrade
- type RunKey
- type RunOutcome
- type RunRecord
- type ScoreRow
- type SpecRecord
- type Store
- func (s *Store) ApproveSpec(skill string, version int) error
- func (s *Store) Cache() llm.Cache
- func (s *Store) ClaimRun(evalID int64, k RunKey) (int64, bool, error)
- func (s *Store) Close() error
- func (s *Store) CreateEval(e EvalRecord) (int64, error)
- func (s *Store) Eval(id int64) (*EvalRecord, error)
- func (s *Store) EvalDir(skill string) (string, error)
- func (s *Store) FinishEval(id int64, status string) error
- func (s *Store) FinishRun(id int64, o RunOutcome) error
- func (s *Store) GeneratedRef(skill string) (string, error)
- func (s *Store) Grades(evalID int64) ([]RunGrade, error)
- func (s *Store) LatestEval(skill string) (*EvalRecord, error)
- func (s *Store) LatestReport(skill string) ([]byte, int64, error)
- func (s *Store) LoadSpec(skill string) (*spec.SkillSpec, int, error)
- func (s *Store) RecordGeneratedRef(skill, ref string) error
- func (s *Store) Report(evalID int64) ([]byte, error)
- func (s *Store) ReportMeta(evalID int64) (ReportMeta, error)
- func (s *Store) Root() string
- func (s *Store) RunDir(skill string, evalID int64, k RunKey) (string, error)
- func (s *Store) Runs(evalID int64) ([]RunRecord, error)
- func (s *Store) SaveGrade(evalID int64, g RunGrade) error
- func (s *Store) SaveReport(evalID int64, doc []byte, m ReportMeta) error
- func (s *Store) SaveScore(evalID int64, sc ScoreRow) error
- func (s *Store) SaveSpec(sp *spec.SkillSpec) (int, error)
- func (s *Store) SaveSuiteCheck(skill, suiteRef string, rows []SuiteCheckRow) error
- func (s *Store) Scores(evalID int64) ([]ScoreRow, error)
- func (s *Store) SkillDir(skill string) (string, error)
- func (s *Store) SpecHistory(skill string) ([]SpecRecord, error)
- func (s *Store) SpecPath(skill string) (string, error)
- func (s *Store) SuiteChecks(skill, suiteRef string) ([]SuiteCheckRow, error)
- func (s *Store) SuiteDir(skill string) (string, error)
- type SuiteCheckRow
Constants ¶
const ( StatusClaimed = "claimed" StatusOK = "ok" StatusFailed = "failed" StatusError = "error" StatusTimeout = "timeout" )
Run statuses. ClaimRun treats StatusOK and StatusFailed as done: a run that executed and whose verifier failed is still a completed measurement — a failed measurement is a measurement — and re-running it would waste a session without changing the answer. StatusError and StatusTimeout mean the run could not be performed at all and are retried, and a claimed-but-unfinished run (StatusClaimed) always is, because a process killed mid-run must not silently drop a session from a denominator.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Condition ¶
type Condition string
Condition identifies which side of a comparison a run belongs to: the skill condition, the baseline condition, or a trigger probe. It is the shared representation store.RunKey and report.ConditionReport both use, so the concept has one type across the eval engine rather than a bare string re-encoded at each consumer.
This is a different vocabulary from a judge's verdict (which candidate won a pairwise comparison — "skill"/"baseline"/"tie") even though the two happen to share two words: a Condition names which run produced a measurement, a verdict names which measurement came out ahead.
type EvalRecord ¶
type EvalRecord struct {
ID int64
Skill string
SpecVersion int
Tier string
SuiteRef string
EngineVersion string
ModelPanel []byte
Seed int64
StartedAt time.Time
FinishedAt time.Time
Status string
}
EvalRecord is one evaluation run: the skill, spec version, and suite it was measured against, plus its lifecycle status.
type ReportMeta ¶
type ReportMeta struct {
Headline float64
PanelComplete bool
// RobustnessGap is nil when it was not computed for this report (the
// strong/floor comparison was not well defined), mirroring
// report.Report.RobustnessGap. A bare float64 here would collapse "not
// computed" and "computed at zero" to the same stored value.
RobustnessGap *float64
}
ReportMeta is the small set of report fields the store indexes directly, alongside the opaque report document itself.
type RunGrade ¶ added in v0.11.0
RunGrade is one session's graded expectations, as stored. Doc is the per-expectation detail, opaque here so this package never imports score.
type RunKey ¶
RunKey identifies one session within an eval: a task, an agent/model pair, the skill/baseline condition, and an attempt number. The UNIQUE constraint on runs is keyed on exactly these fields.
type RunOutcome ¶
type RunOutcome struct {
InputTokens int64
OutputTokens int64
DurationMS int64
AgentVersion string
RateLimited bool
Status string
Error string
ArtifactDir string
}
RunOutcome is what a finished (or failed-to-run) session reported.
type RunRecord ¶
type RunRecord struct {
ID int64
Key RunKey
Outcome RunOutcome
}
RunRecord is one stored run.
type ScoreRow ¶
type ScoreRow struct {
Agent string
Model string
Effectiveness float64
// Healthy is false when this member's adapter failed its probe. Every
// other field on such a row is a zero value that must not be read as a
// real measurement.
Healthy bool
}
ScoreRow is one agent/model pair's computed score for an eval.
type SpecRecord ¶
SpecRecord is one entry in a skill's spec history.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is an open .whetstone/ workspace.
func (*Store) ApproveSpec ¶
ApproveSpec marks one version approved. Approval is per version, so editing a spec requires re-approval — without that the gate approves a spec once and then rubber-stamps everything after it.
func (*Store) ClaimRun ¶
ClaimRun records the intent to execute one run and reports whether it has already finished. Resume is built entirely on this: a finished key is skipped and a claimed-but-unfinished key is handed back for another attempt, because a process killed mid-run must not silently drop a session from a denominator.
func (*Store) CreateEval ¶
func (s *Store) CreateEval(e EvalRecord) (int64, error)
CreateEval starts a new eval record and returns its id.
func (*Store) Eval ¶
func (s *Store) Eval(id int64) (*EvalRecord, error)
Eval returns one eval by id.
func (*Store) FinishEval ¶
FinishEval marks an eval finished with a terminal status and stamps finished_at.
func (*Store) FinishRun ¶
func (s *Store) FinishRun(id int64, o RunOutcome) error
FinishRun records a run's outcome. o.Status is expected to be one of StatusOK, StatusFailed, StatusError, or StatusTimeout: the first two make a later ClaimRun of the same key report done, the latter two make it retry.
func (*Store) GeneratedRef ¶ added in v0.11.0
GeneratedRef returns the ref recorded by RecordGeneratedRef, or the empty string when there is none. An absent row is not an error. A suite written before this table existed has no record. The caller must treat that as possibly reviewed rather than as certainly unreviewed.
func (*Store) Grades ¶ added in v0.11.0
Grades lists every stored grade for an eval, the read side of SaveGrade.
func (*Store) LatestEval ¶
func (s *Store) LatestEval(skill string) (*EvalRecord, error)
LatestEval returns the most recently created eval for a skill.
func (*Store) LatestReport ¶
LatestReport returns the report document for a skill's most recent eval, and that eval's id.
func (*Store) RecordGeneratedRef ¶ added in v0.11.0
RecordGeneratedRef records the content ref of the eval set the generator last wrote for a skill. `suite push` compares the current ref against it: an equal ref means nobody read the file.
func (*Store) ReportMeta ¶
func (s *Store) ReportMeta(evalID int64) (ReportMeta, error)
ReportMeta returns the small set of report fields the store indexes directly for evalID, the read side of SaveReport's ReportMeta write. RobustnessGap round-trips as nil when it was stored as SQL NULL (not computed), never silently as 0 — mirroring the write-side comment on ReportMeta itself.
func (*Store) RunDir ¶
RunDir is the artifact directory for one run, inside the skill's eval sidecar: <evalDir>/runs/<evalID>/<taskID>/<agent>__<model>__<condition>__<attempt>/. skill is validated the same way SkillDir validates it; every RunKey field is sanitised before joining, since each one traces back to untrusted input — and sanitizePathComponent guarantees none of them can contain the "__" used to join them, so the leaf splits back into its four fields unambiguously.
func (*Store) SaveGrade ¶ added in v0.11.0
SaveGrade upserts one session's grade. Upsert rather than insert: a re-graded run recomputes the same key, and a unique-constraint failure there would abort the eval.
func (*Store) SaveReport ¶
func (s *Store) SaveReport(evalID int64, doc []byte, m ReportMeta) error
SaveReport stores a report as opaque JSON bytes, alongside the few fields the store indexes directly. Reports are stored opaque rather than typed so that this package never imports report — report imports score and drift, and importing report back here would create a cycle.
func (*Store) SaveScore ¶
SaveScore upserts one agent/model pair's scores for an eval. Upsert rather than insert: a re-scored eval recomputes the same (eval_id, agent, model) key, and a unique-constraint failure there would abort the run.
func (*Store) SaveSpec ¶
SaveSpec appends a new spec version and rewrites the human-editable YAML. Both, deliberately: the database gives history and approval state, the file gives `whetstone spec edit` something to open.
The database row commits before the file is written, so a file-write failure loses no data — LoadSpec still returns the new version from the row alone. Because of that ordering, SaveSpec still returns the version it committed alongside a non-nil error from the file-write step: recomputing MAX(version)+1 on a naive retry would skip the already-committed version and leave a permanent gap. A caller that sees an error here should treat the returned version as already durable rather than retrying from scratch.
func (*Store) SaveSuiteCheck ¶
func (s *Store) SaveSuiteCheck(skill, suiteRef string, rows []SuiteCheckRow) error
SaveSuiteCheck upserts, per task, whether it was found void the last time the suite was checked.
func (*Store) Scores ¶
Scores lists every agent/model pair's stored scores for an eval, the read side of SaveScore.
func (*Store) SkillDir ¶
SkillDir is the bundle root for a skill. It returns an error rather than a path when skill is not a legal spec name — see skillDirName.
func (*Store) SpecHistory ¶
func (s *Store) SpecHistory(skill string) ([]SpecRecord, error)
SpecHistory lists every stored version, newest first.
func (*Store) SuiteChecks ¶
func (s *Store) SuiteChecks(skill, suiteRef string) ([]SuiteCheckRow, error)
SuiteChecks lists the last-known void status of every checked task in a suite.
type SuiteCheckRow ¶
SuiteCheckRow records whether one task in a suite was found void (its oracle or verifier no longer applies) when the suite was last checked.