store

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 2, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package store owns the .whetstone/ workspace: SQLite index, completion cache, and the on-disk artifact layout. Uses modernc.org/sqlite (pure Go) because the release build is CGO_ENABLED=0.

Index

Constants

View Source
const (
	StatusClaimed = "claimed"
	StatusOK      = "ok"
	StatusFailed  = "failed"
	StatusError   = "error"
	StatusTimeout = "timeout"
)

Run statuses. ClaimRun treats StatusOK and StatusFailed as done: a run that executed and whose verifier failed is still a completed measurement — a failed measurement is a measurement — and re-running it would waste a session without changing the answer. StatusError and StatusTimeout mean the run could not be performed at all and are retried, and a claimed-but-unfinished run (StatusClaimed) always is, because a process killed mid-run must not silently drop a session from a denominator.

Variables

This section is empty.

Functions

This section is empty.

Types

type Condition

type Condition string

Condition identifies which side of a comparison a run belongs to: the skill condition, the baseline condition, or a trigger probe. It is the shared representation store.RunKey and report.ConditionReport both use, so the concept has one type across the eval engine rather than a bare string re-encoded at each consumer.

This is a different vocabulary from a judge's verdict (which candidate won a pairwise comparison — "skill"/"baseline"/"tie") even though the two happen to share two words: a Condition names which run produced a measurement, a verdict names which measurement came out ahead.

type EvalRecord

type EvalRecord struct {
	ID            int64
	Skill         string
	SpecVersion   int
	Tier          string
	SuiteRef      string
	EngineVersion string
	ModelPanel    []byte
	Seed          int64
	StartedAt     time.Time
	FinishedAt    time.Time
	Status        string
}

EvalRecord is one evaluation run: the skill, spec version, and suite it was measured against, plus its lifecycle status.

type ReportMeta

type ReportMeta struct {
	Headline      float64
	PanelComplete bool
	// RobustnessGap is nil when it was not computed for this report (the
	// strong/floor comparison was not well defined), mirroring
	// report.Report.RobustnessGap. A bare float64 here would collapse "not
	// computed" and "computed at zero" to the same stored value.
	RobustnessGap *float64
}

ReportMeta is the small set of report fields the store indexes directly, alongside the opaque report document itself.

type RunGrade added in v0.11.0

type RunGrade struct {
	Key    RunKey
	Passed int
	Total  int
	Doc    []byte
}

RunGrade is one session's graded expectations, as stored. Doc is the per-expectation detail, opaque here so this package never imports score.

type RunKey

type RunKey struct {
	TaskID    string
	Agent     string
	Model     string
	Condition Condition
	Attempt   int
}

RunKey identifies one session within an eval: a task, an agent/model pair, the skill/baseline condition, and an attempt number. The UNIQUE constraint on runs is keyed on exactly these fields.

type RunOutcome

type RunOutcome struct {
	InputTokens  int64
	OutputTokens int64
	DurationMS   int64
	AgentVersion string
	RateLimited  bool
	Status       string
	Error        string
	ArtifactDir  string
}

RunOutcome is what a finished (or failed-to-run) session reported.

type RunRecord

type RunRecord struct {
	ID      int64
	Key     RunKey
	Outcome RunOutcome
}

RunRecord is one stored run.

type ScoreRow

type ScoreRow struct {
	Agent         string
	Model         string
	Effectiveness float64
	// Healthy is false when this member's adapter failed its probe. Every
	// other field on such a row is a zero value that must not be read as a
	// real measurement.
	Healthy bool
}

ScoreRow is one agent/model pair's computed score for an eval.

type SpecRecord

type SpecRecord struct {
	Version   int
	Approved  bool
	CreatedAt time.Time
}

SpecRecord is one entry in a skill's spec history.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is an open .whetstone/ workspace.

func Open

func Open(root string) (*Store, error)

Open creates or opens the workspace under root and applies pending migrations.

func (*Store) ApproveSpec

func (s *Store) ApproveSpec(skill string, version int) error

ApproveSpec marks one version approved. Approval is per version, so editing a spec requires re-approval — without that the gate approves a spec once and then rubber-stamps everything after it.

func (*Store) Cache

func (s *Store) Cache() llm.Cache

Cache returns the completion cache.

func (*Store) ClaimRun

func (s *Store) ClaimRun(evalID int64, k RunKey) (int64, bool, error)

ClaimRun records the intent to execute one run and reports whether it has already finished. Resume is built entirely on this: a finished key is skipped and a claimed-but-unfinished key is handed back for another attempt, because a process killed mid-run must not silently drop a session from a denominator.

func (*Store) Close

func (s *Store) Close() error

Close releases the database handle.

func (*Store) CopyBaseline added in v0.12.0

func (s *Store) CopyBaseline(evalID int64, prior *RunRecord, k RunKey) error

CopyBaseline writes a prior baseline run and its grade into evalID under key k, so the eval scores it without spending a session on it.

func (*Store) CreateEval

func (s *Store) CreateEval(e EvalRecord) (int64, error)

CreateEval starts a new eval record and returns its id.

func (*Store) Eval

func (s *Store) Eval(id int64) (*EvalRecord, error)

Eval returns one eval by id.

func (*Store) EvalDir

func (s *Store) EvalDir(skill string) (string, error)

EvalDir is the eval sidecar directory.

func (*Store) FinishEval

func (s *Store) FinishEval(id int64, status string) error

FinishEval marks an eval finished with a terminal status and stamps finished_at.

func (*Store) FinishRun

func (s *Store) FinishRun(id int64, o RunOutcome) error

FinishRun records a run's outcome. o.Status is expected to be one of StatusOK, StatusFailed, StatusError, or StatusTimeout: the first two make a later ClaimRun of the same key report done, the latter two make it retry.

func (*Store) GeneratedRef added in v0.11.0

func (s *Store) GeneratedRef(skill string) (string, error)

GeneratedRef returns the ref recorded by RecordGeneratedRef, or the empty string when there is none. An absent row is not an error. A suite written before this table existed has no record. The caller must treat that as possibly reviewed rather than as certainly unreviewed.

func (*Store) Grades added in v0.11.0

func (s *Store) Grades(evalID int64) ([]RunGrade, error)

Grades lists every stored grade for an eval, the read side of SaveGrade.

func (*Store) LatestReport

func (s *Store) LatestReport(skill string) ([]byte, int64, error)

LatestReport returns the report document for a skill's most recent eval, and that eval's id.

func (*Store) LoadSpec

func (s *Store) LoadSpec(skill string) (*spec.SkillSpec, int, error)

LoadSpec returns the latest stored spec version for a skill.

func (*Store) RecordGeneratedRef added in v0.11.0

func (s *Store) RecordGeneratedRef(skill, ref string) error

RecordGeneratedRef records the content ref of the eval set the generator last wrote for a skill. `suite push` compares the current ref against it: an equal ref means nobody read the file.

func (*Store) Report

func (s *Store) Report(evalID int64) ([]byte, error)

Report returns the stored report document for one eval.

func (*Store) ReportMeta

func (s *Store) ReportMeta(evalID int64) (ReportMeta, error)

ReportMeta returns the small set of report fields the store indexes directly for evalID, the read side of SaveReport's ReportMeta write. RobustnessGap round-trips as nil when it was stored as SQL NULL (not computed), never silently as 0 — mirroring the write-side comment on ReportMeta itself.

func (*Store) ReusableBaseline added in v0.12.0

func (s *Store) ReusableBaseline(suiteRef, agentVersion string, k RunKey) (*RunRecord, bool, error)

ReusableBaseline finds a finished baseline session from an earlier eval that the caller may reuse instead of spending a session on it.

A baseline installs no skill, so it depends only on the suite, the task, the agent, the model, and the agent's version. Matching all five, plus a status of ok and an age inside baselineReuseWindow, is what makes the prior measurement the same measurement.

func (*Store) Root

func (s *Store) Root() string

Root is the .whetstone directory.

func (*Store) RunDir

func (s *Store) RunDir(skill string, evalID int64, k RunKey) (string, error)

RunDir is the artifact directory for one run, inside the skill's eval sidecar: <evalDir>/runs/<evalID>/<taskID>/<agent>__<model>__<condition>__<attempt>/. skill is validated the same way SkillDir validates it; every RunKey field is sanitised before joining, since each one traces back to untrusted input — and sanitizePathComponent guarantees none of them can contain the "__" used to join them, so the leaf splits back into its four fields unambiguously.

func (*Store) Runs

func (s *Store) Runs(evalID int64) ([]RunRecord, error)

Runs lists every run recorded for an eval, in claim order.

func (*Store) SaveGrade added in v0.11.0

func (s *Store) SaveGrade(evalID int64, g RunGrade) error

SaveGrade upserts one session's grade. Upsert rather than insert: a re-graded run recomputes the same key, and a unique-constraint failure there would abort the eval.

func (*Store) SaveReport

func (s *Store) SaveReport(evalID int64, doc []byte, m ReportMeta) error

SaveReport stores a report as opaque JSON bytes, alongside the few fields the store indexes directly. Reports are stored opaque rather than typed so that this package never imports report — report imports score and drift, and importing report back here would create a cycle.

func (*Store) SaveScore

func (s *Store) SaveScore(evalID int64, sc ScoreRow) error

SaveScore upserts one agent/model pair's scores for an eval. Upsert rather than insert: a re-scored eval recomputes the same (eval_id, agent, model) key, and a unique-constraint failure there would abort the run.

func (*Store) SaveSpec

func (s *Store) SaveSpec(sp *spec.SkillSpec) (int, error)

SaveSpec appends a new spec version and rewrites the human-editable YAML. Both, deliberately: the database gives history and approval state, the file gives `whetstone spec edit` something to open.

The database row commits before the file is written, so a file-write failure loses no data — LoadSpec still returns the new version from the row alone. Because of that ordering, SaveSpec still returns the version it committed alongside a non-nil error from the file-write step: recomputing MAX(version)+1 on a naive retry would skip the already-committed version and leave a permanent gap. A caller that sees an error here should treat the returned version as already durable rather than retrying from scratch.

func (*Store) Scores

func (s *Store) Scores(evalID int64) ([]ScoreRow, error)

Scores lists every agent/model pair's stored scores for an eval, the read side of SaveScore.

func (*Store) SkillDir

func (s *Store) SkillDir(skill string) (string, error)

SkillDir is the bundle root for a skill. It returns an error rather than a path when skill is not a legal spec name — see skillDirName.

func (*Store) SpecHistory

func (s *Store) SpecHistory(skill string) ([]SpecRecord, error)

SpecHistory lists every stored version, newest first.

func (*Store) SpecPath

func (s *Store) SpecPath(skill string) (string, error)

SpecPath is the spec YAML, beside the bundle rather than in the sidecar.

func (*Store) SuiteDir

func (s *Store) SuiteDir(skill string) (string, error)

SuiteDir holds generated task packages and the trigger set.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL