store

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package store owns the .whetstone/ workspace: a SQLite index for versioned specs and the completion cache, plus the on-disk artifact layout.

The SQLite driver is modernc.org/sqlite (pure Go). The release build is CGO_ENABLED=0 across five platform pairs, so a cgo driver is not an option.

Index

Constants

View Source
const (
	StatusClaimed = "claimed"
	StatusOK      = "ok"
	StatusFailed  = "failed"
	StatusError   = "error"
	StatusTimeout = "timeout"
)

Run statuses. ClaimRun treats StatusOK and StatusFailed as done: a run that executed and whose verifier failed is still a completed measurement — a failed measurement is a measurement — and re-running it would waste a session without changing the answer. StatusError and StatusTimeout mean the run could not be performed at all and are retried, and a claimed-but-unfinished run (StatusClaimed) always is, because a process killed mid-run must not silently drop a session from a denominator.

Variables

This section is empty.

Functions

This section is empty.

Types

type Condition

type Condition string

Condition identifies which side of a comparison a run belongs to: the skill condition, the baseline condition, or a trigger probe. It is the shared representation store.RunKey and report.ConditionReport both use, so the concept has one type across the eval engine rather than a bare string re-encoded at each consumer.

This is a different vocabulary from a judge's verdict (which candidate won a pairwise comparison — "skill"/"baseline"/"tie") even though the two happen to share two words: a Condition names which run produced a measurement, a verdict names which measurement came out ahead.

type EvalRecord

type EvalRecord struct {
	ID            int64
	Skill         string
	SpecVersion   int
	Tier          string
	SuiteRef      string
	EngineVersion string
	ModelPanel    []byte
	Seed          int64
	StartedAt     time.Time
	FinishedAt    time.Time
	Status        string
}

EvalRecord is one evaluation run: the skill, spec version, and suite it was measured against, plus its lifecycle status.

type Judgment

type Judgment struct {
	TaskID   string
	Model    string
	Kind     string
	RuleID   string
	Winner   string
	Margin   float64
	Evidence string
	Votes    int
	Swapped  bool
}

Judgment is one LLM-judge (or rule-based) verdict recorded against an eval.

type ReportMeta

type ReportMeta struct {
	Headline      float64
	PanelComplete bool
	// RobustnessGap is nil when it was not computed for this report (the
	// strong/floor comparison was not well defined), mirroring
	// report.Report.RobustnessGap. A bare float64 here would collapse "not
	// computed" and "computed at zero" to the same stored value.
	RobustnessGap *float64
}

ReportMeta is the small set of report fields the store indexes directly, alongside the opaque report document itself.

type RunKey

type RunKey struct {
	TaskID    string
	Agent     string
	Model     string
	Condition Condition
	Attempt   int
}

RunKey identifies one session within an eval: a task, an agent/model pair, the skill/baseline condition, and an attempt number. The UNIQUE constraint on runs is keyed on exactly these fields.

type RunOutcome

type RunOutcome struct {
	// VerifierExit is nil when the verifier never ran — a trigger probe (which
	// has no verifier at all) or a run that failed before reaching it — and
	// the verifier's exit code otherwise, including 0. A caller that reads
	// *VerifierExit == 0 without checking it is non-nil first is the one bug
	// this distinction exists to prevent: "not measured" and "measured and
	// passed" must never collapse into the same value.
	VerifierExit *int
	InputTokens  int64
	OutputTokens int64
	DurationMS   int64
	AgentVersion string
	RateLimited  bool
	Status       string
	Error        string
	ArtifactDir  string
}

RunOutcome is what a finished (or failed-to-run) session reported.

type RunRecord

type RunRecord struct {
	ID      int64
	Key     RunKey
	Outcome RunOutcome
}

RunRecord is one stored run.

type ScoreRow

type ScoreRow struct {
	Agent         string
	Model         string
	TriggerF1     float64
	Reliability   float64
	Uplift        float64
	Efficiency    float64
	Effectiveness float64
	Adherence     float64
	Drift         float64
	Grade         string
	// Healthy is false when this member's adapter failed its probe, mirroring
	// score.PanelEntry.Healthy. Every other field on such a row is a zero
	// value that must not be read as a real measurement.
	Healthy bool
}

ScoreRow is one agent/model pair's computed scores for an eval.

type SpecRecord

type SpecRecord struct {
	Version   int
	Approved  bool
	CreatedAt time.Time
}

SpecRecord is one entry in a skill's spec history.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is an open .whetstone/ workspace.

func Open

func Open(root string) (*Store, error)

Open creates or opens the workspace under root and applies pending migrations.

func (*Store) ApproveSpec

func (s *Store) ApproveSpec(skill string, version int) error

ApproveSpec marks one version approved. Approval is per version, so editing a spec requires re-approval — without that the gate approves a spec once and then rubber-stamps everything after it.

func (*Store) Cache

func (s *Store) Cache() llm.Cache

Cache returns the completion cache.

func (*Store) ClaimRun

func (s *Store) ClaimRun(evalID int64, k RunKey) (int64, bool, error)

ClaimRun records the intent to execute one run and reports whether it has already finished. Resume is built entirely on this: a finished key is skipped and a claimed-but-unfinished key is handed back for another attempt, because a process killed mid-run must not silently drop a session from a denominator.

func (*Store) Close

func (s *Store) Close() error

Close releases the database handle.

func (*Store) ContractPath

func (s *Store) ContractPath(skill string) (string, error)

ContractPath is the compiled drift contract.

func (*Store) CreateEval

func (s *Store) CreateEval(e EvalRecord) (int64, error)

CreateEval starts a new eval record and returns its id.

func (*Store) Eval

func (s *Store) Eval(id int64) (*EvalRecord, error)

Eval returns one eval by id.

func (*Store) EvalDir

func (s *Store) EvalDir(skill string) (string, error)

EvalDir is the sidecar directory. lint.Excluded is the one definition of what does not ship as bundle content; everything eval-only must live under this path so pack's exclusion of it stays correct.

func (*Store) FinishEval

func (s *Store) FinishEval(id int64, status string) error

FinishEval marks an eval finished with a terminal status and stamps finished_at.

func (*Store) FinishRun

func (s *Store) FinishRun(id int64, o RunOutcome) error

FinishRun records a run's outcome. o.Status is expected to be one of StatusOK, StatusFailed, StatusError, or StatusTimeout: the first two make a later ClaimRun of the same key report done, the latter two make it retry.

func (*Store) LatestEval

func (s *Store) LatestEval(skill string) (*EvalRecord, error)

LatestEval returns the most recently created eval for a skill.

func (*Store) LatestReport

func (s *Store) LatestReport(skill string) ([]byte, int64, error)

LatestReport returns the report document for a skill's most recent eval, and that eval's id.

func (*Store) LoadSpec

func (s *Store) LoadSpec(skill string) (*spec.SkillSpec, int, error)

LoadSpec returns the latest stored spec version for a skill.

func (*Store) Report

func (s *Store) Report(evalID int64) ([]byte, error)

Report returns the stored report document for one eval.

func (*Store) ReportMeta

func (s *Store) ReportMeta(evalID int64) (ReportMeta, error)

ReportMeta returns the small set of report fields the store indexes directly for evalID, the read side of SaveReport's ReportMeta write. RobustnessGap round-trips as nil when it was stored as SQL NULL (not computed), never silently as 0 — mirroring the write-side comment on ReportMeta itself.

func (*Store) Root

func (s *Store) Root() string

Root is the .whetstone directory.

func (*Store) RunDir

func (s *Store) RunDir(skill string, evalID int64, k RunKey) (string, error)

RunDir is the artifact directory for one run, inside the skill's eval sidecar: <evalDir>/runs/<evalID>/<taskID>/<agent>__<model>__<condition>__<attempt>/. skill is validated the same way SkillDir validates it; every RunKey field is sanitised before joining, since each one traces back to untrusted input — and sanitizePathComponent guarantees none of them can contain the "__" used to join them, so the leaf splits back into its four fields unambiguously.

func (*Store) Runs

func (s *Store) Runs(evalID int64) ([]RunRecord, error)

Runs lists every run recorded for an eval, in claim order.

func (*Store) SaveJudgment

func (s *Store) SaveJudgment(evalID int64, j Judgment) error

SaveJudgment records one judge verdict against an eval.

func (*Store) SaveReport

func (s *Store) SaveReport(evalID int64, doc []byte, m ReportMeta) error

SaveReport stores a report as opaque JSON bytes, alongside the few fields the store indexes directly. Reports are stored opaque rather than typed so that this package never imports report — report imports score and drift, and importing report back here would create a cycle.

func (*Store) SaveScore

func (s *Store) SaveScore(evalID int64, sc ScoreRow) error

SaveScore upserts one agent/model pair's scores for an eval. Upsert rather than insert: a re-scored eval recomputes the same (eval_id, agent, model) key, and a unique-constraint failure there would abort the run.

func (*Store) SaveSpec

func (s *Store) SaveSpec(sp *spec.SkillSpec) (int, error)

SaveSpec appends a new spec version and rewrites the human-editable YAML. Both, deliberately: the database gives history and approval state, the file gives `whetstone spec edit` something to open.

The database row commits before the file is written, so a file-write failure loses no data — LoadSpec still returns the new version from the row alone. Because of that ordering, SaveSpec still returns the version it committed alongside a non-nil error from the file-write step: recomputing MAX(version)+1 on a naive retry would skip the already-committed version and leave a permanent gap. A caller that sees an error here should treat the returned version as already durable rather than retrying from scratch.

func (*Store) SaveSuiteCheck

func (s *Store) SaveSuiteCheck(skill, suiteRef string, rows []SuiteCheckRow) error

SaveSuiteCheck upserts, per task, whether it was found void the last time the suite was checked.

func (*Store) Scores

func (s *Store) Scores(evalID int64) ([]ScoreRow, error)

Scores lists every agent/model pair's stored scores for an eval, the read side of SaveScore.

func (*Store) SkillDir

func (s *Store) SkillDir(skill string) (string, error)

SkillDir is the bundle root for a skill. It returns an error rather than a path when skill is not a legal spec name — see skillDirName.

func (*Store) SpecHistory

func (s *Store) SpecHistory(skill string) ([]SpecRecord, error)

SpecHistory lists every stored version, newest first.

func (*Store) SpecPath

func (s *Store) SpecPath(skill string) (string, error)

SpecPath is the human-editable spec YAML. It sits beside the bundle rather than inside the eval sidecar because the spec is the authored artifact, not eval scaffolding.

func (*Store) SuiteChecks

func (s *Store) SuiteChecks(skill, suiteRef string) ([]SuiteCheckRow, error)

SuiteChecks lists the last-known void status of every checked task in a suite.

func (*Store) SuiteDir

func (s *Store) SuiteDir(skill string) (string, error)

SuiteDir holds generated task packages and the trigger set.

type SuiteCheckRow

type SuiteCheckRow struct {
	TaskID string
	Void   bool
	Reason string
}

SuiteCheckRow records whether one task in a suite was found void (its oracle or verifier no longer applies) when the suite was last checked.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL