Documentation
¶
Overview ¶
Package quality maps an eval report onto a stored quality record and persists it, so a score can be read back per skill version and compared across versions.
Index ¶
- func RegisterRoutes(api huma.API, store *Store, skills *skill.Store)
- type Executor
- type Record
- type RecordOutput
- type Series
- type SeriesPoint
- type Store
- func (s *Store) GetVersion(ctx context.Context, skillID string, version int) (*Record, error)
- func (s *Store) History(ctx context.Context, skillID string) ([]Record, error)
- func (s *Store) Latest(ctx context.Context, skillID string, version int) (*Record, error)
- func (s *Store) LatestAcrossVersions(ctx context.Context, skillID string) (*Record, error)
- func (s *Store) Upsert(ctx context.Context, rec Record) error
- func (s *Store) WithExecutor(e Executor) *Store
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Executor ¶
type Executor interface {
Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}
Executor is the subset of *pgxpool.Pool that Store's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run a Store write inside a transaction shared with another store (see WithExecutor).
type Record ¶
type Record struct {
SkillID string
Version int
Headline float64
HeadlineCILow float64
HeadlineCIHigh float64
Pillars json.RawMessage
PanelMatrix json.RawMessage
// RobustnessGap is nil when it could not be computed — never a zero. A
// zero means the floor model kept up with the strong one; nil means the
// comparison was not defined for this run.
RobustnessGap *float64
DriftGrade string
DriftBreakdown json.RawMessage
Verified bool
PanelComplete bool
SuiteRef string
EngineVersion string
ModelPanel json.RawMessage
Tier string
// UpliftSource is one of the facts report.Comparable checks before
// treating two reports' scores as a fair comparison — preserved here
// alongside the other Comparable fields (SuiteRef, EngineVersion, Tier,
// ModelPanel, PanelComplete) so a version-over-version trend can tell
// whether its own comparison is valid without re-fetching every report.
UpliftSource string
// JudgeModel mirrors report.Report.JudgeModel — preserved alongside the
// other Comparable fields so BuildSeries can reconstruct a report that
// groups on it. nil for a row written before this field existed, and for
// any run where the caller building the report could not determine which
// model judged it. asReport (internal/quality/series.go) maps nil to one
// shared "unknown judge" value, distinct from every real model name but
// shared across every unknown row — see its doc for why records with no
// recorded judge still group with each other rather than fragmenting.
JudgeModel *string
JobID string
ScoredAt time.Time
// CriticalForbidViolations counts critical-severity forbid-rule
// violations observed across every run in the report. It is the one
// signal in which the evaluation is evidence against the skill rather
// than for it: a version whose runs violated its own contract does not
// clear a held publish however high its headline score.
//
// Zero here means none were observed, which is a real measurement. The
// gate expresses "not measured at all" as a nil *QualityState, never as
// a zero.
CriticalForbidViolations int
// ReportJSON is the worker's full report exactly as posted, kept so the
// detail page can show judge evidence and per-task contract violations —
// facts the aggregates above deliberately compress away. Nil for every
// row written before migration 015, and for any record read by a query
// that does not select it: only Store.GetVersion does, because the
// payload is large and no list or summary needs it.
ReportJSON json.RawMessage
}
Record is one scored measurement of a skill version, as stored in skill_quality.
func FromReport ¶
FromReport maps a report onto a record. It is pure: no database, no clock, no I/O — that is what makes it testable, and what lets a later ingestion endpoint call it before deciding whether to write anything.
func FromReportRaw ¶
FromReportRaw is FromReport plus the report's own serialised bytes. The caller passes what it received rather than re-marshalling r: a round trip through the struct would silently drop any field this binary does not know, and the stored report is meant to outlive this binary's schema.
type RecordOutput ¶
type RecordOutput struct {
SkillID string `json:"skill_id"`
Version int `json:"version"`
Headline float64 `json:"headline_score"`
HeadlineCILow float64 `json:"headline_ci_low"`
HeadlineCIHigh float64 `json:"headline_ci_high"`
Pillars json.RawMessage `json:"pillar_breakdown"`
PanelMatrix json.RawMessage `json:"panel_matrix"`
RobustnessGap *float64 `json:"robustness_gap,omitempty"`
DriftGrade string `json:"drift_grade,omitempty"`
DriftBreakdown json.RawMessage `json:"drift_breakdown"`
Verified bool `json:"verified"`
PanelComplete bool `json:"panel_complete"`
SuiteRef string `json:"suite_ref"`
EngineVersion string `json:"engine_version"`
ModelPanel json.RawMessage `json:"model_panel"`
Tier string `json:"tier"`
UpliftSource string `json:"uplift_source,omitempty"`
JudgeModel *string `json:"judge_model,omitempty"`
JobID string `json:"job_id,omitempty"`
ScoredAt time.Time `json:"scored_at"`
CriticalForbidViolations int `json:"critical_forbid_violations"`
// ReportJSON is json:"-" deliberately. The summary and history endpoints
// share this shape and must stay small; the full report is served only by
// the per-version endpoint, which wraps this struct rather than widening
// it. Struct tags are ignored by the conversion in toRecordOutput, so the
// field must still be present here or that conversion stops compiling —
// which is exactly the drift alarm it exists to be.
ReportJSON json.RawMessage `json:"-"`
}
RecordOutput is the wire shape for a Record.
type Series ¶
type Series struct {
// Key is opaque and stable within one response; the client uses it for
// React keys and nothing else.
Key string `json:"key"`
Current bool `json:"current"`
// Reason is empty on the current series and otherwise names, in the
// engine's own words, why this run cannot be charted with it.
Reason string `json:"reason"`
Points []SeriesPoint `json:"points"`
}
Series is a run of scores that may honestly be charted together.
func BuildSeries ¶
BuildSeries groups a skill's history into runs that report.Comparable considers chartable together. history arrives newest-first (Store.History's order), so the first record's group is the current one.
Every record appears in exactly one series and nothing is dropped: an operator who re-runs against a new panel must be able to see that the score moved because the panel changed, which requires the older run to stay visible with the reason attached.
type SeriesPoint ¶
type SeriesPoint struct {
Version int `json:"version"`
Headline float64 `json:"headline_score"`
HeadlineCILow float64 `json:"headline_ci_low"`
HeadlineCIHigh float64 `json:"headline_ci_high"`
Verified bool `json:"verified"`
ScoredAt time.Time `json:"scored_at"`
}
SeriesPoint is one scored version on a trend line.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store persists Records to skill_quality.
func (*Store) GetVersion ¶
GetVersion returns the score for one specific version, including the full stored report. Ordered by scored_at with id as a deterministic tiebreak (see Latest) so re-fetching one version cannot return different rows on different requests. Returns (nil, nil) when that version has never been scored.
func (*Store) History ¶
History returns every scored row for a skill, across all versions, newest first.
func (*Store) Latest ¶
Latest returns the newest scored row for a skill version, ordered by scored_at with id as a deterministic tiebreak — two rows can share an identical scored_at (timestamp truncation, or two ingestions in one request), and without a secondary key Postgres gives no guarantee which one comes back. A missing row is not an error: it returns (nil, nil).
func (*Store) LatestAcrossVersions ¶
LatestAcrossVersions returns the newest scored row for a skill, across all of its versions, ordered by scored_at with id as a deterministic tiebreak (see Latest). Unlike Latest, it is not pinned to a specific version — a skill's quality badge should keep showing its most recent score even while a newer, not-yet-scored version is the current one; otherwise every publish makes the badge vanish until the next eval lands. A missing row is not an error: it returns (nil, nil).
func (*Store) Upsert ¶
Upsert inserts a new row every time. Despite the name this is deliberately not an update-in-place: history is kept because version-over-version comparison depends on the earlier rows surviving a re-score.
func (*Store) WithExecutor ¶
WithExecutor returns a Store whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.