quality

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package quality maps an eval report onto a stored quality record and persists it, so a score can be read back per skill version and compared across versions.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RegisterRoutes

func RegisterRoutes(api huma.API, store *Store, skills *skill.Store)

RegisterRoutes wires up the read-only quality endpoints: the skill's most recent scored record across all versions, and its full history newest-first for the version-over-version trend.

Types

type Executor

type Executor interface {
	Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
	QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
	Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
}

Executor is the subset of *pgxpool.Pool that Store's queries need. It is satisfied by both *pgxpool.Pool and pgx.Tx, so a caller can run a Store write inside a transaction shared with another store (see WithExecutor).

type Record

type Record struct {
	SkillID string
	Version int

	Headline       float64
	HeadlineCILow  float64
	HeadlineCIHigh float64

	Pillars     json.RawMessage
	PanelMatrix json.RawMessage

	// RobustnessGap is nil when it could not be computed — never a zero. A
	// zero means the floor model kept up with the strong one; nil means the
	// comparison was not defined for this run.
	RobustnessGap  *float64
	DriftGrade     string
	DriftBreakdown json.RawMessage

	Verified      bool
	PanelComplete bool

	SuiteRef      string
	EngineVersion string
	ModelPanel    json.RawMessage
	Tier          string
	// UpliftSource is one of the facts report.Comparable checks before
	// treating two reports' scores as a fair comparison — preserved here
	// alongside the other Comparable fields (SuiteRef, EngineVersion, Tier,
	// ModelPanel, PanelComplete) so a version-over-version trend can tell
	// whether its own comparison is valid without re-fetching every report.
	UpliftSource string
	// JudgeModel mirrors report.Report.JudgeModel — preserved alongside the
	// other Comparable fields so BuildSeries can reconstruct a report that
	// groups on it. nil for a row written before this field existed, and for
	// any run where the caller building the report could not determine which
	// model judged it. asReport (internal/quality/series.go) maps nil to one
	// shared "unknown judge" value, distinct from every real model name but
	// shared across every unknown row — see its doc for why records with no
	// recorded judge still group with each other rather than fragmenting.
	JudgeModel *string
	JobID      string
	ScoredAt   time.Time

	// CriticalForbidViolations counts critical-severity forbid-rule
	// violations observed across every run in the report. It is the one
	// signal in which the evaluation is evidence against the skill rather
	// than for it: a version whose runs violated its own contract does not
	// clear a held publish however high its headline score.
	//
	// Zero here means none were observed, which is a real measurement. The
	// gate expresses "not measured at all" as a nil *QualityState, never as
	// a zero.
	CriticalForbidViolations int

	// ReportJSON is the worker's full report exactly as posted, kept so the
	// detail page can show judge evidence and per-task contract violations —
	// facts the aggregates above deliberately compress away. Nil for every
	// row written before migration 015, and for any record read by a query
	// that does not select it: only Store.GetVersion does, because the
	// payload is large and no list or summary needs it.
	ReportJSON json.RawMessage
}

Record is one scored measurement of a skill version, as stored in skill_quality.

func FromReport

func FromReport(r *report.Report) (Record, error)

FromReport maps a report onto a record. It is pure: no database, no clock, no I/O — that is what makes it testable, and what lets a later ingestion endpoint call it before deciding whether to write anything.

func FromReportRaw

func FromReportRaw(r *report.Report, raw json.RawMessage) (Record, error)

FromReportRaw is FromReport plus the report's own serialised bytes. The caller passes what it received rather than re-marshalling r: a round trip through the struct would silently drop any field this binary does not know, and the stored report is meant to outlive this binary's schema.

type RecordOutput

type RecordOutput struct {
	SkillID                  string          `json:"skill_id"`
	Version                  int             `json:"version"`
	Headline                 float64         `json:"headline_score"`
	HeadlineCILow            float64         `json:"headline_ci_low"`
	HeadlineCIHigh           float64         `json:"headline_ci_high"`
	Pillars                  json.RawMessage `json:"pillar_breakdown"`
	PanelMatrix              json.RawMessage `json:"panel_matrix"`
	RobustnessGap            *float64        `json:"robustness_gap,omitempty"`
	DriftGrade               string          `json:"drift_grade,omitempty"`
	DriftBreakdown           json.RawMessage `json:"drift_breakdown"`
	Verified                 bool            `json:"verified"`
	PanelComplete            bool            `json:"panel_complete"`
	SuiteRef                 string          `json:"suite_ref"`
	EngineVersion            string          `json:"engine_version"`
	ModelPanel               json.RawMessage `json:"model_panel"`
	Tier                     string          `json:"tier"`
	UpliftSource             string          `json:"uplift_source,omitempty"`
	JudgeModel               *string         `json:"judge_model,omitempty"`
	JobID                    string          `json:"job_id,omitempty"`
	ScoredAt                 time.Time       `json:"scored_at"`
	CriticalForbidViolations int             `json:"critical_forbid_violations"`
	// ReportJSON is json:"-" deliberately. The summary and history endpoints
	// share this shape and must stay small; the full report is served only by
	// the per-version endpoint, which wraps this struct rather than widening
	// it. Struct tags are ignored by the conversion in toRecordOutput, so the
	// field must still be present here or that conversion stops compiling —
	// which is exactly the drift alarm it exists to be.
	ReportJSON json.RawMessage `json:"-"`
}

RecordOutput is the wire shape for a Record.

type Series

type Series struct {
	// Key is opaque and stable within one response; the client uses it for
	// React keys and nothing else.
	Key     string `json:"key"`
	Current bool   `json:"current"`
	// Reason is empty on the current series and otherwise names, in the
	// engine's own words, why this run cannot be charted with it.
	Reason string        `json:"reason"`
	Points []SeriesPoint `json:"points"`
}

Series is a run of scores that may honestly be charted together.

func BuildSeries

func BuildSeries(skillName string, history []Record) []Series

BuildSeries groups a skill's history into runs that report.Comparable considers chartable together. history arrives newest-first (Store.History's order), so the first record's group is the current one.

Every record appears in exactly one series and nothing is dropped: an operator who re-runs against a new panel must be able to see that the score moved because the panel changed, which requires the older run to stay visible with the reason attached.

type SeriesPoint

type SeriesPoint struct {
	Version        int       `json:"version"`
	Headline       float64   `json:"headline_score"`
	HeadlineCILow  float64   `json:"headline_ci_low"`
	HeadlineCIHigh float64   `json:"headline_ci_high"`
	Verified       bool      `json:"verified"`
	ScoredAt       time.Time `json:"scored_at"`
}

SeriesPoint is one scored version on a trend line.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store persists Records to skill_quality.

func NewStore

func NewStore(db *pgxpool.Pool) *Store

NewStore builds a Store over db.

func (*Store) GetVersion

func (s *Store) GetVersion(ctx context.Context, skillID string, version int) (*Record, error)

GetVersion returns the score for one specific version, including the full stored report. Ordered by scored_at with id as a deterministic tiebreak (see Latest) so re-fetching one version cannot return different rows on different requests. Returns (nil, nil) when that version has never been scored.

func (*Store) History

func (s *Store) History(ctx context.Context, skillID string) ([]Record, error)

History returns every scored row for a skill, across all versions, newest first.

func (*Store) Latest

func (s *Store) Latest(ctx context.Context, skillID string, version int) (*Record, error)

Latest returns the newest scored row for a skill version, ordered by scored_at with id as a deterministic tiebreak — two rows can share an identical scored_at (timestamp truncation, or two ingestions in one request), and without a secondary key Postgres gives no guarantee which one comes back. A missing row is not an error: it returns (nil, nil).

func (*Store) LatestAcrossVersions

func (s *Store) LatestAcrossVersions(ctx context.Context, skillID string) (*Record, error)

LatestAcrossVersions returns the newest scored row for a skill, across all of its versions, ordered by scored_at with id as a deterministic tiebreak (see Latest). Unlike Latest, it is not pinned to a specific version — a skill's quality badge should keep showing its most recent score even while a newer, not-yet-scored version is the current one; otherwise every publish makes the badge vanish until the next eval lands. A missing row is not an error: it returns (nil, nil).

func (*Store) Upsert

func (s *Store) Upsert(ctx context.Context, rec Record) error

Upsert inserts a new row every time. Despite the name this is deliberately not an update-in-place: history is kept because version-over-version comparison depends on the earlier rows surviving a re-score.

func (*Store) WithExecutor

func (s *Store) WithExecutor(e Executor) *Store

WithExecutor returns a Store whose queries run against e (typically a pgx.Tx) instead of the pool directly, so a caller can compose a write with another store's write in one transaction.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL