report

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Overview

Package report defines the machine-readable result of an eval run: the schema a CI job reads, a later phase posts, and a future UI charts.

Index

Constants

View Source
const SchemaVersion = 1

SchemaVersion is the report.json schema's own version, distinct from the spec version of the skill it measures. Load refuses a report from a newer schema rather than misreading it: a field that changed meaning is worse than a field that is missing.

Variables

This section is empty.

Functions

This section is empty.

Types

type ComposeInput

type ComposeInput struct {
	Skill         string
	SpecVersion   int
	Tier          string
	SuiteRef      string
	EngineVersion string

	ModelPanel    []PanelMember
	PanelComplete bool

	Members []MemberInput
	Tasks   []TaskInput
	// Void lists tasks the oracle gate excluded from scoring, before any run
	// was ever attempted. Their TaskIDs are removed from Tasks before
	// scoring; the list itself survives to the report so a reader can see
	// what was excluded and why.
	//
	// This is a different exclusion from score.TaskPasses.Void, which is a
	// task whose runs all errored — see that doc for why "void" means two
	// different things here and how their report-level visibility currently
	// differs.
	Void []VoidTask

	// JudgeTrusted is whether the judge's calibration cleared the κ floor
	// (a later task selects the floor itself). false demotes UpliftSource to
	// the pass-rate fallback, but JudgeKappa is still recorded — it is the κ
	// that caused the demotion.
	//
	// JudgeKappa is nil when no judge was calibrated for this run at all — a
	// run with no gateway available produces no calibration, and that must
	// not read as a judge that scored κ = 0.0. Compose passes it straight
	// through to Report.JudgeKappa without manufacturing a value.
	JudgeTrusted   bool
	JudgeKappa     *float64
	JudgeLabeledBy string
	// JudgeModel is passed straight through onto Report.JudgeModel — see that
	// field's doc for why only a model name, not a gateway base URL, is
	// available to record here.
	JudgeModel string

	TriggerInferred bool
	// TriggerSource is passed through to Report.TriggerSource unchanged: the
	// single panel member the trigger probes ran on.
	TriggerSource PanelMember
	// TriggerUnknown is score.F1Result.Unknown passed through: the count of
	// trigger probes whose session could not be measured (e.g. it errored) and
	// so were excluded from every quadrant of the trigger confusion matrix
	// rather than counted as a miss.
	TriggerUnknown int
	Unevaluable    int
	// UnevaluableDetail is passed through to the report as given: Compose
	// does not aggregate it per-run or de-duplicate it. The caller owns
	// assembling this list from whatever per-run observations it holds.
	UnevaluableDetail []string

	StartedAt  time.Time
	FinishedAt time.Time

	Iterations []Iteration
}

ComposeInput is everything Compose needs to assemble a Report.

type ConditionReport

type ConditionReport struct {
	Condition store.Condition `json:"condition"`
	Model     string          `json:"model"`
	Passes    int             `json:"passes"`
	Runs      int             `json:"runs"`
}

ConditionReport is the pass rate for one task's condition (skill or baseline), across models.

type Iteration

type Iteration struct {
	N                int      `json:"n"`
	DevEffectiveness float64  `json:"dev_effectiveness"`
	Applied          int      `json:"applied"`
	Pruned           []string `json:"pruned,omitempty"`
	Notes            []string `json:"notes,omitempty"`
}

Iteration is one repair loop's outcome, when the eval ran with repair enabled.

type JudgeNote

type JudgeNote struct {
	Model    string   `json:"model"`
	Winner   string   `json:"winner"`
	Margin   float64  `json:"margin"`
	Evidence []string `json:"evidence,omitempty"`
	Votes    int      `json:"votes"`
}

JudgeNote is one judge verdict summarised for the report.

type MemberInput

type MemberInput struct {
	Member  PanelMember
	Pillars score.Pillars
	// Healthy is false when the member's adapter failed its probe. Such a
	// member contributes nothing to scoring rather than a zero.
	Healthy bool
	Detail  string
	// Drift is this member's per-run drift results, already scored by
	// drift.Score. Empty when the member is unhealthy.
	Drift []drift.Result
	// MetaPartial mirrors runner.Outcome.MetaPartial: true when this member's
	// session metadata was rebuilt from the store's own columns rather than
	// recovered in full. Compose passes it straight through onto MemberReport
	// so a reader can tell a figure rests on partial metadata.
	MetaPartial       bool
	MetaPartialReason string
}

MemberInput is one panel member's raw measurements, from which Compose derives Effectiveness and drift aggregation.

type MemberReport

type MemberReport struct {
	Member        PanelMember   `json:"member"`
	Pillars       score.Pillars `json:"pillars"`
	Effectiveness float64       `json:"effectiveness"`
	Drift         drift.Agg     `json:"drift"`
	DriftGrade    string        `json:"drift_grade"`
	// Healthy is false when the member's adapter failed its probe. Such a
	// member contributes nothing to the headline rather than a zero.
	Healthy bool   `json:"healthy"`
	Detail  string `json:"detail,omitempty"`
	// MetaPartial is true when this member's session metadata was rebuilt from
	// the store's own columns rather than recovered in full (a resumed run
	// whose recorded artifact could not be re-read) — runner.Outcome carries
	// the same flag under the same name. A reader must be able to tell that a
	// figure derived from this member rests on partial metadata.
	MetaPartial       bool   `json:"meta_partial,omitempty"`
	MetaPartialReason string `json:"meta_partial_reason,omitempty"`
}

MemberReport is one panel member's measured result.

type PanelMember

type PanelMember struct {
	Agent      string `json:"agent"`
	Model      string `json:"model"`
	Class      string `json:"class"`
	CLIVersion string `json:"cli_version"`
}

PanelMember identifies one model-panel entry on the report.

type Report

type Report struct {
	SchemaVersion int    `json:"schema_version"`
	Skill         string `json:"skill"`
	SpecVersion   int    `json:"spec_version"`
	Tier          string `json:"tier"`
	SuiteRef      string `json:"suite_ref"`
	EngineVersion string `json:"engine_version"`

	ModelPanel    []PanelMember `json:"model_panel"`
	PanelComplete bool          `json:"panel_complete"`

	Headline   float64    `json:"headline"`
	HeadlineCI [2]float64 `json:"headline_ci"`

	UpliftSource score.UpliftSource `json:"uplift_source"`
	// JudgeKappa is nil when no judge was calibrated for this run, as opposed
	// to a judge calibrated at κ=0 — the two are different facts and a bare
	// float64 cannot distinguish them.
	JudgeKappa     *float64 `json:"judge_kappa,omitempty"`
	JudgeLabeledBy string   `json:"judge_labeled_by,omitempty"`
	// JudgeModel identifies which model judged this run — the judge is a
	// distinct axis of comparability from the model panel being scored: an
	// operator-configurable judge means the same panel, scored on the same
	// suite, can produce a different Headline purely because a different
	// model did the judging. Empty when no judge ran for this report (the
	// pass-rate fallback in UpliftSource), or when the caller building the
	// report could not determine which model served the request.
	//
	// A gateway base URL would sharpen this further — a different base URL
	// can serve a different model behind the same name — but the llm.Gateway
	// interface this package's caller holds is opaque (Complete(ctx, Req)
	// (Res, error), no accessor for the endpoint or model it was constructed
	// with), so that fact is not obtainable at report-build time today. Only
	// the model name is recorded here; see cli/whetstone/eval.go for where a
	// future accessor would need to surface it.
	JudgeModel string `json:"judge_model,omitempty"`

	Members []MemberReport `json:"members"`
	// RobustnessGap is nil when it could not be computed — score.Matrix.ByClass
	// returns ok == false when a class has zero or more than one member, so a
	// strong/floor comparison is not always defined. A pointer rather than a
	// value-plus-flag pair: a zero value would otherwise read as "the floor
	// model kept up", the opposite of "we could not tell", and a two-field
	// representation makes that wrong state constructible by forgetting to set
	// the flag alongside the value.
	RobustnessGap *float64 `json:"robustness_gap,omitempty"`

	Tasks     []TaskReport `json:"tasks"`
	VoidTasks []VoidTask   `json:"void_tasks,omitempty"`

	TriggerInferred bool `json:"trigger_inferred"`
	// TriggerSource is the single panel member the trigger probes actually ran
	// on. Trigger firing is measured once per eval, on this member, and the
	// resulting F1 is then copied into every scored member's Pillars — it is
	// not a per-member measurement, whatever the member's own row in Members
	// might suggest. A zero value means no probes were planned for this tier.
	TriggerSource PanelMember `json:"trigger_source,omitempty"`
	// TriggerUnknown is the count of trigger probes excluded from the trigger
	// confusion matrix because their session could not be measured — see
	// score.Probe.Unknown. Excluded rather than counted as a miss, so an
	// infrastructure failure does not masquerade as a recall failure.
	TriggerUnknown    int      `json:"trigger_unknown,omitempty"`
	Unevaluable       int      `json:"unevaluable"`
	UnevaluableDetail []string `json:"unevaluable_detail,omitempty"`

	StartedAt  time.Time `json:"started_at"`
	FinishedAt time.Time `json:"finished_at"`

	Iterations []Iteration `json:"iterations,omitempty"`
}

Report is the machine-readable result of one eval run against one skill.

func Compose

func Compose(in ComposeInput) (*Report, error)

Compose assembles a Report from raw per-member and per-task measurements: each member's Effectiveness and drift aggregate, the panel-wide headline and its bootstrap confidence interval, the strong/floor robustness gap when it is defined, the judge demotion when the judge is untrusted, and the void-task exclusion — a void task is dropped from Tasks but kept on VoidTasks so the report shows what was excluded and why.

func Load

func Load(r io.Reader) (*Report, error)

Load reads a Report and rejects one written by a newer schema: a field that changed meaning is worse than a field that is missing, so a newer report read by an older binary must fail loudly rather than be misinterpreted.

func (*Report) Comparable

func (r *Report) Comparable(o *Report) (bool, string)

Comparable reports whether two reports measure the same thing, and why not when they do not.

This is the difference between a trend and a misleading chart. A score is a measurement against a specific suite, evaluated by a specific model panel, at a specific tier — change any of those and "the score dropped" is not a fact about the skill. What deliberately does *not* affect comparability is the spec version and the score itself: comparing v3 against v4 is the only question this method exists to answer.

func (*Report) HTML

func (r *Report) HTML(w io.Writer) error

HTML renders r as a single self-contained HTML document: no external stylesheet, script, font, or image, so the report keeps rendering offline and never signals a third party which skills a team evaluated. All model-authored and agent-authored text (task prompts, judge evidence, violation evidence) goes through html/template's default escaping.

func (*Report) Save

func (r *Report) Save(w io.Writer) error

Save writes r as indented JSON: a human opens this file when a score surprises them.

type RunDrift

type RunDrift struct {
	Agent   string `json:"agent"`
	Model   string `json:"model"`
	Attempt int    `json:"attempt"`
	drift.Result
	Violations []drift.Violation `json:"violations,omitempty"`
}

RunDrift is one run's drift measurement. It embeds drift.Result rather than duplicating Components/Adherence/Unevaluable as separate fields, so the per-task view rendered here and the per-member view drift.Aggregate consumes are always the same numbers, not two hand-built copies that can silently disagree.

Agent, alongside Model, is what disambiguates which panel member this run belongs to: a cross-vendor panel (e.g. --agents claude-code,cursor --models opus) can put two members on the same model string, and Model alone cannot tell them apart.

type TaskInput

type TaskInput struct {
	TaskID     string
	Kind       string
	Split      string
	Conditions []ConditionReport
	Drift      []RunDrift
	Judge      []JudgeNote
}

TaskInput is one task's raw results, before Compose excludes void tasks.

type TaskReport

type TaskReport struct {
	TaskID     string            `json:"task_id"`
	Kind       string            `json:"kind"`
	Split      string            `json:"split"`
	Conditions []ConditionReport `json:"conditions,omitempty"`
	Drift      []RunDrift        `json:"drift,omitempty"`
	Judge      []JudgeNote       `json:"judge,omitempty"`
}

TaskReport is one non-void task's results.

type VoidTask

type VoidTask struct {
	TaskID string `json:"task_id"`
	Reason string `json:"reason"`
}

VoidTask is a task excluded from scoring by the oracle gate, listed rather than merely dropped: a suite quietly losing tasks changes what the score means, and the reader has to be able to see it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL