report

package
v0.20.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.

Index

Constants

View Source
const SchemaVersion = 3

SchemaVersion is the report schema. Load refuses a newer schema. Version 2 is the expectation pass rate; version 1 was a geometric mean. Version 3 redefines Delta. See Delta.

Variables

This section is empty.

Functions

This section is empty.

Types

type ComposeInput

type ComposeInput struct {
	Skill         string
	SpecVersion   int
	Tier          string
	SuiteRef      string
	EngineVersion string

	ModelPanel    []PanelMember
	PanelComplete bool

	Members []MemberInput
	Tasks   []TaskInput
	// Void lists evals excluded from scoring before any run was attempted.
	// Their TaskIDs are removed from Tasks; the list survives to the report so
	// a reader can see what was excluded and why.
	Void []VoidTask

	// Baseline is the same score with no skill installed, measured on the
	// primary member only. BaselineMeasured is false when no baseline session
	// ran, which is what separates a zero delta from an absent one.
	Baseline         float64
	BaselineMeasured bool
	BaselineWipeout  bool

	TokensMedian         int64
	TokensMedianBaseline int64

	TriggerF1       float64
	TriggerInferred bool
	TriggerSource   PanelMember
	TriggerUnknown  int

	GraderModel string

	// Dropped lists sessions the judge could not grade.
	Dropped []DroppedGrade
	// ReusedBaselines lists task ids whose baseline came from an earlier eval.
	ReusedBaselines []string

	StartedAt  time.Time
	FinishedAt time.Time
}

ComposeInput is everything Compose needs to assemble a Report.

type ConditionReport

type ConditionReport struct {
	Condition store.Condition `json:"condition"`
	Model     string          `json:"model"`
	Passes    int             `json:"passes"`
	Runs      int             `json:"runs"`
}

ConditionReport tallies expectations for one eval/condition/model.

type DroppedGrade added in v0.12.0

type DroppedGrade struct {
	TaskID    string `json:"task_id"`
	Condition string `json:"condition"`
	Attempt   int    `json:"attempt"`
	Reason    string `json:"reason"`
}

DroppedGrade is a session that ran but whose grade call failed after every retry. It is dropped from the denominator rather than scored as a failure, and listed here so the reader can see what the score did not include.

type GradeNote added in v0.11.0

type GradeNote struct {
	Model        string              `json:"model"`
	Condition    store.Condition     `json:"condition"`
	Attempt      int                 `json:"attempt"`
	Expectations []score.Expectation `json:"expectations,omitempty"`
}

GradeNote is one graded session's expectations and evidence.

type MemberInput

type MemberInput struct {
	Member PanelMember
	// Score is this member's 0–100 mean over its evals' pass rates, from
	// score.MemberScore. Ignored when Healthy is false.
	Score float64
	// Healthy is false when the member's adapter failed its probe. Such a
	// member contributes nothing to scoring rather than a zero.
	Healthy bool
	Detail  string
	// MetaPartial mirrors runner.Outcome.MetaPartial: true when this member's
	// session metadata was rebuilt from the store's own columns rather than
	// recovered in full.
	MetaPartial       bool
	MetaPartialReason string
}

MemberInput is one panel member's measured result, before Compose takes the panel minimum.

type MemberReport

type MemberReport struct {
	Member PanelMember `json:"member"`
	// Effectiveness is this member's 0–100 score. The name predates the
	// pipeline change; renaming it would break stored decoders.
	Effectiveness float64 `json:"effectiveness"`
	// Healthy is false when the member's probe failed.
	Healthy bool   `json:"healthy"`
	Detail  string `json:"detail,omitempty"`
	// MetaPartial is true when session metadata was rebuilt from store columns
	// rather than the run's artifact.
	MetaPartial       bool   `json:"meta_partial,omitempty"`
	MetaPartialReason string `json:"meta_partial_reason,omitempty"`
}

MemberReport is one panel member's measured result.

type PanelMember

type PanelMember struct {
	Agent      string `json:"agent"`
	Model      string `json:"model"`
	Class      string `json:"class"`
	CLIVersion string `json:"cli_version"`
	// Provider names the backend that served the member: empty for
	// Anthropic's own API, otherwise the provider type and the endpoint's
	// host, such as "openrouter" or "openai-compatible:r.openai.azure.com".
	// A changed provider splits the trend like a changed model.
	Provider string `json:"provider,omitempty"`
}

PanelMember identifies one model-panel entry on the report.

type Report

type Report struct {
	SchemaVersion int    `json:"schema_version"`
	Skill         string `json:"skill"`
	SpecVersion   int    `json:"spec_version"`
	Tier          string `json:"tier"`
	SuiteRef      string `json:"suite_ref"`
	EngineVersion string `json:"engine_version"`

	ModelPanel    []PanelMember `json:"model_panel"`
	PanelComplete bool          `json:"panel_complete"`
	// SandboxImage identifies the environment the panel ran in. Empty on a
	// report written before it was recorded.
	SandboxImage string `json:"sandbox_image,omitempty"`

	// Headline is the published 0–100 score: minimum across healthy members.
	Headline float64 `json:"headline"`
	// Baseline is the no-skill measurement, taken on the primary member.
	Baseline float64 `json:"baseline"`
	// Delta is the lift: the primary member's own effectiveness minus that same
	// member's baseline. Before schema 3 it subtracted that baseline from the
	// whole-panel Headline minimum, so a deep-tier run could report a negative
	// lift for a skill that helped. schema_version says which one a stored
	// report carries.
	Delta float64 `json:"delta"`
	// False when no baseline ran, or when the primary member did not score.
	DeltaMeasured bool `json:"delta_measured"`
	// BaselineWipeout is true when the baseline passed no expectation at all.
	BaselineWipeout bool `json:"baseline_wipeout,omitempty"`

	// TokensMedian is the median token spend with the skill installed;
	// TokensMedianBaseline is the same without it.
	TokensMedian         int64 `json:"tokens_median,omitempty"`
	TokensMedianBaseline int64 `json:"tokens_median_baseline,omitempty"`

	// TriggerF1 is the trigger smoke check. Reported beside the headline, not
	// folded into it.
	TriggerF1 float64 `json:"trigger_f1"`

	Members []MemberReport `json:"members"`

	Tasks     []TaskReport `json:"tasks"`
	VoidTasks []VoidTask   `json:"void_tasks,omitempty"`

	DroppedGrades []DroppedGrade `json:"dropped_grades,omitempty"`

	// ReusedBaselines lists the task ids whose baseline session came from an
	// earlier eval rather than a fresh run.
	ReusedBaselines []string `json:"reused_baselines,omitempty"`

	TriggerInferred bool `json:"trigger_inferred"`
	// TriggerSource is the panel member the trigger probes ran on.
	TriggerSource PanelMember `json:"trigger_source,omitempty"`
	// TriggerUnknown is the count of probes excluded from the confusion matrix
	// because their session could not be measured.
	TriggerUnknown int `json:"trigger_unknown,omitempty"`

	// GraderModel identifies which model graded this run.
	GraderModel string `json:"grader_model,omitempty"`

	StartedAt  time.Time `json:"started_at"`
	FinishedAt time.Time `json:"finished_at"`
}

Report is the machine-readable result of one eval run against one skill.

func Compose

func Compose(in ComposeInput) (*Report, error)

Compose assembles a Report from raw per-member and per-eval measurements: the panel-wide headline, the without-skill delta, and the void-eval exclusion — a void eval is dropped from Tasks but kept on VoidTasks so the report shows what was excluded and why.

func Load

func Load(r io.Reader) (*Report, error)

Load reads a Report. Rejects a newer schema version.

func (*Report) Comparable

func (r *Report) Comparable(o *Report) (bool, string)

Comparable reports whether two reports measure the same thing. A changed suite, panel, tier, grader, or engine version makes scores incomparable.

func (*Report) HTML

func (r *Report) HTML(w io.Writer) error

HTML renders r as a self-contained HTML document.

func (*Report) SameConditions added in v0.18.0

func (r *Report) SameConditions(o *Report) (bool, string)

SameConditions reports whether two reports ran under the same conditions, whatever skill each measured. A contest reusing one candidate's report from another contest needs this and not Comparable, because its candidates may be different skills.

func (*Report) Save

func (r *Report) Save(w io.Writer) error

Save writes r as indented JSON.

type TaskInput

type TaskInput struct {
	TaskID     string
	Conditions []ConditionReport
	Grades     []GradeNote
}

TaskInput is one eval's raw results, before Compose excludes void evals.

type TaskReport

type TaskReport struct {
	TaskID     string            `json:"task_id"`
	Conditions []ConditionReport `json:"conditions,omitempty"`
	Grades     []GradeNote       `json:"grades,omitempty"`
}

TaskReport is one non-void eval's results.

type VoidTask

type VoidTask struct {
	TaskID string `json:"task_id"`
	Reason string `json:"reason"`
}

VoidTask is an eval excluded from scoring, listed so the reader can see it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL