report

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 13, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.

Index

Constants

View Source
const SchemaVersion = 2

SchemaVersion is the report schema. Load refuses a newer schema. Version 2 is the expectation pass rate; version 1 was a geometric mean.

Variables

This section is empty.

Functions

This section is empty.

Types

type ComposeInput

type ComposeInput struct {
	Skill         string
	SpecVersion   int
	Tier          string
	SuiteRef      string
	EngineVersion string

	ModelPanel    []PanelMember
	PanelComplete bool

	Members []MemberInput
	Tasks   []TaskInput
	// Void lists evals excluded from scoring before any run was attempted.
	// Their TaskIDs are removed from Tasks; the list survives to the report so
	// a reader can see what was excluded and why.
	Void []VoidTask

	// Baseline is the same score with no skill installed, measured on the
	// primary member only. BaselineMeasured is false when no baseline session
	// ran, which is what separates a zero delta from an absent one.
	Baseline         float64
	BaselineMeasured bool
	BaselineWipeout  bool

	TokensMedian         int64
	TokensMedianBaseline int64

	TriggerF1       float64
	TriggerInferred bool
	TriggerSource   PanelMember
	TriggerUnknown  int

	GraderModel string

	StartedAt  time.Time
	FinishedAt time.Time
}

ComposeInput is everything Compose needs to assemble a Report.

type ConditionReport

type ConditionReport struct {
	Condition store.Condition `json:"condition"`
	Model     string          `json:"model"`
	Passes    int             `json:"passes"`
	Runs      int             `json:"runs"`
}

ConditionReport tallies expectations for one eval/condition/model.

type GradeNote added in v0.11.0

type GradeNote struct {
	Model        string              `json:"model"`
	Condition    store.Condition     `json:"condition"`
	Attempt      int                 `json:"attempt"`
	Expectations []score.Expectation `json:"expectations,omitempty"`
}

GradeNote is one graded session's expectations and evidence.

type MemberInput

type MemberInput struct {
	Member PanelMember
	// Score is this member's 0–100 mean over its evals' pass rates, from
	// score.MemberScore. Ignored when Healthy is false.
	Score float64
	// Healthy is false when the member's adapter failed its probe. Such a
	// member contributes nothing to scoring rather than a zero.
	Healthy bool
	Detail  string
	// MetaPartial mirrors runner.Outcome.MetaPartial: true when this member's
	// session metadata was rebuilt from the store's own columns rather than
	// recovered in full.
	MetaPartial       bool
	MetaPartialReason string
}

MemberInput is one panel member's measured result, before Compose takes the panel minimum.

type MemberReport

type MemberReport struct {
	Member PanelMember `json:"member"`
	// Effectiveness is this member's 0–100 score. The name predates the
	// pipeline change; renaming it would break stored decoders.
	Effectiveness float64 `json:"effectiveness"`
	// Healthy is false when the member's probe failed.
	Healthy bool   `json:"healthy"`
	Detail  string `json:"detail,omitempty"`
	// MetaPartial is true when session metadata was rebuilt from store columns
	// rather than the run's artifact.
	MetaPartial       bool   `json:"meta_partial,omitempty"`
	MetaPartialReason string `json:"meta_partial_reason,omitempty"`
}

MemberReport is one panel member's measured result.

type PanelMember

type PanelMember struct {
	Agent      string `json:"agent"`
	Model      string `json:"model"`
	Class      string `json:"class"`
	CLIVersion string `json:"cli_version"`
}

PanelMember identifies one model-panel entry on the report.

type Report

type Report struct {
	SchemaVersion int    `json:"schema_version"`
	Skill         string `json:"skill"`
	SpecVersion   int    `json:"spec_version"`
	Tier          string `json:"tier"`
	SuiteRef      string `json:"suite_ref"`
	EngineVersion string `json:"engine_version"`

	ModelPanel    []PanelMember `json:"model_panel"`
	PanelComplete bool          `json:"panel_complete"`

	// Headline is the published 0–100 score: minimum across healthy members.
	Headline float64 `json:"headline"`
	// Baseline is the no-skill measurement. DeltaMeasured is false when no
	// baseline ran — a zero delta and an absent delta are different facts.
	Baseline      float64 `json:"baseline"`
	Delta         float64 `json:"delta"`
	DeltaMeasured bool    `json:"delta_measured"`
	// BaselineWipeout is true when the baseline passed no expectation at all.
	BaselineWipeout bool `json:"baseline_wipeout,omitempty"`

	// TokensMedian is the median token spend with the skill installed;
	// TokensMedianBaseline is the same without it.
	TokensMedian         int64 `json:"tokens_median,omitempty"`
	TokensMedianBaseline int64 `json:"tokens_median_baseline,omitempty"`

	// TriggerF1 is the trigger smoke check. Reported beside the headline, not
	// folded into it.
	TriggerF1 float64 `json:"trigger_f1"`

	Members []MemberReport `json:"members"`

	Tasks     []TaskReport `json:"tasks"`
	VoidTasks []VoidTask   `json:"void_tasks,omitempty"`

	TriggerInferred bool `json:"trigger_inferred"`
	// TriggerSource is the panel member the trigger probes ran on.
	TriggerSource PanelMember `json:"trigger_source,omitempty"`
	// TriggerUnknown is the count of probes excluded from the confusion matrix
	// because their session could not be measured.
	TriggerUnknown int `json:"trigger_unknown,omitempty"`

	// GraderModel identifies which model graded this run.
	GraderModel string `json:"grader_model,omitempty"`

	StartedAt  time.Time `json:"started_at"`
	FinishedAt time.Time `json:"finished_at"`
}

Report is the machine-readable result of one eval run against one skill.

func Compose

func Compose(in ComposeInput) (*Report, error)

Compose assembles a Report from raw per-member and per-eval measurements: the panel-wide headline, the without-skill delta, and the void-eval exclusion — a void eval is dropped from Tasks but kept on VoidTasks so the report shows what was excluded and why.

func Load

func Load(r io.Reader) (*Report, error)

Load reads a Report. Rejects a newer schema version.

func (*Report) Comparable

func (r *Report) Comparable(o *Report) (bool, string)

Comparable reports whether two reports measure the same thing. A changed suite, panel, tier, grader, or engine version makes scores incomparable.

func (*Report) HTML

func (r *Report) HTML(w io.Writer) error

HTML renders r as a self-contained HTML document.

func (*Report) Save

func (r *Report) Save(w io.Writer) error

Save writes r as indented JSON.

type TaskInput

type TaskInput struct {
	TaskID     string
	Conditions []ConditionReport
	Grades     []GradeNote
}

TaskInput is one eval's raw results, before Compose excludes void evals.

type TaskReport

type TaskReport struct {
	TaskID     string            `json:"task_id"`
	Conditions []ConditionReport `json:"conditions,omitempty"`
	Grades     []GradeNote       `json:"grades,omitempty"`
}

TaskReport is one non-void eval's results.

type VoidTask

type VoidTask struct {
	TaskID string `json:"task_id"`
	Reason string `json:"reason"`
}

VoidTask is an eval excluded from scoring, listed so the reader can see it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL