Documentation
¶
Overview ¶
Package report defines the machine-readable result of an eval run: the schema a CI job reads, the worker posts, and the UI charts.
Index ¶
Constants ¶
const SchemaVersion = 2
SchemaVersion is the report schema. Load refuses a newer schema. Version 2 is the expectation pass rate; version 1 was a geometric mean.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ComposeInput ¶
type ComposeInput struct {
Skill string
SpecVersion int
Tier string
SuiteRef string
EngineVersion string
ModelPanel []PanelMember
PanelComplete bool
Members []MemberInput
Tasks []TaskInput
// Void lists evals excluded from scoring before any run was attempted.
// Their TaskIDs are removed from Tasks; the list survives to the report so
// a reader can see what was excluded and why.
Void []VoidTask
// Baseline is the same score with no skill installed, measured on the
// primary member only. BaselineMeasured is false when no baseline session
// ran, which is what separates a zero delta from an absent one.
Baseline float64
BaselineMeasured bool
BaselineWipeout bool
TokensMedian int64
TokensMedianBaseline int64
TriggerF1 float64
TriggerInferred bool
TriggerSource PanelMember
TriggerUnknown int
GraderModel string
StartedAt time.Time
FinishedAt time.Time
}
ComposeInput is everything Compose needs to assemble a Report.
type ConditionReport ¶
type ConditionReport struct {
Condition store.Condition `json:"condition"`
Model string `json:"model"`
Passes int `json:"passes"`
Runs int `json:"runs"`
}
ConditionReport tallies expectations for one eval/condition/model.
type GradeNote ¶ added in v0.11.0
type GradeNote struct {
Model string `json:"model"`
Condition store.Condition `json:"condition"`
Attempt int `json:"attempt"`
Expectations []score.Expectation `json:"expectations,omitempty"`
}
GradeNote is one graded session's expectations and evidence.
type MemberInput ¶
type MemberInput struct {
Member PanelMember
// Score is this member's 0–100 mean over its evals' pass rates, from
// score.MemberScore. Ignored when Healthy is false.
Score float64
// Healthy is false when the member's adapter failed its probe. Such a
// member contributes nothing to scoring rather than a zero.
Healthy bool
Detail string
// MetaPartial mirrors runner.Outcome.MetaPartial: true when this member's
// session metadata was rebuilt from the store's own columns rather than
// recovered in full.
MetaPartial bool
MetaPartialReason string
}
MemberInput is one panel member's measured result, before Compose takes the panel minimum.
type MemberReport ¶
type MemberReport struct {
Member PanelMember `json:"member"`
// Effectiveness is this member's 0–100 score. The name predates the
// pipeline change; renaming it would break stored decoders.
Effectiveness float64 `json:"effectiveness"`
// Healthy is false when the member's probe failed.
Healthy bool `json:"healthy"`
Detail string `json:"detail,omitempty"`
// MetaPartial is true when session metadata was rebuilt from store columns
// rather than the run's artifact.
MetaPartial bool `json:"meta_partial,omitempty"`
MetaPartialReason string `json:"meta_partial_reason,omitempty"`
}
MemberReport is one panel member's measured result.
type PanelMember ¶
type PanelMember struct {
Agent string `json:"agent"`
Model string `json:"model"`
Class string `json:"class"`
CLIVersion string `json:"cli_version"`
}
PanelMember identifies one model-panel entry on the report.
type Report ¶
type Report struct {
SchemaVersion int `json:"schema_version"`
Skill string `json:"skill"`
SpecVersion int `json:"spec_version"`
Tier string `json:"tier"`
SuiteRef string `json:"suite_ref"`
EngineVersion string `json:"engine_version"`
ModelPanel []PanelMember `json:"model_panel"`
PanelComplete bool `json:"panel_complete"`
// Headline is the published 0–100 score: minimum across healthy members.
Headline float64 `json:"headline"`
// Baseline is the no-skill measurement. DeltaMeasured is false when no
// baseline ran — a zero delta and an absent delta are different facts.
Baseline float64 `json:"baseline"`
Delta float64 `json:"delta"`
DeltaMeasured bool `json:"delta_measured"`
// BaselineWipeout is true when the baseline passed no expectation at all.
BaselineWipeout bool `json:"baseline_wipeout,omitempty"`
// TokensMedian is the median token spend with the skill installed;
// TokensMedianBaseline is the same without it.
TokensMedian int64 `json:"tokens_median,omitempty"`
TokensMedianBaseline int64 `json:"tokens_median_baseline,omitempty"`
// TriggerF1 is the trigger smoke check. Reported beside the headline, not
// folded into it.
TriggerF1 float64 `json:"trigger_f1"`
Members []MemberReport `json:"members"`
Tasks []TaskReport `json:"tasks"`
VoidTasks []VoidTask `json:"void_tasks,omitempty"`
TriggerInferred bool `json:"trigger_inferred"`
// TriggerSource is the panel member the trigger probes ran on.
TriggerSource PanelMember `json:"trigger_source,omitempty"`
// TriggerUnknown is the count of probes excluded from the confusion matrix
// because their session could not be measured.
TriggerUnknown int `json:"trigger_unknown,omitempty"`
// GraderModel identifies which model graded this run.
GraderModel string `json:"grader_model,omitempty"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
}
Report is the machine-readable result of one eval run against one skill.
func Compose ¶
func Compose(in ComposeInput) (*Report, error)
Compose assembles a Report from raw per-member and per-eval measurements: the panel-wide headline, the without-skill delta, and the void-eval exclusion — a void eval is dropped from Tasks but kept on VoidTasks so the report shows what was excluded and why.
func (*Report) Comparable ¶
Comparable reports whether two reports measure the same thing. A changed suite, panel, tier, grader, or engine version makes scores incomparable.
type TaskInput ¶
type TaskInput struct {
TaskID string
Conditions []ConditionReport
Grades []GradeNote
}
TaskInput is one eval's raw results, before Compose excludes void evals.
type TaskReport ¶
type TaskReport struct {
TaskID string `json:"task_id"`
Conditions []ConditionReport `json:"conditions,omitempty"`
Grades []GradeNote `json:"grades,omitempty"`
}
TaskReport is one non-void eval's results.