Documentation
¶
Overview ¶
Package report defines the machine-readable result of an eval run: the schema a CI job reads, a later phase posts, and a future UI charts.
Index ¶
Constants ¶
const SchemaVersion = 1
SchemaVersion is the report.json schema's own version, distinct from the spec version of the skill it measures. Load refuses a report from a newer schema rather than misreading it: a field that changed meaning is worse than a field that is missing.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ComposeInput ¶
type ComposeInput struct {
Skill string
SpecVersion int
Tier string
SuiteRef string
EngineVersion string
ModelPanel []PanelMember
PanelComplete bool
Members []MemberInput
Tasks []TaskInput
// Void lists tasks the oracle gate excluded from scoring, before any run
// was ever attempted. Their TaskIDs are removed from Tasks before
// scoring; the list itself survives to the report so a reader can see
// what was excluded and why.
//
// This is a different exclusion from score.TaskPasses.Void, which is a
// task whose runs all errored — see that doc for why "void" means two
// different things here and how their report-level visibility currently
// differs.
Void []VoidTask
// JudgeTrusted is whether the judge's calibration cleared the κ floor
// (a later task selects the floor itself). false demotes UpliftSource to
// the pass-rate fallback, but JudgeKappa is still recorded — it is the κ
// that caused the demotion.
//
// JudgeKappa is nil when no judge was calibrated for this run at all — a
// run with no gateway available produces no calibration, and that must
// not read as a judge that scored κ = 0.0. Compose passes it straight
// through to Report.JudgeKappa without manufacturing a value.
JudgeTrusted bool
JudgeKappa *float64
JudgeLabeledBy string
// JudgeModel is passed straight through onto Report.JudgeModel — see that
// field's doc for why only a model name, not a gateway base URL, is
// available to record here.
JudgeModel string
TriggerInferred bool
// TriggerSource is passed through to Report.TriggerSource unchanged: the
// single panel member the trigger probes ran on.
TriggerSource PanelMember
// TriggerUnknown is score.F1Result.Unknown passed through: the count of
// trigger probes whose session could not be measured (e.g. it errored) and
// so were excluded from every quadrant of the trigger confusion matrix
// rather than counted as a miss.
TriggerUnknown int
Unevaluable int
// UnevaluableDetail is passed through to the report as given: Compose
// does not aggregate it per-run or de-duplicate it. The caller owns
// assembling this list from whatever per-run observations it holds.
UnevaluableDetail []string
StartedAt time.Time
FinishedAt time.Time
Iterations []Iteration
}
ComposeInput is everything Compose needs to assemble a Report.
type ConditionReport ¶
type ConditionReport struct {
Condition store.Condition `json:"condition"`
Model string `json:"model"`
Passes int `json:"passes"`
Runs int `json:"runs"`
}
ConditionReport is the pass rate for one task's condition (skill or baseline), across models.
type Iteration ¶
type Iteration struct {
N int `json:"n"`
DevEffectiveness float64 `json:"dev_effectiveness"`
Applied int `json:"applied"`
Pruned []string `json:"pruned,omitempty"`
Notes []string `json:"notes,omitempty"`
}
Iteration is one repair loop's outcome, when the eval ran with repair enabled.
type JudgeNote ¶
type JudgeNote struct {
Model string `json:"model"`
Winner string `json:"winner"`
Margin float64 `json:"margin"`
Evidence []string `json:"evidence,omitempty"`
Votes int `json:"votes"`
}
JudgeNote is one judge verdict summarised for the report.
type MemberInput ¶
type MemberInput struct {
Member PanelMember
Pillars score.Pillars
// Healthy is false when the member's adapter failed its probe. Such a
// member contributes nothing to scoring rather than a zero.
Healthy bool
Detail string
// Drift is this member's per-run drift results, already scored by
// drift.Score. Empty when the member is unhealthy.
Drift []drift.Result
// MetaPartial mirrors runner.Outcome.MetaPartial: true when this member's
// session metadata was rebuilt from the store's own columns rather than
// recovered in full. Compose passes it straight through onto MemberReport
// so a reader can tell a figure rests on partial metadata.
MetaPartial bool
MetaPartialReason string
}
MemberInput is one panel member's raw measurements, from which Compose derives Effectiveness and drift aggregation.
type MemberReport ¶
type MemberReport struct {
Member PanelMember `json:"member"`
Pillars score.Pillars `json:"pillars"`
Effectiveness float64 `json:"effectiveness"`
Drift drift.Agg `json:"drift"`
DriftGrade string `json:"drift_grade"`
// Healthy is false when the member's adapter failed its probe. Such a
// member contributes nothing to the headline rather than a zero.
Healthy bool `json:"healthy"`
Detail string `json:"detail,omitempty"`
// MetaPartial is true when this member's session metadata was rebuilt from
// the store's own columns rather than recovered in full (a resumed run
// whose recorded artifact could not be re-read) — runner.Outcome carries
// the same flag under the same name. A reader must be able to tell that a
// figure derived from this member rests on partial metadata.
MetaPartial bool `json:"meta_partial,omitempty"`
MetaPartialReason string `json:"meta_partial_reason,omitempty"`
}
MemberReport is one panel member's measured result.
type PanelMember ¶
type PanelMember struct {
Agent string `json:"agent"`
Model string `json:"model"`
Class string `json:"class"`
CLIVersion string `json:"cli_version"`
}
PanelMember identifies one model-panel entry on the report.
type Report ¶
type Report struct {
SchemaVersion int `json:"schema_version"`
Skill string `json:"skill"`
SpecVersion int `json:"spec_version"`
Tier string `json:"tier"`
SuiteRef string `json:"suite_ref"`
EngineVersion string `json:"engine_version"`
ModelPanel []PanelMember `json:"model_panel"`
PanelComplete bool `json:"panel_complete"`
Headline float64 `json:"headline"`
HeadlineCI [2]float64 `json:"headline_ci"`
UpliftSource score.UpliftSource `json:"uplift_source"`
// JudgeKappa is nil when no judge was calibrated for this run, as opposed
// to a judge calibrated at κ=0 — the two are different facts and a bare
// float64 cannot distinguish them.
JudgeKappa *float64 `json:"judge_kappa,omitempty"`
JudgeLabeledBy string `json:"judge_labeled_by,omitempty"`
// JudgeModel identifies which model judged this run — the judge is a
// distinct axis of comparability from the model panel being scored: an
// operator-configurable judge means the same panel, scored on the same
// suite, can produce a different Headline purely because a different
// model did the judging. Empty when no judge ran for this report (the
// pass-rate fallback in UpliftSource), or when the caller building the
// report could not determine which model served the request.
//
// A gateway base URL would sharpen this further — a different base URL
// can serve a different model behind the same name — but the llm.Gateway
// interface this package's caller holds is opaque (Complete(ctx, Req)
// (Res, error), no accessor for the endpoint or model it was constructed
// with), so that fact is not obtainable at report-build time today. Only
// the model name is recorded here; see cli/whetstone/eval.go for where a
// future accessor would need to surface it.
JudgeModel string `json:"judge_model,omitempty"`
Members []MemberReport `json:"members"`
// RobustnessGap is nil when it could not be computed — score.Matrix.ByClass
// returns ok == false when a class has zero or more than one member, so a
// strong/floor comparison is not always defined. A pointer rather than a
// value-plus-flag pair: a zero value would otherwise read as "the floor
// model kept up", the opposite of "we could not tell", and a two-field
// representation makes that wrong state constructible by forgetting to set
// the flag alongside the value.
RobustnessGap *float64 `json:"robustness_gap,omitempty"`
Tasks []TaskReport `json:"tasks"`
VoidTasks []VoidTask `json:"void_tasks,omitempty"`
TriggerInferred bool `json:"trigger_inferred"`
// TriggerSource is the single panel member the trigger probes actually ran
// on. Trigger firing is measured once per eval, on this member, and the
// resulting F1 is then copied into every scored member's Pillars — it is
// not a per-member measurement, whatever the member's own row in Members
// might suggest. A zero value means no probes were planned for this tier.
TriggerSource PanelMember `json:"trigger_source,omitempty"`
// TriggerUnknown is the count of trigger probes excluded from the trigger
// confusion matrix because their session could not be measured — see
// score.Probe.Unknown. Excluded rather than counted as a miss, so an
// infrastructure failure does not masquerade as a recall failure.
TriggerUnknown int `json:"trigger_unknown,omitempty"`
Unevaluable int `json:"unevaluable"`
UnevaluableDetail []string `json:"unevaluable_detail,omitempty"`
StartedAt time.Time `json:"started_at"`
FinishedAt time.Time `json:"finished_at"`
Iterations []Iteration `json:"iterations,omitempty"`
}
Report is the machine-readable result of one eval run against one skill.
func Compose ¶
func Compose(in ComposeInput) (*Report, error)
Compose assembles a Report from raw per-member and per-task measurements: each member's Effectiveness and drift aggregate, the panel-wide headline and its bootstrap confidence interval, the strong/floor robustness gap when it is defined, the judge demotion when the judge is untrusted, and the void-task exclusion — a void task is dropped from Tasks but kept on VoidTasks so the report shows what was excluded and why.
func Load ¶
Load reads a Report and rejects one written by a newer schema: a field that changed meaning is worse than a field that is missing, so a newer report read by an older binary must fail loudly rather than be misinterpreted.
func (*Report) Comparable ¶
Comparable reports whether two reports measure the same thing, and why not when they do not.
This is the difference between a trend and a misleading chart. A score is a measurement against a specific suite, evaluated by a specific model panel, at a specific tier — change any of those and "the score dropped" is not a fact about the skill. What deliberately does *not* affect comparability is the spec version and the score itself: comparing v3 against v4 is the only question this method exists to answer.
func (*Report) HTML ¶
HTML renders r as a single self-contained HTML document: no external stylesheet, script, font, or image, so the report keeps rendering offline and never signals a third party which skills a team evaluated. All model-authored and agent-authored text (task prompts, judge evidence, violation evidence) goes through html/template's default escaping.
type RunDrift ¶
type RunDrift struct {
Agent string `json:"agent"`
Model string `json:"model"`
Attempt int `json:"attempt"`
drift.Result
Violations []drift.Violation `json:"violations,omitempty"`
}
RunDrift is one run's drift measurement. It embeds drift.Result rather than duplicating Components/Adherence/Unevaluable as separate fields, so the per-task view rendered here and the per-member view drift.Aggregate consumes are always the same numbers, not two hand-built copies that can silently disagree.
Agent, alongside Model, is what disambiguates which panel member this run belongs to: a cross-vendor panel (e.g. --agents claude-code,cursor --models opus) can put two members on the same model string, and Model alone cannot tell them apart.
type TaskInput ¶
type TaskInput struct {
TaskID string
Kind string
Split string
Conditions []ConditionReport
Drift []RunDrift
Judge []JudgeNote
}
TaskInput is one task's raw results, before Compose excludes void tasks.
type TaskReport ¶
type TaskReport struct {
TaskID string `json:"task_id"`
Kind string `json:"kind"`
Split string `json:"split"`
Conditions []ConditionReport `json:"conditions,omitempty"`
Drift []RunDrift `json:"drift,omitempty"`
Judge []JudgeNote `json:"judge,omitempty"`
}
TaskReport is one non-void task's results.