score

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package score implements the LLM judge protocol: position-swapped pairwise comparisons and per-rule semantic scoring, both anchored on a worked rubric and both requiring a verbatim evidence quote before a verdict counts.

Agent CLIs expose no temperature, so determinism here is bought with protocol rather than sampling parameters: swap the candidates and treat disagreement as a tie (position bias is the largest known failure mode of pairwise LLM judging), require a quote that actually appears in the transcript (a verdict that cannot be checked is not evidence), and spend a third call only when the first two are close enough that one sample of a nondeterministic process is not enough to trust.

Index

Constants

View Source
const DefaultBorderlineMargin = 0.15

DefaultBorderlineMargin is the |margin| threshold below which a third vote is spent. Below it the judge is nearly indifferent, which is exactly where a single sample is worth least; above it, a third call re-confirms a decision already made and spends a session that could measure something else.

View Source
const EfficiencyFloor = 0.3

EfficiencyFloor bounds how far token bloat can pull the score down. At the 0.10 exponent a floored value costs about 11% — a warning, not a verdict, because a verbose skill that works still works.

View Source
const KappaFloor = 0.6

KappaFloor is the minimum Cohen's κ against the calibration set for the judge to be trusted with Uplift. Below it, the judge disagrees with a human often enough that its verdicts are not evidence, and Uplift falls back to the deterministic pass-rate delta instead.

Variables

View Source
var DefaultExponents = Exponents{Trigger: 0.35, Reliability: 0.35, Uplift: 0.20, Efficiency: 0.10}

DefaultExponents is the shipped distribution.

Trigger and Reliability carry 0.35 each because they are the two questions a quality score exists to answer: does the skill fire when it should, and does it work when it fires. Uplift is 0.20 — it is judge-derived, so it is the softest of the four. Efficiency is 0.10, enough to flag bloat and not enough to outrank working.

Uplift is deliberately *only* the judge win-rate. An earlier form averaged it with a pass-rate delta, but pass rate is already the whole of Reliability: that gave pass rate roughly 0.55 effective weight and put two different scales on one axis.

View Source
var ErrNoEvidence = errors.New("score: verdict has no evidence in the transcript")

ErrNoEvidence is returned when a verdict carries no evidence quote, or a quote that does not appear in either transcript. A verdict without a checkable quote is not weak evidence — it is no evidence, and this judge sets 20% of the final score.

Functions

func Bootstrap

func Bootstrap(samples []float64, iters int, seed int64) (lo, hi float64, err error)

Bootstrap computes a 95% percentile bootstrap confidence interval for the mean of samples, by resampling with replacement iters times and taking the 2.5th and 97.5th percentiles of the resampled means.

The source is seeded explicitly with math/rand.New(rand.NewSource(seed)) rather than a shared global generator, so that the same samples, iters, and seed always produce the same interval — a confidence interval that changes between two readings of the same data is not one anyone can quote.

Bootstrap sorts its own copy of samples before resampling, so callers do not need to guarantee arrival order themselves — a caller that built samples by appending from concurrent goroutines under a mutex, for instance, cannot promise a stable order, and rng.Intn(n) indexes by position, so a different order changes which value each draw picks even for the same seed. Sorting by value is the only stable key available at this layer (samples carries no identity beyond the numbers themselves).

func DetectFiring

func DetectFiring(skill string, caps agent.Caps, events []trajectory.Event) (fired, inferred bool)

DetectFiring reports whether the named skill fired anywhere in events, and whether that determination was inferred (from a read of the skill's SKILL.md) rather than explicit (from a Skill tool-call event naming it).

A distractor skill's own read or invocation never counts: DetectFiring only recognizes events that name skill, matched either directly (Name) or via a path whose directory component is the skill name — so reading a distractor's SKILL.md, which is correct behavior on a hard negative, never counts as skill firing.

func Effectiveness

func Effectiveness(p Pillars, e Exponents) (float64, error)

Effectiveness is the weighted geometric mean of the four pillars, ×100.

Geometric rather than arithmetic, and that is the whole design: no pillar can compensate for a zeroed one. A skill that never fires scores zero however reliable it is when someone invokes it by hand.

func Efficiency

func Efficiency(skillMedian, baselineMedian float64) (float64, error)

Efficiency compares median token spend against the baseline's. Spending less than the baseline is not rewarded: the pillar exists to catch bloat, and a bonus for terseness would push skills toward saying too little.

func Kappa

func Kappa(labels, verdicts []string) (float64, error)

Kappa is Cohen's κ between two raters over the same items.

κ rather than raw agreement because two raters who both answer "skill" most of the time agree most of the time by construction. κ subtracts that expected agreement, so it measures what the judge adds over guessing the base rate — which is the only question worth asking of a judge that sets 20% of a score.

κ = (po − pe) / (1 − pe)

func Median

func Median(xs []float64) (float64, error)

Median returns the median of xs, without mutating the caller's slice. Zero samples is an error, not a value: an absent measurement must not be indistinguishable from a real one.

func PassAtK

func PassAtK(n, c, k int) (float64, error)

PassAtK is the unbiased estimator of the probability that k independently drawn runs all pass, given c passes observed in n runs: C(c,k) / C(n,k).

Unbiased matters at these sample sizes. The obvious estimator — "did all k of the runs I have pass" — discards the information in the runs it does not look at and is biased downward at n=2, which is the Full tier. This form is the standard pass^k estimator and is exact for the sampling-without- replacement question actually being asked.

func Reliability

func Reliability(ts []TaskPasses, k int) (float64, error)

Reliability is the mean of PassAtK over tasks.

Mean over tasks, not over pooled runs: a task that happened to get more runs must not weigh more heavily than one that got fewer, or the score drifts with scheduling rather than with the skill.

func UpliftFromJudge

func UpliftFromJudge(vs []Verdict) (float64, error)

UpliftFromJudge is the pairwise win rate, ties counted as half.

func UpliftFromPassRates

func UpliftFromPassRates(skill, baseline float64) float64

UpliftFromPassRates is the documented degrade path for a judge whose κ is below the floor: a pass-rate delta mapped onto [0,1]. It is a worse measurement — it re-uses the signal Reliability already carries — which is why it travels with UpliftPassRate on the report rather than replacing the judge quietly.

Types

type CalDisagreement

type CalDisagreement struct {
	ID       string
	Label    string
	Verdict  string
	Evidence []string
}

CalDisagreement records one item where the judge's verdict did not match the human label. A κ with no examples attached is a number nobody can act on.

type CalItem

type CalItem struct {
	ID       string `yaml:"id"`
	TaskID   string `yaml:"task_id"`
	Prompt   string `yaml:"prompt"`
	Skill    string `yaml:"skill"`
	Baseline string `yaml:"baseline"`
	Label    string `yaml:"label"`
	Note     string `yaml:"note"`
}

CalItem is one labelled calibration pair: a task prompt, a transcript from a session with the skill available, a transcript from a baseline session without it, and a human verdict.

type CalResult

type CalResult struct {
	Kappa         float64
	Agreement     float64
	N             int
	Disagreements []CalDisagreement
	LabeledBy     string
}

CalResult is the outcome of running a judge over a calibration set.

func Calibrate

func Calibrate(ctx context.Context, j *Judge, set *CalSet) (CalResult, error)

Calibrate runs j.Pairwise over every item in set and reports the judge's agreement with the human labels, including every disagreement so a low κ is actionable rather than just alarming.

func (CalResult) JudgeTrusted

func (r CalResult) JudgeTrusted() bool

JudgeTrusted reports whether this calibration result clears the bar to trust the judge with Uplift. An unrun calibration (N == 0) is not trusted: defaulting to trust would make the whole gate opt-in, which is exactly how it gets forgotten.

type CalSet

type CalSet struct {
	LabeledBy string    `yaml:"labeled_by"`
	LabeledAt string    `yaml:"labeled_at"`
	Items     []CalItem `yaml:"items"`
}

CalSet is the calibration file: its items plus who labelled them and when. Provenance travels with the data because a κ computed against author labels is a weaker claim than one against independent labels.

func Calibration

func Calibration() (*CalSet, error)

Calibration reads and parses the embedded calibration set. KnownFields(true) makes an unrecognized YAML key a parse error rather than a silently dropped one — that field would otherwise disappear from every item without anyone noticing.

type Exponents

type Exponents struct {
	Trigger     float64
	Reliability float64
	Uplift      float64
	Efficiency  float64
}

Exponents weight the geometric mean. They sum to 1 so that a perfect skill scores exactly 100 — provided every pillar was actually measured. At the Smoke tier, trigger firing is not probed at all and Uplift has no baseline to compare against, so those two pillars default to their unmeasured neutral values (TriggerF1 = 1.0, Uplift = 0.5) rather than being scored; a flawless Smoke-tier skill tops out around 87, not 100.

type F1Result

type F1Result struct {
	Precision, Recall, F1 float64
	TP, FP, FN, TN        int
	Unknown               int
	// AnyInferred is true if any counted probe's Fired was an inferred
	// detection rather than an explicit one.
	AnyInferred bool
}

F1Result is the trigger precision/recall/F1 over a set of Probes, plus the confusion-matrix counts and provenance flags a report needs.

func TriggerF1

func TriggerF1(ps []Probe) (F1Result, error)

TriggerF1 computes trigger precision/recall/F1 over ps. Unknown probes are excluded from every quadrant. TriggerF1 errors if no measurable positive probe remains, since recall over zero positives is undefined and a measurement with nothing to fire on measures nothing.

type Judge

type Judge struct {
	// contains filtered or unexported fields
}

Judge runs the pairwise and semantic judging protocol against one skill spec's rubric.

func NewJudge

func NewJudge(o JudgeOptions) (*Judge, error)

NewJudge validates options and returns a Judge. A Gateway and a Spec are both required: the rubric is derived from the spec's purpose, and there is nothing to call without a gateway.

func (*Judge) Pairwise

func (j *Judge) Pairwise(ctx context.Context, p Pair) (Verdict, error)

Pairwise compares a skill run against a baseline run of the same task, following the position-swap-and-borderline-revote protocol described on the package doc.

func (*Judge) Semantic

func (j *Judge) Semantic(ctx context.Context, r contract.SemanticRule, s Sample) (float64, []string, error)

Semantic scores one sample against one semantic rule: 1.0 if satisfied and citable, 0.0 otherwise. An unsupported "yes" — satisfied with no quote that actually appears in the transcript — is not evidence of adherence, so it scores 0 rather than 1.

type JudgeOptions

type JudgeOptions struct {
	Gateway llm.Gateway
	Spec    *spec.SkillSpec
	// BorderlineMargin defaults to DefaultBorderlineMargin when zero.
	BorderlineMargin float64
	// MaxVotes caps how many pairwise calls a single Pairwise verdict may spend.
	// Defaults to 3 when zero. The protocol only ever wants a third call when
	// the first two are agreeing but borderline; setting MaxVotes to 2
	// disables that third call entirely, capping every verdict at the swap.
	MaxVotes int
}

JudgeOptions configures a Judge.

type Matrix

type Matrix struct{ Entries []PanelEntry }

Matrix is every member's measured result. Deliberately not called Panel: runner.Panel is the planned list of members, this is what they produced, and one name for both across adjacent packages is how a boundary defect starts.

func (Matrix) ByClass

func (m Matrix) ByClass(c spec.ModelTier) (PanelEntry, bool)

ByClass returns the single healthy entry in Entries whose Member.Class matches c, and whether exactly one was found.

ParsePanel can produce several members of one class on a multi-agent, multi-model panel — it only rejects a panel with zero of a class, not more than one. ByClass reports ok == false for zero matches, more than one, and an unhealthy match: with several members of one capability tier, "the strong member" and "the floor member" are not defined, so a robustness gap computed by comparing them is not defined either; an unhealthy member was never measured at all, so returning it would let a caller read its zero Drift as a real (adherence-zero) result rather than an absent one — the same distinction Headline draws when no member is healthy. Returning whichever member happened to be first in Entries would produce a number indistinguishable from a real one — an absent measurement must be reported as absent, not silently guessed at. A caller that wants a specific member out of a duplicated class should iterate m.Entries directly and say which member it means, rather than asking ByClass to disambiguate for it.

func (Matrix) Headline

func (m Matrix) Headline() (float64, error)

Headline is the minimum Effectiveness across healthy members.

Minimum rather than mean: the claim a published score makes is "this works", and it only works if it works on the weakest model someone will run it on. A member whose adapter failed is excluded rather than scored zero — otherwise an expired token becomes a publish block, which is infrastructure flakiness presented as a quality verdict.

func (Matrix) Mean

func (m Matrix) Mean() (float64, error)

Mean is the arithmetic mean of Effectiveness across healthy members. Unlike Headline it is not the gating statistic — it is report sugar for "how did the panel do on average", which Headline's min-gate deliberately does not answer. As with Headline, an unhealthy member is excluded rather than scored zero, and a panel with no healthy member is an error.

type Member

type Member struct {
	Agent string
	Model string
	Class spec.ModelTier
}

Member identifies one panel entry. score defines its own rather than importing the runner's: scoring must be usable against recorded results with no orchestration in the picture.

type Pair

type Pair struct {
	TaskID   string
	Prompt   string
	Skill    Sample
	Baseline Sample
}

Pair is one task run under both conditions: with the skill available, and without (baseline).

type PanelEntry

type PanelEntry struct {
	Member        Member
	Pillars       Pillars
	Effectiveness float64
	Drift         drift.Agg
	Grade         string
	// Healthy is false when the member's adapter failed its probe. Such a member
	// contributes nothing rather than a zero.
	Healthy bool
	Detail  string
}

PanelEntry is one member's complete result.

type Pillars

type Pillars struct {
	TriggerF1   float64
	Reliability float64
	Uplift      float64
	Efficiency  float64
}

Pillars are the four rates Effectiveness composes, each in [0,1].

func (Pillars) Validate

func (p Pillars) Validate() error

Validate reports whether every pillar is a rate in [0,1]. Effectiveness calls it before computing anything; a caller that needs to check judge-derived or otherwise unhealthy-sourced pillars before they reach a report should call it directly rather than duplicating the range check.

type Probe

type Probe struct {
	Prompt   string
	Positive bool
	Fired    bool
	Inferred bool
	Unknown  bool
	Reason   string
}

Probe is one trigger measurement: did the skill fire on this prompt, and was that prompt one the skill should have fired on (Positive) or a distractor it should have ignored?

Inferred records how Fired was decided: true when it came from a read of SKILL.md rather than an explicit invocation event. The two are not the same measurement — a read cannot distinguish "consulted" from "used" — which is why F1Result carries the flag forward rather than silently mixing them, mirroring the event_source discipline in the analytics path.

Unknown marks a probe whose session could not be measured (e.g. it errored). It is excluded from every quadrant rather than counted as a miss, so infrastructure failures don't masquerade as recall failures.

type Sample

type Sample struct {
	Label      string
	Transcript string
	Outputs    string
}

Sample is one candidate's run on a task: what it did (Transcript) and what it produced (Outputs).

type TaskPasses

type TaskPasses struct {
	TaskID  string
	N       int
	C       int
	Errored int
}

TaskPasses is one task's outcome across its runs: C passes out of N.

Errored counts runs that could not be measured at all — a rate-limit exhaustion or an agent-reported internal error, store.StatusError in the runner's terms — and is deliberately excluded from N: an errored run is neither a pass nor a fail, the same reasoning score.Probe.Unknown applies on the trigger path so infrastructure failures don't masquerade as reliability failures. A task whose runs all errored (N==0, Errored>0) has produced no measurement whatsoever; callers should treat it the way report.VoidTask already treats a task excluded from scoring — dropped from the denominator and listed separately — rather than as a task that failed every run.

func (TaskPasses) Void

func (t TaskPasses) Void() bool

Void reports whether t produced no measurement at all: every run errored, so there is nothing to compute a pass rate from. A caller building the []TaskPasses fed to Reliability must exclude such a task from that slice — the same way report.Compose excludes a report.VoidTask from the tasks it scores — rather than let it reach PassAtK, which refuses N==0.

This is a different exclusion from report.VoidTask, which is a task the oracle gate refused before any run was ever attempted. The two share the same underlying behaviour (drop from the scored denominator, don't count as a failure) but not, today, the same visibility: a report.VoidTask is listed on Report.VoidTasks with its reason, while a task that goes Void here (every run errored) is currently dropped by cli/whetstone's taskPasses with no corresponding report-level record — a reader sees a missing task, not a labelled one. Distinguishing the two kinds in the report is future work; for now, know that "void" means two different things depending on which of these two types you're looking at.

type UpliftSource

type UpliftSource string

UpliftSource records which measurement produced Uplift, so a report never presents the fallback as though it were the judge.

const (
	UpliftJudge    UpliftSource = "judge"
	UpliftPassRate UpliftSource = "passrate-fallback"
)

type Verdict

type Verdict struct {
	Winner   string
	Margin   float64
	Evidence []string
	Votes    int
	Swapped  bool
	Tie      bool
}

Verdict is the outcome of a pairwise comparison. Winner is "skill", "baseline", or "tie".

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL