Documentation
¶
Overview ¶
Package score implements the LLM judge protocol: position-swapped pairwise comparisons and per-rule semantic scoring, both anchored on a worked rubric and both requiring a verbatim evidence quote before a verdict counts.
Agent CLIs expose no temperature, so determinism here is bought with protocol rather than sampling parameters: swap the candidates and treat disagreement as a tie (position bias is the largest known failure mode of pairwise LLM judging), require a quote that actually appears in the transcript (a verdict that cannot be checked is not evidence), and spend a third call only when the first two are close enough that one sample of a nondeterministic process is not enough to trust.
Index ¶
- Constants
- Variables
- func Bootstrap(samples []float64, iters int, seed int64) (lo, hi float64, err error)
- func DetectFiring(skill string, caps agent.Caps, events []trajectory.Event) (fired, inferred bool)
- func Effectiveness(p Pillars, e Exponents) (float64, error)
- func Efficiency(skillMedian, baselineMedian float64) (float64, error)
- func Kappa(labels, verdicts []string) (float64, error)
- func Median(xs []float64) (float64, error)
- func PassAtK(n, c, k int) (float64, error)
- func Reliability(ts []TaskPasses, k int) (float64, error)
- func UpliftFromJudge(vs []Verdict) (float64, error)
- func UpliftFromPassRates(skill, baseline float64) float64
- type CalDisagreement
- type CalItem
- type CalResult
- type CalSet
- type Exponents
- type F1Result
- type Judge
- type JudgeOptions
- type Matrix
- type Member
- type Pair
- type PanelEntry
- type Pillars
- type Probe
- type Sample
- type TaskPasses
- type UpliftSource
- type Verdict
Constants ¶
const DefaultBorderlineMargin = 0.15
DefaultBorderlineMargin is the |margin| threshold below which a third vote is spent. Below it the judge is nearly indifferent, which is exactly where a single sample is worth least; above it, a third call re-confirms a decision already made and spends a session that could measure something else.
const EfficiencyFloor = 0.3
EfficiencyFloor bounds how far token bloat can pull the score down. At the 0.10 exponent a floored value costs about 11% — a warning, not a verdict, because a verbose skill that works still works.
const KappaFloor = 0.6
KappaFloor is the minimum Cohen's κ against the calibration set for the judge to be trusted with Uplift. Below it, the judge disagrees with a human often enough that its verdicts are not evidence, and Uplift falls back to the deterministic pass-rate delta instead.
Variables ¶
var DefaultExponents = Exponents{Trigger: 0.35, Reliability: 0.35, Uplift: 0.20, Efficiency: 0.10}
DefaultExponents is the shipped distribution.
Trigger and Reliability carry 0.35 each because they are the two questions a quality score exists to answer: does the skill fire when it should, and does it work when it fires. Uplift is 0.20 — it is judge-derived, so it is the softest of the four. Efficiency is 0.10, enough to flag bloat and not enough to outrank working.
Uplift is deliberately *only* the judge win-rate. An earlier form averaged it with a pass-rate delta, but pass rate is already the whole of Reliability: that gave pass rate roughly 0.55 effective weight and put two different scales on one axis.
var ErrNoEvidence = errors.New("score: verdict has no evidence in the transcript")
ErrNoEvidence is returned when a verdict carries no evidence quote, or a quote that does not appear in either transcript. A verdict without a checkable quote is not weak evidence — it is no evidence, and this judge sets 20% of the final score.
Functions ¶
func Bootstrap ¶
Bootstrap computes a 95% percentile bootstrap confidence interval for the mean of samples, by resampling with replacement iters times and taking the 2.5th and 97.5th percentiles of the resampled means.
The source is seeded explicitly with math/rand.New(rand.NewSource(seed)) rather than a shared global generator, so that the same samples, iters, and seed always produce the same interval — a confidence interval that changes between two readings of the same data is not one anyone can quote.
Bootstrap sorts its own copy of samples before resampling, so callers do not need to guarantee arrival order themselves — a caller that built samples by appending from concurrent goroutines under a mutex, for instance, cannot promise a stable order, and rng.Intn(n) indexes by position, so a different order changes which value each draw picks even for the same seed. Sorting by value is the only stable key available at this layer (samples carries no identity beyond the numbers themselves).
func DetectFiring ¶
DetectFiring reports whether the named skill fired anywhere in events, and whether that determination was inferred (from a read of the skill's SKILL.md) rather than explicit (from a Skill tool-call event naming it).
A distractor skill's own read or invocation never counts: DetectFiring only recognizes events that name skill, matched either directly (Name) or via a path whose directory component is the skill name — so reading a distractor's SKILL.md, which is correct behavior on a hard negative, never counts as skill firing.
func Effectiveness ¶
Effectiveness is the weighted geometric mean of the four pillars, ×100.
Geometric rather than arithmetic, and that is the whole design: no pillar can compensate for a zeroed one. A skill that never fires scores zero however reliable it is when someone invokes it by hand.
func Efficiency ¶
Efficiency compares median token spend against the baseline's. Spending less than the baseline is not rewarded: the pillar exists to catch bloat, and a bonus for terseness would push skills toward saying too little.
func Kappa ¶
Kappa is Cohen's κ between two raters over the same items.
κ rather than raw agreement because two raters who both answer "skill" most of the time agree most of the time by construction. κ subtracts that expected agreement, so it measures what the judge adds over guessing the base rate — which is the only question worth asking of a judge that sets 20% of a score.
κ = (po − pe) / (1 − pe)
func Median ¶
Median returns the median of xs, without mutating the caller's slice. Zero samples is an error, not a value: an absent measurement must not be indistinguishable from a real one.
func PassAtK ¶
PassAtK is the unbiased estimator of the probability that k independently drawn runs all pass, given c passes observed in n runs: C(c,k) / C(n,k).
Unbiased matters at these sample sizes. The obvious estimator — "did all k of the runs I have pass" — discards the information in the runs it does not look at and is biased downward at n=2, which is the Full tier. This form is the standard pass^k estimator and is exact for the sampling-without- replacement question actually being asked.
func Reliability ¶
func Reliability(ts []TaskPasses, k int) (float64, error)
Reliability is the mean of PassAtK over tasks.
Mean over tasks, not over pooled runs: a task that happened to get more runs must not weigh more heavily than one that got fewer, or the score drifts with scheduling rather than with the skill.
func UpliftFromJudge ¶
UpliftFromJudge is the pairwise win rate, ties counted as half.
func UpliftFromPassRates ¶
UpliftFromPassRates is the documented degrade path for a judge whose κ is below the floor: a pass-rate delta mapped onto [0,1]. It is a worse measurement — it re-uses the signal Reliability already carries — which is why it travels with UpliftPassRate on the report rather than replacing the judge quietly.
Types ¶
type CalDisagreement ¶
CalDisagreement records one item where the judge's verdict did not match the human label. A κ with no examples attached is a number nobody can act on.
type CalItem ¶
type CalItem struct {
ID string `yaml:"id"`
TaskID string `yaml:"task_id"`
Prompt string `yaml:"prompt"`
Skill string `yaml:"skill"`
Baseline string `yaml:"baseline"`
Label string `yaml:"label"`
Note string `yaml:"note"`
}
CalItem is one labelled calibration pair: a task prompt, a transcript from a session with the skill available, a transcript from a baseline session without it, and a human verdict.
type CalResult ¶
type CalResult struct {
Kappa float64
Agreement float64
N int
Disagreements []CalDisagreement
LabeledBy string
}
CalResult is the outcome of running a judge over a calibration set.
func Calibrate ¶
Calibrate runs j.Pairwise over every item in set and reports the judge's agreement with the human labels, including every disagreement so a low κ is actionable rather than just alarming.
func (CalResult) JudgeTrusted ¶
JudgeTrusted reports whether this calibration result clears the bar to trust the judge with Uplift. An unrun calibration (N == 0) is not trusted: defaulting to trust would make the whole gate opt-in, which is exactly how it gets forgotten.
type CalSet ¶
type CalSet struct {
LabeledBy string `yaml:"labeled_by"`
LabeledAt string `yaml:"labeled_at"`
Items []CalItem `yaml:"items"`
}
CalSet is the calibration file: its items plus who labelled them and when. Provenance travels with the data because a κ computed against author labels is a weaker claim than one against independent labels.
func Calibration ¶
Calibration reads and parses the embedded calibration set. KnownFields(true) makes an unrecognized YAML key a parse error rather than a silently dropped one — that field would otherwise disappear from every item without anyone noticing.
type Exponents ¶
Exponents weight the geometric mean. They sum to 1 so that a perfect skill scores exactly 100 — provided every pillar was actually measured. At the Smoke tier, trigger firing is not probed at all and Uplift has no baseline to compare against, so those two pillars default to their unmeasured neutral values (TriggerF1 = 1.0, Uplift = 0.5) rather than being scored; a flawless Smoke-tier skill tops out around 87, not 100.
type F1Result ¶
type F1Result struct {
Precision, Recall, F1 float64
TP, FP, FN, TN int
Unknown int
// AnyInferred is true if any counted probe's Fired was an inferred
// detection rather than an explicit one.
AnyInferred bool
}
F1Result is the trigger precision/recall/F1 over a set of Probes, plus the confusion-matrix counts and provenance flags a report needs.
type Judge ¶
type Judge struct {
// contains filtered or unexported fields
}
Judge runs the pairwise and semantic judging protocol against one skill spec's rubric.
func NewJudge ¶
func NewJudge(o JudgeOptions) (*Judge, error)
NewJudge validates options and returns a Judge. A Gateway and a Spec are both required: the rubric is derived from the spec's purpose, and there is nothing to call without a gateway.
func (*Judge) Pairwise ¶
Pairwise compares a skill run against a baseline run of the same task, following the position-swap-and-borderline-revote protocol described on the package doc.
func (*Judge) Semantic ¶
func (j *Judge) Semantic(ctx context.Context, r contract.SemanticRule, s Sample) (float64, []string, error)
Semantic scores one sample against one semantic rule: 1.0 if satisfied and citable, 0.0 otherwise. An unsupported "yes" — satisfied with no quote that actually appears in the transcript — is not evidence of adherence, so it scores 0 rather than 1.
type JudgeOptions ¶
type JudgeOptions struct {
Gateway llm.Gateway
Spec *spec.SkillSpec
// BorderlineMargin defaults to DefaultBorderlineMargin when zero.
BorderlineMargin float64
// MaxVotes caps how many pairwise calls a single Pairwise verdict may spend.
// Defaults to 3 when zero. The protocol only ever wants a third call when
// the first two are agreeing but borderline; setting MaxVotes to 2
// disables that third call entirely, capping every verdict at the swap.
MaxVotes int
}
JudgeOptions configures a Judge.
type Matrix ¶
type Matrix struct{ Entries []PanelEntry }
Matrix is every member's measured result. Deliberately not called Panel: runner.Panel is the planned list of members, this is what they produced, and one name for both across adjacent packages is how a boundary defect starts.
func (Matrix) ByClass ¶
func (m Matrix) ByClass(c spec.ModelTier) (PanelEntry, bool)
ByClass returns the single healthy entry in Entries whose Member.Class matches c, and whether exactly one was found.
ParsePanel can produce several members of one class on a multi-agent, multi-model panel — it only rejects a panel with zero of a class, not more than one. ByClass reports ok == false for zero matches, more than one, and an unhealthy match: with several members of one capability tier, "the strong member" and "the floor member" are not defined, so a robustness gap computed by comparing them is not defined either; an unhealthy member was never measured at all, so returning it would let a caller read its zero Drift as a real (adherence-zero) result rather than an absent one — the same distinction Headline draws when no member is healthy. Returning whichever member happened to be first in Entries would produce a number indistinguishable from a real one — an absent measurement must be reported as absent, not silently guessed at. A caller that wants a specific member out of a duplicated class should iterate m.Entries directly and say which member it means, rather than asking ByClass to disambiguate for it.
func (Matrix) Headline ¶
Headline is the minimum Effectiveness across healthy members.
Minimum rather than mean: the claim a published score makes is "this works", and it only works if it works on the weakest model someone will run it on. A member whose adapter failed is excluded rather than scored zero — otherwise an expired token becomes a publish block, which is infrastructure flakiness presented as a quality verdict.
func (Matrix) Mean ¶
Mean is the arithmetic mean of Effectiveness across healthy members. Unlike Headline it is not the gating statistic — it is report sugar for "how did the panel do on average", which Headline's min-gate deliberately does not answer. As with Headline, an unhealthy member is excluded rather than scored zero, and a panel with no healthy member is an error.
type Member ¶
Member identifies one panel entry. score defines its own rather than importing the runner's: scoring must be usable against recorded results with no orchestration in the picture.
type Pair ¶
Pair is one task run under both conditions: with the skill available, and without (baseline).
type PanelEntry ¶
type PanelEntry struct {
Member Member
Pillars Pillars
Effectiveness float64
Drift drift.Agg
Grade string
// Healthy is false when the member's adapter failed its probe. Such a member
// contributes nothing rather than a zero.
Healthy bool
Detail string
}
PanelEntry is one member's complete result.
type Pillars ¶
Pillars are the four rates Effectiveness composes, each in [0,1].
func (Pillars) Validate ¶
Validate reports whether every pillar is a rate in [0,1]. Effectiveness calls it before computing anything; a caller that needs to check judge-derived or otherwise unhealthy-sourced pillars before they reach a report should call it directly rather than duplicating the range check.
type Probe ¶
type Probe struct {
Prompt string
Positive bool
Fired bool
Inferred bool
Unknown bool
Reason string
}
Probe is one trigger measurement: did the skill fire on this prompt, and was that prompt one the skill should have fired on (Positive) or a distractor it should have ignored?
Inferred records how Fired was decided: true when it came from a read of SKILL.md rather than an explicit invocation event. The two are not the same measurement — a read cannot distinguish "consulted" from "used" — which is why F1Result carries the flag forward rather than silently mixing them, mirroring the event_source discipline in the analytics path.
Unknown marks a probe whose session could not be measured (e.g. it errored). It is excluded from every quadrant rather than counted as a miss, so infrastructure failures don't masquerade as recall failures.
type Sample ¶
Sample is one candidate's run on a task: what it did (Transcript) and what it produced (Outputs).
type TaskPasses ¶
TaskPasses is one task's outcome across its runs: C passes out of N.
Errored counts runs that could not be measured at all — a rate-limit exhaustion or an agent-reported internal error, store.StatusError in the runner's terms — and is deliberately excluded from N: an errored run is neither a pass nor a fail, the same reasoning score.Probe.Unknown applies on the trigger path so infrastructure failures don't masquerade as reliability failures. A task whose runs all errored (N==0, Errored>0) has produced no measurement whatsoever; callers should treat it the way report.VoidTask already treats a task excluded from scoring — dropped from the denominator and listed separately — rather than as a task that failed every run.
func (TaskPasses) Void ¶
func (t TaskPasses) Void() bool
Void reports whether t produced no measurement at all: every run errored, so there is nothing to compute a pass rate from. A caller building the []TaskPasses fed to Reliability must exclude such a task from that slice — the same way report.Compose excludes a report.VoidTask from the tasks it scores — rather than let it reach PassAtK, which refuses N==0.
This is a different exclusion from report.VoidTask, which is a task the oracle gate refused before any run was ever attempted. The two share the same underlying behaviour (drop from the scored denominator, don't count as a failure) but not, today, the same visibility: a report.VoidTask is listed on Report.VoidTasks with its reason, while a task that goes Void here (every run errored) is currently dropped by cli/whetstone's taskPasses with no corresponding report-level record — a reader sees a missing task, not a labelled one. Distinguishing the two kinds in the report is future work; for now, know that "void" means two different things depending on which of these two types you're looking at.
type UpliftSource ¶
type UpliftSource string
UpliftSource records which measurement produced Uplift, so a report never presents the fallback as though it were the judge.
const ( UpliftJudge UpliftSource = "judge" UpliftPassRate UpliftSource = "passrate-fallback" )