score

package
v0.15.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 17, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DetectFiring

func DetectFiring(skill string, caps agent.Caps, events []agent.Event) (fired, inferred bool)

DetectFiring reports whether skill fired in events, and whether the detection was inferred (a SKILL.md read) rather than explicit (a Skill tool call).

func EvalRate added in v0.11.0

func EvalRate(gs []Grade) (float64, error)

EvalRate is one eval's score across its runs: the mean of the per-run pass rates.

Mean rather than median. At two or three runs a median cannot tell "passed twice of three" from "passed once of three", which is the flakiness the repeat runs exist to find.

func Median

func Median(xs []float64) (float64, error)

Median returns the median of xs, without mutating the caller's slice. Zero samples is an error, not a value: an absent measurement must not be indistinguishable from a real one.

func MedianTokens added in v0.11.0

func MedianTokens(totals []int64) (int64, error)

MedianTokens is the median total token spend across a set of sessions. Reported beside the score rather than inside it: a verbose skill that works still works, and nobody notices a skill tripling its own token bill unless the figure is on the report.

func MemberScore added in v0.11.0

func MemberScore(evalRates []float64) (float64, error)

MemberScore is one panel member's 0–100 score: the mean of its evals' rates.

The mean over evals rather than over pooled expectations. Pooling weighs an eval by how many expectations it happens to carry, so five easy statements added to an easy eval would raise the score without improving the skill.

Types

type Expectation added in v0.11.0

type Expectation struct {
	Text     string `json:"text"`
	Passed   bool   `json:"passed"`
	Evidence string `json:"evidence"`
}

Expectation is one graded statement from an eval's expectations list.

type F1Result

type F1Result struct {
	Precision, Recall, F1 float64
	TP, FP, FN, TN        int
	Unknown               int
	// AnyInferred is true if any counted probe's Fired was an inferred
	// detection rather than an explicit one.
	AnyInferred bool
}

F1Result is the trigger confusion matrix and F1.

func TriggerF1

func TriggerF1(ps []Probe) (F1Result, error)

TriggerF1 computes trigger precision/recall/F1 over ps. Errors if no measurable positive probe remains.

type Grade added in v0.11.0

type Grade struct {
	Expectations []Expectation `json:"expectations"`
	Passed       int           `json:"passed"`
	Total        int           `json:"total"`
	// Model is the model that produced this grade, as the gateway named it in
	// the answer. It is empty when the gateway names none. A gateway that can
	// name its model in advance is read through ModelFor instead, so this
	// field carries the subscription case, where nothing else records a judge.
	Model string `json:"model,omitempty"`
}

Grade is one agent session's expectations, graded.

func (Grade) Rate added in v0.11.0

func (g Grade) Rate() float64

Rate is the fraction of this session's expectations that passed.

type Grader added in v0.11.0

type Grader struct {
	// contains filtered or unexported fields
}

Grader marks an eval's expectations against one session.

func NewGrader added in v0.11.0

func NewGrader(g llm.Gateway) (*Grader, error)

NewGrader returns a Grader backed by g.

func (*Grader) Grade added in v0.11.0

func (g *Grader) Grade(ctx context.Context, expectations []string, r Run) (Grade, error)

Grade marks every expectation against r in one call.

One call rather than one per expectation: the evidence for each verdict is the same transcript, and a per-expectation call multiplies the cost of a score by the length of its longest expectations list.

type Probe

type Probe struct {
	Prompt   string
	Positive bool
	Fired    bool
	Inferred bool
	Unknown  bool
	Reason   string
}

Probe is one trigger measurement. Inferred is true when Fired came from a read of SKILL.md rather than an explicit invocation. Unknown probes are excluded from every quadrant.

type Run added in v0.11.0

type Run struct {
	Prompt         string
	ExpectedOutput string
	Transcript     string
	// Outputs is a rendering of the files the session produced. The grader is
	// told to fail an expectation it can only confirm from the agent's own
	// account, so a session with no outputs rendered here grades harder.
	Outputs string
}

Run is one agent session, as the grader sees it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL