Documentation
¶
Index ¶
- func DetectFiring(skill string, caps agent.Caps, events []agent.Event) (fired, inferred bool)
- func EvalRate(gs []Grade) (float64, error)
- func Median(xs []float64) (float64, error)
- func MedianTokens(totals []int64) (int64, error)
- func MemberScore(evalRates []float64) (float64, error)
- type Expectation
- type F1Result
- type Grade
- type Grader
- type Probe
- type Run
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DetectFiring ¶
DetectFiring reports whether skill fired in events, and whether the detection was inferred (a SKILL.md read) rather than explicit (a Skill tool call).
func EvalRate ¶ added in v0.11.0
EvalRate is one eval's score across its runs: the mean of the per-run pass rates.
Mean rather than median. At two or three runs a median cannot tell "passed twice of three" from "passed once of three", which is the flakiness the repeat runs exist to find.
func Median ¶
Median returns the median of xs, without mutating the caller's slice. Zero samples is an error, not a value: an absent measurement must not be indistinguishable from a real one.
func MedianTokens ¶ added in v0.11.0
MedianTokens is the median total token spend across a set of sessions. Reported beside the score rather than inside it: a verbose skill that works still works, and nobody notices a skill tripling its own token bill unless the figure is on the report.
func MemberScore ¶ added in v0.11.0
MemberScore is one panel member's 0–100 score: the mean of its evals' rates.
The mean over evals rather than over pooled expectations. Pooling weighs an eval by how many expectations it happens to carry, so five easy statements added to an easy eval would raise the score without improving the skill.
Types ¶
type Expectation ¶ added in v0.11.0
type Expectation struct {
Text string `json:"text"`
Passed bool `json:"passed"`
Evidence string `json:"evidence"`
}
Expectation is one graded statement from an eval's expectations list.
type F1Result ¶
type F1Result struct {
Precision, Recall, F1 float64
TP, FP, FN, TN int
Unknown int
// AnyInferred is true if any counted probe's Fired was an inferred
// detection rather than an explicit one.
AnyInferred bool
}
F1Result is the trigger confusion matrix and F1.
type Grade ¶ added in v0.11.0
type Grade struct {
Expectations []Expectation `json:"expectations"`
Passed int `json:"passed"`
Total int `json:"total"`
// Model is the model that produced this grade, as the gateway named it in
// the answer. It is empty when the gateway names none. A gateway that can
// name its model in advance is read through ModelFor instead, so this
// field carries the subscription case, where nothing else records a judge.
Model string `json:"model,omitempty"`
}
Grade is one agent session's expectations, graded.
type Grader ¶ added in v0.11.0
type Grader struct {
// contains filtered or unexported fields
}
Grader marks an eval's expectations against one session.
type Probe ¶
type Probe struct {
Prompt string
Positive bool
Fired bool
Inferred bool
Unknown bool
Reason string
}
Probe is one trigger measurement. Inferred is true when Fired came from a read of SKILL.md rather than an explicit invocation. Unknown probes are excluded from every quadrant.
type Run ¶ added in v0.11.0
type Run struct {
Prompt string
ExpectedOutput string
Transcript string
// Outputs is a rendering of the files the session produced. The grader is
// told to fail an expectation it can only confirm from the agent's own
// account, so a session with no outputs rendered here grades harder.
Outputs string
}
Run is one agent session, as the grader sees it.