Documentation
¶
Overview ¶
Package suite is a skill's eval set: the evals/evals.json tasks it is measured against, the evals/triggers.json queries its description is measured against, and a shipped distractor pack that makes a trigger measurement discriminating.
Index ¶
- Constants
- func Generate(ctx context.Context, gw llm.Gateway, sp *spec.SkillSpec, n int) (*EvalSet, []TriggerQuery, error)
- func GenerateFrom(ctx context.Context, gw llm.Gateway, specs []*spec.SkillSpec, n int) (*EvalSet, []TriggerQuery, error)
- func Ref(dir string) (string, error)
- func ShortRef(ref string) string
- func WriteEvalSet(dir string, set *EvalSet) error
- func WriteTriggerQueries(dir string, qs []TriggerQuery) error
- type Distractor
- type Eval
- type EvalCheck
- type EvalSet
- type TriggerQuery
Constants ¶
const EvalsDir = "evals"
EvalsDir is the eval set's directory name, at the skill root. It is Anthropic's layout, unchanged, so a skill directory moves between skael and skill-creator without a translation step.
Variables ¶
This section is empty.
Functions ¶
func Generate ¶
func Generate(ctx context.Context, gw llm.Gateway, sp *spec.SkillSpec, n int) (*EvalSet, []TriggerQuery, error)
Generate drafts an eval set of n evals from an approved spec, and derives the trigger queries from the spec's own trigger phrases.
One call rather than one per eval: an eval is a prompt and a handful of statements, so the whole set fits in one response, and drafting them together is what stops n evals from being n rewordings of the same one.
func GenerateFrom ¶ added in v0.14.0
func GenerateFrom(ctx context.Context, gw llm.Gateway, specs []*spec.SkillSpec, n int) (*EvalSet, []TriggerQuery, error)
GenerateFrom drafts one eval set covering every spec it is given.
A contest derives from all its candidates: a suite derived from one of them grades the others against that candidate's claims, which is the bias the contest exists to remove. A task only one candidate claims stays in — losing it is a real result.
func Ref ¶
Ref is the content hash of a written suite: every file's relative path and contents, in sorted order.
It exists so that "the same suite" is a computed fact rather than an assumption. A score is only comparable to another score measured against the same tasks, and a trend that silently mixes suites is worse than no trend at all — so a report carries this value and two reports can be asked whether they are comparable.
func WriteEvalSet ¶ added in v0.11.0
WriteEvalSet writes evals/evals.json under dir.
func WriteTriggerQueries ¶ added in v0.11.0
func WriteTriggerQueries(dir string, qs []TriggerQuery) error
WriteTriggerQueries writes evals/triggers.json under dir.
Types ¶
type Distractor ¶
type Distractor struct {
Name string `yaml:"name"`
Description string `yaml:"description"`
// Tier is "near", "mid", or "far".
Tier string `yaml:"tier"`
}
Distractor is one synthetic skill for trigger-precision measurement.
func Distractors ¶
func Distractors() ([]Distractor, error)
Distractors returns the shipped synthetic distractor pack.
type Eval ¶ added in v0.11.0
type Eval struct {
ID int `json:"id"`
Prompt string `json:"prompt"`
ExpectedOutput string `json:"expected_output,omitempty"`
Files []string `json:"files,omitempty"`
Expectations []string `json:"expectations,omitempty"`
}
Eval is one task the skill must handle, from evals/evals.json.
func (*Eval) UnmarshalJSON ¶ added in v0.11.0
UnmarshalJSON reads `assertions` as an alias for `expectations`.
Anthropic contradicts itself: references/schemas.md and grading.json say expectations, SKILL.md says assertions. We write expectations only, and accept both, so a file written by skill-creator loads either way.
type EvalCheck ¶ added in v0.11.0
EvalCheck is one eval's static validation result. It replaces the oracle gate: with no verifier script to prove correct, what remains to check is that the eval can be run and can be scored at all.
type EvalSet ¶ added in v0.11.0
EvalSet is the whole of evals/evals.json.
func LoadEvalSet ¶ added in v0.11.0
LoadEvalSet reads evals/evals.json from dir.
type TriggerQuery ¶ added in v0.11.0
TriggerQuery is one entry of evals/triggers.json: a prompt, and whether the skill must fire on it.
The file is a bare array rather than an object, which is the shape Anthropic's run_eval.py takes through --eval-set. Anthropic never names the file, so the name is ours and the contents are theirs.
func LoadTriggerQueries ¶ added in v0.11.0
func LoadTriggerQueries(dir string) ([]TriggerQuery, error)
LoadTriggerQueries reads evals/triggers.json from dir. A missing file is not an error: a skill can carry evals and no trigger set.
func TriggersFromSpec ¶ added in v0.11.0
func TriggersFromSpec(sp *spec.SkillSpec) []TriggerQuery
TriggersFromSpec derives the trigger queries from an authored spec.