suite

package
v0.21.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 10, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package suite is a skill's eval set: the evals/evals.json tasks it is measured against, the evals/triggers.json queries its description is measured against, and a shipped distractor pack that makes a trigger measurement discriminating.

Index

Constants

View Source
const EvalsDir = "evals"

EvalsDir is the eval set's directory name, at the skill root. It is Anthropic's layout, unchanged, so a skill directory moves between skael and skill-creator without a translation step.

Variables

This section is empty.

Functions

func Generate

func Generate(ctx context.Context, gw llm.Gateway, sp *spec.SkillSpec, n int) (*EvalSet, []TriggerQuery, error)

Generate drafts an eval set of n evals from an approved spec, and derives the trigger queries from the spec's own trigger phrases.

One call rather than one per eval: an eval is a prompt and a handful of statements, so the whole set fits in one response, and drafting them together is what stops n evals from being n rewordings of the same one.

func GenerateFrom added in v0.14.0

func GenerateFrom(ctx context.Context, gw llm.Gateway, specs []*spec.SkillSpec, n int) (*EvalSet, []TriggerQuery, error)

GenerateFrom drafts one eval set covering every spec it is given.

A contest derives from all its candidates: a suite derived from one of them grades the others against that candidate's claims, which is the bias the contest exists to remove. A task only one candidate claims stays in — losing it is a real result.

func Ref

func Ref(dir string) (string, error)

Ref is the content hash of a written suite: every file's relative path and contents, in sorted order.

It exists so that "the same suite" is a computed fact rather than an assumption. A score is only comparable to another score measured against the same tasks, and a trend that silently mixes suites is worse than no trend at all — so a report carries this value and two reports can be asked whether they are comparable.

func ShortRef

func ShortRef(ref string) string

ShortRef abbreviates a ref for display.

func WriteEvalSet added in v0.11.0

func WriteEvalSet(dir string, set *EvalSet) error

WriteEvalSet writes evals/evals.json under dir.

func WriteTriggerQueries added in v0.11.0

func WriteTriggerQueries(dir string, qs []TriggerQuery) error

WriteTriggerQueries writes evals/triggers.json under dir.

Types

type Distractor

type Distractor struct {
	Name        string `yaml:"name"`
	Description string `yaml:"description"`
	// Tier is "near", "mid", or "far".
	Tier string `yaml:"tier"`
}

Distractor is one synthetic skill for trigger-precision measurement.

func Distractors

func Distractors() ([]Distractor, error)

Distractors returns the shipped synthetic distractor pack.

type Eval added in v0.11.0

type Eval struct {
	ID             int      `json:"id"`
	Prompt         string   `json:"prompt"`
	ExpectedOutput string   `json:"expected_output,omitempty"`
	Files          []string `json:"files,omitempty"`
	Expectations   []string `json:"expectations,omitempty"`
}

Eval is one task the skill must handle, from evals/evals.json.

func (*Eval) UnmarshalJSON added in v0.11.0

func (e *Eval) UnmarshalJSON(b []byte) error

UnmarshalJSON reads `assertions` as an alias for `expectations`.

Anthropic contradicts itself: references/schemas.md and grading.json say expectations, SKILL.md says assertions. We write expectations only, and accept both, so a file written by skill-creator loads either way.

type EvalCheck added in v0.11.0

type EvalCheck struct {
	ID     int
	OK     bool
	Void   bool
	Reason string
}

EvalCheck is one eval's static validation result. It replaces the oracle gate: with no verifier script to prove correct, what remains to check is that the eval can be run and can be scored at all.

func Validate added in v0.11.0

func Validate(dir string, set *EvalSet) []EvalCheck

Validate checks every eval against dir, which holds the evals/ directory. A void eval is planned but never scored, so a suite does not silently lose its denominator.

type EvalSet added in v0.11.0

type EvalSet struct {
	SkillName string `json:"skill_name"`
	Evals     []Eval `json:"evals"`
}

EvalSet is the whole of evals/evals.json.

func LoadEvalSet added in v0.11.0

func LoadEvalSet(dir string) (*EvalSet, error)

LoadEvalSet reads evals/evals.json from dir.

type TriggerQuery added in v0.11.0

type TriggerQuery struct {
	Query         string `json:"query"`
	ShouldTrigger bool   `json:"should_trigger"`
}

TriggerQuery is one entry of evals/triggers.json: a prompt, and whether the skill must fire on it.

The file is a bare array rather than an object, which is the shape Anthropic's run_eval.py takes through --eval-set. Anthropic never names the file, so the name is ours and the contents are theirs.

func LoadTriggerQueries added in v0.11.0

func LoadTriggerQueries(dir string) ([]TriggerQuery, error)

LoadTriggerQueries reads evals/triggers.json from dir. A missing file is not an error: a skill can carry evals and no trigger set.

func TriggersFromSpec added in v0.11.0

func TriggersFromSpec(sp *spec.SkillSpec) []TriggerQuery

TriggersFromSpec derives the trigger queries from an authored spec.

Directories

Path Synopsis
Package assets embeds the static suite assets.
Package assets embeds the static suite assets.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL