eval

package
v0.1.0-beta.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

Documentation

Overview

Package eval is Ozy's evaluation harness. It loads the committed corpus (a synthetic downstream MCP catalog plus labeled gold/scenario sets), drives the real broker, search engine, CLI, and MCP seams over it, computes the SPEC.md §14.2 metrics deterministically, and emits a machine-readable run result plus a human-readable benchmark scoreboard gated against thresholds.

Every scenario is data, not Go code: adding a labeled intent means editing a file under evals/data/, never this package. The dataset schema lives in evals/README.md and the metric mathematics in evals/METHODOLOGY.md.

Index

Constants

View Source 
const (
	CategoryLexical     = "lexical"
	CategorySemantic    = "semantic"
	CategoryNoMatch     = "no_match"
	CategoryAmbiguous   = "ambiguous"
	CategoryWrongServer = "wrong_server"
)

Discovery categories. Each labeled intent declares one so metrics can be reported per category and category-specific gates applied.

View Source 
const (
	// OutcomeSuccess: the first call with the given arguments should succeed.
	OutcomeSuccess = "success"
	// OutcomeRepair: the first call fails validation with ExpectedError, and the
	// Corrected arguments then succeed (the repair loop, SPEC.md §14.1).
	OutcomeRepair = "repair"
	// OutcomeError: the call terminates in the structured ExpectedError (e.g. an
	// offline server or schema drift) with no successful repair expected.
	OutcomeError = "error"
)

Invocation outcomes. Each scenario declares exactly one.

View Source 
const (
	KindFind     = "find"
	KindDescribe = "describe"
	KindCall     = "call"
)

Ergonomics case kinds — which broker operation the case exercises.

View Source 
const (
	VerdictPass = "pass"
	VerdictFail = "fail"
)

Verdict values.

View Source 
const (
	FamilyDiscovery   = "discovery"
	FamilyInvocation  = "invocation"
	FamilyErgonomics  = "ergonomics"
	FamilyTokens      = "tokens"
	FamilyPerformance = "performance"
	FamilyCache       = "cache"
)

Family identifiers for scoping a run.

View Source 
const SchemaVersion = "ozy-eval/v1"

SchemaVersion stamps the run-result shape so snapshots from different builds can be compared and migrated.

Variables

This section is empty.

Functions

func Scoreboard 

func Scoreboard(r *RunResult) string

Scoreboard renders the human-readable public benchmark Markdown from a run result. It summarizes each family at a glance; the metric mathematics live in METHODOLOGY.md, which the scoreboard links to rather than inlining.

func WriteScoreboard 

func WriteScoreboard(outDir string, r *RunResult) error

WriteScoreboard regenerates the public benchmark document at outDir/BENCHMARKS.md from the run result, so its prose can never drift from the numbers.

func WriteSnapshot 

func WriteSnapshot(outDir string, r *RunResult) (string, error)

WriteSnapshot writes the run result as a timestamped JSON snapshot under outDir/snapshots/ and refreshes snapshots/baseline.json so trends are git-diffable. It returns the timestamped snapshot path.

Types

type CacheEffectivenessMetrics 

type CacheEffectivenessMetrics struct {
	Estimator              string  `json:"estimator"`
	CacheableOps           int     `json:"cacheableOps"`
	ServedFromCache        int     `json:"servedFromCache"`
	DelegatedOps           int     `json:"delegatedOps"`
	RedundantCallReduction float64 `json:"redundantCallReduction"`
	WorkloadResponseTokens int     `json:"workloadResponseTokens"`
	ExecutedResponseTokens int     `json:"executedResponseTokens"`
	TokensAvoided          int     `json:"tokensAvoided"`
}

CacheEffectivenessMetrics report how much redundant broker work the result cache removes over a fixed repeated-call workload. RedundantCallReduction is the fraction of cacheable operations served from cache instead of re-executed (served / cacheable). The token figures show the downstream/search payload that was re-served from cache rather than re-fetched — work avoided, not agent tokens. Deterministic for a fixed corpus and estimator.

func RunCacheEffectiveness 

func RunCacheEffectiveness(ctx context.Context, corpus *Corpus, est TokenEstimator) (*CacheEffectivenessMetrics, error)

RunCacheEffectiveness drives the fixed workload through the production cache decorator layered over a call-counting broker, classifying each op as a cache hit or a delegated miss by the counter delta.

type CacheGate 

type CacheGate struct {
	RedundantCallReductionMin *float64 `json:"redundantCallReductionMin,omitempty"`
}

CacheGate gates the cache-effectiveness headline (redundant-call reduction).

type Catalog 

type Catalog struct {
	Version     int             `json:"version"`
	Description string          `json:"description"`
	Servers     []CatalogServer `json:"servers"`
	Tools       []CatalogTool   `json:"tools"`
}

Catalog is the synthetic downstream MCP catalog ("the world") the search legs rank against.

type CatalogServer 

type CatalogServer struct {
	ID     string `json:"id"`
	Status string `json:"status"`
}

CatalogServer is one configured downstream server in the corpus.

type CatalogTool 

type CatalogTool struct {
	ToolRef     string         `json:"toolRef"`
	ServerID    string         `json:"serverId"`
	Name        string         `json:"name"`
	Title       string         `json:"title"`
	Description string         `json:"description"`
	InputSchema map[string]any `json:"inputSchema"`
	// ReadOnly mirrors the downstream readOnlyHint annotation. It gates result
	// caching of callTool in the cache-effectiveness family; omitted defaults to
	// false (write-safe).
	ReadOnly bool `json:"readOnly,omitempty"`
}

CatalogTool is one downstream tool entry in the corpus, mirroring the minimum metadata Ozy catalogs (SPEC.md §8).

func (CatalogTool) IndexedText 

func (t CatalogTool) IndexedText() string

IndexedText is the natural-language text embedded for the semantic leg. It mirrors the SPEC.md §10.2 indexed fields a real index run would push to the sidecar, so the eval embeds tools the same way production does.

type Corpus 

type Corpus struct {
	Catalog    Catalog
	Discovery  []DiscoveryCase
	Invocation []InvocationScenario
	Ergonomics []ErgonomicsCase
}

Corpus is the fully loaded, validated eval corpus.

func Load 

func Load(fsys fs.FS) (*Corpus, error)

Load reads and validates the entire corpus from fsys (typically the embedded evals.Data() FS). It fails fast with a path- and field-qualified error on any malformed file or dangling toolRef reference, so a broken dataset never reaches scoring. Structural validation only — semantic-leakage hygiene is reported separately by Hygiene so the loader stays free of the search engine.

func (*Corpus) Store 

func (c *Corpus) Store() (catalog.Store, error)

Store builds an in-memory catalog store populated with the corpus tools, ready for the search engine and broker to rank. Tools are marked online, callable, and fresh so the broker treats the corpus as a healthy live catalog.

type DiscoveryCase 

type DiscoveryCase struct {
	Intent     string   `json:"intent"`
	Category   string   `json:"category"`
	Acceptable []string `json:"acceptable"`
	Rationale  string   `json:"rationale"`
}

DiscoveryCase is one labeled discovery intent: a user phrasing mapped to the acceptable target toolRef(s). An empty Acceptable means the capability is absent and the broker is expected to refuse (no-match correctness).

type DiscoveryGate 

type DiscoveryGate struct {
	Top1Min               *float64 `json:"top1Min,omitempty"`
	Top3Min               *float64 `json:"top3Min,omitempty"`
	MRRMin                *float64 `json:"mrrMin,omitempty"`
	WrongServerRateMax    *float64 `json:"wrongServerRateMax,omitempty"`
	NoMatchCorrectnessMin *float64 `json:"noMatchCorrectnessMin,omitempty"`
	RequiresSemanticLeg   bool     `json:"requiresSemanticLeg,omitempty"`
}

DiscoveryGate is the set of gated metrics for one discovery scope (a category key such as "lexical", or "overall"). *Min fields gate accuracy (higher is better); *Max fields gate rates (lower is better).

type DiscoveryMetrics 

type DiscoveryMetrics struct {
	N                  int     `json:"n"`
	Top1               float64 `json:"top1"`
	Top3               float64 `json:"top3"`
	MRR                float64 `json:"mrr"`
	WrongServerRate    float64 `json:"wrongServerRate,omitempty"`
	NoMatchCorrectness float64 `json:"noMatchCorrectness,omitempty"`
}

DiscoveryMetrics are the SPEC.md §14.2 discovery metrics for a set of cases. WrongServerRate and NoMatchCorrectness are only meaningful for the matching categories and are omitted (zero) elsewhere.

type DiscoveryReport 

type DiscoveryReport struct {
	Overall    DiscoveryMetrics            `json:"overall"`
	ByCategory map[string]DiscoveryMetrics `json:"byCategory"`
}

DiscoveryReport is the discovery family result: headline overall metrics plus a per-category breakdown.

func RunDiscovery 

func RunDiscovery(ctx context.Context, engine *search.Engine, cases []DiscoveryCase) (*DiscoveryReport, error)

RunDiscovery drives the search engine over every discovery case and computes the discovery metrics. It uses the engine's full ranked order (for MRR/top-3) and search.Decide (for no-match correctness), so it measures the same ranking the broker's findTool exposes. Deterministic for a fixed corpus and engine.

type ErgonomicsCase 

type ErgonomicsCase struct {
	Name            string         `json:"name"`
	Kind            string         `json:"kind"`
	Query           string         `json:"query,omitempty"`
	ToolRef         string         `json:"toolRef,omitempty"`
	Arguments       map[string]any `json:"arguments,omitempty"`
	ExpectDecision  string         `json:"expectDecision,omitempty"`
	ExpectErrorType string         `json:"expectErrorType,omitempty"`
	Rationale       string         `json:"rationale"`
}

ErgonomicsCase is one agent-facing input exercised on both the CLI and MCP surfaces for the structural §4.5/§9/§13 conformance checks and CLI↔MCP parity. ExpectDecision / ExpectErrorType are optional intent anchors; the conformance checks run regardless.

type ErgonomicsGate 

type ErgonomicsGate struct {
	DecisionRateMin         *float64 `json:"decisionRateMin,omitempty"`
	InstructionRateMin      *float64 `json:"instructionRateMin,omitempty"`
	ErrorDispositionRateMin *float64 `json:"errorDispositionRateMin,omitempty"`
	WithinBudgetRateMin     *float64 `json:"withinBudgetRateMin,omitempty"`
	ParityRateMin           *float64 `json:"parityRateMin,omitempty"`
}

ErgonomicsGate is the set of gated ergonomics/parity metrics (all *Min).

type ErgonomicsMetrics 

type ErgonomicsMetrics struct {
	N                    int     `json:"n"`
	DecisionRate         float64 `json:"decisionRate"`
	InstructionRate      float64 `json:"instructionRate"`
	ErrorDispositionRate float64 `json:"errorDispositionRate"`
	WithinBudgetRate     float64 `json:"withinBudgetRate"`
	ParityRate           float64 `json:"parityRate"`
}

ErgonomicsMetrics are the structural §4.5/§9/§13 conformance rates plus CLI↔MCP parity, organized by the agent-ergonomics domains: decision clarity (DecisionRate), actionable guidance (InstructionRate), error recoverability (ErrorDispositionRate), response economy (WithinBudgetRate), and surface consistency (ParityRate).

type ErgonomicsReport 

type ErgonomicsReport struct {
	Overall          ErgonomicsMetrics `json:"overall"`
	NonInstructional []string          `json:"nonInstructional,omitempty"`
	BudgetExceeded   []string          `json:"budgetExceeded,omitempty"`
	ParityMismatches []string          `json:"parityMismatches,omitempty"`
}

ErgonomicsReport is the ergonomics family result. The flagged-case lists make regressions self-explaining in the snapshot.

func RunErgonomics 

func RunErgonomics(ctx context.Context, corpus *Corpus) (*ErgonomicsReport, error)

RunErgonomics exercises every ergonomics case on both the CLI (--format json) path and the in-process MCP adapter over one shared broker, applies the structural conformance checks to the broker response, and asserts CLI↔MCP parity. Deterministic for a fixed corpus.

type GateResult 

type GateResult struct {
	Name       string  `json:"name"`
	Comparator string  `json:"comparator"` // "min" or "max"
	Threshold  float64 `json:"threshold"`
	Actual     float64 `json:"actual"`
	Pass       bool    `json:"pass"`
	Skipped    bool    `json:"skipped,omitempty"`
	SkipReason string  `json:"skipReason,omitempty"`
}

GateResult is the outcome of one threshold check.

type HygieneFinding 

type HygieneFinding struct {
	Intent   string `json:"intent"`
	Category string `json:"category"`
	Issue    string `json:"issue"`
}

HygieneFinding flags a gold-set quality problem — currently a "semantic" intent the lexical baseline already wins outright, which means the label is not actually testing semantic matching (it's a lexical freebie).

func Hygiene 

func Hygiene(corpus *Corpus) ([]HygieneFinding, error)

Hygiene runs the lexical-only baseline over the corpus and returns findings where a semantic-paraphrase intent is already won at rank 1 by lexical search. It keeps the "semantic" gold set honest: a genuine semantic case must give the semantic leg something to do. Returns an error only if the engine fails.

type InvocationGate 

type InvocationGate struct {
	ValidArgumentRateMin  *float64 `json:"validArgumentRateMin,omitempty"`
	FirstCallSuccessMin   *float64 `json:"firstCallSuccessMin,omitempty"`
	RepairSuccessMin      *float64 `json:"repairSuccessMin,omitempty"`
	SchemaErrorRateMin    *float64 `json:"schemaErrorRateMin,omitempty"`
	OfflineHandledRateMin *float64 `json:"offlineHandledRateMin,omitempty"`
	ErrorClarityMin       *float64 `json:"errorClarityMin,omitempty"`
}

InvocationGate is the set of gated invocation metrics. All *Min fields gate accuracy (higher is better); any left unset is not gated.

type InvocationMetrics 

type InvocationMetrics struct {
	N                  int     `json:"n"`
	ValidArgumentRate  float64 `json:"validArgumentRate"`
	FirstCallSuccess   float64 `json:"firstCallSuccess"`
	RepairSuccess      float64 `json:"repairSuccess"`
	SchemaErrorRate    float64 `json:"schemaErrorRate"`
	OfflineHandledRate float64 `json:"offlineHandledRate"`
	ErrorClarity       float64 `json:"errorClarity"`
}

InvocationMetrics are the SPEC.md §14.1 invocation/repair metrics over the scenario set. Rates are in [0,1]; a rate whose family has no scenarios is 0.

type InvocationReport 

type InvocationReport struct {
	Overall InvocationMetrics `json:"overall"`
}

InvocationReport is the invocation family result.

func RunInvocation 

func RunInvocation(ctx context.Context, corpus *Corpus) (*InvocationReport, error)

RunInvocation evaluates every invocation scenario over the corpus, driving the real broker.CallTool seam against an in-process fixture downstream. Argument validation and schema-drift detection are modeled at the harness layer (the broker delegates validation downstream today, SPEC.md non-goal); the corrected repair call and the offline path run through the real broker. Deterministic.

type InvocationScenario 

type InvocationScenario struct {
	Name            string         `json:"name"`
	ToolRef         string         `json:"toolRef"`
	Arguments       map[string]any `json:"arguments"`
	ExpectedOutcome string         `json:"expectedOutcome"`
	ExpectedError   string         `json:"expectedError,omitempty"`
	Corrected       map[string]any `json:"corrected,omitempty"`
	LiveSchema      map[string]any `json:"liveSchema,omitempty"`
	Rationale       string         `json:"rationale"`
}

InvocationScenario is one labeled callTool scenario (SPEC.md §14.1). Argument validity is judged against the cataloged tool's inputSchema; LiveSchema, when set, is the drifted downstream schema a TOOL_SCHEMA_CHANGED case detects.

type LatencyReport 

type LatencyReport struct {
	Lexical  *LatencyStats `json:"lexical,omitempty"`
	Fusion   *LatencyStats `json:"fusion,omitempty"`
	EndToEnd *LatencyStats `json:"endToEnd,omitempty"`
	Semantic *LatencyStats `json:"semantic,omitempty"`
}

LatencyReport holds per-path latency for the lexical ranker, RRF fusion, the end-to-end broker findTool path, and (when the leg ran) the semantic path.

func RunLatency 

func RunLatency(ctx context.Context, lexEngine *search.Engine, bk broker.Broker, semEngine *search.Engine) *LatencyReport

RunLatency measures the latency distribution of each retrieval path. lexEngine and bk are the lexical-only baseline; semEngine (optional) measures the real semantic path when the gated leg ran.

type LatencyStats 

type LatencyStats struct {
	N         int     `json:"n"`
	P50Micros float64 `json:"p50Micros"`
	P95Micros float64 `json:"p95Micros"`
	OpsPerSec float64 `json:"opsPerSec"`
}

LatencyStats summarizes a path's latency distribution and throughput. Timings are environment-dependent, so these are reported (with the run's host provenance), never gated.

type Options 

type Options struct {
	// Data is the corpus FS; defaults to the embedded evals.Data().
	Data fs.FS
	// OutDir is where the snapshot and scoreboard are written. Empty means do not
	// write artifacts (in-memory run only).
	OutDir string
	// Families scopes the run; empty runs all known families.
	Families []string
	// Semantic requests the real-model semantic leg (also honored via
	// OZY_EVAL_SEMANTIC=1). Requires SemanticBuilder to actually run.
	Semantic bool
	// SemanticBuilder builds the real semantic provider; see SemanticBuilder docs.
	SemanticBuilder SemanticBuilder
	// Estimator is the token estimator; defaults to DefaultEstimator.
	Estimator TokenEstimator
}

Options configures a harness run.

type Provenance 

type Provenance struct {
	CorpusVersion  int    `json:"corpusVersion"`
	Model          string `json:"model"`
	SemanticRan    bool   `json:"semanticRan"`
	TokenEstimator string `json:"tokenEstimator,omitempty"`
	GitCommit      string `json:"gitCommit"`
	Host           string `json:"host,omitempty"`
}

Provenance records exactly what produced a run so any number is attributable and reproducible (SPEC.md §14 grading discipline).

type RunResult 

type RunResult struct {
	Schema       string                     `json:"schema"`
	GeneratedAt  time.Time                  `json:"generatedAt"`
	Provenance   Provenance                 `json:"provenance"`
	Discovery    *DiscoveryReport           `json:"discovery,omitempty"`
	Invocation   *InvocationReport          `json:"invocation,omitempty"`
	Ergonomics   *ErgonomicsReport          `json:"ergonomics,omitempty"`
	TokenEconomy *TokenEconomyMetrics       `json:"tokenEconomy,omitempty"`
	Performance  *LatencyReport             `json:"performance,omitempty"`
	Cache        *CacheEffectivenessMetrics `json:"cache,omitempty"`
	Hygiene      []HygieneFinding           `json:"hygiene,omitempty"`
	Gates        []GateResult               `json:"gates"`
	Verdict      string                     `json:"verdict"`
}

RunResult is the single structured object a harness run emits: every computed metric, the provenance, the gate outcomes, and the overall verdict. It is both the JSON snapshot and the source the Markdown scoreboard is generated from.

func LoadSnapshot 

func LoadSnapshot(outDir string) (*RunResult, error)

LoadSnapshot reads the latest committed benchmark snapshot from outDir/snapshots/baseline.json for `ozy eval report`.

func Run 

func Run(ctx context.Context, opts Options) (*RunResult, error)

Run loads the corpus, drives the requested families over the real seams, evaluates the gate thresholds, and returns a structured result. When OutDir is set it also writes the JSON snapshot and the BENCHMARKS.md scoreboard.

func (*RunResult) Failed 

func (r *RunResult) Failed() bool

Failed reports whether the run's verdict is fail (used for the process exit status so CI can gate on `ozy eval run`).

func (*RunResult) Render 

func (r *RunResult) Render(format string) string

Render produces the human/concise text form of a run result so `ozy eval` output is readable without --format json. JSON mode marshals the struct.

type SemanticBuilder 

type SemanticBuilder func(ctx context.Context, store catalog.Store, corpus *Corpus) (search.Semantic, func(), error)

SemanticBuilder constructs a semantic provider over the corpus store for the real-model leg, returning the provider, a cleanup func, and an error. It lives behind this hook so internal/eval has no hard dependency on the sidecar; the CLI supplies the real (sidecar-backed) builder. When nil while Semantic is requested, the semantic leg is skipped (recorded), never failed.

type Thresholds 

type Thresholds struct {
	Version    int                      `json:"version"`
	Note       string                   `json:"_note,omitempty"`
	Discovery  map[string]DiscoveryGate `json:"discovery"`
	Invocation *InvocationGate          `json:"invocation,omitempty"`
	Ergonomics *ErgonomicsGate          `json:"ergonomics,omitempty"`
	Tokens     *TokenGate               `json:"tokens,omitempty"`
	Cache      *CacheGate               `json:"cache,omitempty"`
}

Thresholds are the versioned, data-driven gate thresholds. Fields left unset (nil) are not gated, so the gate set can be ratcheted by editing data only.

func LoadThresholds 

func LoadThresholds(fsys fs.FS) (*Thresholds, error)

LoadThresholds reads and parses the gate thresholds from fsys.

func (*Thresholds) EvaluateCache 

func (t *Thresholds) EvaluateCache(m *CacheEffectivenessMetrics) []GateResult

EvaluateCache turns the cache-effectiveness threshold into a gate result.

func (*Thresholds) EvaluateDiscovery 

func (t *Thresholds) EvaluateDiscovery(report *DiscoveryReport, semanticRan bool) []GateResult

EvaluateDiscovery turns the discovery thresholds into gate results against the report. Gates marked requiresSemanticLeg are skipped (not failed) when the run did not exercise the real embedding model. Results are returned in a stable (scope-sorted) order for deterministic snapshots.

func (*Thresholds) EvaluateErgonomics 

func (t *Thresholds) EvaluateErgonomics(r *ErgonomicsReport) []GateResult

EvaluateErgonomics turns the ergonomics thresholds into gate results.

func (*Thresholds) EvaluateInvocation 

func (t *Thresholds) EvaluateInvocation(r *InvocationReport) []GateResult

EvaluateInvocation turns the invocation thresholds into gate results.

func (*Thresholds) EvaluateTokens 

func (t *Thresholds) EvaluateTokens(m *TokenEconomyMetrics) []GateResult

EvaluateTokens turns the token-economy threshold into a gate result.

type TokenEconomyMetrics 

type TokenEconomyMetrics struct {
	Estimator             string  `json:"estimator"`
	DirectStartupTokens   int     `json:"directStartupTokens"`
	OzyStartupTokens      int     `json:"ozyStartupTokens"`
	StartupReductionRatio float64 `json:"startupReductionRatio"`
	DirectTokensToSuccess int     `json:"directTokensToSuccess"`
	OzyTokensToSuccess    int     `json:"ozyTokensToSuccess"`
	LargestPayloadTokens  int     `json:"largestPayloadTokens"`
	BrokerCalls           int     `json:"brokerCalls"`
}

TokenEconomyMetrics are the SPEC.md §13 token-economy numbers, computed deterministically from captured schemas and payloads with a documented, swappable estimator. The headline is the startup reduction: Ozy advertises three tool schemas where direct-MCP loads the entire downstream universe.

func RunTokenEconomy 

func RunTokenEconomy(ctx context.Context, corpus *Corpus, est TokenEstimator) (*TokenEconomyMetrics, error)

RunTokenEconomy measures startup and per-task token economy for the direct-MCP baseline (all downstream schemas) versus the Ozy broker path (three tools plus per-task find→describe→call). Deterministic for a fixed corpus and estimator.

type TokenEstimator 

type TokenEstimator interface {
	// Name identifies the estimator in run provenance.
	Name() string
	// Estimate returns the approximate token count of text.
	Estimate(text string) int
}

TokenEstimator approximates how many model tokens a string costs. Agent context tokens are model-specific, so the estimator is an explicit, swappable seam; the committed numbers record which estimator produced them (SPEC.md §13). A real BPE tokenizer can replace the default without touching the metric code. 

var DefaultEstimator TokenEstimator = heuristicEstimator{}

DefaultEstimator is the estimator used unless a caller swaps it.

type TokenGate 

type TokenGate struct {
	StartupReductionMin *float64 `json:"startupReductionMin,omitempty"`
}

TokenGate gates the token-economy headline (startup reduction ratio).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.