eval

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 6, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Overview

Package eval implements FR-MCP-04's versioned MCP tool-calling evaluation suite: a fixed set of {tool, arguments, expected outcome} cases, embedded into the binary and versioned by directory (v1, and any v2+ added later without touching v1's cases) so a suite someone has pinned to doesn't silently change underneath them.

The suite tests the tool layer itself — the same thing `vegaload mcp eval` runs in CI and the same thing a human could sanity check after upgrading — not an LLM's tool-choosing behavior; evaluating whether a particular model picks the right tool for a prompt is a separate, model-dependent exercise this package deliberately leaves out, and the cases are written against the compiled vegaload binary via the same internal/mcp.Tool.Handler contract (see cmd/vegaload's eval.go), not against an LLM's choices.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RawV1

func RawV1() []byte

RawV1 returns the v1 suite's raw JSON bytes, before any "${WORKDIR}" substitution.

Types

type Caller

type Caller func(ctx context.Context, tool string, arguments json.RawMessage) (any, error)

Caller invokes one MCP tool by name with raw JSON arguments — the same signature internal/mcp.Tool.Handler uses, so cmd/vegaload/eval.go can pass a Caller built directly from mcp.NewTools without this package importing internal/mcp (keeping a one-way dependency: mcp tools are evaluated by this package, this package is never called by them).

type Case

type Case struct {
	Name      string          `json:"name"`
	Tool      string          `json:"tool"`
	Arguments json.RawMessage `json:"arguments"`

	// ExpectError, when true, means the call must return a non-nil
	// error; ExpectContains is ignored in that case.
	ExpectError bool `json:"expect_error,omitempty"`

	// ExpectContains lists fields the JSON-decoded result must contain
	// with exactly these values (a shallow, top-level check — enough
	// for this suite's cases without needing a general JSON-diff).
	ExpectContains map[string]any `json:"expect_contains,omitempty"`
}

Case is one eval case: call Tool with Arguments and check the result against ExpectError/ExpectContains.

type CaseResult

type CaseResult struct {
	Name   string
	Passed bool
	Detail string // empty when Passed; otherwise why it failed
}

CaseResult is one Case's outcome.

func Run

func Run(ctx context.Context, call Caller, cases []Case) []CaseResult

Run calls call for every case and checks its outcome, returning one CaseResult per case in order.

type Suite

type Suite struct {
	Version string `json:"version"`
	Cases   []Case `json:"cases"`
}

Suite is a versioned set of Cases.

func ParseSuite

func ParseSuite(data []byte) (Suite, error)

ParseSuite decodes data (typically RawV1()'s bytes, after any "${WORKDIR}" substitution) into a Suite.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL