Documentation
¶
Overview ¶
Package eval implements FR-MCP-04's versioned MCP tool-calling evaluation suite: a fixed set of {tool, arguments, expected outcome} cases, embedded into the binary and versioned by directory (v1, and any v2+ added later without touching v1's cases) so a suite someone has pinned to doesn't silently change underneath them.
The suite tests the tool layer itself — the same thing `vegaload mcp eval` runs in CI and the same thing a human could sanity check after upgrading — not an LLM's tool-choosing behavior; evaluating whether a particular model picks the right tool for a prompt is a separate, model-dependent exercise this package deliberately leaves out, and the cases are written against the compiled vegaload binary via the same internal/mcp.Tool.Handler contract (see cmd/vegaload's eval.go), not against an LLM's choices.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Raw ¶ added in v0.5.0
Raw returns the raw JSON of the named suite ("v1" or "v2"), and false when there is no such suite.
Types ¶
type Caller ¶
Caller invokes one MCP tool by name with raw JSON arguments — the same signature internal/mcp.Tool.Handler uses, so cmd/vegaload/eval.go can pass a Caller built directly from mcp.NewTools without this package importing internal/mcp (keeping a one-way dependency: mcp tools are evaluated by this package, this package is never called by them).
type Case ¶
type Case struct {
Name string `json:"name"`
Tool string `json:"tool"`
Arguments json.RawMessage `json:"arguments"`
// ExpectError, when true, means the call must return a non-nil
// error; ExpectContains is ignored in that case.
ExpectError bool `json:"expect_error,omitempty"`
// ExpectErrorContains, when set, also requires the error message to
// contain this text. It implies ExpectError (v2).
ExpectErrorContains string `json:"expect_error_contains,omitempty"`
// ExpectFiles lists paths that must exist after the call (v2). A
// case uses it for files a tool writes, such as a JUnit report.
ExpectFiles []string `json:"expect_files,omitempty"`
// ExpectContains lists fields the JSON-decoded result must contain
// with exactly these values. A key is a top-level field name. In v2 a
// key may also be a dotted path into nested objects and arrays, such
// as "baseline.passed" or "checks.0.name" (a number picks an array
// item). There is no general JSON-diff.
ExpectContains map[string]any `json:"expect_contains,omitempty"`
}
Case is one eval case: call Tool with Arguments and check the result against ExpectError/ExpectContains.
type CaseResult ¶
type CaseResult struct {
Name string
Passed bool
Detail string // empty when Passed; otherwise why it failed
}
CaseResult is one Case's outcome.