evalharness

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: MIT Imports: 4 Imported by: 0

Documentation

Overview

Package evalharness drives the shipped binary's `mcp` subcommand over JSON-RPC stdio, exactly like an agent host does. Extracted from cmd/mcpeval so every served-path evaluation tool (mcpeval, selfsweep, giteval) measures the same stack users run instead of the retrieval library.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Result

type Result struct {
	Names []string
	// Files is each symbol's path, relative to the indexed root. Region-level
	// benchmarks score (file, line-start, line-end) tuples, so the path is not
	// optional there the way it is for name-matching probes.
	Files     []string
	Scores    []float32
	Relevance []float32
	// Lines is each symbol's full span ("120-380"); Visible is the span the
	// response actually shows after evidence trimming. They differ exactly
	// when the answer may have been trimmed away, which ranking metrics
	// cannot see — see cmd/deepprobe.
	Lines   []string
	Visible []string
	// Callers and Callees are the graph refs shown for each result. They are
	// what makes a chain followable without a second search, so cmd/chainprobe
	// measures exactly them.
	Callers [][]string
	Callees [][]string
	// RefFiles is the set of files reachable through the graph refs shown for
	// each result — the hops an agent can take without a second search. Kept
	// separate from Files because a probe measuring saved calls has to
	// distinguish what was RETURNED from what merely became reachable.
	RefFiles   [][]string
	Confidence string  // "" when the server omitted retrieval_health
	TopGap     float32 // relative top1-top2 gap as the server computed it
}

Result is one find_context response, reduced to what evaluation needs.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server is one spawned `<bin> mcp --index <path>` process.

func Start

func Start(bin, indexPath string, extraArgs ...string) (*Server, error)

Start spawns the server and completes the MCP initialize handshake.

func (*Server) Find

func (s *Server) Find(query string, maxResults int) (Result, error)

Find returns the full reduced response (names, scores, health).

func (*Server) FindContext

func (s *Server) FindContext(query string, maxResults int) ([]string, error)

FindContext calls the find_context tool and returns ranked qualified names.

func (*Server) FindContextHealth

func (s *Server) FindContextHealth(query string, maxResults int) ([]string, string, error)

FindContextHealth additionally returns the retrieval_health confidence ("" when the server omitted the block, i.e. answered confidently).

func (*Server) FindMarkdown

func (s *Server) FindMarkdown(query string, maxResults int, mode string) (string, error)

FindMarkdown returns the response exactly as an agent host receives it. The other helpers parse the JSON encoding, which is the wrong shape for judging what an agent actually reads.

func (*Server) FindMarkdownFull

func (s *Server) FindMarkdownFull(query string, maxResults int, mode string) (string, error)

FindMarkdownFull is FindMarkdown with evidence trimming disabled, so bodies come back whole. It stands in for expand_context in single-shot harnesses: expand_context addresses a rank inside a live server session, and a harness that spawns a process per call has no session to address.

func (*Server) SetAlpha

func (s *Server) SetAlpha(a string)

SetAlpha sets the seed-vs-PageRank weight as a string ("" = served default, "1" = seeds only, which measures what the graph contributes).

func (*Server) SetAnchorExpand

func (s *Server) SetAnchorExpand(n int)

SetAnchorExpand adds graph neighbours of the top seeds to the candidate pool.

func (*Server) SetFullBodies

func (s *Server) SetFullBodies(n int)

SetFullBodies controls how many top results keep their full body (-1 = all).

func (*Server) SetLiteralSlots

func (s *Server) SetLiteralSlots(n int)

SetLiteralSlots hands the last N answer slots to files where several of the query's identifiers occur together (experiment knob).

func (*Server) SetRerankK

func (s *Server) SetRerankK(k int)

SetRerankK sizes the cross-encoder pool; it must be >= max_results or the tail of a long response never reaches the reranker.

func (*Server) SetSeedK

func (s *Server) SetSeedK(k int)

SeedK, when non-zero, overrides the server's seed-pool size for every subsequent call on this server (experiment knob; 0 = server default).

func (*Server) SetSkipIntent

func (s *Server) SetSkipIntent(v bool)

SetSkipIntent disables the intent ranker for every subsequent call on this server (experiment knob).

func (*Server) SetTestFloor

func (s *Server) SetTestFloor(n int)

SetTestFloor reserves N top answer slots for non-test files (experiment knob).

func (*Server) Stop

func (s *Server) Stop()

Stop kills the server process.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL