Documentation
¶
Overview ¶
Package evalharness drives the shipped binary's `mcp` subcommand over JSON-RPC stdio, exactly like an agent host does. Extracted from cmd/mcpeval so every served-path evaluation tool (mcpeval, selfsweep, giteval) measures the same stack users run instead of the retrieval library.
Index ¶
- type Result
- type Server
- func (s *Server) Find(query string, maxResults int) (Result, error)
- func (s *Server) FindContext(query string, maxResults int) ([]string, error)
- func (s *Server) FindContextHealth(query string, maxResults int) ([]string, string, error)
- func (s *Server) FindMarkdown(query string, maxResults int, mode string) (string, error)
- func (s *Server) FindMarkdownFull(query string, maxResults int, mode string) (string, error)
- func (s *Server) SetAlpha(a string)
- func (s *Server) SetAnchorExpand(n int)
- func (s *Server) SetFullBodies(n int)
- func (s *Server) SetLiteralSlots(n int)
- func (s *Server) SetRerankK(k int)
- func (s *Server) SetSeedK(k int)
- func (s *Server) SetSkipIntent(v bool)
- func (s *Server) SetTestFloor(n int)
- func (s *Server) Stop()
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Result ¶
type Result struct {
Names []string
// Files is each symbol's path, relative to the indexed root. Region-level
// benchmarks score (file, line-start, line-end) tuples, so the path is not
// optional there the way it is for name-matching probes.
Files []string
Scores []float32
Relevance []float32
// Lines is each symbol's full span ("120-380"); Visible is the span the
// response actually shows after evidence trimming. They differ exactly
// when the answer may have been trimmed away, which ranking metrics
// cannot see — see cmd/deepprobe.
Lines []string
Visible []string
// Callers and Callees are the graph refs shown for each result. They are
// what makes a chain followable without a second search, so cmd/chainprobe
// measures exactly them.
Callers [][]string
Callees [][]string
// RefFiles is the set of files reachable through the graph refs shown for
// each result — the hops an agent can take without a second search. Kept
// separate from Files because a probe measuring saved calls has to
// distinguish what was RETURNED from what merely became reachable.
RefFiles [][]string
Confidence string // "" when the server omitted retrieval_health
TopGap float32 // relative top1-top2 gap as the server computed it
}
Result is one find_context response, reduced to what evaluation needs.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is one spawned `<bin> mcp --index <path>` process.
func (*Server) FindContext ¶
FindContext calls the find_context tool and returns ranked qualified names.
func (*Server) FindContextHealth ¶
FindContextHealth additionally returns the retrieval_health confidence ("" when the server omitted the block, i.e. answered confidently).
func (*Server) FindMarkdown ¶
FindMarkdown returns the response exactly as an agent host receives it. The other helpers parse the JSON encoding, which is the wrong shape for judging what an agent actually reads.
func (*Server) FindMarkdownFull ¶
FindMarkdownFull is FindMarkdown with evidence trimming disabled, so bodies come back whole. It stands in for expand_context in single-shot harnesses: expand_context addresses a rank inside a live server session, and a harness that spawns a process per call has no session to address.
func (*Server) SetAlpha ¶
SetAlpha sets the seed-vs-PageRank weight as a string ("" = served default, "1" = seeds only, which measures what the graph contributes).
func (*Server) SetAnchorExpand ¶
SetAnchorExpand adds graph neighbours of the top seeds to the candidate pool.
func (*Server) SetFullBodies ¶
SetFullBodies controls how many top results keep their full body (-1 = all).
func (*Server) SetLiteralSlots ¶
SetLiteralSlots hands the last N answer slots to files where several of the query's identifiers occur together (experiment knob).
func (*Server) SetRerankK ¶
SetRerankK sizes the cross-encoder pool; it must be >= max_results or the tail of a long response never reaches the reranker.
func (*Server) SetSeedK ¶
SeedK, when non-zero, overrides the server's seed-pool size for every subsequent call on this server (experiment knob; 0 = server default).
func (*Server) SetSkipIntent ¶
SetSkipIntent disables the intent ranker for every subsequent call on this server (experiment knob).
func (*Server) SetTestFloor ¶
SetTestFloor reserves N top answer slots for non-test files (experiment knob).