eval

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 5, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

@index Ranking and classification metrics for parser and search evaluation.

@index Golden corpus loading and normalization helpers for parser evaluation.

@index Eval runner orchestration for parser and search benchmark suites.

@index Data schemas for parser golden corpora and search evaluation reports.

@index Search evaluation corpus loading and ranking metric orchestration.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func EdgeKeys 

func EdgeKeys(edges []EvalEdge) []string

@intent project eval edges into stable comparison keys for set-based metrics.

func FalsePositiveRate 

func FalsePositiveRate(ranked []string) float64

@intent treat any returned result for a negative query as one false-positive hit in aggregate eval reporting.

func MRR 

func MRR(ranked []string, relevant map[string]bool) float64

@intent score the position of the first relevant result for ranking evaluation.

func NDCG 

func NDCG(ranked []string, relevant map[string]bool, k int) float64

@intent compare ranked retrieval quality against an ideal ordering using discounted gain.

func NodeKeys 

func NodeKeys(nodes []EvalNode) []string

@intent project eval nodes into stable comparison keys for set-based metrics.

func PrecisionAtK 

func PrecisionAtK(ranked []string, relevant map[string]bool, k int) float64

@intent measure how many of the top-k ranked results are relevant.

func RecallAtK 

func RecallAtK(ranked []string, relevant map[string]bool, k int) float64

@intent measure how much of the relevant set appears within the top-k ranked results.

func WriteGolden 

func WriteGolden(path string, corpus GoldenCorpus) error

@intent persist normalized parser output as a golden snapshot for future comparisons.

Types

type ClassificationMetrics 

type ClassificationMetrics struct {
	TruePositive  int
	FalsePositive int
	FalseNegative int
	Precision     float64
	Recall        float64
	F1            float64
}

ClassificationMetrics holds precision, recall, and F1 counts for one comparison. @intent expose set-based scoring fields used by parser node and edge evaluation.

func ComputeClassification 

func ComputeClassification(expected, actual []string) ClassificationMetrics

ComputeClassification computes set-based precision, recall, and F1 for expected versus actual keys. @intent provide one reusable metric primitive for both parser node and edge evaluation.

type EvalEdge 

type EvalEdge struct {
	Kind string `json:"kind"`
	From string `json:"from"`
	To   string `json:"to"`
}

EvalEdge is the normalized parser edge shape used in golden corpora. @intent capture corpus-stable edge identity for parser comparisons.

func NormalizeEdges 

func NormalizeEdges(edges []model.Edge, nodes []model.Node) []EvalEdge

@intent normalize parsed graph edges into corpus-stable eval records keyed by qualified names.

func (EvalEdge) Key 

func (e EvalEdge) Key() string

Key derives a stable string key for edge-level evaluation comparisons. @intent provide a deterministic identifier for set-based parser metrics.

type EvalNode 

type EvalNode struct {
	ID        string `json:"id"`
	Kind      string `json:"kind"`
	Name      string `json:"name"`
	File      string `json:"file"`
	StartLine int    `json:"start_line"`
	EndLine   int    `json:"end_line,omitempty"`
}

EvalNode is the normalized parser node shape used in golden corpora. @intent capture corpus-stable node identity independent of absolute paths.

func NormalizeNodes 

func NormalizeNodes(nodes []model.Node, baseDir string) []EvalNode

@intent normalize parsed graph nodes into corpus-stable eval records independent of absolute paths.

func (EvalNode) Key 

func (n EvalNode) Key() string

Key derives a stable string key for node-level evaluation comparisons. @intent provide a deterministic identifier for set-based parser metrics.

type GoldenCorpus 

type GoldenCorpus struct {
	Language string     `json:"language"`
	File     string     `json:"file"`
	Nodes    []EvalNode `json:"nodes"`
	Edges    []EvalEdge `json:"edges"`
}

GoldenCorpus stores one parser snapshot for a source file in a language corpus. @intent persist expected parser output for regression comparison.

func LoadGoldenDir 

func LoadGoldenDir(dir string) ([]GoldenCorpus, error)

@intent load every language-specific golden corpus file from the eval corpus directory tree.

type LanguageReport 

type LanguageReport struct {
	Language    string                `json:"language"`
	NodeMetrics ClassificationMetrics `json:"node_metrics"`
	EdgeMetrics ClassificationMetrics `json:"edge_metrics"`
	Files       int                   `json:"files"`
}

LanguageReport summarizes parser accuracy metrics for one language corpus. @intent expose per-language node and edge metrics in the eval report.

func CompareCorpus 

func CompareCorpus(expected, actual GoldenCorpus) LanguageReport

@intent summarize parser accuracy for one language corpus by comparing expected and actual nodes and edges.

type QueryCase 

type QueryCase struct {
	Query    string   `json:"query"`
	Relevant []string `json:"relevant"`
	K        int      `json:"k,omitempty"`
}

QueryCase defines one search evaluation query and its relevant expected results. @intent describe a single ranked-retrieval test case for search evaluation.

type QueryCorpus 

type QueryCorpus struct {
	CorpusDir string      `json:"corpus_dir"`
	Queries   []QueryCase `json:"queries"`
}

QueryCorpus groups search evaluation cases loaded from one corpus directory. @intent represent the full search evaluation suite as one loadable artifact.

func LoadQueryCorpus 

func LoadQueryCorpus(path string) (QueryCorpus, error)

LoadQueryCorpus loads search evaluation cases from a JSON corpus file. @intent ingest the search query corpus before running ranking evaluation.

type QueryDiagnostic 

type QueryDiagnostic struct {
	Query           string   `json:"query"`
	Kind            string   `json:"kind"`
	ResultsReturned int      `json:"results_returned"`
	PAt1            float64  `json:"p_at_1,omitempty"`
	PAt5            float64  `json:"p_at_5,omitempty"`
	RecallAt5       float64  `json:"recall_at_5,omitempty"`
	MRR             float64  `json:"mrr,omitempty"`
	NDCGAt5         float64  `json:"ndcg_at_5,omitempty"`
	FalsePositive   bool     `json:"false_positive,omitempty"`
	TopResults      []string `json:"top_results,omitempty"`
}

QueryDiagnostic records one query's detailed evaluation outcome for debugging regressions. @intent aggregate metric만으로 보이지 않는 개별 쿼리 실패 원인을 보존한다.

type Report 

type Report struct {
	Suite     string           `json:"suite"`
	Languages []LanguageReport `json:"languages,omitempty"`
	Search    *SearchReport    `json:"search,omitempty"`
}

Report bundles parser and search evaluation output into one CLI-facing payload. @intent represent the complete eval result returned to CLI and JSON consumers.

func Run 

func Run(ctx context.Context, opts RunOptions) (*Report, error)

Run executes the requested evaluation suites and writes the chosen report format. @intent provide one orchestration entry point for CLI-driven parser and search evaluation.

type RunOptions 

type RunOptions struct {
	CorpusDir string
	Suite     string
	Format    string
	Update    bool
	Walkers   map[string]*treesitter.Walker
	SearchFn  SearchFunc
	Writer    io.Writer
}

RunOptions collects corpus paths, parser walkers, and output settings for one eval invocation. @intent configure parser/search evaluation suites from CLI without leaking command details.

type SearchFunc 

type SearchFunc func(query string, limit int) ([]string, error)

SearchFunc abstracts search execution so evaluation can run against any backend. @intent decouple ranking metrics from concrete search implementations.

type SearchReport 

type SearchReport struct {
	QueriesTotal           int               `json:"queries_total"`
	AvgPAt1                float64           `json:"avg_p_at_1"`
	AvgPAt3                float64           `json:"avg_p_at_3"`
	AvgPAt5                float64           `json:"avg_p_at_5"`
	AvgRecallAt5           float64           `json:"avg_recall_at_5"`
	AvgMRR                 float64           `json:"avg_mrr"`
	AvgNDCGAt5             float64           `json:"avg_ndcg_at_5"`
	NegativeQueries        int               `json:"negative_queries"`
	NegativeFalsePositives int               `json:"negative_false_positives"`
	NegativePassRate       float64           `json:"negative_pass_rate"`
	PerQuery               []QueryDiagnostic `json:"per_query,omitempty"`
}

SearchReport holds aggregate ranking metrics across the search evaluation corpus. @intent surface average P@K, recall, MRR, and nDCG over all search queries.

func EvaluateQueries 

func EvaluateQueries(cases []QueryCase, searchFn SearchFunc) (SearchReport, error)

@intent execute ranked retrieval metrics across all search evaluation cases.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.