goldensuite

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package goldensuite is the repeatable regression gate: one definition of "our numbers", run against the shipped binary, comparable across runs.

It combines the three measurements that run everywhere and stay stable:

  • gen corpus (hand-authored labels, the target metrics)
  • self-retrieval sweep (label-free, any repo, the repo authors' words)
  • verified negatives (false confidence — invisible to the other two)

giteval is deliberately NOT included: its label quality depends on how deep the clone's history is, so it is a manual tool, not a gate.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ScoreCase

func ScoreCase(c eval.QueryCase, ranked []string) (hit1, hit3, r5, r10 bool)

ScoreCase mirrors cmd/eval: Hit@K is any-of, Recall@K honors min_hits.

Types

type Config

type Config struct {
	GenManifest string      `json:"gen_manifest"`
	Repos       []RepoEntry `json:"repos"`
	// SelfN is how many self-retrieval cases to sample per repo.
	SelfN int `json:"self_n"`
	// SkipIntent turns the intent ranker off for the whole run (A/B knob).
	SkipIntent bool `json:"skip_intent,omitempty"`
}

Config declares what the suite covers. Paths are relative to the config file, so the checked-in definition works on any machine that has the repos.

func LoadConfig

func LoadConfig(path string) (*Config, string, error)

type GenResult

type GenResult struct {
	N     int     `json:"n"`
	Hit1  float64 `json:"hit1"`
	Hit3  float64 `json:"hit3"`
	R5    float64 `json:"r5"`
	R10   float64 `json:"r10"`
	AvgMs int64   `json:"avg_ms"`
}

type NegResult

type NegResult struct {
	N              int     `json:"n"`
	Dropped        int     `json:"dropped_as_present"`
	FalseConfident int     `json:"false_confident"`
	Rate           float64 `json:"rate"`
}

type RepoEntry

type RepoEntry struct {
	Name  string `json:"name"`
	Index string `json:"index"`
}

type RepoResult

type RepoResult struct {
	Self *SelfResult `json:"self,omitempty"`
	Neg  *NegResult  `json:"neg,omitempty"`
}

type Report

type Report struct {
	Generated time.Time             `json:"generated"`
	Gen       *GenResult            `json:"gen,omitempty"`
	Repos     map[string]RepoResult `json:"repos"`
}

Report is one full run, JSON-serializable so runs can be diffed.

func Run

func Run(bin string, cfg *Config, base string, holdout bool, log func(string, ...any)) (*Report, []error)

Run executes the whole suite. Component failures are reported per component rather than aborting: a missing repo must not hide the metrics that did run.

type SelfResult

type SelfResult struct {
	N     int     `json:"n"`
	Hit1  float64 `json:"hit1"`
	Hit5  float64 `json:"hit5"`
	Hit10 float64 `json:"hit10"`
	Miss  float64 `json:"miss"`
	// Hit5ByName accepts any symbol with the same short name as the target.
	// DECISION(2026-07): interface/impl pairs make exact-symbol scoring
	// unfair — the docstring lives on the trait declaration while the code
	// lives in the implementing class, and returning the impl IS the answer
	// an agent needs. os-lib read 0.23 "never retrieved" purely because
	// BasePathImpl.baseName was scored as a miss for BasePath.baseName.
	// Strict stays primary (it is the honest lower bound); this is the
	// upper bound, and a wide gap means decl/impl duplication, not failure.
	Hit5ByName float64 `json:"hit5_by_name"`
	SeedGap    float64 `json:"seed_loss_rate"`
	AvgMs      int64   `json:"avg_ms"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL