Help us shape the future of Go! Take the 2026 Go Developer Survey

goldensuite

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 8, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package goldensuite is the repeatable regression gate: one definition of "our numbers", run against the shipped binary, comparable across runs.

It combines the three measurements that run everywhere and stay stable:

  • gen corpus (hand-authored labels, the target metrics)
  • self-retrieval sweep (label-free, any repo, the repo authors' words)
  • verified negatives (false confidence — invisible to the other two)

giteval is deliberately NOT included: its label quality depends on how deep the clone's history is, so it is a manual tool, not a gate.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ScoreCase

func ScoreCase(c eval.QueryCase, ranked []string) (hit1, hit3, r5, r10 bool)

ScoreCase mirrors cmd/eval: Hit@K is any-of, Recall@K honors min_hits.

Types

type Config

type Config struct {
	GenManifest string      `json:"gen_manifest"`
	Repos       []RepoEntry `json:"repos"`
	// SelfN is how many self-retrieval cases to sample per repo.
	SelfN int `json:"self_n"`
	// SkipIntent turns the intent ranker off for the whole run (A/B knob).
	SkipIntent bool `json:"skip_intent,omitempty"`
}

Config declares what the suite covers. Paths are relative to the config file, so the checked-in definition works on any machine that has the repos.

func LoadConfig

func LoadConfig(path string) (*Config, string, error)

type GenResult

type GenResult struct {
	N     int     `json:"n"`
	Hit1  float64 `json:"hit1"`
	Hit3  float64 `json:"hit3"`
	R5    float64 `json:"r5"`
	R10   float64 `json:"r10"`
	AvgMs int64   `json:"avg_ms"`
}

type NegResult

type NegResult struct {
	N              int     `json:"n"`
	Dropped        int     `json:"dropped_as_present"`
	FalseConfident int     `json:"false_confident"`
	Rate           float64 `json:"rate"`
}

type RepoEntry

type RepoEntry struct {
	Name  string `json:"name"`
	Index string `json:"index"`
}

type RepoResult

type RepoResult struct {
	Self *SelfResult `json:"self,omitempty"`
	Neg  *NegResult  `json:"neg,omitempty"`
}

type Report

type Report struct {
	Generated time.Time             `json:"generated"`
	Gen       *GenResult            `json:"gen,omitempty"`
	Repos     map[string]RepoResult `json:"repos"`
}

Report is one full run, JSON-serializable so runs can be diffed.

func Run

func Run(bin string, cfg *Config, base string, holdout bool, log func(string, ...any)) (*Report, []error)

Run executes the whole suite. Component failures are reported per component rather than aborting: a missing repo must not hide the metrics that did run.

type SelfResult

type SelfResult struct {
	N     int     `json:"n"`
	Hit1  float64 `json:"hit1"`
	Hit5  float64 `json:"hit5"`
	Hit10 float64 `json:"hit10"`
	Miss  float64 `json:"miss"`
	// Hit5ByName accepts any symbol with the same short name as the target.
	// DECISION(2026-07): interface/impl pairs make exact-symbol scoring
	// unfair — the docstring lives on the trait declaration while the code
	// lives in the implementing class, and returning the impl IS the answer
	// an agent needs. os-lib read 0.23 "never retrieved" purely because
	// BasePathImpl.baseName was scored as a miss for BasePath.baseName.
	// Strict stays primary (it is the honest lower bound); this is the
	// upper bound, and a wide gap means decl/impl duplication, not failure.
	Hit5ByName float64 `json:"hit5_by_name"`
	SeedGap    float64 `json:"seed_loss_rate"`
	AvgMs      int64   `json:"avg_ms"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL