Documentation
¶
Overview ¶
Package prregress implements PR-based regression evaluation: given a merged PR, check out the world *before* it landed, build a ckv index over that snapshot, hand the PR's Background to an agent, and compare the agent's plan against what the PR actually did.
Why this exists: ckv eval's recall@k / MRR metrics measure retrieval quality on synthetic queries. They do NOT show whether the system helps an agent reach the right answer end-to-end. PR-regression closes that gap — every passing PR is empirical proof that an agent armed with ckv would have proposed something close to what shipped.
Module layout (one file per concern, all in this package):
types.go — data model + fixture loader (this file) fetcher.go — gh CLI metadata fetch (PR title/body/files) checkout.go — detached git worktree at base_sha ground.go — PR diff → changed file set (truth) agent.go — Claude CLI headless plan generation score.go — LLM-as-judge + file-set F1 runner.go — flow orchestration over a Fixture
Build tag none — uses standard library + gh CLI subprocess + git. Test tag prregress_smoke for the end-to-end PR #70 test that depends on the source repo + Claude CLI + network access being available.
Index ¶
- Constants
- func ExtractBackground(body string) string
- func ExtractExpectedFiles(markdown string) []string
- func ExtractPlanSteps(markdown string) string
- func ExtractPlanSymbols(markdown string) []string
- func FetchDiff(ctx context.Context, e Entry) (string, error)
- func FileSetF1(plan, truth []string) (precision, recall, f1 float64)
- func IntentCosine(ctx context.Context, plan, reference string, e types.Embedder) (float64, error)
- func IntentScore(plan, reference string) float64
- func PlanStepsScore(planSteps, commitMessages string) float64
- func SortedFiles(in []string) []string
- func SummarizeResults(results []Result, threshold float64) string
- func SymbolF1(planSymbols, truthSymbols []string) (precision, recall, f1 float64)
- func TruthFiles(m Meta) []string
- type ChangedFile
- type DeterministicScorer
- type Entry
- type Fixture
- type JudgeScorer
- type Meta
- type Plan
- type PlanAgent
- type Result
- type RunOptions
- type Score
- type Verdict
- type Worktree
Constants ¶
const DefaultThreshold = 0.80
DefaultThreshold is the minimum LLM-judge similarity score that counts as "agent reproduced the intent of the PR."
const FixtureSchemaVersion = "1"
FixtureSchemaVersion is the YAML schema version this package reads/writes. Bump on incompatible changes.
Variables ¶
This section is empty.
Functions ¶
func ExtractBackground ¶
ExtractBackground pulls the "Background" section out of a PR description. If the section isn't present (some teams don't use the pattern), the whole body is returned as a conservative fallback — better to give the agent too much than to give it nothing.
The boundary is "the next header at the same or higher level." Practically: "### Background" runs until the next "### " or "## " or "# ". Anything beyond that header (Solution, Changes, etc.) is stripped so the agent doesn't peek at the answer.
func ExtractExpectedFiles ¶
ExtractExpectedFiles parses the "Expected Changes" section of a markdown plan and returns the file paths the agent listed.
Robust to:
- Different header levels (`##`, `###`, `**Expected Changes**`).
- Singular vs plural ("Expected File", "Expected Changes", etc.).
- Various bullet markers (`-`, `*`, `+`).
- A trailing colon and the agent's free-form description after the path.
Returned slice is deduplicated but order-preserving (first occurrence wins) so the agent's ranking is observable in logs. The caller can sort downstream when needed.
func ExtractPlanSteps ¶
ExtractPlanSteps returns the plan narrative with the "Expected Changes" tail removed so the file list does not dilute step-level token overlap (FileSetF1 already covers that axis). When no "Expected Changes" header is present, the whole markdown is returned unchanged.
func ExtractPlanSymbols ¶
ExtractPlanSymbols pulls likely code-symbol references out of a plan markdown. Order-preserving, dedup. Feeds SymbolF1 alongside the hand-curated Entry.ChangedSymbols truth.
Limitation: regex is heuristic — it picks up Type names mentioned in prose (e.g. "modify ServerConfig"), missed by AST-diff but useful as a first-pass signal. A future iteration could swap to a tree-sitter pass over fenced code blocks for precision.
func FetchDiff ¶
FetchDiff pulls the PR's unified diff via `gh pr diff`. This is the "actual code change" the judge compares the agent's plan against.
gh returns the diff as one large text blob (all files concatenated, standard git diff format). The score layer caps how much of this gets embedded in the judge prompt, so callers don't need to chunk upstream.
func FileSetF1 ¶
FileSetF1 computes precision / recall / F1 of two file path sets. Comparison is path-string equality after normalizing separators (the gh API returns POSIX paths; git on macOS/Linux already uses them). Empty truth → 0; empty plan with non-empty truth → 0/0/0.
func IntentCosine ¶
IntentCosine returns the embedding cosine similarity between two strings. Optional companion to IntentScore — runs only when the caller supplies an embedder (typically the same instance the runner uses for index build, reused for free).
Mock embedders satisfy the interface but their vectors are random hashes of the input; cosine over those is essentially noise. Callers who want a meaningful number must pass a real embedder (e.g. bgeonnx).
Returns NaN-safe value clamped to [-1, 1]. Empty inputs → 0.
func IntentScore ¶
IntentScore measures how well a plan's intent narrative overlaps with a reference intent statement (Entry.IntentGroundTruth when present, else Meta.Title). Unigram token-F1 after lowercase + alphanum/Hangul split + minimal stopword strip. Returns 0..1.
Empty plan or empty reference → 0 (the metric requires both sides). Identical token sets → 1. Disjoint → 0.
Paraphrase invariance is intentionally NOT a goal here: the determinism of token-F1 is the point — reruns produce identical numbers, which is what regression detection needs. The cosine variant (IntentCosine) handles paraphrase when an embedder is available.
func PlanStepsScore ¶
PlanStepsScore compares the plan's step narrative (markdown body sans the "Expected Changes" file list) against the PR's commit-message log. Same token-F1 algorithm as IntentScore but applied to longer strings.
Empty commitMessages → 0; the metric requires both sides. The caller is responsible for joining the multi-commit messages into one string before passing it in (the runner does this once).
func SortedFiles ¶
SortedFiles returns a stable-order copy of the file path slice. Used to keep JSON output diffable across reruns.
func SummarizeResults ¶
SummarizeResults formats a batch outcome for the CLI report. Plain text by default; JSON is the caller's responsibility (just encode the []Result slice).
func SymbolF1 ¶
SymbolF1 mirrors FileSetF1 for symbol identifiers extracted from the plan vs Entry.ChangedSymbols. Normalization: lowercase, runs of non-alphanum collapse to a single `_`, leading/trailing `_` trimmed.
Empty truthSymbols → 0/0/0 (matches FileSetF1's contract). The descriptive truth entries in the fixture ("secp256r1 precompile registration", "systemcontracts init parsers — members") will not match PascalCase plan symbols — that low recall is the intended signal until AST-diff symbol extraction lands.
func TruthFiles ¶
TruthFiles returns the canonical set of files the PR actually changed — used as the ground truth for file-set F1 scoring.
The list comes straight from gh CLI's `files` field, which mirrors what GitHub UI shows on the "Files changed" tab. Pure additions and pure deletions both count. Test files count too: if a PR ships a fix without test coverage that's a real signal an agent's plan can miss.
Why a separate function instead of just reading Meta.Files: future follow-ups will filter (e.g. exclude generated files, normalize renames). Centralizing the projection keeps the runner ignorant of those decisions.
Types ¶
type ChangedFile ¶
type ChangedFile struct {
Path string `json:"path"`
Additions int `json:"additions"`
Deletions int `json:"deletions"`
}
ChangedFile is one file the PR touched, as reported by gh CLI. Line-level diff parsing is deferred to a follow-up (symbol-set F1 would need it; file-set F1 only needs the path).
type DeterministicScorer ¶
type DeterministicScorer struct{}
DeterministicScorer computes every pure-Go metric in the Score (file-set F1, intent token-F1, symbol F1, plan-steps overlap) with NO LLM call — the binary = deterministic half of pr-regression scoring (00 §2.2). It leaves the LLM-judge fields (JudgeScore/JudgeRaw) zero; the agent/session layer injects a JudgeScorer that adds those when an LLM judge is wanted. The diff argument is accepted to satisfy JudgeScorer but is unused here (the deterministic metrics compare plan vs. PR metadata, not raw diff text).
func (DeterministicScorer) Score ¶
func (DeterministicScorer) Score(_ context.Context, e Entry, m Meta, plan Plan, _ string) (Score, error)
Score computes the deterministic plan-vs-PR metrics. Each multi-stage metric is guarded so legacy fixture rows (no IntentGroundTruth, ChangedSymbols, or Commits) emit zero/omitempty fields instead of false positives. IntentCosine is populated separately by RunEntry when a real embedder is configured (see runner.go).
type Entry ¶
type Entry struct {
ID string `yaml:"id"`
Repo string `yaml:"repo"` // owner/name
PRNumber int `yaml:"pr_number"`
SourcePath string `yaml:"source_path"` // local clone path
BaseSHA string `yaml:"base_sha"`
Threshold float64 `yaml:"threshold,omitempty"`
Notes string `yaml:"notes,omitempty"`
IntentGroundTruth string `yaml:"intent_ground_truth,omitempty"`
ChangedSymbols []string `yaml:"changed_symbols,omitempty"`
Category string `yaml:"category,omitempty"`
}
Entry is one PR's fixture row. Fields match testdata/prs.yaml; see that file's header for semantics.
IntentGroundTruth / ChangedSymbols / Category were added in the fixture expansion (4 → 12) so the multi-stage evaluation (E1 intent / E2 symbol-level location) has structured ground truth. All three are optional — legacy entries (pr69 / pr70 / pr72 / pr74) load without them and the score.go consumers fall back to file-set F1 when symbols are absent.
type Fixture ¶
Fixture is the parsed `testdata/prs.yaml` document.
func LoadFixture ¶
LoadFixture reads + validates a PR fixture YAML. Validation is fail-loud: any missing required field on any entry is an error, not a warning, because a malformed entry corrupts the regression report silently.
Portability: every Entry.SourcePath is run through os.ExpandEnv before validation so the YAML can ship with a machine-agnostic `${CKV_STABLENET_PATH}` (or any env-var) placeholder instead of a hard-coded absolute path. Unset placeholders surface as a helpful error during validate() rather than silently expanding to "".
type JudgeScorer ¶
type JudgeScorer interface {
Score(ctx context.Context, e Entry, m Meta, plan Plan, diff string) (Score, error)
}
JudgeScorer evaluates how closely a Plan's intent matches the actual PR change. The combined Score carries the deterministic file/intent/ symbol/plan-step metrics; an agent-layer implementation may also fill the LLM-judge fields (JudgeScore/JudgeRaw). The threshold gate reads JudgeScore, so a deterministic-only run (JudgeScore=0) never "passes" the LLM gate — that's intentional: LLM judging is the agent layer's job.
type Meta ¶
type Meta struct {
Title string `json:"title"`
Body string `json:"body"` // full PR description (Background + Solution + Changes)
Background string `json:"background"` // extracted: the "what's wrong" piece, agent sees this
Files []ChangedFile `json:"files"` // truth: what the PR actually changed
CommitMessages []string `json:"commit_messages,omitempty"` // E3: ground truth for plan-step decomposition (headline + body, one entry per commit)
}
Meta is the slice of GitHub PR data the harness consumes. Captured once via gh CLI and passed forward; we deliberately don't re-fetch during scoring to keep the regression deterministic across reruns.
func FetchMeta ¶
FetchMeta gets the PR's title, body, and changed-file list from GitHub via the `gh` CLI. The user must have run `gh auth login` once; we do not embed a GitHub token.
We deliberately shell out to `gh` instead of using a Go GitHub client because:
- `gh` already manages auth (token in OS keyring, scope renewal)
- it follows the user's GH_HOST / enterprise config
- one binary call is cheaper than pulling in an SDK
Returned Body is the full PR description. Background is the extracted "what's wrong" piece — that's the only thing we hand to the agent (Solution and Changes stay hidden).
type Plan ¶
type Plan struct {
Markdown string `json:"markdown"`
ExpectedFiles []string `json:"expected_files"`
}
Plan is what the agent produced after seeing only Background. Markdown is the free-form rationale; ExpectedFiles is the structured section the agent is asked to emit at the end (parser pulls it back out for file-set F1 scoring).
type PlanAgent ¶
type PlanAgent interface {
Generate(ctx context.Context, e Entry, m Meta, hints []query.Hit) (Plan, error)
}
PlanAgent turns a problem description (PR Background) + ckv search hits into a markdown implementation plan. The agent never sees the PR's Solution / Changes — that's the gap we're measuring.
The interface is the seam where LLM work crosses out of this binary (00 §2.2: binary = deterministic). The ckv binary ships NO concrete PlanAgent — plan generation is inherently an LLM task, so the agent/session layer injects its own implementation. The plan it returns is parsed back into structured form by ExtractExpectedFiles, which stays here as a deterministic, tested parser.
type Result ¶
type Result struct {
Entry Entry `json:"entry"`
Meta Meta `json:"meta"`
Plan Plan `json:"plan"`
Score Score `json:"score"`
Pass bool `json:"pass"` // judge_score >= entry.threshold
Error string `json:"error,omitempty"` // first-error wins; downstream stages skip
}
Result is one Entry's outcome. Persisted to disk for report generation.
func RunEntry ¶
func RunEntry(ctx context.Context, e Entry, opts *RunOptions) Result
RunEntry executes the six-stage flow for one PR. Errors from any stage are folded into Result.Error; later stages skip but the pre-computed parts of Score / Plan / Meta still appear in the report so the user can see how far the pipeline got.
- fetch PR metadata (gh CLI)
- create detached git worktree at base_sha
- ckv build over the worktree
- ckv query (Background as the intent) → top-K hints
- agent generates plan (Background + hints)
- fetch PR diff + score (plan vs diff)
func RunFixture ¶
RunFixture executes RunEntry for every PR in fx. Results come back in the same order as fx.PRs. Individual failures don't abort the batch — each Result captures its own Error string.
type RunOptions ¶
type RunOptions struct {
// Embedder is used to build the per-PR index and answer hint
// queries. Required.
Embedder types.Embedder
// EmbedderName is the value Embedder.Name() returns, captured so
// post-run reports can label which model produced the score.
EmbedderName string
// Agent generates the implementation plan from PR Background +
// search hints. REQUIRED — the binary ships no concrete PlanAgent
// (plan generation is inherently LLM work, injected by the agent
// layer). fill() errors when nil.
Agent PlanAgent
// Scorer grades the plan against the PR. Defaults to
// DeterministicScorer{} (file/intent/symbol/plan-step metrics, no
// LLM). The agent layer may inject an LLM-judge scorer instead.
Scorer JudgeScorer
// TopK is how many ckv hits to hand the agent as "candidates worth
// looking at." Defaults to 10.
TopK int
// IndexBase is the parent directory under which per-PR ckv index
// dirs are created. Defaults to $TMPDIR. Each entry creates
// <IndexBase>/ckv-prregress-<id>-<sha[:12]>/index/.
IndexBase string
}
RunOptions configure how RunFixture / RunEntry operate. The caller must supply an Embedder and a PlanAgent (plan generation is LLM work, excised from the binary per 00 §2.2). Scorer defaults to the deterministic metric scorer; top-K defaults to 10; the index is built into a temp dir beside the worktree.
type Score ¶
type Score struct {
JudgeScore float64 `json:"judge_score"` // 0..1, LLM rubric output (E3 + E4 combined, legacy)
JudgeRaw string `json:"judge_raw,omitempty"`
JudgeError string `json:"judge_error,omitempty"`
FileF1 float64 `json:"file_f1"`
FilePrecision float64 `json:"file_precision"`
FileRecall float64 `json:"file_recall"`
PlanFiles []string `json:"plan_files"`
TruthFiles []string `json:"truth_files"`
// Multi-stage E1/E2/E3 metrics
IntentScore float64 `json:"intent_score,omitempty"` // E1: pure-Go token-F1 vs IntentGroundTruth || Title
IntentCosine float64 `json:"intent_cosine,omitempty"` // E1 (optional): embedder cosine, populated only when RunOptions.Embedder is real
IntentError string `json:"intent_error,omitempty"` // E1 cosine subprocess error, if any
SymbolPrecision float64 `json:"symbol_precision,omitempty"` // E2
SymbolRecall float64 `json:"symbol_recall,omitempty"` // E2
SymbolF1 float64 `json:"symbol_f1,omitempty"` // E2
PlanSymbols []string `json:"plan_symbols,omitempty"` // E2 evidence (extracted from plan)
TruthSymbols []string `json:"truth_symbols,omitempty"` // E2 evidence (= Entry.ChangedSymbols)
PlanStepsScore float64 `json:"plan_steps_score,omitempty"` // E3
}
Score combines the LLM-as-judge score and file-set F1 with the multi-stage decomposition (E1 intent / E2 symbol-level location / E3 plan-step decomposition). All new fields are omitempty so legacy fixture rows without IntentGroundTruth / ChangedSymbols / commit data still produce a clean Score with only the original axes populated.
type Verdict ¶
Verdict is the parsed judge JSON. Score is float [0..1].
func ExtractJudgeVerdict ¶
ExtractJudgeVerdict pulls the first {score, rationale} object out of LLM output. Tolerant to:
- ``` fences (json or plain)
- leading/trailing prose ("Here's my analysis: { ... }")
- score out of range — clamped to [0, 1], not rejected
Returns ok=false if no parseable object found.
type Worktree ¶
type Worktree struct {
Path string // absolute path of the worktree directory
BaseSHA string // resolved full SHA (in case the fixture used a prefix)
// contains filtered or unexported fields
}
Worktree is a detached `git worktree` pointing at base_sha. The original clone's checkout / branch / HEAD is not touched, so the user can keep working on whatever they had checked out.
func CreateWorktree ¶
CreateWorktree spins up a detached worktree at base_sha. Returns a Worktree handle whose Close() removes the worktree (the underlying commits stay in the clone's objects/).
The worktree path is deterministic per (sourcePath, baseSHA), so reruns reuse the same directory and avoid unbounded /tmp growth. We do NOT cache across runs to avoid "stale index" surprises — the caller deletes it via Close() on every eval.
Source Files