Documentation
¶
Index ¶
Constants ¶
const ( MaxHeldOut = 3 MaxIterations = 5 TargetScore = 80.0 )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Benchmarker ¶
type Benchmarker struct {
// contains filtered or unexported fields
}
Benchmarker validates persona quality by generating dry-run reviews and comparing them against held-out originals.
func New ¶
func New(provider llm.Provider) *Benchmarker
New returns a Benchmarker that uses the given LLM provider.
func (*Benchmarker) Run ¶
func (b *Benchmarker) Run(ctx context.Context, persona *analyzer.Persona, heldOut []HeldOutReview) (*Result, *analyzer.Persona, error)
Run performs the benchmark loop: for each iteration it generates dry-run reviews using the persona, compares them with the originals, scores the match, and refines the persona if the score is below the target. It runs at most MaxIterations times. Returns the benchmark result and a potentially refined persona.
type HeldOutReview ¶
HeldOutReview is a review comment withheld from persona building for validation.
func SplitReviews ¶
func SplitReviews(data *ghcrawl.CrawlResult, max int) []HeldOutReview
SplitReviews removes up to max reviews that have non-empty DiffHunks from data and returns them as held-out test samples. It modifies data.Repos in place so the held-out reviews are not visible during persona analysis.
type IterationResult ¶
type IterationResult struct {
Iteration int
Score float64
Feedback string
Pairs []ReviewPair
}
IterationResult holds the outcome of a single benchmark iteration.
type Result ¶
type Result struct {
FinalScore float64
Iterations int
History []IterationResult
}
Result holds the overall benchmark outcome.
Source Files