Documentation
¶
Overview ¶
Package common provides shared utilities for benchmark evaluation.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func JudgeAnswer ¶
func JudgeAnswer(cfg *JudgeConfig, question, expected, got string) (float64, error)
JudgeAnswer uses an LLM to judge if the answer is correct given the reference. Returns a score between 0.0 and 1.0.
Types ¶
type BenchStores ¶
BenchStores wraps a Cortex app instance for benchmark evaluation.
func NewBenchStores ¶
func NewBenchStores() (*BenchStores, error)
NewBenchStores creates an in-memory Cortex instance for benchmarking. If useEmbeddings is true, tries to connect to Ollama for vector search.
func NewBenchStoresWithEmbeddings ¶
func NewBenchStoresWithEmbeddings(cfg embedding.Config) (*BenchStores, error)
NewBenchStoresWithEmbeddings creates a bench store with embedding support.
func (*BenchStores) Close ¶
func (bs *BenchStores) Close() error
Close cleans up the benchmark database.
func (*BenchStores) EmbedQuery ¶
func (bs *BenchStores) EmbedQuery(ctx context.Context, query string) []float32
EmbedQuery generates an embedding for a search query. Returns nil if embeddings are not enabled.
func (*BenchStores) IngestSession ¶
func (bs *BenchStores) IngestSession(ctx context.Context, sessionID, project string, observations []domain.Observation) error
IngestSession creates a session and its observations. If embeddings are enabled, each observation is also embedded.
type BenchmarkResult ¶
type BenchmarkResult struct {
Benchmark string `json:"benchmark"`
Overall float64 `json:"overall_accuracy"`
ByType map[string]float64 `json:"by_type"`
Total int `json:"total_questions"`
Correct int `json:"correct"`
Details []QuestionResult `json:"details,omitempty"`
}
BenchmarkResult holds aggregated benchmark results.
func Aggregate ¶
func Aggregate(results []QuestionResult) BenchmarkResult
Aggregate computes overall and per-type accuracy from question results.
type JudgeConfig ¶
type JudgeConfig struct {
Provider string // "openai" or "anthropic"
APIKey string
Model string // e.g. "gpt-4o" or "claude-sonnet-4-20250514"
}
JudgeConfig configures the LLM-as-Judge evaluator.
func DefaultJudgeConfig ¶
func DefaultJudgeConfig() *JudgeConfig
DefaultJudgeConfig returns config from environment variables.
type QuestionResult ¶
type QuestionResult struct {
ID string `json:"id"`
Type string `json:"type"`
Query string `json:"query"`
Expected string `json:"expected"`
Got string `json:"got"`
Score float64 `json:"score"`
Correct bool `json:"correct"`
}
QuestionResult holds the result of a single question evaluation.
Source Files