Documentation
¶
Overview ¶
Package text evaluates generated text without imposing one shared sample on metrics with different semantic inputs.
Index ¶
Constants ¶
const MetricAnswerRelevance eval.MetricName = "answer_relevance"
MetricAnswerRelevance identifies whether output addresses its input.
const MetricCorrectness eval.MetricName = "correctness"
MetricCorrectness identifies agreement with an explicit reference answer.
const MetricGroundedness eval.MetricName = "groundedness"
MetricGroundedness identifies support from supplied evidence.
Variables ¶
var ErrInvalidSample = errors.New("eval/text: invalid sample")
ErrInvalidSample identifies missing generated-text inputs or evidence.
Functions ¶
This section is empty.
Types ¶
type AnswerRelevanceEvaluator ¶
type AnswerRelevanceEvaluator struct {
// contains filtered or unexported fields
}
AnswerRelevanceEvaluator scores whether generated output addresses its originating input. Groundedness is intentionally evaluated separately.
func NewAnswerRelevanceEvaluator ¶
func NewAnswerRelevanceEvaluator(config ModelEvaluatorConfig) (*AnswerRelevanceEvaluator, error)
NewAnswerRelevanceEvaluator binds the text-specific prompt to the generic model judge.
func (*AnswerRelevanceEvaluator) Evaluate ¶
func (a *AnswerRelevanceEvaluator) Evaluate(ctx context.Context, sample AnswerRelevanceSample) (eval.Report, error)
type AnswerRelevanceSample ¶
AnswerRelevanceSample relates generated output to the input it should answer.
func (AnswerRelevanceSample) Validate ¶
func (a AnswerRelevanceSample) Validate() error
type CorrectnessEvaluator ¶
type CorrectnessEvaluator struct {
// contains filtered or unexported fields
}
CorrectnessEvaluator scores generated output against an explicit reference.
func NewCorrectnessEvaluator ¶
func NewCorrectnessEvaluator(config ModelEvaluatorConfig) (*CorrectnessEvaluator, error)
NewCorrectnessEvaluator binds the reference-aware prompt to the generic model judge.
func (*CorrectnessEvaluator) Evaluate ¶
func (c *CorrectnessEvaluator) Evaluate(ctx context.Context, sample CorrectnessSample) (eval.Report, error)
type CorrectnessSample ¶
type CorrectnessSample struct {
Input string `json:"input"`
Output string `json:"output"`
Reference string `json:"reference"`
}
CorrectnessSample supplies an explicit reference rather than treating retrieved evidence as ground truth.
func (CorrectnessSample) Validate ¶
func (c CorrectnessSample) Validate() error
type GroundednessEvaluator ¶
type GroundednessEvaluator struct {
// contains filtered or unexported fields
}
GroundednessEvaluator scores whether generated output is supported by the supplied evidence.
func NewGroundednessEvaluator ¶
func NewGroundednessEvaluator(config ModelEvaluatorConfig) (*GroundednessEvaluator, error)
NewGroundednessEvaluator binds the evidence-aware prompt to the generic model judge.
func (*GroundednessEvaluator) Evaluate ¶
func (g *GroundednessEvaluator) Evaluate(ctx context.Context, sample GroundednessSample) (eval.Report, error)
type GroundednessSample ¶
type GroundednessSample struct {
Output string `json:"output"`
Evidence []string `json:"evidence"`
}
GroundednessSample keeps evidence separate from generated output so support is not conflated with answer relevance.
func (GroundednessSample) Clone ¶
func (g GroundednessSample) Clone() GroundednessSample
func (GroundednessSample) EvidenceText ¶
func (g GroundednessSample) EvidenceText() string
func (GroundednessSample) Validate ¶
func (g GroundednessSample) Validate() error
type ModelEvaluatorConfig ¶
type ModelEvaluatorConfig struct {
Model chat.Model
PromptTemplate *chatclient.Template
// Threshold is optional. Without one, evaluation produces a score without
// inventing a pass/fail decision.
Threshold *eval.Score
Samples int
}
ModelEvaluatorConfig configures model-backed text metrics. Each evaluator exposes only the prompt variables its own sample contains. Samples greater than one use the median judge score.
Source Files