Documentation
¶
Overview ¶
Package validators provides validation mechanisms for AI model responses. It includes support for both value matching and LLM-based semantic equivalence validation using judge models.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrJudgeNotFound is returned when a judge configuration is not found. ErrJudgeNotFound = errors.New("judge not found") // ErrJudgeVariantNotFound is returned when a judge run variant is not found. ErrJudgeVariantNotFound = errors.New("judge run variant not found") )
var ErrUnsupportedResponseFormatValidation = errors.New("unsupported response format validation")
ErrUnsupportedResponseFormatValidation is returned when a validator cannot handle the response format.
Functions ¶
func GradeVerdict ¶ added in v0.22.0
func GradeVerdict(rules config.ValidationRules, passingVerdicts utils.ValueSet, verdict interface{}) (bool, error)
GradeVerdict grades a judge's raw verdict against the configured passing-verdicts criterion, returning whether the verdict counts as a pass. This is the single grading path used to evaluate judge verdicts of any shape (boolean, score, grade, etc.), so custom verdict formats and passing criteria go through the same logic as the built-in default. The underlying logic is generic (rules, an expected utils.ValueSet, and an actual value), so it also backs schemaValidator's exact-or-explicit-schema matching.
If passingVerdicts specifies an explicit JSON Schema (config.ExplicitSchema), verdict is validated against it directly, without normalization: an explicit schema describes the exact shape of a passing verdict, so applying rules-based normalization (e.g. case-insensitive comparison) would silently change its meaning.
Otherwise, passingVerdicts is treated as a set of literal value(s) that verdict must equal after canonicalization: both the expected value(s) and verdict are canonicalized with rules (the same canonicalization valueMatchValidator uses for ordinary task answers), then compared via a generated "const" (single value) or "enum" (multiple values) schema.
Returns (false, nil) when verdict does not conform to the schema (a legitimate grading failure, not an error), and a non-nil error only for a malformed schema.
Types ¶
type Factory ¶
type Factory struct {
// contains filtered or unexported fields
}
Factory creates and manages validator instances. It provides caching to improve performance.
func NewFactory ¶
func NewFactory(availableJudges []config.JudgeConfig) *Factory
NewFactory creates a new validator factory with the provided judge configurations.
func (*Factory) AssertExists ¶
func (f *Factory) AssertExists(judge config.JudgeSelector) error
AssertExists checks if a judge configuration exists for the given judge selector. Returns an error if the judge configuration does not exist.
func (*Factory) GetValidator ¶
GetValidator returns a validator for the given judge selector. If judge is enabled, returns a cached judge validator; otherwise returns a value match validator.
type SemanticValidationDetails ¶ added in v0.22.0
type SemanticValidationDetails struct {
// Verdict contains the raw verdict produced by the judge.
Verdict interface{}
// JudgeName identifies the judge configuration used.
JudgeName string
// Provider is the name of the AI provider that executed the judge task.
Provider string
// Variant is the name of the judge's configuration variant used.
Variant string
// VariantConfig is the effective variant configuration used by the judge.
VariantConfig config.RunConfig
}
SemanticValidationDetails identifies the judge variant that evaluated a response and the raw verdict it produced.
type ValidationResult ¶
type ValidationResult struct {
// IsCorrect indicates whether the validation passed.
IsCorrect bool
// Title provides a descriptive title for the validation type.
Title string
// Explanation provides an optional explanation of the validation result.
Explanation string
// Usage contains token usage statistics for the validation step when available.
Usage providers.Usage
// ToolCalls contains the per-invocation tool call log for the validation step when available.
ToolCalls []tools.ToolCallSummary
// Semantic contains verdict and provenance details, populated when validation was
// performed by an LLM judge; nil otherwise.
Semantic *SemanticValidationDetails
}
ValidationResult contains the result of a validation check.
type Validator ¶
type Validator interface {
// IsCorrect checks if result matches expected value using the provided validation rules.
// The originalPrompt and expectedResponseFormat provide additional context for semantic validation.
// The logger parameter allows validators to emit structured log messages during validation.
IsCorrect(ctx context.Context, logger logging.Logger, rules config.ValidationRules, expected utils.ValueSet, actual providers.Result, originalPrompt string, expectedResponseFormat config.ResponseFormat) (ValidationResult, error)
// ToCanonical normalizes value for validation using the provided validation rules.
// For string values, applies string normalization rules (case, whitespace, etc.).
// For object values, recursively normalizes all string fields within the object structure.
ToCanonical(rules config.ValidationRules, value interface{}) interface{}
// GetName returns a descriptive user-friendly name for the validator.
GetName() string
// Close cleans up any resources used by the validator.
Close(ctx context.Context) error
}
Validator verifies AI model responses.
func NewJudgeValidator ¶
func NewJudgeValidator(ctx context.Context, judgeConfig *config.JudgeConfig, judgeRunVariant config.RunConfig, availableTools []config.ToolConfig) (Validator, error)
NewJudgeValidator creates a new semantic Validator with the given judge configuration and variant. The judge provider will be initialized from the configuration and used to evaluate responses for semantic equivalence.
func NewSchemaValidator ¶ added in v0.22.0
func NewSchemaValidator() Validator
NewSchemaValidator returns a new Validator that checks results by exact matching or, when the expected value is an explicit JSON Schema, by schema conformance.
func NewValueMatchValidator ¶
func NewValueMatchValidator() Validator
NewValueMatchValidator returns a new Validator that checks results by exact string matching. The validator applies validation rules for case sensitivity and whitespace handling.
Source Files