Documentation
¶
Overview ¶
Package validators provides validation mechanisms for AI model responses. It supports value matching, JSON Schema validation, LLM-based semantic equivalence validation using judge models, and trusted Docker-backed custom validators.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrCustomValidatorNotFound is returned when a selected custom validator is unavailable. ErrCustomValidatorNotFound = errors.New("custom validator not found") // ErrCustomValidatorTemplate is returned when a custom validator template cannot be compiled or rendered. ErrCustomValidatorTemplate = errors.New("invalid custom validator template") // ErrCustomValidatorExecution is returned when a custom validator cannot be run to completion. ErrCustomValidatorExecution = errors.New("custom validator execution failed") // ErrCustomValidatorResponse is returned when a custom validator emits an invalid response. ErrCustomValidatorResponse = errors.New("invalid custom validator response") )
var ( // ErrJudgeNotFound is returned when a judge configuration is not found. ErrJudgeNotFound = errors.New("judge not found") // ErrJudgeVariantNotFound is returned when a judge run variant is not found. ErrJudgeVariantNotFound = errors.New("judge run variant not found") )
var ErrUnsupportedResponseFormatValidation = errors.New("unsupported response format validation")
ErrUnsupportedResponseFormatValidation is returned when a validator cannot handle the response format.
Functions ¶
func CompileCustomValidatorTemplates ¶
func CompileCustomValidatorTemplates(cfg config.ValidatorConfig) error
CompileCustomValidatorTemplates checks that all command, environment, and template-file templates of cfg compile, without requiring any template data.
func GradeVerdict ¶ added in v0.22.0
func GradeVerdict(rules config.ValidationRules, passingVerdicts utils.ValueSet, verdict interface{}) (bool, error)
GradeVerdict grades a judge's raw verdict against the configured passing-verdicts criterion, returning whether the verdict counts as a pass. This is the single grading path used to evaluate judge verdicts of any shape (boolean, score, grade, etc.), so custom verdict formats and passing criteria go through the same logic as the built-in default. The underlying logic is generic (rules, an expected utils.ValueSet, and an actual value), so it also backs schemaValidator's exact-or-explicit-schema matching.
If passingVerdicts specifies an explicit JSON Schema (config.ExplicitSchema), verdict is validated against it directly, without normalization: an explicit schema describes the exact shape of a passing verdict, so applying rules-based normalization (e.g. case-insensitive comparison) would silently change its meaning.
Otherwise, passingVerdicts is treated as a set of literal value(s) that verdict must equal after canonicalization: both the expected value(s) and verdict are canonicalized with rules (the same canonicalization valueMatchValidator uses for ordinary task answers), then compared via a generated "const" (single value) or "enum" (multiple values) schema.
Returns (false, nil) when verdict does not conform to the schema (a legitimate grading failure, not an error), and a non-nil error only for a malformed schema.
Types ¶
type CustomValidator ¶
type CustomValidator interface {
Validator
// IsCorrectWithEnvironment evaluates a response with a trusted validator process that runs in the
// environment of the successful provider attempt; a nil environment provides no task services.
// The validator templates receive the judge-compatible validation context and metadata.
// A valid response rejecting the answer is not an error; execution and protocol failures are.
IsCorrectWithEnvironment(ctx context.Context, logger logging.Logger, rules config.ValidationRules, expected utils.ValueSet, actual providers.Result, originalPrompt string, expectedResponseFormat config.ResponseFormat, environment providers.ExecutionEnvironment, metadata tools.ExecutionTemplateData) (ValidationResult, error)
}
CustomValidator is implemented by externally executed validators that may consume attempt-scoped infrastructure.
type Factory ¶
type Factory struct {
// contains filtered or unexported fields
}
Factory creates and manages validator instances. It provides caching to improve performance.
func NewFactory ¶
func NewFactory(availableJudges []config.JudgeConfig) *Factory
NewFactory creates a validator factory without custom validators.
func NewFactoryWithCustomValidators ¶
func NewFactoryWithCustomValidators(availableJudges []config.JudgeConfig, availableCustomValidators []config.ValidatorConfig) *Factory
NewFactoryWithCustomValidators creates a validator factory with Docker custom validators.
func (*Factory) AssertCustomValidatorExists ¶
AssertCustomValidatorExists checks whether a named custom validator is configured.
func (*Factory) AssertExists ¶
func (f *Factory) AssertExists(judge config.JudgeSelector) error
AssertExists checks if a judge configuration exists for the given judge selector. Returns an error if the judge configuration does not exist.
func (*Factory) GetValidator ¶
func (f *Factory) GetValidator(ctx context.Context, rules config.ValidationRules) (Validator, error)
GetValidator returns a validator for the given validation rules. Selection: custom-validator -> CustomValidator, judge -> JudgeValidator, schema-validation -> SchemaValidator, otherwise ValueMatchValidator.
type SemanticValidationDetails ¶ added in v0.22.0
type SemanticValidationDetails struct {
// Verdict contains the raw verdict produced by the judge.
Verdict interface{}
// JudgeName identifies the judge configuration used.
JudgeName string
// Provider is the name of the AI provider that executed the judge task.
Provider string
// Variant is the name of the judge's configuration variant used.
Variant string
// VariantConfig is the effective variant configuration used by the judge.
VariantConfig config.RunConfig
}
SemanticValidationDetails identifies the judge variant that evaluated a response and the raw verdict it produced.
type ValidationResult ¶
type ValidationResult struct {
// IsCorrect indicates whether the validation passed.
IsCorrect bool
// Title provides a descriptive title for the validation type.
Title string
// Explanation provides an optional explanation of the validation result.
Explanation string
// Usage contains token usage statistics for the validation step when available.
Usage providers.Usage
// ToolCalls contains the per-invocation tool call log for the validation step when available.
ToolCalls []tools.ToolCallSummary
// Semantic contains verdict and provenance details, populated when validation was
// performed by an LLM judge; nil otherwise.
Semantic *SemanticValidationDetails
}
ValidationResult contains the result of a validation check.
type Validator ¶
type Validator interface {
// IsCorrect checks if result matches expected value using the provided validation rules.
// The originalPrompt and expectedResponseFormat provide additional context for semantic validation.
// The logger parameter allows validators to emit structured log messages during validation.
IsCorrect(ctx context.Context, logger logging.Logger, rules config.ValidationRules, expected utils.ValueSet, actual providers.Result, originalPrompt string, expectedResponseFormat config.ResponseFormat) (ValidationResult, error)
// ToCanonical normalizes value for validation using the provided validation rules.
// For string values, applies string normalization rules (case, whitespace, etc.).
// For object values, recursively normalizes all string fields within the object structure.
ToCanonical(rules config.ValidationRules, value interface{}) interface{}
// GetName returns a descriptive user-friendly name for the validator.
GetName() string
// Close cleans up any resources used by the validator.
Close(ctx context.Context) error
}
Validator verifies AI model responses.
func NewJudgeValidator ¶
func NewJudgeValidator(ctx context.Context, judgeConfig *config.JudgeConfig, judgeRunVariant config.RunConfig, availableTools []config.ToolConfig) (Validator, error)
NewJudgeValidator creates a new semantic Validator with the given judge configuration and variant. The judge provider will be initialized from the configuration and used to evaluate responses for semantic equivalence.
func NewSchemaValidator ¶ added in v0.22.0
func NewSchemaValidator() Validator
NewSchemaValidator returns a new Validator that validates the candidate answer against the single expected JSON Schema.
func NewValueMatchValidator ¶
func NewValueMatchValidator() Validator
NewValueMatchValidator returns a new Validator that checks results by exact string matching. The validator applies validation rules for case sensitivity and whitespace handling.
Source Files