Documentation
¶
Overview ¶
Package exact provides deterministic, programmatic evaluators over the typed core/content conversation and eval evidence. Every evaluator here satisfies eval.Evaluator, declares a stable Descriptor with Method eval.MethodProgrammatic, and reaches its verdict from observable facts alone — never from a model judge.
The evaluators are constructor-parameterized: each carries its own target (the required substrings, the tool name, the threshold) rather than reading it from the sample's scenario expectation. This matches the inline usage in the framework design, e.g. exact.RequiredTool("lookup_account").
Discipline honored throughout this package:
- No conversation flattening except the text evaluators' private flattenAssistantText projection, whose result never leaves the package.
- Every failure carries at least one eval.EvidenceRef that resolves to an eval.Evidence the assessment also includes, so eval.Assessment.Validate (which rejects dangling references) passes.
- Diagnostic strings never echo untrusted conversation or tool content; they carry only safe counts, closed-enum tokens, and trusted constructor arguments. Untrusted material is referenced by evidence (message index, redacted excerpt, hash, byte count), never inlined.
- A vacuously configured evaluator (for example RequiredText() with no substrings) yields an eval.Errored assessment with a config_error finding. It never passes.
- Missing required evidence yields eval.Unverified via Descriptor.CheckRequires before any evaluation — never a pass and never a fail.
Index ¶
- func ForbiddenText(substrings ...string) eval.Evaluator
- func ForbiddenTool(name string) eval.Evaluator
- func MaxDuration(limit time.Duration) eval.Evaluator
- func NoToolCall(name string) eval.Evaluator
- func RequiredText(substrings ...string) eval.Evaluator
- func RequiredTool(name string) eval.Evaluator
- func SchemaResult() eval.Evaluator
- func ToolErrorRate(opts ...RateOption) eval.Evaluator
- type RateOption
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ForbiddenText ¶
ForbiddenText returns an evaluator asserting that none of the given substrings appears in the assistant's text output. Constructing it with no substrings is a configuration error: the resulting evaluator yields Errored, never pass.
func ForbiddenTool ¶
ForbiddenTool returns an evaluator asserting that no tool-use block naming tool appears in the conversation. An empty or invalid name is a configuration error: the evaluator yields Errored, never pass.
func MaxDuration ¶
MaxDuration returns an evaluator that fails when the largest recorded timed span exceeds limit. limit must be positive.
func NoToolCall ¶
NoToolCall is an alias for ForbiddenTool, matching the framework design's spelling exact.NoToolCall("issue_refund").
func RequiredText ¶
RequiredText returns an evaluator asserting that every given substring appears somewhere in the assistant's text output. Constructing it with no substrings is a configuration error: the resulting evaluator yields Errored, never pass.
func RequiredTool ¶
RequiredTool returns an evaluator asserting that a tool-use block naming tool appears in the conversation. An empty or invalid name is a configuration error: the evaluator yields Errored, never pass.
func SchemaResult ¶
SchemaResult returns an evaluator that reports whether the sample's structured output satisfied its schema. It has no configuration and is never vacuous.
func ToolErrorRate ¶
func ToolErrorRate(opts ...RateOption) eval.Evaluator
ToolErrorRate returns an evaluator that measures the errored-tool-operation ratio. With no MaxErrorRate option it only reports the measurement and passes; with one it fails when the ratio exceeds the threshold.
Types ¶
type RateOption ¶
type RateOption func(*toolErrorRate)
RateOption configures a ToolErrorRate evaluator.
func MaxErrorRate ¶
func MaxErrorRate(r float64) RateOption
MaxErrorRate sets the maximum acceptable tool error rate. When the observed rate strictly exceeds r the evaluator fails. r must lie within [0,1]; a value outside that range (or NaN) is a configuration error surfaced as Errored.
Source Files