Documentation
¶
Overview ¶
Package judge implements the structured-output model judge: an eval.Evaluator that scores a sample's conversation against a rubric by calling an inference.Client with strict structured output. It is the first eval package permitted to depend on github.com/looprig/inference.
The judge is fail-secure end to end. It builds an inference.Request whose only instructions come from the trusted rubric and whose data is the untrusted conversation (see prompt.go); it requires the model to answer through a strict output schema (see schema.go); and it re-validates the decoded answer locally before trusting it. Every path that cannot produce a well-formed, in-range, provenance-checked score returns a typed error and never an inferred pass. A model that cannot satisfy the schema yields an unsupported/malformed error, not a guessed verdict.
Index ¶
- Constants
- Variables
- func New(r rubric.Rubric, client inference.Client, template inference.Request, ...) eval.Evaluator
- type InferenceError
- type MalformedOutputError
- type MessageIndexError
- type Option
- type QuoteNotFoundError
- type QuotedEvidence
- type RequestInvalidError
- type RubricInvalidError
- type ScoreOutput
- type ScoreRangeError
- type UnsupportedStructuredOutputError
Constants ¶
const ( // ScoreSchemaRevision is the stable revision of the score schema contract. // Bumping the schema's meaning requires a new revision. ScoreSchemaRevision eval.Revision = "score/v1" // MaxReasonBytes bounds the model-authored reason. The reason is untrusted and // never echoed in a diagnostic; this bound keeps a hostile model from // ballooning the response before it is discarded. MaxReasonBytes = 4096 // MaxEvidenceQuotes bounds how many quoted-evidence entries the model may // return. It backs both the bounded evidence instruction in the prompt and the // local validation cap, so the two can never drift. MaxEvidenceQuotes = 8 // MaxQuoteBytes bounds a single evidence quote before provenance is checked. MaxQuoteBytes = 512 )
Variables ¶
var ScoreSchemaV1 = scoreSchema{ // contains filtered or unexported fields }
ScoreSchemaV1 is the version-1 score schema: the wire JSON Schema plus the authoritative local validation of score range, evidence-index bounds, bounded reason length, and quote provenance.
Functions ¶
func New ¶
func New(r rubric.Rubric, client inference.Client, template inference.Request, opts ...Option) eval.Evaluator
New returns a rubric judge as an eval.Evaluator. The template supplies the Model (structured-output-capable) and any System or sampling defaults; the judge fills the request's Messages (the untrusted conversation) and Output (the strict score schema) on each Evaluate. The returned evaluator's Descriptor uses the rubric's Name and Revision and reports Method = MethodModel.
New does not itself reject an invalid rubric or client: an invalid rubric surfaces as a typed error from Evaluate (and from Descriptor().Validate() at a runner's preflight), and a nil client surfaces as an inference failure, so the failure is contained rather than panicking at construction.
Types ¶
type InferenceError ¶
type InferenceError struct {
Cause error
}
InferenceError reports that the inference call itself failed: an unreachable provider, a transport error, a cancelled context, or an exceeded deadline. The underlying error is available via Unwrap (so callers can test for context.DeadlineExceeded) but is never rendered, since it may originate outside the process and carry untrusted content.
func (*InferenceError) Error ¶
func (e *InferenceError) Error() string
func (*InferenceError) Unwrap ¶
func (e *InferenceError) Unwrap() error
type MalformedOutputError ¶
type MalformedOutputError struct {
Reason eval.StructuredErrorReason
Cause error
}
MalformedOutputError reports that the model's structured output could not be extracted or decoded into the score schema: not valid JSON, not the schema shape, or an oversized reason or evidence set. Reason is a closed-enum classification (safe to render); the raw model text is never retained. Cause, when set, is the underlying inference extraction error, available via Unwrap.
func (*MalformedOutputError) Error ¶
func (e *MalformedOutputError) Error() string
func (*MalformedOutputError) Unwrap ¶
func (e *MalformedOutputError) Unwrap() error
type MessageIndexError ¶
MessageIndexError reports that a quoted-evidence entry addressed a message index outside the conversation. Index and Len are safe integers; no conversation content is embedded.
func (*MessageIndexError) Error ¶
func (e *MessageIndexError) Error() string
type Option ¶
type Option func(*options)
Option configures a judge built by New.
func WithMeasurementName ¶
WithMeasurementName overrides the name of the score measurement the judge produces. It defaults to the rubric's Name, so reports key the score on the rubric; a caller scoring several subjects under one rubric can disambiguate.
type QuoteNotFoundError ¶
type QuoteNotFoundError struct {
Index int
}
QuoteNotFoundError reports that a quoted-evidence entry's quote was empty, oversized, or did not appear verbatim in the message it named — a provenance failure. The offending quote is deliberately withheld: it is untrusted conversation-derived text and must not leak through a diagnostic. Only the safe message index is rendered.
func (*QuoteNotFoundError) Error ¶
func (e *QuoteNotFoundError) Error() string
type QuotedEvidence ¶
QuotedEvidence is one verbatim quote the judge cites to justify its score, addressing the conversation message it came from by index. Both fields cross the JSON boundary and are validated immediately; the quote is never rendered back into a diagnostic.
type RequestInvalidError ¶
type RequestInvalidError struct {
Cause error
}
RequestInvalidError reports that the judge assembled a request the inference layer rejected before any model was called (an invalid schema or feature combination). It is a configuration failure, not a verdict. Cause is available via Unwrap but never rendered.
func (*RequestInvalidError) Error ¶
func (e *RequestInvalidError) Error() string
func (*RequestInvalidError) Unwrap ¶
func (e *RequestInvalidError) Unwrap() error
type RubricInvalidError ¶
type RubricInvalidError struct {
Cause error
}
RubricInvalidError reports that the judge was configured with a rubric that does not validate. Cause is the rubric validation error, available via Unwrap; it is drawn from the rubric package's safe vocabulary and carries no content, but the judge still does not render it here to keep the message fixed.
func (*RubricInvalidError) Error ¶
func (e *RubricInvalidError) Error() string
func (*RubricInvalidError) Unwrap ¶
func (e *RubricInvalidError) Unwrap() error
type ScoreOutput ¶
type ScoreOutput struct {
Score float64 `json:"score"`
Reason string `json:"reason"`
Evidence []QuotedEvidence `json:"evidence"`
}
ScoreOutput is the decoded judge answer: a scalar Score, a bounded Reason, and a bounded set of QuotedEvidence. It is the immediate-narrowing target of the model's structured output; nothing else in the judge trusts raw model text.
type ScoreRangeError ¶
ScoreRangeError reports that the decoded score was non-finite or fell outside the rubric's declared range. Score, Min, and Max are safe numbers computed or bounded by the judge, so they are rendered; HasScore distinguishes a non-finite score (whose value is withheld as meaningless) from an in-band but out-of-range one.
func (*ScoreRangeError) Error ¶
func (e *ScoreRangeError) Error() string
type UnsupportedStructuredOutputError ¶
type UnsupportedStructuredOutputError struct {
Cause error
}
UnsupportedStructuredOutputError reports that the configured model cannot satisfy the structured-output contract the judge requires. The judge fails secure: it never falls back to parsing free-form text into a score. Cause is the underlying inference feature error and is available via Unwrap; the message never renders it, since it may carry model metadata.
func (*UnsupportedStructuredOutputError) Error ¶
func (e *UnsupportedStructuredOutputError) Error() string
func (*UnsupportedStructuredOutputError) Unwrap ¶
func (e *UnsupportedStructuredOutputError) Unwrap() error
Source Files