Documentation
¶
Overview ¶
Package judgment defines provider-neutral typed evaluation contracts.
Judgment replaces free-form generation with three strict result shapes: Choice (a definitive pick plus a probability distribution over an enumerable option set), Score (a finite 0-100 rating), and Noul (a strict yes/no verdict). Providers, prompts, caches, and escalation policy stay in adapters or applications; this package owns only the typed contracts and their validation.
Index ¶
- Variables
- func ValidateChoice(request ChoiceRequest, result Choice) error
- func ValidateChoiceRequest(request ChoiceRequest) error
- func ValidateNoul(request NoulRequest, result Noul) error
- func ValidateNoulRequest(request NoulRequest) error
- func ValidateScore(request ScoreRequest, result Score) error
- func ValidateScoreRequest(request ScoreRequest) error
- type Asker
- type Choice
- type ChoiceRequest
- type Chooser
- type Judge
- type Memory
- type Noul
- type NoulRequest
- type Option
- type OptionProbability
- type Probability
- type Route
- type Routed
- type Router
- func (r *Router) Ask(ctx context.Context, scope Scope, request NoulRequest) (Routed[Noul], error)
- func (r *Router) Choose(ctx context.Context, scope Scope, request ChoiceRequest) (Routed[Choice], error)
- func (r *Router) Score(ctx context.Context, scope Scope, request ScoreRequest) (Routed[Score], error)
- func (r *Router) Thresholds() Thresholds
- type Scope
- type Score
- type ScoreRequest
- type Scorer
- type Thresholds
- type Verdict
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidPolicy = errors.New("judgment: invalid escalation policy")
ErrInvalidPolicy reports an escalation router constructed with a nil collaborator or out-of-range thresholds.
var ErrInvalidRequest = errors.New("judgment: invalid request")
ErrInvalidRequest reports a malformed judgment request.
var ErrInvalidResult = errors.New("judgment: invalid result")
ErrInvalidResult reports a malformed judgment result.
Functions ¶
func ValidateChoice ¶
func ValidateChoice(request ChoiceRequest, result Choice) error
ValidateChoice verifies a request/result pair: the request names at least two uniquely identified options, the selected option belongs to the request's option set, the distribution assigns exactly one probability in [0, 1] to every request option, the distribution sums to one within tolerance, and the selected option holds the modal (highest) probability.
Example ¶
request := ChoiceRequest{
Task: "route the incoming request",
Options: []Option{
{ID: "search"},
{ID: "summarize"},
},
}
result := Choice{
Selected: "search",
Distribution: []OptionProbability{
{Option: "search", Probability: 0.75},
{Option: "summarize", Probability: 0.25},
},
}
if err := ValidateChoice(request, result); err != nil {
fmt.Println("invalid choice:", err)
return
}
fmt.Println("selected:", result.Selected)
Output: selected: search
func ValidateChoiceRequest ¶ added in v0.14.0
func ValidateChoiceRequest(request ChoiceRequest) error
ValidateChoiceRequest rejects malformed tasks and option sets with ErrInvalidRequest. It applies the same request rules as ValidateChoice without requiring a result.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/judgment"
)
func main() {
req := judgment.ChoiceRequest{
Task: "Route user request",
Options: []judgment.Option{
{ID: "retrieve", Description: "Search indexed knowledge"},
{ID: "summarize", Description: "Summarize provided context"},
},
}
err := judgment.ValidateChoiceRequest(req)
fmt.Println(err == nil)
}
Output: true
func ValidateNoul ¶
func ValidateNoul(request NoulRequest, result Noul) error
ValidateNoul verifies a request/result pair: the request asks a question and the result carries a known verdict with a valid confidence probability.
Example ¶
request := NoulRequest{Question: "is the build green?"}
result := Noul{Verdict: VerdictYes, Confidence: 0.97}
if err := ValidateNoul(request, result); err != nil {
fmt.Println("invalid noul:", err)
return
}
fmt.Println("gate open:", result.Yes())
Output: gate open: true
func ValidateNoulRequest ¶ added in v0.14.0
func ValidateNoulRequest(request NoulRequest) error
ValidateNoulRequest rejects blank questions with ErrInvalidRequest. It applies the same request rules as ValidateNoul without requiring a result.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/judgment"
)
func main() {
req := judgment.NoulRequest{
Question: "Does the text describe database indexing?",
}
err := judgment.ValidateNoulRequest(req)
fmt.Println(err == nil)
}
Output: true
func ValidateScore ¶
func ValidateScore(request ScoreRequest, result Score) error
ValidateScore verifies a request/result pair: the request names a task and target, the value is finite and within [0, 100], and confidence is a valid probability.
func ValidateScoreRequest ¶ added in v0.14.0
func ValidateScoreRequest(request ScoreRequest) error
ValidateScoreRequest rejects blank tasks or targets with ErrInvalidRequest. It applies the same request rules as ValidateScore without requiring a result.
Example ¶
package main
import (
"fmt"
"github.com/dotcommander/reliquary/judgment"
)
func main() {
req := judgment.ScoreRequest{
Task: "Assess relevance",
Target: "SQLite provides an embedded relational database engine.",
}
err := judgment.ValidateScoreRequest(req)
fmt.Println(err == nil)
}
Output: true
Types ¶
type Asker ¶
type Asker interface {
Ask(ctx context.Context, request NoulRequest) (Noul, error)
}
Asker answers a strict yes/no question with a calibrated probability.
type Choice ¶
type Choice struct {
Selected string
Distribution []OptionProbability
}
Choice is a validated routing decision: the selected option plus a complete probability distribution over the request's option set.
type ChoiceRequest ¶
type ChoiceRequest struct {
// Task describes the decision being routed.
Task string
// Context carries the retrieved facts the judgment is based on.
Context []string
// Options enumerates the allowed targets. A choice request needs at least
// two options with unique, non-blank IDs.
Options []Option
}
ChoiceRequest asks a Chooser to pick exactly one option from Options.
type Chooser ¶
type Chooser interface {
Choose(ctx context.Context, request ChoiceRequest) (Choice, error)
}
Chooser returns a definitive pick from an enumerable option set together with a complete probability distribution over that set.
type Judge ¶
Judge is a provider-neutral evaluator implementing all three judgment primitives. Implementations may compose the individual interfaces.
type Memory ¶
type Memory interface {
// GetChoice returns the cached choice for the exact scope and request.
GetChoice(scope Scope, request ChoiceRequest) (Choice, bool)
// PutChoice caches a validated choice for the scope and request.
PutChoice(scope Scope, request ChoiceRequest, result Choice) error
// GetScore returns the cached score for the exact scope and request.
GetScore(scope Scope, request ScoreRequest) (Score, bool)
// PutScore caches a validated score for the scope and request.
PutScore(scope Scope, request ScoreRequest, result Score) error
// GetNoul returns the cached gate answer for the exact scope and
// request.
GetNoul(scope Scope, request NoulRequest) (Noul, bool)
// PutNoul caches a validated gate answer for the scope and request.
PutNoul(scope Scope, request NoulRequest, result Noul) error
}
Memory is the exact-hit judgment memory the router consults before any provider call. Get methods report whether an entry exists for exactly this scope and request; Put methods store a validated judgment for later exact hits. The judgment/memory package's Store provides this contract through a thin adapter that derives the cache key's primitive and request fingerprint.
type Noul ¶
type Noul struct {
Verdict Verdict
Confidence Probability
}
Noul is a validated gate answer: a strict verdict plus the judge's calibrated confidence in that verdict.
type NoulRequest ¶
NoulRequest asks an Asker one strict yes/no question.
type Option ¶
Option is one enumerable choice target. ID is the stable identifier that judged results reference; Description is optional supporting detail.
type OptionProbability ¶
type OptionProbability struct {
Option string
Probability Probability
}
OptionProbability is the probability mass one choice option received.
type Probability ¶
type Probability float64
Probability is a probability mass in [0, 1].
func (Probability) Valid ¶
func (p Probability) Valid() bool
Valid reports whether p is finite and within [0, 1].
type Route ¶
type Route string
Route names the path that served a judgment.
const ( // RouteCache served the judgment from exact-hit memory without any // provider call. RouteCache Route = "cache" // RouteReflex served the judgment from the reflex provider because its // confidence met the primitive's threshold. RouteReflex Route = "reflex" // RouteDeliberate served the judgment from the deliberative provider // after the reflex confidence fell below the threshold. RouteDeliberate Route = "deliberate" )
type Routed ¶
type Routed[T any] struct { // Judgment is the validated result being served. Judgment T // Route names the path that served the judgment. Route Route // Confidence is the governing confidence of the served judgment: the // selected option's probability for a choice, or the result's // calibrated confidence for a score or verdict. Confidence Probability }
Routed is a served judgment plus the routing metadata that produced it.
type Router ¶
type Router struct {
// contains filtered or unexported fields
}
Router is the cache-then-provider escalation policy for the judgment primitives. Every request is routed through memory first: an exact hit is served without provider calls, and a miss is judged by the fast reflex provider (Tier 1). When the reflex confidence falls strictly below the per-primitive threshold, the request is forwarded to the injected deliberative provider (Tier 2), whose validated answer is served instead. Escalation is one hop: a deliberative answer is served once it validates, whatever its own confidence. Fresh answers are cached for their scope; when caching is the only failure, the served judgment is returned alongside the error.
Provider results are validated before they are served, escalated, or cached, and cache hits are re-validated against the request so a corrupt entry surfaces instead of silently answering. A Router holds no mutable state and is safe for concurrent use; collaborators must be safe for their own concurrent use.
func NewRouter ¶
func NewRouter(reflex, deliberate Judge, mem Memory, thresholds Thresholds) (*Router, error)
NewRouter creates a Router over the injected providers and memory. reflex is the fast Tier-1 judge, deliberate is the heavy Tier-2 judge that below-threshold confidences forward to, and thresholds carries the per-primitive escalation policy. All collaborators are required and caller-owned; the router owns no transport, credentials, or persistent state.
func (*Router) Ask ¶
Ask routes one yes/no question through cache-then-provider escalation: an exact memory hit is served without provider calls; otherwise the reflex provider judges the question, and when the verdict's confidence falls strictly below the Noul threshold the question is forwarded to the deliberative provider, whose validated answer is served. The served answer is cached for the scope; when caching is the only failure, the routed verdict is returned alongside the error.
func (*Router) Choose ¶
func (r *Router) Choose(ctx context.Context, scope Scope, request ChoiceRequest) (Routed[Choice], error)
Choose routes one choice request through cache-then-provider escalation: an exact memory hit is served without provider calls; otherwise the reflex provider judges the request, and when the selected option's probability falls strictly below the Choice threshold the request is forwarded to the deliberative provider, whose validated answer is served. The served answer is cached for the scope; when caching is the only failure, the routed choice is returned alongside the error.
Example ¶
scope := Scope{Tenant: "demo", Schema: "v1"}
mem := newMapMemory()
reflex := &recordingJudge{choice: modalChoice(0.55)}
deliberate := &recordingJudge{choice: modalChoice(0.97)}
router, err := NewRouter(reflex, deliberate, mem, Thresholds{Choice: 0.8, Score: 0.8, Noul: 0.8})
if err != nil {
fmt.Println("router error:", err)
return
}
// The reflex confidence 0.55 falls below the choice threshold 0.8, so
// the request forwards to the deliberative provider.
first, err := router.Choose(context.Background(), scope, validChoiceRequest())
if err != nil {
fmt.Println("routing error:", err)
return
}
fmt.Println(first.Route, first.Judgment.Selected, first.Confidence)
// The identical request is now an exact cache hit.
second, _ := router.Choose(context.Background(), scope, validChoiceRequest())
fmt.Println(second.Route)
Output: deliberate search 0.97 cache
func (*Router) Score ¶
func (r *Router) Score(ctx context.Context, scope Scope, request ScoreRequest) (Routed[Score], error)
Score routes one score request through cache-then-provider escalation: an exact memory hit is served without provider calls; otherwise the reflex provider judges the request, and when the result's confidence falls strictly below the Score threshold the request is forwarded to the deliberative provider, whose validated answer is served. The served answer is cached for the scope; when caching is the only failure, the routed score is returned alongside the error.
func (*Router) Thresholds ¶
func (r *Router) Thresholds() Thresholds
Thresholds returns the router's per-primitive escalation policy.
type Scope ¶
type Scope struct {
// Tenant isolates one customer or deployment's judgments.
Tenant string
// Schema versions the judging pipeline; changing prompts, models, or
// option sets bumps the schema and invalidates older entries.
Schema string
}
Scope identifies the judgment-sharing boundary for one routing call: cached entries are shared only within one tenant and one schema version.
type Score ¶
type Score struct {
Value float64
Confidence Probability
}
Score is a validated rating: a finite value in [0, 100] plus the judge's calibrated confidence in that value.
type ScoreRequest ¶
ScoreRequest asks a Scorer to rate one target on a fixed 0-100 scale.
type Scorer ¶
type Scorer interface {
Score(ctx context.Context, request ScoreRequest) (Score, error)
}
Scorer returns a numeric 0-100 rating for one target given context.
type Thresholds ¶
type Thresholds struct {
// Choice is the minimum selected-option probability a reflex choice
// must reach to be served without deliberation.
Choice Probability
// Score is the minimum confidence a reflex score must reach.
Score Probability
// Noul is the minimum confidence a reflex verdict must reach.
Noul Probability
}
Thresholds are the per-primitive escalation policy values. A reflex judgment escalates to the deliberative provider when its confidence falls strictly below its primitive's threshold: a threshold of zero never escalates, and a threshold of one escalates every judgment short of perfect confidence. Each primitive carries its own threshold so routing policy is calibrated separately for choices, scores, and gates.
func (Thresholds) Validate ¶
func (t Thresholds) Validate() error
Validate reports whether every threshold is a valid probability.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package memory provides the exact-hit judgment memory for the Embed→Judge stack: validated judgment.Choice, judgment.Score, and judgment.Noul results cached under four-part keys (tenant, schema, primitive, request fingerprint) and evicted after caller-configured lifetimes by github.com/jellydator/ttlcache/v3.
|
Package memory provides the exact-hit judgment memory for the Embed→Judge stack: validated judgment.Choice, judgment.Score, and judgment.Noul results cached under four-part keys (tenant, schema, primitive, request fingerprint) and evicted after caller-configured lifetimes by github.com/jellydator/ttlcache/v3. |