Documentation
¶
Overview ¶
Package confidence provides pre-dispatch heuristic scoring for tool-use inputs. The agent uses it to flag tool calls where the model has likely guessed a required parameter — empty strings, placeholder values, "TODO"/"FIXME" tokens, obvious defaults — so the dispatch layer can abstain and ask for clarification instead of acting on shaky arguments.
This is separate from the chain-of-verification (verify_build.go / verifyPayload): verify runs AFTER a tool produces content, checking the output. Confidence runs BEFORE dispatch, checking the input. The two cover different failure modes — verify catches "we built the wrong thing", confidence catches "we're about to act on uncertain inputs".
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ShouldAbstainAt ¶ added in v0.54.0
ShouldAbstainAt is the per-classifier abstention check. A nil or zero threshold falls back to DefaultClassifierThreshold. Returns true when score is strictly less than the resolved threshold.
Used by the agent at vision and router call sites to decide whether to act on the classifier output or route to a clarifying user question (vision) / full-catalog fallback (router).
Out-of-range thresholds are clamped: <=0 falls back to the default (a misconfigured persona shouldn't disable the gate), and >1 clamps to 1.0. The >1 clamp enforces the Persona.Confidence docstring's "can't push the agent into always-abstain territory" promise — without it a persona with `confidence: { router: 2.0 }` would make every score (which clampScore caps at 1.0) strictly less than the threshold and force abstention on every classifier call, defeating the dynamic-catalog / vision-abstention features the operator presumably wanted to tune more aggressively.
Types ¶
type ClassifierKind ¶ added in v0.54.0
type ClassifierKind string
ClassifierKind names a classifier surface that can carry a confidence signal. Strings are stable so they round-trip through the persona YAML's `confidence:` map without translation.
const ( // KindVision is the analyze_image / vision-tool classifier. KindVision ClassifierKind = "vision" // KindRouter is the per-turn tool-group router (see // internal/agent/router.go). Below-threshold confidence routes // to the full-catalog fallback rather than acting on a guess. KindRouter ClassifierKind = "router" )
type Report ¶
type Report struct {
Score Score
MissingKeys []string
WeakKeys []string // present but placeholder-like
Reason string
}
Report captures the outcome of a confidence evaluation.
func Evaluate ¶
Evaluate scores tool-input params against the required key list. Required keys that are missing or placeholder-like subtract weight proportional to their share of the required set. Optional keys are not scored — the caller already opted to treat them as optional.
With no required keys, the score is always 1.0 (nothing to fail on).
func (Report) ShouldAbstain ¶
ShouldAbstain is true when Score is below AbstainThreshold.
type Score ¶
type Score float64
Score is a confidence value in [0.0, 1.0]. 1.0 = fully grounded inputs, every required field carries a concrete value. 0.0 = the input is unsalvageable (empty required fields, obvious placeholders throughout). The default abstention threshold is 0.5; callers override via Dispatch.
const AbstainThreshold Score = 0.5
Threshold below which dispatch should abstain. Tuned against the adversarial eval scenarios; a future golden query set will let us re-calibrate without touching the heuristics themselves.
const DefaultClassifierThreshold Score = 0.5
DefaultClassifierThreshold is used when a persona doesn't override a per-classifier threshold. 0.5 is the historical default applied to the input-grounding AbstainThreshold sibling and reuses the same operator intuition: "anything south of 50 % is a guess; ask".
func ParseClassifierResponse ¶ added in v0.54.0
ParseClassifierResponse extracts a confidence score from an LLM classifier response. The model may return any of:
- a JSON object with a top-level `confidence` field, e.g. `{"groups":[...],"confidence":0.82}` or `{"answer":"…","confidence":0.4}`.
- a JSON array (the older router shape) — no confidence signal, returns ok=false and confidence=1.0 (treat as "model did not opt in to abstention; act on the response").
- free-text prose — no JSON parse, ok=false, confidence=1.0.
Returns (confidence, ok). The bool reports whether the response actually carried a confidence field; a false ok is the contract for "no signal" — the caller should NOT treat that as low confidence, because doing so would penalise the historical no-confidence callers and effectively force abstention everywhere during a roll-out.
Score is clamped to [0, 1] regardless of what the model emits, so a malicious or buggy classifier can't push the agent into either always-abstain (negative) or never-abstain (>1) territory.
Source Files