Documentation
¶
Overview ¶
Package modelprobe runs a bounded capability probe against a model to check, cheaply and objectively, whether it's usable for agentic coding — and whether it's strong enough for complex work.
It has two stages:
Fast-fail gates (tier 1): a battery of single-turn checks. Each presents a system prompt, a user prompt, and a tool set, then validates only the model's FIRST response — did it call the right tool, with the right argument, in the right order, including sprout's bespoke `repo_map` tool that no model was trained on. These are quick and decisive: any miss is a fast fail (the model isn't reliably usable on this platform), and we stop without paying for the costlier complex stage.
Complex task (tier 2): only runs if every gate passes. A multi-turn discovery/analysis task over a small project seeded with distractor and trap files. The model must read a couple of files, find the real root cause, and emit a discrete, sensible TODO list (we stop once it does — we don't have it implement). The todos are captured for evaluation; passing is the signal for primary-grade complex work.
Probing spends real tokens, so callers gate by estimated per-probe cost (WithinCostBudget) and a probe count cap.
Index ¶
Constants ¶
const MaxRequestOutputTokens = 16384
MaxRequestOutputTokens caps per-request completion tokens during probing. Set with headroom so a model with a long todo list or some reasoning preamble isn't truncated mid-answer, while still (a) fitting comfortably inside the 64K-context floor and (b) keeping a single runaway response from blowing past the per-probe cost estimate the budget gate relies on.
const ProbeVersion = "gates+todos+vision-v7"
ProbeVersion identifies the probe scenario/scoring so results can be invalidated when the probe changes.
Variables ¶
This section is empty.
Functions ¶
func EstimatedCostUSD ¶
EstimatedCostUSD estimates the dollar cost of one probe run for a model with the given per-million-token input/output prices.
func IsPermanentError ¶ added in v0.17.17
IsPermanentError reports whether an error from a probe request indicates a definitive failure that will not change on retry, as opposed to a transient transport/5xx/timeout that might succeed next time. Permanent errors are classified as definitive probe failures (Passed=false, Errored=false) so they are persisted and carried forward, rather than retried every run.
HTTP status codes classified as permanent:
- 402 Payment Required (insufficient balance)
- 403 Forbidden
- 404 Not Found (model doesn't exist)
- 405 Method Not Allowed (tool calling not supported)
Missing API key errors ("environment variable X is not set") are also permanent — they require a configuration change, not a retry.
func LimitRequestTokens ¶
func LimitRequestTokens()
LimitRequestTokens caps the provider request completion-token budget for the current process via the env var the provider layer honors. Probe binaries call this once at startup.
func WithinCostBudget ¶
func WithinCostBudget(inputPerMTok, outputPerMTok float64, costKnown bool, maxPerProbe float64) (ok bool, reason string)
WithinCostBudget reports whether a model is cheap enough to probe given a per-probe dollar budget. maxPerProbe <= 0 disables the check. When set, models whose price is unknown OR whose estimated probe cost exceeds the budget are rejected — we only spend on models we can confirm are affordable.
Types ¶
type Result ¶
type Result struct {
Provider string `json:"provider"`
Model string `json:"model"`
// Passed reports the fast-fail gates (tier 1, minimum bar). Complex reports
// the discovery/analysis/todos stage (tier 2), the primary-grade signal.
Passed bool `json:"passed"`
Complex bool `json:"complex,omitempty"`
Skipped bool `json:"skipped,omitempty"`
// Errored marks an inconclusive run: a transport/5xx/timeout error prevented
// a full assessment, so this is NOT a capability verdict. Callers must not
// persist or carry it forward — the model should be re-probed next run.
Errored bool `json:"errored,omitempty"`
// Score is the combined 0..1 score: gates contribute up to 0.5, the complex
// stage the remaining 0.5.
Score float64 `json:"score"`
GateScore float64 `json:"gate_score"`
ComplexScore float64 `json:"complex_score,omitempty"`
// Todos is the model's submitted plan from the complex stage, captured
// verbatim so a human can evaluate whether it actually makes sense.
Todos string `json:"todos,omitempty"`
Vision bool `json:"vision"`
ToolCallOK bool `json:"tool_call_ok"`
Turns int `json:"turns"`
Reason string `json:"reason"`
LatencyMS int64 `json:"latency_ms"`
PromptTokens int `json:"prompt_tokens,omitempty"`
CompletionTokens int `json:"completion_tokens,omitempty"`
ProbedAt string `json:"probed_at"`
ProbeVersion string `json:"probe_version"`
}
Result is the outcome of a probe run.
func Run ¶
Run drives the probe against a client and returns a scored Result. The fast-fail gates always run; the complex stage runs only if every gate passes, so we never pay for the costly stage on a model that fails the minimum bar.
func SkippedResult ¶
SkippedResult builds a Result for a model that was not probed.
Source Files