Documentation
¶
Overview ¶
Package llm wraps the classification model behind a narrow interface so the rest of flakehunter never depends on a specific vendor.
Index ¶
Constants ¶
const ( ProviderOllama = "ollama" ProviderClaude = "claude" ProviderDeterministic = "deterministic" )
Known provider identifiers.
const BaselineDisclaimer = "rule-based baseline (no language model involved)"
BaselineDisclaimer is attached to every verdict this provider produces and is printed by the eval and report commands.
It exists because a number produced by this provider must never be mistaken for a measurement of a language model. Reporting rule-based accuracy as if it were LLM accuracy would misrepresent the tool's headline metric.
const DefaultClaudeModel = "claude-sonnet-4-6"
DefaultClaudeModel is the model used when --model is not given.
const DefaultOllamaModel = "llama3"
DefaultOllamaModel is a small instruct model that fits on a laptop.
const DefaultOllamaURL = "http://localhost:11434"
DefaultOllamaURL is where Ollama listens out of the box.
const RepairPrompt = `` /* 250-byte string literal not displayed */
RepairPrompt is appended on the single retry after a malformed response. It restates only the format requirement, since the analysis itself may have been fine and only the envelope was wrong.
const SchemaJSON = `` /* 337-byte string literal not displayed */
SchemaJSON is the contract shown to the model in the prompt.
const SystemPrompt = `` /* 1829-byte string literal not displayed */
SystemPrompt frames the model's role and, critically, forbids invented evidence.
The citation rule is the load-bearing instruction. A verdict that quotes the log reads as authoritative, so a fabricated quote is worse than no quote at all — it manufactures false confidence. The instruction is reinforced by verdict.VerifyCitations, which drops any citation not literally present; prompting alone is not a guarantee, only a first line of defence.
Variables ¶
var ErrMalformed = errors.New("malformed model response")
ErrMalformed marks a response that could not be parsed into the schema.
Functions ¶
func BuildPrompt ¶
BuildPrompt renders the full user prompt for a request.
func ParseVerdict ¶
ParseVerdict extracts and validates a verdict from a raw model response.
Models wrap JSON in prose or fenced code blocks often enough that demanding a bare object would fail constantly, so the object is located inside the response rather than assumed to be the whole of it. Anything that still does not satisfy the schema is rejected — the caller retries once and then gives up honestly rather than guessing.
Types ¶
type Claude ¶
type Claude struct {
// contains filtered or unexported fields
}
Claude talks to the Anthropic Messages API.
Opt-in rather than default: it needs a key, it costs money per classification, and it ships CI logs off the machine. Worth it when accuracy matters more than those three things.
func NewClaude ¶
NewClaude builds a Claude provider, reading the key from options or the ANTHROPIC_API_KEY environment variable.
type Deterministic ¶
type Deterministic struct {
// contains filtered or unexported fields
}
Deterministic is a transparent keyword-and-pattern classifier.
It is NOT a language model and is not a substitute for one. It exists for two legitimate reasons:
- Tests and CI need a provider that is offline, instant and reproducible.
- Evaluation needs a control. "The model scored 82%" is meaningless on its own; "the model scored 82% where grep scores 71%" is a result. A classifier that cannot beat its own baseline is not earning its inference cost.
Its verdicts are always marked with BaselineDisclaimer so downstream output cannot present them as model output.
func NewDeterministic ¶
func NewDeterministic() *Deterministic
NewDeterministic builds the baseline classifier.
func (*Deterministic) Classify ¶
Classify scores the excerpt against every rule and returns the winning category, with confidence derived from the margin over the runner-up.
func (*Deterministic) Model ¶
func (d *Deterministic) Model() string
Model identifies the "model" — there isn't one, and the string says so.
type Ollama ¶
type Ollama struct {
// contains filtered or unexported fields
}
Ollama talks to a local Ollama daemon.
This is flakehunter's default provider on purpose: it needs no API key, sends no CI logs to a third party, and lets anyone evaluating the tool run the full pipeline immediately. CI logs routinely contain internal hostnames and occasionally leaked secrets, which is a real argument for local inference beyond convenience.
type Options ¶
type Options struct {
Provider string
Model string
BaseURL string
APIKey string
// Temperature is pinned low by callers: classification wants repeatability,
// not creativity.
Temperature float64
}
Options configures provider construction.
type Provider ¶
type Provider interface {
// Name identifies the provider in reports and eval output.
Name() string
// Model identifies the specific model in use.
Model() string
// Classify returns a validated verdict for the excerpt.
Classify(ctx context.Context, req Request) (verdict.Verdict, error)
}
Provider classifies a single flaky-failure excerpt.
The interface is deliberately one method: everything flakehunter needs from a model is "read this log, return this schema". Keeping it that narrow is what makes the deterministic provider a drop-in for evaluation, and what lets a new vendor be added without touching the pipeline.
Source Files