Documentation
¶
Overview ¶
Package verify provides a data-driven conformance harness that certifies a provider behaves correctly before it is relied on in the catalog.
It feeds a set of canonical chat/tool requests to any client.Provider, scores each response against declared expectations (non-empty content, expected tool call, valid JSON arguments, …), and produces a report. Because it takes the Provider interface, the same suite can be run against a live endpoint or against a client.RecorderProvider replaying a recorded baseline cassette — the latter giving a cheap, deterministic regression check without burning tokens.
This certifies behavioral conformance (does the provider actually answer and call tools correctly), which is distinct from the existing parity checks that only confirm registry/wiring.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DiffBaseline ¶
DiffBaseline compares a fresh report against a recorded baseline report and returns the case IDs whose pass/fail verdict regressed (passed in baseline, failed now). This is the "did onboarding this provider change behavior" check; an empty result means no regressions.
func ToolCallF1 ¶
func ToolCallF1(results []CaseResult, _ []Case) (precision, recall, f1 float64)
ToolCallF1 computes the harmonic mean of precision and recall for tool-call behavior across a slice of CaseResults.
Precision = TP / (TP + FP) — of calls made, how many were correct Recall = TP / (TP + FN) — of calls expected, how many were made F1 = 2 * P * R / (P + R)
For each CaseResult:
TP: expected a tool call AND got the right tool (ExpectedTool && CorrectTool) FP: got a tool call but expected none (!ExpectedTool && CalledAnyTool) or got the wrong tool (ExpectedTool && CalledAnyTool && !CorrectTool) FN: expected a tool call but got none (ExpectedTool && !CalledAnyTool) TN: expected no tool call AND got none (ignored in F1)
Returns (1, 1, 1) when there are no tool-call-relevant cases (vacuous).
Types ¶
type Case ¶
type Case struct {
ID string
Messages []client.EyrieMessage
Tools []client.EyrieTool
Expect Expectation
}
Case is a single canonical request plus its expectation.
func CanonicalCases ¶
func CanonicalCases() []Case
CanonicalCases is a small, provider-neutral suite covering the behaviors hawk depends on: basic chat, deterministic content, and tool calling with valid arguments. It is intentionally minimal so it is cheap to run against a live endpoint; extend it per provider as needed.
type CaseResult ¶
type CaseResult struct {
ID string
Passed bool
Failures []string // human-readable reasons it failed
Err string // transport/provider error, if any
Latency time.Duration
ToolCalled string // first tool the model actually called, empty if none
ExpectedTool bool // case declared a ToolName expectation
CalledAnyTool bool // model emitted at least one tool call
CorrectTool bool // model called the expected tool name
}
CaseResult is the verdict for one case.
type Expectation ¶
type Expectation struct {
// NonEmptyContent requires the response to contain assistant text.
NonEmptyContent bool
// ToolName, when set, requires the model to call exactly this tool.
ToolName string
// RequiredArgs lists argument keys that must be present on the tool call.
RequiredArgs []string
// Contains lists case-insensitive substrings the content must include
// (e.g. a deterministic answer like "4").
Contains []string
}
Expectation declares what a correct response to a Case looks like.
Source Files