Documentation
¶
Index ¶
Constants ¶
const DefaultConcurrency = 5
DefaultConcurrency caps in-flight provider calls when the caller doesn't specify one (0 or negative) via RunWithConcurrency.
Variables ¶
This section is empty.
Functions ¶
func Complete ¶ added in v1.13.0
Complete performs a single one-off completion against the given model, dispatched to its provider by the same name-prefix rules Run uses. It's the building block for callers that need one LLM call outside a full eval suite run (e.g. prompt compression).
func WriteReport ¶
WriteReport writes a self-contained HTML report.
Types ¶
type Case ¶
type Case struct {
ID string `json:"id"`
Input string `json:"input"`
Expect map[string]interface{} `json:"expect"`
}
Case is a single expected output.
type Cell ¶
type Cell struct {
Model string
CaseID string
Input string
Output string
Error string
Passed bool
Failed bool
Skipped bool
Duration time.Duration
}
Cell is one model x case result.
type Report ¶
type Report struct {
Suite string
Prompt string
Models []string
Timestamp time.Time
Cells []Cell
Failed int
Total int
}
Report is a full eval run.
func Run ¶
Run executes a suite against each model concurrently and scores results, using DefaultConcurrency in-flight calls at a time.
func RunWithConcurrency ¶ added in v1.8.0
func RunWithConcurrency(ctx context.Context, p *prompt.Template, suite *Suite, models []string, concurrency int) Report
RunWithConcurrency is Run with an explicit cap on in-flight provider calls across all models/cases. Unbounded fan-out (one goroutine per model x case) hammers provider rate limits on large suites even with per-call retry, so callers should keep this at or below what their API keys' tiers allow.
Source Files