Documentation
¶
Index ¶
Constants ¶
View Source
const DefaultConcurrency = 5
DefaultConcurrency caps in-flight provider calls when the caller doesn't specify one (0 or negative) via RunWithConcurrency.
Variables ¶
This section is empty.
Functions ¶
func WriteReport ¶
WriteReport writes a self-contained HTML report.
Types ¶
type Case ¶
type Case struct {
ID string `json:"id"`
Input string `json:"input"`
Expect map[string]interface{} `json:"expect"`
}
Case is a single expected output.
type Cell ¶
type Cell struct {
Model string
CaseID string
Input string
Output string
Error string
Passed bool
Failed bool
Skipped bool
Duration time.Duration
}
Cell is one model x case result.
type Report ¶
type Report struct {
Suite string
Prompt string
Models []string
Timestamp time.Time
Cells []Cell
Failed int
Total int
}
Report is a full eval run.
func Run ¶
Run executes a suite against each model concurrently and scores results, using DefaultConcurrency in-flight calls at a time.
func RunWithConcurrency ¶ added in v1.8.0
func RunWithConcurrency(ctx context.Context, p *prompt.Template, suite *Suite, models []string, concurrency int) Report
RunWithConcurrency is Run with an explicit cap on in-flight provider calls across all models/cases. Unbounded fan-out (one goroutine per model x case) hammers provider rate limits on large suites even with per-call retry, so callers should keep this at or below what their API keys' tiers allow.
Source Files
Click to show internal directories.
Click to hide internal directories.