evals

type AccuracyEval ¶

type AccuracyEval struct {
	EvalName string
	Judge    model.Provider
	Rubric   string
}

AccuracyEval uses an LLM judge to score the accuracy of agent output against expected output.

func (e *AccuracyEval) Name() string

func (e *AccuracyEval) Run(ctx context.Context, actual, expected string) EvalResult

type CaseReport ¶ added in v0.10.0

type CaseReport struct {
	Input    string       `json:"input"`
	Expected string       `json:"expected"`
	Actual   string       `json:"actual"`
	Error    string       `json:"error,omitempty"`
	Results  []EvalResult `json:"results"`
	Score    float64      `json:"score"`  // mean evaluator score for this case
	Passed   bool         `json:"passed"` // all evaluators passed and the target did not error
}

CaseReport is the outcome for a single dataset case: the actual output the target produced and each evaluator's score against the expected output.

type CaseSpec ¶ added in v0.7.0

type CaseSpec struct {
	// Eval selects the evaluator: "exact_match", "contains", or "accuracy".
	Eval string `yaml:"eval"`
	// Name is an optional label; it defaults to the Eval type when empty.
	Name     string `yaml:"name"`
	Input    string `yaml:"input"`
	Expected string `yaml:"expected"`
}

CaseSpec is the YAML representation of a single eval case.

type ContainsEval ¶

type ContainsEval struct {
	EvalName string
}

ContainsEval checks if the input contains the expected substring.

func (e *ContainsEval) Name() string

func (e *ContainsEval) Run(_ context.Context, input, expected string) EvalResult

type Dataset ¶ added in v0.10.0

type Dataset struct {
	Name      string        `json:"name"`
	CreatedAt time.Time     `json:"created_at"`
	Cases     []DatasetCase `json:"cases"`
}

Dataset is a curated set of input/expected pairs used to evaluate an agent or graph. It is the "golden" corpus at the center of the eval-driven loop: capture real runs into a Dataset, then run it against a target and gate on regressions.

func CaptureFromSession(ctx context.Context, store storage.Storage, sessionID, name string) (*Dataset, error)

func LoadDataset(data []byte) (*Dataset, error)

type DatasetCase ¶ added in v0.10.0

type DatasetCase struct {
	Input    string         `json:"input"`
	Expected string         `json:"expected"`
	Metadata map[string]any `json:"metadata,omitempty"`
}

DatasetCase is a single evaluation example: an input and the expected (golden) output, plus optional provenance metadata (e.g. the source session id).

type DatasetReport ¶ added in v0.10.0

type DatasetReport struct {
	Dataset  string       `json:"dataset"`
	RanAt    time.Time    `json:"ran_at"`
	Cases    []CaseReport `json:"cases"`
	Total    int          `json:"total"`
	Passed   int          `json:"passed"`
	AvgScore float64      `json:"avg_score"`
	PassRate float64      `json:"pass_rate"`
}

DatasetReport aggregates a run of a Dataset: per-case outcomes plus the summary metrics the gate compares (mean score and pass rate).

func BaselineFrom(history []ReportSummary) *DatasetReport

func LoadReport(data []byte) (*DatasetReport, error)

type DatasetRunner ¶ added in v0.10.0

type DatasetRunner struct {
	Target     Target
	Evaluators []Eval
}

DatasetRunner executes a Dataset against a Target and scores each produced output with one or more Evaluators. It is the "run" stage of the eval loop.

func (r *DatasetRunner) Run(ctx context.Context, ds *Dataset) (*DatasetReport, error)

type Eval ¶

type Eval interface {
	// Name returns a human-readable name for this eval.
	Name() string
	// Run executes the evaluation and returns a result.
	Run(ctx context.Context, input string, expected string) EvalResult
}

Eval defines a single evaluation case.

type EvalCase ¶

type EvalCase struct {
	Eval     Eval
	Input    string
	Expected string
}

EvalCase pairs an Eval with test data.

type EvalResult ¶

type EvalResult struct {
	Name       string        `json:"name"`
	Score      float64       `json:"score"`
	Passed     bool          `json:"passed"`
	Details    string        `json:"details"`
	Latency    time.Duration `json:"latency"`
	TokensUsed int           `json:"tokens_used,omitempty"`
	Error      string        `json:"error,omitempty"`
}

EvalResult captures the outcome of a single evaluation.

type ExactMatchEval ¶

type ExactMatchEval struct {
	EvalName string
}

ExactMatchEval checks if the agent output exactly matches the expected output.

func (e *ExactMatchEval) Name() string

func (e *ExactMatchEval) Run(_ context.Context, input, expected string) EvalResult

type GateConfig ¶ added in v0.10.0

type GateConfig struct {
	// MinAvgScore fails the gate when the report's mean score is below it.
	MinAvgScore float64 `json:"min_avg_score,omitempty"`
	// MinPassRate fails the gate when the fraction of passing cases is below it.
	MinPassRate float64 `json:"min_pass_rate,omitempty"`
	// MaxRegression fails the gate when the mean score dropped from the baseline by
	// more than this amount (absolute). Ignored when no baseline is supplied.
	MaxRegression float64 `json:"max_regression,omitempty"`
}

GateConfig defines the pass/fail thresholds for an eval run. A zero value gates on nothing (always passes); set the fields relevant to your policy.

type GateResult ¶ added in v0.10.0

type GateResult struct {
	Passed  bool     `json:"passed"`
	Reasons []string `json:"reasons,omitempty"`
}

GateResult is the outcome of applying a GateConfig to a report.

func Gate(report, baseline *DatasetReport, cfg GateConfig) GateResult

func (g GateResult) String() string

type MemReportStore ¶ added in v0.10.0

type MemReportStore struct {
	// contains filtered or unexported fields
}

MemReportStore is an in-memory ReportStore for tests and ephemeral use. It partitions history by the context tenant, mirroring StorageReportStore.

func NewMemReportStore() *MemReportStore

func (m *MemReportStore) History(ctx context.Context, dataset string) ([]ReportSummary, error)

func (m *MemReportStore) SaveReport(ctx context.Context, r *DatasetReport) error

type PerformanceBaseline ¶

type PerformanceBaseline struct {
	MaxLatency      time.Duration
	MaxTotalTokens  int
	MaxPromptTokens int
}

PerformanceBaseline defines expected performance thresholds.

type PerformanceEval ¶

type PerformanceEval struct {
	EvalName string
	RunFunc  func(ctx context.Context) (latency time.Duration, promptTokens int, completionTokens int, err error)
	Baseline *PerformanceBaseline
}

PerformanceEval measures execution latency and token usage, comparing against optional baselines.

func (e *PerformanceEval) Name() string

func (e *PerformanceEval) Run(ctx context.Context, _, _ string) EvalResult

type ReliabilityEval ¶

type ReliabilityEval struct {
	EvalName      string
	ActualCalls   []ToolCallExpectation
	ExpectedCalls []ToolCallExpectation
}

ReliabilityEval verifies the agent made correct tool calls in the correct order.

func (e *ReliabilityEval) Name() string

func (e *ReliabilityEval) Run(_ context.Context, _, _ string) EvalResult

type ReportStore ¶ added in v0.10.0

type ReportStore interface {
	// SaveReport appends r's summary to the dataset's history.
	SaveReport(ctx context.Context, r *DatasetReport) error
	// History returns the dataset's run summaries in chronological (oldest-first)
	// order; empty when there is no history yet.
	History(ctx context.Context, dataset string) ([]ReportSummary, error)
}

ReportStore persists eval-run summaries so scores are queryable over time and a run can gate against its predecessor. Implementations scope history to the context tenant.

type ReportSummary ¶ added in v0.10.0

type ReportSummary struct {
	Dataset  string    `json:"dataset"`
	RanAt    time.Time `json:"ran_at"`
	AvgScore float64   `json:"avg_score"`
	PassRate float64   `json:"pass_rate"`
	Total    int       `json:"total"`
	Passed   int       `json:"passed"`
}

ReportSummary is the compact, trend-friendly record of one eval run kept in history. It is what the gate compares across runs.

type StorageReportStore ¶ added in v0.10.0

type StorageReportStore struct {
	// contains filtered or unexported fields
}

StorageReportStore persists run history as append-only checkpoints (one per run) in a storage.Storage, keyed by a tenant-scoped per-dataset session id. Using checkpoints (rather than a single rewritten record) makes each run an independent row, so there is no read-modify-write over a growing blob.

Concurrency: two runs of the same (tenant, dataset) that race compute the same next seq_num; the (session_id, seq_num) upsert means one overwrites the other's summary at that slot — a single lost run, never corruption of the whole history. Serialize concurrent runs of one dataset if that matters.

It targets the tenant-scoped backends (sqlite, postgres). The experimental adapters that do not yet enforce tenant scoping (PLAN.md P2-002 follow-up) inherit that limitation here too.

func NewStorageReportStore(store storage.Storage) *StorageReportStore

func (s *StorageReportStore) History(ctx context.Context, dataset string) ([]ReportSummary, error)

func (s *StorageReportStore) SaveReport(ctx context.Context, r *DatasetReport) error

type Suite ¶

type Suite struct {
	Name  string
	Evals []EvalCase
}

Suite is a collection of evaluations that run together.

func LoadSuite(data []byte) (*Suite, error)

func (s *Suite) Run(ctx context.Context) SuiteResult

type SuiteResult ¶

type SuiteResult struct {
	SuiteName    string        `json:"suite_name"`
	Results      []EvalResult  `json:"results"`
	TotalEvals   int           `json:"total_evals"`
	Passed       int           `json:"passed"`
	Failed       int           `json:"failed"`
	Errors       int           `json:"errors"`
	AvgScore     float64       `json:"avg_score"`
	TotalLatency time.Duration `json:"total_latency"`
}

SuiteResult aggregates results from running a suite.

func (sr *SuiteResult) Summary() string

type SuiteSpec ¶ added in v0.7.0

type SuiteSpec struct {
	Name  string     `yaml:"name"`
	Cases []CaseSpec `yaml:"cases"`
}

SuiteSpec is the YAML representation of an eval suite. It maps a declarative file to a runnable Suite so evals can be defined without writing Go.

Example:

name: greeting-suite
cases:
  - eval: exact_match
    name: exact-greeting
    input: "hello"
    expected: "hello"
  - eval: contains
    input: "the quick brown fox"
    expected: "quick"

type Target ¶ added in v0.10.0

type Target func(ctx context.Context, input string) (string, error)

Target is the system under evaluation: given a case input it returns the agent's (or graph's) actual output. It abstracts sdk/agent and engine/graph so the runner depends on neither — pass e.g. a closure over agent.Chat.

type ToolCallExpectation ¶

type ToolCallExpectation struct {
	Name string
	Args map[string]any
}

ToolCallExpectation defines an expected tool call.