Help us shape the future of Go! Take the 2026 Go Developer Survey

eval

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 31, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package eval evaluates agents and other LLM tasks against datasets of cases.

A Dataset holds Cases, each an input with an optional expected output and metadata. Dataset.Evaluate runs a Task on every case, with bounded concurrency, repetitions and timeouts, and scores each output with Evaluators. A failing or panicking task or evaluator affects only its own case. The resulting Report aggregates pass rates, scores, token usage and durations, prints as a table or JSON, and compares with a baseline report. Run does the same inside a Go test and fails it below a threshold.

The model follows pydantic-evals (https://pydantic.dev/docs/ai/evals/): evaluators return assertions (pass or fail), scores and labels, each with an optional reason. Tool trajectories are matched as in Google ADK (https://adk.dev/evaluate/criteria/), and SimulateUser drives multi-turn conversations with a simulated user as ADK's user simulation does (https://adk.dev/evaluate/user-sim/). Spans recorded with Tracer support span-based evaluation.

Package eval/replay records and replays HTTP interactions, so provider tests and evaluations run deterministically without network access.

Index

Constants

View Source 
const DefaultConcurrency = 8

DefaultConcurrency bounds the case runs in progress when Concurrency is not set.

View Source 
const DefaultMaxInvocations = 20

DefaultMaxInvocations bounds the user messages of a simulated conversation, as ADK's max_allowed_invocations does.

View Source 
const DefaultStopSignal = "</finished>"

DefaultStopSignal is the reply with which the simulated user ends a conversation.

View Source 
const MetadataToolCalls = "tool_calls"

MetadataToolCalls is the metadata key from which ToolTrajectory reads the expected calls of a case when it is given none, so datasets loaded from files can hold them: a list of objects with "name" and optional "args", like ToolCall in JSON.

Variables

This section is empty.

Functions

func Rouge1 

func Rouge1(candidate, reference string) float64

Rouge1 returns the ROUGE-1 F1 measure of candidate against reference.

func Text 

func Text(v any) string

Text returns the text of an output for evaluators that compare text: a string as is, a byte slice as a string, a fmt.Stringer formatted, and other values as JSON.

func Tracer 

func Tracer() telemetry.Tracer

Tracer returns a tracer that records the spans of each case run, for span-based evaluation. Trace the agent under test with it, for example with agent.WithMiddleware(middleware.Tracing(eval.Tracer())), and the spans of each run appear in EvaluatorContext.Spans. Outside an evaluation, it discards spans.

Types

type Case 

type Case[In, Out any] struct {
	// Name identifies the case in reports. Names are unique within a
	// dataset; Evaluate names unnamed cases "case N", counting from 1.
	Name  string `json:"name,omitempty"`
	Input In     `json:"input"`
	// Expected is the expected output, or nil if the case has none.
	// Evaluators that need it, such as EqualsExpected, skip cases without
	// one.
	Expected *Out `json:"expected,omitempty"`
	// Metadata is free-form data for evaluators, such as a difficulty or
	// a category.
	Metadata map[string]any `json:"metadata,omitempty"`
	// Evaluators run on this case in addition to the dataset's.
	Evaluators []Evaluator `json:"-"`
}

Case is one scenario of a dataset.

type CaseChange 

type CaseChange string

CaseChange classifies how a case changed between two reports. 

const (
	// CaseRegressed means the candidate passed a smaller share of the
	// case's runs.
	CaseRegressed CaseChange = "regressed"
	// CaseImproved means the candidate passed a larger share.
	CaseImproved  CaseChange = "improved"
	CaseUnchanged CaseChange = "unchanged"
	// CaseAdded and CaseRemoved mark cases only in the candidate or only
	// in the baseline.
	CaseAdded   CaseChange = "added"
	CaseRemoved CaseChange = "removed"
)

type CaseDiff 

type CaseDiff struct {
	Name string `json:"name"`
	// Baseline and Candidate are nil for a case missing from the report.
	Baseline  *CaseSummary `json:"baseline,omitempty"`
	Candidate *CaseSummary `json:"candidate,omitempty"`
	Change    CaseChange   `json:"change"`
}

CaseDiff compares one case between two reports.

type CaseResult 

type CaseResult struct {
	Name string `json:"name"`
	// Repeat numbers the run of the case, from 0.
	Repeat   int            `json:"repeat,omitempty"`
	Input    any            `json:"input"`
	Expected any            `json:"expected,omitempty"`
	Output   any            `json:"output,omitempty"`
	Metadata map[string]any `json:"metadata,omitempty"`
	// Evaluations are the results of the evaluators, in order.
	Evaluations []Evaluation `json:"evaluations,omitempty"`
	// EvaluatorErrors lists the evaluators that failed.
	EvaluatorErrors []EvaluatorError `json:"evaluator_errors,omitempty"`
	// Error is the error of the task; its evaluators did not run.
	Error string `json:"error,omitempty"`
	// Status, Turns and Usage come from the task's run result, if any.
	Status   loop.Status   `json:"status,omitempty"`
	Turns    int           `json:"turns,omitempty"`
	Usage    model.Usage   `json:"usage,omitzero"`
	Duration time.Duration `json:"duration"`
}

CaseResult is the outcome of one run of a case.

func (*CaseResult) Failures 

func (c *CaseResult) Failures() []string

Failures describes why the run did not pass.

func (*CaseResult) Passed 

func (c *CaseResult) Passed() bool

Passed reports whether the run passed: the task succeeded, every evaluator ran and every assertion passed.

type CaseSummary 

type CaseSummary struct {
	Name     string  `json:"name"`
	Runs     int     `json:"runs"`
	Passed   int     `json:"passed"`
	PassRate float64 `json:"pass_rate"`
	// Scores is the mean of each score, by name.
	Scores       map[string]float64 `json:"scores,omitempty"`
	Usage        model.Usage        `json:"usage"`
	MeanDuration time.Duration      `json:"mean_duration"`
}

CaseSummary aggregates the runs of one case.

type Comparison 

type Comparison struct {
	Baseline  string `json:"baseline"`
	Candidate string `json:"candidate"`
	PassRate  Delta  `json:"pass_rate"`
	// Assertions and Scores compare the assertion pass rates and the mean
	// scores, by name, of those in either report; one missing from a
	// report is zero there.
	Assertions   map[string]Delta `json:"assertions,omitempty"`
	Scores       map[string]Delta `json:"scores,omitempty"`
	TotalTokens  Delta            `json:"total_tokens"`
	MeanTokens   Delta            `json:"mean_tokens"`
	MeanDuration DurationDelta    `json:"mean_duration"`
	// Cases lists the cases of the candidate in order, then those only in
	// the baseline.
	Cases []CaseDiff `json:"cases"`
}

Comparison compares a candidate report with a baseline, for example a new prompt or model with the current one.

func Compare 

func Compare(baseline, candidate *Report) *Comparison

Compare compares candidate with baseline. Cases are matched by name.

func (*Comparison) Regressions 

func (c *Comparison) Regressions() []CaseDiff

Regressions returns the cases that regressed.

func (*Comparison) String 

func (c *Comparison) String() string

String returns the comparison as WriteText prints it.

func (*Comparison) WriteText 

func (c *Comparison) WriteText(w io.Writer) error

WriteText prints the comparison: the aggregate changes followed by a table of the cases.

type Dataset 

type Dataset[In, Out any] struct {
	Name  string          `json:"name,omitempty"`
	Cases []Case[In, Out] `json:"cases"`
	// Evaluators run on every case.
	Evaluators []Evaluator `json:"-"`
}

Dataset is a named collection of cases for one task.

func LoadDataset 

func LoadDataset[In, Out any](path string) (*Dataset[In, Out], error)

LoadDataset reads a dataset file in the format FormatOf reports. A dataset without a name is named after the file, without its extension. Unknown fields of cases, such as a misspelled "expected", are errors.

A JSON file looks like this; JSON Lines files hold one case object per line: 

{
  "name": "capitals",
  "cases": [
    {"name": "france", "input": "Capital of France?", "expected": "Paris",
     "metadata": {"difficulty": "easy"}}
  ]
}

func ReadDataset 

func ReadDataset[In, Out any](r io.Reader, f Format) (*Dataset[In, Out], error)

ReadDataset reads a dataset in format f.

func (*Dataset[In, Out]) Evaluate 

func (d *Dataset[In, Out]) Evaluate(ctx context.Context, task Task[In, Out], opts ...Option) (*Report, error)

Evaluate runs task on every case of the dataset and evaluates the outputs.

Cases run concurrently, up to Concurrency at a time; the report lists them in dataset order, with the repetitions of a case together. A task or evaluator that fails or panics fails only its case. Evaluate returns an error, along with the report of the runs that finished, only if the dataset is invalid or ctx ends; runs that did not start then fail with the context's error.

func (*Dataset[In, Out]) Save 

func (d *Dataset[In, Out]) Save(path string) error

Save writes the dataset to a file in the format FormatOf reports. Evaluators are not saved.

func (*Dataset[In, Out]) Write 

func (d *Dataset[In, Out]) Write(w io.Writer, f Format) error

Write writes the dataset in format f: indented JSON, or one compact case per line for FormatJSONL, which has no dataset name.

type Delta 

type Delta struct {
	Baseline  float64 `json:"baseline"`
	Candidate float64 `json:"candidate"`
}

Delta is a value in a baseline and a candidate report.

func (Delta) Change 

func (d Delta) Change() float64

Change returns the candidate's value minus the baseline's.

type DurationDelta 

type DurationDelta struct {
	Baseline  time.Duration `json:"baseline"`
	Candidate time.Duration `json:"candidate"`
}

DurationDelta is a duration in a baseline and a candidate report.

type Evaluation 

type Evaluation struct {
	Name string `json:"name"`
	Kind Kind   `json:"kind"`
	// Pass is the outcome of an assertion.
	Pass bool `json:"pass"`
	// Score is the value of a score.
	Score float64 `json:"score"`
	// Label is the value of a label.
	Label string `json:"label"`
	// Reason explains the result.
	Reason string `json:"reason,omitempty"`
	// Evaluator is the name of the evaluator that returned it.
	Evaluator string `json:"evaluator,omitempty"`
}

Evaluation is one result of an evaluator. Assertions decide whether a case passes; scores and labels are aggregated in reports.

func Assertion 

func Assertion(name string, pass bool, reason string) Evaluation

Assertion returns an assertion.

func Label 

func Label(name, value, reason string) Evaluation

Label returns a label.

func Score 

func Score(name string, value float64, reason string) Evaluation

Score returns a score.

func (Evaluation) MarshalJSON 

func (e Evaluation) MarshalJSON() ([]byte, error)

MarshalJSON encodes only the value field of the evaluation's kind.

func (Evaluation) String 

func (e Evaluation) String() string

String formats the evaluation's value.

type Evaluator 

type Evaluator interface {
	Name() string
	Evaluate(ctx context.Context, ec *EvaluatorContext) ([]Evaluation, error)
}

Evaluator scores the output of a case.

Evaluate returns any number of evaluations; none means the evaluator does not apply to the case. An error, or a panic, is reported as an evaluator failure of the case, which then does not pass. Evaluators must be safe for concurrent use.

func Check 

func Check(name string, fn func(ec *EvaluatorContext) (bool, string)) Evaluator

Check returns an evaluator with a single assertion named name, which passes when fn returns true. The string is the reason.

func Completed 

func Completed() Evaluator

Completed returns an evaluator with an assertion, named "Completed", that passes when the run completed, rather than ending at a limit, being interrupted for approval or failing.

func Contains 

func Contains(value any, opts ...MatchOption) Evaluator

Contains returns an evaluator with an assertion, named "Contains", as pydantic-evals defines it: a string output must contain value as a substring, a slice or array must have an element equal to value, and a map must hold every key of the map value with an equal value. Other outputs are compared as their JSON text, with value as a substring.

func DecodesAs 

func DecodesAs[T any]() Evaluator

DecodesAs returns an evaluator with an assertion, named "DecodesAs", that passes when the output's text is a single JSON value that decodes as T without unknown fields. Markdown code fences are removed as JSONSchema does.

func Equals 

func Equals(value any, opts ...MatchOption) Evaluator

Equals returns an evaluator with an assertion, named "Equals", that passes when the output deeply equals value.

func EqualsExpected 

func EqualsExpected(opts ...MatchOption) Evaluator

EqualsExpected returns an evaluator with an assertion, named "EqualsExpected", that passes when the output deeply equals the case's expected output. Cases without one are skipped.

func Func 

func Func(name string, fn func(ctx context.Context, ec *EvaluatorContext) ([]Evaluation, error)) Evaluator

Func returns an evaluator that calls fn.

func HasSpan 

func HasSpan(q SpanQuery) Evaluator

HasSpan returns an evaluator with an assertion, named "HasSpan", that passes when a span recorded during the task matches q. Record spans by tracing the agent with Tracer.

func IsType 

func IsType[T any]() Evaluator

IsType returns an evaluator with an assertion, named "IsType", that passes when the output's dynamic type is T or, if T is an interface, implements it. It suits tasks whose output type is an interface.

func JSONSchema 

func JSONSchema(schema *tool.Schema) Evaluator

JSONSchema returns an evaluator with an assertion, named "JSONSchema", that passes when the output is JSON matching schema. A string output is parsed as JSON, without the fence of a single Markdown code block, as loop.Output does; other outputs are encoded as JSON first.

func LLMJudge 

func LLMJudge(m model.Model, rubric string, opts ...JudgeOption) Evaluator

LLMJudge returns an evaluator that asks m to grade the output against rubric, as pydantic-evals' LLMJudge does (https://pydantic.dev/docs/ai/evals/evaluators/llm-judge/). The judge answers with a Verdict, requested and validated as structured output; it records an assertion with the judge's reason and, with JudgeScore, a score.

Write specific rubrics, one quality per judge, and combine judges with deterministic evaluators. Prefer a different model than the one under test. By default the judge sees only the output; see JudgeInput and JudgeExpected.

func MaxDuration 

func MaxDuration(d time.Duration) Evaluator

MaxDuration returns an evaluator with an assertion, named "MaxDuration", that passes when the task took at most d.

func MaxTokens 

func MaxTokens(n int) Evaluator

MaxTokens returns an evaluator with an assertion, named "MaxTokens", that passes when the run used at most n tokens in total, as model.Usage.TotalTokens counts them.

func MaxToolCalls 

func MaxToolCalls(n int) Evaluator

MaxToolCalls returns an evaluator with an assertion, named "MaxToolCalls", that passes when the run executed at most n tool calls.

func MaxTurns 

func MaxTurns(n int) Evaluator

MaxTurns returns an evaluator with an assertion, named "MaxTurns", that passes when the run took at most n turns (model calls).

func Named 

func Named(name string, e Evaluator) Evaluator

Named renames the evaluations of e: a single evaluation is named name, and several are prefixed with name and an underscore. The evaluator itself is named name too.

func NoSpan 

func NoSpan(q SpanQuery) Evaluator

NoSpan returns an evaluator with an assertion, named "NoSpan", that passes when no span recorded during the task matches q, for example to check that a dangerous tool was not called.

func Regex 

func Regex(pattern string) Evaluator

Regex returns an evaluator with an assertion, named "Regex", that passes when the output's text matches the regular expression pattern. An invalid pattern fails every case with an evaluator error.

func ResponseMatch 

func ResponseMatch(minScore float64) Evaluator

ResponseMatch returns an evaluator that compares the output's text with the expected output's by ROUGE-1, the F1 measure of the words they share, as Google ADK's response_match_score does (https://adk.dev/evaluate/criteria/). It records the score as "ResponseMatch_score" and an assertion, "ResponseMatch", that passes when the score is at least minScore; ADK's default is 0.8. Words are compared case-insensitively, and each CJK character counts as a word. Cases without an expected output are skipped.

func ToolTrajectory 

func ToolTrajectory(match TrajectoryMatch, expected []ToolCall, opts ...TrajectoryOption) Evaluator

ToolTrajectory returns an evaluator with an assertion, named "ToolTrajectory", on the tool calls the run made, as Google ADK's tool_trajectory_avg_score checks them (https://adk.dev/evaluate/criteria/). The calls are those of the assistant messages the run added, in order; calls of sub-agents are not included. Arguments are compared as JSON values, so 1 and 1.0 are equal.

If expected is nil, the expected calls are read from the case metadata under MetadataToolCalls, and cases without them are skipped. Use an empty, non-nil slice to expect no calls.

func TypedCheck 

func TypedCheck[In, Out any](name string, fn func(t Typed[In, Out]) (bool, string)) Evaluator

TypedCheck is Check with the typed view of the context.

func TypedFunc 

func TypedFunc[In, Out any](name string, fn func(ctx context.Context, t Typed[In, Out]) ([]Evaluation, error)) Evaluator

TypedFunc returns an evaluator that calls fn with the typed view of the context. A case whose values have other types fails with an evaluator error.

type EvaluatorContext 

type EvaluatorContext struct {
	// Name is the case name.
	Name string
	// Repeat numbers the run of the case, from 0; see Repeat.
	Repeat int
	Input  any
	Output any
	// Expected is the expected output, or nil if the case has none.
	Expected any
	Metadata map[string]any
	// Result is the run result returned by the task, or nil.
	Result *loop.Result
	// Duration is how long the task took.
	Duration time.Duration
	// Spans are the root spans recorded by Tracer during the task.
	Spans Spans
}

EvaluatorContext is what an evaluator sees of one run of a case.

type EvaluatorError 

type EvaluatorError struct {
	Evaluator string `json:"evaluator"`
	Error     string `json:"error"`
}

EvaluatorError reports an evaluator that returned an error or panicked.

type Format 

type Format string

Format is a dataset file format. 

const (
	// FormatJSON is an object with the dataset's "name" and its "cases".
	FormatJSON Format = "json"
	// FormatJSONL holds one case per line, as JSON Lines.
	FormatJSONL Format = "jsonl"
)

func FormatOf 

func FormatOf(path string) Format

FormatOf returns the format of a dataset file by its extension: FormatJSONL for .jsonl and .ndjson, FormatJSON otherwise.

type JudgeOption 

type JudgeOption func(*judgeConfig)

JudgeOption configures LLMJudge.

func JudgeExpected 

func JudgeExpected() JudgeOption

JudgeExpected shows the case's expected output to the judge, as a reference to compare with.

func JudgeInput 

func JudgeInput() JudgeOption

JudgeInput shows the case input to the judge, for rubrics about how the output answers it.

func JudgeInstructions 

func JudgeInstructions(s string) JudgeOption

JudgeInstructions replaces the judge's system prompt.

func JudgeName 

func JudgeName(name string) JudgeOption

JudgeName names the judge's assertion, and its score with the suffix "_score". It defaults to "LLMJudge".

func JudgeReasoning 

func JudgeReasoning(r *model.Reasoning) JudgeOption

JudgeReasoning enables model reasoning for the judge, which improves grading that needs complex judgment.

func JudgeScore 

func JudgeScore() JudgeOption

JudgeScore asks the judge for a score from 0 to 1 as well, recorded as a score named after the judge with the suffix "_score".

func JudgeSeed 

func JudgeSeed(seed int) JudgeOption

JudgeSeed asks the judge model for reproducible sampling with model.Request.Seed, for providers that support it, such as OpenAI and Gemini; others ignore it.

func JudgeTemperature 

func JudgeTemperature(t float64) JudgeOption

JudgeTemperature sets the judge's sampling temperature; it defaults to 0 for consistent grading. Providers drop it for models that reject it.

type Kind 

type Kind string

Kind is the kind of an Evaluation. 

const (
	// KindAssertion is a check that passes or fails.
	KindAssertion Kind = "assertion"
	// KindScore is a number, by convention between 0 and 1.
	KindScore Kind = "score"
	// KindLabel is a category, such as an error class.
	KindLabel Kind = "label"
)

type MatchOption 

type MatchOption func(*matchConfig)

MatchOption configures how Equals, EqualsExpected and Contains compare strings.

func IgnoreCase 

func IgnoreCase() MatchOption

IgnoreCase compares strings case-insensitively.

func TrimSpace 

func TrimSpace() MatchOption

TrimSpace ignores leading and trailing white space, and compares runs of white space inside strings as a single space.

type Option 

type Option func(*config)

Option configures Dataset.Evaluate and Run.

func Baseline 

func Baseline(r *Report) Option

Baseline makes Run compare the report with baseline, log the comparison and fail the test if a case regressed. Load a saved baseline with LoadReport.

func Concurrency 

func Concurrency(n int) Option

Concurrency bounds the case runs in progress; it defaults to DefaultConcurrency. Use 1 to run cases one after another, for example against rate-limited APIs.

func EvaluatorTimeout 

func EvaluatorTimeout(d time.Duration) Option

EvaluatorTimeout bounds each evaluator call, like Timeout bounds tasks.

func Experiment 

func Experiment(name string) Option

Experiment names the evaluation in its report, for example after the model or prompt version under test.

func MinScore 

func MinScore(name string, mean float64) Option

MinScore makes Run require the mean of the score name to be at least mean, like the per-criterion thresholds of Google ADK. The score must be present.

func OnResult 

func OnResult(fn func(*CaseResult)) Option

OnResult calls fn with each case run as it finishes, one at a time, for example to report progress.

func Repeat 

func Repeat(n int) Option

Repeat runs every case n times, to measure flaky behavior. Reports keep each run and aggregate them per case; see Report.ByCase.

func Threshold 

func Threshold(rate float64) Option

Threshold sets the pass rate Run requires, from 0 to 1; it defaults to 1, every run passing.

func Timeout 

func Timeout(d time.Duration) Option

Timeout bounds each run of the task. A task that does not return when its context ends is abandoned: its case fails with a timeout error and the evaluation goes on without waiting for it.

func WithEvaluators 

func WithEvaluators(es ...Evaluator) Option

WithEvaluators adds evaluators to those of the dataset.

type Report 

type Report struct {
	// Name is the experiment name; see Experiment.
	Name string `json:"name,omitempty"`
	// Dataset is the name of the dataset.
	Dataset string `json:"dataset,omitempty"`
	// Repeat is the number of runs of each case.
	Repeat   int           `json:"repeat"`
	Started  time.Time     `json:"started"`
	Duration time.Duration `json:"duration"`
	Summary  Summary       `json:"summary"`
	// Cases holds every run, in dataset order, with the runs of a case
	// together.
	Cases []*CaseResult `json:"cases"`
}

Report is the outcome of an evaluation.

func LoadReport 

func LoadReport(path string) (*Report, error)

LoadReport reads a report saved with Report.Save. Inputs and outputs are decoded as generic JSON values.

func Run 

func Run[In, Out any](t testing.TB, d *Dataset[In, Out], task Task[In, Out], opts ...Option) *Report

Run evaluates the dataset in a test: it runs Dataset.Evaluate with t.Context(), logs the report and fails the test if the pass rate is below Threshold, a MinScore is not met, or a case regressed from the Baseline. It returns the report, or nil if the evaluation could not run. 

func TestCapitals(t *testing.T) {
	eval.Run(t, dataset, eval.AgentTask(a), eval.Threshold(0.9))
}

func (*Report) ByCase 

func (r *Report) ByCase() []CaseSummary

ByCase aggregates the runs of each case, in dataset order. It shows which cases are flaky when cases are repeated.

func (*Report) Save 

func (r *Report) Save(path string) error

Save writes the report to a JSON file, for example to compare later reports with it.

func (*Report) String 

func (r *Report) String() string

String returns the report as WriteText prints it.

func (*Report) WriteJSON 

func (r *Report) WriteJSON(w io.Writer) error

WriteJSON writes the report as indented JSON.

func (*Report) WriteText 

func (r *Report) WriteText(w io.Writer) error

WriteText prints the report as a table of runs followed by the summary and the reasons of failed runs.

type Scenario 

type Scenario struct {
	// Start is the user's first message.
	Start string `json:"start"`
	// Plan describes what the user wants to achieve and how to answer the
	// agent's questions, for example "Book a table for two at 7pm on Friday.
	// Give the name Ada when asked."
	Plan string `json:"plan"`
	// Persona, if set, describes how the user writes and behaves.
	Persona string `json:"persona,omitempty"`
}

Scenario is the input of SimulateUser: what a simulated user wants from the agent, as in Google ADK's conversation scenarios (https://adk.dev/evaluate/user-sim/).

type SimOption 

type SimOption func(*simConfig)

SimOption configures SimulateUser.

func MaxInvocations 

func MaxInvocations(n int) SimOption

MaxInvocations bounds the user messages of a conversation, including the first; it defaults to DefaultMaxInvocations. The conversation ends after the agent answers the last one.

func SimRunOptions 

func SimRunOptions(opts ...agent.RunOption) SimOption

SimRunOptions configures each run of the agent.

func StopSignal 

func StopSignal(s string) SimOption

StopSignal sets the reply with which the simulated user ends the conversation; it defaults to DefaultStopSignal.

type Span 

type Span struct {
	Name       string             `json:"name"`
	Kind       telemetry.SpanKind `json:"kind"`
	Attributes map[string]any     `json:"attributes,omitempty"`
	Start      time.Time          `json:"start"`
	// Duration is zero for a span that did not end.
	Duration time.Duration `json:"duration"`
	// Error is the text of the error recorded on the span, if any.
	Error    string      `json:"error,omitempty"`
	Events   []SpanEvent `json:"events,omitempty"`
	Children Spans       `json:"children,omitempty"`
	// Parent is nil for a root span.
	Parent *Span `json:"-"`
}

Span is a span recorded by Tracer while a task ran.

type SpanEvent 

type SpanEvent struct {
	Name       string         `json:"name"`
	Attributes map[string]any `json:"attributes,omitempty"`
}

SpanEvent is an event recorded on a span, such as a retry.

type SpanQuery 

type SpanQuery struct {
	Name         string `json:"name,omitempty"`
	NameContains string `json:"name_contains,omitempty"`
	// NamePattern is a regular expression the name must match.
	NamePattern string `json:"name_pattern,omitempty"`
	// Attributes must all be present with equal values; numbers compare by
	// value.
	Attributes map[string]any `json:"attributes,omitempty"`
	// Error, if set, requires the span to have failed (true) or not.
	Error *bool `json:"error,omitempty"`
	// MinDuration and MaxDuration bound the span's duration when positive.
	MinDuration time.Duration `json:"min_duration,omitempty"`
	MaxDuration time.Duration `json:"max_duration,omitempty"`
	// Child and Descendant require a direct child or any descendant to
	// match the query.
	Child      *SpanQuery `json:"child,omitempty"`
	Descendant *SpanQuery `json:"descendant,omitempty"`
	// Ancestor requires any ancestor to match the query.
	Ancestor *SpanQuery `json:"ancestor,omitempty"`
	// Not requires the query not to match.
	Not *SpanQuery `json:"not,omitempty"`
}

SpanQuery selects spans; every condition set must hold. The spans of middleware.Tracing are named after the OpenTelemetry GenAI conventions, such as "invoke_agent {agent}", "chat {model}" and "execute_tool {tool}", with attributes such as gen_ai.tool.name.

func (*SpanQuery) Matches 

func (q *SpanQuery) Matches(sp *Span) bool

Matches reports whether sp matches q.

type Spans 

type Spans []*Span

Spans is a list of spans with their descendants.

func (Spans) All 

func (s Spans) All() iter.Seq[*Span]

All yields the spans and their descendants, depth first.

func (Spans) Any 

func (s Spans) Any(q SpanQuery) bool

Any reports whether a span among s and their descendants matches q.

func (Spans) Find 

func (s Spans) Find(q SpanQuery) []*Span

Find returns the spans, among s and their descendants, that match q.

type Summary 

type Summary struct {
	Runs   int `json:"runs"`
	Passed int `json:"passed"`
	// Errors counts the runs whose task failed.
	Errors   int     `json:"errors"`
	PassRate float64 `json:"pass_rate"`
	// Assertions is the pass rate of each assertion, by name, among the
	// runs that have it.
	Assertions map[string]float64 `json:"assertions,omitempty"`
	// Scores is the mean of each score, by name.
	Scores map[string]float64 `json:"scores,omitempty"`
	// Labels counts the values of each label, by name.
	Labels map[string]map[string]int `json:"labels,omitempty"`
	// Usage is the total usage of all runs.
	Usage        model.Usage   `json:"usage"`
	MeanDuration time.Duration `json:"mean_duration"`
	P50Duration  time.Duration `json:"p50_duration"`
	P95Duration  time.Duration `json:"p95_duration"`
	MaxDuration  time.Duration `json:"max_duration"`
}

Summary aggregates the runs of a report.

func (Summary) MeanTokens 

func (s Summary) MeanTokens() float64

MeanTokens returns the mean total tokens of a run.

type Task 

type Task[In, Out any] func(ctx context.Context, in In) (Out, *loop.Result, error)

Task produces the output for one input. Tasks that run an agent also return its run result, which evaluators such as ToolTrajectory and MaxTokens inspect; other tasks return a nil result. A non-nil error fails the case without running its evaluators.

func AgentTask 

func AgentTask(a agent.Agent, opts ...agent.RunOption) Task[string, string]

AgentTask returns a task that runs a on the input text, without history, and outputs the run's final answer, Result.Output. A run that ends without completing, for example at its turn limit, still produces an output; use Completed to require completion. opts configure each run, as they do for agent.Runner.

func AgentTaskFunc 

func AgentTaskFunc[In, Out any](a agent.Agent, input func(In) *agent.Input, output func(*loop.Result) (Out, error)) Task[In, Out]

AgentTaskFunc returns a task that runs a on the agent input that input builds and derives the output from the run's result with output. A failed run fails the case, with the partial result's usage reported.

func SimulateUser 

func SimulateUser(a agent.Agent, user model.Model, opts ...SimOption) Task[Scenario, string]

SimulateUser returns a task that holds a multi-turn conversation between a and a user simulated by the model user, as Google ADK's user simulation does (https://adk.dev/evaluate/user-sim/). The user opens with Scenario.Start; then, after each answer of the agent, user writes the next message following Scenario.Plan, until it replies with the stop signal or the conversation reaches MaxInvocations.

The output is the agent's last answer. The result is that of the last run, with the usage and turns of all runs and NewMessages holding the whole conversation, so evaluators such as ToolTrajectory and MaxTokens see all of it. The simulated user's own usage is not counted. A run that does not complete, for example one waiting for approval, ends the conversation.

func TypedAgentTask 

func TypedAgentTask[Out any](a agent.Agent, opts ...agent.RunOption) Task[string, Out]

TypedAgentTask returns a task that runs a on the input text, without history, and decodes the final answer as Out, as agent.AskAs does. Configure a with agent.WithOutput[Out]. A run that did not complete, or whose answer does not decode, fails the case.

type ToolCall 

type ToolCall struct {
	Name string `json:"name"`
	// Args are the expected arguments, compared as JSON. Nil matches any
	// arguments.
	Args map[string]any `json:"args,omitempty"`
}

ToolCall is an expected tool call.

func (ToolCall) String 

func (c ToolCall) String() string

type TrajectoryMatch 

type TrajectoryMatch string

TrajectoryMatch selects how ToolTrajectory compares the tool calls of a run with the expected ones. 

const (
	// MatchExact requires exactly the expected calls, in order. It is ADK's
	// EXACT.
	MatchExact TrajectoryMatch = "exact"
	// MatchInOrder requires the expected calls in order; other calls may
	// come between them. It is ADK's IN_ORDER.
	MatchInOrder TrajectoryMatch = "in_order"
	// MatchAnyOrder requires exactly the expected calls, in any order.
	MatchAnyOrder TrajectoryMatch = "any_order"
	// MatchSubset requires the expected calls in any order; other calls may
	// come between them. It is ADK's ANY_ORDER.
	MatchSubset TrajectoryMatch = "subset"
)

type TrajectoryOption 

type TrajectoryOption func(*trajectory)

TrajectoryOption configures ToolTrajectory.

func IgnoreArgs 

func IgnoreArgs() TrajectoryOption

IgnoreArgs matches calls by tool name only.

func PartialArgs 

func PartialArgs() TrajectoryOption

PartialArgs matches calls whose arguments include the expected ones: other arguments, and other fields of nested objects, are ignored. Without it, the arguments must equal the expected ones.

type Typed 

type Typed[In, Out any] struct {
	*EvaluatorContext
	Input    In
	Output   Out
	Expected *Out
}

Typed is an EvaluatorContext with the input, output and expected output of a Dataset[In, Out] typed. Its fields shadow the untyped ones of the embedded context.

type Verdict 

type Verdict struct {
	// Reason explains the verdict; the judge writes it first, so it
	// reasons before it decides.
	Reason string `json:"reason"`
	Pass   bool   `json:"pass"`
	// Score is set when JudgeScore asks for one. The schema bounds it
	// from 0 to 1, and answers outside are sent back to the judge.
	Score float64 `json:"score,omitempty"`
}

Verdict is the structured answer of an LLMJudge.

Source Files

View all Source files

Directories

Path Synopsis
Package replay records HTTP interactions to a cassette file and replays them, so tests of providers, agents and evaluations run deterministically without network access or API keys.
 Package replay records HTTP interactions to a cassette file and replays them, so tests of providers, agents and evaluations run deterministically without network access or API keys.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.