evalrun

package
v1.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2026 License: MIT Imports: 9 Imported by: 1

Documentation

Overview

Package evalrun defines evaluation runs and their results — the execution records of running a suite against a target.

Index

Constants

View Source 
const (
	SettingPassThreshold       = "pass_threshold"
	SettingRegressionThreshold = "regression_threshold"
	SettingConcurrency         = "concurrency"
	SettingTarget              = "target"
	SettingScorers             = "scorers"
	SettingModel               = "model"
	SettingPromptVersionID     = "prompt_version_id"
)

Run.Config keys a run's settings are recorded under. They are flat on purpose: mongo decodes nested maps as bson.D, which serialises as a list.

View Source 
const ScorerErrorPrefix = "scorer error: "

ScorerErrorPrefix starts the Reason of a scorer that could not judge the case: it could not be built from its config, or its Score call failed. Such a result never passed, but it did not fail the output either.

Variables

This section is empty.

Functions

This section is empty.

Types

type Finalization 

type Finalization struct {
	Stats       *ResultStats
	State       RunState // applied only if the stored state is still running
	Error       string   // written when non-empty
	CompletedAt time.Time
}

Finalization is what a runner writes when it stops scheduling a run.

type ListFilter 

type ListFilter struct {
	SuiteID id.SuiteID
	AppID   string
	State   RunState
	Limit   int
	Offset  int
}

ListFilter constrains run listing.

type Quality 

type Quality string

Quality says whether a usage value came from an authoritative receipt. 

const (
	// Unknown includes historical records without accounting provenance.
	Unknown Quality = "unknown"
	// Known identifies an authoritative measured value, including zero.
	Known Quality = "known"
	// Estimated identifies a value inferred from a rate or token estimate.
	Estimated Quality = "estimated"
)

type Result 

type Result struct {
	sentinel.Entity
	ID              id.EvalResultID    `json:"id" bun:"id,pk"`
	RunID           id.EvalRunID       `json:"run_id" bun:"run_id,notnull"`
	CaseID          id.CaseID          `json:"case_id" bun:"case_id,notnull"`
	CaseName        string             `json:"case_name" bun:"case_name,notnull"`
	Status          ResultStatus       `json:"status" bun:"status,notnull"`
	Score           float64            `json:"score" bun:"score,notnull,default:0"`
	Output          string             `json:"output,omitempty" bun:"output"`
	LatencyMs       int                `json:"latency_ms" bun:"latency_ms,notnull,default:0"`
	TokensUsed      int                `json:"tokens_used" bun:"tokens_used,notnull,default:0"`
	Cost            float64            `json:"cost" bun:"cost,notnull,default:0"`
	ScorerResults   []ScorerResult     `json:"scorer_results" bun:"scorer_results,type:jsonb,default:'[]'"`
	Error           string             `json:"error,omitempty" bun:"error"`
	DimensionScores map[string]float64 `json:"dimension_scores,omitempty" bun:"dimension_scores,type:jsonb,default:'{}'"`
	RunTrace        *RunTrace          `json:"run_trace,omitempty" bun:"run_trace,type:jsonb"`
}

Result is the outcome of evaluating a single test case within a run.

type ResultStats 

type ResultStats struct {
	TotalCases      int                `json:"total_cases"`
	Passed          int                `json:"passed"`
	Failed          int                `json:"failed"`
	Errored         int                `json:"errored"`
	PassRate        float64            `json:"pass_rate"`
	AvgScore        float64            `json:"avg_score"`
	AvgLatencyMs    int                `json:"avg_latency_ms"`
	TotalTokens     int                `json:"total_tokens"`
	TotalCost       float64            `json:"total_cost"`
	DimensionScores map[string]float64 `json:"dimension_scores,omitempty"`
}

ResultStats holds aggregate statistics for a run's results.

func Aggregate 

func Aggregate(results []*Result) *ResultStats

Aggregate computes the shared result counters used by stores and recorded runs. Results must contain non-nil entries.

type ResultStatus 

type ResultStatus string

ResultStatus is the outcome status of a single case evaluation. 

const (
	StatusPass  ResultStatus = "pass"
	StatusFail  ResultStatus = "fail"
	StatusError ResultStatus = "error"
)

type Run 

type Run struct {
	sentinel.Entity
	ID              id.EvalRunID       `json:"id" bun:"id,pk"`
	SuiteID         id.SuiteID         `json:"suite_id" bun:"suite_id,notnull"`
	Model           string             `json:"model" bun:"model,notnull"`
	SystemPrompt    string             `json:"system_prompt" bun:"system_prompt"`
	Temperature     float64            `json:"temperature" bun:"temperature,notnull,default:0"`
	TotalCases      int                `json:"total_cases" bun:"total_cases,notnull,default:0"`
	Passed          int                `json:"passed" bun:"passed,notnull,default:0"`
	Failed          int                `json:"failed" bun:"failed,notnull,default:0"`
	PassRate        float64            `json:"pass_rate" bun:"pass_rate,notnull,default:0"`
	AvgScore        float64            `json:"avg_score" bun:"avg_score,notnull,default:0"`
	AvgLatencyMs    int                `json:"avg_latency_ms" bun:"avg_latency_ms,notnull,default:0"`
	TotalTokens     int                `json:"total_tokens" bun:"total_tokens,notnull,default:0"`
	TotalCost       float64            `json:"total_cost" bun:"total_cost,notnull,default:0"`
	AppID           string             `json:"app_id" bun:"app_id,notnull"`
	TargetTenantID  string             `json:"target_tenant_id,omitempty" bun:"target_tenant_id"`
	PersonaRef      string             `json:"persona_ref,omitempty" bun:"persona_ref"`
	Config          map[string]any     `json:"config" bun:"config,type:jsonb,default:'{}'"`
	State           RunState           `json:"state" bun:"state,notnull,default:'running'"`
	Error           string             `json:"error,omitempty" bun:"error"`
	CompletedAt     *time.Time         `json:"completed_at,omitempty" bun:"completed_at"`
	DimensionScores map[string]float64 `json:"dimension_scores,omitempty" bun:"dimension_scores,type:jsonb,default:'{}'"`
}

Run represents a single execution of an evaluation suite.

func (*Run) ApplyStats 

func (r *Run) ApplyStats(st *ResultStats)

ApplyStats copies aggregate counters onto the run. TotalCases is the planned case count and is left alone: a cancelled run has fewer results than cases, and both numbers matter.

type RunState 

type RunState string

RunState is the lifecycle state of an evaluation run. 

const (
	StateRunning   RunState = "running"
	StateCompleted RunState = "completed"
	StateFailed    RunState = "failed"
	StateCancelled RunState = "cancelled"
)

type RunTrace 

type RunTrace struct {
	Steps     []StepTrace `json:"steps,omitempty"`
	ToolCalls []ToolTrace `json:"tool_calls,omitempty"`
}

RunTrace captures agent execution data for persona-aware evaluation.

type ScorerResult 

type ScorerResult struct {
	Usage       *Usage         `json:"usage,omitempty"`
	CriterionID string         `json:"criterion_id,omitempty"`
	Required    bool           `json:"required,omitempty"`
	ScorerName  string         `json:"scorer_name"`
	Score       float64        `json:"score"`
	Passed      bool           `json:"passed"`
	Reason      string         `json:"reason"`
	Dimension   string         `json:"dimension,omitempty"`
	Details     map[string]any `json:"details,omitempty"`
}

ScorerResult captures the output of a single scorer for a case.

func (ScorerResult) Errored 

func (r ScorerResult) Errored() bool

Errored reports whether the scorer could not judge the case, as opposed to judging it and failing it.

type Settings 

type Settings struct {
	PassThreshold       *float64
	RegressionThreshold *float64
	Concurrency         *int
	Target              string
	Scorers             []string
	Model               string
	PromptVersionID     string
}

Settings are what a run was scored with, recorded when it started. A pointer or slice is nil, and a string empty, when the run predates recording: that means "not recorded", which is not the same as today's configuration and must never be shown as if it were.

func SettingsFrom 

func SettingsFrom(config map[string]any) Settings

SettingsFrom reads settings back from Run.Config, tolerating the number and array types each backend decodes into.

func (Settings) Config 

func (s Settings) Config() map[string]any

Config returns the settings as Run.Config entries.

type StepTrace 

type StepTrace struct {
	Index      int    `json:"index"`
	Type       string `json:"type"`
	Output     string `json:"output"`
	TokensUsed int    `json:"tokens_used"`
}

StepTrace records a single reasoning step in an agent run.

type Store 

type Store interface {
	CreateRun(ctx context.Context, run *Run) error
	GetRun(ctx context.Context, runID id.EvalRunID) (*Run, error)
	UpdateRun(ctx context.Context, run *Run) error

	// CancelRun marks a running run cancelled at the given time. It reports
	// false, with no error, when the run exists and is no longer running.
	// The condition is part of the write, so it is safe against a runner
	// finishing at the same moment on another replica.
	CancelRun(ctx context.Context, runID id.EvalRunID, at time.Time) (bool, error)

	// FinalizeRun writes a finished run's counters, dimension scores and
	// completion time, and its error when one is given. It writes the state
	// only if the stored state is still running, and returns the state the
	// store holds afterwards, so a cancel is never overwritten.
	FinalizeRun(ctx context.Context, runID id.EvalRunID, f *Finalization) (RunState, error)

	ListRuns(ctx context.Context, filter *ListFilter) ([]*Run, error)
	ListRunsBySuite(ctx context.Context, suiteID id.SuiteID) ([]*Run, error)

	CreateResult(ctx context.Context, result *Result) error
	CreateResultBatch(ctx context.Context, results []*Result) error
	ListResults(ctx context.Context, runID id.EvalRunID) ([]*Result, error)
	GetResultStats(ctx context.Context, runID id.EvalRunID) (*ResultStats, error)
}

Store defines persistence operations for evaluation runs and results.

type ToolTrace 

type ToolTrace struct {
	ToolName  string `json:"tool_name"`
	Arguments string `json:"arguments"`
	Result    string `json:"result"`
	Error     string `json:"error,omitempty"`
}

ToolTrace records a single tool invocation in an agent run.

type Usage 

type Usage struct {
	Tokens        int64   `json:"tokens"`
	TokensQuality Quality `json:"tokens_quality,omitempty"`
	Cost          float64 `json:"cost"`
	CostQuality   Quality `json:"cost_quality,omitempty"`
	Currency      string  `json:"currency,omitempty"`
}

Usage keeps unknown values distinct from measured zero.

func AggregateUsage 

func AggregateUsage(samples []Usage) (Usage, error)

AggregateUsage combines receipts without turning missing accounting into zero. Different currencies cannot form one monetary total.

func (Usage) CostKnown 

func (u Usage) CostKnown() bool

CostKnown reports whether cost is measured rather than estimated or absent.

func (Usage) Validate 

func (u Usage) Validate() error

Validate checks numeric values and accounting provenance.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.