evalcompose

package
v3.1.0-dev.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: Apache-2.0 Imports: 38 Imported by: 0

Documentation

Overview

Package evalcompose provides the runtime wiring helpers used by the judge and outcome eval tiers: model construction, per-specialist budget metering, fan-out roster assembly, and the in-turn loop guard.

Index

Constants

View Source
const (
	// DefaultMaxModelCallsPerTurn is the per-turn model-call ceiling
	// when a bundle declares 0.
	DefaultMaxModelCallsPerTurn = 100

	// DefaultStopAfterNote is how many more identical calls a model gets
	// after the in-turn note before the turn ends.
	DefaultStopAfterNote = 3

	// LoopNoteKey is the field TurnLoop's note travels under in a tool result.
	LoopNoteKey = "watchdog"
)
View Source
const (
	ProviderGemini          = "gemini"
	ProviderVertex          = "vertex"
	ProviderVertexExpress   = "vertex-express"
	ProviderAnthropic       = "anthropic"
	ProviderAnthropicVertex = "anthropic-vertex"

	EnvAnthropicAPIKey        = "ANTHROPIC_API_KEY" // #nosec G101 -- env var name
	EnvAnthropicVertexProject = "ANTHROPIC_VERTEX_PROJECT_ID"

	DefaultAnthropicModel      = "claude-opus-5"
	DefaultAnthropicSmallModel = "claude-haiku-4-5"
)

Provider aliases and default models for the eval suite.

View Source
const (
	PromptCacheTTL5m = "5m"
	PromptCacheTTL1h = "1h"
)

Prompt-cache breakpoint TTLs.

View Source
const (
	SynthesisName = "_synthesis"
	FallbackName  = "_fallback"
)

Reserved specialist names.

View Source
const DefaultMaxConcurrency = 4

DefaultMaxConcurrency is the fan-out branch concurrency cap when a bundle leaves Fanout.MaxConcurrency unset (0).

View Source
const DetailKey = "mast.usage"

DetailKey is the LLMResponse.CustomMetadata key under which a provider adapter stores a Detailer.

Variables

View Source
var ErrBudgetExceeded = errors.New("budget exceeded")

ErrBudgetExceeded is returned by Meter.Observe when a ceiling is crossed.

View Source
var ErrBudgetRefused = errors.New("budget refused the call")

ErrBudgetRefused is returned by Meter.Allow when a ceiling already makes another model call impossible.

Functions

func Backend

func Backend(provider, modelName string) string

Backend resolves the --provider alias and environment to the pricing backend.

func Build

func Build(spec Spec, opts BuildOptions) (adkagent.Agent, error)

Build turns a Spec into an ADK agent using core-agent's mode constructors (agent.NewTaskAgent / agent.NewSingleTurnAgent), wiring CallGate refusal and stall completion automatically.

func BuildModel

func BuildModel(ctx context.Context, provider, name string, mo ModelOptions) (model.LLM, error)

BuildModel constructs the model.LLM for the given provider alias and model name.

func BuildRoot

func BuildRoot(ctx context.Context, cfg RootConfig) (adkagent.Agent, []tool.Tool, error)

BuildRoot builds the roster and assembles the requested dispatch shape (coordinator or fan-out).

func DiscoverProviders

func DiscoverProviders() ([]profile.Profile, error)

DiscoverProviders loads provider profiles declared in the nearest .agents/config.json if present.

func IsOfflineFake

func IsOfflineFake(name string) bool

IsOfflineFake reports whether name is an offline test double.

func IsProfileProvider

func IsProfileProvider(provider string) bool

IsProfileProvider reports whether provider names a provider profile.

func KVSampler

func KVSampler(ctx context.Context, provider, modelName string) (*kvmetrics.Sampler, error)

KVSampler returns a KV-cache sampler for the self-hosted server a provider profile names, or nil when there is nothing to sample.

func LookupProfile

func LookupProfile(provider string) (profile.Profile, bool, error)

LookupProfile returns the profile a --provider names, extends applied.

func LoopGuard

func LoopGuard(cfg LoopGuardConfig) (*plugin.Plugin, error)

LoopGuard builds the in-turn runaway plugin for the judge eval rig.

func MeterScopes

func MeterScopes(specs []Spec, provider, rootModelName string) map[string]Limits

MeterScopes derives the per-specialist budget scopes for a roster.

func NewRuntimeModel

func NewRuntimeModel(ctx context.Context, provider, name string, mo ModelOptions) (model.LLM, error)

NewRuntimeModel wraps BuildModel in the shared transient-error retry policy.

func RatePer1K

func RatePer1K(provider, modelName string) float64

RatePer1K derives the flat USD-per-1K-total-tokens rate for a model.

func RegisterProfiles

func RegisterProfiles(ps []profile.Profile)

RegisterProfiles sets the declared provider profiles.

func SpecModelName

func SpecModelName(spec Spec, provider, rootModelName string) string

SpecModelName is the model id a spec will actually run on, or "" when it inherits the parent's.

func TierModelName

func TierModelName(provider, rootModelName, tier string) (string, error)

TierModelName maps a `tier:` declaration to a concrete model id for the running provider.

Types

type BudgetConfig

type BudgetConfig struct {
	Limits
	Scopes      map[string]Limits
	FinalReport bool
}

BudgetConfig configures a Meter.

type BuildOptions

type BuildOptions struct {
	Model       model.LLM
	Resolve     ModelResolver
	ResolveTier TierResolver
	Tools       []tool.Tool
	Toolsets    []tool.Toolset
}

BuildOptions carries the runtime bindings a Spec needs to become a concrete ADK agent.

type Bundle

type Bundle struct {
	Name        string
	Description string
	Specialists []string
	Dispatch    string
	Fanout      FanoutConfig
}

Bundle is the workload definition subset used by the evals.

type Call

type Call struct {
	UncachedInputTokens int64
	CachedInputTokens   int64
	CacheWriteTokens    int64
	CacheWrite1hTokens  int64
	OutputTokens        int64
}

Call is the token breakdown of one model call, in the three disjoint input buckets plus output that a price table needs.

type Capability

type Capability string

Capability is what a specialist is allowed to do to the world.

const (
	CapabilityReadOnly       Capability = "read_only"
	CapabilityChangeExecutor Capability = "change_executor"
)

type Detailer

type Detailer interface {
	UsageDetail() (cacheRead, cacheWrite, cacheWrite1h, reasoning *int64, servedModel string)
}

Detailer is the per-call usage detail a provider adapter can report beyond genai.GenerateContentResponseUsageMetadata.

type Dispatch

type Dispatch string

Dispatch selects the root shape BuildRoot assembles.

const (
	DispatchCoordinator Dispatch = "coordinator"
	DispatchGraph       Dispatch = "graph"
	DispatchFanout      Dispatch = "fanout"
	DispatchAuto        Dispatch = "auto"
)

func RosterShape

func RosterShape(specs []Spec) Dispatch

RosterShape reads the dispatch shape out of a roster.

func (Dispatch) Resolve

func (d Dispatch) Resolve(b Bundle) Dispatch

Resolve returns the dispatch shape to build, given the caller's choice and the bundle's own declaration.

type FanoutConfig

type FanoutConfig struct {
	MaxConcurrency int
}

FanoutConfig configures the fan-out dispatch shape.

type Limits

type Limits struct {
	MaxTurns   int
	MaxTokens  int64
	MaxCostUSD float64
	RatePer1K  float64
	Backend    string
	Model      string
	Pricer     Pricer
}

Limits is one set of budget ceilings and the rate used to compute cost against them.

func ModelLimits

func ModelLimits(provider, modelName string) Limits

ModelLimits is the price half of a meter scope for one model.

func (Limits) IsZero

func (l Limits) IsZero() bool

IsZero reports whether l declares neither a ceiling nor a rate.

type LoopGuardConfig

type LoopGuardConfig struct {
	MaxModelCallsPerTurn int
	RepeatThreshold      int
	RepeatStopAfter      int
	OnNote               func(sessionID, toolName string, count int)
	OnStop               func(sessionID, reason string)
}

LoopGuardConfig configures the in-turn runaway guard.

type MCPAllowlist

type MCPAllowlist struct {
	Server string
	Tools  []string
}

MCPAllowlist is the per-MCP-server tool allowlist for a specialist.

type Meter

type Meter struct {
	// contains filtered or unexported fields
}

Meter tracks cumulative session usage against a BudgetConfig and implements agent.CallGate and agent.FinalReportGate.

func NewMeter

func NewMeter(cfg BudgetConfig) *Meter

NewMeter constructs a Meter for a single session.

func (*Meter) Allow

func (m *Meter) Allow(author string) error

Allow implements agent.CallGate, checking author's own scope first and then the session-wide limits before a model call is issued.

func (*Meter) AllowFinalReport

func (m *Meter) AllowFinalReport(author string) bool

AllowFinalReport implements agent.FinalReportGate, granting at most one final report call per author when BudgetConfig.FinalReport is enabled and the author has already spent at least one call.

func (*Meter) Observe

func (m *Meter) Observe(ev *session.Event) error

Observe inspects a session event and updates the running tallies if it carries UsageMetadata.

func (*Meter) ScopeSnapshot

func (m *Meter) ScopeSnapshot(name string) (tokens int64, costUSD float64, turns int, ok bool)

ScopeSnapshot returns the current tally for the named scope and true, or zero values and false if name was not declared in BudgetConfig.Scopes.

func (*Meter) Snapshot

func (m *Meter) Snapshot() (turns int, tokens int64, costUSD float64)

Snapshot returns the current session totals without mutating state.

type Mode

type Mode string

Mode is the ADK v2 agent mode a specialist runs in.

const (
	ModeTask       Mode = "Task"
	ModeSingleTurn Mode = "SingleTurn"
)

type ModelOptions

type ModelOptions struct {
	PromptCache PromptCache
}

ModelOptions configures how BuildModel constructs a model.

type ModelResolver

type ModelResolver func(name string) (model.LLM, error)

ModelResolver turns a specialist's `model:` override into a model.LLM.

func NewModelResolver

func NewModelResolver(ctx context.Context, provider, rootName string, root model.LLM, mo ModelOptions) ModelResolver

NewModelResolver binds a specialist's model override to a concrete model.LLM.

type Pricer

type Pricer interface {
	PriceCall(backend, model string, c Call) (costUSD float64, ok bool)
}

Pricer prices one model call against a per-model rate table.

type PromptCache

type PromptCache struct {
	Enabled *bool  `yaml:"enabled,omitempty"`
	TTL     string `yaml:"ttl,omitempty"`
}

PromptCache configures Claude prompt caching for the model under test.

func (PromptCache) EffectiveTTL

func (p PromptCache) EffectiveTTL() string

EffectiveTTL returns TTL with the 5m default applied.

func (PromptCache) On

func (p PromptCache) On() bool

On resolves Enabled against the default: absent is on.

type RootConfig

type RootConfig struct {
	Bundle             Bundle
	Specs              []Spec
	ProjectInstruction string
	Model              model.LLM
	ModelName          string
	Provider           string
	Tools              []tool.Tool
	Toolsets           []tool.Toolset
	Dispatch           Dispatch
}

RootConfig carries everything BuildRoot needs to turn a bundle and specs into a runnable root agent.

type ScopeSnapshot

type ScopeSnapshot struct {
	Turns    int
	Tokens   int64
	CostUSD  float64
	Unpriced int
}

ScopeSnapshot is a point-in-time reading of one named scope's tally.

type Spec

type Spec struct {
	Name         string
	Description  string
	Mode         Mode
	Model        string
	Tier         string
	Capability   Capability
	Budget       SpecialistBudget
	Tools        ToolAllowlist
	Instruction  string
	OutputSchema *genai.Schema
}

Spec is a specialist definition used by the cost-tier eval check.

type SpecialistBudget

type SpecialistBudget struct {
	MaxTurns            int
	MaxWallclockSeconds int
	MaxCostUSD          float64
}

SpecialistBudget captures per-specialist runtime bounds.

type TierResolver

type TierResolver func(tier string) (model.LLM, error)

TierResolver turns a specialist's `tier:` declaration into a model.LLM.

func NewTierResolver

func NewTierResolver(provider, rootName string, root model.LLM, resolve ModelResolver) TierResolver

NewTierResolver binds a specialist's tier declaration to a concrete model.LLM.

type ToolAllowlist

type ToolAllowlist struct {
	Builtin []string
	MCP     []MCPAllowlist
	Skills  []string
}

ToolAllowlist is the composite allowlist of built-in tools, MCP tools, and skills a specialist may invoke.

func (ToolAllowlist) InheritsAllMCP

func (t ToolAllowlist) InheritsAllMCP() bool

InheritsAllMCP reports whether this allowlist leaves the MCP axis unrestricted (nil slice).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL