Documentation
¶
Overview ¶
Package evalcompose provides the runtime wiring helpers used by the judge and outcome eval tiers: model construction, per-specialist budget metering, fan-out roster assembly, and the in-turn loop guard.
Index ¶
- Constants
- Variables
- func Backend(provider, modelName string) string
- func Build(spec Spec, opts BuildOptions) (adkagent.Agent, error)
- func BuildModel(ctx context.Context, provider, name string, mo ModelOptions) (model.LLM, error)
- func BuildRoot(ctx context.Context, cfg RootConfig) (adkagent.Agent, []tool.Tool, error)
- func DiscoverProviders() ([]profile.Profile, error)
- func IsOfflineFake(name string) bool
- func IsProfileProvider(provider string) bool
- func KVSampler(ctx context.Context, provider, modelName string) (*kvmetrics.Sampler, error)
- func LookupProfile(provider string) (profile.Profile, bool, error)
- func LoopGuard(cfg LoopGuardConfig) (*plugin.Plugin, error)
- func MeterScopes(specs []Spec, provider, rootModelName string) map[string]Limits
- func NewRuntimeModel(ctx context.Context, provider, name string, mo ModelOptions) (model.LLM, error)
- func RatePer1K(provider, modelName string) float64
- func RegisterProfiles(ps []profile.Profile)
- func SpecModelName(spec Spec, provider, rootModelName string) string
- func TierModelName(provider, rootModelName, tier string) (string, error)
- type BudgetConfig
- type BuildOptions
- type Bundle
- type Call
- type Capability
- type Detailer
- type Dispatch
- type FanoutConfig
- type Limits
- type LoopGuardConfig
- type MCPAllowlist
- type Meter
- func (m *Meter) Allow(author string) error
- func (m *Meter) AllowFinalReport(author string) bool
- func (m *Meter) Observe(ev *session.Event) error
- func (m *Meter) ScopeSnapshot(name string) (tokens int64, costUSD float64, turns int, ok bool)
- func (m *Meter) Snapshot() (turns int, tokens int64, costUSD float64)
- type Mode
- type ModelOptions
- type ModelResolver
- type Pricer
- type PromptCache
- type RootConfig
- type ScopeSnapshot
- type Spec
- type SpecialistBudget
- type TierResolver
- type ToolAllowlist
Constants ¶
const ( // DefaultMaxModelCallsPerTurn is the per-turn model-call ceiling // when a bundle declares 0. DefaultMaxModelCallsPerTurn = 100 // DefaultStopAfterNote is how many more identical calls a model gets // after the in-turn note before the turn ends. DefaultStopAfterNote = 3 // LoopNoteKey is the field TurnLoop's note travels under in a tool result. LoopNoteKey = "watchdog" )
const ( ProviderGemini = "gemini" ProviderVertex = "vertex" ProviderVertexExpress = "vertex-express" ProviderAnthropic = "anthropic" ProviderAnthropicVertex = "anthropic-vertex" EnvAnthropicAPIKey = "ANTHROPIC_API_KEY" // #nosec G101 -- env var name EnvAnthropicVertexProject = "ANTHROPIC_VERTEX_PROJECT_ID" DefaultAnthropicModel = "claude-opus-5" DefaultAnthropicSmallModel = "claude-haiku-4-5" )
Provider aliases and default models for the eval suite.
const ( PromptCacheTTL5m = "5m" PromptCacheTTL1h = "1h" )
Prompt-cache breakpoint TTLs.
const ( SynthesisName = "_synthesis" FallbackName = "_fallback" )
Reserved specialist names.
const DefaultMaxConcurrency = 4
DefaultMaxConcurrency is the fan-out branch concurrency cap when a bundle leaves Fanout.MaxConcurrency unset (0).
const DetailKey = "mast.usage"
DetailKey is the LLMResponse.CustomMetadata key under which a provider adapter stores a Detailer.
Variables ¶
var ErrBudgetExceeded = errors.New("budget exceeded")
ErrBudgetExceeded is returned by Meter.Observe when a ceiling is crossed.
var ErrBudgetRefused = errors.New("budget refused the call")
ErrBudgetRefused is returned by Meter.Allow when a ceiling already makes another model call impossible.
Functions ¶
func Build ¶
func Build(spec Spec, opts BuildOptions) (adkagent.Agent, error)
Build turns a Spec into an ADK agent using core-agent's mode constructors (agent.NewTaskAgent / agent.NewSingleTurnAgent), wiring CallGate refusal and stall completion automatically.
func BuildModel ¶
BuildModel constructs the model.LLM for the given provider alias and model name.
func BuildRoot ¶
BuildRoot builds the roster and assembles the requested dispatch shape (coordinator or fan-out).
func DiscoverProviders ¶
DiscoverProviders loads provider profiles declared in the nearest .agents/config.json if present.
func IsOfflineFake ¶
IsOfflineFake reports whether name is an offline test double.
func IsProfileProvider ¶
IsProfileProvider reports whether provider names a provider profile.
func KVSampler ¶
KVSampler returns a KV-cache sampler for the self-hosted server a provider profile names, or nil when there is nothing to sample.
func LookupProfile ¶
LookupProfile returns the profile a --provider names, extends applied.
func LoopGuard ¶
func LoopGuard(cfg LoopGuardConfig) (*plugin.Plugin, error)
LoopGuard builds the in-turn runaway plugin for the judge eval rig.
func MeterScopes ¶
MeterScopes derives the per-specialist budget scopes for a roster.
func NewRuntimeModel ¶
func NewRuntimeModel(ctx context.Context, provider, name string, mo ModelOptions) (model.LLM, error)
NewRuntimeModel wraps BuildModel in the shared transient-error retry policy.
func RegisterProfiles ¶
RegisterProfiles sets the declared provider profiles.
func SpecModelName ¶
SpecModelName is the model id a spec will actually run on, or "" when it inherits the parent's.
func TierModelName ¶
TierModelName maps a `tier:` declaration to a concrete model id for the running provider.
Types ¶
type BudgetConfig ¶
BudgetConfig configures a Meter.
type BuildOptions ¶
type BuildOptions struct {
Model model.LLM
Resolve ModelResolver
ResolveTier TierResolver
Tools []tool.Tool
Toolsets []tool.Toolset
}
BuildOptions carries the runtime bindings a Spec needs to become a concrete ADK agent.
type Bundle ¶
type Bundle struct {
Name string
Description string
Specialists []string
Dispatch string
Fanout FanoutConfig
}
Bundle is the workload definition subset used by the evals.
type Call ¶
type Call struct {
UncachedInputTokens int64
CachedInputTokens int64
CacheWriteTokens int64
CacheWrite1hTokens int64
OutputTokens int64
}
Call is the token breakdown of one model call, in the three disjoint input buckets plus output that a price table needs.
type Capability ¶
type Capability string
Capability is what a specialist is allowed to do to the world.
const ( CapabilityReadOnly Capability = "read_only" CapabilityChangeExecutor Capability = "change_executor" )
type Detailer ¶
type Detailer interface {
UsageDetail() (cacheRead, cacheWrite, cacheWrite1h, reasoning *int64, servedModel string)
}
Detailer is the per-call usage detail a provider adapter can report beyond genai.GenerateContentResponseUsageMetadata.
type Dispatch ¶
type Dispatch string
Dispatch selects the root shape BuildRoot assembles.
func RosterShape ¶
RosterShape reads the dispatch shape out of a roster.
type FanoutConfig ¶
type FanoutConfig struct {
MaxConcurrency int
}
FanoutConfig configures the fan-out dispatch shape.
type Limits ¶
type Limits struct {
MaxTurns int
MaxTokens int64
MaxCostUSD float64
RatePer1K float64
Backend string
Model string
Pricer Pricer
}
Limits is one set of budget ceilings and the rate used to compute cost against them.
func ModelLimits ¶
ModelLimits is the price half of a meter scope for one model.
type LoopGuardConfig ¶
type LoopGuardConfig struct {
MaxModelCallsPerTurn int
RepeatThreshold int
RepeatStopAfter int
OnNote func(sessionID, toolName string, count int)
OnStop func(sessionID, reason string)
}
LoopGuardConfig configures the in-turn runaway guard.
type MCPAllowlist ¶
MCPAllowlist is the per-MCP-server tool allowlist for a specialist.
type Meter ¶
type Meter struct {
// contains filtered or unexported fields
}
Meter tracks cumulative session usage against a BudgetConfig and implements agent.CallGate and agent.FinalReportGate.
func NewMeter ¶
func NewMeter(cfg BudgetConfig) *Meter
NewMeter constructs a Meter for a single session.
func (*Meter) Allow ¶
Allow implements agent.CallGate, checking author's own scope first and then the session-wide limits before a model call is issued.
func (*Meter) AllowFinalReport ¶
AllowFinalReport implements agent.FinalReportGate, granting at most one final report call per author when BudgetConfig.FinalReport is enabled and the author has already spent at least one call.
func (*Meter) Observe ¶
Observe inspects a session event and updates the running tallies if it carries UsageMetadata.
func (*Meter) ScopeSnapshot ¶
ScopeSnapshot returns the current tally for the named scope and true, or zero values and false if name was not declared in BudgetConfig.Scopes.
type ModelOptions ¶
type ModelOptions struct {
PromptCache PromptCache
}
ModelOptions configures how BuildModel constructs a model.
type ModelResolver ¶
ModelResolver turns a specialist's `model:` override into a model.LLM.
func NewModelResolver ¶
func NewModelResolver(ctx context.Context, provider, rootName string, root model.LLM, mo ModelOptions) ModelResolver
NewModelResolver binds a specialist's model override to a concrete model.LLM.
type PromptCache ¶
type PromptCache struct {
Enabled *bool `yaml:"enabled,omitempty"`
TTL string `yaml:"ttl,omitempty"`
}
PromptCache configures Claude prompt caching for the model under test.
func (PromptCache) EffectiveTTL ¶
func (p PromptCache) EffectiveTTL() string
EffectiveTTL returns TTL with the 5m default applied.
func (PromptCache) On ¶
func (p PromptCache) On() bool
On resolves Enabled against the default: absent is on.
type RootConfig ¶
type RootConfig struct {
Bundle Bundle
Specs []Spec
ProjectInstruction string
Model model.LLM
ModelName string
Provider string
Tools []tool.Tool
Toolsets []tool.Toolset
Dispatch Dispatch
}
RootConfig carries everything BuildRoot needs to turn a bundle and specs into a runnable root agent.
type ScopeSnapshot ¶
ScopeSnapshot is a point-in-time reading of one named scope's tally.
type Spec ¶
type Spec struct {
Name string
Description string
Mode Mode
Model string
Tier string
Capability Capability
Budget SpecialistBudget
Tools ToolAllowlist
Instruction string
OutputSchema *genai.Schema
}
Spec is a specialist definition used by the cost-tier eval check.
type SpecialistBudget ¶
SpecialistBudget captures per-specialist runtime bounds.
type TierResolver ¶
TierResolver turns a specialist's `tier:` declaration into a model.LLM.
func NewTierResolver ¶
func NewTierResolver(provider, rootName string, root model.LLM, resolve ModelResolver) TierResolver
NewTierResolver binds a specialist's tier declaration to a concrete model.LLM.
type ToolAllowlist ¶
type ToolAllowlist struct {
Builtin []string
MCP []MCPAllowlist
Skills []string
}
ToolAllowlist is the composite allowlist of built-in tools, MCP tools, and skills a specialist may invoke.
func (ToolAllowlist) InheritsAllMCP ¶
func (t ToolAllowlist) InheritsAllMCP() bool
InheritsAllMCP reports whether this allowlist leaves the MCP axis unrestricted (nil slice).