config

package
v1.1.10 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 27, 2026 License: Apache-2.0 Imports: 13 Imported by: 7

Documentation

Overview

Package config provides configuration management for PromptKit Arena.

This package handles YAML-based configuration loading and validation for:

  • Test scenarios and conversation definitions
  • LLM provider settings and credentials
  • Prompt template configurations
  • Self-play personas and role assignments
  • Default parameters and execution settings

Configuration files are loaded from disk with support for file references, allowing modular organization of scenarios, providers, and prompts.

The package is organized into:

  • types.go: Core type definitions (Config, Scenario, Provider, etc.)
  • loader.go: Loading functions for config files
  • helpers.go: Utility functions for path resolution
  • persona.go: Self-play persona loading
  • validator.go: Configuration validation

Index

Constants

View Source
const (
	LogLevelTrace = "trace"
	LogLevelDebug = "debug"
	LogLevelInfo  = "info"
	LogLevelWarn  = "warn"
	LogLevelError = "error"
)

LogLevel constants for programmatic use.

View Source
const (
	LogFormatJSON = "json"
	LogFormatText = "text"
)

LogFormat constants for programmatic use.

View Source
const (
	DefaultPersonaTemperature = float32(0.7)
	DefaultPersonaSeed        = 42
)

Default values for persona configuration

View Source
const (
	// TurnDetectionModeVAD uses voice activity detection for turn boundaries.
	TurnDetectionModeVAD = "vad"
	// TurnDetectionModeASM uses provider-native (server-side) turn detection.
	TurnDetectionModeASM = "asm"
)

Turn detection mode constants

View Source
const (
	// APIVersion is the Kubernetes-style API version for PromptKit configs
	APIVersion = "promptkit.altairalabs.ai/v1alpha1"

	// SchemaVersion is the version string used in schema URLs and paths
	SchemaVersion = "v1alpha1"
)

Version constants for PromptKit These are the single source of truth for versioning across the codebase

View Source
const SchemaBaseURL = "https://promptkit.altairalabs.ai/schemas/v1alpha1"

SchemaBaseURL is the base URL for PromptKit JSON schemas

View Source
const SchemaLocalPath = "schemas/v1alpha1"

SchemaLocalPath is the path to local schema files (relative to repo root)

Variables

View Source
var SchemaFallbackEnabled = true

SchemaFallbackEnabled controls whether to fall back to local schemas when remote fetch fails

View Source
var SchemaValidationEnabled = true

SchemaValidationEnabled controls whether schema validation is performed Can be disabled for testing or when schemas are not yet published

Functions

func ResolveConfigDir

func ResolveConfigDir(cfg *Config, configFilePath string) string

ResolveConfigDir resolves the base directory for config files with proper defaulting behavior: 1. If cfg.Defaults.ConfigDir is explicitly set, use it 2. Otherwise, if configFilePath is provided, use its directory 3. Otherwise, default to current working directory (".")

This ensures all config file types (prompts, providers, scenarios, tools) are resolved relative to the same base directory.

func ResolveFilePath

func ResolveFilePath(basePath, filePath string) string

ResolveFilePath resolves a file path relative to a base directory

func ResolveOutputPath

func ResolveOutputPath(outDir, filename string) string

ResolveOutputPath resolves an output file path relative to the output directory. If the filename is an absolute path, it is returned as-is. If the filename is empty, an empty string is returned. Otherwise, the filename is joined with the output directory.

func ValidateArenaConfig added in v1.1.2

func ValidateArenaConfig(yamlData []byte) error

ValidateArenaConfig validates an Arena configuration against its schema

func ValidateEval added in v1.1.9

func ValidateEval(yamlData []byte) error

ValidateEval validates an Eval configuration against its schema

func ValidatePersona added in v1.1.2

func ValidatePersona(yamlData []byte) error

ValidatePersona validates a Persona configuration against its schema

func ValidatePromptConfig added in v1.1.2

func ValidatePromptConfig(yamlData []byte) error

ValidatePromptConfig validates a PromptConfig configuration against its schema

func ValidateProvider added in v1.1.2

func ValidateProvider(yamlData []byte) error

ValidateProvider validates a Provider configuration against its schema

func ValidateScenario added in v1.1.2

func ValidateScenario(yamlData []byte) error

ValidateScenario validates a Scenario configuration against its schema

func ValidateTool added in v1.1.2

func ValidateTool(yamlData []byte) error

ValidateTool validates a Tool configuration against its schema

Types

type ArenaConfig

type ArenaConfig struct {
	APIVersion string     `yaml:"apiVersion"`
	Kind       string     `yaml:"kind"`
	Metadata   ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Config     `yaml:"spec"`
}

ArenaConfig represents the main Arena configuration in K8s-style manifest format

type ArenaConfigK8s added in v1.1.2

type ArenaConfigK8s struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Config            `yaml:"spec"`
}

ArenaConfigK8s represents the Arena configuration using full K8s ObjectMeta for unmarshaling This is used internally for compatibility with k8s.io types

type Config

type Config struct {
	// File references for YAML serialization
	PromptConfigs []PromptConfigRef `yaml:"prompt_configs,omitempty"`
	Providers     []ProviderRef     `yaml:"providers"`
	Judges        []JudgeRef        `yaml:"judges,omitempty"`
	JudgeDefaults *JudgeDefaults    `yaml:"judge_defaults,omitempty"`
	Scenarios     []ScenarioRef     `yaml:"scenarios,omitempty"`
	Evals         []EvalRef         `yaml:"evals,omitempty"`
	Tools         []ToolRef         `yaml:"tools,omitempty"`
	MCPServers    []MCPServerConfig `yaml:"mcp_servers,omitempty"`
	StateStore    *StateStoreConfig `yaml:"state_store,omitempty"`
	Defaults      Defaults          `yaml:"defaults"`
	SelfPlay      *SelfPlayConfig   `yaml:"self_play,omitempty"`
	// ProviderGroups maps provider ID to configured group (populated during load)
	ProviderGroups map[string]string `yaml:"-" json:"-"`
	// ProviderCapabilities maps provider ID to its capabilities (populated during load)
	ProviderCapabilities map[string][]string `yaml:"-" json:"-"`

	// Loaded resources (populated by LoadConfig, not serialized)
	LoadedPromptConfigs map[string]*PromptConfigData `yaml:"-" json:"-"` // taskType -> config
	LoadedProviders     map[string]*Provider         `yaml:"-" json:"-"` // provider ID -> provider
	LoadedJudges        map[string]*JudgeTarget      `yaml:"-" json:"-"` // judge name -> resolved target
	LoadedScenarios     map[string]*Scenario         `yaml:"-" json:"-"` // scenario ID -> scenario
	LoadedEvals         map[string]*Eval             `yaml:"-" json:"-"` // eval ID -> eval
	LoadedTools         []ToolData                   `yaml:"-" json:"-"` // list of tool data
	LoadedPersonas      map[string]*UserPersonaPack  `yaml:"-" json:"-"` // persona ID -> persona

	// Base directory for resolving relative paths (set during LoadConfig)
	ConfigDir string `yaml:"-" json:"-"`
}

Config represents the main configuration structure

func LoadConfig

func LoadConfig(filename string) (*Config, error)

LoadConfig loads and validates configuration from a YAML file in K8s-style manifest format. Reads all referenced resource files (scenarios, providers, tools, personas) and populates the Config struct, making it self-contained for programmatic use without physical files.

func (*Config) GetStateStoreConfig

func (c *Config) GetStateStoreConfig() *StateStoreConfig

GetStateStoreConfig returns the state store configuration, creating a default memory config if not specified.

func (*Config) GetStateStoreType

func (c *Config) GetStateStoreType() string

GetStateStoreType returns the configured state store type, defaulting to "memory" if not specified.

type ConfigType added in v1.1.2

type ConfigType string

ConfigType represents the type of configuration file

const (
	ConfigTypeArena        ConfigType = "arena"
	ConfigTypeScenario     ConfigType = "scenario"
	ConfigTypeEval         ConfigType = "eval"
	ConfigTypeProvider     ConfigType = "provider"
	ConfigTypePromptConfig ConfigType = "promptconfig"
	ConfigTypeTool         ConfigType = "tool"
	ConfigTypePersona      ConfigType = "persona"
)

func DetectConfigType added in v1.1.2

func DetectConfigType(yamlData []byte) (ConfigType, error)

DetectConfigType attempts to detect the configuration type from YAML data

type ConfigValidator

type ConfigValidator struct {
	// contains filtered or unexported fields
}

ConfigValidator validates configuration consistency and references

func NewConfigValidator

func NewConfigValidator(cfg *Config) *ConfigValidator

NewConfigValidator creates a new configuration validator

func NewConfigValidatorWithPath

func NewConfigValidatorWithPath(cfg *Config, configPath string) *ConfigValidator

NewConfigValidatorWithPath creates a new configuration validator with a config file path for resolving relative file references

func (*ConfigValidator) GetChecks added in v1.1.5

func (v *ConfigValidator) GetChecks() []ValidationCheck

GetChecks returns the structured validation checks

func (*ConfigValidator) GetErrors added in v1.1.5

func (v *ConfigValidator) GetErrors() []string

GetErrors returns all validation errors as strings

func (*ConfigValidator) GetWarnings

func (v *ConfigValidator) GetWarnings() []string

GetWarnings returns all validation warnings

func (*ConfigValidator) Validate

func (v *ConfigValidator) Validate() error

Validate performs comprehensive validation of the configuration

type ContextMetadata

type ContextMetadata struct {
	Domain       string `json:"domain,omitempty" yaml:"domain,omitempty"`
	UserRole     string `json:"user_role,omitempty" yaml:"user_role,omitempty"`
	ProjectStage string `json:"project_stage,omitempty" yaml:"project_stage,omitempty"`
}

ContextMetadata provides structured context information for scenarios

type ContextPolicy

type ContextPolicy struct {
	// TokenBudget is the maximum tokens for context (0 = unlimited)
	TokenBudget int `json:"token_budget,omitempty" yaml:"token_budget,omitempty"`
	// ReserveForOutput reserves tokens for the response (default 4000)
	ReserveForOutput int `json:"reserve_for_output,omitempty" yaml:"reserve_for_output,omitempty"`
	// Strategy is the truncation strategy: "oldest", "summarize", "relevance", "fail"
	Strategy string `json:"strategy,omitempty" yaml:"strategy,omitempty"`
	// CacheBreakpoints enables Anthropic caching
	CacheBreakpoints bool `json:"cache_breakpoints,omitempty" yaml:"cache_breakpoints,omitempty"`
	// Relevance configures embedding-based truncation when Strategy is "relevance"
	Relevance *RelevanceConfig `json:"relevance,omitempty" yaml:"relevance,omitempty"`
}

ContextPolicy defines context management for a scenario

type CredentialConfig added in v1.1.9

type CredentialConfig struct {
	// APIKey is an explicit API key value (not recommended for production).
	APIKey string `json:"api_key,omitempty" yaml:"api_key,omitempty"`
	// CredentialFile is a path to a file containing the API key.
	CredentialFile string `json:"credential_file,omitempty" yaml:"credential_file,omitempty"`
	// CredentialEnv is the name of an environment variable containing the API key.
	CredentialEnv string `json:"credential_env,omitempty" yaml:"credential_env,omitempty"`
}

CredentialConfig defines how to obtain credentials for a provider. Resolution order: api_key → credential_file → credential_env → default env vars.

type Defaults

type Defaults struct {
	Temperature float32      `yaml:"temperature,omitempty"`
	MaxTokens   int          `yaml:"max_tokens,omitempty"`
	Seed        int          `yaml:"seed,omitempty"`
	Concurrency int          `yaml:"concurrency,omitempty"`
	Output      OutputConfig `yaml:"output,omitempty"`
	// ConfigDir is the base directory for all config files (prompts, providers, scenarios, tools).
	// If not set, defaults to the directory containing the main config file.
	// If the main config file path is not known, defaults to current working directory.
	ConfigDir string   `yaml:"config_dir,omitempty"`
	FailOn    []string `yaml:"fail_on,omitempty"`
	Verbose   bool     `yaml:"verbose,omitempty"`

	// Deprecated fields for backward compatibility (will be removed)
	HTMLReport     string          `yaml:"html_report,omitempty"`
	OutDir         string          `yaml:"out_dir,omitempty"`
	OutputFormats  []string        `yaml:"output_formats,omitempty"`
	MarkdownConfig *MarkdownConfig `yaml:"markdown_config,omitempty"`
}

Defaults contains default configuration values

func (*Defaults) GetOutputConfig added in v1.1.0

func (d *Defaults) GetOutputConfig() OutputConfig

GetOutputConfig returns the effective output configuration, handling backward compatibility

type DuplexConfig added in v1.1.6

type DuplexConfig struct {
	// Timeout is the maximum session duration (e.g., "10m", "5m30s").
	Timeout string `json:"timeout,omitempty" yaml:"timeout,omitempty"`
	// TurnDetection configures how turn boundaries are detected.
	TurnDetection *TurnDetectionConfig `json:"turn_detection,omitempty" yaml:"turn_detection,omitempty"`
	// Resilience configures error handling and retry behavior.
	Resilience *DuplexResilienceConfig `json:"resilience,omitempty" yaml:"resilience,omitempty"`
}

DuplexConfig enables duplex (bidirectional) streaming mode for a scenario. When enabled, audio is streamed in chunks and turn boundaries are detected dynamically.

func (*DuplexConfig) GetResilience added in v1.1.6

func (d *DuplexConfig) GetResilience() *DuplexResilienceConfig

GetResilience returns the resilience config, or nil if not set.

func (*DuplexConfig) GetTimeoutDuration added in v1.1.6

func (d *DuplexConfig) GetTimeoutDuration(defaultTimeout time.Duration) time.Duration

GetTimeoutDuration returns the timeout as a time.Duration, or the default if not set.

func (*DuplexConfig) Validate added in v1.1.6

func (d *DuplexConfig) Validate() error

Validate validates the DuplexConfig settings.

type DuplexResilienceConfig added in v1.1.6

type DuplexResilienceConfig struct {
	// MaxRetries is the number of retry attempts for failed turns (default: 0).
	// Each retry creates a new session if the previous one ended.
	MaxRetries int `json:"max_retries,omitempty" yaml:"max_retries,omitempty"`
	// RetryDelayMs is the delay in milliseconds between retries (default: 1000).
	RetryDelayMs int `json:"retry_delay_ms,omitempty" yaml:"retry_delay_ms,omitempty"`
	// InterTurnDelayMs is the delay in milliseconds between turns (default: 500).
	// This allows the provider to fully process the previous response.
	InterTurnDelayMs int `json:"inter_turn_delay_ms,omitempty" yaml:"inter_turn_delay_ms,omitempty"`
	// SelfplayInterTurnDelayMs is the delay after selfplay turns (default: 1000).
	// Longer delay needed because TTS audio can be lengthy.
	//nolint:lll // JSON/YAML tag names must match field names for clarity
	SelfplayInterTurnDelayMs int `json:"selfplay_inter_turn_delay_ms,omitempty" yaml:"selfplay_inter_turn_delay_ms,omitempty"`
	// PartialSuccessMinTurns is the minimum completed turns to accept partial success (default: 1).
	// If the session ends unexpectedly but this many turns completed, treat as success.
	PartialSuccessMinTurns int `json:"partial_success_min_turns,omitempty" yaml:"partial_success_min_turns,omitempty"`
	// IgnoreLastTurnSessionEnd treats session end on the final turn as success (default: true).
	// Useful when providers may close sessions after completing the expected conversation.
	//nolint:lll // JSON/YAML tag names must match field names for clarity
	IgnoreLastTurnSessionEnd *bool `json:"ignore_last_turn_session_end,omitempty" yaml:"ignore_last_turn_session_end,omitempty"`
}

DuplexResilienceConfig configures error handling and retry behavior for duplex streaming. These settings help handle transient failures and provider-specific behaviors.

func (*DuplexResilienceConfig) GetInterTurnDelayMs added in v1.1.6

func (r *DuplexResilienceConfig) GetInterTurnDelayMs(defaultVal int) int

GetInterTurnDelayMs returns the configured inter-turn delay or the default.

func (*DuplexResilienceConfig) GetMaxRetries added in v1.1.6

func (r *DuplexResilienceConfig) GetMaxRetries(defaultVal int) int

GetMaxRetries returns the configured max retries or the default.

func (*DuplexResilienceConfig) GetPartialSuccessMinTurns added in v1.1.6

func (r *DuplexResilienceConfig) GetPartialSuccessMinTurns(defaultVal int) int

GetPartialSuccessMinTurns returns the configured partial success threshold or the default.

func (*DuplexResilienceConfig) GetRetryDelayMs added in v1.1.6

func (r *DuplexResilienceConfig) GetRetryDelayMs(defaultVal int) int

GetRetryDelayMs returns the configured retry delay or the default.

func (*DuplexResilienceConfig) GetSelfplayInterTurnDelayMs added in v1.1.6

func (r *DuplexResilienceConfig) GetSelfplayInterTurnDelayMs(defaultVal int) int

GetSelfplayInterTurnDelayMs returns the configured selfplay inter-turn delay or the default.

func (*DuplexResilienceConfig) ShouldIgnoreLastTurnSessionEnd added in v1.1.6

func (r *DuplexResilienceConfig) ShouldIgnoreLastTurnSessionEnd(defaultVal bool) bool

ShouldIgnoreLastTurnSessionEnd returns whether to ignore session end on the last turn.

type Eval added in v1.1.9

type Eval struct {
	// ID uniquely identifies this evaluation configuration
	ID string `json:"id" yaml:"id" jsonschema:"required,description=Unique identifier for this evaluation"`
	// Description provides a human-readable explanation of what this eval tests
	Description string `json:"description" yaml:"description" jsonschema:"description=Human-readable description"`
	// Recording specifies the path to the saved conversation to evaluate
	Recording RecordingSource `json:"recording" yaml:"recording" jsonschema:"required,description=Recording source"`
	// Turns configures turn-level assertion evaluation (array format matching Scenario structure)
	Turns []EvalTurnConfig `json:"turns,omitempty" yaml:"turns,omitempty" jsonschema:"description=Turn-level assertions"`
	// ConversationAssertions are evaluated on the entire conversation after replay
	ConversationAssertions []asrt.AssertionConfig `` //nolint:lll
	/* 136-byte string literal not displayed */
	// Tags for categorizing and filtering evaluations
	Tags []string `json:"tags,omitempty" yaml:"tags,omitempty" jsonschema:"description=Tags for categorization"`
	// Mode controls replay behavior: "instant", "realtime", or "accelerated"
	Mode string `` //nolint:lll
	/* 167-byte string literal not displayed */
	// Speed controls playback speed when Mode is "accelerated" (default: 1.0)
	Speed float64 `json:"speed,omitempty" yaml:"speed,omitempty" jsonschema:"description=Playback speed (default 1.0)"`
}

Eval describes an evaluation configuration for replaying and validating saved conversations

func LoadEval added in v1.1.9

func LoadEval(filename string) (*Eval, error)

LoadEval loads and parses an eval configuration from a YAML file in K8s-style manifest format

type EvalAllTurnsConfig added in v1.1.9

type EvalAllTurnsConfig struct {
	Assertions []asrt.AssertionConfig `` //nolint:lll
	/* 126-byte string literal not displayed */
}

EvalAllTurnsConfig contains the assertions for all turns

type EvalConfig added in v1.1.9

type EvalConfig struct {
	APIVersion string     `yaml:"apiVersion"`
	Kind       string     `yaml:"kind"`
	Metadata   ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Eval       `yaml:"spec"`
}

EvalConfig represents an Eval (saved conversation evaluation) configuration in K8s-style manifest format

func (*EvalConfig) GetAPIVersion added in v1.1.9

func (c *EvalConfig) GetAPIVersion() string

GetAPIVersion returns the API version for EvalConfig

func (*EvalConfig) GetKind added in v1.1.9

func (c *EvalConfig) GetKind() string

GetKind returns the kind for EvalConfig

func (*EvalConfig) GetName added in v1.1.9

func (c *EvalConfig) GetName() string

GetName returns the metadata name for EvalConfig

func (*EvalConfig) SetID added in v1.1.9

func (c *EvalConfig) SetID(id string)

SetID sets the ID in the spec for EvalConfig

type EvalConfigK8s added in v1.1.9

type EvalConfigK8s struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Eval              `yaml:"spec"`
}

EvalConfigK8s represents the Eval configuration using full K8s ObjectMeta for unmarshaling

func (*EvalConfigK8s) GetAPIVersion added in v1.1.9

func (c *EvalConfigK8s) GetAPIVersion() string

GetAPIVersion returns the API version for EvalConfigK8s

func (*EvalConfigK8s) GetKind added in v1.1.9

func (c *EvalConfigK8s) GetKind() string

GetKind returns the kind for EvalConfigK8s

func (*EvalConfigK8s) GetName added in v1.1.9

func (c *EvalConfigK8s) GetName() string

GetName returns the metadata name for EvalConfigK8s

func (*EvalConfigK8s) SetID added in v1.1.9

func (c *EvalConfigK8s) SetID(id string)

SetID sets the ID in the spec for EvalConfigK8s

type EvalRef added in v1.1.9

type EvalRef struct {
	File string `yaml:"file"`
}

EvalRef references an eval configuration file

type EvalTurnConfig added in v1.1.9

type EvalTurnConfig struct {
	// AllTurns contains assertions to apply to all assistant turns
	AllTurns *EvalAllTurnsConfig `json:"all_turns,omitempty" yaml:"all_turns,omitempty" jsonschema:"description=Assertions to apply to all assistant messages"` //nolint:lll
}

EvalTurnConfig configures turn-level assertions for evaluations Uses a structure parallel to Scenario turns but with all_turns instead of role

type FragmentRef

type FragmentRef struct {
	Name     string `yaml:"name" json:"name"`
	Path     string `yaml:"path,omitempty" json:"path,omitempty"` // Optional: relative path to fragment file
	Required bool   `yaml:"required" json:"required"`
}

FragmentRef references a reusable prompt fragment Reuses the same structure as prompt.FragmentRef for consistency

type HTMLOutputConfig added in v1.1.0

type HTMLOutputConfig struct {
	File string `yaml:"file,omitempty"` // Custom HTML output file name

}

HTMLOutputConfig contains configuration options for HTML output

type HTTPConfig added in v1.1.2

type HTTPConfig struct {
	URL            string            `json:"url" yaml:"url"`
	Method         string            `json:"method" yaml:"method"`
	HeadersFromEnv []string          `json:"headers_from_env,omitempty" yaml:"headers_from_env,omitempty"`
	TimeoutMs      int               `json:"timeout_ms" yaml:"timeout_ms"`
	Redact         []string          `json:"redact,omitempty" yaml:"redact,omitempty"`
	Headers        map[string]string `json:"headers,omitempty" yaml:"headers,omitempty"`
}

HTTPConfig defines configuration for live HTTP tool execution

type JSONOutputConfig added in v1.1.0

type JSONOutputConfig struct {
}

JSONOutputConfig contains configuration options for JSON output

type JUnitOutputConfig added in v1.1.0

type JUnitOutputConfig struct {
	File string `yaml:"file,omitempty"` // Custom JUnit output file name

}

JUnitOutputConfig contains configuration options for JUnit XML output

type JudgeDefaults added in v1.1.3

type JudgeDefaults struct {
	Prompt         string `yaml:"prompt,omitempty"`
	PromptRegistry string `yaml:"prompt_registry,omitempty"`
}

JudgeDefaults configures default judge prompt selection.

type JudgeRef added in v1.1.3

type JudgeRef struct {
	Name     string `yaml:"name"`            // Judge identifier used in assertions
	Provider string `yaml:"provider"`        // Provider ID reference (must exist in spec.providers)
	Model    string `yaml:"model,omitempty"` // Optional model override for the judge
}

JudgeRef references a judge configuration mapped to a provider. Mirrors self-play role/provider mapping to allow multiple judge targets.

type JudgeTarget added in v1.1.3

type JudgeTarget struct {
	Name     string    // Judge identifier
	Provider *Provider // Resolved provider config
	Model    string    // Effective model (judge override or provider model)
}

JudgeTarget is a resolved judge reference with provider config and effective model.

type LoggingConfig added in v1.1.6

type LoggingConfig struct {
	//nolint:lll // jsonschema tags require single line
	APIVersion string `` /* 126-byte string literal not displayed */
	//nolint:lll // jsonschema tags require single line
	Kind     string            `yaml:"kind" jsonschema:"const=LoggingConfig,title=Kind,description=Resource type identifier"`
	Metadata ObjectMeta        `yaml:"metadata,omitempty" jsonschema:"title=Metadata,description=Resource metadata"`
	Spec     LoggingConfigSpec `yaml:"spec" jsonschema:"title=Spec,description=Logging configuration specification"`
}

LoggingConfig represents logging configuration in K8s-style manifest format. This is used for schema generation and external configuration files.

type LoggingConfigSpec added in v1.1.6

type LoggingConfigSpec struct {
	// DefaultLevel is the default log level for all modules.
	// Supported values: trace, debug, info, warn, error.
	//nolint:lll // jsonschema tags require single line
	DefaultLevel string `` /* 174-byte string literal not displayed */

	// Format specifies the output format.
	// "json" produces machine-parseable JSON logs.
	// "text" produces human-readable text logs.
	//nolint:lll // jsonschema tags require single line
	Format string `yaml:"format,omitempty" jsonschema:"enum=json,enum=text,default=text,title=Format,description=Log output format"`

	// CommonFields are key-value pairs added to every log entry.
	// Useful for environment, service name, cluster, etc.
	//nolint:lll // jsonschema tags require single line
	CommonFields map[string]string `yaml:"commonFields,omitempty" jsonschema:"title=Common Fields,description=Key-value pairs added to every log entry"`

	// Modules configures logging for specific modules.
	// Module names use dot notation (e.g., runtime.pipeline).
	//nolint:lll // jsonschema tags require single line
	Modules []ModuleLoggingConfig `yaml:"modules,omitempty" jsonschema:"title=Modules,description=Per-module logging configuration"`
}

LoggingConfigSpec defines the logging configuration parameters.

func DefaultLoggingConfig added in v1.1.6

func DefaultLoggingConfig() LoggingConfigSpec

DefaultLoggingConfig returns a LoggingConfigSpec with sensible defaults.

func (*LoggingConfigSpec) Validate added in v1.1.6

func (c *LoggingConfigSpec) Validate() error

Validate validates the LoggingConfigSpec.

type MCPServerConfig

type MCPServerConfig struct {
	Name    string            `yaml:"name"`
	Command string            `yaml:"command"`
	Args    []string          `yaml:"args,omitempty"`
	Env     map[string]string `yaml:"env,omitempty"`
}

MCPServerConfig represents configuration for an MCP server

type MarkdownConfig deprecated added in v1.1.0

type MarkdownConfig struct {
	IncludeDetails    bool `yaml:"include_details"`     // Include detailed test information
	ShowOverview      bool `yaml:"show_overview"`       // Show executive overview section
	ShowResultsMatrix bool `yaml:"show_results_matrix"` // Show results matrix table
	ShowFailedTests   bool `yaml:"show_failed_tests"`   // Show failed tests section
	ShowCostSummary   bool `yaml:"show_cost_summary"`   // Show cost analysis section
}

Deprecated: Use MarkdownOutputConfig instead

type MarkdownOutputConfig added in v1.1.0

type MarkdownOutputConfig struct {
	File              string `yaml:"file,omitempty"`      // Custom markdown output file name
	IncludeDetails    bool   `yaml:"include_details"`     // Include detailed test information
	ShowOverview      bool   `yaml:"show_overview"`       // Show executive overview section
	ShowResultsMatrix bool   `yaml:"show_results_matrix"` // Show results matrix table
	ShowFailedTests   bool   `yaml:"show_failed_tests"`   // Show failed tests section
	ShowCostSummary   bool   `yaml:"show_cost_summary"`   // Show cost analysis section
}

MarkdownOutputConfig contains configuration options for markdown output formatting

type ModuleLoggingConfig added in v1.1.6

type ModuleLoggingConfig struct {
	// Name is the module name pattern using dot notation.
	// Examples: "runtime", "runtime.pipeline", "providers.openai".
	// More specific names take precedence over less specific ones.
	//nolint:lll // jsonschema tags require single line
	Name string `yaml:"name" jsonschema:"title=Name,description=Module name pattern using dot notation (e.g. runtime.pipeline)"`

	// Level is the log level for this module.
	// Overrides the default level for matching loggers.
	//nolint:lll // jsonschema tags require single line
	Level string `` /* 128-byte string literal not displayed */

	// Fields are additional key-value pairs added to logs from this module.
	//nolint:lll // jsonschema tags require single line
	Fields map[string]string `yaml:"fields,omitempty" jsonschema:"title=Fields,description=Additional fields added to logs from this module"`
}

ModuleLoggingConfig configures logging for a specific module.

type ObjectMeta added in v1.1.2

type ObjectMeta struct {
	Name        string            `yaml:"name,omitempty" jsonschema:"title=Name,description=Name of the resource"`
	Namespace   string            `yaml:"namespace,omitempty" jsonschema:"title=Namespace,description=Namespace for the resource"`
	Labels      map[string]string `yaml:"labels,omitempty" jsonschema:"title=Labels,description=Key-value pairs for organizing resources"`
	Annotations map[string]string `yaml:"annotations,omitempty" jsonschema:"title=Annotations,description=Additional metadata"`
}

ObjectMeta is a simplified metadata structure for PromptKit configs Based on K8s ObjectMeta but with YAML-friendly tags and optional fields

type OutputConfig added in v1.1.0

type OutputConfig struct {
	Dir       string                `yaml:"dir"`                 // Base output directory
	Formats   []string              `yaml:"formats"`             // List of enabled formats: json, html, markdown, junit
	JSON      *JSONOutputConfig     `yaml:"json,omitempty"`      // JSON-specific configuration
	HTML      *HTMLOutputConfig     `yaml:"html,omitempty"`      // HTML-specific configuration
	Markdown  *MarkdownOutputConfig `yaml:"markdown,omitempty"`  // Markdown-specific configuration
	JUnit     *JUnitOutputConfig    `yaml:"junit,omitempty"`     // JUnit-specific configuration
	Recording *RecordingConfig      `yaml:"recording,omitempty"` // Session recording configuration
}

OutputConfig contains configuration for all output formats

func (*OutputConfig) GetHTMLOutputConfig added in v1.1.0

func (o *OutputConfig) GetHTMLOutputConfig() *HTMLOutputConfig

GetHTMLOutputConfig returns the HTML configuration with defaults

func (*OutputConfig) GetJUnitOutputConfig added in v1.1.0

func (o *OutputConfig) GetJUnitOutputConfig() *JUnitOutputConfig

GetJUnitOutputConfig returns the JUnit configuration with defaults

func (*OutputConfig) GetMarkdownOutputConfig added in v1.1.0

func (o *OutputConfig) GetMarkdownOutputConfig() *MarkdownOutputConfig

GetMarkdownOutputConfig returns the markdown configuration with defaults

type PersonaConfig

type PersonaConfig struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       UserPersonaPack   `yaml:"spec"`
}

PersonaConfig represents a K8s-style persona configuration manifest

type PersonaConfigSchema added in v1.1.2

type PersonaConfigSchema struct {
	APIVersion string          `yaml:"apiVersion"`
	Kind       string          `yaml:"kind"`
	Metadata   ObjectMeta      `yaml:"metadata,omitempty"`
	Spec       UserPersonaPack `yaml:"spec"`
}

PersonaConfigSchema represents a Persona configuration for schema generation with simplified ObjectMeta

type PersonaDefaults

type PersonaDefaults struct {
	Temperature *float32 `json:"temperature,omitempty" yaml:"temperature,omitempty"`
	Seed        *int     `json:"seed,omitempty" yaml:"seed,omitempty"`
}

PersonaDefaults contains default LLM parameters for the persona. Pointer types are used to distinguish "not set" (nil) from "explicitly set to 0".

type PersonaRef

type PersonaRef struct {
	File string `yaml:"file"`
}

PersonaRef references a persona file

type PersonaStyle

type PersonaStyle struct {
	Verbosity      string   `json:"verbosity" yaml:"verbosity"`             // "short", "medium", "long"
	ChallengeLevel string   `json:"challenge_level" yaml:"challenge_level"` // "low", "medium", "high"
	FrictionTags   []string `json:"friction_tags" yaml:"friction_tags"`     // e.g., ["skeptical", "impatient", "detail-oriented"]
}

PersonaStyle defines behavioral characteristics for the user persona

type PlatformConfig added in v1.1.9

type PlatformConfig struct {
	// Type is the platform type: "bedrock", "vertex", or "azure".
	Type string `json:"type,omitempty" yaml:"type,omitempty"`
	// Region is the cloud region (e.g., "us-west-2", "us-central1").
	Region string `json:"region,omitempty" yaml:"region,omitempty"`
	// Project is the cloud project ID (required for Vertex).
	Project string `json:"project,omitempty" yaml:"project,omitempty"`
	// Endpoint is an optional custom endpoint URL.
	Endpoint string `json:"endpoint,omitempty" yaml:"endpoint,omitempty"`
	// AdditionalConfig holds platform-specific settings.
	AdditionalConfig map[string]interface{} `json:"additional_config,omitempty" yaml:"additional_config,omitempty"`
}

PlatformConfig defines platform-specific settings for hyperscaler hosting. Platforms are hosting layers (bedrock, vertex, azure) that determine auth and endpoints, while provider type determines message/response handling.

type Pricing

type Pricing struct {
	InputCostPer1K  float64 `json:"input_cost_per_1k" yaml:"input_cost_per_1k"`
	OutputCostPer1K float64 `json:"output_cost_per_1k" yaml:"output_cost_per_1k"`
}

Pricing defines cost per 1K tokens for input and output

type PromptConfigData

type PromptConfigData struct {
	FilePath string            // relative to ConfigDir
	Config   interface{}       // parsed prompt configuration (*prompt.Config at runtime)
	TaskType string            // extracted from Config.Spec.TaskType
	Vars     map[string]string // Variable overrides from arena.yaml
}

PromptConfigData holds a loaded prompt configuration with its file path

type PromptConfigRef

type PromptConfigRef struct {
	ID   string            `yaml:"id"`
	File string            `yaml:"file"`
	Vars map[string]string `yaml:"vars,omitempty"`
}

PromptConfigRef references a prompt builder configuration

type PromptConfigSchema added in v1.1.2

type PromptConfigSchema struct {
	APIVersion string      `yaml:"apiVersion"`
	Kind       string      `yaml:"kind"`
	Metadata   ObjectMeta  `yaml:"metadata,omitempty"`
	Spec       prompt.Spec `yaml:"spec"`
}

PromptConfigSchema represents a PromptConfig in K8s-style manifest format for schema generation

type Provider

type Provider struct {
	ID               string                 `json:"id" yaml:"id"`
	Type             string                 `json:"type" yaml:"type"`
	Model            string                 `json:"model" yaml:"model"`
	BaseURL          string                 `json:"base_url,omitempty" yaml:"base_url,omitempty"`
	RateLimit        RateLimit              `json:"rate_limit,omitempty" yaml:"rate_limit,omitempty"`
	Defaults         ProviderDefaults       `json:"defaults,omitempty" yaml:"defaults,omitempty"`
	Pricing          Pricing                `json:"pricing,omitempty" yaml:"pricing,omitempty"`
	PricingCorrectAt string                 `json:"pricing_correct_at,omitempty" yaml:"pricing_correct_at,omitempty"`
	IncludeRawOutput bool                   `json:"include_raw_output,omitempty" yaml:"include_raw_output,omitempty"` // Include raw API requests in output for debugging
	AdditionalConfig map[string]interface{} `json:"additional_config,omitempty" yaml:"additional_config,omitempty"`   // Additional provider-specific configuration
	Credential       *CredentialConfig      `json:"credential,omitempty" yaml:"credential,omitempty"`
	Platform         *PlatformConfig        `json:"platform,omitempty" yaml:"platform,omitempty"`
	// Capabilities lists what this provider supports: text, streaming, vision, tools, json, audio, video, documents
	Capabilities []string `json:"capabilities,omitempty" yaml:"capabilities,omitempty"`
}

Provider defines API connection and defaults

func LoadProvider

func LoadProvider(filename string) (*Provider, error)

LoadProvider loads and parses a provider configuration from a YAML file in K8s-style manifest format

type ProviderConfig

type ProviderConfig struct {
	APIVersion string     `yaml:"apiVersion"`
	Kind       string     `yaml:"kind"`
	Metadata   ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Provider   `yaml:"spec"`
}

ProviderConfig represents a Provider configuration in K8s-style manifest format

func (*ProviderConfig) GetAPIVersion

func (c *ProviderConfig) GetAPIVersion() string

K8s manifest interface implementation for ProviderConfig

func (*ProviderConfig) GetKind

func (c *ProviderConfig) GetKind() string

func (*ProviderConfig) GetName

func (c *ProviderConfig) GetName() string

func (*ProviderConfig) SetID

func (c *ProviderConfig) SetID(id string)

type ProviderConfigK8s added in v1.1.2

type ProviderConfigK8s struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Provider          `yaml:"spec"`
}

ProviderConfigK8s is the K8s-compatible version for unmarshaling

func (*ProviderConfigK8s) GetAPIVersion added in v1.1.2

func (c *ProviderConfigK8s) GetAPIVersion() string

K8s manifest interface implementation for ProviderConfigK8s

func (*ProviderConfigK8s) GetKind added in v1.1.2

func (c *ProviderConfigK8s) GetKind() string

func (*ProviderConfigK8s) GetName added in v1.1.2

func (c *ProviderConfigK8s) GetName() string

func (*ProviderConfigK8s) SetID added in v1.1.2

func (c *ProviderConfigK8s) SetID(id string)

type ProviderDefaults

type ProviderDefaults struct {
	Temperature float32 `json:"temperature" yaml:"temperature"`
	TopP        float32 `json:"top_p" yaml:"top_p"`
	MaxTokens   int     `json:"max_tokens" yaml:"max_tokens"`
}

ProviderDefaults defines default parameters for a provider

type ProviderRef

type ProviderRef struct {
	File  string `yaml:"file"`
	Group string `yaml:"group,omitempty"`
}

ProviderRef references a provider configuration file

type RateLimit

type RateLimit struct {
	RPS   int `json:"rps" yaml:"rps"`
	Burst int `json:"burst" yaml:"burst"`
}

RateLimit defines rate limiting parameters

type RecordingConfig added in v1.1.6

type RecordingConfig struct {
	Enabled bool   `yaml:"enabled"`       // Enable session recording
	Dir     string `yaml:"dir,omitempty"` // Subdirectory within output.dir (default: "recordings")
}

RecordingConfig contains configuration for session recording. Session recordings capture the complete event stream for replay and analysis.

type RecordingSource added in v1.1.9

type RecordingSource struct {
	// Path to the recording file (relative to eval config file or absolute)
	Path string `json:"path" yaml:"path" jsonschema:"required,description=Path to recording file"`
	// Type hints the recording format: "session", "arena_output", "transcript", or "generic"
	// If omitted, format is auto-detected from file extension
	Type string `` //nolint:lll
	/* 182-byte string literal not displayed */
}

RecordingSource specifies where to load the saved conversation from

type RedisConfig

type RedisConfig struct {
	// Address of the Redis server (e.g., "localhost:6379")
	Address string `yaml:"address"`

	// Password for Redis authentication (optional)
	Password string `yaml:"password,omitempty"`

	// Database number (0-15, default is 0)
	Database int `yaml:"database,omitempty"`

	// TTL for conversation state (e.g., "24h", "7d"). Default is "24h"
	TTL string `yaml:"ttl,omitempty"`

	// Prefix for Redis keys (default is "promptkit")
	Prefix string `yaml:"prefix,omitempty"`
}

RedisConfig contains Redis-specific configuration

type RelevanceConfig added in v1.1.6

type RelevanceConfig struct {
	// Provider specifies the embedding provider: "openai" or "gemini"
	Provider string `json:"provider" yaml:"provider"`

	// Model optionally overrides the default embedding model for the provider
	Model string `json:"model,omitempty" yaml:"model,omitempty"`

	// MinRecentMessages always keeps the N most recent messages regardless of relevance.
	// Default: 3
	MinRecentMessages int `json:"min_recent_messages,omitempty" yaml:"min_recent_messages,omitempty"`

	// AlwaysKeepSystemRole keeps all system role messages regardless of score.
	// Default: true
	AlwaysKeepSystemRole *bool `json:"always_keep_system_role,omitempty" yaml:"always_keep_system_role,omitempty"`

	// SimilarityThreshold is the minimum score (0.0-1.0) to consider a message relevant.
	// Messages below this threshold are dropped first. Default: 0.0 (no threshold)
	SimilarityThreshold float64 `json:"similarity_threshold,omitempty" yaml:"similarity_threshold,omitempty"`

	// QuerySource determines what text to compare messages against.
	// Values: "last_user" (default), "last_n", "custom"
	QuerySource string `json:"query_source,omitempty" yaml:"query_source,omitempty"`

	// LastNCount is the number of messages to use when QuerySource is "last_n".
	// Default: 3
	LastNCount int `json:"last_n_count,omitempty" yaml:"last_n_count,omitempty"`

	// CustomQuery is the query text when QuerySource is "custom".
	CustomQuery string `json:"custom_query,omitempty" yaml:"custom_query,omitempty"`

	// CacheEmbeddings enables caching of embeddings across truncation calls.
	// Default: false
	CacheEmbeddings bool `json:"cache_embeddings,omitempty" yaml:"cache_embeddings,omitempty"`
}

RelevanceConfig configures embedding-based relevance truncation. Used when ContextPolicy.Strategy is "relevance".

func (*RelevanceConfig) Validate added in v1.1.9

func (r *RelevanceConfig) Validate() error

Validate validates the RelevanceConfig settings.

type Scenario

type Scenario struct {
	ID              string                 `json:"id" yaml:"id"`
	TaskType        string                 `json:"task_type" yaml:"task_type"`
	Mode            string                 `json:"mode,omitempty" yaml:"mode,omitempty"`
	Description     string                 `json:"description" yaml:"description"`
	ContextMetadata *ContextMetadata       `json:"context_metadata,omitempty" yaml:"context_metadata,omitempty"`
	Turns           []TurnDefinition       `json:"turns" yaml:"turns"`
	Context         map[string]interface{} `json:"context,omitempty" yaml:"context,omitempty"`
	Constraints     map[string]interface{} `json:"constraints,omitempty" yaml:"constraints,omitempty"`
	ToolPolicy      *ToolPolicy            `json:"tool_policy,omitempty" yaml:"tool_policy,omitempty"`
	// ProvidersOverride: If empty, uses all arena providers.
	Providers     []string `json:"providers,omitempty" yaml:"providers,omitempty"`
	ProviderGroup string   `json:"provider_group,omitempty" yaml:"provider_group,omitempty"`
	// RequiredCapabilities filters providers to only those supporting all listed capabilities.
	// Valid values: text, streaming, vision, tools, json, audio, video, documents
	RequiredCapabilities []string `json:"required_capabilities,omitempty" yaml:"required_capabilities,omitempty"`
	// Enable streaming for all turns by default.
	Streaming bool `json:"streaming,omitempty" yaml:"streaming,omitempty"`
	// Context management policy for long conversations.
	ContextPolicy *ContextPolicy `json:"context_policy,omitempty" yaml:"context_policy,omitempty"`
	// Assertions evaluated after the entire conversation completes.
	ConversationAssertions []asrt.AssertionConfig `json:"conversation_assertions,omitempty" yaml:"conversation_assertions,omitempty"` //nolint:lll
	// Duplex enables bidirectional streaming mode for voice/audio scenarios.
	Duplex *DuplexConfig `json:"duplex,omitempty" yaml:"duplex,omitempty"`
}

Scenario describes user turns, context, and validation constraints

func LoadScenario

func LoadScenario(filename string) (*Scenario, error)

LoadScenario loads and parses a scenario from a YAML file in K8s-style manifest format

func (*Scenario) ShouldStreamTurn

func (s *Scenario) ShouldStreamTurn(turnIndex int) bool

ShouldStreamTurn returns whether streaming should be used for a specific turn. It checks the turn's streaming override first, then falls back to the scenario's streaming setting.

func (*Scenario) Validate added in v1.1.9

func (s *Scenario) Validate() error

Validate validates the Scenario and all nested configurations.

type ScenarioConfig

type ScenarioConfig struct {
	APIVersion string     `yaml:"apiVersion"`
	Kind       string     `yaml:"kind"`
	Metadata   ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Scenario   `yaml:"spec"`
}

ScenarioConfig represents a Scenario in K8s-style manifest format

func (*ScenarioConfig) GetAPIVersion

func (c *ScenarioConfig) GetAPIVersion() string

K8s manifest interface implementation for ScenarioConfig

func (*ScenarioConfig) GetKind

func (c *ScenarioConfig) GetKind() string

func (*ScenarioConfig) GetName

func (c *ScenarioConfig) GetName() string

func (*ScenarioConfig) SetID

func (c *ScenarioConfig) SetID(id string)

type ScenarioConfigK8s added in v1.1.2

type ScenarioConfigK8s struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Scenario          `yaml:"spec"`
}

ScenarioConfigK8s is the K8s-compatible version for unmarshaling

func (*ScenarioConfigK8s) GetAPIVersion added in v1.1.2

func (c *ScenarioConfigK8s) GetAPIVersion() string

K8s manifest interface implementation for ScenarioConfigK8s

func (*ScenarioConfigK8s) GetKind added in v1.1.2

func (c *ScenarioConfigK8s) GetKind() string

func (*ScenarioConfigK8s) GetName added in v1.1.2

func (c *ScenarioConfigK8s) GetName() string

func (*ScenarioConfigK8s) SetID added in v1.1.2

func (c *ScenarioConfigK8s) SetID(id string)

type ScenarioRef

type ScenarioRef struct {
	File string `yaml:"file"`
}

ScenarioRef references a scenario file

type SchemaValidationError added in v1.1.2

type SchemaValidationError struct {
	Field       string
	Description string
	Value       interface{}
}

SchemaValidationError represents a validation error from JSON schema validation

func (SchemaValidationError) Error added in v1.1.2

func (e SchemaValidationError) Error() string

Error implements the error interface

type SchemaValidationResult added in v1.1.2

type SchemaValidationResult struct {
	Valid  bool
	Errors []SchemaValidationError
}

SchemaValidationResult contains the results of schema validation

func ValidateWithLocalSchema added in v1.1.2

func ValidateWithLocalSchema(yamlData []byte, configType ConfigType, schemaDir string) (*SchemaValidationResult, error)

ValidateWithLocalSchema validates YAML data against a local JSON schema file

func ValidateWithSchema added in v1.1.2

func ValidateWithSchema(yamlData []byte, configType ConfigType) (*SchemaValidationResult, error)

ValidateWithSchema validates YAML data against a JSON schema

type SelfPlayConfig

type SelfPlayConfig struct {
	Enabled  bool                `yaml:"enabled"`
	Personas []PersonaRef        `yaml:"personas"`
	Roles    []SelfPlayRoleGroup `yaml:"roles"`
}

SelfPlayConfig configures self-play functionality

type SelfPlayRoleGroup

type SelfPlayRoleGroup struct {
	ID       string `yaml:"id"`
	Provider string `yaml:"provider"` // Provider ID reference (must exist in spec.providers)
}

SelfPlayRoleGroup defines user LLM configuration for self-play

type StateStoreConfig

type StateStoreConfig struct {
	// Type specifies the state store implementation: "memory" or "redis"
	Type string `yaml:"type"`

	// Redis configuration (only used when Type is "redis")
	Redis *RedisConfig `yaml:"redis,omitempty"`
}

StateStoreConfig represents configuration for conversation state storage

type TTSConfig added in v1.1.6

type TTSConfig struct {
	// Provider is the TTS provider (e.g., "openai", "elevenlabs", "cartesia", "mock").
	Provider string `json:"provider" yaml:"provider"`
	// Voice is the voice ID to use for synthesis.
	Voice string `json:"voice" yaml:"voice"`
	// AudioFiles is a list of PCM audio files to use for mock TTS provider.
	// When provider is "mock", these files are loaded and rotated through for each synthesis call.
	// Paths are relative to the scenario file location.
	AudioFiles []string `json:"audio_files,omitempty" yaml:"audio_files,omitempty"`
	// SampleRate is the sample rate of the TTS output in Hz.
	// Default is 24000 for most TTS providers. For mock provider with pre-recorded files,
	// set this to match the actual file sample rate (e.g., 16000 for 16kHz PCM files).
	SampleRate int `json:"sample_rate,omitempty" yaml:"sample_rate,omitempty"`
}

TTSConfig configures text-to-speech for self-play audio generation in duplex mode.

func (*TTSConfig) Validate added in v1.1.6

func (t *TTSConfig) Validate() error

Validate validates the TTSConfig settings.

type ToolConfigSchema added in v1.1.2

type ToolConfigSchema struct {
	APIVersion string     `yaml:"apiVersion"`
	Kind       string     `yaml:"kind"`
	Metadata   ObjectMeta `yaml:"metadata,omitempty"`
	Spec       ToolSpec   `yaml:"spec"`
}

ToolConfigSchema represents a Tool configuration for schema generation with simplified ObjectMeta

type ToolData

type ToolData struct {
	FilePath string
	Data     []byte
}

ToolData holds raw tool configuration data

type ToolPolicy

type ToolPolicy struct {
	ToolChoice          string   `json:"tool_choice" yaml:"tool_choice"` // "auto" | "required" | "none"
	MaxToolCallsPerTurn int      `json:"max_tool_calls_per_turn" yaml:"max_tool_calls_per_turn"`
	MaxTotalToolCalls   int      `json:"max_total_tool_calls" yaml:"max_total_tool_calls"`
	Blocklist           []string `json:"blocklist,omitempty" yaml:"blocklist,omitempty"`
}

ToolPolicy defines constraints for tool usage in scenarios

type ToolRef

type ToolRef struct {
	File string `yaml:"file"`
}

ToolRef references a tool configuration file

type ToolSpec added in v1.1.2

type ToolSpec struct {
	Name         string      `json:"name" yaml:"name"`
	Description  string      `json:"description" yaml:"description"`
	InputSchema  interface{} `json:"input_schema" yaml:"input_schema"`   // JSON Schema Draft-07
	OutputSchema interface{} `json:"output_schema" yaml:"output_schema"` // JSON Schema Draft-07
	Mode         string      `json:"mode" yaml:"mode"`                   // "mock" | "live"
	TimeoutMs    int         `json:"timeout_ms" yaml:"timeout_ms"`
	MockResult   interface{} `json:"mock_result,omitempty" yaml:"mock_result,omitempty"`     // Static mock data
	MockTemplate string      `json:"mock_template,omitempty" yaml:"mock_template,omitempty"` // Template for dynamic mocks
	HTTPConfig   *HTTPConfig `json:"http,omitempty" yaml:"http,omitempty"`                   // Live HTTP configuration
}

ToolSpec represents a tool descriptor (re-exported from runtime/tools for schema generation)

type TurnContentPart added in v1.1.0

type TurnContentPart struct {
	Type string `json:"type" yaml:"type"`                     // "text", "image", "audio", "video"
	Text string `json:"text,omitempty" yaml:"text,omitempty"` // Text content (for type=text)

	// For media content
	Media *TurnMediaContent `json:"media,omitempty" yaml:"media,omitempty"`
}

TurnContentPart represents a content part in a scenario turn (simplified for YAML configuration)

type TurnDefinition

type TurnDefinition struct {
	Role    string `json:"role" yaml:"role"` // "user", "assistant", or provider selector like "claude-user" (only for self-play turns)
	Content string `json:"content,omitempty" yaml:"content,omitempty"`

	// Multimodal content parts (text, images, audio, video)
	// If Parts is non-empty, it takes precedence over Content.
	Parts []TurnContentPart `json:"parts,omitempty" yaml:"parts,omitempty"`

	// Self-play specific fields (when role is a provider selector like "claude-user")
	Persona       string  `json:"persona,omitempty" yaml:"persona,omitempty"`               // Persona ID for self-play
	Turns         int     `json:"turns,omitempty" yaml:"turns,omitempty"`                   // Number of user messages to generate
	AssistantTemp float32 `json:"assistant_temp,omitempty" yaml:"assistant_temp,omitempty"` // Override assistant temperature
	UserTemp      float32 `json:"user_temp,omitempty" yaml:"user_temp,omitempty"`           // Override user temperature
	Seed          int     `json:"seed,omitempty" yaml:"seed,omitempty"`                     // Override seed

	// TTS configures text-to-speech for self-play audio generation in duplex mode.
	// When set, self-play generates audio responses instead of text.
	TTS *TTSConfig `json:"tts,omitempty" yaml:"tts,omitempty"`

	// Streaming control - if nil, uses scenario-level streaming setting
	Streaming *bool `json:"streaming,omitempty" yaml:"streaming,omitempty"` // Override streaming for this turn

	// Turn-level assertions (for testing only)
	Assertions []asrt.AssertionConfig `json:"assertions,omitempty" yaml:"assertions,omitempty"`
}

TurnDefinition represents a single conversation turn definition

type TurnDetectionConfig added in v1.1.6

type TurnDetectionConfig struct {
	// Mode specifies the turn detection method: "vad" (voice activity detection) or "asm" (provider-native).
	Mode string `json:"mode,omitempty" yaml:"mode,omitempty"`
	// VAD contains voice activity detection settings (used when Mode is "vad").
	VAD *VADConfig `json:"vad,omitempty" yaml:"vad,omitempty"`
}

TurnDetectionConfig configures turn detection for duplex mode.

func (*TurnDetectionConfig) Validate added in v1.1.6

func (t *TurnDetectionConfig) Validate() error

Validate validates the TurnDetectionConfig settings.

type TurnMediaContent added in v1.1.0

type TurnMediaContent struct {
	FilePath         string `json:"file_path,omitempty" yaml:"file_path,omitempty"`                 // Relative path to media file (resolved at test time)
	URL              string `json:"url,omitempty" yaml:"url,omitempty"`                             // External URL (http/https)
	Data             string `json:"data,omitempty" yaml:"data,omitempty"`                           // Base64-encoded data (for inline media)
	StorageReference string `json:"storage_reference,omitempty" yaml:"storage_reference,omitempty"` // Storage backend reference (for externalized media)
	MIMEType         string `json:"mime_type" yaml:"mime_type"`                                     // MIME type (e.g., "image/jpeg")
	Detail           string `json:"detail,omitempty" yaml:"detail,omitempty"`                       // Detail level for images: "low", "high", "auto"
	Caption          string `json:"caption,omitempty" yaml:"caption,omitempty"`                     // Optional caption/description
}

TurnMediaContent represents media content in a turn definition

type UserPersonaPack

type UserPersonaPack struct {
	ID             string `json:"id" yaml:"id"`
	Description    string `json:"description" yaml:"description"`
	PromptActivity string `json:"prompt_activity,omitempty" yaml:"prompt_activity,omitempty"` // DEPRECATED: Legacy prompt builder reference

	// NEW: Template system (preferred)
	Fragments      []FragmentRef     `json:"fragments,omitempty" yaml:"fragments,omitempty"`             // Fragment references for composition
	SystemTemplate string            `json:"system_template,omitempty" yaml:"system_template,omitempty"` // Template with {{variables}}
	RequiredVars   []string          `json:"required_vars,omitempty" yaml:"required_vars,omitempty"`     // Variables that must be provided
	OptionalVars   map[string]string `json:"optional_vars,omitempty" yaml:"optional_vars,omitempty"`     // Variables with default values

	// LEGACY: Backward compatibility
	SystemPrompt string `json:"system_prompt,omitempty" yaml:"system_prompt,omitempty"` // Plain text system prompt (no variables)

	Goals       []string        `json:"goals" yaml:"goals"`
	Constraints []string        `json:"constraints" yaml:"constraints"`
	Style       PersonaStyle    `json:"style" yaml:"style"`
	Defaults    PersonaDefaults `json:"defaults" yaml:"defaults"`
}

UserPersonaPack defines how the User LLM behaves in self-play scenarios

func LoadPersona

func LoadPersona(filename string) (*UserPersonaPack, error)

LoadPersona loads a user persona from a YAML or JSON file

func (*UserPersonaPack) BuildSystemPrompt

func (p *UserPersonaPack) BuildSystemPrompt(region string, contextVars map[string]string) (string, error)

BuildSystemPrompt builds the system prompt using template system or falls back to legacy methods. Priority order: 1. system_template + fragments (new, preferred) 2. system_prompt (legacy, plain text) 3. prompt_activity (deprecated, will be removed)

type VADConfig added in v1.1.6

type VADConfig struct {
	// SilenceThresholdMs is the silence duration in milliseconds to trigger turn end (default: 500).
	SilenceThresholdMs int `json:"silence_threshold_ms,omitempty" yaml:"silence_threshold_ms,omitempty"`
	// MinSpeechMs is the minimum speech duration before silence counts (default: 1000).
	MinSpeechMs int `json:"min_speech_ms,omitempty" yaml:"min_speech_ms,omitempty"`
	// MaxTurnDurationS forces turn end after this duration in seconds (default: 60).
	MaxTurnDurationS int `json:"max_turn_duration_s,omitempty" yaml:"max_turn_duration_s,omitempty"`
}

VADConfig configures voice activity detection for turn detection.

func (*VADConfig) Validate added in v1.1.6

func (v *VADConfig) Validate() error

Validate validates the VADConfig settings.

type ValidationCheck added in v1.1.5

type ValidationCheck struct {
	Name    string   // Name of the check (e.g., "Prompt configs exist")
	Passed  bool     // Whether the check passed
	Issues  []string // List of specific issues if failed
	Warning bool     // Whether this is a warning (not an error)
}

ValidationCheck represents a single validation check result

type ValidationError added in v1.1.6

type ValidationError struct {
	Field   string
	Message string
	Value   string
}

ValidationError represents a configuration validation error.

func (*ValidationError) Error added in v1.1.6

func (e *ValidationError) Error() string

type ValidationResult added in v1.1.5

type ValidationResult struct {
	Checks []ValidationCheck
}

ValidationResult contains structured validation results

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL