Documentation
¶
Overview ¶
Package config provides configuration management for PromptKit Arena.
This package handles YAML-based configuration loading and validation for:
- Test scenarios and conversation definitions
- LLM provider settings and credentials
- Prompt template configurations
- Self-play personas and role assignments
- Default parameters and execution settings
Configuration files are loaded from disk with support for file references, allowing modular organization of scenarios, providers, and prompts.
The package is organized into:
- types.go: Core type definitions (Config, Scenario, Provider, etc.)
- loader.go: Loading functions for config files
- helpers.go: Utility functions for path resolution
- persona.go: Self-play persona loading
- validator.go: Configuration validation
Index ¶
- func ResolveConfigDir(cfg *Config, configFilePath string) string
- func ResolveFilePath(basePath, filePath string) string
- func ResolveOutputPath(outDir, filename string) string
- type ArenaConfig
- type Config
- type ConfigValidator
- type ContextMetadata
- type ContextPolicy
- type Defaults
- type FragmentRef
- type HTMLOutputConfig
- type JSONOutputConfig
- type JUnitOutputConfig
- type MCPServerConfig
- type MarkdownConfigdeprecated
- type MarkdownOutputConfig
- type OutputConfig
- type PersonaConfig
- type PersonaDefaults
- type PersonaRef
- type PersonaStyle
- type Pricing
- type PromptConfigData
- type PromptConfigRef
- type Provider
- type ProviderConfig
- type ProviderDefaults
- type ProviderRef
- type RateLimit
- type RedisConfig
- type Scenario
- type ScenarioConfig
- type ScenarioRef
- type SelfPlayConfig
- type SelfPlayRoleGroup
- type StateStoreConfig
- type ToolData
- type ToolPolicy
- type ToolRef
- type TurnContentPart
- type TurnDefinition
- type TurnMediaContent
- type UserPersonaPack
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ResolveConfigDir ¶
ResolveConfigDir resolves the base directory for config files with proper defaulting behavior: 1. If cfg.Defaults.ConfigDir is explicitly set, use it 2. Otherwise, if configFilePath is provided, use its directory 3. Otherwise, default to current working directory (".")
This ensures all config file types (prompts, providers, scenarios, tools) are resolved relative to the same base directory.
func ResolveFilePath ¶
ResolveFilePath resolves a file path relative to a base directory
func ResolveOutputPath ¶
ResolveOutputPath resolves an output file path relative to the output directory. If the filename is an absolute path, it is returned as-is. If the filename is empty, an empty string is returned. Otherwise, the filename is joined with the output directory.
Types ¶
type ArenaConfig ¶
type ArenaConfig struct {
APIVersion string `yaml:"apiVersion"`
Kind string `yaml:"kind"`
Metadata metav1.ObjectMeta `yaml:"metadata,omitempty"`
Spec Config `yaml:"spec"`
}
ArenaConfig represents the main Arena configuration in K8s-style manifest format
type Config ¶
type Config struct {
// File references for YAML serialization
PromptConfigs []PromptConfigRef `yaml:"prompt_configs,omitempty"`
Providers []ProviderRef `yaml:"providers"`
Scenarios []ScenarioRef `yaml:"scenarios"`
Tools []ToolRef `yaml:"tools,omitempty"`
MCPServers []MCPServerConfig `yaml:"mcp_servers,omitempty"`
StateStore *StateStoreConfig `yaml:"state_store,omitempty"`
Defaults Defaults `yaml:"defaults"`
SelfPlay *SelfPlayConfig `yaml:"self_play,omitempty"`
// Loaded resources (populated by LoadConfig, not serialized)
LoadedPromptConfigs map[string]*PromptConfigData `yaml:"-" json:"-"` // taskType -> config
LoadedProviders map[string]*Provider `yaml:"-" json:"-"` // provider ID -> provider
LoadedScenarios map[string]*Scenario `yaml:"-" json:"-"` // scenario ID -> scenario
LoadedTools []ToolData `yaml:"-" json:"-"` // list of tool data
LoadedPersonas map[string]*UserPersonaPack `yaml:"-" json:"-"` // persona ID -> persona
// Base directory for resolving relative paths (set during LoadConfig)
ConfigDir string `yaml:"-" json:"-"`
}
Config represents the main configuration structure
func LoadConfig ¶
LoadConfig loads and validates configuration from a YAML file in K8s-style manifest format. Reads all referenced resource files (scenarios, providers, tools, personas) and populates the Config struct, making it self-contained for programmatic use without physical files.
func (*Config) GetStateStoreConfig ¶
func (c *Config) GetStateStoreConfig() *StateStoreConfig
GetStateStoreConfig returns the state store configuration, creating a default memory config if not specified.
func (*Config) GetStateStoreType ¶
GetStateStoreType returns the configured state store type, defaulting to "memory" if not specified.
type ConfigValidator ¶
type ConfigValidator struct {
// contains filtered or unexported fields
}
ConfigValidator validates configuration consistency and references
func NewConfigValidator ¶
func NewConfigValidator(cfg *Config) *ConfigValidator
NewConfigValidator creates a new configuration validator
func NewConfigValidatorWithPath ¶
func NewConfigValidatorWithPath(cfg *Config, configPath string) *ConfigValidator
NewConfigValidatorWithPath creates a new configuration validator with a config file path for resolving relative file references
func (*ConfigValidator) GetWarnings ¶
func (v *ConfigValidator) GetWarnings() []string
GetWarnings returns all validation warnings
func (*ConfigValidator) Validate ¶
func (v *ConfigValidator) Validate() error
Validate performs comprehensive validation of the configuration
type ContextMetadata ¶
type ContextMetadata struct {
Domain string `json:"domain,omitempty" yaml:"domain,omitempty"`
UserRole string `json:"user_role,omitempty" yaml:"user_role,omitempty"`
ProjectStage string `json:"project_stage,omitempty" yaml:"project_stage,omitempty"`
}
ContextMetadata provides structured context information for scenarios
type ContextPolicy ¶
type ContextPolicy struct {
TokenBudget int `json:"token_budget,omitempty" yaml:"token_budget,omitempty"` // Max tokens (0 = unlimited, default)
ReserveForOutput int `json:"reserve_for_output,omitempty" yaml:"reserve_for_output,omitempty"` // Reserve for response (default 4000)
Strategy string `json:"strategy,omitempty" yaml:"strategy,omitempty"` // "oldest", "summarize", "relevance", "fail"
CacheBreakpoints bool `json:"cache_breakpoints,omitempty" yaml:"cache_breakpoints,omitempty"` // Enable Anthropic caching
}
ContextPolicy defines context management for a scenario
type Defaults ¶
type Defaults struct {
Temperature float32 `yaml:"temperature"`
MaxTokens int `yaml:"max_tokens"`
Seed int `yaml:"seed"`
Concurrency int `yaml:"concurrency"`
Output OutputConfig `yaml:"output"`
// ConfigDir is the base directory for all config files (prompts, providers, scenarios, tools).
// If not set, defaults to the directory containing the main config file.
// If the main config file path is not known, defaults to current working directory.
ConfigDir string `yaml:"config_dir"`
FailOn []string `yaml:"fail_on"`
Verbose bool `yaml:"verbose"`
// Deprecated fields for backward compatibility (will be removed)
HTMLReport string `yaml:"html_report,omitempty"`
OutDir string `yaml:"out_dir,omitempty"`
OutputFormats []string `yaml:"output_formats,omitempty"`
MarkdownConfig *MarkdownConfig `yaml:"markdown_config,omitempty"`
}
Defaults contains default configuration values
func (*Defaults) GetOutputConfig ¶ added in v1.1.0
func (d *Defaults) GetOutputConfig() OutputConfig
GetOutputConfig returns the effective output configuration, handling backward compatibility
type FragmentRef ¶
type FragmentRef struct {
Name string `yaml:"name" json:"name"`
Path string `yaml:"path,omitempty" json:"path,omitempty"` // Optional: relative path to fragment file
Required bool `yaml:"required" json:"required"`
}
FragmentRef references a reusable prompt fragment Reuses the same structure as prompt.FragmentRef for consistency
type HTMLOutputConfig ¶ added in v1.1.0
type HTMLOutputConfig struct {
File string `yaml:"file,omitempty"` // Custom HTML output file name
}
HTMLOutputConfig contains configuration options for HTML output
type JSONOutputConfig ¶ added in v1.1.0
type JSONOutputConfig struct {
}
JSONOutputConfig contains configuration options for JSON output
type JUnitOutputConfig ¶ added in v1.1.0
type JUnitOutputConfig struct {
File string `yaml:"file,omitempty"` // Custom JUnit output file name
}
JUnitOutputConfig contains configuration options for JUnit XML output
type MCPServerConfig ¶
type MCPServerConfig struct {
Name string `yaml:"name"`
Command string `yaml:"command"`
Args []string `yaml:"args,omitempty"`
Env map[string]string `yaml:"env,omitempty"`
}
MCPServerConfig represents configuration for an MCP server
type MarkdownConfig
deprecated
added in
v1.1.0
type MarkdownConfig struct {
IncludeDetails bool `yaml:"include_details"` // Include detailed test information
ShowOverview bool `yaml:"show_overview"` // Show executive overview section
ShowResultsMatrix bool `yaml:"show_results_matrix"` // Show results matrix table
ShowFailedTests bool `yaml:"show_failed_tests"` // Show failed tests section
ShowCostSummary bool `yaml:"show_cost_summary"` // Show cost analysis section
}
Deprecated: Use MarkdownOutputConfig instead
type MarkdownOutputConfig ¶ added in v1.1.0
type MarkdownOutputConfig struct {
File string `yaml:"file,omitempty"` // Custom markdown output file name
IncludeDetails bool `yaml:"include_details"` // Include detailed test information
ShowOverview bool `yaml:"show_overview"` // Show executive overview section
ShowResultsMatrix bool `yaml:"show_results_matrix"` // Show results matrix table
ShowFailedTests bool `yaml:"show_failed_tests"` // Show failed tests section
ShowCostSummary bool `yaml:"show_cost_summary"` // Show cost analysis section
}
MarkdownOutputConfig contains configuration options for markdown output formatting
type OutputConfig ¶ added in v1.1.0
type OutputConfig struct {
Dir string `yaml:"dir"` // Base output directory
Formats []string `yaml:"formats"` // List of enabled formats: json, html, markdown, junit
JSON *JSONOutputConfig `yaml:"json,omitempty"` // JSON-specific configuration
HTML *HTMLOutputConfig `yaml:"html,omitempty"` // HTML-specific configuration
Markdown *MarkdownOutputConfig `yaml:"markdown,omitempty"` // Markdown-specific configuration
JUnit *JUnitOutputConfig `yaml:"junit,omitempty"` // JUnit-specific configuration
}
OutputConfig contains configuration for all output formats
func (*OutputConfig) GetHTMLOutputConfig ¶ added in v1.1.0
func (o *OutputConfig) GetHTMLOutputConfig() *HTMLOutputConfig
GetHTMLOutputConfig returns the HTML configuration with defaults
func (*OutputConfig) GetJUnitOutputConfig ¶ added in v1.1.0
func (o *OutputConfig) GetJUnitOutputConfig() *JUnitOutputConfig
GetJUnitOutputConfig returns the JUnit configuration with defaults
func (*OutputConfig) GetMarkdownOutputConfig ¶ added in v1.1.0
func (o *OutputConfig) GetMarkdownOutputConfig() *MarkdownOutputConfig
GetMarkdownOutputConfig returns the markdown configuration with defaults
type PersonaConfig ¶
type PersonaConfig struct {
APIVersion string `yaml:"apiVersion"`
Kind string `yaml:"kind"`
Metadata metav1.ObjectMeta `yaml:"metadata,omitempty"`
Spec UserPersonaPack `yaml:"spec"`
}
PersonaConfig represents a K8s-style persona configuration manifest
type PersonaDefaults ¶
type PersonaDefaults struct {
Temperature float32 `json:"temperature" yaml:"temperature"`
Seed int `json:"seed" yaml:"seed"`
}
PersonaDefaults contains default LLM parameters for the persona
type PersonaRef ¶
type PersonaRef struct {
File string `yaml:"file"`
}
PersonaRef references a persona file
type PersonaStyle ¶
type PersonaStyle struct {
Verbosity string `json:"verbosity" yaml:"verbosity"` // "short", "medium", "long"
ChallengeLevel string `json:"challenge_level" yaml:"challenge_level"` // "low", "medium", "high"
FrictionTags []string `json:"friction_tags" yaml:"friction_tags"` // e.g., ["skeptical", "impatient", "detail-oriented"]
}
PersonaStyle defines behavioral characteristics for the user persona
type Pricing ¶
type Pricing struct {
InputCostPer1K float64 `json:"input_cost_per_1k" yaml:"input_cost_per_1k"`
OutputCostPer1K float64 `json:"output_cost_per_1k" yaml:"output_cost_per_1k"`
}
Pricing defines cost per 1K tokens for input and output
type PromptConfigData ¶
type PromptConfigData struct {
FilePath string // relative to ConfigDir
Config interface{} // parsed prompt configuration (*prompt.PromptConfig at runtime)
TaskType string // extracted from Config.Spec.TaskType
Vars map[string]string // Variable overrides from arena.yaml
}
PromptConfigData holds a loaded prompt configuration with its file path
type PromptConfigRef ¶
type PromptConfigRef struct {
ID string `yaml:"id"`
File string `yaml:"file"`
Vars map[string]string `yaml:"vars,omitempty"`
}
PromptConfigRef references a prompt builder configuration
type Provider ¶
type Provider struct {
ID string `json:"id" yaml:"id"`
Type string `json:"type" yaml:"type"`
Model string `json:"model" yaml:"model"`
BaseURL string `json:"base_url" yaml:"base_url"`
RateLimit RateLimit `json:"rate_limit" yaml:"rate_limit"`
Defaults ProviderDefaults `json:"defaults" yaml:"defaults"`
Pricing Pricing `json:"pricing" yaml:"pricing"`
PricingCorrectAt string `json:"pricing_correct_at" yaml:"pricing_correct_at"`
IncludeRawOutput bool `json:"include_raw_output" yaml:"include_raw_output"` // Include raw API requests in output for debugging
AdditionalConfig map[string]interface{} `json:"additional_config,omitempty" yaml:"additional_config,omitempty"` // Additional provider-specific configuration
}
Provider defines API connection and defaults
func LoadProvider ¶
LoadProvider loads and parses a provider configuration from a YAML file in K8s-style manifest format
type ProviderConfig ¶
type ProviderConfig struct {
APIVersion string `yaml:"apiVersion"`
Kind string `yaml:"kind"`
Metadata metav1.ObjectMeta `yaml:"metadata,omitempty"`
Spec Provider `yaml:"spec"`
}
ProviderConfig represents a Provider in K8s-style manifest format
func (*ProviderConfig) GetAPIVersion ¶
func (c *ProviderConfig) GetAPIVersion() string
K8s manifest interface implementation for ProviderConfig
func (*ProviderConfig) GetKind ¶
func (c *ProviderConfig) GetKind() string
func (*ProviderConfig) GetName ¶
func (c *ProviderConfig) GetName() string
func (*ProviderConfig) SetID ¶
func (c *ProviderConfig) SetID(id string)
type ProviderDefaults ¶
type ProviderDefaults struct {
Temperature float32 `json:"temperature" yaml:"temperature"`
TopP float32 `json:"top_p" yaml:"top_p"`
MaxTokens int `json:"max_tokens" yaml:"max_tokens"`
}
ProviderDefaults defines default parameters for a provider
type ProviderRef ¶
type ProviderRef struct {
File string `yaml:"file"`
}
ProviderRef references a provider configuration file
type RedisConfig ¶
type RedisConfig struct {
// Address of the Redis server (e.g., "localhost:6379")
Address string `yaml:"address"`
// Password for Redis authentication (optional)
Password string `yaml:"password,omitempty"`
// Database number (0-15, default is 0)
Database int `yaml:"database,omitempty"`
// TTL for conversation state (e.g., "24h", "7d"). Default is "24h"
TTL string `yaml:"ttl,omitempty"`
// Prefix for Redis keys (default is "promptkit")
Prefix string `yaml:"prefix,omitempty"`
}
RedisConfig contains Redis-specific configuration
type Scenario ¶
type Scenario struct {
ID string `json:"id" yaml:"id"`
TaskType string `json:"task_type" yaml:"task_type"`
Mode string `json:"mode,omitempty" yaml:"mode,omitempty"`
Description string `json:"description" yaml:"description"`
ContextMetadata *ContextMetadata `json:"context_metadata,omitempty" yaml:"context_metadata,omitempty"`
Turns []TurnDefinition `json:"turns" yaml:"turns"`
Context map[string]interface{} `json:"context,omitempty" yaml:"context,omitempty"`
Constraints map[string]interface{} `json:"constraints,omitempty" yaml:"constraints,omitempty"`
ToolPolicy *ToolPolicy `json:"tool_policy,omitempty" yaml:"tool_policy,omitempty"`
Providers []string `json:"providers,omitempty" yaml:"providers,omitempty"` // Optional: override which providers to test. If empty, uses all arena providers.
Streaming bool `json:"streaming,omitempty" yaml:"streaming,omitempty"` // Enable streaming for all turns by default
ContextPolicy *ContextPolicy `json:"context_policy,omitempty" yaml:"context_policy,omitempty"` // Context management for long conversations
}
Scenario describes user turns, context, and validation constraints
func LoadScenario ¶
LoadScenario loads and parses a scenario from a YAML file in K8s-style manifest format
func (*Scenario) ShouldStreamTurn ¶
ShouldStreamTurn returns whether streaming should be used for a specific turn. It checks the turn's streaming override first, then falls back to the scenario's streaming setting.
type ScenarioConfig ¶
type ScenarioConfig struct {
APIVersion string `yaml:"apiVersion"`
Kind string `yaml:"kind"`
Metadata metav1.ObjectMeta `yaml:"metadata,omitempty"`
Spec Scenario `yaml:"spec"`
}
ScenarioConfig represents a Scenario in K8s-style manifest format
func (*ScenarioConfig) GetAPIVersion ¶
func (c *ScenarioConfig) GetAPIVersion() string
K8s manifest interface implementation for ScenarioConfig
func (*ScenarioConfig) GetKind ¶
func (c *ScenarioConfig) GetKind() string
func (*ScenarioConfig) GetName ¶
func (c *ScenarioConfig) GetName() string
func (*ScenarioConfig) SetID ¶
func (c *ScenarioConfig) SetID(id string)
type ScenarioRef ¶
type ScenarioRef struct {
File string `yaml:"file"`
}
ScenarioRef references a scenario file
type SelfPlayConfig ¶
type SelfPlayConfig struct {
Enabled bool `yaml:"enabled"`
Personas []PersonaRef `yaml:"personas"`
Roles []SelfPlayRoleGroup `yaml:"roles"`
}
SelfPlayConfig configures self-play functionality
type SelfPlayRoleGroup ¶
type SelfPlayRoleGroup struct {
ID string `yaml:"id"`
Provider string `yaml:"provider"` // Provider ID reference (must exist in spec.providers)
}
SelfPlayRoleGroup defines user LLM configuration for self-play
type StateStoreConfig ¶
type StateStoreConfig struct {
// Type specifies the state store implementation: "memory" or "redis"
Type string `yaml:"type"`
// Redis configuration (only used when Type is "redis")
Redis *RedisConfig `yaml:"redis,omitempty"`
}
StateStoreConfig represents configuration for conversation state storage
type ToolPolicy ¶
type ToolPolicy struct {
ToolChoice string `json:"tool_choice" yaml:"tool_choice"` // "auto" | "required" | "none"
MaxToolCallsPerTurn int `json:"max_tool_calls_per_turn" yaml:"max_tool_calls_per_turn"`
MaxTotalToolCalls int `json:"max_total_tool_calls" yaml:"max_total_tool_calls"`
Blocklist []string `json:"blocklist,omitempty" yaml:"blocklist,omitempty"`
}
ToolPolicy defines constraints for tool usage in scenarios
type ToolRef ¶
type ToolRef struct {
File string `yaml:"file"`
}
ToolRef references a tool configuration file
type TurnContentPart ¶ added in v1.1.0
type TurnContentPart struct {
Type string `json:"type" yaml:"type"` // "text", "image", "audio", "video"
Text string `json:"text,omitempty" yaml:"text,omitempty"` // Text content (for type=text)
// For media content
Media *TurnMediaContent `json:"media,omitempty" yaml:"media,omitempty"`
}
TurnContentPart represents a content part in a scenario turn (simplified for YAML configuration)
type TurnDefinition ¶
type TurnDefinition struct {
Role string `json:"role" yaml:"role"` // "user", "assistant", or provider selector like "claude-user" (only for self-play turns)
Content string `json:"content,omitempty" yaml:"content,omitempty"`
// Multimodal content parts (text, images, audio, video)
// If Parts is non-empty, it takes precedence over Content.
Parts []TurnContentPart `json:"parts,omitempty" yaml:"parts,omitempty"`
// Self-play specific fields (when role is a provider selector like "claude-user")
Persona string `json:"persona,omitempty" yaml:"persona,omitempty"` // Persona ID for self-play
Turns int `json:"turns,omitempty" yaml:"turns,omitempty"` // Number of user messages to generate
AssistantTemp float32 `json:"assistant_temp,omitempty" yaml:"assistant_temp,omitempty"` // Override assistant temperature
UserTemp float32 `json:"user_temp,omitempty" yaml:"user_temp,omitempty"` // Override user temperature
Seed int `json:"seed,omitempty" yaml:"seed,omitempty"` // Override seed
// Streaming control - if nil, uses scenario-level streaming setting
Streaming *bool `json:"streaming,omitempty" yaml:"streaming,omitempty"` // Override streaming for this turn
// Turn-level assertions (for testing only)
Assertions []assertions.AssertionConfig `json:"assertions,omitempty" yaml:"assertions,omitempty"`
}
TurnDefinition represents a single conversation turn definition
type TurnMediaContent ¶ added in v1.1.0
type TurnMediaContent struct {
FilePath string `json:"file_path,omitempty" yaml:"file_path,omitempty"` // Relative path to media file (resolved at test time)
URL string `json:"url,omitempty" yaml:"url,omitempty"` // External URL (http/https)
Data string `json:"data,omitempty" yaml:"data,omitempty"` // Base64-encoded data (for inline media)
MIMEType string `json:"mime_type" yaml:"mime_type"` // MIME type (e.g., "image/jpeg")
Detail string `json:"detail,omitempty" yaml:"detail,omitempty"` // Detail level for images: "low", "high", "auto"
Caption string `json:"caption,omitempty" yaml:"caption,omitempty"` // Optional caption/description
}
TurnMediaContent represents media content in a turn definition
type UserPersonaPack ¶
type UserPersonaPack struct {
ID string `json:"id" yaml:"id"`
Description string `json:"description" yaml:"description"`
PromptActivity string `json:"prompt_activity,omitempty" yaml:"prompt_activity,omitempty"` // DEPRECATED: Legacy prompt builder reference
// NEW: Template system (preferred)
Fragments []FragmentRef `json:"fragments,omitempty" yaml:"fragments,omitempty"` // Fragment references for composition
SystemTemplate string `json:"system_template,omitempty" yaml:"system_template,omitempty"` // Template with {{variables}}
RequiredVars []string `json:"required_vars,omitempty" yaml:"required_vars,omitempty"` // Variables that must be provided
OptionalVars map[string]string `json:"optional_vars,omitempty" yaml:"optional_vars,omitempty"` // Variables with default values
// LEGACY: Backward compatibility
SystemPrompt string `json:"system_prompt,omitempty" yaml:"system_prompt,omitempty"` // Plain text system prompt (no variables)
Goals []string `json:"goals" yaml:"goals"`
Constraints []string `json:"constraints" yaml:"constraints"`
Style PersonaStyle `json:"style" yaml:"style"`
Defaults PersonaDefaults `json:"defaults" yaml:"defaults"`
}
UserPersonaPack defines how the User LLM behaves in self-play scenarios
func LoadPersona ¶
func LoadPersona(filename string) (*UserPersonaPack, error)
LoadPersona loads a user persona from a YAML or JSON file
func (*UserPersonaPack) BuildSystemPrompt ¶
func (p *UserPersonaPack) BuildSystemPrompt(region string, contextVars map[string]string) (string, error)
BuildSystemPrompt builds the system prompt using template system or falls back to legacy methods. Priority order: 1. system_template + fragments (new, preferred) 2. system_prompt (legacy, plain text) 3. prompt_activity (deprecated, will be removed)