config

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 2, 2025 License: Apache-2.0 Imports: 10 Imported by: 7

Documentation

Overview

Package config provides configuration management for PromptKit Arena.

This package handles YAML-based configuration loading and validation for:

  • Test scenarios and conversation definitions
  • LLM provider settings and credentials
  • Prompt template configurations
  • Self-play personas and role assignments
  • Default parameters and execution settings

Configuration files are loaded from disk with support for file references, allowing modular organization of scenarios, providers, and prompts.

The package is organized into:

  • types.go: Core type definitions (Config, Scenario, Provider, etc.)
  • loader.go: Loading functions for config files
  • helpers.go: Utility functions for path resolution
  • persona.go: Self-play persona loading
  • validator.go: Configuration validation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ResolveConfigDir 

func ResolveConfigDir(cfg *Config, configFilePath string) string

ResolveConfigDir resolves the base directory for config files with proper defaulting behavior: 1. If cfg.Defaults.ConfigDir is explicitly set, use it 2. Otherwise, if configFilePath is provided, use its directory 3. Otherwise, default to current working directory (".")

This ensures all config file types (prompts, providers, scenarios, tools) are resolved relative to the same base directory.

func ResolveFilePath 

func ResolveFilePath(basePath, filePath string) string

ResolveFilePath resolves a file path relative to a base directory

func ResolveOutputPath 

func ResolveOutputPath(outDir, filename string) string

ResolveOutputPath resolves an output file path relative to the output directory. If the filename is an absolute path, it is returned as-is. If the filename is empty, an empty string is returned. Otherwise, the filename is joined with the output directory.

Types

type ArenaConfig 

type ArenaConfig struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Config            `yaml:"spec"`
}

ArenaConfig represents the main Arena configuration in K8s-style manifest format

type Config 

type Config struct {
	// File references for YAML serialization
	PromptConfigs []PromptConfigRef `yaml:"prompt_configs,omitempty"`
	Providers     []ProviderRef     `yaml:"providers"`
	Scenarios     []ScenarioRef     `yaml:"scenarios"`
	Tools         []ToolRef         `yaml:"tools,omitempty"`
	MCPServers    []MCPServerConfig `yaml:"mcp_servers,omitempty"`
	StateStore    *StateStoreConfig `yaml:"state_store,omitempty"`
	Defaults      Defaults          `yaml:"defaults"`
	SelfPlay      *SelfPlayConfig   `yaml:"self_play,omitempty"`

	// Loaded resources (populated by LoadConfig, not serialized)
	LoadedPromptConfigs map[string]*PromptConfigData `yaml:"-" json:"-"` // taskType -> config
	LoadedProviders     map[string]*Provider         `yaml:"-" json:"-"` // provider ID -> provider
	LoadedScenarios     map[string]*Scenario         `yaml:"-" json:"-"` // scenario ID -> scenario
	LoadedTools         []ToolData                   `yaml:"-" json:"-"` // list of tool data
	LoadedPersonas      map[string]*UserPersonaPack  `yaml:"-" json:"-"` // persona ID -> persona

	// Base directory for resolving relative paths (set during LoadConfig)
	ConfigDir string `yaml:"-" json:"-"`
}

Config represents the main configuration structure

func LoadConfig 

func LoadConfig(filename string) (*Config, error)

LoadConfig loads and validates configuration from a YAML file in K8s-style manifest format. Reads all referenced resource files (scenarios, providers, tools, personas) and populates the Config struct, making it self-contained for programmatic use without physical files.

func (*Config) GetStateStoreConfig 

func (c *Config) GetStateStoreConfig() *StateStoreConfig

GetStateStoreConfig returns the state store configuration, creating a default memory config if not specified.

func (*Config) GetStateStoreType 

func (c *Config) GetStateStoreType() string

GetStateStoreType returns the configured state store type, defaulting to "memory" if not specified.

type ConfigValidator 

type ConfigValidator struct {
	// contains filtered or unexported fields
}

ConfigValidator validates configuration consistency and references

func NewConfigValidator 

func NewConfigValidator(cfg *Config) *ConfigValidator

NewConfigValidator creates a new configuration validator

func NewConfigValidatorWithPath 

func NewConfigValidatorWithPath(cfg *Config, configPath string) *ConfigValidator

NewConfigValidatorWithPath creates a new configuration validator with a config file path for resolving relative file references

func (*ConfigValidator) GetWarnings 

func (v *ConfigValidator) GetWarnings() []string

GetWarnings returns all validation warnings

func (*ConfigValidator) Validate 

func (v *ConfigValidator) Validate() error

Validate performs comprehensive validation of the configuration

type ContextMetadata 

type ContextMetadata struct {
	Domain       string `json:"domain,omitempty" yaml:"domain,omitempty"`
	UserRole     string `json:"user_role,omitempty" yaml:"user_role,omitempty"`
	ProjectStage string `json:"project_stage,omitempty" yaml:"project_stage,omitempty"`
}

ContextMetadata provides structured context information for scenarios

type ContextPolicy 

type ContextPolicy struct {
	TokenBudget      int    `json:"token_budget,omitempty" yaml:"token_budget,omitempty"`             // Max tokens (0 = unlimited, default)
	ReserveForOutput int    `json:"reserve_for_output,omitempty" yaml:"reserve_for_output,omitempty"` // Reserve for response (default 4000)
	Strategy         string `json:"strategy,omitempty" yaml:"strategy,omitempty"`                     // "oldest", "summarize", "relevance", "fail"
	CacheBreakpoints bool   `json:"cache_breakpoints,omitempty" yaml:"cache_breakpoints,omitempty"`   // Enable Anthropic caching
}

ContextPolicy defines context management for a scenario

type Defaults 

type Defaults struct {
	Temperature float32 `yaml:"temperature"`
	MaxTokens   int     `yaml:"max_tokens"`
	Seed        int     `yaml:"seed"`
	Concurrency int     `yaml:"concurrency"`
	HTMLReport  string  `yaml:"html_report"`
	OutDir      string  `yaml:"out_dir"`
	// ConfigDir is the base directory for all config files (prompts, providers, scenarios, tools).
	// If not set, defaults to the directory containing the main config file.
	// If the main config file path is not known, defaults to current working directory.
	ConfigDir string   `yaml:"config_dir"`
	FailOn    []string `yaml:"fail_on"`
	Verbose   bool     `yaml:"verbose"`
}

Defaults contains default configuration values

type FragmentRef 

type FragmentRef struct {
	Name     string `yaml:"name" json:"name"`
	Path     string `yaml:"path,omitempty" json:"path,omitempty"` // Optional: relative path to fragment file
	Required bool   `yaml:"required" json:"required"`
}

FragmentRef references a reusable prompt fragment Reuses the same structure as prompt.FragmentRef for consistency

type MCPServerConfig 

type MCPServerConfig struct {
	Name    string            `yaml:"name"`
	Command string            `yaml:"command"`
	Args    []string          `yaml:"args,omitempty"`
	Env     map[string]string `yaml:"env,omitempty"`
}

MCPServerConfig represents configuration for an MCP server

type PersonaConfig 

type PersonaConfig struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       UserPersonaPack   `yaml:"spec"`
}

PersonaConfig represents a K8s-style persona configuration manifest

type PersonaDefaults 

type PersonaDefaults struct {
	Temperature float32 `json:"temperature" yaml:"temperature"`
	Seed        int     `json:"seed" yaml:"seed"`
}

PersonaDefaults contains default LLM parameters for the persona

type PersonaRef 

type PersonaRef struct {
	File string `yaml:"file"`
}

PersonaRef references a persona file

type PersonaStyle 

type PersonaStyle struct {
	Verbosity      string   `json:"verbosity" yaml:"verbosity"`             // "short", "medium", "long"
	ChallengeLevel string   `json:"challenge_level" yaml:"challenge_level"` // "low", "medium", "high"
	FrictionTags   []string `json:"friction_tags" yaml:"friction_tags"`     // e.g., ["skeptical", "impatient", "detail-oriented"]
}

PersonaStyle defines behavioral characteristics for the user persona

type Pricing 

type Pricing struct {
	InputCostPer1K  float64 `json:"input_cost_per_1k" yaml:"input_cost_per_1k"`
	OutputCostPer1K float64 `json:"output_cost_per_1k" yaml:"output_cost_per_1k"`
}

Pricing defines cost per 1K tokens for input and output

type PromptConfigData 

type PromptConfigData struct {
	FilePath string      // relative to ConfigDir
	Config   interface{} // parsed prompt configuration (*prompt.PromptConfig at runtime)
	TaskType string      // extracted from Config.Spec.TaskType
}

PromptConfigData holds a loaded prompt configuration with its file path

type PromptConfigRef 

type PromptConfigRef struct {
	ID   string `yaml:"id"`
	File string `yaml:"file"`
}

PromptConfigRef references a prompt builder configuration

type Provider 

type Provider struct {
	ID               string           `json:"id" yaml:"id"`
	Type             string           `json:"type" yaml:"type"`
	Model            string           `json:"model" yaml:"model"`
	BaseURL          string           `json:"base_url" yaml:"base_url"`
	RateLimit        RateLimit        `json:"rate_limit" yaml:"rate_limit"`
	Defaults         ProviderDefaults `json:"defaults" yaml:"defaults"`
	Pricing          Pricing          `json:"pricing" yaml:"pricing"`
	PricingCorrectAt string           `json:"pricing_correct_at" yaml:"pricing_correct_at"`
	IncludeRawOutput bool             `json:"include_raw_output" yaml:"include_raw_output"` // Include raw API requests in output for debugging
}

Provider defines API connection and defaults

func LoadProvider 

func LoadProvider(filename string) (*Provider, error)

LoadProvider loads and parses a provider configuration from a YAML file in K8s-style manifest format

type ProviderConfig 

type ProviderConfig struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Provider          `yaml:"spec"`
}

ProviderConfig represents a Provider in K8s-style manifest format

func (*ProviderConfig) GetAPIVersion 

func (c *ProviderConfig) GetAPIVersion() string

K8s manifest interface implementation for ProviderConfig

func (*ProviderConfig) GetKind 

func (c *ProviderConfig) GetKind() string

func (*ProviderConfig) GetName 

func (c *ProviderConfig) GetName() string

func (*ProviderConfig) SetID 

func (c *ProviderConfig) SetID(id string)

type ProviderDefaults 

type ProviderDefaults struct {
	Temperature float32 `json:"temperature" yaml:"temperature"`
	TopP        float32 `json:"top_p" yaml:"top_p"`
	MaxTokens   int     `json:"max_tokens" yaml:"max_tokens"`
}

ProviderDefaults defines default parameters for a provider

type ProviderRef 

type ProviderRef struct {
	File string `yaml:"file"`
}

ProviderRef references a provider configuration file

type RateLimit 

type RateLimit struct {
	RPS   int `json:"rps" yaml:"rps"`
	Burst int `json:"burst" yaml:"burst"`
}

RateLimit defines rate limiting parameters

type RedisConfig 

type RedisConfig struct {
	// Address of the Redis server (e.g., "localhost:6379")
	Address string `yaml:"address"`

	// Password for Redis authentication (optional)
	Password string `yaml:"password,omitempty"`

	// Database number (0-15, default is 0)
	Database int `yaml:"database,omitempty"`

	// TTL for conversation state (e.g., "24h", "7d"). Default is "24h"
	TTL string `yaml:"ttl,omitempty"`

	// Prefix for Redis keys (default is "promptkit")
	Prefix string `yaml:"prefix,omitempty"`
}

RedisConfig contains Redis-specific configuration

type Scenario 

type Scenario struct {
	ID              string                 `json:"id" yaml:"id"`
	TaskType        string                 `json:"task_type" yaml:"task_type"`
	Mode            string                 `json:"mode,omitempty" yaml:"mode,omitempty"`
	Description     string                 `json:"description" yaml:"description"`
	ContextMetadata *ContextMetadata       `json:"context_metadata,omitempty" yaml:"context_metadata,omitempty"`
	Turns           []TurnDefinition       `json:"turns" yaml:"turns"`
	Context         map[string]interface{} `json:"context,omitempty" yaml:"context,omitempty"`
	Constraints     map[string]interface{} `json:"constraints,omitempty" yaml:"constraints,omitempty"`
	ToolPolicy      *ToolPolicy            `json:"tool_policy,omitempty" yaml:"tool_policy,omitempty"`
	Providers       []string               `json:"providers,omitempty" yaml:"providers,omitempty"`           // Optional: override which providers to test. If empty, uses all arena providers.
	Streaming       bool                   `json:"streaming,omitempty" yaml:"streaming,omitempty"`           // Enable streaming for all turns by default
	ContextPolicy   *ContextPolicy         `json:"context_policy,omitempty" yaml:"context_policy,omitempty"` // Context management for long conversations
}

Scenario describes user turns, context, and validation constraints

func LoadScenario 

func LoadScenario(filename string) (*Scenario, error)

LoadScenario loads and parses a scenario from a YAML file in K8s-style manifest format

func (*Scenario) ShouldStreamTurn 

func (s *Scenario) ShouldStreamTurn(turnIndex int) bool

ShouldStreamTurn returns whether streaming should be used for a specific turn. It checks the turn's streaming override first, then falls back to the scenario's streaming setting.

type ScenarioConfig 

type ScenarioConfig struct {
	APIVersion string            `yaml:"apiVersion"`
	Kind       string            `yaml:"kind"`
	Metadata   metav1.ObjectMeta `yaml:"metadata,omitempty"`
	Spec       Scenario          `yaml:"spec"`
}

ScenarioConfig represents a Scenario in K8s-style manifest format

func (*ScenarioConfig) GetAPIVersion 

func (c *ScenarioConfig) GetAPIVersion() string

K8s manifest interface implementation for ScenarioConfig

func (*ScenarioConfig) GetKind 

func (c *ScenarioConfig) GetKind() string

func (*ScenarioConfig) GetName 

func (c *ScenarioConfig) GetName() string

func (*ScenarioConfig) SetID 

func (c *ScenarioConfig) SetID(id string)

type ScenarioRef 

type ScenarioRef struct {
	File string `yaml:"file"`
}

ScenarioRef references a scenario file

type SelfPlayConfig 

type SelfPlayConfig struct {
	Enabled  bool                `yaml:"enabled"`
	Personas []PersonaRef        `yaml:"personas"`
	Roles    []SelfPlayRoleGroup `yaml:"roles"`
}

SelfPlayConfig configures self-play functionality

type SelfPlayRoleGroup 

type SelfPlayRoleGroup struct {
	ID       string `yaml:"id"`
	Provider string `yaml:"provider"` // Provider ID reference (must exist in spec.providers)
}

SelfPlayRoleGroup defines user LLM configuration for self-play

type StateStoreConfig 

type StateStoreConfig struct {
	// Type specifies the state store implementation: "memory" or "redis"
	Type string `yaml:"type"`

	// Redis configuration (only used when Type is "redis")
	Redis *RedisConfig `yaml:"redis,omitempty"`
}

StateStoreConfig represents configuration for conversation state storage

type ToolData 

type ToolData struct {
	FilePath string
	Data     []byte
}

ToolData holds raw tool configuration data

type ToolPolicy 

type ToolPolicy struct {
	ToolChoice          string   `json:"tool_choice" yaml:"tool_choice"` // "auto" | "required" | "none"
	MaxToolCallsPerTurn int      `json:"max_tool_calls_per_turn" yaml:"max_tool_calls_per_turn"`
	MaxTotalToolCalls   int      `json:"max_total_tool_calls" yaml:"max_total_tool_calls"`
	Blocklist           []string `json:"blocklist,omitempty" yaml:"blocklist,omitempty"`
}

ToolPolicy defines constraints for tool usage in scenarios

type ToolRef 

type ToolRef struct {
	File string `yaml:"file"`
}

ToolRef references a tool configuration file

type TurnDefinition 

type TurnDefinition struct {
	Role    string `json:"role" yaml:"role"` // "user", "assistant", or provider selector like "claude-user" (only for self-play turns)
	Content string `json:"content,omitempty" yaml:"content,omitempty"`

	// Self-play specific fields (when role is a provider selector like "claude-user")
	Persona       string  `json:"persona,omitempty" yaml:"persona,omitempty"`               // Persona ID for self-play
	Turns         int     `json:"turns,omitempty" yaml:"turns,omitempty"`                   // Number of user messages to generate
	AssistantTemp float32 `json:"assistant_temp,omitempty" yaml:"assistant_temp,omitempty"` // Override assistant temperature
	UserTemp      float32 `json:"user_temp,omitempty" yaml:"user_temp,omitempty"`           // Override user temperature
	Seed          int     `json:"seed,omitempty" yaml:"seed,omitempty"`                     // Override seed

	// Streaming control - if nil, uses scenario-level streaming setting
	Streaming *bool `json:"streaming,omitempty" yaml:"streaming,omitempty"` // Override streaming for this turn

	// Turn-level assertions (for testing only)
	Assertions []validators.ValidatorConfig `json:"assertions,omitempty" yaml:"assertions,omitempty"`
}

TurnDefinition represents a single conversation turn definition

type UserPersonaPack 

type UserPersonaPack struct {
	ID             string `json:"id" yaml:"id"`
	Description    string `json:"description" yaml:"description"`
	PromptActivity string `json:"prompt_activity,omitempty" yaml:"prompt_activity,omitempty"` // DEPRECATED: Legacy prompt builder reference

	// NEW: Template system (preferred)
	Fragments      []FragmentRef     `json:"fragments,omitempty" yaml:"fragments,omitempty"`             // Fragment references for composition
	SystemTemplate string            `json:"system_template,omitempty" yaml:"system_template,omitempty"` // Template with {{variables}}
	RequiredVars   []string          `json:"required_vars,omitempty" yaml:"required_vars,omitempty"`     // Variables that must be provided
	OptionalVars   map[string]string `json:"optional_vars,omitempty" yaml:"optional_vars,omitempty"`     // Variables with default values

	// LEGACY: Backward compatibility
	SystemPrompt string `json:"system_prompt,omitempty" yaml:"system_prompt,omitempty"` // Plain text system prompt (no variables)

	Goals       []string        `json:"goals" yaml:"goals"`
	Constraints []string        `json:"constraints" yaml:"constraints"`
	Style       PersonaStyle    `json:"style" yaml:"style"`
	Defaults    PersonaDefaults `json:"defaults" yaml:"defaults"`
}

UserPersonaPack defines how the User LLM behaves in self-play scenarios

func LoadPersona 

func LoadPersona(filename string) (*UserPersonaPack, error)

LoadPersona loads a user persona from a YAML or JSON file

func (*UserPersonaPack) BuildSystemPrompt 

func (p *UserPersonaPack) BuildSystemPrompt(region string, contextVars map[string]string) (string, error)

BuildSystemPrompt builds the system prompt using template system or falls back to legacy methods. Priority order: 1. system_template + fragments (new, preferred) 2. system_prompt (legacy, plain text) 3. prompt_activity (deprecated, will be removed)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.