Documentation
¶
Overview ¶
Package profile is the canonical reader for v7 benchmark profile YAML files under scripts/benchmark/profiles/. The schema is frozen by SD-010 §3 (Harness × Model Matrix Benchmark); additive fields require a spec amendment.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Limits ¶
type Limits struct {
MaxOutputTokens int `yaml:"max_output_tokens"`
ContextTokens int `yaml:"context_tokens"`
RateLimitRPM int `yaml:"rate_limit_rpm"`
RateLimitTPM int `yaml:"rate_limit_tpm"`
}
Limits captures provider-side ceilings. rate_limit_* are informational in v1 (D6 forbids concurrency > 1) and reserved for the follow-up scheduler.
type ModelServerInfo ¶ added in v0.9.29
type ModelServerInfo struct {
Quantization string `json:"quantization,omitempty"`
LoadedContextLength int `json:"loaded_context_length,omitempty"`
MaxContextLength int `json:"max_context_length,omitempty"`
Source string `json:"source,omitempty"` // URL queried
}
ModelServerInfo is populated at run time by querying the local model server (e.g. lmstudio /api/v0/models/<id>). Fields are empty/zero when the server does not expose them.
type Pricing ¶
type Pricing struct {
InputUSDPerMTok float64 `yaml:"input_usd_per_mtok"`
OutputUSDPerMTok float64 `yaml:"output_usd_per_mtok"`
CachedInputUSDPerMTok float64 `yaml:"cached_input_usd_per_mtok"`
}
Pricing is the single source of truth for cost reconciliation. Units are USD per million tokens.
type Profile ¶
type Profile struct {
ID string `yaml:"id"`
Provider Provider `yaml:"provider"`
Pricing Pricing `yaml:"pricing"`
Limits Limits `yaml:"limits"`
Sampling Sampling `yaml:"sampling"`
Versioning Versioning `yaml:"versioning"`
// Path is the filesystem path the profile was loaded from. Not part of
// the YAML; populated by Load / LoadDir for diagnostics and `profiles
// list` output.
Path string `yaml:"-"`
}
Profile is the in-memory shape of a frozen v1 profile YAML.
type Provider ¶
type Provider struct {
Type ProviderType `yaml:"type"`
Model string `yaml:"model"`
BaseURL string `yaml:"base_url"`
APIKeyEnv string `yaml:"api_key_env"`
}
Provider describes how the harness adapter should reach the model API.
type ProviderType ¶
type ProviderType string
ProviderType enumerates the provider client paths supported by the matrix runner. SD-010 §3 forbids silent override; the adapter selects its client path by exact match on this field.
const ( ProviderAnthropic ProviderType = "anthropic" ProviderOpenAI ProviderType = "openai" ProviderOpenAICompat ProviderType = "openai-compat" ProviderOpenRouter ProviderType = "openrouter" ProviderOMLX ProviderType = "omlx" ProviderLMStudio ProviderType = "lmstudio" ProviderOllama ProviderType = "ollama" ProviderGoogle ProviderType = "google" ProviderRapidMLX ProviderType = "rapid-mlx" ProviderVLLM ProviderType = "vllm" )
type Sampling ¶
type Sampling struct {
Temperature float64 `yaml:"temperature"`
Reasoning string `yaml:"reasoning,omitempty"`
TopP *float64 `yaml:"top_p,omitempty"`
TopK *int `yaml:"top_k,omitempty"`
MinP *float64 `yaml:"min_p,omitempty"`
}
Sampling is opaque to the runner; passed verbatim to the adapter's apply_profile step. Reasoning is a free-form string ("low" | "medium" | "high" | "" depending on the family). Pointer fields are omitted when nil so server defaults apply.
type Versioning ¶
Versioning records when the profile was authored and which provider snapshot the adapter resolved at apply_profile time.
Source Files