Documentation
¶
Index ¶
- func RenderMarkdown(benchmark Benchmark) string
- func WriteBenchmarkJSON(path string, benchmark Benchmark) error
- func WriteMarkdown(path string, benchmark Benchmark) error
- func WritePairJSON(path string, pair PairResult) error
- type ArtifactPaths
- type Benchmark
- type BenchmarkMetadata
- type BenchmarkSummary
- type ConfigurationSummary
- type DeltaSummary
- type NumericSummary
- type PairPaths
- type PairResult
- type PairSpec
- type ProtocolMetrics
- type RunSnapshot
- type ScorecardMetrics
- type TokenDelta
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RenderMarkdown ¶
RenderMarkdown renders a benchmark document as a stable Markdown report.
func WriteBenchmarkJSON ¶
WriteBenchmarkJSON writes benchmark.json to disk.
func WriteMarkdown ¶
WriteMarkdown writes benchmark.md to disk.
func WritePairJSON ¶
func WritePairJSON(path string, pair PairResult) error
WritePairJSON writes one pair result to disk.
Types ¶
type ArtifactPaths ¶
type ArtifactPaths struct {
PairDir string
WithoutSkillRunsDir string
WithSkillRunsDir string
PairJSONPath string
}
ArtifactPaths describes the output layout for one paired benchmark case.
func PairArtifactPaths ¶
func PairArtifactPaths(root, scenarioID, model string, repeat int) ArtifactPaths
PairArtifactPaths returns the standard directory layout for one pair.
type Benchmark ¶
type Benchmark struct {
Metadata BenchmarkMetadata `json:"metadata"`
Pairs []PairResult `json:"pairs"`
Summary BenchmarkSummary `json:"summary"`
}
Benchmark is the machine-readable output for a skill-delta experiment run.
func BuildBenchmark ¶
func BuildBenchmark(metadata BenchmarkMetadata, pairs []PairResult) Benchmark
BuildBenchmark constructs a benchmark document from metadata and pair results.
func ReadBenchmarkJSON ¶
ReadBenchmarkJSON reads a benchmark document from disk.
type BenchmarkMetadata ¶
type BenchmarkMetadata struct {
Suite string `json:"suite"`
GeneratedAt string `json:"generated_at"`
RunsDir string `json:"runs_dir,omitempty"`
Repeats int `json:"repeats"`
Provider string `json:"provider,omitempty"`
Scenarios []string `json:"scenarios,omitempty"`
Models []string `json:"models,omitempty"`
NoSkillPrompt string `json:"no_skill_prompt,omitempty"`
WithSkillPrompt string `json:"with_skill_prompt,omitempty"`
InfraBenchVersion string `json:"infra_bench_version,omitempty"`
InfraBenchCommit string `json:"infra_bench_commit,omitempty"`
ContractVersion string `json:"contract_version,omitempty"`
PromptVersion string `json:"prompt_version,omitempty"`
SkillVersion string `json:"skill_version,omitempty"`
}
BenchmarkMetadata describes the benchmark suite and provenance information.
type BenchmarkSummary ¶
type BenchmarkSummary struct {
PairCount int `json:"pair_count"`
WithoutSkill ConfigurationSummary `json:"without_skill"`
WithSkill ConfigurationSummary `json:"with_skill"`
Delta DeltaSummary `json:"delta"`
}
BenchmarkSummary is the aggregate view of all pair results.
func Aggregate ¶
func Aggregate(pairs []PairResult) BenchmarkSummary
Aggregate computes benchmark summary statistics across all pair results.
type ConfigurationSummary ¶
type ConfigurationSummary struct {
PassRatePct NumericSummary `json:"pass_rate_pct"`
CompliancePct NumericSummary `json:"compliance_pct"`
DurationSeconds NumericSummary `json:"duration_seconds"`
TotalTokens NumericSummary `json:"total_tokens"`
EstimatedCostUSD NumericSummary `json:"estimated_cost_usd"`
Score NumericSummary `json:"score"`
}
ConfigurationSummary aggregates one benchmark configuration across all pairs.
type DeltaSummary ¶
type DeltaSummary struct {
PassRatePct float64 `json:"pass_rate_pct"`
CompliancePct float64 `json:"compliance_pct"`
DurationSeconds float64 `json:"duration_seconds"`
TotalTokens float64 `json:"total_tokens"`
EstimatedCostUSD float64 `json:"estimated_cost_usd"`
Score float64 `json:"score"`
}
DeltaSummary stores configuration deltas as with-skill minus without-skill.
type NumericSummary ¶
type NumericSummary struct {
Mean float64 `json:"mean"`
Stddev float64 `json:"stddev"`
Min float64 `json:"min"`
Max float64 `json:"max"`
}
NumericSummary is a standard aggregate for one numeric metric.
type PairPaths ¶
type PairPaths struct {
WithoutSkillRunDir string `json:"without_skill_run_dir,omitempty"`
WithSkillRunDir string `json:"with_skill_run_dir,omitempty"`
WithoutSkillTranscript string `json:"without_skill_transcript,omitempty"`
WithSkillTranscript string `json:"with_skill_transcript,omitempty"`
WithoutSkillEvidenceDir string `json:"without_skill_evidence_dir,omitempty"`
WithSkillEvidenceDir string `json:"with_skill_evidence_dir,omitempty"`
WithoutSkillScorecard string `json:"without_skill_scorecard,omitempty"`
WithSkillScorecard string `json:"with_skill_scorecard,omitempty"`
}
PairPaths links a pair result back to the underlying local artifacts.
type PairResult ¶
type PairResult struct {
ScenarioID string `json:"scenario_id"`
Model string `json:"model"`
Provider string `json:"provider,omitempty"`
Repeat int `json:"repeat"`
WithoutSkill RunSnapshot `json:"without_skill"`
WithSkill RunSnapshot `json:"with_skill"`
VerdictDelta string `json:"verdict_delta"`
DurationDeltaSeconds float64 `json:"duration_delta_seconds"`
CostDeltaUSD float64 `json:"cost_delta_usd"`
ComplianceDeltaPct float64 `json:"compliance_delta_pct"`
ScoreDelta float64 `json:"score_delta"`
TokenDelta TokenDelta `json:"token_delta"`
Paths PairPaths `json:"paths,omitempty"`
}
PairResult is one without-skill vs with-skill comparison for a single scenario/model/repeat tuple.
func BuildDryRunPair ¶
func BuildDryRunPair(spec PairSpec) PairResult
BuildDryRunPair returns a placeholder pair result for command and pipeline verification when the underlying scenario execution is skipped.
func BuildPairResult ¶
func BuildPairResult(withoutSkillDir, withSkillDir string) (PairResult, error)
BuildPairResult loads two run artifact directories and returns a normalized skill-delta pair result.
func LoadPairs ¶
func LoadPairs(root string) ([]PairResult, error)
LoadPairs walks a benchmark directory and loads all pair.json files.
type PairSpec ¶
type PairSpec struct {
ScenarioID string
Model string
Provider string
Repeat int
Paths ArtifactPaths
}
PairSpec identifies one scenario/model/repeat pair.
type ProtocolMetrics ¶
type ProtocolMetrics struct {
PrescribeCount int `json:"prescribe_count"`
ReportCount int `json:"report_count"`
OrphanedPrescriptions int `json:"orphaned_prescriptions"`
DeclinedCount int `json:"declined_count"`
ChecksPassed int `json:"checks_passed"`
ChecksTotal int `json:"checks_total"`
VerdictCoveragePct float64 `json:"verdict_coverage_pct"`
ComplianceRatePct float64 `json:"compliance_rate_pct"`
}
ProtocolMetrics captures protocol-compliance specific counters for one run.
type RunSnapshot ¶
type RunSnapshot struct {
RunID string `json:"run_id,omitempty"`
Passed bool `json:"passed"`
ExitCode int `json:"exit_code,omitempty"`
DurationSeconds float64 `json:"duration_seconds"`
Turns int `json:"turns"`
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
EstimatedCostUSD float64 `json:"estimated_cost_usd"`
ChecksPassed int `json:"checks_passed"`
ChecksTotal int `json:"checks_total"`
Protocol ProtocolMetrics `json:"protocol"`
Scorecard ScorecardMetrics `json:"scorecard"`
Metadata map[string]string `json:"metadata,omitempty"`
}
RunSnapshot holds the normalized metrics for a single benchmark run.
type ScorecardMetrics ¶
type ScorecardMetrics struct {
Available bool `json:"available"`
Score *float64 `json:"score,omitempty"`
Band string `json:"band,omitempty"`
Signals []string `json:"signals,omitempty"`
SignalCounts map[string]int `json:"signal_counts,omitempty"`
}
ScorecardMetrics captures scorecard output derived from benchmark evidence.
type TokenDelta ¶
type TokenDelta struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
}
TokenDelta captures the change in token use between configurations.
Source Files