bench

package
v0.1.0-beta.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package bench implements the scenario benchmark harness: a controlled, local-first, reproducible comparison of direct-MCP vs Ozy-brokered agent performance against a frozen incident scenario.

Index

Constants

View Source 
const (
	LedgerKindSystemPrompt     = "system_prompt"
	LedgerKindAgentInstruction = "agent_instruction"
	LedgerKindToolSchema       = "tool_schema"
	LedgerKindToolCall         = "tool_call"
	LedgerKindToolResult       = "tool_result"
	LedgerKindAssistantMessage = "assistant_message"
	LedgerKindUserMessage      = "user_message"
	LedgerKindError            = "error"
	LedgerKindFinalAnswer      = "final_answer"
)

LedgerKind values categorize ledger entries by their role in the context.

View Source 
const (
	TokenSourceMeasured  = "measured"
	TokenSourceEstimated = "estimated"
)

TokenSource values label how token counts were obtained.

View Source 
const Version = "0.1.0-dev"

Version is the build version reported by `ozy-bench --version`.

Variables

This section is empty.

Functions

func ComputeFixtureHash 

func ComputeFixtureHash(dir string) (string, error)

ComputeFixtureHash returns a SHA-256 hash of the fixture directory contents, used to detect fixture changes across runs.

func Execute 

func Execute(args []string, stdout, stderr io.Writer) int

Execute runs the CLI with the given args and writers, returning the process exit code.

func GenerateIncidentDB 

func GenerateIncidentDB(targetDir string) error

GenerateIncidentDB creates the read-only incident SQLite database with rows for suspended accounts incorrectly invoiced on 2026-06-14.

func ResolveRunCount 

func ResolveRunCount(cliRuns int, cfg *ScenarioConfig) int

ResolveRunCount returns the effective run count: CLI flag, then BENCH_RUNS env, then scenario config default.

func SanitizeBaseURL 

func SanitizeBaseURL(raw string) string

SanitizeBaseURL strips path and query components, keeping only scheme://host[:port].

func ServeMCP 

func ServeMCP(toolset, fixtureDir string) error

ServeMCP creates an MCP server for the given toolset and serves it over stdio. fixtureDir is used by toolsets that need access to the fixture directory (code-search, git, incident-db, filesystem).

func WriteBreakdown 

func WriteBreakdown(path string, bd *ContextBreakdown) error

WriteBreakdown writes a context breakdown as indented JSON.

func WriteComparison 

func WriteComparison(dir string, c *SurfaceComparison) error

WriteComparison writes the surface comparison as JSON and Markdown.

func WriteGradingResult 

func WriteGradingResult(path string, result *GradingResult) error

WriteGradingResult writes the grading result as JSON to path.

func WriteProvenance 

func WriteProvenance(path string, record *EnvironmentRecord) error

WriteProvenance writes the environment record as JSON to the given path.

func WriteSurfaceMetrics 

func WriteSurfaceMetrics(path string, m *SurfaceMetrics) error

WriteSurfaceMetrics writes surface metrics to path.

Types

type CategoryTokens 

type CategoryTokens struct {
	SystemPrompt        int `json:"systemPrompt"`
	ToolDefinitions     int `json:"toolDefinitions"`
	ToolResults         int `json:"toolResults"`
	FileContents        int `json:"fileContents"`
	ConversationHistory int `json:"conversationHistory"`
	CurrentUserMessage  int `json:"currentUserMessage"`
	AssistantPrefill    int `json:"assistantPrefill"`
	Uncategorized       int `json:"uncategorized"`
	TotalInput          int `json:"totalInput"`
	TotalOutput         int `json:"totalOutput"`
}

CategoryTokens is ContextSpy's per-request input/output token split by role in the context window. Fields mirror the requests table's tokens_* columns.

type ContextBreakdown 

type ContextBreakdown struct {
	Session  string          `json:"session"`
	Requests []RequestTokens `json:"requests"`
	Totals   CategoryTokens  `json:"totals"`
	Tools    []ToolTokens    `json:"tools,omitempty"`
}

ContextBreakdown is the per-run capture: every request's breakdown (the tool surface is re-sent each turn, so any single request shows its share), the summed totals, and the per-tool schema cost.

type ContextSpy 

type ContextSpy struct {
	// contains filtered or unexported fields
}

ContextSpy drives a ContextSpy proxy over its HTTP API: it opens a named session per run (so the proxy tags that run's model requests with it) and reads the per-run token breakdown back from the API. It is best-effort — when CONTEXTSPY_API is unset, New returns a disabled spy whose methods no-op so the bench still runs; it just won't emit context-breakdown.json.

This is the containerized path: the bench runs hermetically in Docker and reaches a host-side ContextSpy over HTTP (e.g. host.docker.internal:5173), never touching the host filesystem.

func NewContextSpy 

func NewContextSpy() *ContextSpy

NewContextSpy reads CONTEXTSPY_API (e.g. "http://host.docker.internal:5173"). The spy is disabled (a no-op) when it is unset.

func (*ContextSpy) Breakdown 

func (c *ContextSpy) Breakdown(ctx context.Context, name string) (*ContextBreakdown, error)

Breakdown fetches the active session's per-request breakdown and per-tool token totals from the ContextSpy API. Returns nil (no error) when the spy is disabled, no session is active, or the session captured no requests.

func (*ContextSpy) EndSession 

func (c *ContextSpy) EndSession(ctx context.Context)

EndSession closes the active session. Best-effort.

func (*ContextSpy) StartSession 

func (c *ContextSpy) StartSession(ctx context.Context, name string)

StartSession opens a named session and remembers its id, so requests during the run are tagged with it. Best-effort: on failure the run proceeds with no active session (Breakdown then no-ops). Runs are sequential, so a single active session id on the struct is sufficient.

type CriterionResult 

type CriterionResult struct {
	Name   string `json:"name"`
	Pass   bool   `json:"pass"`
	Detail string `json:"detail"`
}

CriterionResult is a single pass/fail check with a description.

type EnvironmentRecord 

type EnvironmentRecord struct {
	ModelName     string   `json:"modelName"`
	ModelBaseURL  string   `json:"modelBaseURL"`
	Temperature   string   `json:"temperature"`
	MaxTokens     int      `json:"maxTokens"`
	ContextWindow int      `json:"contextWindow"`
	Timestamp     string   `json:"timestamp"`
	OzyGitSHA     string   `json:"ozyGitSHA"`
	ScenarioHash  string   `json:"scenarioHash"`
	Modes         []string `json:"modes"`
	RunCount      int      `json:"runCount"`
}

EnvironmentRecord captures the model and runtime provenance for a benchmark run.

func BuildProvenance 

func BuildProvenance(cfg *ScenarioConfig, modes []string, runCount int) (*EnvironmentRecord, error)

BuildProvenance collects the model and runtime metadata for the run.

type FixtureMeta 

type FixtureMeta struct {
	CulpritHash    string
	CulpritSubject string
	TargetDir      string
}

FixtureMeta holds resolved metadata after generating the acme-billing fixture.

func GenerateFixture 

func GenerateFixture(targetDir string) (*FixtureMeta, error)

GenerateFixture creates the deterministic acme-billing fixture in targetDir. It writes Java source files, documentation, initializes a git repository, creates a multi-commit history, resolves the culprit commit hash, and returns FixtureMeta.

type GradingResult 

type GradingResult struct {
	Overall         bool              `json:"overall"`
	Criteria        []CriterionResult `json:"criteria"`
	ForbiddenChecks []CriterionResult `json:"forbidden_checks"`
}

GradingResult captures per-criterion pass/fail and overall verdict.

func Grade 

func Grade(gt *GroundTruth, finalAnswer string, toolCalls []ToolCallLog, culpritHash string) *GradingResult

Grade scores the final answer and tool-call log against ground truth. It returns a deterministic pass/fail — no model is used.

type GroundTruth 

type GroundTruth struct {
	RootCauseFile        string   `json:"root_cause_file"`
	RootCauseFunction    string   `json:"root_cause_function"`
	CulpritCommitSubject string   `json:"culprit_commit_subject"`
	ExpectedTest         string   `json:"expected_test"`
	ExpectedPatchFile    string   `json:"expected_patch_file"`
	ForbiddenBehaviors   []string `json:"forbidden_behaviors"`
}

GroundTruth defines the expected answer and forbidden behaviors for a scenario.

func LoadGroundTruth 

func LoadGroundTruth(path string) (*GroundTruth, error)

LoadGroundTruth reads ground_truth.json from path.

type LedgerItem 

type LedgerItem struct {
	RunID                  string `json:"run_id"`
	Mode                   string `json:"mode"`
	Phase                  string `json:"phase"`
	Source                 string `json:"source"`
	Kind                   string `json:"kind"`
	Server                 string `json:"server,omitempty"`
	Tool                   string `json:"tool,omitempty"`
	Bytes                  int    `json:"bytes"`
	TokenCount             int    `json:"token_count"`
	TokenSource            string `json:"token_source"`
	IncludedInModelContext bool   `json:"included_in_model_context"`
}

LedgerItem is a single entry in the JSONL context ledger.

type LedgerWriter 

type LedgerWriter struct {
	// contains filtered or unexported fields
}

LedgerWriter appends LedgerItems to a JSONL file.

func NewLedgerWriter 

func NewLedgerWriter(path string) (*LedgerWriter, error)

NewLedgerWriter creates a ledger writer that appends JSON lines to path.

func (*LedgerWriter) Append 

func (w *LedgerWriter) Append(item *LedgerItem) error

Append writes one ledger item as a JSON line.

func (*LedgerWriter) Close 

func (w *LedgerWriter) Close() error

Close flushes and closes the underlying file.

type Orchestrator 

type Orchestrator struct {
	Scenario   *ScenarioConfig
	FixtureDir string
	RunDir     string
	NumRuns    int
	Mode       string // "direct", "ozy", or "both"
}

Orchestrator manages the full benchmark lifecycle.

func (*Orchestrator) Run 

func (o *Orchestrator) Run(ctx context.Context) error

Run executes the full benchmark.

type RequestTokens 

type RequestTokens struct {
	Index int `json:"index"`
	CategoryTokens
}

RequestTokens is one captured request's breakdown, ordered within the run.

type RunResult 

type RunResult struct {
	RunID       string         `json:"runId"`
	Mode        string         `json:"mode"`
	Success     bool           `json:"success"`
	TimedOut    bool           `json:"timedOut"`
	DurationSec float64        `json:"durationSec"`
	Transcript  string         `json:"-"`
	FinalAnswer string         `json:"finalAnswer"`
	ToolCalls   []ToolCallLog  `json:"toolCalls"`
	Grading     *GradingResult `json:"grading,omitempty"`
}

RunResult captures the output of a single agent run.

type Runner 

type Runner struct {
	OpenCodePath string
	ConfigPath   string
	WorkDir      string // volume-mounted dir for artifacts
	FixtureDir   string // OpenCode's working directory
	Timeout      time.Duration
	Spy          *ContextSpy // per-run ContextSpy session control (best-effort)
}

Runner launches an agent with a task prompt and captures output.

func NewRunner 

func NewRunner(configPath, workDir, fixtureDir string, timeout time.Duration) *Runner

NewRunner creates a runner for the given config.

func (*Runner) Run 

func (r *Runner) Run(ctx context.Context, mode, runID, taskPrompt string) (*RunResult, error)

Run launches OpenCode in non-interactive mode with the task prompt. It creates a project-level opencode.json and .opencode/mcp.json in the work directory, derived from environment variables — never touching user config.

type ScenarioConfig 

type ScenarioConfig struct {
	Name     string `json:"name"`
	TaskFile string `json:"taskFile"`
	Fixture  string `json:"fixture"`

	Model struct {
		NameEnv       string `json:"nameEnv"`
		BaseURLEnv    string `json:"baseURLEnv"`
		APIKeyEnv     string `json:"apiKeyEnv"`
		Temperature   string `json:"temperature"`
		MaxTokens     int    `json:"maxTokens"`
		ContextWindow int    `json:"contextWindow"`
	} `json:"model"`

	AgentConfigs struct {
		Direct string `json:"direct"`
		Ozy    string `json:"ozy"`
	} `json:"agentConfigs"`

	Limits struct {
		Runs           int `json:"runs"`
		TimeoutSeconds int `json:"timeoutSeconds"`
	} `json:"limits"`

	GroundTruth string `json:"groundTruth"`

	ForbiddenTools     []string `json:"forbiddenTools"`
	ForbiddenBehaviors []string `json:"forbiddenBehaviors"`

	// BaseDir is the directory containing the scenario config file.
	// All relative paths (TaskFile, Fixture, GroundTruth) are resolved
	// relative to this directory.
	BaseDir string `json:"-"`
}

ScenarioConfig is the typed JSONC configuration for a benchmark scenario.

func LoadScenario 

func LoadScenario(path string) (*ScenarioConfig, error)

LoadScenario reads, parses, and resolves a JSONC scenario configuration from path. It substitutes {env:VAR} references using os.LookupEnv.

func (*ScenarioConfig) ResolvePath 

func (cfg *ScenarioConfig) ResolvePath(rel string) string

ResolvePath resolves a relative path against the scenario's base directory.

func (*ScenarioConfig) ScenarioHash 

func (cfg *ScenarioConfig) ScenarioHash() (string, error)

ScenarioHash computes a deterministic SHA-256 hash of the scenario's definition. The hash incorporates the task prompt content so it changes when the scenario definition or prompt changes.

type SurfaceComparison 

type SurfaceComparison struct {
	Direct    SurfaceMetrics `json:"direct"`
	Ozy       SurfaceMetrics `json:"ozy"`
	Reduction struct {
		ToolCount    int     `json:"toolCount"`
		SchemaTokens int     `json:"schemaTokens"`
		Ratio        float64 `json:"ratio"`
	} `json:"reduction"`
}

SurfaceComparison compares the startup surfaces of direct and ozy modes.

func CompareSurfaces 

func CompareSurfaces(direct, ozy *SurfaceMetrics) *SurfaceComparison

CompareSurfaces computes the reduction ratio between direct and ozy surfaces.

type SurfaceMetrics 

type SurfaceMetrics struct {
	Mode                   string        `json:"mode"`
	ToolsVisible           int           `json:"toolsVisible"`
	SchemaBytes            int           `json:"schemaBytes"`
	SchemaTokens           int           `json:"schemaTokens"`
	IrrelevantSchemaTokens int           `json:"irrelevantSchemaTokens"`
	Tools                  []ToolSurface `json:"tools"`
}

SurfaceMetrics captures the startup tool surface for one mode.

func MeasureSurface 

func MeasureSurface(mode string, tools []ToolSurface, estimator eval.TokenEstimator) *SurfaceMetrics

MeasureSurface enumerates the tools advertised at startup for the given mode. It takes the list of tool surfaces discovered during MCP initialization.

type ToolCallLog 

type ToolCallLog struct {
	Tool   string `json:"tool"`
	Server string `json:"server,omitempty"`
}

ToolCallLog records a tool invocation during the agent run.

type ToolSurface 

type ToolSurface struct {
	Name         string `json:"name"`
	Server       string `json:"server"`
	Description  string `json:"description"`
	SchemaBytes  int    `json:"schemaBytes"`
	SchemaTokens int    `json:"schemaTokens"`
}

ToolSurface describes a single advertised tool at startup.

type ToolTokens 

type ToolTokens struct {
	Tool             string `json:"tool"`
	DefinitionTokens int    `json:"definitionTokens"`
	ResultTokens     int    `json:"resultTokens"`
	Occurrences      int    `json:"occurrences,omitempty"`
}

ToolTokens is a per-tool schema/result token total across the run, summed over every request that advertised or called the tool. Occurrences is not exposed by the HTTP API (it was a SQL COUNT) and stays zero.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.