benchmark

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 1, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package benchmark executes and reports website-planned DeepSWE campaigns.

Index

Constants

View Source 
const (
	DeepSWECommit          = "e016041a6ccf8da29906afc9a3f5a8df940a1f78"
	PierVersion            = "0.3.0"
	CodexClientVersion     = "0.144.6"
	ClaudeClientVersion    = "2.1.207"
	ReportSchemaVersion    = "benchmark-report-v4"
	StorageSchemaVersion   = "benchmark-store-v1"
	TreatmentSchemaVersion = "benchmark-treatment-v1"
	DeepSWETrialsSourceURL = "https://deepswe.datacurve.ai/artifacts/v1.1/trials.json"
)

Pinned benchmark inputs and artifact schema versions.

View Source 
const (
	TreatmentInstructionsOnly      = "instructions-only"
	TreatmentInstructionsAndSkills = "instructions-and-skills"
)

Treatment modes define the files injected into the provider workspace.

View Source 
const (
	// ArmBaseline is a native provider execution without Agent Layer.
	ArmBaseline = "baseline"
	// ArmTreatment is an execution with an immutable Agent Layer bundle.
	ArmTreatment = "treatment"
)

Variables

View Source 
var ErrConfirmationRequired = errors.New("benchmark paid execution requires confirmation")

ErrConfirmationRequired is returned before any paid model invocation.

Functions

func NewEventID 

func NewEventID() (string, error)

NewEventID returns a random, path-safe execution event identity.

func RenderHTML 

func RenderHTML(report Report) ([]byte, error)

RenderHTML produces an offline document from already-derived report data.

func RenderJSON 

func RenderJSON(report Report) ([]byte, error)

RenderJSON renders canonical report JSON before the presentation document.

func TaskTreeChecksum 

func TaskTreeChecksum(root string) (string, error)

TaskTreeChecksum returns Pier 0.3.0's deterministic task-directory identity.

Types

type AttemptResult 

type AttemptResult struct {
	SchemaVersion         string    `json:"schema_version"`
	EventID               string    `json:"event_id"`
	Attempt               int       `json:"attempt"`
	Task                  string    `json:"task"`
	Status                string    `json:"status"`
	Error                 string    `json:"error,omitempty"`
	F2PPassed             int       `json:"f2p_passed"`
	F2PTotal              int       `json:"f2p_total"`
	F2PScore              float64   `json:"f2p_score"`
	PartialScore          float64   `json:"partial_score"`
	Reward                float64   `json:"reward"`
	CostUSD               *float64  `json:"cost_usd,omitempty"`
	CostMinUSD            *float64  `json:"cost_min_usd,omitempty"`
	CostMaxUSD            *float64  `json:"cost_max_usd,omitempty"`
	CostKind              string    `json:"cost_kind"`
	DurationSeconds       *float64  `json:"duration_seconds,omitempty"`
	TaskChecksum          string    `json:"task_checksum"`
	StartedAt             time.Time `json:"started_at"`
	FinishedAt            time.Time `json:"finished_at"`
	Provider              string    `json:"provider"`
	PublishedModel        string    `json:"published_model"`
	RuntimeModel          string    `json:"runtime_model"`
	ReasoningEffort       string    `json:"reasoning_effort"`
	ProviderClientVersion string    `json:"provider_client_version"`
	DispatchConformant    bool      `json:"dispatch_conformant"`
	PatchBytes            int64     `json:"patch_bytes"`
	VerifierBuildFailed   bool      `json:"verifier_build_failed"`
	BuildErrorExcerpt     string    `json:"build_error_excerpt,omitempty"`
	CoordinatorCostUSD    *float64  `json:"coordinator_cost_usd,omitempty"`
	CoordinatorCostMinUSD *float64  `json:"coordinator_cost_min_usd,omitempty"`
	CoordinatorCostMaxUSD *float64  `json:"coordinator_cost_max_usd,omitempty"`
	ChildCostUSD          *float64  `json:"child_cost_usd,omitempty"`
	ChildCostMinUSD       *float64  `json:"child_cost_min_usd,omitempty"`
	ChildCostMaxUSD       *float64  `json:"child_cost_max_usd,omitempty"`
	InvocationCount       int       `json:"invocation_count"`
}

AttemptResult is the normalized immutable record used by campaign analysis. A score of zero is successful evidence; Status is the success indicator.

func (AttemptResult) CostBounds 

func (result AttemptResult) CostBounds() (float64, float64, error)

CostBounds returns exact cost twice or the explicitly recorded range.

func (AttemptResult) Validate 

func (result AttemptResult) Validate() error

Validate prevents malformed or incomplete evidence from entering analysis.

type BaselineOptions 

type BaselineOptions struct {
	RepoRoot        string
	PlanPath        string
	PlanJSON        []byte
	TaskConcurrency int
	Confirmed       bool
}

BaselineOptions configures a bare-model run from a website-exported plan.

type BaselineOutcome 

type BaselineOutcome struct {
	PlanID       string
	StateDir     string
	EstimatedUSD float64
	ActualUSD    float64
	Completed    int
	Required     int
	Summary      *BaselineSummary
}

BaselineOutcome reports the immutable plan identity and current baseline.

func CheckBaseline 

func CheckBaseline(ctx context.Context, options BaselineOptions) (BaselineOutcome, error)

CheckBaseline validates the exported plan and every local execution prerequisite without making a provider call.

func RunBaseline 

func RunBaseline(ctx context.Context, options BaselineOptions, executor TaskExecutor) (BaselineOutcome, error)

RunBaseline executes or reuses the bare-model repetitions specified by the exported plan. It never runs an Agent Layer treatment.

type BaselineSummary 

type BaselineSummary struct {
	SchemaVersion          string                `json:"schema_version"`
	PlanID                 string                `json:"plan_id"`
	Model                  string                `json:"model"`
	Reasoning              string                `json:"reasoning"`
	PublishedHarnesses     []string              `json:"published_harnesses"`
	LocalHarness           string                `json:"local_harness"`
	PublishedComparable    bool                  `json:"published_comparable"`
	PublishedMean          float64               `json:"published_mean"`
	FreshBaselineMean      float64               `json:"fresh_baseline_mean"`
	FreshMinusPublished    float64               `json:"fresh_minus_published"`
	DecisionThreshold      float64               `json:"decision_threshold"`
	ActualBaselineCostUSD  ObservedCostRange     `json:"actual_baseline_cost_usd"`
	EstimatedBaselineSpend float64               `json:"estimated_baseline_spend_usd"`
	CompletedAt            time.Time             `json:"completed_at"`
	Tasks                  []BaselineTaskSummary `json:"tasks"`
	Limitations            []string              `json:"limitations"`
}

BaselineSummary compares the fresh equal-task baseline with the published target evidence carried by the planner export.

type BaselineTaskSummary 

type BaselineTaskSummary struct {
	Task          string            `json:"task"`
	Repetitions   int               `json:"repetitions"`
	PublishedMean float64           `json:"published_mean"`
	FreshMean     float64           `json:"fresh_mean"`
	Difference    float64           `json:"difference"`
	CostUSD       ObservedCostRange `json:"cost_usd"`
}

BaselineTaskSummary preserves each task's planned repetition count and observed equal-repetition baseline mean.

type CampaignReportOptions 

type CampaignReportOptions struct {
	RepoRoot            string
	PlanPath            string
	PlanJSON            []byte
	LegacyAnalysisPaths []string
}

CampaignReportOptions selects a plan and, only for plans exported before cost-axis provenance existed, explicit canonical analysis documents.

type CampaignReportOutcome 

type CampaignReportOutcome struct {
	PlanID            string
	Versions          int
	Report            Report
	JSONPath          string
	HTMLPath          string
	Analyses          []string
	SkippedTreatments []string
}

CampaignReportOutcome identifies the generated canonical report artifacts.

func BuildCampaignReport 

func BuildCampaignReport(options CampaignReportOptions) (CampaignReportOutcome, error)

BuildCampaignReport derives statistics and presentation artifacts from immutable arm evidence. It never makes provider calls.

type ExecutionRequest 

type ExecutionRequest struct {
	RepoRoot     string
	EvidenceDir  string
	EventID      string
	Attempt      int
	Task         string
	Model        Model
	Effort       string
	Arm          string
	Bundle       *TreatmentBundle
	TaskChecksum string
}

ExecutionRequest identifies one plan-selected repetition and its evidence destination.

type Model 

type Model struct {
	Name                  string `json:"name"`
	PublishedIdentifier   string `json:"published_identifier"`
	RuntimeIdentifier     string `json:"runtime_identifier"`
	Adapter               string `json:"adapter"`
	ProviderClientVersion string `json:"provider_client_version"`
}

Model defines a published model family and its native Pier adapter.

func ParseModelSelection 

func ParseModelSelection(value string) (Model, string, error)

ParseModelSelection validates the stable family:effort identity.

type ObservedBaselineTaskReport 

type ObservedBaselineTaskReport struct {
	Task                    string    `json:"task"`
	Repetitions             int       `json:"repetitions"`
	Scores                  []float64 `json:"scores"`
	Mean                    float64   `json:"mean"`
	SampleVariance          float64   `json:"sample_variance"`
	VerifierBuildFailedRuns int       `json:"verifier_build_failed_runs"`
}

ObservedBaselineTaskReport is the shared bare-model evidence for one task.

type ObservedCampaignReport 

type ObservedCampaignReport struct {
	PlanID                string                       `json:"plan_id"`
	Model                 string                       `json:"model"`
	Reasoning             string                       `json:"reasoning"`
	BaselineLabel         string                       `json:"baseline_label"`
	TaskCount             int                          `json:"task_count"`
	RunsPerArm            int                          `json:"runs_per_arm"`
	SignificanceLevel     float64                      `json:"two_sided_significance_level"`
	EqualTaskWeighting    bool                         `json:"equal_task_weighting"`
	BaselineMean          float64                      `json:"baseline_mean"`
	BaselineStandardError float64                      `json:"baseline_standard_error"`
	BaselineCost          ObservedCostRange            `json:"baseline_cost"`
	CampaignCost          ObservedCostRange            `json:"campaign_cost"`
	CostAxis              ObservedCostAxis             `json:"cost_axis"`
	BaselineTasks         []ObservedBaselineTaskReport `json:"baseline_tasks"`
	Versions              []ObservedVersionReport      `json:"versions"`
	Warnings              []string                     `json:"warnings,omitempty"`
}

ObservedCampaignReport is the executive report model for a shared baseline and an ordered series of skills/instructions versions.

type ObservedCostAxis 

type ObservedCostAxis struct {
	Scale                        string  `json:"scale"`
	ReferenceConfiguration       string  `json:"reference_configuration"`
	ReferenceSnapshotSHA256      string  `json:"reference_snapshot_sha256"`
	ReferenceEstimatedArmCostUSD float64 `json:"reference_estimated_arm_cost_usd"`
	RoundingIncrementUSD         float64 `json:"rounding_increment_usd"`
	MaximumUSD                   float64 `json:"maximum_usd"`
}

ObservedCostAxis records the campaign-specific, reproducible cost domain.

type ObservedCostRange 

type ObservedCostRange struct {
	Midpoint float64 `json:"midpoint"`
	Minimum  float64 `json:"minimum"`
	Maximum  float64 `json:"maximum"`
}

ObservedCostRange is the bounded provider cost for one report scope.

type ObservedTaskReport 

type ObservedTaskReport struct {
	Task                             string    `json:"task"`
	RepetitionsPerArm                int       `json:"repetitionsPerArm"`
	BaselineScores                   []float64 `json:"baselineScores"`
	TreatmentScores                  []float64 `json:"treatmentScores"`
	BaselineMean                     float64   `json:"baselineMean"`
	TreatmentMean                    float64   `json:"treatmentMean"`
	Difference                       float64   `json:"difference"`
	BaselineSampleVariance           float64   `json:"baselineSampleVariance"`
	TreatmentSampleVariance          float64   `json:"treatmentSampleVariance"`
	BaselineVerifierBuildFailedRuns  int       `json:"baselineVerifierBuildFailedRuns"`
	TreatmentVerifierBuildFailedRuns int       `json:"treatmentVerifierBuildFailedRuns"`
}

ObservedTaskReport contains the repeated score evidence for one selected task in the source analysis artifact.

type ObservedVersionReport 

type ObservedVersionReport struct {
	GeneratedAt               time.Time                   `json:"generated_at"`
	Label                     string                      `json:"label"`
	Verdict                   string                      `json:"verdict"`
	Mean                      float64                     `json:"mean"`
	StandardError             float64                     `json:"standard_error"`
	ObservedDifference        float64                     `json:"observed_difference"`
	DecisionThreshold         float64                     `json:"decision_threshold"`
	DifferenceStandardError   float64                     `json:"difference_standard_error"`
	EffectiveDegreesOfFreedom float64                     `json:"effective_degrees_of_freedom"`
	TCriticalValue            float64                     `json:"t_critical_value"`
	Cost                      ObservedCostRange           `json:"cost"`
	CostMultiple              float64                     `json:"cost_multiple"`
	CostMultipleMinimum       float64                     `json:"cost_multiple_minimum"`
	CostMultipleMaximum       float64                     `json:"cost_multiple_maximum"`
	InvocationCount           int                         `json:"invocation_count"`
	DispatchConformantRuns    int                         `json:"dispatch_conformant_runs"`
	TotalRuns                 int                         `json:"total_runs"`
	AgentTimeoutMultiplier    float64                     `json:"agent_timeout_multiplier"`
	Tasks                     []ObservedVersionTaskReport `json:"tasks"`
	Limitations               []string                    `json:"limitations"`
}

ObservedVersionReport is one skills/instructions version compared with the campaign's shared bare-model baseline.

type ObservedVersionTaskReport 

type ObservedVersionTaskReport struct {
	Task                    string    `json:"task"`
	Scores                  []float64 `json:"scores"`
	Mean                    float64   `json:"mean"`
	Difference              float64   `json:"difference"`
	SampleVariance          float64   `json:"sample_variance"`
	VerifierBuildFailedRuns int       `json:"verifier_build_failed_runs"`
}

ObservedVersionTaskReport is one campaign version's evidence for one task.

type PierExecutor 

type PierExecutor struct{}

PierExecutor invokes the pinned official Pier adapter once.

func (PierExecutor) Execute 

func (PierExecutor) Execute(ctx context.Context, request ExecutionRequest) (AttemptResult, error)

Execute runs one task and promotes sanitized evidence before returning.

type Report 

type Report struct {
	SchemaVersion    string                  `json:"schema_version"`
	ComparisonID     string                  `json:"comparison_id"`
	GeneratedAt      time.Time               `json:"generated_at"`
	ObservedCampaign *ObservedCampaignReport `json:"observed_campaign"`
	Limitations      []string                `json:"limitations"`
}

Report is the canonical derived model shared by JSON and HTML rendering.

func BuildObservedCampaignReport 

func BuildObservedCampaignReport(documents ...[]byte) (Report, error)

BuildObservedCampaignReport validates ordered observed-arm analyses sharing one baseline and converts them into the canonical campaign report.

type TaskExecutor 

type TaskExecutor interface {
	Execute(context.Context, ExecutionRequest) (AttemptResult, error)
}

TaskExecutor is the testable boundary around one paid task execution.

type TreatmentBundle 

type TreatmentBundle struct {
	Root              string            `json:"root"`
	Manifest          TreatmentManifest `json:"manifest"`
	ManifestHash      string            `json:"manifest_hash"`
	LinuxBinary       string            `json:"linux_binary"`
	LinuxBinarySHA256 string            `json:"linux_binary_sha256"`
	AdapterPath       string            `json:"adapter_path"`
	AdapterSHA256     string            `json:"adapter_sha256"`
	TemplatesCommit   string            `json:"templates_commit,omitempty"`
	TemplatesDirty    bool              `json:"templates_dirty"`
}

TreatmentBundle is the secret-free, immutable effective Agent Layer input.

func BuildTreatmentBundle 

func BuildTreatmentBundle(repoRoot, targetArch, mode string, model Model, effort string) (*TreatmentBundle, error)

BuildTreatmentBundle stages the canonical shipped templates, synchronizes that isolated stage, and includes only its provider-effective projections. Repository-local .agent-layer customizations are never treatment inputs. The caller owns cleanup after the task adapter has consumed the bundle.

type TreatmentFile 

type TreatmentFile struct {
	Path   string `json:"path"`
	SHA256 string `json:"sha256"`
}

TreatmentFile is the content-addressed declaration for one injected file.

type TreatmentManifest 

type TreatmentManifest struct {
	SchemaVersion          string          `json:"schema_version"`
	Mode                   string          `json:"mode"`
	AgentTimeoutMultiplier float64         `json:"agent_timeout_multiplier"`
	Files                  []TreatmentFile `json:"files"`
	RequiredRoles          []string        `json:"required_dispatch_roles"`
}

TreatmentManifest names only files that were actually injected. It cannot contain .env, project memory, runtime state, temporary data, or credentials.

type TreatmentOptions 

type TreatmentOptions struct {
	RepoRoot        string
	PlanPath        string
	PlanJSON        []byte
	Label           string
	TaskConcurrency int
	Confirmed       bool
}

TreatmentOptions configures one immutable skills-and-instructions campaign version for a website-exported plan.

type TreatmentOutcome 

type TreatmentOutcome struct {
	PlanID       string
	TreatmentID  string
	Label        string
	StateDir     string
	Completed    int
	Required     int
	Missing      int
	ProviderCall bool
}

TreatmentOutcome describes the selected immutable treatment and its cached execution progress.

func CheckTreatment 

func CheckTreatment(ctx context.Context, options TreatmentOptions) (TreatmentOutcome, error)

CheckTreatment validates the plan, baseline, and current treatment bundle without making a provider call.

func RunTreatment 

func RunTreatment(ctx context.Context, options TreatmentOptions, executor TaskExecutor) (TreatmentOutcome, error)

RunTreatment executes only the missing repetitions for the current immutable Agent Layer bundle. Existing evidence is never overwritten.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.