benchmark

package
v0.17.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2026 License: MIT Imports: 41 Imported by: 0

Documentation

Overview

Package benchmark executes and reports content-addressed DeepSWE studies.

Index

Constants

View Source 
const (
	DeepSWECommit          = "e016041a6ccf8da29906afc9a3f5a8df940a1f78"
	PierVersion            = "0.3.0"
	CodexClientVersion     = "0.146.0"
	ClaudeClientVersion    = "2.1.207"
	ReportSchemaVersion    = "benchmark-report-v5"
	StorageSchemaVersion   = "benchmark-store-v1"
	TreatmentSchemaVersion = "benchmark-treatment-v2"
	DeepSWETrialsSourceURL = "https://deepswe.datacurve.ai/artifacts/v1.1/trials.json"
)

Pinned benchmark inputs and artifact schema versions.

View Source 
const (
	TreatmentInstructionsOnly      = "instructions-only"
	TreatmentInstructionsAndSkills = "instructions-and-skills"
)

Treatment modes define the files injected into the provider workspace.

View Source 
const (
	// ArmBaseline is a native provider execution without Agent Layer.
	ArmBaseline = "baseline"
	// ArmTreatment is an execution with an immutable Agent Layer bundle.
	ArmTreatment = "treatment"
)

Variables

This section is empty.

Functions

func NewEventID 

func NewEventID() (string, error)

NewEventID returns a random, path-safe execution event identity.

func TaskTreeChecksum 

func TaskTreeChecksum(root string) (string, error)

TaskTreeChecksum returns Pier 0.3.0's deterministic task-directory identity.

Types

type AttemptResult 

type AttemptResult struct {
	SchemaVersion         string    `json:"schema_version"`
	EventID               string    `json:"event_id"`
	Attempt               int       `json:"attempt"`
	Task                  string    `json:"task"`
	Status                string    `json:"status"`
	Error                 string    `json:"error,omitempty"`
	F2PPassed             int       `json:"f2p_passed"`
	F2PTotal              int       `json:"f2p_total"`
	F2PScore              float64   `json:"f2p_score"`
	PartialScore          float64   `json:"partial_score"`
	Reward                float64   `json:"reward"`
	CostUSD               *float64  `json:"cost_usd,omitempty"`
	CostMinUSD            *float64  `json:"cost_min_usd,omitempty"`
	CostMaxUSD            *float64  `json:"cost_max_usd,omitempty"`
	CostKind              string    `json:"cost_kind"`
	DurationSeconds       *float64  `json:"duration_seconds,omitempty"`
	TaskChecksum          string    `json:"task_checksum"`
	EnvironmentIdentity   string    `json:"task_environment_identity,omitempty"`
	StartedAt             time.Time `json:"started_at"`
	FinishedAt            time.Time `json:"finished_at"`
	Provider              string    `json:"provider"`
	PublishedModel        string    `json:"published_model"`
	RuntimeModel          string    `json:"runtime_model"`
	ReasoningEffort       string    `json:"reasoning_effort"`
	ProviderClientVersion string    `json:"provider_client_version"`
	DispatchConformant    bool      `json:"dispatch_conformant"`
	PatchBytes            int64     `json:"patch_bytes"`
	VerifierBuildFailed   bool      `json:"verifier_build_failed"`
	BuildErrorExcerpt     string    `json:"build_error_excerpt,omitempty"`
	CoordinatorCostUSD    *float64  `json:"coordinator_cost_usd,omitempty"`
	CoordinatorCostMinUSD *float64  `json:"coordinator_cost_min_usd,omitempty"`
	CoordinatorCostMaxUSD *float64  `json:"coordinator_cost_max_usd,omitempty"`
	ChildCostUSD          *float64  `json:"child_cost_usd,omitempty"`
	ChildCostMinUSD       *float64  `json:"child_cost_min_usd,omitempty"`
	ChildCostMaxUSD       *float64  `json:"child_cost_max_usd,omitempty"`
	InvocationCount       int       `json:"invocation_count"`
	InvocationWorkers     int       `json:"invocation_workers,omitempty"`
}

AttemptResult is the normalized immutable record used by campaign analysis. A score of zero is successful evidence; Status is the success indicator.

func (AttemptResult) CostBounds 

func (result AttemptResult) CostBounds() (float64, float64, error)

CostBounds returns exact cost twice or the explicitly recorded range.

func (AttemptResult) Validate 

func (result AttemptResult) Validate() error

Validate prevents malformed or incomplete evidence from entering analysis.

type AuthenticationPreflight added in v0.17.1 

type AuthenticationPreflight struct {
	Provider             string    `json:"provider"`
	Check                string    `json:"check"`
	AuthenticationMethod string    `json:"authentication_method"`
	VerifiedAt           time.Time `json:"verified_at"`
}

AuthenticationPreflight is invocation provenance for a successful provider credential check. It is not part of study, arm, or treatment identity.

type ExecutionRequest 

type ExecutionRequest struct {
	RepoRoot               string
	EvidenceDir            string
	EventID                string
	Attempt                int
	Task                   string
	Model                  Model
	Effort                 string
	Arm                    string
	Bundle                 *TreatmentBundle
	TaskChecksum           string
	EnvironmentIdentity    string
	AgentTimeoutMultiplier float64
	PreflightOnly          bool
	// ResumeFailedInfrastructure is set only by the study scheduler for a new
	// user-authorized benchmark invocation. It permits a fresh event after an
	// earlier immutable infrastructure-failure receipt; it never retries a cell
	// within the failed invocation.
	ResumeFailedInfrastructure bool
	// contains filtered or unexported fields
}

ExecutionRequest identifies one plan-selected repetition and its evidence destination.

type Model 

type Model struct {
	Name                  string `json:"name"`
	PublishedIdentifier   string `json:"published_identifier"`
	RuntimeIdentifier     string `json:"runtime_identifier"`
	Adapter               string `json:"adapter"`
	ProviderClientVersion string `json:"provider_client_version"`
}

Model defines a published model family and its native Pier adapter.

func ParseModelSelection 

func ParseModelSelection(value string) (Model, string, error)

ParseModelSelection validates the stable family:effort identity.

type ObservedCostRange 

type ObservedCostRange struct {
	Midpoint float64 `json:"midpoint"`
	Minimum  float64 `json:"minimum"`
	Maximum  float64 `json:"maximum"`
}

ObservedCostRange is the bounded provider cost for one study scope.

type PierExecutor 

type PierExecutor struct{}

PierExecutor invokes the pinned official Pier adapter once.

func (PierExecutor) Execute 

func (PierExecutor) Execute(ctx context.Context, request ExecutionRequest) (AttemptResult, error)

Execute runs one task and promotes sanitized evidence before returning.

func (PierExecutor) Preflight added in v0.17.0 

func (PierExecutor) Preflight(ctx context.Context, request ExecutionRequest) error

Preflight executes the real Pier container and treatment setup without invoking the provider model.

type ReadinessAuditOptions added in v0.17.0 

type ReadinessAuditOptions struct {
	RepoRoot        string
	TaskConcurrency int
}

ReadinessAuditOptions configures the non-paid audit of every task in the pinned DeepSWE checkout.

type ReadinessAuditOutcome added in v0.17.0 

type ReadinessAuditOutcome struct {
	Schema        string               `json:"schema"`
	DeepSWECommit string               `json:"deep_swe_commit"`
	Required      int                  `json:"required"`
	Validated     int                  `json:"validated"`
	Certified     int                  `json:"certified"`
	Failed        int                  `json:"failed"`
	Blocked       int                  `json:"blocked"`
	Tasks         []ReadinessAuditTask `json:"tasks"`
}

ReadinessAuditOutcome summarizes the complete task-catalog preflight.

func CheckAllTaskReadiness added in v0.17.0 

func CheckAllTaskReadiness(ctx context.Context, options ReadinessAuditOptions) (ReadinessAuditOutcome, error)

CheckAllTaskReadiness validates and certifies every task in the pinned DeepSWE checkout without invoking a provider model or Pier.

type ReadinessAuditTask added in v0.17.0 

type ReadinessAuditTask struct {
	Task   string `json:"task"`
	Status string `json:"status"`
	Error  string `json:"error,omitempty"`
}

ReadinessAuditTask records the preflight result for one DeepSWE task.

type StudyComparisonReport added in v0.17.0 

type StudyComparisonReport struct {
	Left               string   `json:"left"`
	Right              string   `json:"right"`
	Available          bool     `json:"available"`
	UnavailableReason  string   `json:"unavailable_reason,omitempty"`
	Difference         *float64 `json:"difference,omitempty"`
	Variance           *float64 `json:"variance,omitempty"`
	StandardError      *float64 `json:"standard_error,omitempty"`
	DegreesOfFreedom   *float64 `json:"degrees_of_freedom,omitempty"`
	Statistic          *float64 `json:"statistic,omitempty"`
	RawTwoSidedPValue  *float64 `json:"raw_two_sided_p_value,omitempty"`
	HolmAdjustedPValue *float64 `json:"holm_adjusted_p_value,omitempty"`
}

StudyComparisonReport records the fixed-selection comparison of two experiments.

type StudyExecutionProvenance added in v0.17.0 

type StudyExecutionProvenance struct {
	Command                   string `json:"reproduction_command"`
	CLI                       string `json:"cli"`
	Harness                   string `json:"harness"`
	HarnessVersion            string `json:"harness_version"`
	TaskConcurrency           int    `json:"task_concurrency"`
	TaskContainerArchitecture string `json:"task_container_architecture"`
}

StudyExecutionProvenance records the public runner and pinned evaluation harness.

type StudyExperimentProgress added in v0.17.0 

type StudyExperimentProgress struct {
	Name      string
	Identity  string
	Completed int
	Required  int
	Missing   int
}

StudyExperimentProgress reports cached and missing cells for one experiment.

type StudyExperimentReport added in v0.17.0 

type StudyExperimentReport struct {
	Name                      string                   `json:"name"`
	Identity                  string                   `json:"identity"`
	Model                     string                   `json:"model"`
	Reasoning                 string                   `json:"reasoning"`
	InputHashes               map[string]string        `json:"immutable_inputs"`
	ResourceContract          map[string]any           `json:"resource_contract"`
	ProviderClients           []string                 `json:"provider_client_versions"`
	AuthenticationPreflight   *AuthenticationPreflight `json:"authentication_preflight,omitempty"`
	WorkerCountObserved       int                      `json:"observed_worker_count"`
	CompletedCells            int                      `json:"completed_cells"`
	RequiredCells             int                      `json:"required_cells"`
	Score                     *float64                 `json:"calibrated_score,omitempty"`
	ObservedCost              ObservedCostRange        `json:"observed_cost"`
	InvocationCount           int                      `json:"invocation_count"`
	DispatchConformantRuns    int                      `json:"dispatch_conformant_runs"`
	WorkflowNoncomplianceRuns int                      `json:"workflow_noncompliance_runs"`
	Tasks                     []StudyTaskReport        `json:"tasks"`
	BundleManifest            *TreatmentManifest       `json:"immutable_bundle_manifest,omitempty"`
	LinuxBinarySHA256         string                   `json:"linux_binary_sha256,omitempty"`
	AdapterSHA256             string                   `json:"adapter_sha256,omitempty"`
	SourceCommit              string                   `json:"source_commit,omitempty"`
	SourceDirty               bool                     `json:"source_dirty,omitempty"`
	ComparabilityWarnings     []string                 `json:"comparability_warnings"`
}

StudyExperimentReport records one content-addressed experiment and its cells.

type StudyHolmFamily added in v0.17.0 

type StudyHolmFamily struct {
	Method  string   `json:"method"`
	Size    int      `json:"size"`
	Members []string `json:"members"`
}

StudyHolmFamily describes the multiple-comparison adjustment applied to a study.

type StudyOptions added in v0.17.0 

type StudyOptions struct {
	RepoRoot        string
	StudyPath       string
	TaskConcurrency int
	Tasks           []string
	DryRun          bool
	// OnPrepared receives validated cached/missing progress after the full
	// runtime preflight and before any inference call.
	OnPrepared func(StudyOutcome) error
	// OnCellComplete is a serial CLI-facing progress hook. Library code never
	// writes stdout, and completed cells remain observable even if a later cell
	// fails.
	OnCellComplete func(ObservedCostRange)
}

StudyOptions configures the single public DeepSWE benchmark workflow. TaskConcurrency and Tasks scope one invocation only; neither affects identity.

type StudyOutcome added in v0.17.0 

type StudyOutcome struct {
	StudyID                  string
	SelectionID              string
	Required                 int
	Completed                int
	Missing                  int
	HasBareExperiment        bool
	BarePublishedEstimateUSD *float64
	ObservedInvocationCost   ObservedCostRange
	JSONPath                 string
	HTMLPath                 string
	Experiments              []StudyExperimentProgress
}

StudyOutcome is the user-facing progress summary for a study invocation.

func RunStudy added in v0.17.0 

func RunStudy(ctx context.Context, options StudyOptions, executor TaskExecutor) (StudyOutcome, error)

RunStudy is intentionally the only paid public entry point. The command itself is authorization; callers must use DryRun when they need the no-inference path.

type StudyReport added in v0.17.0 

type StudyReport struct {
	SchemaVersion string                   `json:"schema_version"`
	StudyID       string                   `json:"study_id"`
	SelectionID   string                   `json:"selection_id"`
	GeneratedAt   time.Time                `json:"generated_at"`
	Execution     StudyExecutionProvenance `json:"execution"`
	Selection     StudySelectionProvenance `json:"selection"`
	Experiments   []StudyExperimentReport  `json:"experiments"`
	Comparisons   []StudyComparisonReport  `json:"comparisons"`
	HolmFamily    StudyHolmFamily          `json:"holm_family"`
	Limitations   []string                 `json:"limitations"`
}

StudyReport is deliberately independent from MatrixReport. A study is the public reproducibility boundary: it records exactly its declared experiments and every task in its fixed selection, including cells that are still missing.

type StudySelectionProvenance added in v0.17.0 

type StudySelectionProvenance struct {
	Model         string            `json:"model"`
	Reasoning     string            `json:"reasoning"`
	SnapshotURL   string            `json:"snapshot_url"`
	SnapshotSHA   string            `json:"snapshot_sha256"`
	TaskChecksums map[string]string `json:"task_checksums"`
	Environments  map[string]string `json:"task_environment_identities"`
}

StudySelectionProvenance records the immutable selection inputs used by a study.

type StudyTaskReport added in v0.17.0 

type StudyTaskReport struct {
	Task                 string            `json:"task"`
	RepetitionsRequired  int               `json:"repetitions_required"`
	RepetitionsCompleted int               `json:"repetitions_completed"`
	F2PMean              *float64          `json:"f2p_mean,omitempty"`
	SampleVariance       *float64          `json:"sample_variance,omitempty"`
	CalibrationIntercept float64           `json:"calibration_intercept"`
	CalibrationSlope     float64           `json:"calibration_slope"`
	Weight               float64           `json:"weight"`
	EffectiveCoefficient float64           `json:"effective_coefficient"`
	CalibratedMean       *float64          `json:"calibrated_mean,omitempty"`
	WeightedContribution *float64          `json:"weighted_contribution,omitempty"`
	ObservedCost         ObservedCostRange `json:"observed_cost"`
	MissingAttempts      []int             `json:"missing_attempts,omitempty"`
}

StudyTaskReport records one selected task's repetitions and score evidence.

type TaskExecutor 

type TaskExecutor interface {
	Execute(context.Context, ExecutionRequest) (AttemptResult, error)
}

TaskExecutor is the testable boundary around one paid task execution.

type TreatmentBundle 

type TreatmentBundle struct {
	Root              string            `json:"root"`
	Manifest          TreatmentManifest `json:"manifest"`
	ManifestHash      string            `json:"manifest_hash"`
	LinuxArchitecture string            `json:"linux_architecture"`
	LinuxBinary       string            `json:"linux_binary"`
	LinuxBinarySHA256 string            `json:"linux_binary_sha256"`
	AdapterPath       string            `json:"adapter_path"`
	AdapterSHA256     string            `json:"adapter_sha256"`
	TemplatesCommit   string            `json:"templates_commit,omitempty"`
	TemplatesDirty    bool              `json:"templates_dirty"`
	// CredentialNames names the *only* host values that may cross the task
	// boundary. Values are never part of a bundle, manifest, or pin.
	CredentialNames   []string `json:"credential_names,omitempty"`
	RuntimeSourceKind string   `json:"runtime_source_kind"`
	RuntimeVersion    string   `json:"runtime_version,omitempty"`
}

TreatmentBundle is the secret-free, immutable effective Agent Layer input.

func BuildStudyTreatmentBundle added in v0.17.0 

func BuildStudyTreatmentBundle(repoRoot string, experiment preparedStudyExperiment) (*TreatmentBundle, error)

BuildStudyTreatmentBundle stages the immutable input snapshot prepared for an experiment. A distributed study is reproducible from those declared bytes and the runner assets embedded in this executable alone. Its runtime target is the certified task container, not the machine preparing the study.

type TreatmentDispatchConfig added in v0.17.0 

type TreatmentDispatchConfig struct {
	Schema        string                    `toml:"schema" json:"schema"`
	PlanReviewers []TreatmentDispatchTarget `toml:"plan_reviewers" json:"plan_reviewers"`
	Implementer   TreatmentDispatchTarget   `toml:"implementer" json:"implementer"`
	CodeReviewer  TreatmentDispatchTarget   `toml:"code_reviewer" json:"code_reviewer"`
}

TreatmentDispatchConfig assigns exact execution targets to workflow roles.

type TreatmentDispatchTarget added in v0.17.0 

type TreatmentDispatchTarget struct {
	Agent           string `toml:"agent" json:"agent"`
	Model           string `toml:"model" json:"model"`
	ReasoningEffort string `toml:"reasoning_effort" json:"reasoning_effort"`
}

TreatmentDispatchTarget is one exact Agent Dispatch execution identity.

type TreatmentFile 

type TreatmentFile struct {
	Path   string `json:"path"`
	SHA256 string `json:"sha256"`
}

TreatmentFile is the content-addressed declaration for one injected file.

type TreatmentManifest 

type TreatmentManifest struct {
	SchemaVersion          string                  `json:"schema_version"`
	Mode                   string                  `json:"mode"`
	AgentTimeoutMultiplier float64                 `json:"agent_timeout_multiplier"`
	Files                  []TreatmentFile         `json:"files"`
	RequiredRoles          []string                `json:"required_dispatch_roles"`
	DispatchConfig         TreatmentDispatchConfig `json:"dispatch_config,omitempty"`
}

TreatmentManifest names only files that were actually injected. It cannot contain .env, project memory, runtime state, temporary data, or credentials.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.