pilot

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source 
const Schema = "agentic.eval.agent/v1alpha1"

Schema identifies the pilot record contract.

Variables

This section is empty.

Functions

func BuildCommand 

func BuildCommand(s CommandSpec) (*exec.Cmd, error)

BuildCommand constructs the isolated Codex command.

func DigestString 

func DigestString(value string) string

DigestString returns the SHA-256 digest of value.

func FocusToolCalls 

func FocusToolCalls(events Events) int

FocusToolCalls returns successful focus-call count for terminal events. The legacy fallback preserves support for older bare tool-call event records.

func PreflightFocus 

func PreflightFocus(ctx context.Context, binary, workspace string) error

PreflightFocus proves that the exact server configured for a focus run can initialize and advertises the focus tool before an agent is charged a run.

func SanitizeText 

func SanitizeText(value string) string

SanitizeText removes private filesystem roots from evidence.

func SanitizeTranscript 

func SanitizeTranscript(in []json.RawMessage) []json.RawMessage

SanitizeTranscript sanitizes each transcript event.

func SkillDiscoveryEvidence 

func SkillDiscoveryEvidence(events Events, skillName string) bool

SkillDiscoveryEvidence reports whether the transcript contains a concrete read or load of the named skill's SKILL.md. Generic prose or a tool call is not sufficient evidence of model discovery.

func ValidateMatrix 

func ValidateMatrix(scenarios []Scenario, runs []Run) error

ValidateMatrix enforces the complete paired pilot design before aggregation.

func WorkspaceDigest 

func WorkspaceDigest(root string) (string, error)

WorkspaceDigest hashes the sorted relative paths and contents, excluding Git metadata and private pilot output. It is stable across checkout roots.

Types

type CommandSpec 

type CommandSpec struct {
	Binary    string
	Workspace string
	Prompt    string
	Focus     bool
	CodexHome string
}

CommandSpec is the external-agent invocation contract. The runner never interprets or modifies the agent's patch; it only records its output.

type ConditionSummary 

type ConditionSummary struct {
	Runs                  int     `json:"runs"`
	Qualifying            int     `json:"qualifying"`
	ObligationCoverage    int     `json:"obligation_coverage"`
	ObligationTotal       int     `json:"obligation_total"`
	EvidenceBytes         []int64 `json:"evidence_bytes"`
	ToolCalls             []int   `json:"tool_calls"`
	DurationMS            []int64 `json:"duration_ms"`
	OperatorInterventions int     `json:"operator_interventions"`
}

ConditionSummary aggregates runs for one condition.

type Events 

type Events struct {
	Usage     map[string]any
	Raw       []json.RawMessage
	ToolCalls int
}

Events contains parsed agent events and aggregate tool counts.

func ParseJSONL 

func ParseJSONL(stdout []byte) (Events, error)

ParseJSONL parses Codex JSONL events.

func RunOnce 

func RunOnce(ctx context.Context, spec CommandSpec) (Events, string, string, int64, error)

RunOnce executes one pilot command and captures its evidence.

type FocusMetric 

type FocusMetric struct {
	ErrorCategories                   map[string]int
	Calls, FailedCalls, FirstPosition int
	Refresh, Evidence                 bool
	FocusResultFollowedByEdit         bool
	RefreshCompleted                  bool
}

FocusMetric records terminal go_context outcomes and objective workflow signals. Calls counts successful completions; FailedCalls counts failed completions. The legacy Refresh and Evidence fields remain for compatible consumers and are derived without trusting agent prose.

func FocusMetrics 

func FocusMetrics(events Events) FocusMetric

FocusMetrics returns strict focus adoption metrics from agent events. It counts only completed MCP events and recognizes edits through the explicit file_change event, not through message wording.

type ObligationResult 

type ObligationResult struct {
	Obligation string `json:"obligation"`
	Status     string `json:"status"`
	Evidence   string `json:"evidence"`
}

ObligationResult records evidence for one scenario obligation.

func ScoreObligations 

func ScoreObligations(s Scenario, r Run) []ObligationResult

ScoreObligations evaluates scenario obligations from recorded evidence only. Agent prose is evidence, never an assertion of completion by itself.

type Run 

type Run struct {
	SchemaVersion             string             `json:"schema_version"`
	ScenarioID                string             `json:"scenario_id"`
	TaskID                    string             `json:"task_id"`
	Condition                 string             `json:"condition"`
	Repetition                int                `json:"repetition"`
	Model                     string             `json:"model"`
	Reasoning                 string             `json:"reasoning"`
	SourceSHA256              string             `json:"source_sha256"`
	BinarySHA256              string             `json:"binary_sha256"`
	WorkspaceSHA256           string             `json:"workspace_sha256"`
	Prompt                    string             `json:"prompt"`
	Transcript                []json.RawMessage  `json:"transcript"`
	Patch                     string             `json:"patch"`
	PatchSHA256               string             `json:"patch_sha256"`
	Acceptance                string             `json:"acceptance"`
	ProcessError              string             `json:"process_error,omitempty"`
	Stderr                    string             `json:"stderr,omitempty"`
	AcceptanceEvidenceSHA256  string             `json:"acceptance_evidence_sha256"`
	ScopeViolations           []string           `json:"scope_violations"`
	EvidenceBytes             int64              `json:"evidence_bytes"`
	ToolCalls                 int                `json:"tool_calls"`
	FocusToolCalls            int                `json:"focus_tool_calls"`
	FocusFailedCalls          int                `json:"focus_failed_calls"`
	FocusErrorCategories      map[string]int     `json:"focus_error_categories"`
	FirstFocusCallPosition    int                `json:"first_focus_call_position"`
	RefreshUse                bool               `json:"refresh_use"`
	FocusEvidenceUse          bool               `json:"focus_evidence_use"`
	FocusResultFollowedByEdit bool               `json:"focus_result_followed_by_edit"`
	RefreshCompleted          bool               `json:"refresh_completed"`
	FocusDelivery             string             `json:"focus_delivery"`
	DurationMS                int64              `json:"duration_ms"`
	OperatorIntervention      bool               `json:"operator_intervention"`
	Uncertainty               []string           `json:"uncertainty"`
	Qualifying                bool               `json:"qualifying"`
	DecisionObligations       []ObligationResult `json:"decision_obligations"`
	TranscriptSHA256          string             `json:"transcript_sha256"`
}

Run records one isolated agent evaluation.

func LoadRun 

func LoadRun(path string) (Run, error)

LoadRun decodes and validates one pilot record.

type Scenario 

type Scenario struct {
	SchemaVersion string   `json:"schema_version"`
	ID            string   `json:"id"`
	TaskID        string   `json:"task_id"`
	Prompt        string   `json:"prompt"`
	Obligations   []string `json:"obligations"`
}

Scenario describes one evaluated repository task.

func LoadScenario 

func LoadScenario(path string) (Scenario, error)

LoadScenario decodes and validates a scenario manifest.

func LoadScenarios 

func LoadScenarios(root string) ([]Scenario, error)

LoadScenarios loads the required two scenario manifests.

type Summary 

type Summary struct {
	SchemaVersion string                      `json:"schema_version"`
	Runs          int                         `json:"runs"`
	ByCondition   map[string]ConditionSummary `json:"by_condition"`
}

Summary aggregates pilot runs.

func Summarize 

func Summarize(runs []Run) Summary

Summarize aggregates the supplied pilot runs.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.