evals

package
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 23, 2026 License: MIT Imports: 39 Imported by: 0

Documentation

Overview

Package evals ports Pi's documentation eval tooling: the task plan, the comparison command and its Docker runner, the paired comparison report, the harness's prompt and environment helpers, and the fixtures the evals use.

Go has no Vitest. The vitest-evals harness that drives an agent Session (createPiCodingAgentHarness) is not ported here. stubgen:omit CreateDockerContext stubgen:omit CreatePiCodingAgentHarness stubgen:omit PiCodingAgentHarnessWithOutput stubgen:omit PiCodingAgentInput

Index

Constants

View Source
const (
	OpenAIProviderID    = "acme"
	OpenAIModelID       = "acme-chat"
	OpenAIProbePrompt   = "Reply with ACME_OK."
	OpenAIProbeResponse = "ACME_OK"
	StreamProviderID    = "acme-stream"
	StreamModelID       = "acme-stream-chat"
	StreamProbePrompt   = "Reply with ACME_STREAM_OK."
	StreamProbeResponse = "ACME_STREAM_OK"
)

Fixture identities of the OpenAI-compatible (openai) and custom NDJSON (stream) Acme APIs.

View Source
const DocsEvalSuffix = ".docs.eval.go"

DocsEvalSuffix ends the name of a documentation eval file. Pi's is ".docs.eval.ts"; PiG's suites are Go files.

View Source
const PiSessionSnapshotArtifact = "piSessionJsonl"

PiSessionSnapshotArtifact is the harness artifact that carries the Session JSONL snapshot.

View Source
const StreamAPIDocumentation = `{
  "name": "Acme Streaming API",
  "request": {
    "method": "POST",
    "path": "/generate",
    "headers": {
      "content-type": "application/json",
      "x-acme-key": "resolved credential"
    },
    "body": {
      "model": "` + StreamModelID + `",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        }
      ],
      "stream": true
    }
  },
  "response": {
    "contentType": "application/x-ndjson",
    "events": [
      {
        "type": "text_delta",
        "text": "Hello"
      },
      {
        "type": "usage",
        "input_tokens": 3,
        "output_tokens": 2
      },
      {
        "type": "done",
        "reason": "stop"
      }
    ]
  }
}
`

StreamAPIDocumentation describes the stream fixture's API to the agent under evaluation.

Variables

View Source
var DocumentationEvalTools = [...]string{"read", "write", "edit", "grep", "find", "ls"}

DocumentationEvalTools are the tools documentation evals run with; they exclude shell and unrestricted network tools.

DocumentationVariants lists the variants in plan order.

Functions

func AgentDirEnvironment

func AgentDirEnvironment() string

AgentDirEnvironment is the environment variable that selects the agent directory (D2).

func ApplyIsolatedEnvironment

func ApplyIsolatedEnvironment(home, agentDir string) func()

ApplyIsolatedEnvironment removes the runner's PI_EVAL_* variables and points HOME, USERPROFILE and the agent directory at the isolated paths. The returned function restores every variable it changed.

func DiscoverCases

func DiscoverCases(ctx context.Context, docker DockerContext, variant DocumentationVariant, files, vitestArgs []string) (string, error)

DiscoverCases lists the cases of files in the variant's image and returns the path of the discovery report.

func ExcludePiDocumentation

func ExcludePiDocumentation(defaultPrompt string) (string, error)

ExcludePiDocumentation removes the documentation section from the default system prompt. It fails when the section or the working-directory section after it is missing.

func FormatEvalComparisonReport

func FormatEvalComparisonReport(report EvalComparisonReport) string

FormatEvalComparisonReport renders report for the terminal. It returns "" for a report without comparisons.

func LoadConfiguredModelRuntime

func LoadConfiguredModelRuntime(ctx context.Context, agentDir string) (*coding.ModelRuntime, error)

LoadConfiguredModelRuntime creates a ModelRuntime from agentDir's models.json, auth.json and models-store.json without model network access. The caller owns ModelRuntime.Close.

func RPCSessionStats

func RPCSessionStats(ctx context.Context, pigPath, dir string, args, env []string) ([]byte, error)

RPCSessionStats starts pig in RPC mode with args and env in dir and returns the data of get_session_stats.

func RequireEvalAuthFile

func RequireEvalAuthFile(provider string) (string, error)

RequireEvalAuthFile returns the path of the agent directory's auth.json after checking that it is a JSON object with a credential for provider.

func RunEntrypoint

func RunEntrypoint(ctx context.Context, config EntrypointConfig, args []string, stdout io.Writer) (int, error)

RunEntrypoint runs the container entry point: it checks the image, prepares the agent directory, lists or runs the selected suites and returns the process exit code.

func RunTask

func RunTask(ctx context.Context, docker DockerContext, task EvalTask) (string, error)

RunTask runs one isolated task in a fresh container. A failed run is not an error: the result is the path of the Vitest report when the run wrote one, and "" otherwise.

func VerifySystemPrompt

func VerifySystemPrompt(systemPrompt string, options PiCodingAgentHarnessOptions) (string, error)

VerifySystemPrompt checks that systemPrompt kept its rules and has the documentation section the variant expects. It returns systemPrompt unchanged when the options expect nothing.

func WriteDiscovery

func WriteDiscovery(outputPath string, tests []DiscoveredTest) error

WriteDiscovery writes the discovered tests as JSON to outputPath.

func WriteVitestReport

func WriteVitestReport(outputPath string, report VitestReport) error

WriteVitestReport writes report as JSON to outputPath, 0o666 before the umask like Vitest's outputFile.

Types

type AcmeServer

type AcmeServer struct {
	// contains filtered or unexported fields
}

AcmeServer is a loopback HTTP fixture that accepts only correctly authenticated, well-formed streaming requests and records whether the last accepted request was the probe prompt.

func CreateAcmeServer

func CreateAcmeServer(mode AcmeServerMode) *AcmeServer

CreateAcmeServer returns a stopped fixture server for mode.

func (*AcmeServer) BaseURL

func (server *AcmeServer) BaseURL() string

BaseURL is the OpenAI-compatible base URL, Origin()+"/v1".

func (*AcmeServer) Origin

func (server *AcmeServer) Origin() string

Origin is the server's http://127.0.0.1:<port> origin, or "" before Start.

func (*AcmeServer) Reset

func (server *AcmeServer) Reset()

Reset clears the probe flag.

func (*AcmeServer) Start

func (server *AcmeServer) Start() error

Start listens on an ephemeral loopback port.

func (*AcmeServer) Stop

func (server *AcmeServer) Stop(ctx context.Context) error

Stop stops accepting connections and waits for in-flight requests, as Node's server.close does.

func (*AcmeServer) ValidRequestReceived

func (server *AcmeServer) ValidRequestReceived() bool

ValidRequestReceived reports whether the last accepted request was the probe.

type AcmeServerMode

type AcmeServerMode string

AcmeServerMode selects the fixture API.

const (
	AcmeServerModeOpenAI AcmeServerMode = "openai"
	AcmeServerModeStream AcmeServerMode = "stream"
)

type AddedModel

type AddedModel struct {
	Model                   ModelFields `json:"model"`
	ExistingModelsPreserved bool        `json:"existingModelsPreserved"`
}

AddedModel is a model that models.json added to a built-in provider.

type AddedModelOutput

type AddedModelOutput struct {
	Result AddedModelResult `json:"result"`
}

AddedModelOutput is the structured output of an added-model eval.

func InspectAddedModel

func InspectAddedModel(ctx context.Context, runtime *coding.ModelRuntime, providerID, modelID string) (AddedModelOutput, error)

InspectAddedModel checks that models.json added modelID to the built-in provider providerID without dropping any of the provider's built-in models.

type AddedModelResult

type AddedModelResult struct {
	*AddedModel
	Error string `json:"error,omitempty"`
}

AddedModelResult holds either the added model or an error message.

type AgentRun

type AgentRun struct {
	Response     string
	SystemPrompt string
	AgentDir     string
	Workspace    string
	// Model and PigPath identify the Session's model and the pig binary, for outputs that start more pig processes.
	Model            PiCodingAgentModelSelection
	PigPath          string
	Messages         []map[string]any
	ExtensionErrors  []json.RawMessage
	SuccessfulTools  map[string]string
	SessionStatistic SessionStatistics
}

AgentRun is what a harness output callback inspects after the last step: the run's answer, the system prompt the model received, the isolated agent and workspace directories, the final Session messages as RPC JSON, and what the extension host reported.

type BlockedPair

type BlockedPair struct {
	EvalSet   string   `json:"evalSet"`
	CaseID    string   `json:"caseId"`
	Model     string   `json:"model"`
	RunNumber int      `json:"runNumber"`
	Reasons   []string `json:"reasons"`
}

BlockedPair is a pair excluded from comparison, with the reasons.

type BuiltImage

type BuiltImage struct {
	Name string `json:"name"`
	ID   string `json:"id"`
}

BuiltImage names one documentation-variant image and its content ID.

type BuiltImages

type BuiltImages struct {
	WithoutDocs BuiltImage `json:"without_docs"`
	WithDocs    BuiltImage `json:"with_docs"`
}

BuiltImages holds the image of each documentation variant.

func BuildImages

func BuildImages(ctx context.Context, command Command, packageRoot, repositoryRoot string) (BuiltImages, error)

BuildImages builds the without_docs and with_docs images from the Dockerfile under packageRoot with repositoryRoot as the build context, then reads each image ID.

func (BuiltImages) Variant

func (images BuiltImages) Variant(variant DocumentationVariant) BuiltImage

Variant returns the image of variant.

type Case

type Case struct {
	Name string
	Run  func(ctx context.Context, run RunFunc) error
}

Case is one eval case. A returned error fails the case (a Vitest assertion failure).

type Command

type Command interface {
	Execute(ctx context.Context, name string, args []string, capture bool) (status int, stdout string, err error)
}

Command runs one external command and reports its exit status. Capture returns stdout and leaves stderr attached to the caller; otherwise all three streams stay attached.

type CustomTool

type CustomTool struct {
	bridge.Tool
	// Execute runs the tool; it is called on the harness side with the model's arguments.
	Execute func(ctx context.Context, toolCallID string, params map[string]any) (CustomToolResult, error)
}

CustomTool is a tool the harness serves to the agent through the bridge extension (harness.ts customTools).

type CustomToolResult

type CustomToolResult struct {
	Content   string
	Details   any
	Terminate bool
}

CustomToolResult is a custom tool's result (AgentToolResult content, details and terminate).

type DiscoveredEvalCase

type DiscoveredEvalCase struct {
	File     string `json:"file"`
	FullName string `json:"fullName"`
	EvalSet  string `json:"evalSet"`
	CaseID   string `json:"caseId"`
}

DiscoveredEvalCase is one Vitest eval case named "<eval set> > <case>".

func ParseDiscoveredCases

func ParseDiscoveredCases(value any) ([]DiscoveredEvalCase, error)

ParseDiscoveredCases validates the decoded JSON list of discovered cases and derives each case identity.

type DiscoveredTest

type DiscoveredTest struct {
	Name string `json:"name"`
	File string `json:"file"`
}

DiscoveredTest is one `vitest list --json` entry.

type DockerContext

type DockerContext struct {
	Command           Command
	PackageRoot       string
	Images            BuiltImages
	ArtifactDirectory string
	AuthPath          string
	Provider          string
	Model             string
	RunsPerVariant    int
}

DockerContext carries what every container run needs.

type DocumentationVariant

type DocumentationVariant string

DocumentationVariant selects whether the Pi documentation section stays in the system prompt.

const (
	DocumentationVariantWithoutDocs DocumentationVariant = "without_docs"
	DocumentationVariantWithDocs    DocumentationVariant = "with_docs"
)

func ResolveDocumentationVariant

func ResolveDocumentationVariant(value ...string) (DocumentationVariant, error)

ResolveDocumentationVariant validates value, which defaults to PI_EVAL_VARIANT when omitted.

type EntrypointConfig

type EntrypointConfig struct {
	// PackageRoot is the evals package directory as discovered files report it.
	PackageRoot string
	// Runner is the root-only evaluator binary.
	Runner string
	// Binaries are the executables the sandboxed agent runs; each must exist.
	Binaries []string
	// AuthSource is the credential file the host mounts.
	AuthSource string
	// HostAgentDir receives the credentials for the harness.
	HostAgentDir string
	// ArtifactDir is where reports are written.
	ArtifactDir string
	Suites      Runner
}

EntrypointConfig locates the image's contents.

type EvalCli

type EvalCli struct {
	PackageRoot    string
	RepositoryRoot string
	Command        Command
	Environment    map[string]string
	Stdout         io.Writer
	Now            func() time.Time
	// DocumentationEvals lists the documentation eval files when no file is named; nil walks PackageRoot/evals.
	DocumentationEvals func() []string
}

EvalCli is one documentation comparison run: where its package and repository are, how it runs commands, and its process inputs.

func (EvalCli) RunEvalCli

func (cli EvalCli) RunEvalCli(ctx context.Context, args []string) (int, error)

RunEvalCli runs the documentation comparison for args and returns the process exit code: 1 when any pair is blocked, 0 otherwise. A configuration or infrastructure failure is returned as an error.

type EvalCliOptions

type EvalCliOptions struct {
	Provider       string
	Model          string
	RunsPerVariant int
	RequestedFiles []string
	DiscoveryArgs  []string
}

EvalCliOptions are the parsed command-line options of the documentation comparison.

func ParseEvalCli

func ParseEvalCli(args []string, environment ...map[string]string) (EvalCliOptions, error)

ParseEvalCli parses the arguments after the program name. The model comes from --provider and --model, which must come together, or from PI_PROVIDER and PI_MODEL in environment (the process environment when omitted).

type EvalComparisonFlag

type EvalComparisonFlag string

EvalComparisonFlag marks a notable comparison result.

const (
	EvalComparisonFlagNoLift             EvalComparisonFlag = "no-lift"
	EvalComparisonFlagNegativeDelta      EvalComparisonFlag = "negative-delta"
	EvalComparisonFlagControlSaturated   EvalComparisonFlag = "control-saturated"
	EvalComparisonFlagTreatmentSaturated EvalComparisonFlag = "treatment-saturated"
	EvalComparisonFlagFlaky              EvalComparisonFlag = "flaky"
)

type EvalComparisonReport

type EvalComparisonReport struct {
	SchemaVersion     int                  `json:"schemaVersion"`
	ProtocolDigest    string               `json:"protocolDigest"`
	Control           DocumentationVariant `json:"control"`
	Treatment         DocumentationVariant `json:"treatment"`
	Comparisons       []EvalSetComparison  `json:"comparisons"`
	BlockedPairs      []BlockedPair        `json:"blockedPairs"`
	OperationalTotals []VariantTotals      `json:"operationalTotals"`
}

EvalComparisonReport is the paired comparison of the documentation variants.

func SummarizeEvalObservations

func SummarizeEvalObservations(protocolDigest string, expectedRuns []ExpectedEvalRun, observations []EvalObservation) EvalComparisonReport

SummarizeEvalObservations pairs each case run's control and treatment observations and compares the variants per eval set. A pair is blocked unless the design expects one run of each variant and each produced exactly one scored observation; an eval set with a blocked pair withholds its pass rates.

type EvalMetrics

type EvalMetrics struct {
	InputTokens      *float64 `json:"inputTokens,omitempty"`
	OutputTokens     *float64 `json:"outputTokens,omitempty"`
	CacheReadTokens  *float64 `json:"cacheReadTokens,omitempty"`
	CacheWriteTokens *float64 `json:"cacheWriteTokens,omitempty"`
	TotalTokens      *float64 `json:"totalTokens,omitempty"`
	ToolCalls        *float64 `json:"toolCalls,omitempty"`
	TotalMs          *float64 `json:"totalMs,omitempty"`
	EstimatedCostUsd *float64 `json:"estimatedCostUsd,omitempty"`
}

EvalMetrics are a run's measured costs. A nil metric was not measured, which is distinct from zero.

type EvalObservation

type EvalObservation struct {
	EvalRunIdentity
	EvalMetrics
	Outcome EvalOutcome `json:"outcome"`
	Score   *float64    `json:"score,omitempty"`
}

EvalObservation is one observed run. Score is set exactly when Outcome is EvalOutcomeScored.

func ErroredObservation

func ErroredObservation(task EvalTask) EvalObservation

ErroredObservation is the observation of a task that produced no report.

func ReadTaskObservation

func ReadTaskObservation(task EvalTask, reportPath, artifactDirectory string) (EvalObservation, error)

ReadTaskObservation reads the single-case Vitest report of task, persists its Session snapshot under artifactDirectory, and classifies the run. A report that cannot be read, holds another case, or reports another model is an errored observation; only a failure to persist the snapshot is returned as an error.

type EvalOutcome

type EvalOutcome string

EvalOutcome classifies a run.

const (
	EvalOutcomeScored   EvalOutcome = "scored"
	EvalOutcomeUnscored EvalOutcome = "unscored"
	EvalOutcomeSkipped  EvalOutcome = "skipped"
	EvalOutcomePending  EvalOutcome = "pending"
	EvalOutcomeErrored  EvalOutcome = "errored"
)

func ClassifyCaseStatus

func ClassifyCaseStatus(status ReportCaseStatus) EvalOutcome

ClassifyCaseStatus maps a failed case to errored, a skipped, to-do or disabled case to skipped, and a pending case to pending. It returns "" for any other status.

type EvalProject

type EvalProject string

EvalProject selects the docs or host suites (vitest.evals.config.ts projects).

const (
	// EvalProjectDocs runs the *.docs.eval.go suites; each case runs in a container per documentation variant.
	EvalProjectDocs EvalProject = "docs"
	// EvalProjectHost runs every other suite on this machine.
	EvalProjectHost EvalProject = "host"
)

type EvalRunIdentity

type EvalRunIdentity struct {
	EvalSet   string               `json:"evalSet"`
	CaseID    string               `json:"caseId"`
	Variant   DocumentationVariant `json:"variant"`
	Model     string               `json:"model"`
	RunNumber int                  `json:"runNumber"`
}

EvalRunIdentity identifies one run of a case.

type EvalSetComparison

type EvalSetComparison struct {
	EvalSet           string               `json:"evalSet"`
	TotalPairs        int                  `json:"totalPairs"`
	EligiblePairs     int                  `json:"eligiblePairs"`
	BlockedPairs      int                  `json:"blockedPairs"`
	ControlPassRate   *float64             `json:"controlPassRate"`
	TreatmentPassRate *float64             `json:"treatmentPassRate"`
	Lift              *float64             `json:"lift"`
	Flags             []EvalComparisonFlag `json:"flags"`
	TotalTokens       PairedMetricSummary  `json:"totalTokens"`
	ToolCalls         PairedMetricSummary  `json:"toolCalls"`
	TotalMs           PairedMetricSummary  `json:"totalMs"`
	EstimatedCostUsd  PairedMetricSummary  `json:"estimatedCostUsd"`
}

EvalSetComparison compares the variants over one eval set. Pass rates and lift are nil unless every pair is eligible.

type EvalTask

type EvalTask struct {
	DiscoveredEvalCase
	Variant   DocumentationVariant `json:"variant"`
	Model     string               `json:"model"`
	RunNumber int                  `json:"runNumber"`
}

EvalTask is one isolated run of a case under one variant, model and repetition.

func CreateTaskPlan

func CreateTaskPlan(cases []DiscoveredEvalCase, model string, runsPerVariant int) ([]EvalTask, error)

CreateTaskPlan creates one task per case, repetition and variant. Odd repetitions run without_docs first and even repetitions with_docs first.

type ExecCommand

type ExecCommand struct{ Dir string }

ExecCommand runs commands with os/exec in Dir.

func (ExecCommand) Execute

func (command ExecCommand) Execute(ctx context.Context, name string, args []string, capture bool) (int, string, error)

Execute implements Command. A command that cannot start is an error; one that ends without an exit status counts as status 1.

type ExpectedEvalRun

type ExpectedEvalRun = EvalRunIdentity

ExpectedEvalRun is a run the plan expects.

type ExpectedTool

type ExpectedTool struct {
	Name      string
	Arguments map[string]any
}

ExpectedTool is a tool a ToolCallJudge requires. A nil Arguments matches any arguments.

type HarnessRun

type HarnessRun struct {
	// Name is the harness name (createHarness name), "pi-coding-agent" unless the options name it.
	Name      string
	Output    any
	Events    []TranscriptEvent
	Metadata  map[string]any
	Usage     UsageSummary
	TotalMs   float64
	Artifacts map[string]any
	Errors    []error
}

HarnessRun is the normalized result of one run.

func RunPiCodingAgent

func RunPiCodingAgent(ctx context.Context, input PiCodingAgentInput, options PiCodingAgentHarnessOptions) (result *HarnessRun, failure error)

RunPiCodingAgent runs input against a fresh PiG Session in an isolated root and normalizes the result. The root, home, agent directory, workspace and Session directory are new for each run and removed afterwards.

type HarnessRunError

type HarnessRunError struct {
	Err error
	Run *HarnessRun
}

HarnessRunError is a failed run that still measured something (attachHarnessRunToError).

func (*HarnessRunError) Error

func (failure *HarnessRunError) Error() string

func (*HarnessRunError) Unwrap

func (failure *HarnessRunError) Unwrap() error

type Judge

type Judge interface {
	Name() string
	Assess(output any, calls []ToolCall) (JudgeScore, error)
}

Judge scores a run's output and tool calls.

func StructuredOutputJudge

func StructuredOutputJudge(expected any) (Judge, error)

StructuredOutputJudge scores a JSON object output against expected: strict field equality, an error field fails the run, and extra fields fail it (allowExtras false). The score is the fraction of matching fields.

func ToolCallJudge

func ToolCallJudge(expected ...ExpectedTool) Judge

ToolCallJudge requires every expected tool to be called, in any order, with strictly equal arguments when given; extra calls are allowed.

type JudgeScore

type JudgeScore struct {
	Name     string         `json:"name"`
	Score    float64        `json:"score"`
	Metadata map[string]any `json:"metadata,omitempty"`
}

JudgeScore is one judge's verdict.

type ModelCostFields

type ModelCostFields struct {
	Input      float64 `json:"input"`
	Output     float64 `json:"output"`
	CacheRead  float64 `json:"cacheRead"`
	CacheWrite float64 `json:"cacheWrite"`
}

ModelCostFields is a model's per-token pricing.

type ModelFields

type ModelFields struct {
	ID            string          `json:"id"`
	Name          string          `json:"name"`
	Provider      string          `json:"provider"`
	Reasoning     bool            `json:"reasoning"`
	Input         []string        `json:"input"`
	Cost          ModelCostFields `json:"cost"`
	ContextWindow int             `json:"contextWindow"`
	MaxTokens     int             `json:"maxTokens"`
}

ModelFields is the model metadata an eval compares.

type OperationalMetricTotal

type OperationalMetricTotal struct {
	AvailableRuns int      `json:"availableRuns"`
	Total         *float64 `json:"total"`
}

OperationalMetricTotal sums a metric over the runs that measured it. Total is nil when no run did.

type PairedMetricSummary

type PairedMetricSummary struct {
	EligiblePairs int      `json:"eligiblePairs"`
	ControlMean   *float64 `json:"controlMean"`
	TreatmentMean *float64 `json:"treatmentMean"`
	MeanDelta     *float64 `json:"meanDelta"`
}

PairedMetricSummary compares a metric over the pairs that measured it in both variants.

type PiCodingAgentHarnessOptions

type PiCodingAgentHarnessOptions struct {
	Name                    string
	Model                   *PiCodingAgentModelSelection
	NoTools                 string
	Tools                   []string
	CustomTools             []CustomTool
	WorkspaceFiles          map[string]string
	TransformSystemPrompt   func(defaultPrompt string) (string, error)
	ExpectedPiDocumentation *bool
	Output                  PiCodingAgentOutput
	PigPath                 string
	ExtensionPath           string
	// contains filtered or unexported fields
}

PiCodingAgentHarnessOptions configures an eval harness. A nil Tools keeps the Session default. PigPath and ExtensionPath name the pig binary and the cmd/pig-eval-extension binary; empty values read PI_EVAL_PIG and PI_EVAL_EXTENSION, and the pig binary falls back to PATH.

func CreatePiDocumentationEvalHarness

func CreatePiDocumentationEvalHarness(options ...PiCodingAgentHarnessOptions) (PiCodingAgentHarnessOptions, error)

CreatePiDocumentationEvalHarness returns the harness options of the documentation variant named by PI_EVAL_VARIANT. It fails outside the isolated container sandbox (PI_EVAL_CONTAINER=1 with a sandbox identity). Tools default to DocumentationEvalTools; the without_docs variant strips the documentation section.

type PiCodingAgentInput

type PiCodingAgentInput []PiCodingAgentStep

PiCodingAgentInput is the steps of one run. A plain string input is PromptInput.

func PromptInput

func PromptInput(content string) PiCodingAgentInput

PromptInput is the input that sends one prompt.

type PiCodingAgentModelSelection

type PiCodingAgentModelSelection struct {
	Provider string `json:"provider"`
	ID       string `json:"id"`
}

PiCodingAgentModelSelection names the provider and model an eval runs.

func ResolveModelSelection

func ResolveModelSelection(explicitModel *PiCodingAgentModelSelection, environment ...map[string]string) (PiCodingAgentModelSelection, error)

ResolveModelSelection prefers explicitModel and otherwise reads PI_PROVIDER and PI_MODEL from environment, which defaults to the process environment.

type PiCodingAgentOutput

type PiCodingAgentOutput func(ctx context.Context, run AgentRun) (any, error)

PiCodingAgentOutput computes a run's structured output (PiCodingAgentHarnessWithOutput.output).

type PiCodingAgentStep

type PiCodingAgentStep struct {
	Type    PiCodingAgentStepType `json:"type"`
	Content string                `json:"content,omitempty"`
}

PiCodingAgentStep is one step of a PiCodingAgentInput.

type PiCodingAgentStepType

type PiCodingAgentStepType string

PiCodingAgentStepType selects what a step of an input does.

const (
	// PiCodingAgentStepPrompt sends Content to the agent.
	PiCodingAgentStepPrompt PiCodingAgentStepType = "prompt"
	// PiCodingAgentStepReload reloads the Session's resources and extensions.
	PiCodingAgentStepReload PiCodingAgentStepType = "reload"
)

type ProviderProbe

type ProviderProbe struct {
	ValidRequestReceived bool                  `json:"validRequestReceived"`
	Model                ModelFields           `json:"model"`
	Response             ProviderProbeResponse `json:"response"`
}

ProviderProbe is a completed probe of a configured provider.

type ProviderProbeResponse

type ProviderProbeResponse struct {
	Text         string `json:"text"`
	StopReason   string `json:"stopReason"`
	InputTokens  int    `json:"inputTokens"`
	OutputTokens int    `json:"outputTokens"`
}

ProviderProbeResponse summarizes the probe completion.

type ProviderRuntimeOutput

type ProviderRuntimeOutput struct {
	Result ProviderRuntimeResult `json:"result"`
}

ProviderRuntimeOutput is the structured output of a provider eval.

func InspectProvider

func InspectProvider(ctx context.Context, runtime *coding.ModelRuntime, scenario ProviderScenario) ProviderRuntimeOutput

InspectProvider reloads the runtime's configuration offline, then completes the scenario's context with the configured model. Configuration and lookup failures are structured errors.

type ProviderRuntimeResult

type ProviderRuntimeResult struct {
	*ProviderProbe
	Error string `json:"error,omitempty"`
}

ProviderRuntimeResult holds either a probe or an error message.

type ProviderScenario

type ProviderScenario struct {
	ProviderID           string
	ModelID              string
	CreateContext        func() ai.Context
	Options              ai.StreamOptions
	ValidRequestReceived func() bool
}

ProviderScenario names the configured model to probe and how to recognize the fixture's probe request.

type ReportCaseStatus

type ReportCaseStatus string

ReportCaseStatus is a Vitest assertion status.

const (
	ReportCaseStatusPassed   ReportCaseStatus = "passed"
	ReportCaseStatusFailed   ReportCaseStatus = "failed"
	ReportCaseStatusSkipped  ReportCaseStatus = "skipped"
	ReportCaseStatusPending  ReportCaseStatus = "pending"
	ReportCaseStatusTodo     ReportCaseStatus = "todo"
	ReportCaseStatusDisabled ReportCaseStatus = "disabled"
)

type RunFunc

type RunFunc func(ctx context.Context, input PiCodingAgentInput) (*HarnessRun, error)

RunFunc runs input through the suite's harness, records the run for the report and applies the suite's judges (the `run` fixture of vitest-evals).

type Runner

type Runner struct {
	Suites      []Suite
	PackageRoot string
}

Runner lists and runs suites. PackageRoot is the evals package directory as the container reports it.

func (Runner) List

func (runner Runner) List(selection Selection) []DiscoveredTest

List is `vitest list --json`: the selected cases named "<suite> > <case>" with their absolute file.

func (Runner) Run

func (runner Runner) Run(ctx context.Context, selection Selection) VitestReport

Run runs the selected cases in order and returns the JSON report. Suite setup failures fail every case of the suite. The report's Success is false when a case failed.

type Selection

type Selection struct {
	Project EvalProject
	// Files are paths relative to the package root; empty selects every file of the project.
	Files []string
	// NamePattern matches "<suite> <case>" (Vitest -t); nil matches all.
	NamePattern *regexp.Regexp
}

Selection narrows the suites and cases a runner lists or runs.

func ParseEntrypointArgs

func ParseEntrypointArgs(args []string) (discover bool, selection Selection, err error)

ParseEntrypointArgs reads the arguments the host passes (docker.ts discoverCases and runTask): --discover, --project, eval files and a test name pattern.

type SessionStatistics

type SessionStatistics struct {
	SessionFile string
	ToolCalls   float64
	Input       float64
	Output      float64
	CacheRead   float64
	CacheWrite  float64
	Total       float64
	Cost        float64
}

SessionStatistics are the Session totals the agent process reports (get_session_stats).

type Suite

type Suite struct {
	Name    string
	File    string
	Harness func(ctx context.Context, input PiCodingAgentInput) (*HarnessRun, error)
	Judges  []Judge
	// JudgeThreshold is the average score a case needs; nil is Vitest-evals' default of 1. NoJudgeThreshold records
	// scores without failing (judgeThreshold: null).
	JudgeThreshold   *float64
	NoJudgeThreshold bool
	Cases            []Case
	// Setup runs before the suite's cases and returns its cleanup (beforeAll/afterAll). The cleanup runs even when Setup
	// fails.
	Setup func(ctx context.Context) (func(context.Context) error, error)
	// BeforeEach runs before every case (beforeEach).
	BeforeEach func()
}

Suite is a describeEval: a harness, judges and cases. File is the path under the evals package, ending in DocsEvalSuffix for a documentation suite.

func (Suite) Project

func (suite Suite) Project() EvalProject

Project is the project the suite belongs to.

type ToolCall

type ToolCall struct {
	Name      string         `json:"name"`
	Arguments map[string]any `json:"arguments,omitempty"`
	Status    string         `json:"status"`
	Result    any            `json:"result,omitempty"`
	Error     map[string]any `json:"error,omitempty"`
}

ToolCall is a transcript tool call joined with its result (@vitest-evals/core toolCalls).

func ToolCalls

func ToolCalls(events []TranscriptEvent) []ToolCall

ToolCalls joins each tool_call event with the first tool_result of the same call ID.

type TranscriptEvent

type TranscriptEvent = map[string]any

TranscriptEvent is one normalized transcript event.

type UsageSummary

type UsageSummary struct {
	Provider     string         `json:"provider"`
	Model        string         `json:"model"`
	InputTokens  float64        `json:"inputTokens"`
	OutputTokens float64        `json:"outputTokens"`
	TotalTokens  float64        `json:"totalTokens"`
	ToolCalls    float64        `json:"toolCalls"`
	Metadata     map[string]any `json:"metadata"`
}

UsageSummary is a run's model usage.

type VariantTotals

type VariantTotals struct {
	Variant          DocumentationVariant   `json:"variant"`
	Runs             int                    `json:"runs"`
	InputTokens      OperationalMetricTotal `json:"inputTokens"`
	OutputTokens     OperationalMetricTotal `json:"outputTokens"`
	CacheReadTokens  OperationalMetricTotal `json:"cacheReadTokens"`
	CacheWriteTokens OperationalMetricTotal `json:"cacheWriteTokens"`
	TotalTokens      OperationalMetricTotal `json:"totalTokens"`
	ToolCalls        OperationalMetricTotal `json:"toolCalls"`
	TotalMs          OperationalMetricTotal `json:"totalMs"`
	EstimatedCostUsd OperationalMetricTotal `json:"estimatedCostUsd"`
}

VariantTotals are one variant's operational totals over every observed run.

type VitestReport

type VitestReport struct {
	NumTotalTests   int          `json:"numTotalTests"`
	NumPassedTests  int          `json:"numPassedTests"`
	NumFailedTests  int          `json:"numFailedTests"`
	NumPendingTests int          `json:"numPendingTests"`
	NumTodoTests    int          `json:"numTodoTests"`
	StartTime       int64        `json:"startTime"`
	Success         bool         `json:"success"`
	TestResults     []fileResult `json:"testResults"`
}

VitestReport is the JSON reporter's output.

Directories

Path Synopsis
Package bridge connects the eval harness to the extension that runs inside the agent process under test.
Package bridge connects the eval harness to the extension that runs inside the agent process under test.
Package evalsuites holds the eval suites of Pi's packages/evals/evals directory, written against the PiG Session harness.
Package evalsuites holds the eval suites of Pi's packages/evals/evals directory, written against the PiG Session harness.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL