Documentation
¶
Overview ¶
Package evals ports Pi's documentation eval tooling: the task plan, the comparison command and its Docker runner, the paired comparison report, the harness's prompt and environment helpers, and the fixtures the evals use.
Go has no Vitest. The vitest-evals harness that drives an agent Session (createPiCodingAgentHarness) is not ported here. stubgen:omit CreateDockerContext stubgen:omit CreatePiCodingAgentHarness stubgen:omit PiCodingAgentHarnessWithOutput stubgen:omit PiCodingAgentInput
Index ¶
- Constants
- Variables
- func AgentDirEnvironment() string
- func ApplyIsolatedEnvironment(home, agentDir string) func()
- func DiscoverCases(ctx context.Context, docker DockerContext, variant DocumentationVariant, ...) (string, error)
- func ExcludePiDocumentation(defaultPrompt string) (string, error)
- func FormatEvalComparisonReport(report EvalComparisonReport) string
- func LoadConfiguredModelRuntime(ctx context.Context, agentDir string) (*coding.ModelRuntime, error)
- func RPCSessionStats(ctx context.Context, pigPath, dir string, args, env []string) ([]byte, error)
- func RequireEvalAuthFile(provider string) (string, error)
- func RunEntrypoint(ctx context.Context, config EntrypointConfig, args []string, stdout io.Writer) (int, error)
- func RunTask(ctx context.Context, docker DockerContext, task EvalTask) (string, error)
- func VerifySystemPrompt(systemPrompt string, options PiCodingAgentHarnessOptions) (string, error)
- func WriteDiscovery(outputPath string, tests []DiscoveredTest) error
- func WriteVitestReport(outputPath string, report VitestReport) error
- type AcmeServer
- type AcmeServerMode
- type AddedModel
- type AddedModelOutput
- type AddedModelResult
- type AgentRun
- type BlockedPair
- type BuiltImage
- type BuiltImages
- type Case
- type Command
- type CustomTool
- type CustomToolResult
- type DiscoveredEvalCase
- type DiscoveredTest
- type DockerContext
- type DocumentationVariant
- type EntrypointConfig
- type EvalCli
- type EvalCliOptions
- type EvalComparisonFlag
- type EvalComparisonReport
- type EvalMetrics
- type EvalObservation
- type EvalOutcome
- type EvalProject
- type EvalRunIdentity
- type EvalSetComparison
- type EvalTask
- type ExecCommand
- type ExpectedEvalRun
- type ExpectedTool
- type HarnessRun
- type HarnessRunError
- type Judge
- type JudgeScore
- type ModelCostFields
- type ModelFields
- type OperationalMetricTotal
- type PairedMetricSummary
- type PiCodingAgentHarnessOptions
- type PiCodingAgentInput
- type PiCodingAgentModelSelection
- type PiCodingAgentOutput
- type PiCodingAgentStep
- type PiCodingAgentStepType
- type ProviderProbe
- type ProviderProbeResponse
- type ProviderRuntimeOutput
- type ProviderRuntimeResult
- type ProviderScenario
- type ReportCaseStatus
- type RunFunc
- type Runner
- type Selection
- type SessionStatistics
- type Suite
- type ToolCall
- type TranscriptEvent
- type UsageSummary
- type VariantTotals
- type VitestReport
Constants ¶
const ( OpenAIProviderID = "acme" OpenAIModelID = "acme-chat" OpenAIProbePrompt = "Reply with ACME_OK." OpenAIProbeResponse = "ACME_OK" StreamProviderID = "acme-stream" StreamModelID = "acme-stream-chat" StreamProbePrompt = "Reply with ACME_STREAM_OK." StreamProbeResponse = "ACME_STREAM_OK" )
Fixture identities of the OpenAI-compatible (openai) and custom NDJSON (stream) Acme APIs.
const DocsEvalSuffix = ".docs.eval.go"
DocsEvalSuffix ends the name of a documentation eval file. Pi's is ".docs.eval.ts"; PiG's suites are Go files.
const PiSessionSnapshotArtifact = "piSessionJsonl"
PiSessionSnapshotArtifact is the harness artifact that carries the Session JSONL snapshot.
const StreamAPIDocumentation = `{ "name": "Acme Streaming API", "request": { "method": "POST", "path": "/generate", "headers": { "content-type": "application/json", "x-acme-key": "resolved credential" }, "body": { "model": "` + StreamModelID + `", "messages": [ { "role": "user", "content": "Hello" } ], "stream": true } }, "response": { "contentType": "application/x-ndjson", "events": [ { "type": "text_delta", "text": "Hello" }, { "type": "usage", "input_tokens": 3, "output_tokens": 2 }, { "type": "done", "reason": "stop" } ] } } `
StreamAPIDocumentation describes the stream fixture's API to the agent under evaluation.
Variables ¶
var DocumentationEvalTools = [...]string{"read", "write", "edit", "grep", "find", "ls"}
DocumentationEvalTools are the tools documentation evals run with; they exclude shell and unrestricted network tools.
var DocumentationVariants = [...]DocumentationVariant{DocumentationVariantWithoutDocs, DocumentationVariantWithDocs}
DocumentationVariants lists the variants in plan order.
Functions ¶
func AgentDirEnvironment ¶
func AgentDirEnvironment() string
AgentDirEnvironment is the environment variable that selects the agent directory (D2).
func ApplyIsolatedEnvironment ¶
func ApplyIsolatedEnvironment(home, agentDir string) func()
ApplyIsolatedEnvironment removes the runner's PI_EVAL_* variables and points HOME, USERPROFILE and the agent directory at the isolated paths. The returned function restores every variable it changed.
func DiscoverCases ¶
func DiscoverCases(ctx context.Context, docker DockerContext, variant DocumentationVariant, files, vitestArgs []string) (string, error)
DiscoverCases lists the cases of files in the variant's image and returns the path of the discovery report.
func ExcludePiDocumentation ¶
ExcludePiDocumentation removes the documentation section from the default system prompt. It fails when the section or the working-directory section after it is missing.
func FormatEvalComparisonReport ¶
func FormatEvalComparisonReport(report EvalComparisonReport) string
FormatEvalComparisonReport renders report for the terminal. It returns "" for a report without comparisons.
func LoadConfiguredModelRuntime ¶
LoadConfiguredModelRuntime creates a ModelRuntime from agentDir's models.json, auth.json and models-store.json without model network access. The caller owns ModelRuntime.Close.
func RPCSessionStats ¶
RPCSessionStats starts pig in RPC mode with args and env in dir and returns the data of get_session_stats.
func RequireEvalAuthFile ¶
RequireEvalAuthFile returns the path of the agent directory's auth.json after checking that it is a JSON object with a credential for provider.
func RunEntrypoint ¶
func RunEntrypoint(ctx context.Context, config EntrypointConfig, args []string, stdout io.Writer) (int, error)
RunEntrypoint runs the container entry point: it checks the image, prepares the agent directory, lists or runs the selected suites and returns the process exit code.
func RunTask ¶
RunTask runs one isolated task in a fresh container. A failed run is not an error: the result is the path of the Vitest report when the run wrote one, and "" otherwise.
func VerifySystemPrompt ¶
func VerifySystemPrompt(systemPrompt string, options PiCodingAgentHarnessOptions) (string, error)
VerifySystemPrompt checks that systemPrompt kept its rules and has the documentation section the variant expects. It returns systemPrompt unchanged when the options expect nothing.
func WriteDiscovery ¶
func WriteDiscovery(outputPath string, tests []DiscoveredTest) error
WriteDiscovery writes the discovered tests as JSON to outputPath.
func WriteVitestReport ¶
func WriteVitestReport(outputPath string, report VitestReport) error
WriteVitestReport writes report as JSON to outputPath, 0o666 before the umask like Vitest's outputFile.
Types ¶
type AcmeServer ¶
type AcmeServer struct {
// contains filtered or unexported fields
}
AcmeServer is a loopback HTTP fixture that accepts only correctly authenticated, well-formed streaming requests and records whether the last accepted request was the probe prompt.
func CreateAcmeServer ¶
func CreateAcmeServer(mode AcmeServerMode) *AcmeServer
CreateAcmeServer returns a stopped fixture server for mode.
func (*AcmeServer) BaseURL ¶
func (server *AcmeServer) BaseURL() string
BaseURL is the OpenAI-compatible base URL, Origin()+"/v1".
func (*AcmeServer) Origin ¶
func (server *AcmeServer) Origin() string
Origin is the server's http://127.0.0.1:<port> origin, or "" before Start.
func (*AcmeServer) Start ¶
func (server *AcmeServer) Start() error
Start listens on an ephemeral loopback port.
func (*AcmeServer) Stop ¶
func (server *AcmeServer) Stop(ctx context.Context) error
Stop stops accepting connections and waits for in-flight requests, as Node's server.close does.
func (*AcmeServer) ValidRequestReceived ¶
func (server *AcmeServer) ValidRequestReceived() bool
ValidRequestReceived reports whether the last accepted request was the probe.
type AcmeServerMode ¶
type AcmeServerMode string
AcmeServerMode selects the fixture API.
const ( AcmeServerModeOpenAI AcmeServerMode = "openai" AcmeServerModeStream AcmeServerMode = "stream" )
type AddedModel ¶
type AddedModel struct {
Model ModelFields `json:"model"`
ExistingModelsPreserved bool `json:"existingModelsPreserved"`
}
AddedModel is a model that models.json added to a built-in provider.
type AddedModelOutput ¶
type AddedModelOutput struct {
Result AddedModelResult `json:"result"`
}
AddedModelOutput is the structured output of an added-model eval.
func InspectAddedModel ¶
func InspectAddedModel(ctx context.Context, runtime *coding.ModelRuntime, providerID, modelID string) (AddedModelOutput, error)
InspectAddedModel checks that models.json added modelID to the built-in provider providerID without dropping any of the provider's built-in models.
type AddedModelResult ¶
type AddedModelResult struct {
*AddedModel
Error string `json:"error,omitempty"`
}
AddedModelResult holds either the added model or an error message.
type AgentRun ¶
type AgentRun struct {
Response string
SystemPrompt string
AgentDir string
Workspace string
// Model and PigPath identify the Session's model and the pig binary, for outputs that start more pig processes.
Model PiCodingAgentModelSelection
PigPath string
Messages []map[string]any
ExtensionErrors []json.RawMessage
SuccessfulTools map[string]string
SessionStatistic SessionStatistics
}
AgentRun is what a harness output callback inspects after the last step: the run's answer, the system prompt the model received, the isolated agent and workspace directories, the final Session messages as RPC JSON, and what the extension host reported.
type BlockedPair ¶
type BlockedPair struct {
EvalSet string `json:"evalSet"`
CaseID string `json:"caseId"`
Model string `json:"model"`
RunNumber int `json:"runNumber"`
Reasons []string `json:"reasons"`
}
BlockedPair is a pair excluded from comparison, with the reasons.
type BuiltImage ¶
BuiltImage names one documentation-variant image and its content ID.
type BuiltImages ¶
type BuiltImages struct {
WithoutDocs BuiltImage `json:"without_docs"`
WithDocs BuiltImage `json:"with_docs"`
}
BuiltImages holds the image of each documentation variant.
func BuildImages ¶
func BuildImages(ctx context.Context, command Command, packageRoot, repositoryRoot string) (BuiltImages, error)
BuildImages builds the without_docs and with_docs images from the Dockerfile under packageRoot with repositoryRoot as the build context, then reads each image ID.
func (BuiltImages) Variant ¶
func (images BuiltImages) Variant(variant DocumentationVariant) BuiltImage
Variant returns the image of variant.
type Command ¶
type Command interface {
Execute(ctx context.Context, name string, args []string, capture bool) (status int, stdout string, err error)
}
Command runs one external command and reports its exit status. Capture returns stdout and leaves stderr attached to the caller; otherwise all three streams stay attached.
type CustomTool ¶
type CustomTool struct {
bridge.Tool
// Execute runs the tool; it is called on the harness side with the model's arguments.
Execute func(ctx context.Context, toolCallID string, params map[string]any) (CustomToolResult, error)
}
CustomTool is a tool the harness serves to the agent through the bridge extension (harness.ts customTools).
type CustomToolResult ¶
CustomToolResult is a custom tool's result (AgentToolResult content, details and terminate).
type DiscoveredEvalCase ¶
type DiscoveredEvalCase struct {
File string `json:"file"`
FullName string `json:"fullName"`
EvalSet string `json:"evalSet"`
CaseID string `json:"caseId"`
}
DiscoveredEvalCase is one Vitest eval case named "<eval set> > <case>".
func ParseDiscoveredCases ¶
func ParseDiscoveredCases(value any) ([]DiscoveredEvalCase, error)
ParseDiscoveredCases validates the decoded JSON list of discovered cases and derives each case identity.
type DiscoveredTest ¶
DiscoveredTest is one `vitest list --json` entry.
type DockerContext ¶
type DockerContext struct {
Command Command
PackageRoot string
Images BuiltImages
ArtifactDirectory string
AuthPath string
Provider string
Model string
RunsPerVariant int
}
DockerContext carries what every container run needs.
type DocumentationVariant ¶
type DocumentationVariant string
DocumentationVariant selects whether the Pi documentation section stays in the system prompt.
const ( DocumentationVariantWithoutDocs DocumentationVariant = "without_docs" DocumentationVariantWithDocs DocumentationVariant = "with_docs" )
func ResolveDocumentationVariant ¶
func ResolveDocumentationVariant(value ...string) (DocumentationVariant, error)
ResolveDocumentationVariant validates value, which defaults to PI_EVAL_VARIANT when omitted.
type EntrypointConfig ¶
type EntrypointConfig struct {
// PackageRoot is the evals package directory as discovered files report it.
PackageRoot string
// Runner is the root-only evaluator binary.
Runner string
// Binaries are the executables the sandboxed agent runs; each must exist.
Binaries []string
// AuthSource is the credential file the host mounts.
AuthSource string
// HostAgentDir receives the credentials for the harness.
HostAgentDir string
// ArtifactDir is where reports are written.
ArtifactDir string
Suites Runner
}
EntrypointConfig locates the image's contents.
type EvalCli ¶
type EvalCli struct {
PackageRoot string
RepositoryRoot string
Command Command
Environment map[string]string
Stdout io.Writer
Now func() time.Time
// DocumentationEvals lists the documentation eval files when no file is named; nil walks PackageRoot/evals.
DocumentationEvals func() []string
}
EvalCli is one documentation comparison run: where its package and repository are, how it runs commands, and its process inputs.
func (EvalCli) RunEvalCli ¶
RunEvalCli runs the documentation comparison for args and returns the process exit code: 1 when any pair is blocked, 0 otherwise. A configuration or infrastructure failure is returned as an error.
type EvalCliOptions ¶
type EvalCliOptions struct {
Provider string
Model string
RunsPerVariant int
RequestedFiles []string
DiscoveryArgs []string
}
EvalCliOptions are the parsed command-line options of the documentation comparison.
func ParseEvalCli ¶
func ParseEvalCli(args []string, environment ...map[string]string) (EvalCliOptions, error)
ParseEvalCli parses the arguments after the program name. The model comes from --provider and --model, which must come together, or from PI_PROVIDER and PI_MODEL in environment (the process environment when omitted).
type EvalComparisonFlag ¶
type EvalComparisonFlag string
EvalComparisonFlag marks a notable comparison result.
const ( EvalComparisonFlagNoLift EvalComparisonFlag = "no-lift" EvalComparisonFlagNegativeDelta EvalComparisonFlag = "negative-delta" EvalComparisonFlagControlSaturated EvalComparisonFlag = "control-saturated" EvalComparisonFlagTreatmentSaturated EvalComparisonFlag = "treatment-saturated" EvalComparisonFlagFlaky EvalComparisonFlag = "flaky" )
type EvalComparisonReport ¶
type EvalComparisonReport struct {
SchemaVersion int `json:"schemaVersion"`
ProtocolDigest string `json:"protocolDigest"`
Control DocumentationVariant `json:"control"`
Treatment DocumentationVariant `json:"treatment"`
Comparisons []EvalSetComparison `json:"comparisons"`
BlockedPairs []BlockedPair `json:"blockedPairs"`
OperationalTotals []VariantTotals `json:"operationalTotals"`
}
EvalComparisonReport is the paired comparison of the documentation variants.
func SummarizeEvalObservations ¶
func SummarizeEvalObservations(protocolDigest string, expectedRuns []ExpectedEvalRun, observations []EvalObservation) EvalComparisonReport
SummarizeEvalObservations pairs each case run's control and treatment observations and compares the variants per eval set. A pair is blocked unless the design expects one run of each variant and each produced exactly one scored observation; an eval set with a blocked pair withholds its pass rates.
type EvalMetrics ¶
type EvalMetrics struct {
InputTokens *float64 `json:"inputTokens,omitempty"`
OutputTokens *float64 `json:"outputTokens,omitempty"`
CacheReadTokens *float64 `json:"cacheReadTokens,omitempty"`
CacheWriteTokens *float64 `json:"cacheWriteTokens,omitempty"`
TotalTokens *float64 `json:"totalTokens,omitempty"`
ToolCalls *float64 `json:"toolCalls,omitempty"`
TotalMs *float64 `json:"totalMs,omitempty"`
EstimatedCostUsd *float64 `json:"estimatedCostUsd,omitempty"`
}
EvalMetrics are a run's measured costs. A nil metric was not measured, which is distinct from zero.
type EvalObservation ¶
type EvalObservation struct {
EvalRunIdentity
EvalMetrics
Outcome EvalOutcome `json:"outcome"`
Score *float64 `json:"score,omitempty"`
}
EvalObservation is one observed run. Score is set exactly when Outcome is EvalOutcomeScored.
func ErroredObservation ¶
func ErroredObservation(task EvalTask) EvalObservation
ErroredObservation is the observation of a task that produced no report.
func ReadTaskObservation ¶
func ReadTaskObservation(task EvalTask, reportPath, artifactDirectory string) (EvalObservation, error)
ReadTaskObservation reads the single-case Vitest report of task, persists its Session snapshot under artifactDirectory, and classifies the run. A report that cannot be read, holds another case, or reports another model is an errored observation; only a failure to persist the snapshot is returned as an error.
type EvalOutcome ¶
type EvalOutcome string
EvalOutcome classifies a run.
const ( EvalOutcomeScored EvalOutcome = "scored" EvalOutcomeUnscored EvalOutcome = "unscored" EvalOutcomeSkipped EvalOutcome = "skipped" EvalOutcomePending EvalOutcome = "pending" EvalOutcomeErrored EvalOutcome = "errored" )
func ClassifyCaseStatus ¶
func ClassifyCaseStatus(status ReportCaseStatus) EvalOutcome
ClassifyCaseStatus maps a failed case to errored, a skipped, to-do or disabled case to skipped, and a pending case to pending. It returns "" for any other status.
type EvalProject ¶
type EvalProject string
EvalProject selects the docs or host suites (vitest.evals.config.ts projects).
const ( // EvalProjectDocs runs the *.docs.eval.go suites; each case runs in a container per documentation variant. EvalProjectDocs EvalProject = "docs" // EvalProjectHost runs every other suite on this machine. EvalProjectHost EvalProject = "host" )
type EvalRunIdentity ¶
type EvalRunIdentity struct {
EvalSet string `json:"evalSet"`
CaseID string `json:"caseId"`
Variant DocumentationVariant `json:"variant"`
Model string `json:"model"`
RunNumber int `json:"runNumber"`
}
EvalRunIdentity identifies one run of a case.
type EvalSetComparison ¶
type EvalSetComparison struct {
EvalSet string `json:"evalSet"`
TotalPairs int `json:"totalPairs"`
EligiblePairs int `json:"eligiblePairs"`
BlockedPairs int `json:"blockedPairs"`
ControlPassRate *float64 `json:"controlPassRate"`
TreatmentPassRate *float64 `json:"treatmentPassRate"`
Lift *float64 `json:"lift"`
Flags []EvalComparisonFlag `json:"flags"`
TotalTokens PairedMetricSummary `json:"totalTokens"`
ToolCalls PairedMetricSummary `json:"toolCalls"`
TotalMs PairedMetricSummary `json:"totalMs"`
EstimatedCostUsd PairedMetricSummary `json:"estimatedCostUsd"`
}
EvalSetComparison compares the variants over one eval set. Pass rates and lift are nil unless every pair is eligible.
type EvalTask ¶
type EvalTask struct {
DiscoveredEvalCase
Variant DocumentationVariant `json:"variant"`
Model string `json:"model"`
RunNumber int `json:"runNumber"`
}
EvalTask is one isolated run of a case under one variant, model and repetition.
func CreateTaskPlan ¶
func CreateTaskPlan(cases []DiscoveredEvalCase, model string, runsPerVariant int) ([]EvalTask, error)
CreateTaskPlan creates one task per case, repetition and variant. Odd repetitions run without_docs first and even repetitions with_docs first.
type ExecCommand ¶
type ExecCommand struct{ Dir string }
ExecCommand runs commands with os/exec in Dir.
type ExpectedEvalRun ¶
type ExpectedEvalRun = EvalRunIdentity
ExpectedEvalRun is a run the plan expects.
type ExpectedTool ¶
ExpectedTool is a tool a ToolCallJudge requires. A nil Arguments matches any arguments.
type HarnessRun ¶
type HarnessRun struct {
// Name is the harness name (createHarness name), "pi-coding-agent" unless the options name it.
Name string
Output any
Events []TranscriptEvent
Metadata map[string]any
Usage UsageSummary
TotalMs float64
Artifacts map[string]any
Errors []error
}
HarnessRun is the normalized result of one run.
func RunPiCodingAgent ¶
func RunPiCodingAgent(ctx context.Context, input PiCodingAgentInput, options PiCodingAgentHarnessOptions) (result *HarnessRun, failure error)
RunPiCodingAgent runs input against a fresh PiG Session in an isolated root and normalizes the result. The root, home, agent directory, workspace and Session directory are new for each run and removed afterwards.
type HarnessRunError ¶
type HarnessRunError struct {
Err error
Run *HarnessRun
}
HarnessRunError is a failed run that still measured something (attachHarnessRunToError).
func (*HarnessRunError) Error ¶
func (failure *HarnessRunError) Error() string
func (*HarnessRunError) Unwrap ¶
func (failure *HarnessRunError) Unwrap() error
type Judge ¶
type Judge interface {
Name() string
Assess(output any, calls []ToolCall) (JudgeScore, error)
}
Judge scores a run's output and tool calls.
func StructuredOutputJudge ¶
StructuredOutputJudge scores a JSON object output against expected: strict field equality, an error field fails the run, and extra fields fail it (allowExtras false). The score is the fraction of matching fields.
func ToolCallJudge ¶
func ToolCallJudge(expected ...ExpectedTool) Judge
ToolCallJudge requires every expected tool to be called, in any order, with strictly equal arguments when given; extra calls are allowed.
type JudgeScore ¶
type JudgeScore struct {
Name string `json:"name"`
Score float64 `json:"score"`
Metadata map[string]any `json:"metadata,omitempty"`
}
JudgeScore is one judge's verdict.
type ModelCostFields ¶
type ModelCostFields struct {
Input float64 `json:"input"`
Output float64 `json:"output"`
CacheRead float64 `json:"cacheRead"`
CacheWrite float64 `json:"cacheWrite"`
}
ModelCostFields is a model's per-token pricing.
type ModelFields ¶
type ModelFields struct {
ID string `json:"id"`
Name string `json:"name"`
Provider string `json:"provider"`
Reasoning bool `json:"reasoning"`
Input []string `json:"input"`
Cost ModelCostFields `json:"cost"`
ContextWindow int `json:"contextWindow"`
MaxTokens int `json:"maxTokens"`
}
ModelFields is the model metadata an eval compares.
type OperationalMetricTotal ¶
type OperationalMetricTotal struct {
AvailableRuns int `json:"availableRuns"`
Total *float64 `json:"total"`
}
OperationalMetricTotal sums a metric over the runs that measured it. Total is nil when no run did.
type PairedMetricSummary ¶
type PairedMetricSummary struct {
EligiblePairs int `json:"eligiblePairs"`
ControlMean *float64 `json:"controlMean"`
TreatmentMean *float64 `json:"treatmentMean"`
MeanDelta *float64 `json:"meanDelta"`
}
PairedMetricSummary compares a metric over the pairs that measured it in both variants.
type PiCodingAgentHarnessOptions ¶
type PiCodingAgentHarnessOptions struct {
Name string
Model *PiCodingAgentModelSelection
NoTools string
Tools []string
CustomTools []CustomTool
WorkspaceFiles map[string]string
TransformSystemPrompt func(defaultPrompt string) (string, error)
ExpectedPiDocumentation *bool
Output PiCodingAgentOutput
PigPath string
ExtensionPath string
// contains filtered or unexported fields
}
PiCodingAgentHarnessOptions configures an eval harness. A nil Tools keeps the Session default. PigPath and ExtensionPath name the pig binary and the cmd/pig-eval-extension binary; empty values read PI_EVAL_PIG and PI_EVAL_EXTENSION, and the pig binary falls back to PATH.
func CreatePiDocumentationEvalHarness ¶
func CreatePiDocumentationEvalHarness(options ...PiCodingAgentHarnessOptions) (PiCodingAgentHarnessOptions, error)
CreatePiDocumentationEvalHarness returns the harness options of the documentation variant named by PI_EVAL_VARIANT. It fails outside the isolated container sandbox (PI_EVAL_CONTAINER=1 with a sandbox identity). Tools default to DocumentationEvalTools; the without_docs variant strips the documentation section.
type PiCodingAgentInput ¶
type PiCodingAgentInput []PiCodingAgentStep
PiCodingAgentInput is the steps of one run. A plain string input is PromptInput.
func PromptInput ¶
func PromptInput(content string) PiCodingAgentInput
PromptInput is the input that sends one prompt.
type PiCodingAgentModelSelection ¶
PiCodingAgentModelSelection names the provider and model an eval runs.
func ResolveModelSelection ¶
func ResolveModelSelection(explicitModel *PiCodingAgentModelSelection, environment ...map[string]string) (PiCodingAgentModelSelection, error)
ResolveModelSelection prefers explicitModel and otherwise reads PI_PROVIDER and PI_MODEL from environment, which defaults to the process environment.
type PiCodingAgentOutput ¶
PiCodingAgentOutput computes a run's structured output (PiCodingAgentHarnessWithOutput.output).
type PiCodingAgentStep ¶
type PiCodingAgentStep struct {
Type PiCodingAgentStepType `json:"type"`
Content string `json:"content,omitempty"`
}
PiCodingAgentStep is one step of a PiCodingAgentInput.
type PiCodingAgentStepType ¶
type PiCodingAgentStepType string
PiCodingAgentStepType selects what a step of an input does.
const ( // PiCodingAgentStepPrompt sends Content to the agent. PiCodingAgentStepPrompt PiCodingAgentStepType = "prompt" // PiCodingAgentStepReload reloads the Session's resources and extensions. PiCodingAgentStepReload PiCodingAgentStepType = "reload" )
type ProviderProbe ¶
type ProviderProbe struct {
ValidRequestReceived bool `json:"validRequestReceived"`
Model ModelFields `json:"model"`
Response ProviderProbeResponse `json:"response"`
}
ProviderProbe is a completed probe of a configured provider.
type ProviderProbeResponse ¶
type ProviderProbeResponse struct {
Text string `json:"text"`
StopReason string `json:"stopReason"`
InputTokens int `json:"inputTokens"`
OutputTokens int `json:"outputTokens"`
}
ProviderProbeResponse summarizes the probe completion.
type ProviderRuntimeOutput ¶
type ProviderRuntimeOutput struct {
Result ProviderRuntimeResult `json:"result"`
}
ProviderRuntimeOutput is the structured output of a provider eval.
func InspectProvider ¶
func InspectProvider(ctx context.Context, runtime *coding.ModelRuntime, scenario ProviderScenario) ProviderRuntimeOutput
InspectProvider reloads the runtime's configuration offline, then completes the scenario's context with the configured model. Configuration and lookup failures are structured errors.
type ProviderRuntimeResult ¶
type ProviderRuntimeResult struct {
*ProviderProbe
Error string `json:"error,omitempty"`
}
ProviderRuntimeResult holds either a probe or an error message.
type ProviderScenario ¶
type ProviderScenario struct {
ProviderID string
ModelID string
CreateContext func() ai.Context
Options ai.StreamOptions
ValidRequestReceived func() bool
}
ProviderScenario names the configured model to probe and how to recognize the fixture's probe request.
type ReportCaseStatus ¶
type ReportCaseStatus string
ReportCaseStatus is a Vitest assertion status.
const ( ReportCaseStatusPassed ReportCaseStatus = "passed" ReportCaseStatusFailed ReportCaseStatus = "failed" ReportCaseStatusSkipped ReportCaseStatus = "skipped" ReportCaseStatusPending ReportCaseStatus = "pending" ReportCaseStatusTodo ReportCaseStatus = "todo" ReportCaseStatusDisabled ReportCaseStatus = "disabled" )
type RunFunc ¶
type RunFunc func(ctx context.Context, input PiCodingAgentInput) (*HarnessRun, error)
RunFunc runs input through the suite's harness, records the run for the report and applies the suite's judges (the `run` fixture of vitest-evals).
type Runner ¶
Runner lists and runs suites. PackageRoot is the evals package directory as the container reports it.
func (Runner) List ¶
func (runner Runner) List(selection Selection) []DiscoveredTest
List is `vitest list --json`: the selected cases named "<suite> > <case>" with their absolute file.
type Selection ¶
type Selection struct {
Project EvalProject
// Files are paths relative to the package root; empty selects every file of the project.
Files []string
// NamePattern matches "<suite> <case>" (Vitest -t); nil matches all.
NamePattern *regexp.Regexp
}
Selection narrows the suites and cases a runner lists or runs.
type SessionStatistics ¶
type SessionStatistics struct {
SessionFile string
ToolCalls float64
Input float64
Output float64
CacheRead float64
CacheWrite float64
Total float64
Cost float64
}
SessionStatistics are the Session totals the agent process reports (get_session_stats).
type Suite ¶
type Suite struct {
Name string
File string
Harness func(ctx context.Context, input PiCodingAgentInput) (*HarnessRun, error)
Judges []Judge
// JudgeThreshold is the average score a case needs; nil is Vitest-evals' default of 1. NoJudgeThreshold records
// scores without failing (judgeThreshold: null).
JudgeThreshold *float64
NoJudgeThreshold bool
Cases []Case
// Setup runs before the suite's cases and returns its cleanup (beforeAll/afterAll). The cleanup runs even when Setup
// fails.
Setup func(ctx context.Context) (func(context.Context) error, error)
// BeforeEach runs before every case (beforeEach).
BeforeEach func()
}
Suite is a describeEval: a harness, judges and cases. File is the path under the evals package, ending in DocsEvalSuffix for a documentation suite.
func (Suite) Project ¶
func (suite Suite) Project() EvalProject
Project is the project the suite belongs to.
type ToolCall ¶
type ToolCall struct {
Name string `json:"name"`
Arguments map[string]any `json:"arguments,omitempty"`
Status string `json:"status"`
Result any `json:"result,omitempty"`
Error map[string]any `json:"error,omitempty"`
}
ToolCall is a transcript tool call joined with its result (@vitest-evals/core toolCalls).
func ToolCalls ¶
func ToolCalls(events []TranscriptEvent) []ToolCall
ToolCalls joins each tool_call event with the first tool_result of the same call ID.
type TranscriptEvent ¶
TranscriptEvent is one normalized transcript event.
type UsageSummary ¶
type UsageSummary struct {
Provider string `json:"provider"`
Model string `json:"model"`
InputTokens float64 `json:"inputTokens"`
OutputTokens float64 `json:"outputTokens"`
TotalTokens float64 `json:"totalTokens"`
ToolCalls float64 `json:"toolCalls"`
Metadata map[string]any `json:"metadata"`
}
UsageSummary is a run's model usage.
type VariantTotals ¶
type VariantTotals struct {
Variant DocumentationVariant `json:"variant"`
Runs int `json:"runs"`
InputTokens OperationalMetricTotal `json:"inputTokens"`
OutputTokens OperationalMetricTotal `json:"outputTokens"`
CacheReadTokens OperationalMetricTotal `json:"cacheReadTokens"`
CacheWriteTokens OperationalMetricTotal `json:"cacheWriteTokens"`
TotalTokens OperationalMetricTotal `json:"totalTokens"`
ToolCalls OperationalMetricTotal `json:"toolCalls"`
TotalMs OperationalMetricTotal `json:"totalMs"`
EstimatedCostUsd OperationalMetricTotal `json:"estimatedCostUsd"`
}
VariantTotals are one variant's operational totals over every observed run.
type VitestReport ¶
type VitestReport struct {
NumTotalTests int `json:"numTotalTests"`
NumPassedTests int `json:"numPassedTests"`
NumFailedTests int `json:"numFailedTests"`
NumPendingTests int `json:"numPendingTests"`
NumTodoTests int `json:"numTodoTests"`
StartTime int64 `json:"startTime"`
Success bool `json:"success"`
TestResults []fileResult `json:"testResults"`
}
VitestReport is the JSON reporter's output.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bridge connects the eval harness to the extension that runs inside the agent process under test.
|
Package bridge connects the eval harness to the extension that runs inside the agent process under test. |
|
Package evalsuites holds the eval suites of Pi's packages/evals/evals directory, written against the PiG Session harness.
|
Package evalsuites holds the eval suites of Pi's packages/evals/evals directory, written against the PiG Session harness. |