execution

package
v0.38.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 17, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package execution: git resource materialization for the per-task workspace.

Git resources let task authors check out a clean copy of a local git repository into the workspace before the agent runs. This is useful when developing skills that live inside the same repo whose code they need to reason about (e.g. azure-sdk-for-rust): rather than hand-crafting fixtures, the task can point at the local clone and get an isolated checkout for each run.

Currently only the "worktree" strategy is supported. It uses `git worktree add --detach` against the local source repo, which is cheap (shares the same .git object store) and does not require network access. Additional strategies (HTTPS clone, ssh, submodules) can be added later behind the same GitResource interface.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ResolveWorkDir added in v0.37.0

func ResolveWorkDir(workspaceDir, workDir string) (string, error)

ResolveWorkDir returns the effective working directory for the agent session. When workDir is empty, the workspace root is returned. Otherwise workDir is joined under workspaceDir and verified not to escape it via path traversal. `..` segments are rejected outright (even if they would resolve to a path still inside the workspace) so callers cannot rely on filesystem normalization to hide intent.

func ShutdownSharedClient added in v0.37.0

func ShutdownSharedClient(_ context.Context) error

ShutdownSharedClient stops the underlying Copilot SDK process if a shared client was ever constructed. Safe to call multiple times. Should be invoked once from the top-level command after all engines have been Shutdown and all graders have completed.

func UpdateOutcomeUsage

func UpdateOutcomeUsage(outcome *models.EvaluationOutcome, engine AgentEngine)

UpdateOutcomeUsage replaces fallback per-turn usage data in the outcome with authoritative post-shutdown usage data from the engine, then re-aggregates the digest-level usage totals. Call after engine.Shutdown().

Types

type AgentEngine

type AgentEngine interface {
	// Initialize sets up the engine
	Initialize(ctx context.Context) error

	// Execute runs a test with the given stimulus. The caller controls
	// cancellation and deadlines through ctx.
	Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)

	// Shutdown cleans up resources. It is safe to call multiple times;
	// subsequent calls after the first are no-ops. After Shutdown returns,
	// SessionUsage results include data from session termination events.
	Shutdown(ctx context.Context) error

	// SessionUsage returns the final usage stats for a session, including
	// data from session.shutdown events that fire during Shutdown().
	// Returns nil if no usage data is available for the given session.
	SessionUsage(sessionID string) *models.UsageStats
}

AgentEngine is the interface for executing test prompts

type CopilotClient

type CopilotClient interface {
	// CreateSession maps to [copilot.Client.CreateSession]
	CreateSession(ctx context.Context, config *copilot.SessionConfig) (CopilotSession, error)

	// GetAuthStatus maps to [copilot.Client.GetAuthStatus]
	GetAuthStatus(ctx context.Context) (*copilot.GetAuthStatusResponse, error)

	// Start maps to [copilot.Client.Start]
	Start(ctx context.Context) error

	// Stop maps to [copilot.Client.Stop]
	Stop() error

	// ResumeSessionWithOptions maps to [copilot.Client.ResumeSessionWithOptions]
	ResumeSessionWithOptions(ctx context.Context, sessionID string, config *copilot.ResumeSessionConfig) (CopilotSession, error)

	// DeleteSession maps to [copilot.Client.DeleteSession]
	DeleteSession(ctx context.Context, sessionID string) error

	// ListModels maps to [copilot.Client.ListModels]
	ListModels(ctx context.Context) ([]copilot.ModelInfo, error)
}

CopilotClient is just an interface over *copilot.Client

func SharedClient added in v0.37.0

func SharedClient(opts SharedClientOptions) CopilotClient

SharedClient returns a lazily-constructed, process-wide CopilotClient.

Rationale (#135 R2): the embedded Copilot CLI process is expensive to spawn / tear down. Now that all per-call state (workdir, model, MCP servers, skill dirs, system message) is provided to CreateSession and ResumeSessionWithOptions, a single SDK client can serve compatible CopilotEngine instances and graders within a `waza run`. Startup-only CLIArgs (for example --model) are part of the compatibility key.

The client is started lazily on first use by CopilotEngine.Initialize (or by an explicit [Start] caller) and is stopped exactly once via ShutdownSharedClient. Engines built on top of the shared client must not call client.Stop() themselves.

Tests that need an isolated client can either construct one directly with [newCopilotClient] (package-private) or pass a custom NewCopilotClient factory via CopilotEngineBuilderOptions.

type CopilotEngine

type CopilotEngine struct {
	// contains filtered or unexported fields
}

CopilotEngine integrates with GitHub Copilot SDK

func (*CopilotEngine) CopilotClient added in v0.37.0

func (e *CopilotEngine) CopilotClient() CopilotClient

CopilotClient returns the underlying Copilot SDK client backing this engine. Graders that need to spin up their own grading session (e.g. prompt graders) can use this to avoid spawning a fresh SDK process per invocation. Callers must NOT call Stop() on the returned client.

func (*CopilotEngine) DeleteSession added in v0.37.0

func (e *CopilotEngine) DeleteSession(ctx context.Context, sessionID string) error

DeleteSession removes a persistent session created via Execute (with EphemeralSession=false) and stops tracking it for shutdown cleanup. It is used by callers that own a long-lived session, such as the responder, to tear it down promptly rather than waiting for engine Shutdown.

func (*CopilotEngine) Execute

Execute runs a test with Copilot SDK

func (*CopilotEngine) Initialize

func (e *CopilotEngine) Initialize(ctx context.Context) error

Initialize sets up the Copilot client

func (*CopilotEngine) ListModels added in v0.28.0

func (e *CopilotEngine) ListModels(ctx context.Context) ([]copilot.ModelInfo, error)

ListModels returns the available models from the Copilot backend.

func (*CopilotEngine) SessionUsage

func (e *CopilotEngine) SessionUsage(sessionID string) *models.UsageStats

SessionUsage returns the final usage stats for a session. Call after Shutdown() to get data from session.shutdown events (ModelMetrics, TotalPremiumRequests). When BYOK was active for this session, the returned stats include sanitized provider metadata so downstream consumers can label PremiumRequests accurately (custom endpoint vs Copilot MaaS).

func (*CopilotEngine) SetKeepWorkspace added in v0.29.0

func (e *CopilotEngine) SetKeepWorkspace(keep bool)

SetKeepWorkspace enables or disables workspace preservation on shutdown.

func (*CopilotEngine) Shutdown

func (e *CopilotEngine) Shutdown(ctx context.Context) error

Shutdown cleans up resources, deleting session and workspace data. It is safe to call multiple times; subsequent calls after the first are no-ops that return the original error.

type CopilotEngineBuilder

type CopilotEngineBuilder struct {
	// contains filtered or unexported fields
}

CopilotEngineBuilder builds a CopilotEngine with options

func NewCopilotEngineBuilder

func NewCopilotEngineBuilder(defaultModelID string, options *CopilotEngineBuilderOptions) *CopilotEngineBuilder

NewCopilotEngineBuilder creates a builder for CopilotEngine

  • defaultModelID - used if no model ID is specified in session creation. Can be blank, which means the copilot CLI will choose its own fallback model.

When `options.NewCopilotClient` is nil (the production path), the engine is wired to the process-wide shared client returned by SharedClient. The caller MUST invoke ShutdownSharedClient once at the top level after every engine has been Shutdown — engines built on top of the shared client will not stop it themselves. See docs/design/135-improve-concurrency.md (R2).

When a `NewCopilotClient` factory is provided (test path), the engine constructs its own client and stops it during Shutdown as before.

func (*CopilotEngineBuilder) Build

func (b *CopilotEngineBuilder) Build() *CopilotEngine

type CopilotEngineBuilderOptions

type CopilotEngineBuilderOptions struct {
	NewCopilotClient func(clientOptions *copilot.ClientOptions) CopilotClient
}

type CopilotSession

type CopilotSession interface {
	// Disconnect maps to [copilot.Session.Disconnect]. It closes the session and releases resources, however it
	// doesn't delete data and the session is still resumable until deleted via [copilot.Client.DeleteSession].
	Disconnect() error

	// On maps to [copilot.Session.On]
	On(handler copilot.SessionEventHandler) func()

	// SendAndWait maps to [copilot.Session.SendAndWait]
	SendAndWait(ctx context.Context, options copilot.MessageOptions) (*copilot.SessionEvent, error)

	// SessionID returns [copilot.Session.SessionID]
	SessionID() string
}

CopilotSession is just an interface over *copilot.Session

type ExecutionRequest

type ExecutionRequest struct {
	ModelID      string
	Message      string
	Context      map[string]any
	Resources    []ResourceFile
	GitResources []models.GitResource
	WorkDir      string
	Instructions []InstructionFile
	Tools        []copilot.Tool

	MessageMode MessageMode
	Streaming   bool

	// FirstEventTimeout bounds how long Execute waits for the FIRST session
	// event before treating the run as a session-start hang and aborting it
	// with a distinct error. It is intentionally much shorter than the overall
	// turn deadline (the ctx passed to Execute), which must be sized for the
	// slowest legitimate full turn and therefore cannot also catch a
	// no-first-turn wedge quickly. Zero disables the check (the prior behavior).
	FirstEventTimeout time.Duration

	// EphemeralSession keeps one-off sessions out of engine shutdown tracking.
	// New ephemeral sessions are deleted at the end of Execute. Resumed
	// ephemeral sessions are only disconnected because the caller does not own
	// the original session data.
	EphemeralSession bool

	// SkipWorkspaceCapture avoids post-run workspace snapshots for callers that
	// only need model/tool output, such as prompt graders.
	SkipWorkspaceCapture bool

	SessionID    string
	WorkspaceDir string // Reuse an existing workspace directory (for follow-up prompts)
	SkillName    string

	// TaskName and TaskDescription carry test-case metadata so mock engines can
	// echo them, enabling output_contains expectations that reference task-level
	// concepts (e.g., "recursive") without a real model.
	TaskName        string
	TaskDescription string

	SourceDir  string   // used when looking for workspace items via relative path, like skills.
	SkillPaths []string // Directories to search for skills
	NoSkills   bool     // When true, skip all skill loading
	// SuppressSkillBody prevents full target skill content from being appended
	// while still allowing skill discovery and compact summaries.
	SuppressSkillBody bool

	// MCPServers configures MCP servers for the session. Keys are server names,
	// values follow the copilot SDK MCPServerConfig format (type/command/args).
	MCPServers map[string]copilot.MCPServerConfig

	// PermissionHandler called when the copilot SDK wants to determine if a tool can be used.
	// Default: allows all tools.
	PermissionHandler copilot.PermissionHandlerFunc

	// CancelOnSkillInvocation, when true, causes the execution context to be
	// canceled as soon as a SkillInvoked event is received. This allows trigger
	// tests to terminate early once the skill invocation they care about has been
	// detected, avoiding unnecessary wait for the agent to finish its full turn.
	CancelOnSkillInvocation bool
}

ExecutionRequest represents a test execution request

type ExecutionResponse

type ExecutionResponse struct {
	FinalOutput      string
	Events           []agentevent.Event
	ModelID          string
	SkillInvocations []SkillInvocation
	DurationMs       int64
	ToolCalls        []models.ToolCall
	ErrorMsg         string
	Success          bool
	WorkspaceDir     string            // Path to workspace directory (for file grading)
	WorkspaceFiles   map[string][]byte // Post-execution workspace file contents captured before session disconnect
	SessionID        string            // Copilot session ID
	Usage            *models.UsageStats
}

ExecutionResponse represents the result of an execution.

Events carries engine-neutral agentevent.Event values rather than a concrete SDK type so additional agent engines (Claude Code, Codex, generic CLI) can plug in without leaking their native event types through this API. The Copilot engine wraps each SDK event into an Event via internal/copilotevents.FromSDK at the engine boundary; consumers that still need SDK-typed access unwrap via copilotevents.ToSDK or copilotevents.AsSDKEvent.

func (*ExecutionResponse) ContainsText

func (r *ExecutionResponse) ContainsText(text string) bool

ContainsText checks if output contains text (case-insensitive)

func (*ExecutionResponse) ExtractMessages

func (r *ExecutionResponse) ExtractMessages() []string

ExtractMessages gets all non-empty assistant messages from events.

Copilot SDK events are unwrapped through copilotevents.AsSDKEvent so the existing Content helper can extract structured fields. Events produced by other engines fall back to agentevent.Event.Text, which delegates to the engine payload's optional agentevent.TextProvider implementation. This keeps ExtractMessages truly engine-neutral: a non-Copilot engine that emits KindAssistantMessage with a TextProvider payload will surface here instead of being silently dropped.

type GitResource added in v0.37.0

type GitResource interface {
	// Cleanup removes the materialized resource (e.g. `git worktree
	// remove`). It is safe to call multiple times; subsequent calls are
	// no-ops.
	Cleanup(ctx context.Context) error
}

GitResource represents a materialized git resource that must be cleaned up when the engine shuts down. Implementations are returned from CloneGitResources.

func CloneGitResources added in v0.37.0

func CloneGitResources(ctx context.Context, resources []models.GitResource, workspaceDir string) ([]GitResource, error)

CloneGitResources materializes each git resource into workspaceDir and returns handles that the caller is responsible for cleaning up via Cleanup(). If any resource fails to materialize, previously created resources in this call are cleaned up before returning the error.

The caller must clean up resources before removing workspaceDir, because some strategies (worktree) keep bookkeeping inside the source repository's .git/worktrees/ that needs `git worktree remove` to invalidate cleanly.

type InstructionFile added in v0.33.0

type InstructionFile struct {
	Path    string
	Content []byte
}

InstructionFile represents a file whose content should be applied as agent instructions.

type MessageMode added in v0.33.0

type MessageMode string

MessageMode controls how a prompt is submitted to an existing session.

const (
	// MessageModeEnqueue sends the prompt using the Copilot SDK's enqueue mode.
	MessageModeEnqueue MessageMode = "enqueue"
)

type MockEngine

type MockEngine struct {
	// contains filtered or unexported fields
}

MockEngine is a simple mock implementation for testing

func NewMockEngine

func NewMockEngine(modelID string) *MockEngine

NewMockEngine creates a new mock engine

func (*MockEngine) Execute

func (*MockEngine) Initialize

func (m *MockEngine) Initialize(ctx context.Context) error

func (*MockEngine) SessionUsage

func (m *MockEngine) SessionUsage(sessionID string) *models.UsageStats

func (*MockEngine) SetKeepWorkspace added in v0.29.0

func (m *MockEngine) SetKeepWorkspace(keep bool)

SetKeepWorkspace enables or disables workspace preservation on shutdown.

func (*MockEngine) Shutdown

func (m *MockEngine) Shutdown(ctx context.Context) error

type ResourceFile

type ResourceFile struct {
	Path    string
	Content []byte
}

ResourceFile represents a file resource

type SessionEventsCollector

type SessionEventsCollector struct {
	// SkillInvocations is a chronological list of skills invoked during the session
	SkillInvocations []SkillInvocation
	// contains filtered or unexported fields
}

func NewSessionEventsCollector

func NewSessionEventsCollector() *SessionEventsCollector

NewSessionEventsCollector creates a new SessionEvents.

func (*SessionEventsCollector) Done

func (coll *SessionEventsCollector) Done() <-chan struct{}

Done returns the channel that is closed when the session completes.

func (*SessionEventsCollector) ErrorMessage

func (coll *SessionEventsCollector) ErrorMessage() string

ErrorMessage returns the error message, if any.

func (*SessionEventsCollector) FirstEvent added in v0.37.0

func (coll *SessionEventsCollector) FirstEvent() <-chan struct{}

FirstEvent returns a channel that is closed when the FIRST session event of any type is received. A healthy session emits events promptly once the agent starts its first turn; a session-start hang (the embedded engine launches but never produces a turn) emits nothing at all. Callers use this to arm a short "time to first event" deadline distinct from the overall turn timeout, so a no-first-turn wedge is caught in seconds instead of running out the full (necessarily large) turn budget.

func (*SessionEventsCollector) On

On is a callback, intended to be passed to copilot.Session.On to receive events in real-time.

func (*SessionEventsCollector) OutputParts

func (coll *SessionEventsCollector) OutputParts() []string

OutputParts returns the collected output text parts.

func (*SessionEventsCollector) SessionEvents

func (coll *SessionEventsCollector) SessionEvents() []copilot.SessionEvent

SessionEvents returns the collected session events.

func (*SessionEventsCollector) SetOnSkillInvoked added in v0.28.0

func (coll *SessionEventsCollector) SetOnSkillInvoked(fn func(SkillInvocation))

SetOnSkillInvoked registers a callback that fires every time a SkillInvoked event is received. The callback runs synchronously inside On(), so it can safely cancel a context to abort an in-flight SendAndWait.

func (*SessionEventsCollector) ToolCalls

func (coll *SessionEventsCollector) ToolCalls() []models.ToolCall

ToolCalls goes through the list of session events and correlates tool starts with Success. The resulting tool calls are not cached - if you're going to use it repeatedly you should store it locally.

type SessionUsageCollector

type SessionUsageCollector struct {
	// contains filtered or unexported fields
}

SessionUsageCollector tracks token and premium request usage from Copilot SDK session events. Its On method implements copilot.SessionEventHandler and should be registered via session.On(collector.On).

Usage data arrives through two channels:

  • Per-turn events (AssistantUsage) — accumulated as a fallback.
  • Session termination events (SessionIdle, SessionShutdown) — authoritative totals that override per-turn data when available.

func NewSessionUsageCollector

func NewSessionUsageCollector() *SessionUsageCollector

func (*SessionUsageCollector) On

On handles a single session event, extracting any usage data it carries. Pass this method to session.On as a copilot.SessionEventHandler.

func (*SessionUsageCollector) UsageStats

func (s *SessionUsageCollector) UsageStats() *models.UsageStats

UsageStats returns the collected usage statistics. Returns nil if no usage data was collected. Session-level data (from SessionIdle/SessionShutdown) is preferred as the authoritative source; per-turn accumulated data (from AssistantUsage) is used as fallback.

type SharedClientOptions added in v0.37.0

type SharedClientOptions struct {
	// LogLevel passed through to the underlying copilot.Client. Defaults to
	// "error" when blank.
	LogLevel string
	// CLIArgs passed through to the underlying copilot.Client. Calls with the
	// same CLIArgs share one process; calls with different CLIArgs get separate
	// processes because CLIArgs are startup-only.
	CLIArgs []string
}

SharedClientOptions configures a lazily-constructed process-wide Copilot SDK client returned by SharedClient. Clients are shared by CLIArgs key; within each key, only the first call wins and subsequent calls receive the already-built client regardless of options.

type SkillInvocation

type SkillInvocation struct {
	// Name of the invoked skill
	Name string
	// Path of the invoked SKILL.md
	Path string
}

type WorkspaceKeeper added in v0.29.0

type WorkspaceKeeper interface {
	SetKeepWorkspace(keep bool)
}

WorkspaceKeeper is an optional interface that engines can implement to support preserving temp workspaces after execution (for debugging).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL