Documentation
¶
Overview ¶
Package execution: git resource materialization for the per-task workspace.
Git resources let task authors check out a clean copy of a local git repository into the workspace before the agent runs. This is useful when developing skills that live inside the same repo whose code they need to reason about (e.g. azure-sdk-for-rust): rather than hand-crafting fixtures, the task can point at the local clone and get an isolated checkout for each run.
Currently only the "worktree" strategy is supported. It uses `git worktree add --detach` against the local source repo, which is cheap (shares the same .git object store) and does not require network access. Additional strategies (HTTPS clone, ssh, submodules) can be added later behind the same GitResource interface.
Index ¶
- func ResolveWorkDir(workspaceDir, workDir string) (string, error)
- func ShutdownSharedClient(_ context.Context) error
- func UpdateOutcomeUsage(outcome *models.EvaluationOutcome, engine AgentEngine)
- type AgentEngine
- type CopilotClient
- type CopilotEngine
- func (e *CopilotEngine) CopilotClient() CopilotClient
- func (e *CopilotEngine) DeleteSession(ctx context.Context, sessionID string) error
- func (e *CopilotEngine) Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)
- func (e *CopilotEngine) Initialize(ctx context.Context) error
- func (e *CopilotEngine) ListModels(ctx context.Context) ([]copilot.ModelInfo, error)
- func (e *CopilotEngine) SessionUsage(sessionID string) *models.UsageStats
- func (e *CopilotEngine) SetKeepWorkspace(keep bool)
- func (e *CopilotEngine) Shutdown(ctx context.Context) error
- type CopilotEngineBuilder
- type CopilotEngineBuilderOptions
- type CopilotSession
- type ExecutionRequest
- type ExecutionResponse
- type GitResource
- type InstructionFile
- type MessageMode
- type MockEngine
- func (m *MockEngine) Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)
- func (m *MockEngine) Initialize(ctx context.Context) error
- func (m *MockEngine) SessionUsage(sessionID string) *models.UsageStats
- func (m *MockEngine) SetKeepWorkspace(keep bool)
- func (m *MockEngine) Shutdown(ctx context.Context) error
- type ResourceFile
- type SessionEventsCollector
- func (coll *SessionEventsCollector) Done() <-chan struct{}
- func (coll *SessionEventsCollector) ErrorMessage() string
- func (coll *SessionEventsCollector) FirstEvent() <-chan struct{}
- func (coll *SessionEventsCollector) On(event copilot.SessionEvent)
- func (coll *SessionEventsCollector) OutputParts() []string
- func (coll *SessionEventsCollector) SessionEvents() []copilot.SessionEvent
- func (coll *SessionEventsCollector) SetOnSkillInvoked(fn func(SkillInvocation))
- func (coll *SessionEventsCollector) ToolCalls() []models.ToolCall
- type SessionUsageCollector
- type SharedClientOptions
- type SkillInvocation
- type WorkspaceKeeper
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ResolveWorkDir ¶ added in v0.37.0
ResolveWorkDir returns the effective working directory for the agent session. When workDir is empty, the workspace root is returned. Otherwise workDir is joined under workspaceDir and verified not to escape it via path traversal. `..` segments are rejected outright (even if they would resolve to a path still inside the workspace) so callers cannot rely on filesystem normalization to hide intent.
func ShutdownSharedClient ¶ added in v0.37.0
ShutdownSharedClient stops the underlying Copilot SDK process if a shared client was ever constructed. Safe to call multiple times. Should be invoked once from the top-level command after all engines have been Shutdown and all graders have completed.
func UpdateOutcomeUsage ¶
func UpdateOutcomeUsage(outcome *models.EvaluationOutcome, engine AgentEngine)
UpdateOutcomeUsage replaces fallback per-turn usage data in the outcome with authoritative post-shutdown usage data from the engine, then re-aggregates the digest-level usage totals. Call after engine.Shutdown().
Types ¶
type AgentEngine ¶
type AgentEngine interface {
// Initialize sets up the engine
Initialize(ctx context.Context) error
// Execute runs a test with the given stimulus. The caller controls
// cancellation and deadlines through ctx.
Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)
// Shutdown cleans up resources. It is safe to call multiple times;
// subsequent calls after the first are no-ops. After Shutdown returns,
// SessionUsage results include data from session termination events.
Shutdown(ctx context.Context) error
// SessionUsage returns the final usage stats for a session, including
// data from session.shutdown events that fire during Shutdown().
// Returns nil if no usage data is available for the given session.
SessionUsage(sessionID string) *models.UsageStats
}
AgentEngine is the interface for executing test prompts
type CopilotClient ¶
type CopilotClient interface {
// CreateSession maps to [copilot.Client.CreateSession]
CreateSession(ctx context.Context, config *copilot.SessionConfig) (CopilotSession, error)
// GetAuthStatus maps to [copilot.Client.GetAuthStatus]
GetAuthStatus(ctx context.Context) (*copilot.GetAuthStatusResponse, error)
// Start maps to [copilot.Client.Start]
Start(ctx context.Context) error
// Stop maps to [copilot.Client.Stop]
Stop() error
// ResumeSessionWithOptions maps to [copilot.Client.ResumeSessionWithOptions]
ResumeSessionWithOptions(ctx context.Context, sessionID string, config *copilot.ResumeSessionConfig) (CopilotSession, error)
// DeleteSession maps to [copilot.Client.DeleteSession]
DeleteSession(ctx context.Context, sessionID string) error
// ListModels maps to [copilot.Client.ListModels]
ListModels(ctx context.Context) ([]copilot.ModelInfo, error)
}
CopilotClient is just an interface over *copilot.Client
func SharedClient ¶ added in v0.37.0
func SharedClient(opts SharedClientOptions) CopilotClient
SharedClient returns a lazily-constructed, process-wide CopilotClient.
Rationale (#135 R2): the embedded Copilot CLI process is expensive to spawn / tear down. Now that all per-call state (workdir, model, MCP servers, skill dirs, system message) is provided to CreateSession and ResumeSessionWithOptions, a single SDK client can serve compatible CopilotEngine instances and graders within a `waza run`. Startup-only CLIArgs (for example --model) are part of the compatibility key.
The client is started lazily on first use by CopilotEngine.Initialize (or by an explicit [Start] caller) and is stopped exactly once via ShutdownSharedClient. Engines built on top of the shared client must not call client.Stop() themselves.
Tests that need an isolated client can either construct one directly with [newCopilotClient] (package-private) or pass a custom NewCopilotClient factory via CopilotEngineBuilderOptions.
type CopilotEngine ¶
type CopilotEngine struct {
// contains filtered or unexported fields
}
CopilotEngine integrates with GitHub Copilot SDK
func (*CopilotEngine) CopilotClient ¶ added in v0.37.0
func (e *CopilotEngine) CopilotClient() CopilotClient
CopilotClient returns the underlying Copilot SDK client backing this engine. Graders that need to spin up their own grading session (e.g. prompt graders) can use this to avoid spawning a fresh SDK process per invocation. Callers must NOT call Stop() on the returned client.
func (*CopilotEngine) DeleteSession ¶ added in v0.37.0
func (e *CopilotEngine) DeleteSession(ctx context.Context, sessionID string) error
DeleteSession removes a persistent session created via Execute (with EphemeralSession=false) and stops tracking it for shutdown cleanup. It is used by callers that own a long-lived session, such as the responder, to tear it down promptly rather than waiting for engine Shutdown.
func (*CopilotEngine) Execute ¶
func (e *CopilotEngine) Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)
Execute runs a test with Copilot SDK
func (*CopilotEngine) Initialize ¶
func (e *CopilotEngine) Initialize(ctx context.Context) error
Initialize sets up the Copilot client
func (*CopilotEngine) ListModels ¶ added in v0.28.0
ListModels returns the available models from the Copilot backend.
func (*CopilotEngine) SessionUsage ¶
func (e *CopilotEngine) SessionUsage(sessionID string) *models.UsageStats
SessionUsage returns the final usage stats for a session. Call after Shutdown() to get data from session.shutdown events (ModelMetrics, TotalPremiumRequests). When BYOK was active for this session, the returned stats include sanitized provider metadata so downstream consumers can label PremiumRequests accurately (custom endpoint vs Copilot MaaS).
func (*CopilotEngine) SetKeepWorkspace ¶ added in v0.29.0
func (e *CopilotEngine) SetKeepWorkspace(keep bool)
SetKeepWorkspace enables or disables workspace preservation on shutdown.
type CopilotEngineBuilder ¶
type CopilotEngineBuilder struct {
// contains filtered or unexported fields
}
CopilotEngineBuilder builds a CopilotEngine with options
func NewCopilotEngineBuilder ¶
func NewCopilotEngineBuilder(defaultModelID string, options *CopilotEngineBuilderOptions) *CopilotEngineBuilder
NewCopilotEngineBuilder creates a builder for CopilotEngine
- defaultModelID - used if no model ID is specified in session creation. Can be blank, which means the copilot CLI will choose its own fallback model.
When `options.NewCopilotClient` is nil (the production path), the engine is wired to the process-wide shared client returned by SharedClient. The caller MUST invoke ShutdownSharedClient once at the top level after every engine has been Shutdown — engines built on top of the shared client will not stop it themselves. See docs/design/135-improve-concurrency.md (R2).
When a `NewCopilotClient` factory is provided (test path), the engine constructs its own client and stops it during Shutdown as before.
func (*CopilotEngineBuilder) Build ¶
func (b *CopilotEngineBuilder) Build() *CopilotEngine
type CopilotEngineBuilderOptions ¶
type CopilotEngineBuilderOptions struct {
NewCopilotClient func(clientOptions *copilot.ClientOptions) CopilotClient
}
type CopilotSession ¶
type CopilotSession interface {
// Disconnect maps to [copilot.Session.Disconnect]. It closes the session and releases resources, however it
// doesn't delete data and the session is still resumable until deleted via [copilot.Client.DeleteSession].
Disconnect() error
// On maps to [copilot.Session.On]
On(handler copilot.SessionEventHandler) func()
// SendAndWait maps to [copilot.Session.SendAndWait]
SendAndWait(ctx context.Context, options copilot.MessageOptions) (*copilot.SessionEvent, error)
// SessionID returns [copilot.Session.SessionID]
SessionID() string
}
CopilotSession is just an interface over *copilot.Session
type ExecutionRequest ¶
type ExecutionRequest struct {
ModelID string
Message string
Context map[string]any
Resources []ResourceFile
GitResources []models.GitResource
WorkDir string
Instructions []InstructionFile
Tools []copilot.Tool
MessageMode MessageMode
Streaming bool
// FirstEventTimeout bounds how long Execute waits for the FIRST session
// event before treating the run as a session-start hang and aborting it
// with a distinct error. It is intentionally much shorter than the overall
// turn deadline (the ctx passed to Execute), which must be sized for the
// slowest legitimate full turn and therefore cannot also catch a
// no-first-turn wedge quickly. Zero disables the check (the prior behavior).
FirstEventTimeout time.Duration
// EphemeralSession keeps one-off sessions out of engine shutdown tracking.
// New ephemeral sessions are deleted at the end of Execute. Resumed
// ephemeral sessions are only disconnected because the caller does not own
// the original session data.
EphemeralSession bool
// SkipWorkspaceCapture avoids post-run workspace snapshots for callers that
// only need model/tool output, such as prompt graders.
SkipWorkspaceCapture bool
SessionID string
WorkspaceDir string // Reuse an existing workspace directory (for follow-up prompts)
SkillName string
// TaskName and TaskDescription carry test-case metadata so mock engines can
// echo them, enabling output_contains expectations that reference task-level
// concepts (e.g., "recursive") without a real model.
TaskName string
TaskDescription string
SourceDir string // used when looking for workspace items via relative path, like skills.
SkillPaths []string // Directories to search for skills
NoSkills bool // When true, skip all skill loading
// SuppressSkillBody prevents full target skill content from being appended
// while still allowing skill discovery and compact summaries.
SuppressSkillBody bool
// MCPServers configures MCP servers for the session. Keys are server names,
// values follow the copilot SDK MCPServerConfig format (type/command/args).
MCPServers map[string]copilot.MCPServerConfig
// PermissionHandler called when the copilot SDK wants to determine if a tool can be used.
// Default: allows all tools.
PermissionHandler copilot.PermissionHandlerFunc
// CancelOnSkillInvocation, when true, causes the execution context to be
// canceled as soon as a SkillInvoked event is received. This allows trigger
// tests to terminate early once the skill invocation they care about has been
// detected, avoiding unnecessary wait for the agent to finish its full turn.
CancelOnSkillInvocation bool
}
ExecutionRequest represents a test execution request
type ExecutionResponse ¶
type ExecutionResponse struct {
FinalOutput string
Events []agentevent.Event
ModelID string
SkillInvocations []SkillInvocation
DurationMs int64
ToolCalls []models.ToolCall
ErrorMsg string
Success bool
WorkspaceDir string // Path to workspace directory (for file grading)
WorkspaceFiles map[string][]byte // Post-execution workspace file contents captured before session disconnect
SessionID string // Copilot session ID
Usage *models.UsageStats
}
ExecutionResponse represents the result of an execution.
Events carries engine-neutral agentevent.Event values rather than a concrete SDK type so additional agent engines (Claude Code, Codex, generic CLI) can plug in without leaking their native event types through this API. The Copilot engine wraps each SDK event into an Event via internal/copilotevents.FromSDK at the engine boundary; consumers that still need SDK-typed access unwrap via copilotevents.ToSDK or copilotevents.AsSDKEvent.
func (*ExecutionResponse) ContainsText ¶
func (r *ExecutionResponse) ContainsText(text string) bool
ContainsText checks if output contains text (case-insensitive)
func (*ExecutionResponse) ExtractMessages ¶
func (r *ExecutionResponse) ExtractMessages() []string
ExtractMessages gets all non-empty assistant messages from events.
Copilot SDK events are unwrapped through copilotevents.AsSDKEvent so the existing Content helper can extract structured fields. Events produced by other engines fall back to agentevent.Event.Text, which delegates to the engine payload's optional agentevent.TextProvider implementation. This keeps ExtractMessages truly engine-neutral: a non-Copilot engine that emits KindAssistantMessage with a TextProvider payload will surface here instead of being silently dropped.
type GitResource ¶ added in v0.37.0
type GitResource interface {
// Cleanup removes the materialized resource (e.g. `git worktree
// remove`). It is safe to call multiple times; subsequent calls are
// no-ops.
Cleanup(ctx context.Context) error
}
GitResource represents a materialized git resource that must be cleaned up when the engine shuts down. Implementations are returned from CloneGitResources.
func CloneGitResources ¶ added in v0.37.0
func CloneGitResources(ctx context.Context, resources []models.GitResource, workspaceDir string) ([]GitResource, error)
CloneGitResources materializes each git resource into workspaceDir and returns handles that the caller is responsible for cleaning up via Cleanup(). If any resource fails to materialize, previously created resources in this call are cleaned up before returning the error.
The caller must clean up resources before removing workspaceDir, because some strategies (worktree) keep bookkeeping inside the source repository's .git/worktrees/ that needs `git worktree remove` to invalidate cleanly.
type InstructionFile ¶ added in v0.33.0
InstructionFile represents a file whose content should be applied as agent instructions.
type MessageMode ¶ added in v0.33.0
type MessageMode string
MessageMode controls how a prompt is submitted to an existing session.
const ( // MessageModeEnqueue sends the prompt using the Copilot SDK's enqueue mode. MessageModeEnqueue MessageMode = "enqueue" )
type MockEngine ¶
type MockEngine struct {
// contains filtered or unexported fields
}
MockEngine is a simple mock implementation for testing
func NewMockEngine ¶
func NewMockEngine(modelID string) *MockEngine
NewMockEngine creates a new mock engine
func (*MockEngine) Execute ¶
func (m *MockEngine) Execute(ctx context.Context, req *ExecutionRequest) (*ExecutionResponse, error)
func (*MockEngine) Initialize ¶
func (m *MockEngine) Initialize(ctx context.Context) error
func (*MockEngine) SessionUsage ¶
func (m *MockEngine) SessionUsage(sessionID string) *models.UsageStats
func (*MockEngine) SetKeepWorkspace ¶ added in v0.29.0
func (m *MockEngine) SetKeepWorkspace(keep bool)
SetKeepWorkspace enables or disables workspace preservation on shutdown.
type ResourceFile ¶
ResourceFile represents a file resource
type SessionEventsCollector ¶
type SessionEventsCollector struct {
// SkillInvocations is a chronological list of skills invoked during the session
SkillInvocations []SkillInvocation
// contains filtered or unexported fields
}
func NewSessionEventsCollector ¶
func NewSessionEventsCollector() *SessionEventsCollector
NewSessionEventsCollector creates a new SessionEvents.
func (*SessionEventsCollector) Done ¶
func (coll *SessionEventsCollector) Done() <-chan struct{}
Done returns the channel that is closed when the session completes.
func (*SessionEventsCollector) ErrorMessage ¶
func (coll *SessionEventsCollector) ErrorMessage() string
ErrorMessage returns the error message, if any.
func (*SessionEventsCollector) FirstEvent ¶ added in v0.37.0
func (coll *SessionEventsCollector) FirstEvent() <-chan struct{}
FirstEvent returns a channel that is closed when the FIRST session event of any type is received. A healthy session emits events promptly once the agent starts its first turn; a session-start hang (the embedded engine launches but never produces a turn) emits nothing at all. Callers use this to arm a short "time to first event" deadline distinct from the overall turn timeout, so a no-first-turn wedge is caught in seconds instead of running out the full (necessarily large) turn budget.
func (*SessionEventsCollector) On ¶
func (coll *SessionEventsCollector) On(event copilot.SessionEvent)
On is a callback, intended to be passed to copilot.Session.On to receive events in real-time.
func (*SessionEventsCollector) OutputParts ¶
func (coll *SessionEventsCollector) OutputParts() []string
OutputParts returns the collected output text parts.
func (*SessionEventsCollector) SessionEvents ¶
func (coll *SessionEventsCollector) SessionEvents() []copilot.SessionEvent
SessionEvents returns the collected session events.
func (*SessionEventsCollector) SetOnSkillInvoked ¶ added in v0.28.0
func (coll *SessionEventsCollector) SetOnSkillInvoked(fn func(SkillInvocation))
SetOnSkillInvoked registers a callback that fires every time a SkillInvoked event is received. The callback runs synchronously inside On(), so it can safely cancel a context to abort an in-flight SendAndWait.
func (*SessionEventsCollector) ToolCalls ¶
func (coll *SessionEventsCollector) ToolCalls() []models.ToolCall
ToolCalls goes through the list of session events and correlates tool starts with Success. The resulting tool calls are not cached - if you're going to use it repeatedly you should store it locally.
type SessionUsageCollector ¶
type SessionUsageCollector struct {
// contains filtered or unexported fields
}
SessionUsageCollector tracks token and premium request usage from Copilot SDK session events. Its On method implements copilot.SessionEventHandler and should be registered via session.On(collector.On).
Usage data arrives through two channels:
- Per-turn events (AssistantUsage) — accumulated as a fallback.
- Session termination events (SessionIdle, SessionShutdown) — authoritative totals that override per-turn data when available.
func NewSessionUsageCollector ¶
func NewSessionUsageCollector() *SessionUsageCollector
func (*SessionUsageCollector) On ¶
func (s *SessionUsageCollector) On(event copilot.SessionEvent)
On handles a single session event, extracting any usage data it carries. Pass this method to session.On as a copilot.SessionEventHandler.
func (*SessionUsageCollector) UsageStats ¶
func (s *SessionUsageCollector) UsageStats() *models.UsageStats
UsageStats returns the collected usage statistics. Returns nil if no usage data was collected. Session-level data (from SessionIdle/SessionShutdown) is preferred as the authoritative source; per-turn accumulated data (from AssistantUsage) is used as fallback.
type SharedClientOptions ¶ added in v0.37.0
type SharedClientOptions struct {
// "error" when blank.
LogLevel string
// same CLIArgs share one process; calls with different CLIArgs get separate
// processes because CLIArgs are startup-only.
CLIArgs []string
}
SharedClientOptions configures a lazily-constructed process-wide Copilot SDK client returned by SharedClient. Clients are shared by CLIArgs key; within each key, only the first call wins and subsequent calls receive the already-built client regardless of options.
type SkillInvocation ¶
type WorkspaceKeeper ¶ added in v0.29.0
type WorkspaceKeeper interface {
SetKeepWorkspace(keep bool)
}
WorkspaceKeeper is an optional interface that engines can implement to support preserving temp workspaces after execution (for debugging).