computer

package
v0.220.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 30, 2026 License: Apache-2.0 Imports: 36 Imported by: 0

README

computer

What - the computer bounded context: reading the accessibility tree, pressing labelled controls, capturing screenshots, moving the pointer and typing. Why - desktop control needs platform-native calls, coordinate scaling and a rate limit, none of which belong in the agent loop. How - domain/ defines the accessibility and observation contracts, and infrastructure/ provides the platform bridges and the screenshot stream.

How it plugs in

  • NewTools(...) builds Computer, GetLatestFrame, RecordStart and RecordStop from the manifests in tools/, and the container registers them into the tools registry.
  • The accessibility provider is macOS-only today. Other platforms return an unsupported error and the tool falls back to screenshot guidance, so a native failure never takes down the CLI.
  • PublishedEvent maps this context's chat events to the AG-UI CUSTOM events it publishes (computer_use_paused, computer_use_resumed, screen_recording). Headless passes it to its run encoder.
  • A file lock keeps two computer-use sessions from driving the same desktop at once. It is a no-op on Windows.

Documentation

Index

Constants

View Source 
const (
	ToolComputer       = "Computer"
	ToolGetLatestFrame = "GetLatestFrame"
	ToolRecordStart    = "RecordStart"
	ToolRecordStop     = "RecordStop"
)

Names of the computer tools, as their manifests declare them.

Variables

This section is empty.

Functions

func NewTools 

func NewTools(cfg *config.Config, frames FrameSourceLookup, annotator agentdomain.ImageAnnotator, recorder *ScreenRecorder) map[string]agentdomain.Tool

NewTools builds the computer-use tool set. GetLatestFrame is always included (it also serves non-screen frame sources such as cameras); the Computer tool is gated on computer_use.enabled and RecordStart/RecordStop, which share recorder, on computer_use.recording.enabled.

func PublishedEvent added in v0.219.0 

func PublishedEvent(event agentdomain.ChatEvent) (agui.CustomEvent, bool)

PublishedEvent maps the computer context's chat events to the AG-UI CUSTOM events it publishes, so a client can show a pause or a recording indicator.

func ScaleAPIToScreen 

func ScaleAPIToScreen(apiX, apiY, apiWidth, apiHeight, screenWidth, screenHeight int) (int, int, error)

ScaleAPIToScreen converts coordinates from API space (the model's screenshot) to screen space (actual display) with proportional scaling. Frames are resized uniformly (ScreenshotToolConfig.FitDims), so the X and Y factors are equal in practice; independent factors are kept for identity mapping when no resize happened.

Parameters:

  • apiX, apiY: Coordinates from the model's response (in screenshot space)
  • apiWidth, apiHeight: Screenshot dimensions from FitDims
  • screenWidth, screenHeight: Actual logical screen dimensions

Returns:

  • screenX, screenY: Coordinates in actual screen space for mouse operations
  • error when the input lies outside the API space; the message stays in API space so a model that never saw the real display cannot misread it as screen bounds

Example: 

API screenshot: 1024x768
Actual screen: 2048x1536
API coordinate (512, 384) → Screen coordinate (1024, 768)

func ToolNames added in v0.216.0 

func ToolNames() []string

ToolNames returns the names of the computer tools, whether or not config enables them, so a custom tool can never take one.

Types

type ComputerTool 

type ComputerTool struct {
	// contains filtered or unexported fields
}

ComputerTool is the single computer-use tool: it drives the mouse, keyboard, and screen through one action-based interface.

func NewComputerTool 

func NewComputerTool(cfg *config.Config, limiter rateLimiter) *ComputerTool

NewComputerTool creates the Computer tool.

func (*ComputerTool) Definition 

func (t *ComputerTool) Definition() sdk.ChatCompletionTool

Definition returns the tool definition for the LLM

func (*ComputerTool) Execute 

func (t *ComputerTool) Execute(ctx context.Context, args map[string]any) (*agentdomain.ToolExecutionResult, error)

Execute performs the action and returns the observation.

func (*ComputerTool) FormatForLLM 

func (t *ComputerTool) FormatForLLM(result *agentdomain.ToolExecutionResult) string

FormatForLLM formats the result for LLM consumption

func (*ComputerTool) FormatPreview 

func (t *ComputerTool) FormatPreview(result *agentdomain.ToolExecutionResult) string

FormatPreview returns a short preview of the result for UI display

func (*ComputerTool) FormatResult 

func (t *ComputerTool) FormatResult(result *agentdomain.ToolExecutionResult, formatType agentdomain.FormatterType) string

FormatResult formats the result based on the requested format type

func (*ComputerTool) IsEnabled 

func (t *ComputerTool) IsEnabled() bool

IsEnabled returns whether the tool is enabled

func (*ComputerTool) Manifest added in v0.216.0 

func (t *ComputerTool) Manifest() agentdomain.ToolManifest

Manifest returns the tool's manifest so the tool registry knows its policy.

func (*ComputerTool) RequiresApproval added in v0.216.0 

func (t *ComputerTool) RequiresApproval(args map[string]any, _ agentdomain.AgentMode) bool

RequiresApproval gates the Computer tool per computer_use.approval: under "destructive" only input actions (click, type, key, ...) need approval, observations such as a screenshot do not.

func (*ComputerTool) ShouldAlwaysExpand 

func (t *ComputerTool) ShouldAlwaysExpand() bool

ShouldAlwaysExpand determines if tool results should always be expanded in UI

func (*ComputerTool) ShouldCollapseArg 

func (t *ComputerTool) ShouldCollapseArg(key string) bool

ShouldCollapseArg determines if an argument should be collapsed in display

func (*ComputerTool) Validate 

func (t *ComputerTool) Validate(args map[string]any) error

Validate checks if the tool arguments are valid

type Executor 

type Executor struct {
	// contains filtered or unexported fields
}

Executor performs computer-use actions through the display controller. Action and observation coordinates are in the frame coordinate space (the space screenshots and their annotations use).

func NewExecutor 

func NewExecutor(cfg *config.Config) *Executor

NewExecutor creates an Executor.

func (*Executor) Do 

func (e *Executor) Do(ctx context.Context, a computerdomain.Action) (*computerdomain.Observation, error)

Do executes one action and returns the resulting observation.

type FrameSourceLookup 

type FrameSourceLookup interface {
	FrameSource(name string) (agentdomain.FrameSource, bool)
	FrameSourceNames() []string
}

FrameSourceLookup resolves named frame sources; the tool registry satisfies it.

type GetLatestFrameTool 

type GetLatestFrameTool struct {
	// contains filtered or unexported fields
}

GetLatestFrameTool retrieves the latest frame from a named frame source (the screen ring buffer or a configured directory source), either as a raw image or as annotated text for models without vision.

func NewGetLatestFrameTool 

func NewGetLatestFrameTool(cfg *config.Config, sources frameSourceLookup, annotator agentdomain.ImageAnnotator) *GetLatestFrameTool

NewGetLatestFrameTool creates a new tool that reads from the frame sources

func (*GetLatestFrameTool) Definition 

func (t *GetLatestFrameTool) Definition() sdk.ChatCompletionTool

Definition returns the tool definition for the LLM

func (*GetLatestFrameTool) Execute 

func (t *GetLatestFrameTool) Execute(ctx context.Context, args map[string]any) (*agentdomain.ToolExecutionResult, error)

Execute retrieves the latest frame from the requested source

func (*GetLatestFrameTool) FormatForLLM 

func (t *GetLatestFrameTool) FormatForLLM(result *agentdomain.ToolExecutionResult) string

FormatForLLM formats the result for LLM consumption

func (*GetLatestFrameTool) FormatPreview 

func (t *GetLatestFrameTool) FormatPreview(result *agentdomain.ToolExecutionResult) string

FormatPreview returns a short preview of the result for UI display

func (*GetLatestFrameTool) FormatResult 

func (t *GetLatestFrameTool) FormatResult(result *agentdomain.ToolExecutionResult, formatType agentdomain.FormatterType) string

FormatResult formats tool execution results for different contexts

func (*GetLatestFrameTool) IsEnabled 

func (t *GetLatestFrameTool) IsEnabled() bool

IsEnabled returns whether this tool is enabled

func (*GetLatestFrameTool) Manifest added in v0.216.0 

func (t *GetLatestFrameTool) Manifest() agentdomain.ToolManifest

Manifest returns the tool's manifest so the tool registry knows its policy.

func (*GetLatestFrameTool) RequiresApproval added in v0.216.0 

func (t *GetLatestFrameTool) RequiresApproval(map[string]any, agentdomain.AgentMode) bool

RequiresApproval gates GetLatestFrame per computer_use.approval; reading a frame is an observation.

func (*GetLatestFrameTool) ShouldAlwaysExpand 

func (t *GetLatestFrameTool) ShouldAlwaysExpand() bool

ShouldAlwaysExpand determines if tool results should always be expanded in UI

func (*GetLatestFrameTool) ShouldCollapseArg 

func (t *GetLatestFrameTool) ShouldCollapseArg(key string) bool

ShouldCollapseArg determines if an argument should be collapsed in display

func (*GetLatestFrameTool) Validate 

func (t *GetLatestFrameTool) Validate(args map[string]any) error

Validate checks if the tool arguments are valid

type RecordingStatus added in v0.208.0 

type RecordingStatus struct {
	Path            string                `json:"path"`
	Region          computerdomain.Region `json:"region"` // frame space
	FrameWidth      int                   `json:"frame_width"`
	FrameHeight     int                   `json:"frame_height"`
	DurationSeconds float64               `json:"duration_seconds,omitempty"`
	SizeBytes       int64                 `json:"size_bytes,omitempty"`
	Capped          bool                  `json:"capped,omitempty"`
	Message         string                `json:"message"`
}

RecordingStatus is the result data of RecordStart and RecordStop.

type ScreenRecorder added in v0.208.0 

type ScreenRecorder struct {
	// contains filtered or unexported fields
}

ScreenRecorder runs at most one ffmpeg screen recording at a time, machine wide: a lock file under ~/.infer/run keeps a second infer process from starting a competing capture. The process outlives the RecordStart tool call; RecordStop, the max_duration cap (ffmpeg -t), or Close on CLI shutdown ends it. While it runs it is a background job, so a headless run waits for it instead of exiting.

func NewScreenRecorder added in v0.208.0 

func NewScreenRecorder(cfg *config.Config, notifier agentdomain.UINotifier, jobs scheddomain.JobSubmitter) *ScreenRecorder

NewScreenRecorder creates the recorder shared by RecordStart and RecordStop. jobs may be nil, in which case a recording is not tracked as a background job.

func (*ScreenRecorder) Close added in v0.208.0 

func (r *ScreenRecorder) Close()

Close finalizes any active recording and refuses new ones. The container calls it on shutdown (normal exit, SIGINT, SIGTERM).

func (*ScreenRecorder) Start added in v0.208.0 

func (r *ScreenRecorder) Start(ctx context.Context, req recordRequest) (RecordingStatus, error)

Start begins a recording. Resolving the capture area and ffmpeg (which may download it) happens before taking the lock, so Close is never held up.

func (*ScreenRecorder) Stop added in v0.208.0 

func (r *ScreenRecorder) Stop() (RecordingStatus, error)

Stop finalizes the active recording, or the one the time cap already finished, and reports the file.

Source Files

View all Source files

Directories

Path Synopsis
accessibility
capture
clipboard
clipboard/text
display
vlm

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.