computer

package
v0.182.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package computer is the computer-use capability: the Computer tool drives the mouse, keyboard, and screen through robotgo; GetLatestFrame serves frames from the screen ring buffer and configured frame sources.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewTools 

func NewTools(cfg *config.Config, frames FrameSourceLookup, annotator agentdomain.ImageAnnotator) map[string]agentdomain.Tool

NewTools builds the computer-use tool set. GetLatestFrame is always included (it also serves non-screen frame sources such as cameras); the Computer tool is gated on computer_use.enabled.

func ScaleAPIToScreen 

func ScaleAPIToScreen(apiX, apiY, apiWidth, apiHeight, screenWidth, screenHeight int) (int, int, error)

ScaleAPIToScreen converts coordinates from API space (the model's screenshot) to screen space (actual display) with proportional scaling. Frames are resized uniformly (ScreenshotToolConfig.FitDims), so the X and Y factors are equal in practice; independent factors are kept for identity mapping when no resize happened.

Parameters:

  • apiX, apiY: Coordinates from the model's response (in screenshot space)
  • apiWidth, apiHeight: Screenshot dimensions from FitDims
  • screenWidth, screenHeight: Actual logical screen dimensions

Returns:

  • screenX, screenY: Coordinates in actual screen space for mouse operations
  • error when the input lies outside the API space; the message stays in API space so a model that never saw the real display cannot misread it as screen bounds

Example: 

API screenshot: 1024x768
Actual screen: 2048x1536
API coordinate (512, 384) → Screen coordinate (1024, 768)

Types

type ComputerTool 

type ComputerTool struct {
	// contains filtered or unexported fields
}

ComputerTool is the single computer-use tool: it drives the mouse, keyboard, and screen through one action-based interface.

func NewComputerTool 

func NewComputerTool(cfg *config.Config, limiter rateLimiter) *ComputerTool

NewComputerTool creates the Computer tool.

func (*ComputerTool) Definition 

func (t *ComputerTool) Definition() sdk.ChatCompletionTool

Definition returns the tool definition for the LLM

func (*ComputerTool) Execute 

func (t *ComputerTool) Execute(ctx context.Context, args map[string]any) (*agentdomain.ToolExecutionResult, error)

Execute performs the action and returns the observation.

func (*ComputerTool) FormatForLLM 

func (t *ComputerTool) FormatForLLM(result *agentdomain.ToolExecutionResult) string

FormatForLLM formats the result for LLM consumption

func (*ComputerTool) FormatPreview 

func (t *ComputerTool) FormatPreview(result *agentdomain.ToolExecutionResult) string

FormatPreview returns a short preview of the result for UI display

func (*ComputerTool) FormatResult 

func (t *ComputerTool) FormatResult(result *agentdomain.ToolExecutionResult, formatType agentdomain.FormatterType) string

FormatResult formats the result based on the requested format type

func (*ComputerTool) IsEnabled 

func (t *ComputerTool) IsEnabled() bool

IsEnabled returns whether the tool is enabled

func (*ComputerTool) ShouldAlwaysExpand 

func (t *ComputerTool) ShouldAlwaysExpand() bool

ShouldAlwaysExpand determines if tool results should always be expanded in UI

func (*ComputerTool) ShouldCollapseArg 

func (t *ComputerTool) ShouldCollapseArg(key string) bool

ShouldCollapseArg determines if an argument should be collapsed in display

func (*ComputerTool) Validate 

func (t *ComputerTool) Validate(args map[string]any) error

Validate checks if the tool arguments are valid

type Executor 

type Executor struct {
	// contains filtered or unexported fields
}

Executor performs computer-use actions through the display controller. Action and observation coordinates are in the frame coordinate space (the space screenshots and their annotations use).

func NewExecutor 

func NewExecutor(cfg *config.Config) *Executor

NewExecutor creates an Executor.

func (*Executor) Do 

func (e *Executor) Do(ctx context.Context, a computerdomain.Action) (*computerdomain.Observation, error)

Do executes one action and returns the resulting observation.

type FrameSourceLookup 

type FrameSourceLookup interface {
	FrameSource(name string) (agentdomain.FrameSource, bool)
	FrameSourceNames() []string
}

FrameSourceLookup resolves named frame sources; the tool registry satisfies it.

type GetLatestFrameTool 

type GetLatestFrameTool struct {
	// contains filtered or unexported fields
}

GetLatestFrameTool retrieves the latest frame from a named frame source (the screen ring buffer or a configured directory source), either as a raw image or as annotated text for models without vision.

func NewGetLatestFrameTool 

func NewGetLatestFrameTool(cfg *config.Config, sources frameSourceLookup, annotator agentdomain.ImageAnnotator) *GetLatestFrameTool

NewGetLatestFrameTool creates a new tool that reads from the frame sources

func (*GetLatestFrameTool) Definition 

func (t *GetLatestFrameTool) Definition() sdk.ChatCompletionTool

Definition returns the tool definition for the LLM

func (*GetLatestFrameTool) Execute 

func (t *GetLatestFrameTool) Execute(ctx context.Context, args map[string]any) (*agentdomain.ToolExecutionResult, error)

Execute retrieves the latest frame from the requested source

func (*GetLatestFrameTool) FormatForLLM 

func (t *GetLatestFrameTool) FormatForLLM(result *agentdomain.ToolExecutionResult) string

FormatForLLM formats the result for LLM consumption

func (*GetLatestFrameTool) FormatPreview 

func (t *GetLatestFrameTool) FormatPreview(result *agentdomain.ToolExecutionResult) string

FormatPreview returns a short preview of the result for UI display

func (*GetLatestFrameTool) FormatResult 

func (t *GetLatestFrameTool) FormatResult(result *agentdomain.ToolExecutionResult, formatType agentdomain.FormatterType) string

FormatResult formats tool execution results for different contexts

func (*GetLatestFrameTool) IsEnabled 

func (t *GetLatestFrameTool) IsEnabled() bool

IsEnabled returns whether this tool is enabled

func (*GetLatestFrameTool) ShouldAlwaysExpand 

func (t *GetLatestFrameTool) ShouldAlwaysExpand() bool

ShouldAlwaysExpand determines if tool results should always be expanded in UI

func (*GetLatestFrameTool) ShouldCollapseArg 

func (t *GetLatestFrameTool) ShouldCollapseArg(key string) bool

ShouldCollapseArg determines if an argument should be collapsed in display

func (*GetLatestFrameTool) Validate 

func (t *GetLatestFrameTool) Validate(args map[string]any) error

Validate checks if the tool arguments are valid

Source Files

View all Source files

Directories

Path Synopsis
Package domain holds the computer-use bounded context's contracts: actions the agent can perform, targets they act on, observations they return, and the capabilities the platform supports.
 Package domain holds the computer-use bounded context's contracts: actions the agent can perform, targets they act on, observations they return, and the capabilities the platform supports.
accessibility
Package accessibility provides crash-isolated platform accessibility-tree access for the computer-use capability.
 Package accessibility provides crash-isolated platform accessibility-tree access for the computer-use capability.
clipboard
Package clipboard provides cgo-free clipboard access.
 Package clipboard provides cgo-free clipboard access.
clipboard/text
Package text provides a CGO-free, cross-platform clipboard text writer that shells out to the platform's native clipboard utility (pbcopy, wl-copy, xclip, xsel, or clip).
 Package text provides a CGO-free, cross-platform clipboard text writer that shells out to the platform's native clipboard utility (pbcopy, wl-copy, xclip, xsel, or clip).
display
vlm
Package vlm provides image annotation (image -> scene summary + numbered element list) via a side-call to a vision model through the inference gateway.
 Package vlm provides image annotation (image -> scene summary + numbered element list) via a side-call to a vision model through the inference gateway.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.