computer

package
v0.245.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2026 License: Apache-2.0 Imports: 36 Imported by: 0

README

computer

What - the computer bounded context: reading the accessibility tree, pressing labelled controls, capturing screenshots, moving the pointer and typing. Why - desktop control needs platform-native calls, coordinate scaling and a rate limit, none of which belong in the agent loop. How - domain/ defines the accessibility and observation contracts, and infrastructure/ provides the platform bridges and the screenshot stream.

How it plugs in

  • NewTools(...) builds Computer, GetLatestFrame, RecordStart and RecordStop from the manifests in tools/, and the container registers them into the tools registry.
  • The accessibility provider runs in a helper subprocess: AXUIElement on macOS, AT-SPI2 over D-Bus on Linux X11. Wayland and Windows return an unsupported error and the tool falls back to screenshot guidance, so a native failure never takes down the CLI. The X11 window lookup it uses to resolve a target also serves window recording.
  • PublishedEvent maps this context's chat events onto the run stream: the screenRecording state key, and the computer_use activity each Computer pointer or keyboard action publishes before it runs. Headless passes it to its run encoder. A pause or resume publishes nothing: pausing stops the run and resuming starts a new one, which clients see through the run frames.
  • A file lock keeps two computer-use sessions from driving the same desktop at once. It is a no-op on Windows.

Documentation

Index

Constants

View Source 
const (
	ToolComputer       = "Computer"
	ToolGetLatestFrame = "GetLatestFrame"
	ToolRecordStart    = "RecordStart"
	ToolRecordStop     = "RecordStop"
)

Names of the computer tools, as their manifests declare them.

Variables

This section is empty.

Functions

func NewTools 

func NewTools(cfg *config.Config, frames FrameSourceLookup, annotator agentdomain.ImageAnnotator, recorder *ScreenRecorder, notifier agentdomain.UINotifier, screen computerdomain.ScreenControl) map[string]agentdomain.Tool

NewTools builds the computer-use tool set. GetLatestFrame is always included (it also serves non-screen frame sources such as cameras); the Computer tool is gated on computer_use.enabled and RecordStart/RecordStop, which share recorder, on computer_use.recording.enabled. notifier may be nil, in which case the tools publish nothing. The screen control parks the Computer tool's input actions while a person holds the screen.

func PublishedEvent added in v0.219.0 

func PublishedEvent(event agentdomain.ChatEvent) (agui.Published, bool)

PublishedEvent maps the computer context's chat events into the run state and activity they publish: the screenRecording key and the computer-use actions a client renders live. A paused or resumed computer use ends and starts runs instead, which clients see through the run frames, so it publishes nothing.

func ScaleAPIToScreen 

func ScaleAPIToScreen(apiX, apiY, apiWidth, apiHeight, screenWidth, screenHeight int) (int, int, error)

ScaleAPIToScreen converts coordinates from API space (the model's screenshot) to screen space (actual display) with proportional scaling. Frames are resized uniformly (ScreenshotToolConfig.FitDims), so the X and Y factors are equal in practice; independent factors are kept for identity mapping when no resize happened.

Parameters:

  • apiX, apiY: Coordinates from the model's response (in screenshot space)
  • apiWidth, apiHeight: Screenshot dimensions from FitDims
  • screenWidth, screenHeight: Actual logical screen dimensions

Returns:

  • screenX, screenY: Coordinates in actual screen space for mouse operations
  • error when the input lies outside the API space; the message stays in API space so a model that never saw the real display cannot misread it as screen bounds

Example: 

API screenshot: 1024x768
Actual screen: 2048x1536
API coordinate (512, 384) → Screen coordinate (1024, 768)

func ToolNames added in v0.216.0 

func ToolNames() []string

ToolNames returns the names of the computer tools, whether or not config enables them, so a custom tool can never take one.

Types

type ComputerTool 

type ComputerTool struct {
	// contains filtered or unexported fields
}

ComputerTool is the single computer-use tool: it drives the mouse, keyboard, and screen through one action-based interface.

func NewComputerTool 

func NewComputerTool(cfg *config.Config, limiter rateLimiter, notifier agentdomain.UINotifier, screen computerdomain.ScreenControl) *ComputerTool

NewComputerTool creates the Computer tool. The notifier receives the computer-use activity each pointer or keyboard action publishes before it runs. The screen control parks the tool's input actions while a person holds the screen.

func (*ComputerTool) Definition 

func (t *ComputerTool) Definition() sdk.ChatCompletionTool

Definition returns the tool definition for the LLM

func (*ComputerTool) Execute 

func (t *ComputerTool) Execute(ctx context.Context, args map[string]any) (*agentdomain.ToolExecutionResult, error)

Execute performs the action and returns the observation.

func (*ComputerTool) FormatForLLM 

func (t *ComputerTool) FormatForLLM(result *agentdomain.ToolExecutionResult) string

FormatForLLM formats the result for LLM consumption

func (*ComputerTool) FormatPreview 

func (t *ComputerTool) FormatPreview(result *agentdomain.ToolExecutionResult) string

FormatPreview returns a short preview of the result for UI display

func (*ComputerTool) FormatResult 

func (t *ComputerTool) FormatResult(result *agentdomain.ToolExecutionResult, formatType agentdomain.FormatterType) string

FormatResult formats the result based on the requested format type

func (*ComputerTool) IsEnabled 

func (t *ComputerTool) IsEnabled() bool

IsEnabled returns whether the tool is enabled

func (*ComputerTool) Manifest added in v0.216.0 

func (t *ComputerTool) Manifest() agentdomain.ToolManifest

Manifest returns the tool's manifest so the tool registry knows its policy.

func (*ComputerTool) RequiresApproval added in v0.216.0 

func (t *ComputerTool) RequiresApproval(args map[string]any, _ agentdomain.AgentMode) bool

RequiresApproval gates the Computer tool per computer_use.approval: under "destructive" only input actions (click, type, key, ...) need approval, observations such as a screenshot do not.

func (*ComputerTool) Validate 

func (t *ComputerTool) Validate(args map[string]any) error

Validate checks if the tool arguments are valid

type Executor 

type Executor struct {
	// contains filtered or unexported fields
}

Executor performs computer-use actions through the display controller. Action and observation coordinates are in the frame coordinate space (the space screenshots and their annotations use).

func NewExecutor 

func NewExecutor(cfg *config.Config, notifier agentdomain.UINotifier, screen computerdomain.ScreenControl) *Executor

NewExecutor creates an Executor. The screen control, when wired, parks input actions while a person holds the screen.

func (*Executor) Do 

func (e *Executor) Do(ctx context.Context, a computerdomain.Action) (*computerdomain.Observation, error)

Do executes one action and returns the resulting observation.

type FrameSourceLookup 

type FrameSourceLookup interface {
	FrameSource(name string) (agentdomain.FrameSource, bool)
	FrameSourceNames() []string
}

FrameSourceLookup resolves named frame sources; the tool registry satisfies it.

type GetLatestFrameTool 

type GetLatestFrameTool struct {
	// contains filtered or unexported fields
}

GetLatestFrameTool retrieves the latest frame from a named frame source (the screen ring buffer or a configured directory source), either as a raw image or as annotated text for models without vision.

func NewGetLatestFrameTool 

func NewGetLatestFrameTool(cfg *config.Config, sources FrameSourceLookup, annotator agentdomain.ImageAnnotator) *GetLatestFrameTool

NewGetLatestFrameTool creates a new tool that reads from the frame sources

func (*GetLatestFrameTool) Definition 

func (t *GetLatestFrameTool) Definition() sdk.ChatCompletionTool

Definition returns the tool definition for the LLM

func (*GetLatestFrameTool) Execute 

func (t *GetLatestFrameTool) Execute(ctx context.Context, args map[string]any) (*agentdomain.ToolExecutionResult, error)

Execute retrieves the latest frame from the requested source

func (*GetLatestFrameTool) FormatForLLM 

func (t *GetLatestFrameTool) FormatForLLM(result *agentdomain.ToolExecutionResult) string

FormatForLLM formats the result for LLM consumption

func (*GetLatestFrameTool) FormatPreview 

func (t *GetLatestFrameTool) FormatPreview(result *agentdomain.ToolExecutionResult) string

FormatPreview returns a short preview of the result for UI display

func (*GetLatestFrameTool) FormatResult 

func (t *GetLatestFrameTool) FormatResult(result *agentdomain.ToolExecutionResult, formatType agentdomain.FormatterType) string

FormatResult formats tool execution results for different contexts

func (*GetLatestFrameTool) IsEnabled 

func (t *GetLatestFrameTool) IsEnabled() bool

IsEnabled returns whether this tool is enabled

func (*GetLatestFrameTool) Manifest added in v0.216.0 

func (t *GetLatestFrameTool) Manifest() agentdomain.ToolManifest

Manifest returns the tool's manifest so the tool registry knows its policy.

func (*GetLatestFrameTool) RequiresApproval added in v0.216.0 

func (t *GetLatestFrameTool) RequiresApproval(map[string]any, agentdomain.AgentMode) bool

RequiresApproval gates GetLatestFrame per computer_use.approval; reading a frame is an observation.

func (*GetLatestFrameTool) Validate 

func (t *GetLatestFrameTool) Validate(args map[string]any) error

Validate checks if the tool arguments are valid

type RecordingStatus added in v0.208.0 

type RecordingStatus struct {
	Path            string                `json:"path"`
	Region          computerdomain.Region `json:"region"` // frame space
	FrameWidth      int                   `json:"frame_width"`
	FrameHeight     int                   `json:"frame_height"`
	DurationSeconds float64               `json:"duration_seconds,omitempty"`
	SizeBytes       int64                 `json:"size_bytes,omitempty"`
	Capped          bool                  `json:"capped,omitempty"`
	Message         string                `json:"message"`
}

RecordingStatus is the result data of RecordStart and RecordStop.

type ScreenControl added in v0.245.0 

type ScreenControl struct {
	// contains filtered or unexported fields
}

ScreenControl keeps the take-over invariant this context owns: exactly one actor, the person at the keyboard or the agent, drives the screen at a time. HandBack wakes the executor actions waiting on the hold. Drive takes no screen lock - the person is the authority on the worker's screen.

func NewScreenControl added in v0.245.0 

func NewScreenControl(cfg *config.Config, notifier agentdomain.UINotifier) *ScreenControl

NewScreenControl creates the screen control. The notifier may be nil, in which case take-overs publish nothing.

func (*ScreenControl) Drive added in v0.245.0 

func (s *ScreenControl) Drive(ctx context.Context, input computerdomain.ScreenInput) error

Drive performs one person input on the held screen, in the frame space the Screen view sends. It runs without the tool layer - no approval, no judge, no conversation entry - because the person at the keyboard is the authority.

func (*ScreenControl) HandBack added in v0.245.0 

func (s *ScreenControl) HandBack() bool

HandBack releases the screen to the agent, lets go of every action the executor parked on Held and reports whether the screen was held.

func (*ScreenControl) Held added in v0.245.0 

func (s *ScreenControl) Held() bool

Held reports whether the person currently drives the screen.

func (*ScreenControl) TakeOver added in v0.245.0 

func (s *ScreenControl) TakeOver() error

TakeOver claims the screen for the person. A second claim, from another tab of the session or another web viewer, is refused.

func (*ScreenControl) WaitRelease added in v0.245.0 

func (s *ScreenControl) WaitRelease(ctx context.Context) error

WaitRelease blocks while the person drives the screen. It re-checks held after every release, so a claim taken in between parks again.

type ScreenRecorder added in v0.208.0 

type ScreenRecorder struct {
	// contains filtered or unexported fields
}

ScreenRecorder runs at most one ffmpeg screen recording at a time, machine wide: a lock file under ~/.infer/run keeps a second infer process from starting a competing capture. The process outlives the RecordStart tool call; RecordStop, the max_duration cap (ffmpeg -t), or Close on CLI shutdown ends it. While it runs it is a background job, so a headless run waits for it instead of exiting.

func NewScreenRecorder added in v0.208.0 

func NewScreenRecorder(cfg *config.Config, notifier agentdomain.UINotifier, jobs scheddomain.JobSubmitter) *ScreenRecorder

NewScreenRecorder creates the recorder shared by RecordStart and RecordStop. jobs may be nil, in which case a recording is not tracked as a background job.

func (*ScreenRecorder) Close added in v0.208.0 

func (r *ScreenRecorder) Close()

Close finalizes any active recording, waits for one a Stop is finalizing, and refuses new ones. The container calls it on shutdown (normal exit, SIGINT, SIGTERM).

func (*ScreenRecorder) Start added in v0.208.0 

func (r *ScreenRecorder) Start(ctx context.Context, req recordRequest) (RecordingStatus, error)

Start begins a recording. Resolving the capture area and ffmpeg (which may download it) happens before taking the lock, so Close is never held up.

func (*ScreenRecorder) Stop added in v0.208.0 

func (r *ScreenRecorder) Stop() (RecordingStatus, error)

Stop finalizes the active recording, or the one the time cap already finished, and reports the file. ffmpeg finalizes outside the lock, and finishing keeps RecordStart refusing until it is done.

Source Files

View all Source files

Directories

Path Synopsis
accessibility
capture
clipboard
clipboard/text
display
vlm

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.