Documentation
¶
Overview ¶
Package agent provides the ComputerUse orchestrator — a screenshot→analyze→act→verify loop that lets the agent control the desktop GUI. It calls the LLM directly (bypassing the provider.Message type, which doesn't support multimodal image inputs) to analyze screenshots and decide the next action.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ComputerAction ¶
type ComputerAction struct {
Action string `json:"action"` // click | double-click | right-click | type | key | scroll | wait | done | fail
X int `json:"x,omitempty"`
Y int `json:"y,omitempty"`
Text string `json:"text,omitempty"`
Key string `json:"key,omitempty"` // Enter, Tab, Escape, etc.
Clicks int `json:"clicks,omitempty"` // scroll: positive=down, negative=up
MS int `json:"ms,omitempty"` // wait: milliseconds to wait
Summary string `json:"summary,omitempty"` // done/fail
Reason string `json:"reason,omitempty"` // fail
}
ComputerAction is a structured action the LLM returns during the computer-use loop.
type ComputerUse ¶
type ComputerUse struct {
// contains filtered or unexported fields
}
ComputerUse runs a visual desktop-automation loop: screenshot → LLM analysis → execute action → verify → repeat. It makes direct HTTP calls to the OpenAI-compatible provider so it can send base64-encoded screenshot images (the standard Message type only supports plain text).
func NewComputerUse ¶
func NewComputerUse(apiKey, baseURL, model string) *ComputerUse
NewComputerUse creates a ComputerUse loop driver.
Source Files