Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrScreenNotHeld = errors.New("the screen is not held")
ErrScreenNotHeld refuses an input for a screen nobody holds, so a stale tab cannot drive what it handed back.
Functions ¶
func IsInputAction ¶ added in v0.245.0
func IsInputAction(kind ActionKind) bool
IsInputAction reports whether the agent executes the kind by driving the pointer, keyboard or clipboard, so it waits while the person holds the screen. Observations capture or read and never wait.
Types ¶
type Action ¶
type Action struct {
Kind ActionKind
Target *Target
Text string // type: the text to type
Label string // press: exact accessibility label
Scope string // accessibility/press: frontmost, dock, menubar, app:<name>, or pid:<n>
Combo string // key: a combo such as "cmd+shift+t"
Button string // click kinds: left (default), right, middle
Direction string // scroll: vertical (default) or horizontal
Amount int // scroll: wheel clicks; negative reverses direction
}
Action is one computer-use request: what to do and, where relevant, on what target and with what input.
type ActionKind ¶
type ActionKind string
ActionKind enumerates what the Computer tool can do.
const ( ActionScreenshot ActionKind = "screenshot" ActionAccessibility ActionKind = "accessibility" ActionCursor ActionKind = "cursor" ActionMove ActionKind = "move" ActionClick ActionKind = "click" ActionDoubleClick ActionKind = "double_click" ActionTripleClick ActionKind = "triple_click" ActionScroll ActionKind = "scroll" ActionType ActionKind = "type" ActionKey ActionKind = "key" ActionPress ActionKind = "press" )
type Image ¶
type Image struct {
Data string // base64-encoded
MimeType string
Path string // on-disk copy, reachable by ImageDecode for non-vision models
}
Image is a captured frame.
type Observation ¶
type Observation struct {
Message string // LLM-facing outcome description
CursorX int
CursorY int
Width int // frame-space dimensions
Height int
Image *Image // set by screenshot actions
Elements []UIElement // set by accessibility observations
}
Observation is what the computer reports back after an action.
type ScreenControl ¶ added in v0.245.0
type ScreenControl interface {
// TakeOver claims the screen for the person. It fails while the person
// already holds it, from another tab of the same session or another one.
TakeOver() error
// HandBack releases the screen to the agent and reports whether it was
// held. Releasing an unheld screen is a no-op.
HandBack() bool
// Held reports whether the person currently drives the screen.
Held() bool
// WaitRelease blocks while the person drives the screen and returns when
// the agent may act again, or the context's error when its turn ends.
WaitRelease(ctx context.Context) error
// Drive performs one person input on the held screen in frame space,
// and refuses with ErrScreenNotHeld when nobody holds it.
Drive(ctx context.Context, input ScreenInput) error
}
ScreenControl owns the take-over invariant: exactly one actor, the person at the keyboard or the agent, drives a screen at a time. TakeOver claims it for the person, HandBack hands it to the agent and WaitRelease blocks one agent input action until the person lets go.
type ScreenInput ¶ added in v0.245.0
type ScreenInput struct {
Kind ActionKind `json:"kind"` // move, click, double_click, scroll, type, key
X int `json:"x,omitempty"`
Y int `json:"y,omitempty"`
Text string `json:"text,omitempty"` // type: the text to type
Combo string `json:"combo,omitempty"` // key: a combo such as "ctrl+shift+t" or a key name
Button string `json:"button,omitempty"` // click kinds: left (default), right, middle
Direction string `json:"direction,omitempty"` // scroll: vertical (default) or horizontal
Amount int `json:"amount,omitempty"` // scroll: wheel clicks, negative reverses
}
ScreenInput is one person-driven input on the session's screen, in frame coordinate space (the space the Screen view's image and its X-Frame-Width and X-Frame-Height headers use): what to do and, per kind, on what.
func (ScreenInput) Action ¶ added in v0.245.0
func (s ScreenInput) Action() (Action, error)
Action builds the action this input drives the screen with, so the display dispatch stays one code path with the Computer tool's and validates both.
type Target ¶
type Target struct {
X int
Y int
// Region, on a screenshot action, captures only this rectangle.
Region *Region
}
Target locates an action on screen, in the frame coordinate space (the same space screenshots and their annotations use).
type UIElement ¶ added in v0.179.0
type UIElement struct {
Role string `json:"role"`
Label string `json:"label"`
State string `json:"state"`
BBox [4]int `json:"bbox"`
}
UIElement is one compact accessibility-tree observation. BBox is [left, top, right, bottom] in the same frame coordinate space as Computer screenshots and pointer actions.
Source Files