computer_use

package
v0.17.21 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package computer_use denylist loader.

This file defines the thread-safe Loader that parses the embedded default denylist (denylist.json) plus an optional per-user override file, then classifies a ForegroundInfo against the merged effective list.

Override semantics: an override entry with the same BundleID or WindowClassRegex as a default entry REPLACES the default (override wins). A new override entry with no matching default is ADDED. Overrides keep FromOverride=true so callers can distinguish source.

Index

Constants

View Source 
const AnthropicComputerToolName = "computer_20241022"

AnthropicComputerToolName is the name of Anthropic's native computer use tool.

View Source 
const DefaultOverridePath = "~/.config/sprout/computer_use_denylist_overrides.json"

DefaultOverridePath is the default per-user override file path.

Variables

View Source 
var ErrAlreadyHalted = errors.New("computer-use already halted")

ErrAlreadyHalted is returned by Halt() when the panic key is triggered a second time while already halted.

View Source 
var ErrDestructiveAppBlocked = errors.New("destructive app action blocked by user")

ErrDestructiveAppBlocked is returned when the user denies a destructive-app action via the approval cascade.

View Source 
var ErrForegroundUnavailable = errors.New("foreground-app detection not supported on this platform")

ErrForegroundUnavailable is returned by GetForegroundApp on platforms where foreground-window detection is not supported (Wayland, Windows, headless). Callers should log and skip the denylist gate gracefully.

View Source 
var ErrPanicKeyHalted = errors.New("computer-use halted by panic key")

ErrPanicKeyHalted is returned by all ComputerBackend methods when the panic key has been triggered and computer-use actions are blocked.

View Source 
var ErrRateLimited = fmt.Errorf("computer-use action rate limit exceeded")

ErrRateLimited is returned when the action-rate cap is exceeded. It exists so callers / tests can distinguish a safety stop from a backend failure.

Functions

func AddAllowEntry added in v0.16.19 

func AddAllowEntry(loader *Loader, bundleID, windowClassRegex string) error

AddAllowEntry adds an "allow: true" override entry for the given app key. If bundleID is non-empty, the entry matches by bundle_id; otherwise by windowClassRegex. The override file is updated and the loader reloaded. Safe to call concurrently.

func GOOSName added in v0.16.19 

func GOOSName() string

GOOSName returns runtime.GOOS, exposed via this package to keep platform-specific files independent of import duplication.

func Handlers 

func Handlers() []tools.ToolHandler

Handlers returns the computer-use tool handlers in a stable order. The agent registers these into its execution registry (and derives LLM definitions from each handler's Definition()) only when computer use is enabled in config. Kept here so the unexported handler structs don't need to leak.

func IsChordDisabled added in v0.16.19 

func IsChordDisabled(chord string) bool

IsChordDisabled reports whether the chord string is the explicit-off sentinel "disabled". Used by the registration code to skip starting the watcher entirely.

func KillProcessGroup added in v0.16.19 

func KillProcessGroup(p *os.Process) error

KillProcessGroup sends SIGKILL to the process group rooted at p so any children the process forked are killed alongside it.

func NewAuditingBackend 

func NewAuditingBackend(inner ComputerBackend, dir, sessionID string) (*auditingBackend, error)

NewAuditingBackend wraps inner, writing audit records to <dir>/<sessionID>.jsonl. The directory is created if missing. If the log cannot be opened, the error is returned and the caller should fall back to the unwrapped backend rather than failing the whole feature.

func NewRateLimitedBackend 

func NewRateLimitedBackend(inner ComputerBackend, maxPerMin int) *rateLimitedBackend

NewRateLimitedBackend wraps inner with a cap of maxPerMin actions per rolling 60s window. A maxPerMin <= 0 disables the cap.

func RecordSafetyEvent 

func RecordSafetyEvent(action string, args map[string]any)

RecordSafetyEvent records a safety-related event (e.g. per-session opt-in consent) to the audit log when the active backend is an auditingBackend. It is a no-op when the backend is not audit-wrapped (e.g. MockBackend in tests or when the audit directory could not be created). This lets callers in the agent layer emit opt-in / denial events without needing to know whether the audit decorator is present.

func SetBackend 

func SetBackend(b ComputerBackend)

SetBackend sets the active backend for all tool handlers.

func SetBackendPreActionHook added in v0.16.19 

func SetBackendPreActionHook(fn func(action string, args map[string]any) error)

SetBackendPreActionHook is a helper for callers outside this package (e.g., pkg/agent) to install a pre-action hook when the backend is wrapped by an auditingBackend. It is a no-op for other backend types.

func SetDestructiveAppPrompter added in v0.16.19 

func SetDestructiveAppPrompter(p DestructiveAppPrompter)

SetDestructiveAppPrompter installs the prompter used by classifyAndPrompt. Pass nil to revert to the safe no-op default.

func SetProcessGroup added in v0.16.19 

func SetProcessGroup(cmd *exec.Cmd)

SetProcessGroup makes cmd start in a new process group so that later signals can target the whole group (the process and any children it forks).

func ToolNames 

func ToolNames() []string

ToolNames returns the names of all computer-use tools. Used by the agent's dispatch-layer guard to reject these tools for any persona other than computer_user.

func TranslateAnthropicAction 

func TranslateAnthropicAction(action string, params map[string]any) (any, error)

TranslateAnthropicAction converts an Anthropic computer_20241022 action into calls to our ComputerBackend interface.

Anthropic's tool uses a single "action" parameter with sub-params: 

action: "screenshot", "mouse_move", "left_click", "right_click", "middle_click",
  "double_click", "triple_click", "left_click_drag", "type", "key", "hold_key",
  "scroll", "wait"
coordinate:       [x, y] for mouse actions
to_coordinate:    [x, y] for drag end point
text:             for type action
key:              for key / hold_key actions
scroll_direction: "up", "down", "left", "right" for scroll
amount:           for scroll
milliseconds:     for wait

The return value is optional structured output (e.g. screenshot data for the "screenshot" action). Callers should inspect it for action-specific results.

func TriggerPanicKey added in v0.16.19 

func TriggerPanicKey(reason string) error

TriggerPanicKey fires the panic key on the registered backend. It is a no-op when no PanicableBackend has been registered (e.g. computer use is disabled). Returns nil on first trigger, ErrAlreadyHalted on subsequent triggers. The reason is recorded to the audit log.

func TriggerPanicKeyFromChord added in v0.16.19 

func TriggerPanicKeyFromChord() error

TriggerPanicKeyFromChord is called by a ChordWatcher when the OS chord is detected. It is a thin wrapper around TriggerPanicKey with a fixed reason of "os_chord" so audit logs can distinguish OS-chord triggers from programmatic ones.

Types

type AuditRecord 

type AuditRecord struct {
	Time   string         `json:"time"` // RFC3339, supplied by the caller's clock
	Action string         `json:"action"`
	Args   map[string]any `json:"args,omitempty"`
	Err    string         `json:"error,omitempty"`
}

AuditRecord is one line in a session's computer-use audit log. Screenshots are recorded by size only (not contents) to keep the log small and avoid persisting potentially sensitive screen captures.

type Category added in v0.16.19 

type Category string

Category classifies WHY an app is on the denylist. 

const (
	CategoryFinancial       Category = "financial"
	CategorySystem          Category = "system"
	CategoryDestructive     Category = "destructive"
	CategoryPasswordManager Category = "password_manager"
)

type ChordWatcher added in v0.16.19 

type ChordWatcher interface {
	// Start begins watching for the chord. The watcher should call
	// TriggerPanicKey("os_chord") when the chord is detected. Returns
	// an error if the watcher cannot start (e.g. xdotool not installed).
	// Start is best-effort: callers should log the error and continue.
	Start(ctx context.Context) error
	// Stop halts the watcher and releases any resources. Safe to call
	// even if Start was never called or failed.
	Stop()
}

ChordWatcher watches the OS for a keyboard chord and triggers the panic key when detected. Implementations live in platform-specific files:

  • panic_key_chord_darwin.go (macOS, osascript-based)
  • panic_key_chord_linux.go (Linux, xdotool-based)
  • panic_key_chord_other.go (stub for all other platforms)

func ActiveChordWatcher added in v0.16.19 

func ActiveChordWatcher() ChordWatcher

ActiveChordWatcher returns the currently registered watcher (nil if none).

func NewChordWatcher added in v0.16.19 

func NewChordWatcher(chord string) ChordWatcher

NewChordWatcher returns a platform-specific ChordWatcher for the given chord string (e.g. "ctrl+shift+escape"). The chord is parsed into a list of required keys (modifiers + main key). When chord is "disabled", the returned watcher is a no-op whose Start/Stop never do anything.

The parser is permissive: keys are lowercased and trimmed. Unknown keys are kept as-is so the platform-specific matcher can decide. Modifiers ("ctrl", "shift", "alt", "meta"/"cmd"/"super") are separated from the main key.

func SetActiveChordWatcher added in v0.16.19 

func SetActiveChordWatcher(w ChordWatcher) ChordWatcher

SetActiveChordWatcher swaps the active watcher, stopping any previous one. Returns the previously-active watcher (for tests).

type Classification added in v0.16.19 

type Classification struct {
	Category     Category
	Reason       string
	MatchedEntry DenylistEntry
	FromOverride bool
}

Classification is the result of matching a ForegroundInfo against the effective denylist.

func (Classification) IsBlocked added in v0.16.19 

func (c Classification) IsBlocked() bool

IsBlocked returns true when the classification indicates a match.

type ComputerBackend 

type ComputerBackend interface {
	Screenshot(region *Rect) (image []byte, dims Size, err error)
	MouseClick(x, y int, button MouseButton, double bool) error
	MouseDrag(from, to Point, button MouseButton) error
	MoveTo(x, y int) error
	KeyboardType(text string) error
	KeyboardPress(key string) error
	Scroll(dir ScrollDir, amount int, at *Point) error
}

ComputerBackend is the platform-specific interface for desktop control. Phase 2 (SP-063-2) will implement this for macOS, Linux, Windows.

func GetBackend 

func GetBackend() ComputerBackend

GetBackend returns the current backend.

func NewPlatformBackend 

func NewPlatformBackend() (ComputerBackend, error)

NewPlatformBackend returns the best available real backend for the current host, or a descriptive error when the platform/toolchain can't support computer use. Callers that get an error should keep the default MockBackend and surface the message to the user (e.g. "install cliclick").

type DenylistEntry added in v0.16.19 

type DenylistEntry struct {
	// BundleID is the macOS bundle identifier (e.g., "com.apple.Safari").
	BundleID string

	// WindowClassRegex is the X11 WM_CLASS[Name] pattern.
	WindowClassRegex string

	// WindowTitleRegex is an optional title pattern that must ALSO match.
	WindowTitleRegex string

	// Category is one of the Category constants.
	Category Category

	// Reason is a human-readable explanation.
	Reason string

	// FromOverride is true when the entry came from the user override file.
	// Populated at load time, not serialized.
	FromOverride bool

	// Allow is true when this entry is an explicit "allow" sentinel.
	// An override entry with Allow:true removes matching default entries
	// from the effective denylist (the user explicitly whitelisted the app).
	Allow bool
	// contains filtered or unexported fields
}

DenylistEntry is one entry in the effective denylist.

type DestructiveAppDecision added in v0.16.19 

type DestructiveAppDecision int

DestructiveAppDecision is the user's choice when a destructive-app action is intercepted by the gate. 

const (
	// DestructiveAppDeny blocks the action entirely.
	DestructiveAppDeny DestructiveAppDecision = iota
	// DestructiveAppAllowOnce allows this single invocation.
	DestructiveAppAllowOnce
	// DestructiveAppAllowAlways allows this invocation and persists the
	// app to the override allowlist so future sessions skip the prompt.
	DestructiveAppAllowAlways
)

func (DestructiveAppDecision) String added in v0.16.19 

func (d DestructiveAppDecision) String() string

String returns a stable lowercase identifier for the decision.

type DestructiveAppPrompter added in v0.16.19 

type DestructiveAppPrompter interface {
	PromptDestructiveApp(ctx context.Context, action string, args map[string]any, cls Classification) DestructiveAppDecision
}

DestructiveAppPrompter is the interface the agent side implements to prompt the user when a denylisted app is detected. Tests can mock this.

func GetDestructiveAppPrompter added in v0.16.19 

func GetDestructiveAppPrompter() DestructiveAppPrompter

GetDestructiveAppPrompter returns the current prompter, or a no-op default that always returns DestructiveAppDeny when none is set.

type ForegroundInfo added in v0.16.19 

type ForegroundInfo struct {
	// AppName is the human-readable application name.
	AppName string

	// BundleID is the macOS bundle identifier (e.g., "com.apple.Safari").
	// Empty on Linux.
	BundleID string

	// WindowClass is the X11 WM_CLASS[Name] field (e.g., "Navigator").
	// Empty on macOS.
	WindowClass string

	// WindowTitle is the active window's title.
	WindowTitle string
}

ForegroundInfo describes the app currently in the foreground.

func GetForegroundApp added in v0.16.19 

func GetForegroundApp() (ForegroundInfo, error)

GetForegroundApp returns the foreground-app tuple for the current platform. Returns ErrForegroundUnavailable when detection is not supported. Implementations live in platform-specific files:

  • foreground_darwin.go (macOS, osascript-based)
  • foreground_linux.go (Linux/X11, xdotool + wmctrl)
  • foreground_other.go (stub for all other platforms)

type Loader added in v0.16.19 

type Loader struct {
	// contains filtered or unexported fields
}

Loader is a thread-safe cache of the parsed default + override denylist.

func DefaultLoader added in v0.16.19 

func DefaultLoader() *Loader

DefaultLoader returns the singleton denylist loader.

func (*Loader) IsDestructiveApp added in v0.16.19 

func (l *Loader) IsDestructiveApp(fg ForegroundInfo) Classification

IsDestructiveApp classifies a foreground-app tuple against the effective denylist. Entries with Allow:true short-circuit to "not destructive" (empty Classification).

func (*Loader) OverridePath added in v0.16.19 

func (l *Loader) OverridePath() string

OverridePath returns the current override file path.

func (*Loader) Reload added in v0.16.19 

func (l *Loader) Reload() error

Reload reloads the default + override lists. Tests call this after modifying the override file via SetOverridePath.

func (*Loader) SetOverridePath added in v0.16.19 

func (l *Loader) SetOverridePath(path string)

SetOverridePath sets the override file path. Next Reload() reads from new path.

type MockBackend 

type MockBackend struct {
	Records                []MockBackendRecord
	OverrideScreenshotData []byte
	OverrideScreenshotDims Size
	OverrideError          error
	// contains filtered or unexported fields
}

MockBackend implements ComputerBackend for testing.

func (*MockBackend) KeyboardPress 

func (m *MockBackend) KeyboardPress(key string) error

func (*MockBackend) KeyboardType 

func (m *MockBackend) KeyboardType(text string) error

func (*MockBackend) MouseClick 

func (m *MockBackend) MouseClick(x, y int, button MouseButton, double bool) error

func (*MockBackend) MouseDrag 

func (m *MockBackend) MouseDrag(from, to Point, button MouseButton) error

func (*MockBackend) MoveTo 

func (m *MockBackend) MoveTo(x, y int) error

func (*MockBackend) Screenshot 

func (m *MockBackend) Screenshot(region *Rect) ([]byte, Size, error)

func (*MockBackend) Scroll 

func (m *MockBackend) Scroll(dir ScrollDir, amount int, at *Point) error

type MockBackendRecord 

type MockBackendRecord struct {
	Action string
	Args   map[string]any
}

MockBackendRecord records a single call for test assertions.

type MouseButton 

type MouseButton string

MouseButton specifies which mouse button. 

const (
	MouseLeft   MouseButton = "left"
	MouseRight  MouseButton = "right"
	MouseMiddle MouseButton = "middle"
)

type PanicableBackend added in v0.16.19 

type PanicableBackend struct {
	// contains filtered or unexported fields
}

PanicableBackend wraps a ComputerBackend so that in-flight subprocess actions can be killed when the panic key is pressed. It sits between the subprocess backend and the rate-limit decorator: 

real → panicable → rateLimited → auditing

func GlobalPanicable added in v0.16.19 

func GlobalPanicable() *PanicableBackend

GlobalPanicable returns the registered PanicableBackend, or nil if none has been registered. Exported for tests and any future signal handler that needs direct access.

func NewPanicableBackend added in v0.16.19 

func NewPanicableBackend(inner ComputerBackend) *PanicableBackend

NewPanicableBackend wraps inner with panic-key support. It installs package-level hooks so that the subprocess backend's runWithCtx can report the in-flight process to this decorator.

func (*PanicableBackend) Halt added in v0.16.19 

func (p *PanicableBackend) Halt(reason string) error

Halt signals the panic key was pressed. It kills any in-flight subprocess and records the halt state. Subsequent actions return ErrPanicKeyHalted. Calling Halt() while already halted returns ErrAlreadyHalted and records a "panic_key_duplicate" audit event.

func (*PanicableBackend) HaltReason added in v0.16.19 

func (p *PanicableBackend) HaltReason() string

HaltReason returns the reason string passed to the most recent Halt() call. Safe to call concurrently.

func (*PanicableBackend) HaltedAt added in v0.16.19 

func (p *PanicableBackend) HaltedAt() time.Time

HaltedAt returns the time when Halt() was most recently called. Safe to call concurrently.

func (*PanicableBackend) IsHalted added in v0.16.19 

func (p *PanicableBackend) IsHalted() bool

IsHalted reports whether the panic key has been triggered.

func (*PanicableBackend) KeyboardPress added in v0.16.19 

func (p *PanicableBackend) KeyboardPress(key string) error

func (*PanicableBackend) KeyboardType added in v0.16.19 

func (p *PanicableBackend) KeyboardType(text string) error

func (*PanicableBackend) MouseClick added in v0.16.19 

func (p *PanicableBackend) MouseClick(x, y int, button MouseButton, double bool) error

func (*PanicableBackend) MouseDrag added in v0.16.19 

func (p *PanicableBackend) MouseDrag(from, to Point, button MouseButton) error

func (*PanicableBackend) MoveTo added in v0.16.19 

func (p *PanicableBackend) MoveTo(x, y int) error

func (*PanicableBackend) Reset added in v0.16.19 

func (p *PanicableBackend) Reset()

Reset clears the halted state after the user acknowledges the halt. Records a "panic_key_reset" audit event. No-op (with no audit event) when the backend is not currently halted.

func (*PanicableBackend) Screenshot added in v0.16.19 

func (p *PanicableBackend) Screenshot(region *Rect) ([]byte, Size, error)

func (*PanicableBackend) Scroll added in v0.16.19 

func (p *PanicableBackend) Scroll(dir ScrollDir, amount int, at *Point) error

type PlatformSupport 

type PlatformSupport struct {
	Supported bool   `json:"supported"`
	OS        string `json:"os"`
	Reason    string `json:"reason,omitempty"` // populated when Supported is false
}

PlatformSupport describes whether the current host can run computer use and, if not, why. Used by the "Test connection" diagnostic and the persona activation check.

func CheckPlatformSupport 

func CheckPlatformSupport() PlatformSupport

CheckPlatformSupport reports whether a real backend can be constructed without actually taking control of the desktop.

type Point 

type Point struct {
	X, Y int
}

Point represents a coordinate.

type Rect 

type Rect struct {
	X, Y, Width, Height int
}

Rect represents a rectangular region on screen.

type ScrollDir 

type ScrollDir string

ScrollDir specifies scroll direction. 

const (
	ScrollUp    ScrollDir = "up"
	ScrollDown  ScrollDir = "down"
	ScrollLeft  ScrollDir = "left"
	ScrollRight ScrollDir = "right"
)

type Size 

type Size struct {
	Width, Height int
}

Size represents dimensions.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.