uiauto

package
v0.703.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 7, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package uiauto is the Pando Desktop Controller: it wires the platform-independent internal/uiauto/core building blocks (Element, Selector, Locator, SnapshotStore, Backend, ActionResolver, ...) to a concrete accessibility backend and exposes a single Manager the desktop_* agent tools drive.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Registry

func Registry() *core.Registry

Registry returns the process-wide backend Registry, so platform packages and tests can register or resolve backends without reaching into an unexported package variable.

func ResetShared

func ResetShared()

ResetShared discards the shared Manager singleton so the next Shared() call rebuilds it from the current configuration. Exposed for tests.

Types

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager is the Desktop Controller runtime. Since Block R (2026-08-30) it no longer binds to a single resolved core.Backend: it holds the set of backends actually available in this session -- an "OS" backend (atspi/ uia/ax, or null when none is available) and, independently, the "cdp" backend when registered -- and routes each operation to whichever one should actually serve it (see backendForScope/backendForElement). It never performs an OS call itself; all of that lives behind core.Backend.

func NewManager

func NewManager(opts Options) (*Manager, error)

NewManager resolves the backends named by opts.Backend (falling back to the "null" OS backend when resolution fails, so a Manager can always be constructed) and builds the SnapshotStore/physical-input layer around them.

func Shared

func Shared() (*Manager, error)

Shared returns the process-wide Manager singleton, lazily building it (or rebuilding it, on relevant config change) from config.Get().InternalTools, mirroring browser_session.go's shared-session pattern.

func (*Manager) ActionTimeout

func (m *Manager) ActionTimeout() time.Duration

ActionTimeout returns the configured default action/wait timeout.

func (*Manager) Apps

func (m *Manager) Apps(ctx context.Context) ([]core.AppInfo, error)

Apps lists running applications, filtered by the allow/deny policy.

func (*Manager) BackendName

func (m *Manager) BackendName() string

BackendName returns the name of the resolved default (OS) backend ("null" when none is available). It does not reflect whether a "cdp" backend is also available for routing -- see IsPinned/CdpAvailable.

func (*Manager) Capabilities

func (m *Manager) Capabilities() core.Capabilities

Capabilities reports what the default (OS) backend can actually do, in this session, merged with the physical-input/screen-capture layers. This is the same signal Manager reported before per-scope routing existed; it intentionally never folds in the CDP backend's capabilities, since those only apply to browser-scoped operations. Use CapabilitiesFor(ctx, scope) when the caller has a specific scope in mind (e.g. a browser window), so a reported capability always matches the backend that would actually serve that scope.

func (*Manager) CapabilitiesFor

func (m *Manager) CapabilitiesFor(ctx context.Context, scope core.Scope) core.Capabilities

CapabilitiesFor reports the capabilities of whichever backend backendForScope would route scope to, merged with the same physical- input/screen-capture layers Capabilities() uses. A browser-window scope with a live CDP session reports that session's real capabilities (Accessibility/UIInspection/UIActions/Events); any other scope reports exactly Capabilities(). This never overclaims: a capability is only true here when the backend that would actually handle scope can offer it.

func (*Manager) CdpAvailable

func (m *Manager) CdpAvailable() bool

CdpAvailable reports whether this Manager holds a resolved "cdp" backend eligible for routing (false when pinned to a different backend, or "cdp" is not registered). It says nothing about whether a browser session is currently live -- callers that need that should call CapabilitiesFor with a browser-app scope, or just try the operation: the CDP backend itself always answers honestly (APP_NOT_FOUND) when no session exists.

func (*Manager) Click

func (m *Manager) Click(ctx context.Context, ref core.ElementRef) (*core.ActionResult, error)

Click resolves ref and performs a click (native invoke, physical fallback), on the backend that produced ref.

func (*Manager) ClickAt

func (m *Manager) ClickAt(ctx context.Context, x, y int) (*core.ActionResult, error)

ClickAt performs a raw, ref-less coordinate click via physical input, bypassing the accessibility tree entirely -- the action path a model uses after looking at a (optionally grid-annotated) desktop_screenshot and picking a point. It is gated on Options.AllowPhysicalInput (the same knob that gates every other physical-input fallback) and validates (x,y) against the real display bounds before anything reaches the OS. The caller (internal/llm/tools/desktop_click_at.go) is responsible for the permission.Service prompt and for marking the response "source":"vision".

func (*Manager) Close

func (m *Manager) Close() error

Close releases both resolved backends' resources.

func (*Manager) DefaultDepth

func (m *Manager) DefaultDepth() int

DefaultDepth returns the configured default Observe depth.

func (*Manager) Find

func (m *Manager) Find(ctx context.Context, scope core.Scope, selectorStr string, limit int) ([]*core.Element, *core.Snapshot, error)

Find resolves selectorStr within scope, returning up to limit matches (limit <= 0 means backend default) and storing them as a fresh Snapshot so the caller gets qualified refs back. Routed exactly like Observe: one backendForScope decision for the whole call, stamped onto every result element's Backend field.

func (*Manager) Focus

func (m *Manager) Focus(ctx context.Context, ref core.ElementRef) (*core.ActionResult, error)

Focus resolves ref and focuses it (native focus, physical click fallback), on the backend that produced ref.

func (*Manager) IsPinned

func (m *Manager) IsPinned() bool

IsPinned reports whether opts.Backend explicitly pinned a single backend, disabling per-scope routing.

func (*Manager) Key

func (m *Manager) Key(ctx context.Context, ref core.ElementRef, key string) (*core.ActionResult, error)

Key sends a key/chord. When ref is empty, the key is sent globally (physical input only, backend-independent); otherwise it targets the resolved element on the backend that produced it.

func (*Manager) MaxNodes

func (m *Manager) MaxNodes() int

MaxNodes returns the configured element budget for Observe/render calls.

func (*Manager) Observe

func (m *Manager) Observe(ctx context.Context, scope core.Scope, depth int) (*core.Snapshot, error)

Observe builds a Snapshot of scope, walking the backend's Children tree up to depth levels (falling back to opts.DefaultDepth when depth <= 0) and opts.MaxNodes elements, assigning qualified e1..eN refs as it goes. Which backend serves scope is decided once, by backendForScope, and used for every Windows/Children call in this Observe -- and stamped onto every element's Backend field, so a later action against any ref this call returns routes back to the same backend (see backendForElement).

func (*Manager) Read

func (m *Manager) Read(ctx context.Context, ref core.ElementRef) (*core.Element, error)

Read resolves ref against the SnapshotStore and returns the element, enforcing the app policy for its owning application.

func (*Manager) Screenshot

func (m *Manager) Screenshot(ctx context.Context, target string, grid bool) ([]byte, string, error)

Screenshot captures target ("screen" (default), "window:<id>", or a qualified element ref meaning "crop to its bounds") via internal/uiauto/screen, honors Options.ScreenshotScale, encodes the result as PNG, and passes it through the shared internal/imageopt pipeline exactly as the other image-producing tools do (see internal/llm/tools/browser_screenshot.go, image_crop.go). When grid is true, a light coordinate grid + axis labels (internal/uiauto/vision. DrawGrid) is overlaid before any scaling, so labels always read real, unscaled screen coordinates -- the coordinates desktop_click_at expects.

func (*Manager) ScreenshotScale

func (m *Manager) ScreenshotScale() float64

ScreenshotScale returns the configured screenshot scaling factor.

func (*Manager) Scroll

func (m *Manager) Scroll(ctx context.Context, ref core.ElementRef, amount int) (*core.ActionResult, error)

Scroll resolves ref and scrolls it by amount (native scroll, physical fallback), on the backend that produced ref.

func (*Manager) SemanticAvailable

func (m *Manager) SemanticAvailable() bool

SemanticAvailable reports whether the resolved backend can plausibly answer this session's desktop_observe/desktop_find calls at all (Accessibility+UIActions capabilities). It is a coarse, cheap signal tools use to nudge the model toward the semantic path first -- vision fallback is genuinely a fallback, not a shortcut -- without forcing an actual (expensive) Find call on every desktop_click_at invocation. A false result means the semantic path is guaranteed unusable right now; a true result does not guarantee a specific selector will resolve, only that trying it is worthwhile before reaching for coordinates.

func (*Manager) Type

func (m *Manager) Type(ctx context.Context, ref core.ElementRef, text string) (*core.ActionResult, error)

Type resolves ref and enters text (native setvalue/type, physical fallback), on the backend that produced ref.

func (*Manager) Wait

func (m *Manager) Wait(ctx context.Context, scope core.Scope, selectorStr string, cond core.Condition, timeout time.Duration) (*core.Element, error)

Wait polls selectorStr within scope until cond is satisfied or timeout elapses (timeout <= 0 uses opts.ActionTimeout), against whichever backend backendForScope routes scope to.

func (*Manager) Windows

func (m *Manager) Windows(ctx context.Context, appID string) ([]core.WindowInfo, error)

Windows lists the windows of appID (or all apps when appID is empty), enforcing the allow/deny policy for appID. When appID names the CDP virtual browser app (or is empty, meaning "every app"), the connected browser's pages are included alongside the OS backend's windows; otherwise only the OS backend is asked, so a native appID never triggers a pointless CDP round trip.

type Options

type Options struct {
	// Backend selects the backend to resolve from the Registry: "auto" (the
	// default), or an explicit name ("atspi", "uia", "ax", "cdp", "null").
	Backend string

	// MaxNodes caps how many elements a single Observe call pulls from the
	// backend and how many a rendered response shows.
	MaxNodes int
	// DefaultDepth is used by Observe when the caller passes depth <= 0.
	DefaultDepth int

	// ActionTimeout is the default timeout for a single action/wait.
	ActionTimeout time.Duration
	// SnapshotTTL is how long an observed snapshot stays resolvable.
	SnapshotTTL time.Duration

	// AllowPhysicalInput enables the ActionResolver's synthetic
	// mouse/keyboard fallback when a native action is unsupported or fails.
	AllowPhysicalInput bool

	// AllowedApps, when non-empty, restricts every operation to apps whose
	// id or name matches (case-insensitively) one of these entries.
	AllowedApps []string
	// DeniedApps blocks apps whose id or name matches, regardless of
	// AllowedApps.
	DeniedApps []string

	// ScreenshotScale downsizes desktop_screenshot output on top of the
	// shared imageopt pipeline. 1.0 means "no extra scaling".
	ScreenshotScale float64

	// Inert makes the Manager a fully no-op automation surface: no
	// physical-input layer and no screen capture, so every OS-touching
	// entry point reports PLATFORM_NOT_SUPPORTED. It is set by
	// OptionsFromConfig when the user pins the "null" backend, which is
	// how the desktop policy says "no real desktop automation". Backend
	// == "null" alone does NOT imply it: uiauto's own tests pin "null" as
	// a neutral accessibility backend while injecting a fake physical
	// input layer, and that must keep working.
	Inert bool
}

Options configures a Manager.

func OptionsFromConfig

func OptionsFromConfig(it config.InternalToolsConfig) Options

OptionsFromConfig converts the relevant InternalToolsConfig fields into Options.

Directories

Path Synopsis
Package core provides the platform-independent building blocks of the Pando Desktop Controller: a normalized accessibility-tree element model, a selector DSL, snapshot/ref bookkeeping, action resolution and rendering for LLM consumption.
Package core provides the platform-independent building blocks of the Pando Desktop Controller: a normalized accessibility-tree element model, a selector DSL, snapshot/ref bookkeeping, action resolution and rendering for LLM consumption.
Package events defines the backend-agnostic UI event subscription model for the Pando Desktop Controller.
Package events defines the backend-agnostic UI event subscription model for the Pando Desktop Controller.
Package input implements the cross-platform physical (synthetic) mouse/keyboard input layer of the Pando Desktop Controller.
Package input implements the cross-platform physical (synthetic) mouse/keyboard input layer of the Pando Desktop Controller.
platform
browser
Package browser implements the Chrome DevTools Protocol accessibility backend (registered as "cdp") for the Pando Desktop Controller (internal/uiauto): when the UI in question is a Chrome/Edge/Chromium page, it serves the normalized core.Element model straight from the CDP Accessibility/DOM domains of an already-running browser session, instead of the OS accessibility API.
Package browser implements the Chrome DevTools Protocol accessibility backend (registered as "cdp") for the Pando Desktop Controller (internal/uiauto): when the UI in question is a Chrome/Edge/Chromium page, it serves the normalized core.Element model straight from the CDP Accessibility/DOM domains of an already-running browser session, instead of the OS accessibility API.
darwin
Package darwin implements the macOS AXUIElement core.Backend ("ax") for the Pando Desktop Controller, described in Phase 5 of pando/plans/desktop_controller_uiauto_plan.md.
Package darwin implements the macOS AXUIElement core.Backend ("ax") for the Pando Desktop Controller, described in Phase 5 of pando/plans/desktop_controller_uiauto_plan.md.
linux
Package linux implements the Linux AT-SPI2 accessibility backend for the Pando Desktop Controller (internal/uiauto).
Package linux implements the Linux AT-SPI2 accessibility backend for the Pando Desktop Controller (internal/uiauto).
windows
Package windows implements the Windows UI Automation (UIA) accessibility backend for the Pando Desktop Controller (internal/uiauto), Phase 4 of [pando/plans/desktop_controller_uiauto_plan.md].
Package windows implements the Windows UI Automation (UIA) accessibility backend for the Pando Desktop Controller (internal/uiauto), Phase 4 of [pando/plans/desktop_controller_uiauto_plan.md].
Package portal implements the shared plumbing for talking to the XDG desktop portal (org.freedesktop.portal.Desktop) that internal/uiauto's Linux Wayland input (internal/uiauto/input) and screen capture (internal/uiauto/screen) packages both need: the well-known bus/object names, org.freedesktop.portal.Request handle-token generation and Response-signal handling, response-code -> core.DesktopError mapping, and consent timeouts.
Package portal implements the shared plumbing for talking to the XDG desktop portal (org.freedesktop.portal.Desktop) that internal/uiauto's Linux Wayland input (internal/uiauto/input) and screen capture (internal/uiauto/screen) packages both need: the well-known bus/object names, org.freedesktop.portal.Request handle-token generation and Response-signal handling, response-code -> core.DesktopError mapping, and consent timeouts.
Package screen implements the cross-platform screen capture layer of the Pando Desktop Controller, used by Manager.Screenshot (internal/uiauto/manager.go).
Package screen implements the cross-platform screen capture layer of the Pando Desktop Controller, used by Manager.Screenshot (internal/uiauto/manager.go).
Package vision implements the Pando Desktop Controller's vision-fallback path: when a region of the screen exposes no usable accessibility semantics (a canvas app, a remote desktop window, a game, an app with a broken accessibility implementation), the agent still needs a way to act.
Package vision implements the Pando Desktop Controller's vision-fallback path: when a region of the screen exposes no usable accessibility semantics (a canvas app, a remote desktop window, a game, an app with a broken accessibility implementation), the agent still needs a way to act.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL