Documentation
¶
Overview ¶
Package uiauto is the Pando Desktop Controller: it wires the platform-independent internal/uiauto/core building blocks (Element, Selector, Locator, SnapshotStore, Backend, ActionResolver, ...) to a concrete accessibility backend and exposes a single Manager the desktop_* agent tools drive.
Index ¶
- func Registry() *core.Registry
- func ResetShared()
- type Manager
- func (m *Manager) ActionTimeout() time.Duration
- func (m *Manager) Apps(ctx context.Context) ([]core.AppInfo, error)
- func (m *Manager) BackendName() string
- func (m *Manager) Capabilities() core.Capabilities
- func (m *Manager) CapabilitiesFor(ctx context.Context, scope core.Scope) core.Capabilities
- func (m *Manager) CdpAvailable() bool
- func (m *Manager) Click(ctx context.Context, ref core.ElementRef) (*core.ActionResult, error)
- func (m *Manager) ClickAt(ctx context.Context, x, y int) (*core.ActionResult, error)
- func (m *Manager) Close() error
- func (m *Manager) DefaultDepth() int
- func (m *Manager) Find(ctx context.Context, scope core.Scope, selectorStr string, limit int) ([]*core.Element, *core.Snapshot, error)
- func (m *Manager) Focus(ctx context.Context, ref core.ElementRef) (*core.ActionResult, error)
- func (m *Manager) IsPinned() bool
- func (m *Manager) Key(ctx context.Context, ref core.ElementRef, key string) (*core.ActionResult, error)
- func (m *Manager) MaxNodes() int
- func (m *Manager) Observe(ctx context.Context, scope core.Scope, depth int) (*core.Snapshot, error)
- func (m *Manager) Read(ctx context.Context, ref core.ElementRef) (*core.Element, error)
- func (m *Manager) Screenshot(ctx context.Context, target string, grid bool) ([]byte, string, error)
- func (m *Manager) ScreenshotScale() float64
- func (m *Manager) Scroll(ctx context.Context, ref core.ElementRef, amount int) (*core.ActionResult, error)
- func (m *Manager) SemanticAvailable() bool
- func (m *Manager) Type(ctx context.Context, ref core.ElementRef, text string) (*core.ActionResult, error)
- func (m *Manager) Wait(ctx context.Context, scope core.Scope, selectorStr string, cond core.Condition, ...) (*core.Element, error)
- func (m *Manager) Windows(ctx context.Context, appID string) ([]core.WindowInfo, error)
- type Options
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Registry ¶
Registry returns the process-wide backend Registry, so platform packages and tests can register or resolve backends without reaching into an unexported package variable.
func ResetShared ¶
func ResetShared()
ResetShared discards the shared Manager singleton so the next Shared() call rebuilds it from the current configuration. Exposed for tests.
Types ¶
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager is the Desktop Controller runtime. Since Block R (2026-08-30) it no longer binds to a single resolved core.Backend: it holds the set of backends actually available in this session -- an "OS" backend (atspi/ uia/ax, or null when none is available) and, independently, the "cdp" backend when registered -- and routes each operation to whichever one should actually serve it (see backendForScope/backendForElement). It never performs an OS call itself; all of that lives behind core.Backend.
func NewManager ¶
NewManager resolves the backends named by opts.Backend (falling back to the "null" OS backend when resolution fails, so a Manager can always be constructed) and builds the SnapshotStore/physical-input layer around them.
func Shared ¶
Shared returns the process-wide Manager singleton, lazily building it (or rebuilding it, on relevant config change) from config.Get().InternalTools, mirroring browser_session.go's shared-session pattern.
func (*Manager) ActionTimeout ¶
ActionTimeout returns the configured default action/wait timeout.
func (*Manager) BackendName ¶
BackendName returns the name of the resolved default (OS) backend ("null" when none is available). It does not reflect whether a "cdp" backend is also available for routing -- see IsPinned/CdpAvailable.
func (*Manager) Capabilities ¶
func (m *Manager) Capabilities() core.Capabilities
Capabilities reports what the default (OS) backend can actually do, in this session, merged with the physical-input/screen-capture layers. This is the same signal Manager reported before per-scope routing existed; it intentionally never folds in the CDP backend's capabilities, since those only apply to browser-scoped operations. Use CapabilitiesFor(ctx, scope) when the caller has a specific scope in mind (e.g. a browser window), so a reported capability always matches the backend that would actually serve that scope.
func (*Manager) CapabilitiesFor ¶
CapabilitiesFor reports the capabilities of whichever backend backendForScope would route scope to, merged with the same physical- input/screen-capture layers Capabilities() uses. A browser-window scope with a live CDP session reports that session's real capabilities (Accessibility/UIInspection/UIActions/Events); any other scope reports exactly Capabilities(). This never overclaims: a capability is only true here when the backend that would actually handle scope can offer it.
func (*Manager) CdpAvailable ¶
CdpAvailable reports whether this Manager holds a resolved "cdp" backend eligible for routing (false when pinned to a different backend, or "cdp" is not registered). It says nothing about whether a browser session is currently live -- callers that need that should call CapabilitiesFor with a browser-app scope, or just try the operation: the CDP backend itself always answers honestly (APP_NOT_FOUND) when no session exists.
func (*Manager) Click ¶
func (m *Manager) Click(ctx context.Context, ref core.ElementRef) (*core.ActionResult, error)
Click resolves ref and performs a click (native invoke, physical fallback), on the backend that produced ref.
func (*Manager) ClickAt ¶
ClickAt performs a raw, ref-less coordinate click via physical input, bypassing the accessibility tree entirely -- the action path a model uses after looking at a (optionally grid-annotated) desktop_screenshot and picking a point. It is gated on Options.AllowPhysicalInput (the same knob that gates every other physical-input fallback) and validates (x,y) against the real display bounds before anything reaches the OS. The caller (internal/llm/tools/desktop_click_at.go) is responsible for the permission.Service prompt and for marking the response "source":"vision".
func (*Manager) DefaultDepth ¶
DefaultDepth returns the configured default Observe depth.
func (*Manager) Find ¶
func (m *Manager) Find(ctx context.Context, scope core.Scope, selectorStr string, limit int) ([]*core.Element, *core.Snapshot, error)
Find resolves selectorStr within scope, returning up to limit matches (limit <= 0 means backend default) and storing them as a fresh Snapshot so the caller gets qualified refs back. Routed exactly like Observe: one backendForScope decision for the whole call, stamped onto every result element's Backend field.
func (*Manager) Focus ¶
func (m *Manager) Focus(ctx context.Context, ref core.ElementRef) (*core.ActionResult, error)
Focus resolves ref and focuses it (native focus, physical click fallback), on the backend that produced ref.
func (*Manager) IsPinned ¶
IsPinned reports whether opts.Backend explicitly pinned a single backend, disabling per-scope routing.
func (*Manager) Key ¶
func (m *Manager) Key(ctx context.Context, ref core.ElementRef, key string) (*core.ActionResult, error)
Key sends a key/chord. When ref is empty, the key is sent globally (physical input only, backend-independent); otherwise it targets the resolved element on the backend that produced it.
func (*Manager) Observe ¶
Observe builds a Snapshot of scope, walking the backend's Children tree up to depth levels (falling back to opts.DefaultDepth when depth <= 0) and opts.MaxNodes elements, assigning qualified e1..eN refs as it goes. Which backend serves scope is decided once, by backendForScope, and used for every Windows/Children call in this Observe -- and stamped onto every element's Backend field, so a later action against any ref this call returns routes back to the same backend (see backendForElement).
func (*Manager) Read ¶
Read resolves ref against the SnapshotStore and returns the element, enforcing the app policy for its owning application.
func (*Manager) Screenshot ¶
Screenshot captures target ("screen" (default), "window:<id>", or a qualified element ref meaning "crop to its bounds") via internal/uiauto/screen, honors Options.ScreenshotScale, encodes the result as PNG, and passes it through the shared internal/imageopt pipeline exactly as the other image-producing tools do (see internal/llm/tools/browser_screenshot.go, image_crop.go). When grid is true, a light coordinate grid + axis labels (internal/uiauto/vision. DrawGrid) is overlaid before any scaling, so labels always read real, unscaled screen coordinates -- the coordinates desktop_click_at expects.
func (*Manager) ScreenshotScale ¶
ScreenshotScale returns the configured screenshot scaling factor.
func (*Manager) Scroll ¶
func (m *Manager) Scroll(ctx context.Context, ref core.ElementRef, amount int) (*core.ActionResult, error)
Scroll resolves ref and scrolls it by amount (native scroll, physical fallback), on the backend that produced ref.
func (*Manager) SemanticAvailable ¶
SemanticAvailable reports whether the resolved backend can plausibly answer this session's desktop_observe/desktop_find calls at all (Accessibility+UIActions capabilities). It is a coarse, cheap signal tools use to nudge the model toward the semantic path first -- vision fallback is genuinely a fallback, not a shortcut -- without forcing an actual (expensive) Find call on every desktop_click_at invocation. A false result means the semantic path is guaranteed unusable right now; a true result does not guarantee a specific selector will resolve, only that trying it is worthwhile before reaching for coordinates.
func (*Manager) Type ¶
func (m *Manager) Type(ctx context.Context, ref core.ElementRef, text string) (*core.ActionResult, error)
Type resolves ref and enters text (native setvalue/type, physical fallback), on the backend that produced ref.
func (*Manager) Wait ¶
func (m *Manager) Wait(ctx context.Context, scope core.Scope, selectorStr string, cond core.Condition, timeout time.Duration) (*core.Element, error)
Wait polls selectorStr within scope until cond is satisfied or timeout elapses (timeout <= 0 uses opts.ActionTimeout), against whichever backend backendForScope routes scope to.
func (*Manager) Windows ¶
Windows lists the windows of appID (or all apps when appID is empty), enforcing the allow/deny policy for appID. When appID names the CDP virtual browser app (or is empty, meaning "every app"), the connected browser's pages are included alongside the OS backend's windows; otherwise only the OS backend is asked, so a native appID never triggers a pointless CDP round trip.
type Options ¶
type Options struct {
// Backend selects the backend to resolve from the Registry: "auto" (the
// default), or an explicit name ("atspi", "uia", "ax", "cdp", "null").
Backend string
// MaxNodes caps how many elements a single Observe call pulls from the
// backend and how many a rendered response shows.
MaxNodes int
// DefaultDepth is used by Observe when the caller passes depth <= 0.
DefaultDepth int
// ActionTimeout is the default timeout for a single action/wait.
ActionTimeout time.Duration
// SnapshotTTL is how long an observed snapshot stays resolvable.
SnapshotTTL time.Duration
// AllowPhysicalInput enables the ActionResolver's synthetic
// mouse/keyboard fallback when a native action is unsupported or fails.
AllowPhysicalInput bool
// AllowedApps, when non-empty, restricts every operation to apps whose
// id or name matches (case-insensitively) one of these entries.
AllowedApps []string
// DeniedApps blocks apps whose id or name matches, regardless of
// AllowedApps.
DeniedApps []string
// ScreenshotScale downsizes desktop_screenshot output on top of the
// shared imageopt pipeline. 1.0 means "no extra scaling".
ScreenshotScale float64
// Inert makes the Manager a fully no-op automation surface: no
// physical-input layer and no screen capture, so every OS-touching
// entry point reports PLATFORM_NOT_SUPPORTED. It is set by
// OptionsFromConfig when the user pins the "null" backend, which is
// how the desktop policy says "no real desktop automation". Backend
// == "null" alone does NOT imply it: uiauto's own tests pin "null" as
// a neutral accessibility backend while injecting a fake physical
// input layer, and that must keep working.
Inert bool
}
Options configures a Manager.
func OptionsFromConfig ¶
func OptionsFromConfig(it config.InternalToolsConfig) Options
OptionsFromConfig converts the relevant InternalToolsConfig fields into Options.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package core provides the platform-independent building blocks of the Pando Desktop Controller: a normalized accessibility-tree element model, a selector DSL, snapshot/ref bookkeeping, action resolution and rendering for LLM consumption.
|
Package core provides the platform-independent building blocks of the Pando Desktop Controller: a normalized accessibility-tree element model, a selector DSL, snapshot/ref bookkeeping, action resolution and rendering for LLM consumption. |
|
Package events defines the backend-agnostic UI event subscription model for the Pando Desktop Controller.
|
Package events defines the backend-agnostic UI event subscription model for the Pando Desktop Controller. |
|
Package input implements the cross-platform physical (synthetic) mouse/keyboard input layer of the Pando Desktop Controller.
|
Package input implements the cross-platform physical (synthetic) mouse/keyboard input layer of the Pando Desktop Controller. |
|
platform
|
|
|
browser
Package browser implements the Chrome DevTools Protocol accessibility backend (registered as "cdp") for the Pando Desktop Controller (internal/uiauto): when the UI in question is a Chrome/Edge/Chromium page, it serves the normalized core.Element model straight from the CDP Accessibility/DOM domains of an already-running browser session, instead of the OS accessibility API.
|
Package browser implements the Chrome DevTools Protocol accessibility backend (registered as "cdp") for the Pando Desktop Controller (internal/uiauto): when the UI in question is a Chrome/Edge/Chromium page, it serves the normalized core.Element model straight from the CDP Accessibility/DOM domains of an already-running browser session, instead of the OS accessibility API. |
|
darwin
Package darwin implements the macOS AXUIElement core.Backend ("ax") for the Pando Desktop Controller, described in Phase 5 of pando/plans/desktop_controller_uiauto_plan.md.
|
Package darwin implements the macOS AXUIElement core.Backend ("ax") for the Pando Desktop Controller, described in Phase 5 of pando/plans/desktop_controller_uiauto_plan.md. |
|
linux
Package linux implements the Linux AT-SPI2 accessibility backend for the Pando Desktop Controller (internal/uiauto).
|
Package linux implements the Linux AT-SPI2 accessibility backend for the Pando Desktop Controller (internal/uiauto). |
|
windows
Package windows implements the Windows UI Automation (UIA) accessibility backend for the Pando Desktop Controller (internal/uiauto), Phase 4 of [pando/plans/desktop_controller_uiauto_plan.md].
|
Package windows implements the Windows UI Automation (UIA) accessibility backend for the Pando Desktop Controller (internal/uiauto), Phase 4 of [pando/plans/desktop_controller_uiauto_plan.md]. |
|
Package portal implements the shared plumbing for talking to the XDG desktop portal (org.freedesktop.portal.Desktop) that internal/uiauto's Linux Wayland input (internal/uiauto/input) and screen capture (internal/uiauto/screen) packages both need: the well-known bus/object names, org.freedesktop.portal.Request handle-token generation and Response-signal handling, response-code -> core.DesktopError mapping, and consent timeouts.
|
Package portal implements the shared plumbing for talking to the XDG desktop portal (org.freedesktop.portal.Desktop) that internal/uiauto's Linux Wayland input (internal/uiauto/input) and screen capture (internal/uiauto/screen) packages both need: the well-known bus/object names, org.freedesktop.portal.Request handle-token generation and Response-signal handling, response-code -> core.DesktopError mapping, and consent timeouts. |
|
Package screen implements the cross-platform screen capture layer of the Pando Desktop Controller, used by Manager.Screenshot (internal/uiauto/manager.go).
|
Package screen implements the cross-platform screen capture layer of the Pando Desktop Controller, used by Manager.Screenshot (internal/uiauto/manager.go). |
|
Package vision implements the Pando Desktop Controller's vision-fallback path: when a region of the screen exposes no usable accessibility semantics (a canvas app, a remote desktop window, a game, an app with a broken accessibility implementation), the agent still needs a way to act.
|
Package vision implements the Pando Desktop Controller's vision-fallback path: when a region of the screen exposes no usable accessibility semantics (a canvas app, a remote desktop window, a game, an app with a broken accessibility implementation), the agent still needs a way to act. |