Documentation
¶
Overview ¶
Package uitree renders an accessibility tree into the compact, uid-annotated text form that the agent reads and acts on.
It is shared by browser-use (CDP Accessibility.getFullAXTree) and computer-use (macOS AXUIElement). Both produce a tree of role/name/value/state nodes, so both get the same on-screen vocabulary: an agent that has learned to read a browser snapshot can read a native app snapshot with no new concepts.
Callers adapt their backend's node shape into []Node and call Build. The uid minting, generation stamping, role vocabulary, state rendering and elision all live here so the two consumers cannot drift.
See internal-doc/computer-use-design.md §3.1.
Index ¶
Constants ¶
const DefaultMaxLines = 400
DefaultMaxLines bounds a snapshot so one enormous window cannot blow the context window.
Variables ¶
var ContextRoles = map[string]bool{ "heading": true, "img": true, "image": true, "alert": true, "dialog": true, "status": true, "tabpanel": true, "cell": true, "columnheader": true, "rowheader": true, "listitem": true, "window": true, "sheet": true, "group": true, "toolbar": true, "statictext": true, }
ContextRoles are shown without a uid to give the model structure.
var InteractiveRoles = map[string]bool{ "button": true, "link": true, "textbox": true, "searchbox": true, "checkbox": true, "radio": true, "combobox": true, "listbox": true, "option": true, "menuitem": true, "menuitemcheckbox": true, "menuitemradio": true, "tab": true, "switch": true, "slider": true, "spinbutton": true, "textfield": true, "textarea": true, "MenuListPopup": true, "menubutton": true, "popupbutton": true, "incrementor": true, "disclosuretriangle": true, "row": true, "colorwell": true, }
InteractiveRoles are roles that receive a uid and can be targeted by an action. Aligned with what Codex/Claude snapshots mark as actionable.
Both the CDP and the macOS AX vocabularies are normalized into these names by their adapters (AXButton → button, AXTextField → textbox, …).
Functions ¶
func RenderIdentityAndActions ¶
RenderIdentityAndActions exposes native semantic identifiers and secondary AX actions without making the model guess action names. It is separate from RenderStates so existing state semantics stay unchanged for browser trees.
func RenderStates ¶
RenderStates renders the interesting states of a node as a " (a, b=c)" suffix.
Types ¶
type Node ¶
type Node struct {
ID string
Role string
Name string
Value string
States []State
// SemanticID is a backend-provided stable identifier such as an AXIdentifier.
// Actions are named secondary accessibility actions; click/AXPress is omitted.
SemanticID string
Actions []string
ChildIDs []string
Ref int64
Ignored bool
}
Node is a backend-neutral accessibility node.
Ref is the backend's own handle for the element (a CDP backendDOMNodeId, an AX element index); Build only stores it, never interprets it. A node is only eligible for a uid when Ref != 0, since a uid the backend cannot resolve back to an element is worse than no uid at all.
type Snapshot ¶
type Snapshot struct {
Text string
UIDs map[string]int64 // uid → Node.Ref
Gen int
// Refs is the reverse binding (Ref → uid) for every element in this
// snapshot. Pass it back as `known` on the next Build for the same surface.
Refs map[int64]string
// NextUID is the first uid number this snapshot did not use.
NextUID int
}
Snapshot is one serialized tree state.
A uid names an *element*, not a position. It is bound to a node's Ref for as long as that element keeps appearing, and is **never reused** once the element is gone. That is what makes "absent from the latest snapshot" mean "stale".
This is load-bearing, and the obvious implementation gets it wrong. If each snapshot restarts its uid sequence at e1, a uid is silently *rebound* rather than invalidated: the model reads `[e1] button "New Note"`, the tree changes, the next snapshot mints `[e1] button "Delete All Notes"`, and an action carrying the remembered `e1` resolves cleanly — to the wrong button. Presence in the latest map is then a perfect disguise for staleness, and the check that is supposed to prevent a misdirected click is the thing that permits it.
Binding uid↔Ref fixes it in both directions at once: a surviving element keeps its uid (so a remembered uid stays valid, because it really is the same element, and so consecutive snapshots diff to nothing), while a departed element's uid is retired forever (so a remembered uid for it is simply absent, which resolveUID already rejects).
Refs and NextUID carry the binding forward to the next Build on the session.
func Build ¶
func Build(nodes []Node, filter string, gen int, maxLines int, known map[int64]string, uidBase int) *Snapshot
Build serializes a tree into compact uid-annotated text.
filter "interactive" (default) emits interactive + context nodes; "all" also emits static text. Nodes are walked from every root (a node no other node claims as a child) in document order, so uids are stable for a stable tree.
known is the previous snapshot's Ref→uid binding for this surface (nil on the first call): an element still present keeps its uid. uidBase is the session's monotonic counter, from which brand-new elements are numbered; the caller advances it to Snapshot.NextUID.
Passing known=nil and uidBase=0 every time reintroduces uid rebinding — see the Snapshot doc comment for why that is a correctness bug and not a cosmetic one.
Source Files