Documentation
¶
Overview ¶
Package browser gives an agentloop sandbox a `browser` global that drives a real browser: navigate, click, type, read text, and — via Set-of-Marks — enumerate the page's interactive elements as numbered ids the model can act on without ever producing a CSS selector or a pixel coordinate.
The package holds no browser dependency of its own. Everything here talks to a Driver, a ten-method seam an implementation satisfies; the chromedp-backed one lives in the sibling module github.com/mind-vm/agentloop/browser/chrome, which is a separate Go module precisely so agentloop's own dependency set stays small for the deployments that never open a browser.
Wiring follows the same shape as ext's packs — an application closes an agentloop.Capability over its Driver:
chrome := chrome.New(chrome.Options{})
defer chrome.Close()
cap := agentloop.Capability{
Name: "browser",
Description: "Drive a real browser",
Build: func(bc agentloop.BuildContext) ([]sandbox.Pack, error) {
return []sandbox.Pack{browser.Pack(bc.Ctx, chrome, nil)}, nil
},
}
The Driver is the application's to own and close: Capability.Build has no teardown hook, so a per-session browser belongs to a SandboxBuilder (whose Build returns a cleanup func), and a shared one belongs to the process.
Set-of-Marks ¶
A vision model can describe a screenshot but cannot name a CSS selector, and coordinates it invents are unreliable. Set-of-Marks fixes the vocabulary problem: mark() enumerates every visible interactive element, draws a numbered box over each, and returns {id, tag, role, name, x, y, w, h} for all of them. The model picks an id; clickMark(id) / typeMark(id, text) resolve it back to the DOM node. That listing is useful on its own — an agent with no vision backend at all can navigate a page from the returned metadata, which is why mark() does not require one.
Ids are assigned in document order at mark() time and are stable only until the next mark(): any DOM change renumbers them. A navigation drops them entirely, and the pack enforces that — clickMark after goto errors rather than clicking whatever now holds that number.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Pack ¶
Pack exposes a `browser` global that drives d, one page, one call at a time. Every primitive is synchronous: it returns when the browser has finished the step, so a script reads as a straight sequence of page actions.
Every call is policy-gated under tool name "browser", with the primitive's name in args["action"] and, for goto, the target in args["url"] — sandbox.DefaultPolicy runs browser URLs through the same rules it applies to fetch, so an agent cannot reach a private address by opening it in a tab instead. Note that this gates navigation, not the page: a page the agent has been allowed to open can redirect itself, and browser.eval() can set location. Treat the allowlist as a statement about where the agent may steer, not as a network boundary.
vision may be nil, in which case ask() and askMarks() are neither registered nor advertised; mark() works either way. ctx is the per-run context — it bounds every call and is what the driver aborts on when the turn ends.
Example ¶
ExamplePack shows the wiring an application supplies: a Driver it owns and closes, closed over by a Capability. Nothing here needs a real browser — swap the fake for chrome.New(chrome.Options{}) from the browser/chrome module and the same Capability drives Chrome.
driver := &fakeDriver{} // in production: chrome.New(chrome.Options{})
cap := agentloop.Capability{
Name: "browser",
Description: "Drive a real browser",
Build: func(bc agentloop.BuildContext) ([]sandbox.Pack, error) {
// bc.Ctx is the per-run context: it bounds every browser
// call without outliving, or ending, the browser itself.
return []sandbox.Pack{browser.Pack(bc.Ctx, driver, nil)}, nil
},
}
builder := &agentloop.DefaultSandboxBuilder{
Capabilities: append(agentloop.DefaultCapabilities(nil, ""), cap),
}
_ = builder
// What the model then writes, turn by turn:
s := sandbox.New(browser.Pack(context.Background(), driver, nil))
s.SetPolicy(sandbox.DefaultPolicy{URLAllowPrefixes: []string{"https://example.com/"}})
_, _ = s.Execute(`
browser.goto("https://example.com/search");
var field = browser.mark().find(m => m.role === "textbox" || m.tag === "input");
browser.typeMark(field.id, "socks");
`)
Types ¶
type Driver ¶
type Driver interface {
Navigate(ctx context.Context, url string) error
// Click dispatches a real mouse click at the centre of the first
// element matching the CSS selector, waiting for it first.
Click(ctx context.Context, selector string) error
// SendKeys focuses the first element matching the CSS selector and
// types text into it as key events.
SendKeys(ctx context.Context, selector, text string) error
// WaitVisible blocks until the CSS selector matches an element
// that is present and visible.
WaitVisible(ctx context.Context, selector string) error
// WaitReady blocks until the CSS selector matches an element that
// is present in the DOM, visible or not.
WaitReady(ctx context.Context, selector string) error
// Text returns the visible text of the first element matching the
// CSS selector.
Text(ctx context.Context, selector string) (string, error)
// Value returns the `value` property of the first element matching
// the CSS selector.
Value(ctx context.Context, selector string) (string, error)
// Eval evaluates expr in the page and decodes its completion value
// into out, which is a pointer with encoding/json semantics. A nil
// out evaluates expr and discards the result.
Eval(ctx context.Context, expr string, out any) error
// KeyEvent types text into whatever currently has focus, as key
// events the page sees as real input.
KeyEvent(ctx context.Context, text string) error
// Screenshot captures the current viewport as PNG bytes.
Screenshot(ctx context.Context) ([]byte, error)
}
Driver is the seam between the browser pack and a real browser. It is deliberately small — ten methods, all synchronous, all taking the per-run context — so an implementation on top of chromedp, Playwright, WebDriver, or a test fake is a short file.
Every method blocks until the browser has finished the step, including any navigation it triggered. Callers hold no session handle: a Driver is bound to exactly one page, and the pack drives that page in sequence.
Implementations must honour ctx cancellation by aborting the in-flight command, and must not tear the browser down when it fires — ctx is the agent's turn, which is shorter than the browser's life.
type Mark ¶
type Mark struct {
ID int `json:"id"`
Tag string `json:"tag"`
Role string `json:"role"`
Name string `json:"name"`
X int `json:"x"`
Y int `json:"y"`
W int `json:"w"`
H int `json:"h"`
}
Mark is one interactive element found by mark(). X and Y are the element's centre in viewport coordinates, W and H its size, all in CSS pixels rounded to integers. Name is the element's best available accessible label — aria-label, alt, title, placeholder, value, or trimmed inner text, in that order — truncated to 80 characters.
type Vision ¶
Vision answers a natural-language question about a screenshot. The pack takes one as a callback rather than importing a model client, so this package stays free of any vision-provider dependency — an application closes it over whichever model it already has configured (agentloop's own llm.Client, Gemini, whatever).
A nil Vision is valid and common: the pack then registers no ask() / askMarks() and advertises neither in the prompt, so the model is never shown a primitive that would fail. mark() still works — its structured element listing is what most navigation actually needs.
Source Files
Directories