Documentation
¶
Overview ¶
internal/browser/session/cdp_executor.go This file implements the humanoid.Executor interface using the Chrome DevTools Protocol (CDP). It acts as a bridge, translating the browser-agnostic commands from the humanoid simulation (like "dispatch mouse event" or "get element geometry") into concrete chromedp actions that can be executed by the browser session.
The cdpExecutor is responsible for the low-level details of interacting with the browser, such as constructing CDP command parameters and evaluating JavaScript snippets for tasks like geometry retrieval. It ensures that all browser interactions initiated by the humanoid are correctly executed and their results are returned in the expected format.
internal/browser/session/context_utils.go
internal/browser/session/harvester.go This file implements the Harvester, a component responsible for collecting network traffic and console log data from a browser session. It listens to events from the Chrome DevTools Protocol (CDP), such as network requests, responses, and console API calls.
The collected data is then aggregated and transformed into standard formats, specifically a HAR (HTTP Archive) file for network traffic and a structured log format for console messages. The Harvester also provides functionality to wait for network idle periods, which is crucial for synchronizing automation scripts with page loading activity.
internal/browser/session/interaction.go This file implements the high-level user interaction methods for a Session, such as navigating to a URL, clicking elements, typing text, and scrolling. These methods serve as the primary API for controlling the browser page.
When a `humanoid` is attached to the session, these actions are delegated to it to simulate realistic, human-like behavior. If no humanoid is present, the methods fall back to using standard, direct `chromedp` actions. This design allows for a flexible execution model, supporting both simple automation and sophisticated, human-like interaction simulation through the same API.
Each method is responsible for managing its own operational context and timeouts, ensuring that long-running actions (like navigation) can be cancelled without terminating the entire session.
internal/browser/session/interactor.go This file defines the Interactor, a component responsible for performing automated, intelligent, and recursive interactions with a web page. Its goal is to explore a web application's state space by discovering and interacting with UI elements like a human user would.
The core logic operates in a depth-first recursive manner: 1. Discover all new, unique, and interactable elements on the current page. 2. Prioritize these elements (e.g., fill in input fields before clicking submit). 3. Interact with a subset of these elements (e.g., click, type, select). 4. After an interaction, wait for the page to stabilize (e.g., for JS updates). 5. Recurse to the next depth level to discover new elements that may have appeared.
To avoid data races and handle dynamic content robustly, the Interactor uses an atomic JavaScript-based discovery mechanism. It tags elements in the DOM, takes a snapshot of their state, and then uses unique selectors for subsequent interactions. This ensures that interactions target the correct elements, even if the DOM is modified between discovery and action.
internal/browser/session/interfaces.go
internal/browser/session/session.go This file defines the `Session` struct, which represents a single, isolated browser tab. It serves as the primary orchestrator for all browser-related activities, integrating various components like the `Harvester` for data collection, the `Humanoid` for realistic user interaction, and the `Interactor` for automated page exploration.
The Session manages the lifecycle of the browser tab, from initialization to closure. It provides a high-level API for actions (e.g., Navigate, Click, Type) and artifact collection, while also implementing the lower-level `ActionExecutor` interface required by its sub-components. This centralizes context management and error handling, providing a robust foundation for browser automation tasks.
Index ¶
- Variables
- func CalculateHeaderSize(headers network.Headers) int64
- func CombineContext(ctx1, ctx2 context.Context) (context.Context, context.CancelFunc)
- func ConvertCDPCookies(cookieHeader string) []schemas.HARCookie
- func ConvertCDPHeaders(headers network.Headers) []schemas.NVPair
- func ConvertCDPTimings(t *network.ResourceTiming) schemas.Timings
- func Detach(ctx context.Context) context.Context
- func GetHeader(headers network.Headers, key string) string
- func IsTextMime(mimeType string) bool
- type ActionExecutor
- type Harvester
- type Interactor
- type Session
- func (s *Session) AddFinding(ctx context.Context, finding schemas.Finding) error
- func (s *Session) Click(ctx context.Context, selector string) error
- func (s *Session) Close(ctx context.Context) error
- func (s *Session) CollectArtifacts(ctx context.Context) (*schemas.Artifacts, error)
- func (s *Session) DispatchMouseEvent(ctx context.Context, data schemas.MouseEventData) error
- func (s *Session) DispatchStructuredKey(ctx context.Context, data schemas.KeyEventData) error
- func (s *Session) ExecuteScript(ctx context.Context, script string, args []interface{}) (json.RawMessage, error)
- func (s *Session) ExposeFunction(ctx context.Context, name string, function interface{}) error
- func (s *Session) GetContext() context.Context
- func (s *Session) GetElementGeometry(ctx context.Context, selector string) (*schemas.ElementGeometry, error)
- func (s *Session) ID() string
- func (s *Session) Initialize(ctx context.Context, taintTemplate, taintConfig string) error
- func (s *Session) InjectScriptPersistently(ctx context.Context, script string) error
- func (s *Session) Interact(ctx context.Context, config schemas.InteractionConfig) error
- func (s *Session) Navigate(ctx context.Context, url string) error
- func (s *Session) RunActions(ctx context.Context, actions ...chromedp.Action) error
- func (s *Session) RunBackgroundActions(ctx context.Context, actions ...chromedp.Action) error
- func (s *Session) ScrollPage(ctx context.Context, direction string) error
- func (s *Session) SendKeys(ctx context.Context, keys string) error
- func (s *Session) SetOnClose(fn func())
- func (s *Session) Sleep(ctx context.Context, d time.Duration) error
- func (s *Session) Submit(ctx context.Context, selector string) error
- func (s *Session) Type(ctx context.Context, selector string, text string) error
- func (s *Session) WaitForAsync(ctx context.Context, milliseconds int) error
- type StabilizationFunc
Constants ¶
This section is empty.
Variables ¶
var ErrStaleElement = errors.New("element is stale or detached from the document")
ErrStaleElement is a sentinel error used to indicate that an interaction failed because the target DOM element is no longer attached to the document or has otherwise become invalid, typically due to a navigation or a dynamic DOM update.
Functions ¶
func CalculateHeaderSize ¶
func CombineContext ¶
CombineContext creates a new context that is a child of a primary context (`ctx1`) but is also cancelled when a secondary context (`ctx2`) is cancelled. This is a critical utility for working with `chromedp`, where `ctx1` is typically the long-lived session or allocator context carrying essential connection values, and `ctx2` is a shorter-lived operational context with a specific deadline (e.g., for a single navigation or action).
By deriving the new context from `ctx1`, it inherits all necessary values. By monitoring `ctx2` in a separate goroutine, it ensures that the combined context respects the operational timeout of `ctx2`.
Returns the combined context and its cancel function. The caller is responsible for calling the cancel function to release the monitoring goroutine.
func ConvertCDPCookies ¶
Updated return type to []schemas.HARCookie to match the HAR schema requirements.
func ConvertCDPTimings ¶
func ConvertCDPTimings(t *network.ResourceTiming) schemas.Timings
ConvertCDPTimings converts network.ResourceTiming to HAR Timings format.
func Detach ¶
Detach creates and returns a new context that is "detached" from its parent's lifecycle. The new context inherits all the values of the parent `ctx` but will not be cancelled when `ctx` is cancelled, nor will it expire when `ctx`'s deadline is reached.
This is particularly useful for launching background tasks or performing cleanup operations that need to continue running even after the originating operation's context has been cancelled.
func GetHeader ¶
GetHeader performs a case insensitive search for a header and returns its string value.
func IsTextMime ¶
Types ¶
type ActionExecutor ¶
type ActionExecutor interface {
// RunActions executes a sequence of browser actions within a given operational
// context. The implementation is responsible for combining this context with
// the long-lived session context to ensure actions have the necessary CDP
// connection information.
RunActions(ctx context.Context, actions ...chromedp.Action) error
// RunBackgroundActions executes a sequence of browser actions in a "detached"
// context. This is crucial for tasks that must not be cancelled when the
// parent operational context finishes, such as asynchronously fetching a
// response body after a request has already completed.
RunBackgroundActions(ctx context.Context, actions ...chromedp.Action) error
}
ActionExecutor defines a generic interface for executing browser actions, abstracting the underlying implementation (which is `chromedp` in this case). This allows components like the `Interactor` and `Harvester` to request browser operations without being directly coupled to the `Session` struct or `chromedp`.
type Harvester ¶
type Harvester struct {
// contains filtered or unexported fields
}
Harvester is responsible for collecting network traffic (as a HAR file) and console log messages from a browser session. It attaches listeners to the Chrome DevTools Protocol (CDP) and processes events related to network requests, page lifecycle, and console output.
func NewHarvester ¶
func NewHarvester(ctx context.Context, logger *zap.Logger, captureBodies bool, executor ActionExecutor) *Harvester
NewHarvester creates and initializes a new Harvester. It requires a parent context for its own lifecycle, a logger, a flag to enable body capture, and an ActionExecutor to safely perform CDP actions in the background (e.g., fetching response bodies).
func (*Harvester) Start ¶
Start attaches the Harvester's listeners to the CDP event stream of the provided context (typically the session context). From this point on, it will capture all relevant network and console events.
func (*Harvester) Stop ¶
Stop gracefully shuts down the Harvester. It cancels its internal context, waits for any in-flight background operations (like fetching response bodies) to complete, and then generates the final HAR file and console log slice from the collected data.
func (*Harvester) WaitNetworkIdle ¶
WaitNetworkIdle blocks until there have been no active network requests for a specified `quietPeriod`. This is a crucial synchronization primitive for browser automation, allowing scripts to wait until a page has fully loaded all its dynamic resources before proceeding.
It works by monitoring the number of in-flight requests. When the count drops to zero, a timer is started. If the count remains at zero until the timer fires, the network is considered idle. If a new request starts, the timer is stopped and reset.
type Interactor ¶
type Interactor struct {
// contains filtered or unexported fields
}
Interactor orchestrates the automated, intelligent exploration of a web page. It discovers interactive elements, prioritizes them, and performs actions (clicking, typing) in a recursive, depth-first manner to uncover new states of the application.
func NewInteractor ¶
func NewInteractor( logger *zap.Logger, rng *rand.Rand, h *humanoid.Humanoid, stabilizeFn StabilizationFunc, executor ActionExecutor, sessionCtx context.Context, ) *Interactor
NewInteractor creates and initializes a new Interactor.
Parameters:
- logger: The logger for the Interactor to use.
- rng: The random number generator to use for element shuffling.
- h: An optional `humanoid.Humanoid` instance for realistic interactions. If nil, the Interactor will fall back to direct CDP actions.
- stabilizeFn: A function that will be called to wait for the page to stabilize after interactions.
- executor: An `ActionExecutor` (typically the Session) used to run browser actions.
- sessionCtx: The master context for the browser session, used for cleanup tasks that must outlive specific interaction operations.
func (*Interactor) Interact ¶
func (i *Interactor) Interact(ctx context.Context, config schemas.InteractionConfig) error
Interact is the main entry point for starting the automated interaction process. It initiates a recursive, depth-first exploration of the web page, starting at depth 0.
Parameters:
- ctx: The context for the entire interaction sequence. A timeout should be set on this context to prevent indefinite execution.
- config: The configuration that governs the interaction, such as the maximum recursion depth and the number of actions to perform at each depth.
Returns an error if a critical part of the interaction fails or if the context is cancelled.
type Session ¶
type Session struct {
// contains filtered or unexported fields
}
Session represents a single, isolated browser tab and serves as the primary orchestrator for all browser-related activities. It implements the `schemas.SessionContext` interface, providing a high-level API for browser interactions, and the `ActionExecutor` interface, allowing its sub-components to perform browser actions safely.
The Session integrates and manages the lifecycle of various components, including:
- Harvester: For collecting network traffic and console logs.
- Humanoid: For simulating realistic user interactions.
- Interactor: For automated, intelligent exploration of the page.
func NewSession ¶
func NewSession( ctx context.Context, cancel context.CancelFunc, cfg config.Interface, persona schemas.Persona, logger *zap.Logger, onClose func(), findingsChan chan<- schemas.Finding, ) (*Session, error)
NewSession creates a new, uninitialized Session.
Parameters:
- ctx: The master context that governs the entire lifetime of the session. This should be a context derived from `chromedp.NewContext`.
- cancel: The cancel function for the master context.
- cfg: The application configuration.
- persona: The user persona to be emulated in this session.
- logger: The logger for the session to use.
- onClose: An optional callback function to be executed when the session is closed.
- findingsChan: A channel for reporting security findings discovered during the session.
func (*Session) AddFinding ¶
AddFinding sends a finding to the central findings channel safely.
func (*Session) Click ¶
Click performs a click action on the DOM element matching the specified selector. If a `humanoid` is attached, it will perform a realistic `IntelligentClick`, which includes human-like mouse movements, pauses, and physical inaccuracies. Otherwise, it falls back to a direct `chromedp.Click` action after ensuring the element is visible.
Parameters:
- ctx: The context for the click operation.
- selector: The CSS selector of the element to click.
Returns an error if the element is not found, the click fails, or the operation is cancelled.
func (*Session) CollectArtifacts ¶
CollectArtifacts gathers the HAR, DOM, Console Logs, and Storage state.
func (*Session) DispatchMouseEvent ¶
DispatchMouseEvent directly dispatches a mouse event. Delegates to the executor.
func (*Session) DispatchStructuredKey ¶
DispatchStructuredKey dispatches a structured key event. Delegates to the executor.
func (*Session) ExecuteScript ¶
func (s *Session) ExecuteScript(ctx context.Context, script string, args []interface{}) (json.RawMessage, error)
ExecuteScript runs JS in the current document context.
Updated signature to match schemas.SessionContext interface (InvalidIfaceAssign error).
func (*Session) ExposeFunction ¶
ExposeFunction allows Go functions to be called from the browser's JavaScript context.
func (*Session) GetContext ¶
GetContext returns the underlying master context for the session.
func (*Session) GetElementGeometry ¶
func (s *Session) GetElementGeometry(ctx context.Context, selector string) (*schemas.ElementGeometry, error)
GetElementGeometry retrieves geometry. Delegates to the executor.
func (*Session) Initialize ¶
Initialize performs the setup for the session, preparing it for interaction. This is a critical step that must be called before any other actions are taken.
The process includes:
- Establishing the initial CDP connection to the browser tab.
- Starting the `Harvester` to begin collecting network and console data.
- Applying browser stealth evasions and emulating the specified `Persona`.
- Initializing the `Humanoid` and `Interactor` components.
- Injecting the IAST (Interactive Application Security Testing) taint analysis shim script, if configured.
- Setting any custom HTTP headers.
- Positioning the mouse cursor at the center of the viewport.
Parameters:
- ctx: An operational context with a timeout for the initialization process.
- taintTemplate: The JavaScript template for the taint tracking shim.
- taintConfig: The JSON configuration for the taint tracking shim.
Returns an error if any part of the initialization fails.
func (*Session) InjectScriptPersistently ¶
InjectScriptPersistently adds a script to evaluate on new documents.
func (*Session) Interact ¶
Interact initiates an automated, exploratory interaction with the current page. It delegates the complex logic of discovering and interacting with elements to the `Interactor` component. This method is the entry point for running a full, automated crawl-and-interact sequence on a page.
Parameters:
- ctx: The context for the entire automated interaction sequence.
- config: The configuration specifying the parameters for the interaction, such as maximum depth and element filtering.
Returns an error if the interaction fails or is cancelled.
func (*Session) Navigate ¶
Navigate instructs the browser to load the specified URL. This is a comprehensive action that includes not only the navigation itself but also a subsequent "stabilization" period, where it waits for network activity to cease.
If a `humanoid` is attached to the session, it will also perform a realistic cognitive pause after the page has stabilized.
Parameters:
- ctx: The context for the navigation operation. A timeout specific to navigation will be derived from this context.
- url: The URL to navigate to.
Returns an error if the navigation fails, times out, or is cancelled.
func (*Session) RunActions ¶
RunActions executes a series of `chromedp.Action` funcs for the session. It implements the `ActionExecutor` interface and is the primary method for running CDP commands.
It robustly handles context management by combining the long-lived session context (which holds the CDP connection values) with a shorter-lived operational context (which provides a deadline for the specific action). It also includes logic to correctly prioritize context cancellation errors and to ignore spurious "context canceled" errors that can occur when an action triggers a page navigation.
func (*Session) RunBackgroundActions ¶
RunBackgroundActions executes a series of `chromedp.Action` funcs in a way that is detached from the main session's lifecycle. It implements the `ActionExecutor` interface.
This is critical for operations that must complete even if the session or the originating operational context is cancelled. For example, the `Harvester` uses this to fetch response bodies in the background, ensuring that the fetches can complete even if the user navigates away from the page or closes the session.
It works by creating a `valueOnlyContext` that inherits the necessary CDP connection info but ignores cancellation signals from the main session context.
func (*Session) ScrollPage ¶
ScrollPage simulates scrolling the page in a given direction ("up", "down", "top", "bottom"). It executes a JavaScript `window.scrollBy` or `window.scrollTo` command to perform the scroll. If a `humanoid` is attached, it performs a brief, realistic pause after the scroll action.
func (*Session) SetOnClose ¶
func (s *Session) SetOnClose(fn func())
SetOnClose allows a callback function to be set after the Session has been created. This is typically used by the `BrowserManager` to register a function that cleans up its internal session map when a session is closed.
func (*Session) Sleep ¶
Sleep pauses execution for a specified duration (convenience wrapper respecting context).
func (*Session) Submit ¶
Submit attempts to submit a form associated with the given selector. This is typically equivalent to pressing Enter in an input field or clicking a submit button. It uses the `chromedp.Submit` action.
func (*Session) Type ¶
Type simulates typing text into the DOM element matching the specified selector. Before typing, it ensures the element is visible and robustly clears any existing content using JavaScript.
If a `humanoid` is attached, it simulates human-like typing, complete with realistic inter-key delays, probabilistic typos, and typo corrections. Otherwise, it falls back to a direct `chromedp.SendKeys` action.
Parameters:
- ctx: The context for the entire typing operation.
- selector: The CSS selector of the input element.
- text: The text to type.
Returns an error if the element cannot be found or interacted with, or if the operation is cancelled.
func (*Session) WaitForAsync ¶
WaitForAsync pauses the execution for a specified number of milliseconds. This can be used to wait for asynchronous operations on the page to complete (e.g., an animation or a delayed API call).
If a `humanoid` is attached, it will use `Hesitate` to simulate a more realistic pause with subtle mouse drift. Otherwise, it falls back to a simple `Sleep`. The wait can be cancelled by the provided context.
type StabilizationFunc ¶
StabilizationFunc defines the signature for a function that can be used to wait for the page to become stable (e.g., waiting for network idle or for specific animations to complete).
Source Files