wee

package module
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 31, 2024 License: GPL-3.0 Imports: 39 Imported by: 3

README

wee

Wee is based on go-rod, and provides clean interface to write crawler.

Examples

package main

import (
	"fmt"

	"github.com/coghost/wee"
)

func main() {
	bot := wee.NewBotDefault()
	defer bot.BlockInCleanUp()

	const (
		url     = `http://www.baidu.com`
		input   = `input[id="kw"]`
		keyword = "golang"
		items   = `div.result[id] h3>a`
		noItems = `div.this_should_not_existed`
	)

	bot.MustOpen(url)
	bot.MustInput(input, "golang", wee.WithSubmit(true))

	got := bot.MustAnyElem([]string{items, noItems})
	if got == noItems {
		fmt.Printf("%s has no results\n", keyword)
		return
	}

	elems, err := bot.Elems(items)
	if err != nil {
		fmt.Printf("get items failed: %v\n", err)
	}

	for _, elem := range elems {
		fmt.Printf("%s - %s\n", elem.MustText(), *elem.MustAttribute("href"))
	}
}

See examples folder for more detailed examples.

Issues

panic: context canceled

context canceled issue usually happend when elem is bound with Timeout on bot.Elem(xxx), you can use

elem.CancelTimeout().Timeout(xxx) to rewrite previous value.

Documentation

Index

Constants

View Source 
const (
	SEP       = "@@@"
	IFrameSep = "$$$"
)
View Source 
const (
	// ZeroToSec no timeout in second
	ZeroToSec = 0
	// NapToSec a nap timeout in second
	NapToSec = 2
	// ShortToSec short timeout in second
	ShortToSec = 10
	// MediumToSec medium timeout in second
	MediumToSec = 20
	// LongToSec long timeout in second
	LongToSec = 60
)
View Source 
const (
	PT1Sec  = 1
	PT10Sec = 10
	PT20Sec = 20
	PT60Sec = 60
	// PT10MilliSec a very short timeout in millisecond
	PT10MilliSec = 0.01
)
View Source 
const (
	LongScrollStep     = 32
	MediumScrollStep   = 16
	ShortScrollStep    = 8
	QuickScrollStep    = 4
	DirectlyScrollStep = 1
)
View Source 
const (
	ActivatePageRetry = 10
)
View Source 
const PaintRects = "--show-paint-rects"

PaintRects is a flag to show paint rectangles in the browser

View Source 
const (
	// SlowMotionMillis is the default slow motion duration in milliseconds
	SlowMotionMillis = 500
)

Variables

View Source 
var (
	// ErrNoSelectorClicked is returned when no cookie acceptance selectors could be clicked successfully.
	ErrNoSelectorClicked = errors.New("failed to click any cookie acceptance selectors")

	// ErrPageLoadAfterCookies is returned when the page fails to load after attempting to accept cookies.
	ErrPageLoadAfterCookies = errors.New("page failed to load after accepting cookies")
)
View Source 
var (
	ErrSelectorEmpty   = errors.New("selector is empty")
	ErrNotInteractable = errors.New("elem not interactable")
)
View Source 
var (
	ErrCannotActivateOpenedPage = errors.New("cannot activate latest opened page")
	ErrMissingCookieFile        = errors.New("missing cookie file")
)
View Source 
var (
	ErrEmptySelector  = errors.New("empty selector")
	ErrGetElemsByText = errors.New("get elems not support by text")
	ErrCannotFindElem = errors.New("cannot find elem of selector")
)
View Source 
var ErrElemShapeBox = errors.New("cannot get element box by shape.Box()")
View Source 
var ErrEmptyCookieStr = errors.New("no cookie str found")
View Source 
var ErrGetTextAfterInput = errors.New("cannot get text after input")

Functions

func BindBotLanucher 

func BindBotLanucher(bot *Bot, options ...BotOption)

BindBotLanucher launches the browser and page for the bot. This is used when we create a bot first and launch the browser elsewhere.

func Blocked 

func Blocked()

Blocked prompts the user to decide whether to quit or continue indefinitely. It calls QuitOnTimeout with a negative value to trigger the interactive prompt.

func Confirm 

func Confirm(msg ...string) bool

Confirm displays an interactive confirmation prompt and returns the user's choice as a boolean. It uses pterm's DefaultInteractiveConfirm to show the prompt.

Parameters:

  • msg: Optional. A custom message for the confirmation prompt. If not provided, a default message is used.

Returns:

  • bool: true if the user confirms, false otherwise.

func ErrCannotFindSelector 

func ErrCannotFindSelector(sel string) error

func Filenamify 

func Filenamify(name string) string

Filenamify converts a string into a valid filename by replacing illegal characters.

Parameters:

  • name: The input string to be converted into a valid filename.

Returns:

  • A string with illegal filename characters replaced by underscores.

This function replaces common illegal filename characters (such as /, \, ", :, *, ?, <, >, |) with underscores. It's useful for sanitizing strings that will be used as filenames, ensuring they are compatible with most file systems. The function uses strutil.Replaces to perform the replacements based on a predefined map of illegal characters to underscores.

func FirstOrDefault 

func FirstOrDefault[T any](dft T, args ...T) T

FirstOrDefault returns the first value from the variadic args if provided, otherwise returns the default value. It uses a generic type T to work with any data type.

Parameters:

  • dft: The default value to return if no args are provided.
  • args: A variadic parameter of type T.

Returns:

  • The first value from args if provided, otherwise the default value dft.

func FloatAorB added in v0.1.4 

func FloatAorB(a, b float64) float64

func ForceQuitBrowser added in v0.1.4 

func ForceQuitBrowser(browserName string, retry int) error

ForceQuitBrowser attempts to close a browser by killing its process.

Parameters:

  • browserName: The name of the browser process to be killed.
  • retry: The number of times to retry killing the process if unsuccessful.

Returns:

  • An error if the process couldn't be killed after all retry attempts, nil otherwise.

This function repeatedly tries to kill the specified browser process using the KillProcess function. If unsuccessful, it will retry up to the specified number of times, with a random sleep interval between attempts (using RandSleepNap). If all attempts fail, it returns the last error encountered.

func IntAorB added in v0.1.3 

func IntAorB(a, b int) int

func IsJSON 

func IsJSON(str string) bool

IsJSON checks if a string is valid JSON.

Parameters:

  • str: The input string to be checked.

Returns:

  • A boolean indicating whether the input string is valid JSON.

This function attempts to unmarshal the input string into a json.RawMessage. If the unmarshal operation succeeds (returns nil error), the function returns true, indicating that the input is valid JSON. Otherwise, it returns false.

func IsZeroVal 

func IsZeroVal(x interface{}) bool

IsZeroVal checks if a value of any type is its zero value.

Parameters:

  • x: The interface{} to be checked.

Returns:

  • A boolean indicating whether x is its zero value.

This function checks if x is nil or if it's equal to its type's zero value. It uses reflect.DeepEqual to compare x with the zero value of its type. The function works for all types, including primitives, structs, and pointers.

func KillProcess added in v0.1.4 

func KillProcess(name string) error

KillProcess terminates a process with the given name.

Parameters:

  • name: The name of the process to be terminated.

Returns:

  • An error if the process list couldn't be retrieved or if the process couldn't be killed. Returns nil if the process was successfully terminated or if no matching process was found.

This function iterates through all running processes, compares their names with the provided name, and attempts to kill the first matching process. If multiple processes have the same name, only the first encountered will be terminated. The function stops searching after killing one process.

func MustStrToFloat 

func MustStrToFloat(raw string, keptChars string) float64

MustStrToFloat converts a string to a float64, keeping only specified characters. It panics if the conversion fails.

Parameters:

  • raw: The input string to be converted.
  • keptChars: A string containing characters to be kept along with digits.

Returns:

  • A float64 representation of the filtered string.

This function first calls StrToNumChars to filter the input string, then uses cast.ToFloat64 to convert the result to a float64. If the conversion fails, it will panic.

func MustStringify 

func MustStringify(data interface{}) string

MustStringify converts an interface{} to its JSON string representation. It's similar to Stringify but panics if an error occurs during the conversion.

Parameters:

  • data: The interface{} to be converted to a JSON string.

Returns:

  • A string containing the JSON representation of the input data.

This function calls Stringify internally and panics if an error is returned. It should be used only when you're certain that the conversion will succeed, or if you want to halt execution on failure.

func NameFromURL added in v0.1.4 

func NameFromURL(uri string) string

NameFromURL extracts a filename-safe string from a URL, excluding the domain.

Parameters:

  • uri: The input URL string to be processed.

Returns:

  • A string derived from the URL path, suitable for use as a filename.

This function attempts to parse the input URL and extract a meaningful name from its path. It removes the scheme and domain, trims leading and trailing slashes, and handles empty paths. If the resulting name is empty, it returns "homepage". The final result is passed through the Filenamify function to ensure it's safe for use as a filename.

Example: 

https://en.wikipedia.org/wiki/Main_Page ==> wiki_Main_Page

If URL parsing fails, the function returns the original input string.

func NewBrowser 

func NewBrowser(opts ...BrowserOptionFunc) (*launcher.Launcher, *rod.Browser)

func NewChromeExtension 

func NewChromeExtension(line, savePath string) (string, error)

NewChromeExtension will create two files from line, and save to savePath

  • background.js
  • manifest.json

line format is: "host:port:username:password:<OTHER>"

func NewLauncher 

func NewLauncher(opts ...BrowserOptionFunc) *launcher.Launcher

func NewStringSlice 

func NewStringSlice(raw string, fixStep int, randomStep ...bool) []string

NewStringSlice splits a raw string into a slice of strings.

Parameters:

  • raw: The input string to be split into a slice.
  • fixStep: The base number of characters for each substring.
  • randomStep: Optional boolean to enable random step mode. If true, the step size will vary randomly.

Returns:

  • A slice of strings, where each string has approximately fixStep characters.

The function splits the input string into substrings. If randomStep is enabled, the actual number of characters in each substring may vary slightly from fixStep. This variation is determined by random chance, potentially adding or subtracting 1 or 2 characters from fixStep.

func NewUserMode 

func NewUserMode(opts ...BrowserOptionFunc) (*launcher.Launcher, *rod.Browser)

func NoEcho added in v0.1.4 

func NoEcho(...any)

NoEcho is a no-op function that accepts any number of arguments and does nothing.

This function can be used as a placeholder or to explicitly ignore values. It's particularly useful in situations where you need to satisfy an interface that requires a function, but you don't actually need to do anything with the arguments.

Parameters:

  • ...any: Variadic parameter that can accept any type and any number of arguments.

Returns:

  • Nothing.

func NormalizeSliceIndex added in v0.1.4 

func NormalizeSliceIndex(length, index int) int

NormalizeSliceIndex adjusts an index to be within the bounds of a slice or array.

Parameters:

  • length: The length of the slice or array.
  • index: The input index to be normalized.

Returns:

  • An integer representing the normalized index.

If length is 0, it returns -1. For negative indices, it wraps around from the end (e.g., -1 becomes length-1). If the index is out of bounds, it normalizes to the valid range [0, length-1].

func ParseCURLCookies added in v0.1.4 

func ParseCURLCookies(raw string) ([]*proto.NetworkCookieParam, error)

ParseCURLCookies converts a cURL-formatted cookie string into NetworkCookieParam objects.

It extracts cookies from the "Cookie" header in the cURL string, parses each cookie, and creates NetworkCookieParam objects with appropriate attributes based on the URL.

Parameters:

  • raw: string - The raw cURL-formatted cookie string.

Returns:

  • []*proto.NetworkCookieParam: Slice of parsed cookie objects.
  • error: Error if parsing fails or no cookies are found.

Usage: 

cookies, err := ParseCURLCookies(curlString)

func QuitIfY 

func QuitIfY()

QuitIfY prompts the user to quit or continue the program. It displays an interactive confirmation dialog with the current date, asking if the user wants to quit (Y) or continue (N). If the user chooses to quit, the program exits immediately. The default option is set to continue (N).

func QuitOnTimeout 

func QuitOnTimeout(args ...int)

QuitOnTimeout waits for a specified number of seconds before quitting the program.

  • If no argument is provided, it uses a default timeout of 3 seconds.
  • If the timeout is 0, it returns immediately without any action.
  • If the timeout is negative, it prompts the user to quit or continue.
  • Otherwise, it displays a spinner for the specified duration before quitting.

Parameters:

  • args: Variadic parameter for timeout in seconds. If not provided, default is used.

func RandFloatX1k 

func RandFloatX1k(min, max float64) int

RandFloatX1k returns a random integer value between (min * 1000) and (max * 1000).

Parameters:

  • min: The minimum value of the range (in seconds).
  • max: The maximum value of the range (in seconds).

Returns:

  • An integer representing milliseconds, randomly chosen between min*1000 and max*1000.

If min equals max, it returns min * 1000 as an integer. The function uses math.Round to ensure accurate conversion from seconds to milliseconds.

func RandSleep 

func RandSleep(min, max float64, msg ...string) int

RandSleep sleeps for a random duration between min and max seconds. It returns the actual sleep duration in milliseconds.

func RandSleepNap added in v0.1.3 

func RandSleepNap() (sleepMills int)

RandSleepNap sleeps for a random duration between 1 and 2 seconds. It returns the actual sleep duration in milliseconds.

func RandSleepPT1Ms added in v0.1.3 

func RandSleepPT1Ms() (sleepMills int)

RandSleepPT1Ms sleeps for a random duration between 0.01 and 0.1 milliseconds. It returns the actual sleep duration in milliseconds.

func RetryIn3 added in v0.1.4 

func RetryIn3(retryableFunc retry.RetryableFunc) error

RetryIn3 attempts to execute a retryable function up to 3 times with a 1-second delay between attempts.

Parameters:

  • retryableFunc: A function of type retry.RetryableFunc to be executed with retry logic.

Returns:

  • An error if all retry attempts fail, or nil if the function succeeds within 3 attempts.

This function uses the retry.Do method with the following options:

  • LastErrorOnly(true): Only the last error is returned if all attempts fail.
  • Attempts(3): The function will be attempted a maximum of 3 times.
  • Delay(time.Second*1): There's a 1-second delay between each attempt.

func SleepN added in v0.1.4 

func SleepN(n float64) int

SleepN sleeps for a random duration between n and n*1.1 seconds. It returns the actual sleep duration in milliseconds.

func SleepPT100Ms added in v0.1.4 

func SleepPT100Ms()

SleepPT100Ms sleeps 0.1~0.2s

func SleepPT250Ms added in v0.1.4 

func SleepPT250Ms()

SleepPT250Ms sleeps 0.25~0.3s

func SleepPT500Ms added in v0.1.4 

func SleepPT500Ms()

SleepPT500Ms sleeps 0.5~0.6s

func SleepWithSpin 

func SleepWithSpin(timeInSeconds int, args ...string)

SleepWithSpin displays a spinner for a specified number of seconds with an optional message. It shows a visual indicator of waiting and then exits after the specified duration.

Parameters:

  • timeInSeconds: The duration to display the spinner, in seconds.
  • args: Optional. A custom message to display. If not provided, a default message is used.

func StrAorB 

func StrAorB(a, b string) string

StrAorB returns the first non-empty string between two input strings.

Parameters:

  • a: The first string to check.
  • b: The second string to check.

Returns:

  • If 'a' is not empty, it returns 'a'.
  • If 'a' is empty, it returns 'b', regardless of whether 'b' is empty or not.

This function is useful for providing a fallback value when the primary string might be empty.

func StrToNumChars 

func StrToNumChars(raw string, keptChars string) string

StrToNumChars extracts numeric characters and specified kept characters from a string.

Parameters:

  • raw: The input string to be processed.
  • keptChars: A string containing additional characters to be kept along with digits.

Returns:

  • A string containing only digits and the specified kept characters.

This function creates a regular expression pattern to match digits and the specified kept characters, finds all matching substrings in the input, and joins them together.

func Stringify 

func Stringify(data interface{}) (string, error)

Stringify converts an interface{} to its JSON string representation.

Parameters:

  • data: The interface{} to be converted to a JSON string.

Returns:

  • A string containing the JSON representation of the input data.
  • An error if the conversion to JSON fails.

This function uses json.Marshal to convert the input data to a JSON byte slice, then converts the byte slice to a string. If the marshaling process fails, it returns an empty string and the error from json.Marshal.

func TermScanf added in v0.1.4 

func TermScanf(msg ...string) string

TermScanf prompts the user for text input and returns the input as a string. It uses pterm's DefaultInteractiveTextInput to display the prompt and capture user input.

Parameters:

  • msg: Optional. A custom message for the input prompt. If not provided, a default prompt is used.

Returns:

  • string: The user's input as a string.

func TermScanfInt added in v0.1.4 

func TermScanfInt(msg ...string) int

TermScanfInt prompts the user for numeric input and returns the input as an integer. It uses pterm's DefaultInteractiveTextInput to display the prompt and capture user input, then converts the input to an integer using cast.ToInt.

Parameters:

  • msg: Optional. A custom message for the input prompt. If not provided, a default prompt is used.

Returns:

  • int: The user's input converted to an integer.

Types

type Bot 

type Bot struct {
	// UniqueID is the identifier of a bot.
	UniqueID string
	// contains filtered or unexported fields
}

func NewBot 

func NewBot(options ...BotOption) *Bot

func NewBotDefault 

func NewBotDefault(options ...BotOption) *Bot

NewBotDefault creates a bot with a default launcher/browser. The launcher/browser passed in will be ignored.

func NewBotForDebug 

func NewBotForDebug(options ...BotOption) *Bot

NewBotForDebug creates a bot which launches browser with option: `--show-paint-rects`,

refer: https://peter.sh/experiments/chromium-command-line-switches/#show-paint-rects

func NewBotHeadless added in v0.1.4 

func NewBotHeadless(options ...BotOption) *Bot

NewBotHeadless sets headless with NewBotDefault

func NewBotUserMode 

func NewBotUserMode(options ...BotOption) *Bot

NewBotUserMode creates a bot that connects to the system Chrome browser.r

func NewBotWithOptionsOnly 

func NewBotWithOptionsOnly(options ...BotOption) *Bot

NewBotWithOptionsOnly creates a new Bot instance with options only, without creating a page.

func (*Bot) AcceptCookies added in v0.1.4 

func (b *Bot) AcceptCookies(cookies ...string) error

AcceptCookies attempts to accept cookies by clicking elements matching the provided selectors.

Parameters:

  • cookies: Variadic string parameter of CSS selectors for cookie acceptance elements.

Returns:

  • error: nil if at least one selector was clicked and page loaded successfully, otherwise an error.

The function tries all selectors even if some fail, logging warnings for individual failures. It returns an error if no selectors were clicked or if the final page load fails.

func (*Bot) ActivateLastOpenedPage added in v0.1.4 

func (b *Bot) ActivateLastOpenedPage(pagesBefore rod.Pages, retry int) error

func (*Bot) ActivatePage added in v0.1.4 

func (b *Bot) ActivatePage(page *rod.Page) error

ActivatePage activates a page instead of current.

func (*Bot) ActivatePageByURLRegex 

func (b *Bot) ActivatePageByURLRegex(jsRegex string, retry int) error

ActivatePageByURLRegex

usage: 

if v := regexStr; v != "" {
	return c.activatePageByURLRegex(v)
}

func (*Bot) AllAttrs 

func (b *Bot) AllAttrs(selector string, opts ...ElemOptionFunc) ([]string, error)

func (*Bot) AllElementsAttrMap 

func (b *Bot) AllElementsAttrMap(elems []*rod.Element, opts ...ElemOptionFunc) []map[string]string

func (*Bot) AnyElem 

func (b *Bot) AnyElem(selectors []string, opts ...ElemOptionFunc) (string, error)

func (*Bot) AnyElemWithTimeout 

func (b *Bot) AnyElemWithTimeout(selectors []string, opts ...ElemOptionFunc) (string, error)

func (*Bot) BlockInCleanUp added in v0.1.3 

func (b *Bot) BlockInCleanUp()

BlockInCleanUp performs cleanup and blocks execution.

func (*Bot) Browser 

func (b *Bot) Browser() *rod.Browser

func (*Bot) Cleanup 

func (b *Bot) Cleanup()

Cleanup closes the opened page and quits the browser in non-userMode. In userMode, by default it will skip cleanup.

func (*Bot) ClearStorageCookies added in v0.1.4 

func (b *Bot) ClearStorageCookies()

ClearStorageCookies calls StorageClearCookies, clears all history cookies.

func (*Bot) Click 

func (b *Bot) Click(selector string, opts ...ElemOptionFunc) error

Click finds an element using the given selector and attempts to click it.

This function performs the following steps:

  1. Locates an element matching the provided CSS selector.
  2. If found, attempts to click the element using the Bot's ClickElem method.

The function uses the Bot's configured timeout settings and other properties for element selection and interaction behavior. It supports customization through optional ElemOptionFunc arguments, which can modify both the element selection process and the click behavior.

If no element matches the selector, the function returns an ErrCannotFindSelector error. If an element is found but cannot be clicked (e.g., not interactable or covered by another element), the error from the ClickElem method is returned.

Usage: 

err := bot.Click("#submit-button")
if err != nil {
    // Handle error (element not found, not clickable, etc.)
}

This method is useful for simulating user interactions in web automation tasks, particularly when dealing with dynamic content or complex DOM structures. It encapsulates the logic of finding and interacting with elements, providing a simple interface for common click operations.

func (*Bot) ClickElem 

func (b *Bot) ClickElem(elem *rod.Element, opts ...ElemOptionFunc) error

ClickElem attempts to click the given element. It performs the following steps:

  1. Optionally highlights the element.
  2. Ensures the element is interactable (scrolls into view if necessary).
  3. Attempts to click the element using the left mouse button.
  4. If the click fails due to a timeout or invisibility, it falls back to clicking via JavaScript.

Parameters:

  • elem: The rod.Element to be clicked.
  • opts: Optional ElemOptionFunc to customize the click behavior.

Options:

  • handleCoverByEsc: If true, attempts to handle covered elements by pressing Escape.
  • highlight: If true, highlights the element before clicking.
  • clickByScript: If true, uses JavaScript to perform the click instead of simulating a mouse click.

Returns:

  • An error if the click operation fails, nil otherwise.

func (*Bot) ClickElemWithScript 

func (b *Bot) ClickElemWithScript(elem *rod.Element, opts ...ElemOptionFunc) error

ClickElemWithScript attempts to click the given element using JavaScript. This method is particularly useful when standard clicking methods fail due to element positioning, overlays, or other DOM-related issues.

The function performs the following steps:

  1. Optionally highlights the element for visual feedback.
  2. Executes a JavaScript click event on the element.
  3. Checks if the element is still interactable after the click.

Options:

  • timeout: Duration to wait for the click operation to complete (default: MediumToSec).
  • highlight: If true, highlights the element before clicking (default: true).

Note:

  • This method cancels the timeout after the initial JavaScript execution to allow for potential page reloads or redirects triggered by the click.
  • If the element becomes non-interactable (e.g., removed from DOM) after the click, it's considered a successful operation, assuming the click caused a page change.

func (*Bot) ClickWithScript 

func (b *Bot) ClickWithScript(selector string, opts ...ElemOptionFunc) error

func (*Bot) ClosePopover 

func (b *Bot) ClosePopover(sel string) (int, error)

ClosePopover attempts to close popovers matching the given selector. It finds all elements matching the selector, checks if they're interactable, highlights them, and tries to click each one.

Parameters:

  • sel: CSS selector string for the popover element(s).

Returns:

  • int: Number of popovers successfully closed.
  • error: nil if successful, otherwise an error describing the failure.

The function stops and returns an error if it encounters any issues during the process. If some popovers are closed before an error occurs, it returns the partial count along with the error.

Usage example: 

closed, err := bot.ClosePopover(".modal-close-button")
if err != nil {
    log.Printf("Error closing popovers: %v", err)
} else {
    log.Printf("Successfully closed %d popovers", closed)
}

func (*Bot) ClosePopovers added in v0.1.4 

func (b *Bot) ClosePopovers(popovers ...string) int

func (*Bot) CookieFile added in v0.1.2 

func (b *Bot) CookieFile() string

func (*Bot) CurrentURL added in v0.1.4 

func (b *Bot) CurrentURL() string

func (*Bot) CustomizePage 

func (b *Bot) CustomizePage()

func (*Bot) DOMStable added in v0.1.4 

func (b *Bot) DOMStable(d time.Duration, diff float64) error

func (*Bot) DisableImages 

func (b *Bot) DisableImages()

func (*Bot) DumpCookies 

func (b *Bot) DumpCookies() (string, error)

DumpCookies saves the current cookies of the bot's page to a file on disk.

This function performs the following operations:

1. Ensures the cookie file exists:

  • If b.cookieFile is empty, it generates a filename based on the current URL.
  • Creates any necessary parent directories for the cookie file.

2. Retrieves and marshals cookies:

  • Calls b.page.MustCookies() to get all cookies from the current page.
  • Marshals these cookies into JSON format.

3. Writes cookies to file:

  • Opens or creates the cookie file with 0644 permissions (rw-r--r--).
  • Writes the JSON-encoded cookie data to the file.

Returns:

  • string: The full path to the cookie file where the cookies were saved.
  • error: An error if any of the following occur:
  • Failure to ensure the cookie file (e.g., permission issues)
  • Failure to marshal cookies to JSON
  • Failure to write to the cookie file

Usage: 

filepath, err := bot.DumpCookies()
if err != nil {
    log.Fatalf("Failed to dump cookies: %v", err)
}
fmt.Printf("Cookies saved to: %s\n", filepath)

Notes:

  • This function is useful for persisting session data between bot runs.
  • The saved cookies can be later loaded using the LoadCookies method.
  • If b.cookieFile is not set, the function will automatically generate a filename based on the current URL, stored in the default or specified cookie folder.
  • Ensure the bot has necessary permissions to write to the cookie directory.

func (*Bot) DumpXHR 

func (b *Bot) DumpXHR(ptn []string, handler func(h *rod.Hijack))

func (*Bot) Elem 

func (b *Bot) Elem(selector string, opts ...ElemOptionFunc) (*rod.Element, error)

Elem get element by selector.

func (*Bot) ElemAttr 

func (b *Bot) ElemAttr(selector string, opts ...ElemOptionFunc) (string, error)

func (*Bot) ElemByIndex 

func (b *Bot) ElemByIndex(selector string, index int) (*rod.Element, error)

ElemByIndex get elem without wait.

func (*Bot) ElemByText 

func (b *Bot) ElemByText(selector string, opts ...ElemOptionFunc) (*rod.Element, error)

ElemByText by text content.

Available layouts:

  • when selector is like `div.abc@@@txt`, will use contains
  • when selector is like `div.abc@@@---@@@txt`, will use exact match

func (*Bot) ElementAttr 

func (b *Bot) ElementAttr(elem *rod.Element, opts ...ElemOptionFunc) (string, error)

func (*Bot) ElementAttrMap 

func (b *Bot) ElementAttrMap(elem *rod.Element, opts ...ElemOptionFunc) map[string]string

func (*Bot) Elems 

func (b *Bot) Elems(selector string, opts ...ElemOptionFunc) ([]*rod.Element, error)

Elems get all elements immediately.

WARN: when use Elems(MustElems), please ensure following conditions

  • !!! selector cannot be empty
  • ??? selector cannot contain SEP: `@@@`, which means shouldn't by Text

func (*Bot) ElemsByText 

func (b *Bot) ElemsByText(selector string, opts ...ElemOptionFunc) ([]*rod.Element, error)

func (*Bot) Eval added in v0.1.3 

func (b *Bot) Eval(script string) (*proto.RuntimeRemoteObject, error)

func (*Bot) FocusAndHighlight 

func (b *Bot) FocusAndHighlight(elem *rod.Element)

func (*Bot) GetElemBox 

func (b *Bot) GetElemBox(elem *rod.Element) (*proto.DOMRect, error)

func (*Bot) GetScrollHeight 

func (b *Bot) GetScrollHeight() (float64, error)

func (*Bot) GetWindowInnerHeight 

func (b *Bot) GetWindowInnerHeight() float64

func (*Bot) Highlight 

func (b *Bot) Highlight(elem *rod.Element, show, hide float64, style string, count int) float64

func (*Bot) HighlightElem 

func (b *Bot) HighlightElem(elem *rod.Element)

func (*Bot) Hijack 

func (b *Bot) Hijack(patterns []string, networkResourceType proto.NetworkResourceType, handler HijackHandler, continueRequest bool)

func (*Bot) HijackAny 

func (b *Bot) HijackAny(resources []string, handler HijackHandler, continueRequest bool)

HijackAny

func (*Bot) IframeElem 

func (b *Bot) IframeElem(iframe, selector string, opts ...ElemOptionFunc) (*rod.Element, error)

func (*Bot) Input 

func (b *Bot) Input(sel, text string, opts ...ElemOptionFunc) (string, error)

Input first clear all content, and then input text content.

func (*Bot) InputSelect 

func (b *Bot) InputSelect(sel, text string, opts ...ElemOptionFunc) error

func (*Bot) Interactable 

func (b *Bot) Interactable(elem *rod.Element) error

func (*Bot) LoadCookies 

func (b *Bot) LoadCookies(filepath string) ([]*proto.NetworkCookieParam, error)

LoadCookies loads cookies from a file or from previously stored cURL data and converts them into a format usable by the bot.

Parameters:

  • filepath: string - The path to the cookie file. If empty, the function will attempt to generate a filename based on the bot's current URL.

Returns:

  • []*proto.NetworkCookieParam: A slice of NetworkCookieParam objects representing the loaded cookies.
  • error: An error if the loading process fails at any step.

Function behavior: 1. Retrieves raw cookie data:

  • If copyAsCURLCookies is not empty, it uses this data.
  • Otherwise, it reads from the specified file or an auto-generated file based on the current URL.

2. Determines the format of the cookie data:

  • If the data is in JSON format, it parses it as proto.NetworkCookie objects.
  • If not JSON, it assumes cURL format and parses accordingly.

3. For JSON format:

  • Unmarshals the data into proto.NetworkCookie objects.
  • Converts these into proto.NetworkCookieParam objects, filling in additional fields.

4. For cURL format:

  • Calls ParseCURLCookies to handle the conversion.

Usage: 

cookies, err := bot.LoadCookies("path/to/cookies.json")
if err != nil {
    log.Fatalf("Failed to load cookies: %v", err)
}
// Use the cookies with the bot...

Notes:

  • This function is flexible and can handle both JSON-formatted cookie files (typically saved by DumpCookies) and cURL-formatted cookie strings.
  • When loading from a file, ensure the bot has read permissions for the cookie file.
  • If filepath is empty and the bot can't determine the current URL, this will result in an error.
  • The returned cookies are in a format ready to be used with the bot's page or browser instance.
  • Any parsing or file reading errors will be returned, allowing the caller to handle them appropriately.

See also: - DumpCookies: For saving cookies to a file. - ParseCURLCookies: For the specific handling of cURL-formatted cookie strings.

func (*Bot) LogTimeSpent added in v0.1.4 

func (b *Bot) LogTimeSpent(start time.Time, skips ...int)

func (*Bot) MakeElemInteractable added in v0.1.4 

func (b *Bot) MakeElemInteractable(elem *rod.Element, byEsc bool) error

func (*Bot) MarkElem 

func (b *Bot) MarkElem(timeout time.Duration, elem *rod.Element, style string)

func (*Bot) MarkElems 

func (b *Bot) MarkElems(timeout time.Duration, elems ...*rod.Element)

func (*Bot) MustAcceptCookies 

func (b *Bot) MustAcceptCookies(cookies ...string)

func (*Bot) MustActivatePage added in v0.1.4 

func (b *Bot) MustActivatePage(page *rod.Page)

func (*Bot) MustAllElemsAttrs 

func (b *Bot) MustAllElemsAttrs(selector string, opts ...ElemOptionFunc) []string

func (*Bot) MustAnyElem 

func (b *Bot) MustAnyElem(selectors []string, opts ...ElemOptionFunc) string

func (*Bot) MustClick 

func (b *Bot) MustClick(selector string, opts ...ElemOptionFunc)

func (*Bot) MustClickAll 

func (b *Bot) MustClickAll(selectors []string, opts ...ElemOptionFunc)

func (*Bot) MustClickAndWait 

func (b *Bot) MustClickAndWait(selector string, opts ...ElemOptionFunc)

func (*Bot) MustClickElem 

func (b *Bot) MustClickElem(elem *rod.Element, opts ...ElemOptionFunc)

func (*Bot) MustClickElemAndWait 

func (b *Bot) MustClickElemAndWait(elem *rod.Element, opts ...ElemOptionFunc)

func (*Bot) MustClickSequentially added in v0.1.4 

func (b *Bot) MustClickSequentially(selectors ...string)

MustClickSequentially clicks on a series of elements specified by selectors in order. It waits 500ms between clicks and ensures DOM stability after all clicks.

Parameters:

  • selectors: Variadic string parameter of CSS selectors to be clicked in sequence.

This function will panic if any click operation fails.

func (*Bot) MustDOMStable added in v0.1.4 

func (b *Bot) MustDOMStable()

func (*Bot) MustElem 

func (b *Bot) MustElem(selector string, opts ...ElemOptionFunc) *rod.Element

func (*Bot) MustElemAttr 

func (b *Bot) MustElemAttr(selector string, opts ...ElemOptionFunc) string

func (*Bot) MustElems 

func (b *Bot) MustElems(selector string, opts ...ElemOptionFunc) []*rod.Element

func (*Bot) MustElemsForAllSelectors 

func (b *Bot) MustElemsForAllSelectors(selectors []string, opts ...ElemOptionFunc) []*rod.Element

func (*Bot) MustEval 

func (b *Bot) MustEval(script string) string

MustEval a wrapper with MediumTo to rod.Page.MustEval

refer: https://github.com/go-rod/rod/blob/main/examples_test.go#L53 

@param script `() => console.log("hello world")` or "`(a, b) => a + b`, 1, 2"
@return string

func (*Bot) MustIframeElem 

func (b *Bot) MustIframeElem(iframe, selector string, opts ...ElemOptionFunc) *rod.Element

func (*Bot) MustInput 

func (b *Bot) MustInput(sel, text string, opts ...ElemOptionFunc) string

func (*Bot) MustOpen 

func (b *Bot) MustOpen(uri string)

func (*Bot) MustPressEscape 

func (b *Bot) MustPressEscape(elem *rod.Element)

func (*Bot) MustScrollToBottom 

func (b *Bot) MustScrollToBottom(opts ...ElemOptionFunc)

func (*Bot) MustScrollToTop 

func (b *Bot) MustScrollToTop()

func (*Bot) MustScrollToXY 

func (b *Bot) MustScrollToXY(x, y float64)

func (*Bot) MustWaitLoad 

func (b *Bot) MustWaitLoad()

func (*Bot) NotNilElem 

func (b *Bot) NotNilElem(sel string, opts []ElemOptionFunc) (*rod.Element, error)

func (*Bot) Open 

func (b *Bot) Open(url string, timeouts ...time.Duration) error

func (*Bot) OpenAndSetCookies added in v0.1.4 

func (b *Bot) OpenAndSetCookies(uri string, timeouts ...time.Duration) error

OpenAndSetCookies open uri with cookies.

typically with following steps:

func (*Bot) OpenElem added in v0.1.4 

func (b *Bot) OpenElem(elem *rod.Element, opts ...ElemOptionFunc) error

func (*Bot) OpenInNewTab added in v0.1.4 

func (b *Bot) OpenInNewTab(elem *rod.Element) error

OpenInNewTab opens an element (usually a link/button) in a new tab by Ctrl+Click.

func (*Bot) OpenURLInNewTab added in v0.1.4 

func (b *Bot) OpenURLInNewTab(uri string) error

OpenURLInNewTab opens url in a new tab and activates it

func (*Bot) Page 

func (b *Bot) Page() *rod.Page

func (*Bot) ParseCURLCookiesFromFile added in v0.1.4 

func (b *Bot) ParseCURLCookiesFromFile(filePath string) ([]*proto.NetworkCookieParam, error)

func (*Bot) Press 

func (b *Bot) Press(elem *rod.Element, keys ...input.Key) error

func (*Bot) PressEscape 

func (b *Bot) PressEscape(elem *rod.Element) error

func (*Bot) PressPageDown added in v0.1.4 

func (b *Bot) PressPageDown(times int) error

func (*Bot) ResetToOriginalPage 

func (b *Bot) ResetToOriginalPage() error

func (*Bot) Scroll 

func (b *Bot) Scroll(x, y float64, step int) error

Scroll Scroll with mouse.

func (*Bot) ScrollLikeHuman 

func (b *Bot) ScrollLikeHuman(offsetX, offsetY float64, opts ...ElemOptionFunc) error

func (*Bot) ScrollToBottom added in v0.1.4 

func (b *Bot) ScrollToBottom(opts ...ElemOptionFunc) error

func (*Bot) ScrollToElemDirectly 

func (b *Bot) ScrollToElemDirectly(elem *rod.Element) error

func (*Bot) ScrollToElement 

func (b *Bot) ScrollToElement(elem *rod.Element, opts ...ElemOptionFunc)

ScrollToElement just a do the best operation

func (*Bot) ScrollToElementE added in v0.1.4 

func (b *Bot) ScrollToElementE(elem *rod.Element, opts ...ElemOptionFunc) error

func (*Bot) SetPanicWith 

func (b *Bot) SetPanicWith(panicWith PanicByType)

func (*Bot) SetTimeout 

func (b *Bot) SetTimeout()

func (*Bot) TimeTrack added in v0.1.4 

func (b *Bot) TimeTrack(start time.Time, skip int)

func (*Bot) TryFocusElem added in v0.1.4 

func (b *Bot) TryFocusElem(elem *rod.Element, scrollToRight bool) error

func (*Bot) TryScrollToBottom added in v0.1.4 

func (b *Bot) TryScrollToBottom(opts ...ElemOptionFunc)

TryScrollToBottom just try scroll to bottom, error will be ignored.

func (*Bot) TypeCharsOneByOne 

func (b *Bot) TypeCharsOneByOne(elem *rod.Element, value string)

func (*Bot) WaitURLContains added in v0.1.4 

func (b *Bot) WaitURLContains(str string, timeouts ...float64) error

WaitURLContains uses `decodeURIComponent(window.location.href).includes` to check if url has str or not, Scenario:

  • if a page's loading status can be determined by url

type BotOption 

type BotOption func(*Bot)

func AcceptLanguage added in v0.1.4 

func AcceptLanguage(s string) BotOption

func Browser added in v0.1.4 

func Browser(brw *rod.Browser) BotOption

func ClearCookies added in v0.1.4 

func ClearCookies(b bool) BotOption

func CopyAsCURLCookies added in v0.1.4 

func CopyAsCURLCookies(b []byte) BotOption

CopyAsCURLCookies reads the raw content `Copy as cURL` from clipboard

func ForceCleanup added in v0.1.4 

func ForceCleanup(b bool) BotOption

func Headless 

func Headless(b bool) BotOption

func Humanized added in v0.1.4 

func Humanized(b bool) BotOption

func Launcher added in v0.1.4 

func Launcher(l *launcher.Launcher) BotOption

func Logger added in v0.1.4 

func Logger(l *zap.Logger) BotOption

func StealthMode added in v0.1.4 

func StealthMode(b bool) BotOption

func TrackTime added in v0.1.4 

func TrackTime(b bool) BotOption

TrackTime tracktime shows logs with level `debug`.

func UserAgent added in v0.1.4 

func UserAgent(s string) BotOption

func UserDataDir added in v0.1.4 

func UserDataDir(s string) BotOption

func UserMode added in v0.1.4 

func UserMode(b bool) BotOption

func WithCookieFile 

func WithCookieFile(s string) BotOption

WithCookieFile uses the cookie file specified, cookie file should be json array.

func WithCookieFolder added in v0.1.2 

func WithCookieFolder(s string) BotOption

WithCookieFolder will load cookies from folder passed in.

func WithCookies 

func WithCookies(b bool) BotOption

WithCookies will automatically load cookies from current dir for ".cookies/xxx".

func WithHighlightTimes 

func WithHighlightTimes(i int) BotOption

func WithLeftPosition added in v0.1.4 

func WithLeftPosition(i int) BotOption

WithLeftPosition sets the left position of the browser window.

func WithPage 

func WithPage(b bool) BotOption

func WithPanicBy 

func WithPanicBy(i PanicByType) BotOption

func WithPopovers 

func WithPopovers(popovers ...string) BotOption

WithPopovers is useful when popovers appear randomly.

type BrowserOptionFunc 

type BrowserOptionFunc func(o *BrowserOptions)

func BrowserExtensions added in v0.1.4 

func BrowserExtensions(extArr ...string) BrowserOptionFunc

BrowserExtensions dirs of extensions, only unpacked extension is supported.

func BrowserFlags added in v0.1.4 

func BrowserFlags(arr ...string) BrowserOptionFunc

BrowserFlags flags those are not pre-defined

func BrowserHeadless added in v0.1.4 

func BrowserHeadless(b bool) BrowserOptionFunc

func BrowserIgnoreCertErrors added in v0.1.4 

func BrowserIgnoreCertErrors(b bool) BrowserOptionFunc

func BrowserIncognito added in v0.1.4 

func BrowserIncognito(b bool) BrowserOptionFunc

func BrowserNoDefaultDevice added in v0.1.4 

func BrowserNoDefaultDevice(b bool) BrowserOptionFunc

func BrowserPaintRects added in v0.1.4 

func BrowserPaintRects(b bool) BrowserOptionFunc

func BrowserProxy added in v0.1.4 

func BrowserProxy(s string) BrowserOptionFunc

func BrowserSlowMotionDelay added in v0.1.4 

func BrowserSlowMotionDelay(i int) BrowserOptionFunc

func BrowserUserDataDir added in v0.1.4 

func BrowserUserDataDir(s string) BrowserOptionFunc

func LaunchLeakless added in v0.1.4 

func LaunchLeakless(b bool) BrowserOptionFunc

type BrowserOptions 

type BrowserOptions struct {
	// contains filtered or unexported fields
}

type ElemOptionFunc 

type ElemOptionFunc func(o *ElemOptions)

func ClearBeforeInput added in v0.1.3 

func ClearBeforeInput(b bool) ElemOptionFunc

func DisableHandleCoverByEsc 

func DisableHandleCoverByEsc() ElemOptionFunc

func EndWithEscape added in v0.1.3 

func EndWithEscape(b bool) ElemOptionFunc

func OpenInTab added in v0.1.4 

func OpenInTab(b bool) ElemOptionFunc

func Trigger added in v0.1.4 

func Trigger(b bool) ElemOptionFunc

func WithAttr 

func WithAttr(s string) ElemOptionFunc

func WithAttrMap 

func WithAttrMap(m map[string]string) ElemOptionFunc

func WithCaseInsensitive 

func WithCaseInsensitive(b bool) ElemOptionFunc

func WithClickByScript 

func WithClickByScript(b bool) ElemOptionFunc

func WithHighlight 

func WithHighlight(b bool) ElemOptionFunc

func WithHumanized 

func WithHumanized(b bool) ElemOptionFunc

func WithIframe 

func WithIframe(page *rod.Page) ElemOptionFunc

func WithIndex 

func WithIndex(i int) ElemOptionFunc

func WithRetries added in v0.1.4 

func WithRetries(i uint) ElemOptionFunc

func WithRoot 

func WithRoot(root *rod.Element) ElemOptionFunc

func WithScrollAsHuman 

func WithScrollAsHuman(ah *ScrollAsHuman) ElemOptionFunc

func WithSelectorType 

func WithSelectorType(t rod.SelectorType) ElemOptionFunc

func WithSteps 

func WithSteps(i int) ElemOptionFunc

func WithSubmit 

func WithSubmit(b bool) ElemOptionFunc

func WithTimeout 

func WithTimeout(t float64) ElemOptionFunc

func WithWaitStable 

func WithWaitStable(b bool) ElemOptionFunc

type ElemOptions 

type ElemOptions struct {
	// contains filtered or unexported fields
}

type HijackHandler 

type HijackHandler func(*rod.Hijack)

HijackHandler hijacks network resources, you can disable hijacked resources by setting the return value to true.

type PanicByType 

type PanicByType int
const (
	PanicByDft PanicByType = iota
	PanicByDump
	PanicByLogError
)

type ScrollAsHuman 

type ScrollAsHuman struct {
	// contains filtered or unexported fields
}

func NewScrollAsHuman added in v0.1.4 

func NewScrollAsHuman(long, short, up float64) *ScrollAsHuman

Source Files

View all Source files

Directories

Path Synopsis
examples
basic/baidu command
basic/google command
basic/sannysoft command
blocket command
books command
express command
hijack command
incolumitas command
jandan command
modal command
nfcloud command
qiyueba command
usermode command
wikipedia command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.