scrape

package
v0.3.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 11, 2026 License: GPL-3.0 Imports: 8 Imported by: 0

Documentation

Overview

Package scrape fetches pages through Ether's host HTTP service (so the manifest network allowlist applies) and pulls text and links out of simple HTML. It rate-limits per host to stay polite. For anything trickier than listing pages, parse the body in plugin code.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Matches 

func Matches(page string, re *regexp.Regexp) [][]string

Matches returns the capturing groups of every regex match — the escape hatch for listing pages with regular shapes.

func Text 

func Text(pageHTML string) string

Text strips markup down to readable text.

Types

type Fetcher 

type Fetcher struct {
	Host  etherplugin.Host
	Delay time.Duration // between requests to one host; default 250ms
	// contains filtered or unexported fields
}

Fetcher wraps a plugin Host with a polite per-host delay.

func (*Fetcher) Get 

func (f *Fetcher) Get(ctx context.Context, rawURL string) (string, int, error)

Get fetches a URL through the host and returns the body. 

type Link struct {
	URL  string
	Text string
}

Link is one anchor: absolute URL plus its visible text. 

func Links(pageHTML, base string) []Link

Links extracts anchors, resolving relative hrefs against base.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.