Documentation
¶
Overview ¶
Package fetch provides URL fetching backends for the url tool. Supports DefuddleCLI (local mode) and BrowserGateway (remote mode) with daily caching.
Plane: shared
Index ¶
Constants ¶
const MaxContentChars = 30_000
MaxContentChars is the rune limit applied by TruncateContent.
Variables ¶
This section is empty.
Functions ¶
func BinaryFetchError ¶ added in v1.2.1
BinaryFetchError returns an error telling the agent to use a download tool. Suggests aria2c if available, then wget, then curl.
func IsBinaryBody ¶ added in v1.2.1
IsBinaryBody checks the first 8KB for null bytes.
func IsBinaryContentType ¶ added in v1.2.1
IsBinaryContentType returns true for content types that are never human-readable text.
func TruncateContent ¶
TruncateContent truncates content to MaxContentChars runes.
Types ¶
type Backend ¶
Backend controls how HTML is fetched and converted to markdown.
func NewBrowserGatewayBackend ¶
NewBrowserGatewayBackend creates a backend that fetches via browser-gateway.
func NewCachedFetchBackend ¶
NewCachedFetchBackend creates a CachedFetchBackend wrapping the given fallback. If the cache directory cannot be created, returns the fallback directly (caching disabled).
func NewDefuddleCLIBackend ¶
func NewDefuddleCLIBackend() Backend
NewDefuddleCLIBackend creates a backend that fetches URLs via Go's HTTP client. HTML pages are passed to defuddle for parsing; non-HTML content is returned raw. Requires defuddle to be installed and on PATH.
type CachedFetchBackend ¶
type CachedFetchBackend struct {
// contains filtered or unexported fields
}
CachedFetchBackend wraps a Backend with a file-based daily cache. Cache key: sanitized URL + today's date. Cache dir: ~/.cache/organon/scrapes/.
Source Files