Documentation
¶
Overview ¶
Package fetch retrieves web pages for the read_url tool. It exposes a single Fetcher interface with backends selected by config: "firecrawl" (hosted scrape API with JS rendering) and "raw" (plain HTTP + text extraction). Failures are classified into ErrorKind so the tool layer can give the model an honest, actionable message instead of a generic error.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Error ¶
type Error struct {
Kind ErrorKind
StatusCode int // HTTP status that triggered the failure (service or target), 0 if none
FinalURL string // best-known final URL, "" if unknown
Msg string
}
Error is a classified fetch failure. FinalURL carries the best-known resolved URL even on failure: a shortlink redirect often resolves before the target page turns out to be bot-protected, and "this link points to X" is still a useful answer.
type ErrorKind ¶
type ErrorKind string
ErrorKind classifies fetch failures. The tool layer maps each kind to a model-facing message that steers away from retry/search spirals.
const ( KindInvalidURL ErrorKind = "invalid_url" // malformed or non-http(s) URL KindRefused ErrorKind = "refused" // service refuses the site by policy (e.g. Reddit) KindBlocked ErrorKind = "blocked" // captcha / anti-bot wall (e.g. marketplaces) KindNotFound ErrorKind = "not_found" // target 404/410 KindTimeout ErrorKind = "timeout" // deadline exceeded KindRateLimited ErrorKind = "rate_limited" // 429 from the service KindQuota ErrorKind = "quota" // fetch service out of credits KindConfig ErrorKind = "config" // bad/missing API key, unwired backend KindUpstream ErrorKind = "upstream" // 5xx, malformed response, network error // KindForbiddenNetwork is the raw backend's SSRF guard: the URL resolves // to a loopback/link-local/private address the network policy forbids. KindForbiddenNetwork ErrorKind = "forbidden_network" )
type Fetcher ¶
Fetcher retrieves a single URL. Implementations classify failures as *Error where possible; any other error means a programmer/transport fault.
type Result ¶
type Result struct {
Content string
Title string // "" when unknown
FinalURL string // after redirects; equals the requested URL if none
StatusCode int // target page HTTP status when known, 0 otherwise
}
Result is a successfully fetched page. Content is the full extracted markdown/plain text — truncation is a presentation concern of the tool layer, not the fetcher.
Source Files