Documentation
¶
Overview ¶
Package crawl is Hanzo Crawl: fetch one URL and return its readable content as markdown, in-process, in Go.
It replaces the dial to a separate Crawl4AI deployment. That service was named in config but did not exist — crawl.hanzo.svc.cluster.local was NXDOMAIN — so every scrape returned {success:false, "no such host"} while the surface in front of it answered 200. This package is the same move already made for the SEARCH half in clients/websearch/search.go, which replaced a SearXNG pod with in-process Go: one fewer non-Go dependency, one fewer thing that can be down, and no network hop for work that is a fetch and a parse.
The engine is three orthogonal steps, each in its own file and each testable without the others:
Fetch (this file) URL → HTML bytes, over guarded HTTP extract (extract.go) HTML → the readable subtree + metadata render (markdown.go) node → markdown
Fetching untrusted URLs from inside the cluster is the security boundary ¶
The caller supplies the URL, so this is a server-side request forgery primitive by construction, and it runs INSIDE the cluster with an in-namespace service DNS and a cloud metadata endpoint on 169.254.169.254 that hands out credentials to anyone who asks. A naive fetcher here is a credential exfiltration hole, not a bug. The separate service got network policy for free by being a separate pod; folding it in means this package must carry the guard itself.
So: only http/https, and every address actually dialed must be a public unicast address. The check is in the DIALER, not on the hostname, because checking a hostname up front and then letting the transport resolve it again is a time-of-check/time-of-use gap — DNS rebinding walks straight through it. Dialing is the only moment the real destination is known, and that is where it is enforced. Redirects are followed but re-enter the same dialer, so a public URL that 302s to 169.254.169.254 is refused at the hop that matters.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrBlocked = errors.New("crawl: refused to dial a non-public address")
ErrBlocked is returned when a URL resolves to an address this package refuses to dial. It is deliberately distinguishable: a caller may want to report a blocked fetch differently from a site that was merely down, and a log line that cannot tell them apart makes an attack look like an outage.
Functions ¶
func Mount ¶
Mount registers /v1/crawl.
The gate mirrors /v1/websearch/search exactly, and it is not optional here. This surface fetches a URL the caller chooses, from inside the cluster — an open one is a proxy into the private network, and the address guard in crawl.go is the second line of that defence, not the first. A caller is admitted with EITHER a validated principal (a signed-in user, already authenticated and metered) OR the shared service key. Neither ⇒ refused. An unset key 503s rather than defaulting open, so a misconfigured deploy fails closed and loudly.
Types ¶
type Document ¶
type Document struct {
URL string `json:"url"`
Title string `json:"title,omitempty"`
Markdown string `json:"markdown"`
Metadata map[string]any `json:"metadata,omitempty"`
}
Document is the crawled page.
type Page ¶
type Page struct {
// URL is the FINAL url after redirects, not the one requested. Callers cite it,
// so it must be where the content actually came from.
URL string
// Title is the document title, best-effort from <title> or og:title.
Title string
// Markdown is the readable content. Empty is a legitimate result for a page
// that is genuinely all chrome; it is not an error.
Markdown string
// Metadata carries what the document said about itself (description, og:*,
// language, status). Map-shaped because it is passed through to a JSON contract
// that predates this package and accepts arbitrary keys.
Metadata map[string]any
}
Page is one crawled document. It is what every surface in front of this package renders, so it holds the extracted content and the provenance needed to cite it — not the raw HTML, which no caller has wanted and which would keep the whole response alive in memory.
func Fetch ¶
Fetch retrieves one URL and returns its readable content.
It returns an error only when there is nothing to read — a refused address, an unreachable host, a non-2xx status, a body that is not a document. A page that is reachable but yields little content is a Page with short Markdown, because "this page is mostly navigation" is a true answer and a caller that treats it as a failure would retry forever.
type Request ¶
type Request struct {
URL string `json:"url"`
}
Request is the /v1/crawl body. One URL per call: batching would make the response a partial-failure envelope that every caller then has to unpack, and no caller has asked for more than one.
type Response ¶
type Response struct {
Success bool `json:"success"`
Data *Document `json:"data,omitempty"`
Error string `json:"error,omitempty"`
}
Response is the /v1/crawl body.
Success is a field rather than an HTTP status because "the page could not be fetched" is a normal outcome of asking about a URL, not a fault of the request: the caller sent a well-formed ask and gets a well-formed answer saying the page was unreachable. Reserving non-2xx for auth and malformed input keeps a caller's error handling honest — a 200 means the surface worked, and Success says what it found.
Source Files