Documentation
¶
Overview ¶
Package crawl is any web page turned into clean markdown a model can read.
Hanzo Crawl: fetch one URL and return its readable content as markdown, in-process, in Go.
It replaces the dial to a separate Crawl4AI deployment. That service was named in config but did not exist — crawl.hanzo.svc.cluster.local was NXDOMAIN — so every scrape returned {success:false, "no such host"} while the surface in front of it answered 200. This package is the same move already made for the SEARCH half in clients/websearch/search.go, which replaced a SearXNG pod with in-process Go: one fewer non-Go dependency, one fewer thing that can be down, and no network hop for work that is a fetch and a parse.
The engine is three orthogonal steps, each in its own file and each testable without the others:
Fetch (this file) URL → HTML bytes, over guarded HTTP extract (extract.go) HTML → the readable subtree + metadata render (markdown.go) node → markdown
Fetching untrusted URLs from inside the cluster is the security boundary ¶
The caller supplies the URL, so this is a server-side request forgery primitive by construction, and it runs INSIDE the cluster with an in-namespace service DNS and a cloud metadata endpoint on 169.254.169.254 that hands out credentials to anyone who asks. A naive fetcher here is a credential exfiltration hole, not a bug. The separate service got network policy for free by being a separate pod; folding it in means this package must carry the guard itself.
So: only http/https, and every address actually dialed must be a public unicast address. The check is in the DIALER, not on the hostname, because checking a hostname up front and then letting the transport resolve it again is a time-of-check/time-of-use gap — DNS rebinding walks straight through it. Dialing is the only moment the real destination is known, and that is where it is enforced. Redirects are followed but re-enter the same dialer, so a public URL that 302s to 169.254.169.254 is refused at the hop that matters.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrBlocked = errors.New("crawl: refused to dial a non-public address")
ErrBlocked is returned when a URL resolves to an address this package refuses to dial. It is deliberately distinguishable: a caller may want to report a blocked fetch differently from a site that was merely down, and a log line that cannot tell them apart makes an attack look like an outage.
Functions ¶
func Bind ¶
Bind installs the process-wide archive. A nil or absent VFS leaves crawling fully functional and unarchived, which is the correct posture for a deployment with no object store configured — fail soft, never fail closed, since nothing about answering a fetch depends on keeping it.
func Mount ¶
Mount registers /v1/crawl.
The gate mirrors /v1/websearch/search exactly, and it is not optional here. This surface fetches a URL the caller chooses, from inside the cluster — an open one is a proxy into the private network, and the address guard in crawl.go is the second line of that defence, not the first. A caller is admitted with EITHER a validated principal (a signed-in user, already authenticated and metered) OR the shared service key. Neither ⇒ refused. An unset key 503s rather than defaulting open, so a misconfigured deploy fails closed and loudly.
Types ¶
type Archive ¶
type Archive struct {
// contains filtered or unexported fields
}
Archive keeps pages under a scope.
func (*Archive) Get ¶
Get returns a previously archived page.
Any failure — missing, unreachable, corrupt — is reported the same way: not found. A caller's only sensible response to each is to fetch, so distinguishing them at this seam would create a decision nobody makes differently.
func (*Archive) Put ¶
Put files a page under the scope, keyed by the url that was REQUESTED.
requested is a separate argument from p.URL on purpose: Fetch returns where the page landed after redirects, and keying by that would file every redirecting page under an address no caller ever asks for. Get would then miss it forever — a cache that silently never hits for exactly the pages that redirect, which is a large share of the real web.
type Page ¶
type Page struct {
// URL is the FINAL url after redirects, not the one requested. Callers cite it,
// so it must be where the content actually came from.
URL string
// Title is the document title, best-effort from <title> or og:title.
Title string
// Markdown is the readable content. Empty is a legitimate result for a page
// that is genuinely all chrome; it is not an error.
Markdown string
// Metadata carries what the document said about itself (description, og:*,
// language, status). Map-shaped because it is passed through to a JSON contract
// that predates this package and accepts arbitrary keys.
Metadata map[string]any
}
Page is one crawled document. It is what every surface in front of this package renders, so it holds the extracted content and the provenance needed to cite it — not the raw HTML, which no caller has wanted and which would keep the whole response alive in memory.
func Fetch ¶
Fetch retrieves one URL and returns its readable content.
It returns an error only when there is nothing to read — a refused address, an unreachable host, a non-2xx status, a body that is not a document. A page that is reachable but yields little content is a Page with short Markdown, because "this page is mostly navigation" is a true answer and a caller that treats it as a failure would retry forever.
func Read ¶
Read is the door: return the archived page if we have one, otherwise fetch it and keep it.
A hit skips the network entirely, which is the point — the same URL is read repeatedly across a research loop, a re-ask, and a re-index, and paying for it once is both faster and politer to the origin.
Source Files