websearch

Overview ¶

Native Go meta-search — the SEARCH half of /v1/websearch, replacing the reverse proxy to the (retired) SearXNG pod.

Rationale (HIP-0106 preserved): web search stays Hanzo-native with NO external search SaaS — never Brave/Serper/Tavily/Jina/Cohere and never a paid search-API key. SearXNG earned its keep as keyless meta-search over public engines; this file does the SAME job in-process in Go, so the stock Python metasearch pod is gone (one fewer non-Go dependency) and search no longer 502s when it is down.

It queries keyless public engines directly over native Go HTTP, parses their HTML with x/net/html, and returns the SearXNG JSON contract the LibreChat searxng client decodes verbatim: {results:[{url,title,content,...}]}. So the chat server needs NO change — searxngInstanceUrl already points at this surface.

Composable by construction: an engine is {name, build(query)→URL, parse(HTML) →results}. metaSearch runs the ENABLED engines concurrently and merges+dedupes by normalized URL. WEBSEARCH_ENGINES selects them (default "bing", verified datacenter-tolerant from the cluster egress); adding one is a registry entry, not new plumbing. Any engine that fails or gets bot-challenged contributes zero and never fails the request — search degrades to fewer results, never to a 5xx.

Package websearch exposes Hanzo-native Web Search + Scrape on the unified cloud-api /v1 plane, so hanzo.chat's web_search agent tool runs entirely on Hanzo infrastructure with NO external SaaS provider, per HIP-0106.

hanzo.chat (LibreChat fork) implements web_search as a fixed 3-stage pipeline whose provider contracts are frozen by the upstream client (@librechat/agents tools/search). The only self-hostable, key-less-to-a-SaaS providers it accepts are:

• search provider "searxng" → GET {searxngInstanceUrl}/search?q=&format=json ← {results:[{url,title,content,img_src?}]}
• scraper provider "firecrawl" → POST {firecrawlApiUrl}/{version}/scrape body {url,formats} ← {success,data:{markdown,metadata}} (reranker is optional; we omit it — provider+scraper is sufficient.)

This subsystem serves BOTH contracts under /v1/websearch, backed by Hanzo's own services — never a third-party search API:

• GET /v1/websearch/search SearXNG-shaped. Served NATIVELY in-process by a keyless Go meta-search (search.go) — no SearXNG pod, no search SaaS.
• POST /v1/websearch/v1/scrape Firecrawl-shaped. Served NATIVELY in-process (also /v1/websearch/scrape) by clients/crawl — fetch, extract, render — returning {success,data:{markdown,metadata}}.

Both halves are now in-process Go, for the same reason and by the same shape: a keyless meta-search here, a fetch-and-extract in clients/crawl. Neither has a pod to be down. Scrape previously dialled a separate crawler at crawl.hanzo.svc that did NOT exist — the name was NXDOMAIN — so this surface answered 200 while every scrape inside it returned success:false. clients/crawl is the same capability with no network hop and no second deployment to keep alive.

The chat server calls these SERVER-SIDE in-cluster, so point searxngInstanceUrl / firecrawlApiUrl at this surface (public api.hanzo.ai/v1 or the internal cloud-api svc DNS — same binary either way).

AUTH: two callers, two ONE-WAY-equivalent gates, never an open proxy —

• SEARCH (/v1/websearch/search) admits EITHER a validated principal (principal.Validated — X-User-Id minted by the identity middleware from a verified JWT: the signed-in console user via the /cloud bearer proxy) OR the shared service key WEBSEARCH_API_KEY as X-API-Key (the hanzo.chat server, which reaches cloud service-to-service with no user principal). A caller with neither is refused.
• SCRAPE (/v1/websearch/*/scrape) requires the shared key as a Bearer (the chat server path only; the console surfaces scrape read-only, does not drive it).

An unset key 503s and any missing/mismatched key 401s on the key path; a request with a validated principal never needs the key. So neither surface is ever an open proxy, and the signed-in console user reaches search without the shared key.