cloudscraper-go

cloudscraper-go

A native Go library and CLI to fetch anti-bot–protected pages — TLS/JA3 fingerprint spoofing via uTLS, no Python, no headless browser for the common cases. A single static binary.

Status: the full roadmap (M1–M5) is implemented and CI-green — a browser TLS fingerprint (verifiable against tls.peet.ws), hot sessions with retries + proxy, a bounded concurrent crawler, a server daemon, and an MCP server so AI agents can call it as a tool. Scope limits are documented, not hidden.

Problem

Cloudflare, Akamai and friends increasingly block by how you connect, not who you say you are. Go's net/http has a distinctive TLS ClientHello (JA3/JA4) that these systems flag instantly — so a plain http.Get gets 403 / Access Denied on sites a real browser loads fine. cloudscraper-go makes the Go client's TLS handshake indistinguishable from a real browser's, which clears the most common "blocked by fingerprint" cases without spinning up Chrome.

How it works

The core is a custom http.RoundTripper that performs the TLS handshake with a browser's ClientHello (via uTLS), then speaks HTTP/2 or HTTP/1.1 over that connection. Cookies and redirects are handled by the standard net/http.Client.

flowchart LR
  App["Your code / CLI"] -->|http.Client| RT["uTLS RoundTripper"]
  RT -->|"browser ClientHello (JA3/JA4)"| TLS["TLS handshake"]
  TLS -->|ALPN| H{h2?}
  H -->|yes| H2["golang.org/x/net/http2"]
  H -->|no| H1["HTTP/1.1"]
  H2 --> Site[(Protected site)]
  H1 --> Site

Quick start

CLI

go install github.com/maarkN/cloudscraper-go/cmd/cloudscraper@latest

# Fetch a page with a Chrome fingerprint (body -> stdout, status/headers -> stderr)
cloudscraper fetch https://example.com --dump-headers

# Prove the fingerprint: what JA3/JA4 does the server actually see?
cloudscraper fingerprint --profile chrome

fingerprint output (real):

profile:       chrome
http_version:  h2
ja3_hash:      96f0044b9649e05e8e079d698cc65a77
ja4:           t13d1516h2_8daaf6152771_d8a2da3f94cd
user_agent:    Mozilla/5.0 (...) Chrome/131.0.0.0 Safari/537.36

Switch --profile firefox and the JA3/JA4 changes to Firefox's — the cipher and extension ordering is genuinely different, not just the User-Agent.

CLI reference

Run cloudscraper <command> --help for the authoritative list. All fetching commands take a browser --profile and can send extra request headers.

cloudscraper fetch <url> — GET one URL. The body goes to stdout; status and headers go to stderr (so you can pipe the body cleanly).

cloudscraper crawl <url>... — fetch many URLs concurrently.

cloudscraper fingerprint — probe tls.peet.ws and print the JA3/JA4 the server sees. Flag: -p, --profile.

cloudscraper mcp — run the MCP server over stdio (see the AI agents section below). Flag: -p, --profile.

Custom headers (Authorization, X-Api-Key, anything)

Pass -H "Name: Value" as many times as you need. It works on fetch and crawl:

# A bearer token, an API key and an Accept header on a single fetch
cloudscraper fetch https://api.example.com/v1/me \
  -H "Referer: https://api.example.com/" \
  -H "Authorization: Bearer eyJhbGciOi..." \
  -H "X-Api-Key: 1234567890" \
  -H "Accept: application/json"

# The same auth header applied to every URL in a concurrent crawl
cloudscraper crawl -c 8 -H "Authorization: Bearer $TOKEN" \
  https://api.example.com/a https://api.example.com/b

From the library it is cloudscraper.WithHeader("Authorization", "Bearer ..."), and you can pass it as many times as you like when constructing the client.

When headers are enough, and when they aren't

The right headers matter as much as the fingerprint. A lot of Cloudflare rules block on request shape, not TLS: a missing Referer (hotlink / anti-scraping rules), a missing API auth header, or a non-browser Accept / Sec-Fetch-*. Those are exactly what -H fixes. A common path from 403 Sorry, you have been blocked to 200 looks like:

cloudscraper fetch "https://api.example.com/v1/search?q=..." --profile firefox \
  -H "Referer: https://api.example.com/" \        # clears a Referer-gated Cloudflare rule
  -H "Authorization: Guest" \                     # some public APIs still want an auth header
  -H "Accept: application/json, text/plain, */*"

It is enough when the block is header- or IP-based: a browser fingerprint plus the right -H headers get you through, and the response is the real content (check the body, not just the status).

It is not enough when the site serves a JavaScript challenge ("Just a moment...") or keys on the HTTP/2 frame fingerprint, which is still Go's (see Scope & limitations below). Pure Go cannot clear those, and the tool says so rather than pretending.

Library

package main

import (
	"context"
	"fmt"

	"github.com/maarkN/cloudscraper-go/pkg/cloudscraper"
)

func main() {
	client, _ := cloudscraper.New(cloudscraper.WithProfile("chrome"))
	resp, err := client.Get(context.Background(), "https://example.com")
	if err != nil {
		panic(err)
	}
	fmt.Println(resp.Proto, resp.StatusCode, len(resp.Body))
}

Options: WithProfile, WithTimeout, WithoutRedirects, WithHeader, WithProxy, WithRetries.

Sessions, retries & proxy (M2)

A Client is a hot session: its cookie jar persists across calls, so a login or challenge cookie set on one request rides along on the next.

client, _ := cloudscraper.New(
	cloudscraper.WithProfile("chrome"),
	cloudscraper.WithRetries(3),                       // network / 429 / 5xx, exp. backoff + jitter
	cloudscraper.WithProxy("http://user:pass@host:8080"), // http CONNECT or socks5://
)
a, _ := client.Get(ctx, "https://site/login")   // sets a cookie
b, _ := client.Get(ctx, "https://site/account") // cookie is reused automatically

Retries honour Retry-After and the request context; they only replay requests whose body can be rewound. From the CLI: --retries and --proxy.

Concurrent crawling (M3)

Fetch many URLs with a bounded worker pool, per-host rate limiting and optional Prometheus metrics — cancellable and with backpressure:

import "github.com/maarkN/cloudscraper-go/internal/crawl"

crawler := crawl.New(fetcher, crawl.Options{
	Concurrency: 8,    // bounded in-flight fetches (semaphore)
	PerHostRPS:  2,    // token-bucket rate limit per host
	Metrics:     crawl.NewMetrics(prometheus.DefaultRegisterer),
})
results, err := crawler.Crawl(ctx, urls) // results keep input order; per-URL errors captured

From the CLI:

cloudscraper crawl -c 8 --rps 2 https://a.example https://b.example https://c.example

A single failing URL never aborts the others; a cancelled context stops the whole crawl. Fetcher is a one-method interface, so the crawler is unit-tested with a fake — no network.

Server mode (M4)

Run it as a daemon that keeps solved sessions hot — each ?session= id maps to its own long-lived, cookie-warm client:

go run ./cmd/server -addr :8080 -idle-ttl 10m

curl "localhost:8080/fetch?url=https://example.com&session=a"   # solves once
curl "localhost:8080/fetch?url=https://example.com/about&session=a" # reuses it

Idle sessions are evicted by a background janitor; SIGINT/SIGTERM triggers a graceful drain. Handlers depend on a small Doer/Factory pair, so they're unit-tested without network.

🤖 Use it from an AI agent (MCP)

cloudscraper mcp runs a Model Context Protocol server over stdio (built on the official modelcontextprotocol/go-sdk), so any MCP client — Claude Desktop, IDEs — can read anti-bot–protected pages as a tool. Point the client at the binary:

{
  "mcpServers": {
    "cloudscraper": { "command": "cloudscraper", "args": ["mcp"] }
  }
}

Tools exposed:

This is the native Go counterpart to the MCP server in cloudscraper.js — same idea, single static binary, no Python.

Scope & limitations (read this)

Being explicit about what pure Go can and can't do is a feature, not a caveat.

What this means today: the JA3/JA4 layer matches a real browser, which clears the common "blocked by IP/TLS fingerprint" cases. Two known gaps are documented rather than hidden:

• HTTP/2 frame fingerprint — cloudscraper fingerprint still reports Go's akamai_fingerprint, because we use the standard x/net/http2. Matching Chrome's h2 SETTINGS/frame order needs a customized HTTP/2 layer (roadmap).
• Header order — net/http reorders headers on the wire, so the byte-exact browser header order isn't reproduced yet.

For heavy JS challenges, the plan is a pluggable fallback (headless via chromedp, or an external solver) — deliberately out of the fast path.

Design decisions & trade-offs

• uTLS over a headless browser. For the fingerprint layer, uTLS is orders of magnitude cheaper than driving Chrome, and ships as one static binary. Headless is reserved as an opt-in fallback for challenges Go can't evaluate.
• Compose http.RoundTripper, don't fork net/http. Cookies, redirects and the Client API come for free; we only replace the transport. The cost is the two gaps above (h2 frames, header order), which a fork would fix — a considered trade-off for M1, revisited later.
• Dial per request (for now). Simple and correct; session reuse / pooling is M2/M3, where it belongs alongside rate limiting and backpressure.
• Fail honestly. https-only today; non-https returns a clear error rather than pretending.

Benchmarks

Measured on loopback (a local TLS server) so the numbers isolate the library's own cost — uTLS handshake + HTTP + gzip inflate — with no network variance, and are reproducible:

go test -run '^$' -bench BenchmarkGet -benchmem ./pkg/cloudscraper

Intel i7-4850HQ (4C/8T), Go 1.26. Concurrency yields ~2.4× throughput. Each request currently opens a fresh TLS connection, so the uTLS handshake dominates the per-request time — connection pooling is the clear next optimization. (Hot sessions already avoid re-solving the anti-bot challenge; pooling would additionally avoid re-handshaking.)

For context, the Python approach this replaces — the original cloudscraper.js — spawned a python process per request (interpreter cold-start + a freshly created scraper), a cost measured in hundreds of milliseconds to seconds. cloudscraper-go is a single static binary with no per-request process spawn.

Roadmap

• M1 — uTLS transport: Chrome/Firefox ClientHello, verified against tls.peet.ws. CLI fetch + fingerprint.
• M2 — Sessions: cookie reuse across requests, retries with exponential backoff + jitter (honouring Retry-After), and http/socks5 proxy support.
• M3 — Concurrent crawler: bounded worker pool (errgroup + semaphore), per-host token-bucket rate limiting, cancellable context + backpressure, and Prometheus metrics. CLI crawl.
• M4 — Server mode: HTTP daemon that keeps sessions hot (per-session clients), with /fetch, session close, Prometheus /metrics, idle eviction and graceful shutdown.
• M5 — Agent bridge: an MCP server (cloudscraper mcp) exposing fetch_protected_url (clean Markdown) and get_cookies — the native Go counterpart to cloudscraper.js's agent story.

Development

make build       # build both binaries into ./bin
make test        # go test -race -short ./...   (offline, deterministic)
make test-net    # full suite incl. network fingerprint tests
make lint        # golangci-lint
make bench       # network benchmark
make fingerprint # build + run `cloudscraper fingerprint`

License

MIT — see LICENSE.

Credits

TLS fingerprinting by refraction-networking/utls. Inspired by the Python cloudscraper and the author's cloudscraper.js.