Directories
¶
| Path | Synopsis |
|---|---|
|
Package antibot implements an adaptive anti-bot engine with block detection, stealth profiles, and human behaviour simulation.
|
Package antibot implements an adaptive anti-bot engine with block detection, stealth profiles, and human behaviour simulation. |
|
Package apiserver implements a REST and WebSocket API server for ScrapeGoat, enabling microservice deployment and programmatic access to crawling, extraction, and job management.
|
Package apiserver implements a REST and WebSocket API server for ScrapeGoat, enabling microservice deployment and programmatic access to crawling, extraction, and job management. |
|
Package changedetect implements a content monitoring engine that detects changes on web pages using multiple diffing strategies and notifications.
|
Package changedetect implements a content monitoring engine that detects changes on web pages using multiple diffing strategies and notifications. |
|
Package crawlgraph implements a directed graph of URLs discovered during crawling, with support for SQLite persistence, multiple export formats, and replay strategies for re-crawling changed or failed pages.
|
Package crawlgraph implements a directed graph of URLs discovered during crawling, with support for SQLite persistence, multiple export formats, and replay strategies for re-crawling changed or failed pages. |
|
fingerprint
Package fingerprint makes ScrapeGoat's TLS handshake and HTTP headers look like the browser it claims to be.
|
Package fingerprint makes ScrapeGoat's TLS handshake and HTTP headers look like the browser it claims to be. |
|
Package llmextract provides a schema-driven LLM extraction pipeline that uses large language models to extract structured data from HTML content.
|
Package llmextract provides a schema-driven LLM extraction pipeline that uses large language models to extract structured data from HTML content. |
|
Package mcp implements a Model Context Protocol (MCP) server that exposes ScrapeGoat's crawling, extraction, and SEO tools to AI agents such as Claude, GPT-4, Cursor, and Cline via JSON-RPC 2.0 over stdio or HTTP/SSE.
|
Package mcp implements a Model Context Protocol (MCP) server that exposes ScrapeGoat's crawling, extraction, and SEO tools to AI agents such as Claude, GPT-4, Cursor, and Cline via JSON-RPC 2.0 over stdio or HTTP/SSE. |
|
Package observability exposes ScrapeGoat's operational metrics.
|
Package observability exposes ScrapeGoat's operational metrics. |
|
sdk
Package plugin/sdk provides helper types and functions for building ScrapeGoat plugins using the init() registration pattern.
|
Package plugin/sdk provides helper types and functions for building ScrapeGoat plugins using the init() registration pattern. |
|
Package safety enforces the network trust boundary for every outbound fetch.
|
Package safety enforces the network trust boundary for every outbound fetch. |
|
Package testutil holds helpers shared across ScrapeGoat's test packages.
|
Package testutil holds helpers shared across ScrapeGoat's test packages. |
|
Package transforms extends the pipeline with schema validation, data transformation, and enrichment middlewares.
|
Package transforms extends the pipeline with schema validation, data transformation, and enrichment middlewares. |
Click to show internal directories.
Click to hide internal directories.