web_fetch

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 29, 2026 License: Apache-2.0, MIT Imports: 26 Imported by: 0

Documentation

Overview

Package web_fetch provides a public URL content fetcher with SSRF protection.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FetchURLContent 

func FetchURLContent(ctx context.Context, rawURL string) (string, error)

FetchURLContent preserves the existing package-level API for the chat pipeline.

func PageFetchTimeout added in v0.9.0 

func PageFetchTimeout() time.Duration

PageFetchTimeout is the shared budget for one agent web_fetch page read. The HTTP client, the fetch context, and chromedp all use this value. Chromedp's deadline is a child of the fetch context, so the budgets do not add.

WEKNORA_WEB_FETCH_TIMEOUT accepts a Go duration (150s, 3m) or a bare positive number of seconds (180), the same pattern as WEKNORA_AGENT_LLM_TIMEOUT. Empty, invalid, and non-positive values use defaultPageFetchTimeout.

Types

type ErrorCode 

type ErrorCode string

ErrorCode identifies the stage and class of a fetch failure. 

const (
	ErrorInvalidURL         ErrorCode = "invalid_url"
	ErrorDNS                ErrorCode = "dns_failed"
	ErrorTimeout            ErrorCode = "connection_timeout"
	ErrorTLS                ErrorCode = "tls_failed"
	ErrorHTTP403            ErrorCode = "http_403"
	ErrorHTTP429            ErrorCode = "http_429"
	ErrorHTTP5xx            ErrorCode = "http_5xx"
	ErrorHTTPStatus         ErrorCode = "http_status"
	ErrorSSRFRejected       ErrorCode = "ssrf_rejected"
	ErrorRedirectRejected   ErrorCode = "redirect_rejected"
	ErrorRead               ErrorCode = "read_failed"
	ErrorHTMLParse          ErrorCode = "html_parse_failed"
	ErrorEmptyContent       ErrorCode = "empty_content"
	ErrorConnection         ErrorCode = "connection_failed"
	ErrorBodyTooLarge       ErrorCode = "body_too_large"
	ErrorUnsupportedContent ErrorCode = "unsupported_content"
	ErrorSnapshotExpired    ErrorCode = "snapshot_expired"
)

Fetch failure codes distinguish retryable network errors from permanent failures.

func ErrorDetails 

func ErrorDetails(err error) (ErrorCode, bool, string)

ErrorDetails returns stable fields suitable for tool responses and logs.

type FetchError 

type FetchError struct {
	Code      ErrorCode
	Retryable bool
	Err       error
}

FetchError carries stable, machine-readable failure details.

func (*FetchError) Error 

func (e *FetchError) Error() string

func (*FetchError) Unwrap 

func (e *FetchError) Unwrap() error

type Fetcher 

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher fetches and extracts public web pages through an SSRF-safe client.

func NewFetcher 

func NewFetcher() *Fetcher

NewFetcher creates a production fetcher with DNS and redirect SSRF guards.

func NewPipelineFetcher 

func NewPipelineFetcher() *Fetcher

NewPipelineFetcher creates an HTTP-only fetcher for the chat pipeline. It keeps the pre-refactor 15s timeout and does not launch Chromium.

func (*Fetcher) Fetch 

func (f *Fetcher) Fetch(ctx context.Context, rawURL string) (string, error)

Fetch downloads a page and returns clean text content.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.