Documentation
¶
Overview ¶
Package webfetch provides an HTTP fetch tool with content extraction for agent use.
Index ¶
Constants ¶
View Source
const ( // ModeText extracts readable text from the HTML. ModeText = "text" // ModeHTML returns raw HTML truncated to max length. ModeHTML = "html" // ModeMarkdown returns simplified markdown. ModeMarkdown = "markdown" )
Variables ¶
View Source
var ErrBlockedURL = errors.New("URL targets a blocked internal/private network address")
ErrBlockedURL is returned when a URL targets an internal or private network address.
Functions ¶
func ValidateURLForP2P ¶
ValidateURLForP2P checks that a URL is safe in a P2P context. It blocks file:// schemes and URLs resolving to internal/private network addresses.
Types ¶
type FetchResult ¶
type FetchResult struct {
URL string `json:"url"`
Title string `json:"title"`
Content string `json:"content"`
ContentLength int `json:"content_length"`
Truncated bool `json:"truncated"`
}
FetchResult holds the extracted content from a fetched web page.
func Fetch ¶
func Fetch(ctx context.Context, rawURL string, mode string, maxLength int, p2pSafe bool) (*FetchResult, error)
Fetch downloads a web page and extracts content in the specified mode. Supported modes: "text" (default), "html" (raw HTML), "markdown" (simplified). maxLength controls the maximum character length of the returned content. When p2pSafe is true, each redirect target is validated before following.
Source Files
Click to show internal directories.
Click to hide internal directories.