fetcher

package
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 30, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package fetcher provides HTTP content fetching and processing functionality.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type FetchRequest 

type FetchRequest struct {
	URL        string
	MaxLength  *int
	StartIndex *int
	Raw        bool
}

FetchRequest holds the parameters for a fetch request

type HTTPFetcher 

type HTTPFetcher struct {
	// contains filtered or unexported fields
}

HTTPFetcher handles HTTP requests and content retrieval

func NewHTTPFetcher 

func NewHTTPFetcher(
	httpClient *http.Client,
	robotsChecker *robots.Checker,
	contentProcessor *processor.ContentProcessor,
	userAgent string,
) *HTTPFetcher

NewHTTPFetcher creates a new HTTP fetcher instance

func (*HTTPFetcher) FetchURL 

func (f *HTTPFetcher) FetchURL(req *FetchRequest) (string, error)

FetchURL retrieves and processes content from the specified URL

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.