Documentation
¶
Overview ¶
Package webfetch reads a URL as markdown for a caller that pays per token: it asks the site for a form meant for agents, follows the llms.txt conventions, and falls back to the readable part of the HTML.
The package follows the /llms.txt proposal for content negotiation, for the `.md` variants of a URL, and for the `llms.txt` file itself. A Rule in the Registry gives a site its own selector, its own furniture list, or a mirror, for the pages the generic pass reads badly.
Index ¶
Constants ¶
const ( // MethodNegotiated marks content the site served in a form meant for // agents. MethodNegotiated = "negotiated" // MethodAlternate marks a `rel="alternate" type="text/markdown"` link, // from the HTML or from a `Link:` header. MethodAlternate = "alternate" // MethodVariant marks the `.md` variant of the URL. MethodVariant = "md-variant" // MethodLlmsTxt marks an `llms.txt` file or one of its entries. MethodLlmsTxt = "llms.txt" // MethodRulePrefix opens the name of the site rule that served the page. MethodRulePrefix = "rule:" // MethodGeneric marks HTML with its furniture stripped. MethodGeneric = "generic" )
Method names the branch Read took.
const ( // UserAgent identifies the command to a site, and names the product token // a robots.txt file matches. UserAgent = "agentic-web-fetch/1.0 (+https://github.com/kafo-dev/agentic-tools)" // DefaultTimeout bounds one request. DefaultTimeout = 32 * time.Second )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher gets pages with colly, under the rules the site publishes in its robots.txt file. A Fetcher holds one collector, so the rules it has read serve every request and a site answers for its robots.txt once.
type Options ¶
type Options struct {
// Timeout bounds one request; zero and below keep [DefaultTimeout].
Timeout time.Duration
// UserAgent replaces [UserAgent] when it carries text.
UserAgent string
}
Options configures a Fetcher. The zero value keeps every default.
type Page ¶
type Page struct {
// URL is the address the response came from, after redirects.
URL *url.URL
// Status is the HTTP status code.
Status int
// Type is the media type of the body, without its parameters.
Type string
// Header holds the response headers.
Header http.Header
// Body holds the response body.
Body []byte
}
Page is one HTTP response.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds the rules, indexed by host.
type Result ¶
type Result struct {
// URL is the address the content came from, after redirects and
// negotiation.
URL string
// Method names the branch that produced the content: one of
// [MethodNegotiated], [MethodAlternate], [MethodVariant], [MethodLlmsTxt]
// and [MethodGeneric], or [MethodRulePrefix] followed by a rule's name.
Method string
// Content holds the markdown.
Content string
}
Result holds one page's content and the branch that produced it.
type Rule ¶
type Rule struct {
// Name labels the rule in the reader's output, which reports it as
// `rule:<name>`.
Name string
// Hosts lists the hosts the rule serves. An entry starting with `*.`
// serves that domain and every subdomain of it.
Hosts []string
// Mirror rewrites a page URL into one that serves the same content in a
// form an agent reads without cleaning: a raw file, an API, a mirror that
// allows scraping. It reports false when the URL has no mirror.
Mirror func(u *url.URL) (string, bool)
// Select holds the CSS selector of the readable part. An empty selector
// leaves the choice to the generic pass.
Select string
// Drop lists CSS selectors of page furniture to remove before conversion,
// in addition to the elements the generic pass always drops.
Drop []string
}
Rule tells the reader how to read one site's pages.
Source Files