Documentation
¶
Overview ¶
Package feed reads the feeds recurring digests are built from: RSS 2.0 (and the RSS 1.0 and 0.9x it grew from), Atom 1.0 and JSON Feed 1.0 and 1.1. It parses with the standard library's XML and JSON decoders and golang.org/x/net/html (already a dependency) rather than a feed library: a digest needs a dozen fields per item, and parsing them ourselves keeps the limits (size, items, nesting) and the failure messages under our control. It also finds the feed a web page links to (discover.go) and fetches feeds through the SSRF guard (fetch.go).
Index ¶
- Constants
- func Excerpt(src string) string
- func FirstLines(data []byte, n int) []string
- func GuardText(address string, err error) (i18n.Text, bool)
- func IsNotFeed(err error) bool
- func LooksLikeHTML(data []byte) bool
- func NormalizeURL(raw string) (string, error)
- func ParseDate(s string) (time.Time, bool)
- func TextOf(err error, address string) i18n.Text
- func Words(text string, n int) string
- type AddressError
- type Feed
- type FeedLink
- type FetchError
- type Fetcher
- type Found
- type Hop
- type Item
- type NoFeedLinkError
- type NotFeedError
- type Request
- type Response
- type Result
Constants ¶
const ( Timeout = 15 * time.Second MaxBytes = 5 << 20 MaxRedirects = 3 )
Limits of a read (brief 3.1): 15 seconds, 5 MB, up to 3 redirects.
const ( ErrUnreachable = "unreachable" // DNS, connection refused or reset, TLS ErrTimeout = "timeout" ErrBlocked = "blocked" // the SSRF guard refused the address ErrTooLarge = "too_large" ErrStatus = "status" // an HTTP error status ErrRedirects = "redirects" ErrAddress = "address" // not a usable URL )
Error kinds.
const ( KindRSS = "rss" KindAtom = "atom" KindJSON = "json" )
Kinds of feed.
const MaxItems = 200
MaxItems is how many items of a feed are kept, newest first. A feed's back catalogue can be long; a digest only ever looks at its newest posts.
Variables ¶
This section is empty.
Functions ¶
func Excerpt ¶
Excerpt turns a feed's HTML (a description, a summary, content) into plain text: tags, scripts and styles go, entities are decoded and whitespace collapses. WordPress's "The post … appeared first on …" footer and a trailing "[…]" are dropped.
func FirstLines ¶
FirstLines returns up to n of the document's first non-empty lines, each at most 120 characters, for error messages.
func GuardText ¶
GuardText is why the guard refused an address it checked before reading, when it refused it for where it points (a private address): false for its other answers, which keep their own words.
func LooksLikeHTML ¶
LooksLikeHTML reports whether a document is a web page.
func NormalizeURL ¶
NormalizeURL turns what someone typed ("blog.acme.io", "blog.acme.io/feed/") into an absolute http(s) URL.
func ParseDate ¶
ParseDate reads the date formats feeds use: RFC 1123 and its variants (RSS), RFC 3339 (Atom, JSON Feed, dc:date) and a few common slips.
Types ¶
type AddressError ¶
type AddressError struct {
// Text is the same sentence, to write in a reader's language.
Text i18n.Text
// contains filtered or unexported fields
}
AddressError is an address that can't be a feed's: empty, not an address, or not http(s).
func (*AddressError) Error ¶
func (e *AddressError) Error() string
type Feed ¶
type Feed struct {
Kind string
Title string
// Link is the site the feed belongs to.
Link string
Items []Item
}
Feed is a parsed feed.
type FeedLink ¶
type FeedLink struct {
URL string
Title string
// Kind is the feed's kind from the link's type (rss, atom, json).
Kind string
}
FeedLink is a feed a page links to in its head.
type FetchError ¶
type FetchError struct {
Kind string
// URL is the address that failed (the redirect target when it was one).
URL string
Status int
// Detail is the English sentence ("blog.acme.io didn't answer: …"), and
// Text the same sentence as a message, for a reader's language.
Detail string
Text i18n.Text
Hops []Hop
}
FetchError explains why a read failed, in terms a person can act on.
func AsFetchError ¶
func AsFetchError(err error) (*FetchError, bool)
AsFetchError returns err as a *FetchError.
func (*FetchError) Error ¶
func (e *FetchError) Error() string
func (*FetchError) Host ¶
func (e *FetchError) Host() string
Host of the address that failed ("blog.acme.io").
type Fetcher ¶
type Fetcher struct {
Policy netguard.Policy
// Transport replaces the guarded transport (tests, and the demo's
// recorded sites); the address checks still run.
Transport http.RoundTripper
Timeout time.Duration
MaxBytes int64
UserAgent string
}
Fetcher reads feeds, pages and images from addresses people type, through the same SSRF guard as importers and webhooks: every address (the first and each redirect's) is checked before connecting, and again at connect time against the resolved IP, so a name can't point at loopback, private, link-local or metadata addresses.
func (*Fetcher) Find ¶
Find reads an address as a feed and, when it is a web page, reads the feeds the page links to in its head (blog.acme.io finds blog.acme.io/feed/). The first one that parses wins.
type Found ¶
type Found struct {
*Result
// Page is the page address the feed was linked from ("" when the
// address was the feed itself).
Page string
}
Found is the feed found from an address someone typed.
type Hop ¶
type Hop struct {
URL string `json:"url"`
Status int `json:"status"`
Location string `json:"location,omitempty"`
}
Hop is one address answered on the way: a redirect or the final answer.
type Item ¶
type Item struct {
// Key identifies the item across reads: its guid or id, else its link,
// so an edited post isn't sent twice.
Key string
URL string
Title string
Author string
Published *time.Time
// Categories and Tags are the item's terms. A term is a tag when the
// feed says so (an RSS category domain or an Atom scheme naming a tag
// taxonomy); JSON Feed tags are categories.
Categories []string
Tags []string
// Excerpt is plain text: HTML, scripts and styles are stripped.
Excerpt string
// Image is the item's picture (media, enclosure, else the first image
// of its content) and ImageAlt that image's own alt text, when it has one.
Image string
ImageAlt string
}
Item is one entry of a feed.
type NoFeedLinkError ¶
type NoFeedLinkError struct{ Page string }
NoFeedLinkError means a page links to no feed.
func (*NoFeedLinkError) Error ¶
func (e *NoFeedLinkError) Error() string
func (*NoFeedLinkError) Text ¶
func (e *NoFeedLinkError) Text() i18n.Text
Text is "blog.acme.io is a web page that links to no feed".
type NotFeedError ¶
type NotFeedError struct {
// HTML is true when the document is a web page.
HTML bool
// Lines are the document's first lines, for showing what answered.
Lines []string
// Reason describes what was found instead ("an XML document whose root is <urlset>").
Reason string
// Code names the reason for its message (notFeed.<code>), and Arg is
// what the message names: the parser's own words, or the root element.
Code string
Arg string
}
NotFeedError means the document isn't a feed: an HTML page (a maintenance page, the site's home page), or XML or JSON of another kind.
func (*NotFeedError) Error ¶
func (e *NotFeedError) Error() string
type Request ¶
type Request struct {
URL string
ETag string
Modified string
// Accept overrides the feed Accept header (images).
Accept string
}
Request is a conditional read.
type Response ¶
type Response struct {
// URL is the final address, after redirects.
URL string
Status int
NotModified bool
Body []byte
ContentType string
ETag string
Modified string
Hops []Hop
// Moved is the target of a permanent redirect (301, 308) on the way.
Moved string
Duration time.Duration
}
Response is what answered.