feed

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: AGPL-3.0 Imports: 26 Imported by: 0

Documentation

Overview

Package feed reads the feeds recurring digests are built from: RSS 2.0 (and the RSS 1.0 and 0.9x it grew from), Atom 1.0 and JSON Feed 1.0 and 1.1. It parses with the standard library's XML and JSON decoders and golang.org/x/net/html (already a dependency) rather than a feed library: a digest needs a dozen fields per item, and parsing them ourselves keeps the limits (size, items, nesting) and the failure messages under our control. It also finds the feed a web page links to (discover.go) and fetches feeds through the SSRF guard (fetch.go).

Index

Constants

View Source
const (
	Timeout      = 15 * time.Second
	MaxBytes     = 5 << 20
	MaxRedirects = 3
)

Limits of a read (brief 3.1): 15 seconds, 5 MB, up to 3 redirects.

View Source
const (
	ErrUnreachable = "unreachable" // DNS, connection refused or reset, TLS
	ErrTimeout     = "timeout"
	ErrBlocked     = "blocked" // the SSRF guard refused the address
	ErrTooLarge    = "too_large"
	ErrStatus      = "status" // an HTTP error status
	ErrRedirects   = "redirects"
	ErrAddress     = "address" // not a usable URL
)

Error kinds.

View Source
const (
	KindRSS  = "rss"
	KindAtom = "atom"
	KindJSON = "json"
)

Kinds of feed.

View Source
const MaxItems = 200

MaxItems is how many items of a feed are kept, newest first. A feed's back catalogue can be long; a digest only ever looks at its newest posts.

Variables

This section is empty.

Functions

func Excerpt

func Excerpt(src string) string

Excerpt turns a feed's HTML (a description, a summary, content) into plain text: tags, scripts and styles go, entities are decoded and whitespace collapses. WordPress's "The post … appeared first on …" footer and a trailing "[…]" are dropped.

func FirstLines

func FirstLines(data []byte, n int) []string

FirstLines returns up to n of the document's first non-empty lines, each at most 120 characters, for error messages.

func GuardText

func GuardText(address string, err error) (i18n.Text, bool)

GuardText is why the guard refused an address it checked before reading, when it refused it for where it points (a private address): false for its other answers, which keep their own words.

func IsNotFeed

func IsNotFeed(err error) bool

IsNotFeed reports whether err says the document isn't a feed.

func LooksLikeHTML

func LooksLikeHTML(data []byte) bool

LooksLikeHTML reports whether a document is a web page.

func NormalizeURL

func NormalizeURL(raw string) (string, error)

NormalizeURL turns what someone typed ("blog.acme.io", "blog.acme.io/feed/") into an absolute http(s) URL.

func ParseDate

func ParseDate(s string) (time.Time, bool)

ParseDate reads the date formats feeds use: RFC 1123 and its variants (RSS), RFC 3339 (Atom, JSON Feed, dc:date) and a few common slips.

func TextOf

func TextOf(err error, address string) i18n.Text

TextOf is why a read failed, as a message, when err knows it: a FetchError, an AddressError, or (for the address it was read at) a page that isn't a feed or links to none. It is zero for other errors.

func Words

func Words(text string, n int) string

Words shortens text to at most n words, ending with "…" when it was cut.

Types

type AddressError

type AddressError struct {

	// Text is the same sentence, to write in a reader's language.
	Text i18n.Text
	// contains filtered or unexported fields
}

AddressError is an address that can't be a feed's: empty, not an address, or not http(s).

func (*AddressError) Error

func (e *AddressError) Error() string

type Feed

type Feed struct {
	Kind  string
	Title string
	// Link is the site the feed belongs to.
	Link  string
	Items []Item
}

Feed is a parsed feed.

func Parse

func Parse(data []byte, base *url.URL) (*Feed, error)

Parse reads a feed. base resolves relative links (the feed's own URL).

type FeedLink struct {
	URL   string
	Title string
	// Kind is the feed's kind from the link's type (rss, atom, json).
	Kind string
}

FeedLink is a feed a page links to in its head.

func Discover

func Discover(page []byte, base *url.URL) []FeedLink

Discover lists the feeds a page links to with <link rel="alternate">, in the page's order. Comment feeds come last: WordPress links a site's comments feed right after its posts feed.

type FetchError

type FetchError struct {
	Kind string
	// URL is the address that failed (the redirect target when it was one).
	URL    string
	Status int
	// Detail is the English sentence ("blog.acme.io didn't answer: …"), and
	// Text the same sentence as a message, for a reader's language.
	Detail string
	Text   i18n.Text
	Hops   []Hop
}

FetchError explains why a read failed, in terms a person can act on.

func AsFetchError

func AsFetchError(err error) (*FetchError, bool)

AsFetchError returns err as a *FetchError.

func (*FetchError) Error

func (e *FetchError) Error() string

func (*FetchError) Host

func (e *FetchError) Host() string

Host of the address that failed ("blog.acme.io").

type Fetcher

type Fetcher struct {
	Policy netguard.Policy
	// Transport replaces the guarded transport (tests, and the demo's
	// recorded sites); the address checks still run.
	Transport http.RoundTripper
	Timeout   time.Duration
	MaxBytes  int64
	UserAgent string
}

Fetcher reads feeds, pages and images from addresses people type, through the same SSRF guard as importers and webhooks: every address (the first and each redirect's) is checked before connecting, and again at connect time against the resolved IP, so a name can't point at loopback, private, link-local or metadata addresses.

func (*Fetcher) Find

func (f *Fetcher) Find(ctx context.Context, address string) (*Found, error)

Find reads an address as a feed and, when it is a web page, reads the feeds the page links to in its head (blog.acme.io finds blog.acme.io/feed/). The first one that parses wins.

func (*Fetcher) Get

func (f *Fetcher) Get(ctx context.Context, r Request) (*Response, error)

Get reads an address, following up to MaxRedirects redirects, each target checked by the guard first. A 304 answer to a conditional request is NotModified with no body.

func (*Fetcher) Read

func (f *Fetcher) Read(ctx context.Context, r Request) (*Result, error)

Read fetches and parses a feed. A document that isn't a feed fails with a *NotFeedError carrying its first lines.

type Found

type Found struct {
	*Result
	// Page is the page address the feed was linked from ("" when the
	// address was the feed itself).
	Page string
}

Found is the feed found from an address someone typed.

type Hop

type Hop struct {
	URL      string `json:"url"`
	Status   int    `json:"status"`
	Location string `json:"location,omitempty"`
}

Hop is one address answered on the way: a redirect or the final answer.

type Item

type Item struct {
	// Key identifies the item across reads: its guid or id, else its link,
	// so an edited post isn't sent twice.
	Key       string
	URL       string
	Title     string
	Author    string
	Published *time.Time
	// Categories and Tags are the item's terms. A term is a tag when the
	// feed says so (an RSS category domain or an Atom scheme naming a tag
	// taxonomy); JSON Feed tags are categories.
	Categories []string
	Tags       []string
	// Excerpt is plain text: HTML, scripts and styles are stripped.
	Excerpt string
	// Image is the item's picture (media, enclosure, else the first image
	// of its content) and ImageAlt that image's own alt text, when it has one.
	Image    string
	ImageAlt string
}

Item is one entry of a feed.

type NoFeedLinkError

type NoFeedLinkError struct{ Page string }

NoFeedLinkError means a page links to no feed.

func (*NoFeedLinkError) Error

func (e *NoFeedLinkError) Error() string

func (*NoFeedLinkError) Text

func (e *NoFeedLinkError) Text() i18n.Text

Text is "blog.acme.io is a web page that links to no feed".

type NotFeedError

type NotFeedError struct {
	// HTML is true when the document is a web page.
	HTML bool
	// Lines are the document's first lines, for showing what answered.
	Lines []string
	// Reason describes what was found instead ("an XML document whose root is <urlset>").
	Reason string
	// Code names the reason for its message (notFeed.<code>), and Arg is
	// what the message names: the parser's own words, or the root element.
	Code string
	Arg  string
}

NotFeedError means the document isn't a feed: an HTML page (a maintenance page, the site's home page), or XML or JSON of another kind.

func (*NotFeedError) Error

func (e *NotFeedError) Error() string

func (*NotFeedError) Text

func (e *NotFeedError) Text(address string) i18n.Text

Text is why the address isn't a feed, as the sentence "blog.acme.io/feed/ isn't a feed: the document is empty", for an address written without its scheme.

type Request

type Request struct {
	URL      string
	ETag     string
	Modified string
	// Accept overrides the feed Accept header (images).
	Accept string
}

Request is a conditional read.

type Response

type Response struct {
	// URL is the final address, after redirects.
	URL         string
	Status      int
	NotModified bool
	Body        []byte
	ContentType string
	ETag        string
	Modified    string
	Hops        []Hop
	// Moved is the target of a permanent redirect (301, 308) on the way.
	Moved    string
	Duration time.Duration
}

Response is what answered.

type Result

type Result struct {
	*Response
	// Feed is nil when the feed didn't change (NotModified).
	Feed *Feed
}

Result is a read feed with what answered.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL