Documentation
¶
Overview ¶
Package feed fetches and converts syndication feeds.
The fetch side is a bounded worker pool (Run) over per-feed fetches (fetchOne); results flow to a single collector goroutine that performs ALL database writes — the channel is the queue, and services need no mutexes. Failure is per-feed and never escapes: panics become EPANIC, errors become classified FeedUpdates feeding exponential backoff (1h doubling to a 24h cap, reset by any successful contact including a 304).
Politeness is deliberate and layered: conditional GET, per-host serialization, honest User-Agent (per-feed overrides are the operator's explicit call), Accept and Accept-Language headers, a 10 MiB body cap, and HTTP/1.1 only — CDN bot-mitigation fingerprints the h2 connection itself, and a batch fetcher gains nothing from h2 (see docs/design.md).
The content pipeline order is LOAD-BEARING:
select body -> Sanitize (strip, absolutize, structural policy)
-> NormalizeTypography -> Highlight -> keyword filters
Sanitize establishes the invariants everything downstream assumes; typography and highlighting never touch code; filters match what a reader would see. Filters, strip selectors, and fetch overrides live only in config and are overlaid onto DB-sourced feeds at the start of every Run.
Index ¶
Constants ¶
const ( DateFromFeed = "feed" // gofeed parsed it (standard layouts) DateRescued = "rescued" // a loose layout matched, config timezone DateNone = "" // no usable date: fetch-time fallback applies )
Date-resolution tiers, reported by resolvePublished and shown by `firehose test` so a date-blind feed announces itself at onboarding instead of days later as a "just now" ghost in the river.
Variables ¶
This section is empty.
Functions ¶
func CheckConfig ¶
CheckConfig validates the feed-level parts of the config that only this package can check (strip selectors compile). Called by `firehose check`.
Types ¶
type Fetcher ¶
type Fetcher struct {
// Now is injectable for tests (WTF-style testable time).
Now func() time.Time
// Force ignores per-feed backoff gates: every feed is attempted now.
Force bool
// contains filtered or unexported fields
}
Fetcher runs one fetch pass over all due feeds.
func NewFetcher ¶
func NewFetcher(cfg *firehose.Config, feeds firehose.FeedService, items firehose.ItemService) *Fetcher
NewFetcher constructs a Fetcher over the given services.
func (*Fetcher) Run ¶
Run fetches all due feeds (those not gated by backoff), fanning out to cfg.Fetch.Concurrency workers and collecting all writes on this goroutine — the single-writer invariant. It returns the first cache-write error encountered; per-feed fetch/parse failures are not errors here, they are recorded as feed health status.
type Probe ¶
type Probe struct {
RequestURL string
FinalURL string
Hops []ProbeHop // redirect chain, in order
ChainPermanent bool // every hop 301/308: a real fetch would persist FinalURL
Timezone string // zone assumed for zoneless rescued dates
Status int
Proto string // negotiated protocol (expect HTTP/1.1 by design)
ContentType string
ETag string
LastModified string
Server string
BodyBytes int
BodySnippet string // first bytes of an unparseable body (block pages!)
FeedType string
FeedVersion string
FeedTitle string
ItemCount int
// Date visibility: how every item's published time resolved. Unparsed
// items fall to the fetch-time fallback and may never age out of the
// display window — the probe's job is to say so at onboarding.
DatesFeed int
DatesRescued int
DatesUnparsed int
First *ProbeItem
}
Probe is the result of a diagnostic single-feed fetch (`firehose test`). It ignores the cache entirely: no conditional headers, nothing written.
func RunProbe ¶
func RunProbe(ctx context.Context, fetch firehose.FetchConfig, preq ProbeRequest) (*Probe, error)
RunProbe fetches and analyses a single feed URL with the configured fetch politeness (User-Agent, Accept, Accept-Language, timeout) but no cache interaction. A non-nil Probe is always returned, carrying whatever was learned before the failure; err classifies the failure.
type ProbeItem ¶
type ProbeItem struct {
Title string
Link string
GUID string
Published time.Time
PublishedTier string // DateFromFeed | DateRescued | DateNone
PublishedRaw string // verbatim raw date string (diagnostics)
FullContent bool // content:encoded present
Words int
LeadImage string
}
ProbeItem is the analysis of the first item, run through the same sanitize/inference pipeline the fetcher uses.
type ProbeRequest ¶
type ProbeRequest struct {
URL string
UserAgent string // overrides fetch.UserAgent when set
Headers map[string]string // set verbatim, last
Timezone string // zone for zoneless rescued dates; UTC when unset
}
ProbeRequest describes the diagnostic request: URL plus the same optional overrides a per-feed config block can apply, so `firehose test` can bisect exactly what a hostile CDN wants.
Source Files