Documentation
¶
Overview ¶
Package site ingests a published website the project itself publishes, over HTTP and with no credential (spec 0008).
The security boundary is publication: anything this package can fetch, the world can already read. That holds only while no credential can reach a request, so the fetcher has no way to accept one and strips any that appear (R-KB-12).
Index ¶
Constants ¶
const ( DefaultMaxBytes = 2 << 20 DefaultPause = 250 * time.Millisecond DefaultTimeout = 30 * time.Second )
Defaults for the bounds R-KB-15 requires (spec 0008 D7).
const MinRouteStops = 5
MinRouteStops is the fewest stops a route may keep and still be indexed (spec 0008 Q-5).
Variables ¶
var ( // ErrBadOrigin is returned for a configured origin that is not a bare HTTPS // scheme and host. ErrBadOrigin = errors.NewSentinel("site.bad_origin", "site: origin must be https://host with nothing else") // ErrOffOrigin is returned for a URL, or a redirect, that leaves the origin. ErrOffOrigin = errors.NewSentinel("site.off_origin", "site: refused a URL off the origin") // ErrTooLarge is returned for a page over the size bound. ErrTooLarge = errors.NewSentinel("site.too_large", "site: page exceeds the size bound") // ErrNotFound is returned for a page the site no longer serves, which is // skipped rather than fatal (spec 0008 section 6). ErrNotFound = errors.NewSentinel("site.not_found", "site: page not found") )
var DefaultCutoff = time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
DefaultCutoff is the publication date before which a post is not indexed (R-KB-14c).
var DefaultLanes = []string{"orienteering", "pioneering", "first-light", "campfire", "soapbox", "the-expedition"}
DefaultLanes are the editorial lanes of phpboyscout.uk. Every lane is in scope (spec 0008 section 3.4, reversed 2026-08-11).
var ErrBadLane = errors.NewSentinel("site.bad_lane", "site: a lane must be a lowercase slug")
ErrBadLane is returned for a configured lane that is not a lowercase slug.
var ErrNoContent = errors.NewSentinel("site.no_content", "site: page has no content region")
ErrNoContent is returned for a page without the content region its kind expects. Falling back to <main> would index the theme's chrome (R-KB-16).
Functions ¶
Types ¶
type Article ¶
type Article struct {
Sections []Section
}
Article is a post's content region, split at its second- and third-level headings to match the docs' chunk model.
func ExtractArticle ¶
ExtractArticle reads the article content region of a post.
type Entry ¶
type Entry struct {
URL string
Title string
// Lane is the lane slug a post was listed under; empty for a route.
Lane string
Published time.Time
}
Entry is one page a feed declares.
type Fetcher ¶
type Fetcher struct {
// contains filtered or unexported fields
}
Fetcher fetches pages from one origin, one request at a time.
func NewFetcher ¶
NewFetcher returns a fetcher locked to origin, which must be https://host.
type Listing ¶
type Listing struct {
Posts []Entry
Routes []Entry
// Excluded counts posts outside the window or with no usable date.
Excluded int
// OffOrigin counts declared links that left the origin, and were dropped.
OffOrigin int
}
Listing is what discovery found. Only a listing Discover returned without an error is complete, and only a complete one may mark a page absent (Q-3).
type Option ¶
type Option func(*options)
Option configures a Fetcher. There is deliberately no option that supplies a credential.
func WithRootCAs ¶
WithRootCAs replaces the system roots, for tests against httptest.
func WithTimeout ¶
WithTimeout bounds one request, redirects included.
func WithVersion ¶
WithVersion names the bot's version in the User-Agent.
type Page ¶
type Page struct {
// URL is where the body came from, after any same-origin redirect.
URL string
Body []byte
Validators Validators
NotModified bool
}
Page is one fetched page.
type Route ¶
Route is a guided topic page reduced to what R-KB-14d allows it to say: its title, its framing sentence and its ordered stops. Stop blurbs are dropped.
func ExtractRoute ¶
ExtractRoute reads a guided topic page, resolving each stop against pageURL.
type Section ¶
Section is one heading section of an article. The intro before the first heading has an empty Heading and Anchor, and cites the page itself.
type Validators ¶
Validators are a page's HTTP cache validators, sent back on the next fetch so an unchanged page costs no body.