site

package
v0.15.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package site ingests a published website the project itself publishes, over HTTP and with no credential (spec 0008).

The security boundary is publication: anything this package can fetch, the world can already read. That holds only while no credential can reach a request, so the fetcher has no way to accept one and strips any that appear (R-KB-12).

Index

Constants

View Source
const (
	DefaultMaxBytes = 2 << 20
	DefaultPause    = 250 * time.Millisecond
	DefaultTimeout  = 30 * time.Second
)

Defaults for the bounds R-KB-15 requires (spec 0008 D7).

View Source
const MinRouteStops = 5

MinRouteStops is the fewest stops a route may keep and still be indexed (spec 0008 Q-5).

Variables

View Source
var (
	// ErrBadOrigin is returned for a configured origin that is not a bare HTTPS
	// scheme and host.
	ErrBadOrigin = errors.NewSentinel("site.bad_origin", "site: origin must be https://host with nothing else")

	// ErrOffOrigin is returned for a URL, or a redirect, that leaves the origin.
	ErrOffOrigin = errors.NewSentinel("site.off_origin", "site: refused a URL off the origin")

	// ErrTooLarge is returned for a page over the size bound.
	ErrTooLarge = errors.NewSentinel("site.too_large", "site: page exceeds the size bound")

	// ErrNotFound is returned for a page the site no longer serves, which is
	// skipped rather than fatal (spec 0008 section 6).
	ErrNotFound = errors.NewSentinel("site.not_found", "site: page not found")
)
View Source
var DefaultCutoff = time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)

DefaultCutoff is the publication date before which a post is not indexed (R-KB-14c).

View Source
var DefaultLanes = []string{"orienteering", "pioneering", "first-light", "campfire", "soapbox", "the-expedition"}

DefaultLanes are the editorial lanes of phpboyscout.uk. Every lane is in scope (spec 0008 section 3.4, reversed 2026-08-11).

View Source
var ErrBadLane = errors.NewSentinel("site.bad_lane", "site: a lane must be a lowercase slug")

ErrBadLane is returned for a configured lane that is not a lowercase slug.

View Source
var ErrNoContent = errors.NewSentinel("site.no_content", "site: page has no content region")

ErrNoContent is returned for a page without the content region its kind expects. Falling back to <main> would index the theme's chrome (R-KB-16).

Functions

func IsLane

func IsLane(lane string) bool

IsLane reports whether lane is a lane slug discovery will accept.

func LaneLabel

func LaneLabel(slug string) string

LaneLabel names a lane slug as the site does, with its note where it has one.

Types

type Article

type Article struct {
	Sections []Section
}

Article is a post's content region, split at its second- and third-level headings to match the docs' chunk model.

func ExtractArticle

func ExtractArticle(body []byte) (Article, error)

ExtractArticle reads the article content region of a post.

type Entry

type Entry struct {
	URL   string
	Title string
	// Lane is the lane slug a post was listed under; empty for a route.
	Lane      string
	Published time.Time
}

Entry is one page a feed declares.

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher fetches pages from one origin, one request at a time.

func NewFetcher

func NewFetcher(origin string, opts ...Option) (*Fetcher, error)

NewFetcher returns a fetcher locked to origin, which must be https://host.

func (*Fetcher) Get

func (f *Fetcher) Get(ctx context.Context, rawURL string, prev Validators) (Page, error)

Get fetches rawURL, which must be on the origin. A 404 or 410 is ErrNotFound, and a 304 against prev is a Page with NotModified set.

func (*Fetcher) Origin

func (f *Fetcher) Origin() string

Origin is the scheme and host this fetcher is locked to, without a slash.

type Listing

type Listing struct {
	Posts  []Entry
	Routes []Entry
	// Excluded counts posts outside the window or with no usable date.
	Excluded int
	// OffOrigin counts declared links that left the origin, and were dropped.
	OffOrigin int
}

Listing is what discovery found. Only a listing Discover returned without an error is complete, and only a complete one may mark a page absent (Q-3).

func Discover

func Discover(ctx context.Context, f *Fetcher, lanes []string, cutoff time.Time) (Listing, error)

Discover reads the lane feeds and the topics feed (R-KB-14, R-KB-14a). Posts published before cutoff, or with no usable date, are never listed, so they are never fetched (R-KB-14c). A feed that fails is reported in the error, and what the others listed is still returned.

type Option

type Option func(*options)

Option configures a Fetcher. There is deliberately no option that supplies a credential.

func WithMaxBytes

func WithMaxBytes(n int64) Option

WithMaxBytes bounds a page's body.

func WithPause

func WithPause(d time.Duration) Option

WithPause sets the gap between one request finishing and the next starting.

func WithRootCAs

func WithRootCAs(pool *x509.CertPool) Option

WithRootCAs replaces the system roots, for tests against httptest.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout bounds one request, redirects included.

func WithVersion

func WithVersion(v string) Option

WithVersion names the bot's version in the User-Agent.

type Page

type Page struct {
	// URL is where the body came from, after any same-origin redirect.
	URL         string
	Body        []byte
	Validators  Validators
	NotModified bool
}

Page is one fetched page.

type Route

type Route struct {
	Title string
	Strap string
	Stops []Stop
}

Route is a guided topic page reduced to what R-KB-14d allows it to say: its title, its framing sentence and its ordered stops. Stop blurbs are dropped.

func ExtractRoute

func ExtractRoute(pageURL string, body []byte) (Route, error)

ExtractRoute reads a guided topic page, resolving each stop against pageURL.

func (Route) Within

func (r Route) Within(listed map[string]bool) (Route, bool)

Within keeps the stops discovery listed, so a route cannot lead past an excluded lane or the cutoff (R-KB-14e). A route left with fewer than MinRouteStops is not a guide, and ok is false.

type Section

type Section struct {
	Heading     string
	HeadingPath string
	Anchor      string
	Body        string
}

Section is one heading section of an article. The intro before the first heading has an empty Heading and Anchor, and cites the page itself.

type Stop

type Stop struct {
	URL   string
	Title string
}

Stop is one entry on a guided topic page's reading route.

type Validators

type Validators struct {
	ETag         string
	LastModified string
}

Validators are a page's HTTP cache validators, sent back on the next fetch so an unchanged page costs no body.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL