siteindex

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package siteindex keeps a published site's posts and guides in the index (spec 0008). It joins pkg/site, which fetches with no credential, to pkg/index, which is why it is a package of its own: the index reaches forge, and the fetcher must not.

Index

Constants

View Source
const DefaultMaxPages = 500

DefaultMaxPages bounds how many pages one run fetches (R-KB-15). The site declared 241 on 2026-10-30.

Variables

This section is empty.

Functions

This section is empty.

Types

type Indexer

type Indexer struct {
	// contains filtered or unexported fields
}

Indexer refreshes one site.

func New

func New(f *site.Fetcher, w *index.Writer, opts ...Option) *Indexer

New builds an indexer for the site f is locked to.

func (*Indexer) Origin

func (ix *Indexer) Origin() string

Origin is the site this indexer refreshes.

func (*Indexer) Refresh

func (ix *Indexer) Refresh(ctx context.Context, full bool) (Report, error)

Refresh brings the site's pages up to date. With full set it ignores cache validators, so every page is fetched, though unchanged text is still not re-chunked.

A site that cannot be reached, or a page that fails, is recorded on the site and in the report, and is never an error: the bot answers from what it holds (spec 0008 section 6). Only a failure to write the index is returned.

type Option

type Option func(*Indexer)

Option configures an Indexer.

func WithCutoff

func WithCutoff(t time.Time) Option

WithCutoff sets the publication date before which a post is not indexed.

func WithLanes

func WithLanes(lanes ...string) Option

WithLanes sets the lanes whose posts are indexed.

func WithMaxPages

func WithMaxPages(n int) Option

WithMaxPages bounds how many pages one run fetches.

type Report

type Report struct {
	Origin string

	// Posts and Routes count what was indexed or confirmed this run.
	Posts  int
	Routes int

	Indexed   int
	Unchanged int

	// Missing counts listed pages the site answered 404 for. They keep their
	// last good copy until the listing drops them.
	Missing int

	// Failed maps a page URL to why it could not be indexed.
	Failed map[string]string

	// Removed names the pages this run took out of the index, and Pending
	// those that recent listings have omitted (spec 0008 Q-3).
	Removed []string
	Pending []string

	// Problem says why the run was incomplete, empty when it was not.
	Problem string
}

Report is what one run of a site did.

func (Report) Complete

func (r Report) Complete() bool

Complete reports whether every feed was read and nothing was cut short, so the run was allowed to count absences.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL