indexer

package
v0.15.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package indexer builds and refreshes the knowledge index.

It joins the three pieces that have existed separately until now: discovery decides what may be indexed, a fetcher gets the content, and the store holds it. Everything it depends on is an interface, so the whole cycle runs in a test without a forge, a clone or a network.

Index

Constants

View Source
const DefaultInterval = time.Hour

DefaultInterval is how often the corpus is re-read when configuration says nothing.

Documentation changes on the timescale of a working day, not a minute, and every refresh fetches each source that may have moved. An hour keeps the index current enough to answer from without turning the bot into a polling load on the forge.

Variables

View Source
var ErrNotStarted = errors.NewSentinel("indexer.not_started", "indexer: not started")

ErrNotStarted is reported by readiness before the service has run.

Functions

func Collectable

func Collectable(name string, size int64) bool

Collectable reports whether a file is worth reading into memory.

Filtering before the read rather than after is what keeps an indexing run proportional to the documentation rather than to the repository: a vendored blob or a generated dump costs the same to read as to index, and answers nothing either way.

Types

type CloneFetcher

type CloneFetcher struct {
	// contains filtered or unexported fields
}

CloneFetcher keeps a shallow clone of each source on disk and brings it up to date by fast-forward, so an unchanged source costs one round trip rather than a clone (spec 0033 D1, D3).

The clones are a cache of public content: the corpus predicate runs before a source is fetched, and Prune removes what stops qualifying.

func NewCloneFetcher

func NewCloneFetcher(dir, token string, opts ...CloneOption) *CloneFetcher

NewCloneFetcher builds a fetcher keeping its clones under dir. Without WithSSH it clones over HTTPS with token.

func (*CloneFetcher) Fetch

func (f *CloneFetcher) Fetch(ctx context.Context, r forge.Repository) (*Snapshot, error)

Fetch brings a repository's clone up to date and reports its HEAD. The tree is read only when the snapshot is.

func (*CloneFetcher) Prune added in v0.14.0

func (f *CloneFetcher) Prune(ctx context.Context, keep []string) error

Prune removes every clone whose source is not in keep (spec 0033 D4).

type CloneOption added in v0.14.0

type CloneOption func(*CloneFetcher)

CloneOption configures a CloneFetcher.

func WithForge added in v0.15.0

func WithForge(forgeURL string) CloneOption

WithForge names the forge a clone URL must be on before a credential is sent with it. Without it, an SSH fetcher and an HTTPS fetcher with a token refuse every URL.

func WithReleases added in v0.15.0

func WithReleases(r Releases) CloneOption

WithReleases reads code from each source's latest release. Without it, code comes from the default branch.

func WithSSH added in v0.14.0

func WithSSH(ssh SSH) CloneOption

WithSSH clones over SSH. There is no HTTPS fallback: a source whose SSH URL is missing or names another host fails.

type Discoverer

type Discoverer interface {
	Discover(ctx context.Context) ([]corpus.Source, error)
}

Discoverer enumerates the corpus and reports a verdict for every candidate.

Declared here rather than taken as *corpus.Discoverer so the indexing cycle is testable against a fixed set of sources, without standing up three forge capabilities to exercise a loop that only consumes their result.

type Fetcher

type Fetcher interface {
	Fetch(ctx context.Context, r forge.Repository) (*Snapshot, error)
}

Fetcher gets a repository's indexable content.

An interface rather than a clone, so the indexing cycle is testable without a network — and so the clone implementation is one replaceable piece rather than something threaded through the whole package. Fetch is called from more than one goroutine at once, for different repositories.

type Indexer

type Indexer struct {
	// contains filtered or unexported fields
}

Indexer builds and refreshes the index.

func New

func New(d Discoverer, f Fetcher, w *index.Writer, db *store.DB) *Indexer

New builds an indexer.

func (*Indexer) AddSites added in v0.15.0

func (ix *Indexer) AddSites(sites ...SiteRefresher)

AddSites adds published sites to every run.

func (*Indexer) Build

func (ix *Indexer) Build(ctx context.Context) (Report, error)

Build indexes every qualifying source from scratch.

func (*Indexer) Refresh

func (ix *Indexer) Refresh(ctx context.Context) (Report, error)

Refresh re-indexes only the sources whose commit has moved.

func (*Indexer) SiteOrigins added in v0.15.0

func (ix *Indexer) SiteOrigins() []string

SiteOrigins names the published sites every run refreshes.

func (*Indexer) Sources

func (ix *Indexer) Sources(ctx context.Context) ([]corpus.Source, error)

Sources reports every candidate and the verdict reached about it.

Rejections included, because a listing has to explain an absence: an operator asking why infra is not indexed should get the clause that excluded it rather than silence.

func (*Indexer) StaleCount added in v0.14.0

func (ix *Indexer) StaleCount(ctx context.Context, now time.Time, interval time.Duration) (int, error)

StaleCount counts the qualifying sources Stale would report (spec 0033 D7).

func (*Indexer) Status

func (ix *Indexer) Status(ctx context.Context) ([]SourceStatus, error)

Status reports what is indexed, per source.

type Pruner added in v0.14.0

type Pruner interface {
	Prune(ctx context.Context, keep []string) error
}

Pruner is a Fetcher that keeps something per source, and can discard what belongs to sources no longer qualifying (spec 0033 D4).

type Releases added in v0.15.0

type Releases interface {
	LatestRelease(ctx context.Context, path string) (string, error)
}

Releases names a source's latest release, returning forge.ErrReleaseNotFound when it has none (spec 0035 D2).

type Report

type Report struct {
	Sources   int
	Indexed   int
	Skipped   int
	Documents int
	Chunks    int

	// Deferred counts sources a stopped run never reached. They keep their
	// previous index. Distinct from Skipped, which means unchanged.
	Deferred int

	// Stopped says why the run stopped short, empty when it did not.
	Stopped string

	// Failed names the sources that could not be indexed, and why. A run
	// continues past a failure, so this is how an operator learns one happened.
	Failed map[string]string

	// Sites holds each published site's own report.
	Sites []siteindex.Report
}

Report summarises an indexing run.

type SSH added in v0.14.0

type SSH struct {
	Key        string
	KnownHosts string
}

SSH is the key a fetcher clones with and the host keys it trusts (spec 0033 D2).

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service keeps the index current on a schedule.

It calls the same Refresh the CLI command calls, so the scheduled and manual routes cannot drift apart.

func NewService

func NewService(ix *Indexer, opts ...ServiceOption) *Service

NewService builds a refresher over an indexer.

func (*Service) Failures

func (s *Service) Failures() int64

Failures counts cycles that could not complete.

func (*Service) Live

func (s *Service) Live() error

Live reports whether the process should keep running. It always may.

A refresh failing does not make the service unhealthy: the index it already holds is still answerable, and a restart would not reach the forge either. The loop has no other way to end — it runs until Stop closes it — so there is no unexpected exit for liveness to catch. Refresh health is reported through the counters and `index status`, which is where an operator can act on it.

func (*Service) Ready

func (s *Service) Ready() error

Ready reports whether the index is usable.

Staleness is deliberately not a readiness failure. It is the same judgement as a reconnect that lost events: an index that has not refreshed recently is still usable, answering from it with a caveat beats not answering, and taking the bot out of service would not fix a forge nobody can reach. Staleness is reported through `index status` and the refresh counters instead.

func (*Service) Refreshes

func (s *Service) Refreshes() int64

Refreshes counts successful cycles.

func (*Service) Running

func (s *Service) Running() bool

Running reports whether a refresh is in progress.

func (*Service) StaleCount added in v0.14.0

func (s *Service) StaleCount(ctx context.Context) (int, error)

StaleCount counts the sources unconfirmed for too many of this service's intervals (spec 0033 D7).

func (*Service) Start

func (s *Service) Start(ctx context.Context) error

Start begins refreshing.

It returns immediately and refreshes in the background, including the first one. Indexing the corpus takes minutes, and a controller waiting on Start would report the whole daemon as failing to come up while it was working correctly.

func (*Service) Stop

func (s *Service) Stop(ctx context.Context)

Stop ends the refresh loop and waits for any refresh already running.

Waiting rather than returning immediately: a refresh is mid-write when shutdown arrives, and returning would let the process exit between writing a document and recording the commit it came from, leaving the index claiming to hold content it does not. The wait is bounded by the shutdown deadline in ctx, after which the refresh is cancelled and the transaction rolls back.

Safe unstarted and safe to call twice.

type ServiceOption

type ServiceOption func(*Service)

ServiceOption configures a Service.

func WithInterval

func WithInterval(d time.Duration) ServiceOption

WithInterval sets how often the index refreshes.

func WithLogger

func WithLogger(l *slog.Logger) ServiceOption

WithLogger sets the logger.

type SiteRefresher added in v0.15.0

type SiteRefresher interface {
	Origin() string
	Refresh(ctx context.Context, full bool) (siteindex.Report, error)
}

SiteRefresher keeps one published site current. Every run refreshes each site after the repositories, so the scheduled and manual routes cover sites alike (spec 0008 D1).

type Snapshot

type Snapshot struct {
	// CommitSHA is the default branch head, which docs, README and changelog
	// come from.
	CommitSHA string

	// CodeCommitSHA is the latest release's commit, or the branch head when the
	// source has no release. CodeRef is that release's tag, empty for the branch.
	CodeCommitSHA string
	CodeRef       string

	// CodeErr is a release lookup that failed for a reason other than "no
	// release". Read then returns no code, and the code already indexed stays
	// at the last release (spec 0035 D9).
	CodeErr error

	// Read returns path to content, already filtered to what the corpus
	// indexes, plus the code commit's go.mod (spec 0036 D7). Separate from the
	// fetch so an unchanged source is never walked (spec 0033 D3).
	Read func(ctx context.Context) (map[string][]byte, error)
	// contains filtered or unexported fields
}

Snapshot is a repository's default branch, and the commit its code is read from (spec 0035).

type SourceStatus

type SourceStatus struct {
	Path      string
	CommitSHA string

	// CodeCommitSHA and CodeRef are the release code was indexed from; CodeRef
	// is empty when code comes from the branch (spec 0035 D6).
	CodeCommitSHA string
	CodeRef       string

	Documents   int
	Chunks      int
	LastIndexed time.Time
	LastError   string

	// Refreshed is when the source was last confirmed current.
	Refreshed time.Time

	// Deferred reports that the last run stopped before reaching it.
	Deferred bool

	// Leaving names a site's pages that recent listings have omitted, which
	// leave the index unless a listing names them again (spec 0008 Q-3).
	Leaving []string
}

SourceStatus is what `index status` reports for one source.

func (SourceStatus) Stale added in v0.14.0

func (s SourceStatus) Stale(now time.Time, interval time.Duration) bool

Stale reports whether the source has gone unconfirmed for longer than spec 0033 D7 allows at this refresh interval.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL