Documentation
¶
Overview ¶
Package indexer builds and refreshes the knowledge index.
It joins the three pieces that have existed separately until now: discovery decides what may be indexed, a fetcher gets the content, and the store holds it. Everything it depends on is an interface, so the whole cycle runs in a test without a forge, a clone or a network.
Index ¶
Constants ¶
const DefaultInterval = time.Hour
DefaultInterval is how often the corpus is re-read when configuration says nothing.
Documentation changes on the timescale of a working day, not a minute, and every refresh clones the repositories that moved. An hour keeps the index current enough to answer from without turning the bot into a polling load on the forge.
Variables ¶
var ErrNotStarted = errors.New("indexer: not started")
ErrNotStarted is reported by readiness before the service has run.
Functions ¶
func Collectable ¶
Collectable reports whether a file is worth reading into memory.
Filtering before the read rather than after is what keeps an indexing run proportional to the documentation rather than to the repository: a vendored blob or a generated dump costs the same to read as to index, and answers nothing either way.
Types ¶
type CloneFetcher ¶
type CloneFetcher struct {
// contains filtered or unexported fields
}
CloneFetcher gets repository content by shallow-cloning in memory.
In memory rather than to disk deliberately. The corpus predicate is a security boundary, and content that never touches the filesystem cannot be left behind by a crash between the clone and the check that should have rejected it.
func NewCloneFetcher ¶
func NewCloneFetcher(token string) *CloneFetcher
NewCloneFetcher builds a fetcher over go/repo.
func (*CloneFetcher) Fetch ¶
func (f *CloneFetcher) Fetch(ctx context.Context, r forge.Repository) (*Snapshot, error)
Fetch clones a repository and returns the files the corpus indexes.
Filtering happens here rather than after: a file the corpus will not index is never read into memory, so the size of an indexing run tracks the documentation rather than the repository.
type Discoverer ¶
Discoverer enumerates the corpus and reports a verdict for every candidate.
Declared here rather than taken as *corpus.Discoverer so the indexing cycle is testable against a fixed set of sources, without standing up three forge capabilities to exercise a loop that only consumes their result.
type Fetcher ¶
Fetcher gets a repository's indexable content.
An interface rather than a clone, so the indexing cycle is testable without a network — and so the shallow-clone implementation is one replaceable piece rather than something threaded through the whole package.
type Indexer ¶
type Indexer struct {
// contains filtered or unexported fields
}
Indexer builds and refreshes the index.
type Report ¶
type Report struct {
Sources int
Indexed int
Skipped int
Documents int
Chunks int
// Failed names the sources that could not be indexed, and why. A run
// continues past a failure, so this is how an operator learns one happened.
Failed map[string]string
}
Report summarises an indexing run.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service keeps the index current on a schedule.
It calls the same Refresh the CLI command calls, so the scheduled and manual routes cannot drift apart.
func NewService ¶
func NewService(ix *Indexer, opts ...ServiceOption) *Service
NewService builds a refresher over an indexer.
func (*Service) Live ¶
Live reports whether the process should keep running. It always may.
A refresh failing does not make the service unhealthy: the index it already holds is still answerable, and a restart would not reach the forge either. The loop has no other way to end — it runs until Stop closes it — so there is no unexpected exit for liveness to catch. Refresh health is reported through the counters and `index status`, which is where an operator can act on it.
func (*Service) Ready ¶
Ready reports whether the index is usable.
Staleness is deliberately not a readiness failure. It is the same judgement as a reconnect that lost events: an index that has not refreshed recently is still usable, answering from it with a caveat beats not answering, and taking the bot out of service would not fix a forge nobody can reach. Staleness is reported through `index status` and the refresh counters instead.
func (*Service) Start ¶
Start begins refreshing.
It returns immediately and refreshes in the background, including the first one. Indexing the corpus takes minutes, and a controller waiting on Start would report the whole daemon as failing to come up while it was working correctly.
func (*Service) Stop ¶
Stop ends the refresh loop and waits for any refresh already running.
Waiting rather than returning immediately: a refresh is mid-write when shutdown arrives, and returning would let the process exit between writing a document and recording the commit it came from, leaving the index claiming to hold content it does not. The wait is bounded by the shutdown deadline in ctx, after which the refresh is cancelled and the transaction rolls back.
Safe unstarted and safe to call twice.
type ServiceOption ¶
type ServiceOption func(*Service)
ServiceOption configures a Service.
func WithInterval ¶
func WithInterval(d time.Duration) ServiceOption
WithInterval sets how often the index refreshes.
Source Files