Documentation
¶
Overview ¶
Package indexer builds and refreshes the knowledge index.
It joins the three pieces that have existed separately until now: discovery decides what may be indexed, a fetcher gets the content, and the store holds it. Everything it depends on is an interface, so the whole cycle runs in a test without a forge, a clone or a network.
Index ¶
- Constants
- Variables
- func Collectable(name string, size int64) bool
- type CloneFetcher
- type CloneOption
- type Discoverer
- type Fetcher
- type Indexer
- func (ix *Indexer) Build(ctx context.Context) (Report, error)
- func (ix *Indexer) Refresh(ctx context.Context) (Report, error)
- func (ix *Indexer) Sources(ctx context.Context) ([]corpus.Source, error)
- func (ix *Indexer) StaleCount(ctx context.Context, now time.Time, interval time.Duration) (int, error)
- func (ix *Indexer) Status(ctx context.Context) ([]SourceStatus, error)
- type Pruner
- type Report
- type SSH
- type Service
- func (s *Service) Failures() int64
- func (s *Service) Live() error
- func (s *Service) Ready() error
- func (s *Service) Refreshes() int64
- func (s *Service) Running() bool
- func (s *Service) StaleCount(ctx context.Context) (int, error)
- func (s *Service) Start(ctx context.Context) error
- func (s *Service) Stop(ctx context.Context)
- type ServiceOption
- type Snapshot
- type SourceStatus
Constants ¶
const DefaultInterval = time.Hour
DefaultInterval is how often the corpus is re-read when configuration says nothing.
Documentation changes on the timescale of a working day, not a minute, and every refresh fetches each source that may have moved. An hour keeps the index current enough to answer from without turning the bot into a polling load on the forge.
Variables ¶
var ErrNotStarted = errors.NewSentinel("indexer.not_started", "indexer: not started")
ErrNotStarted is reported by readiness before the service has run.
Functions ¶
func Collectable ¶
Collectable reports whether a file is worth reading into memory.
Filtering before the read rather than after is what keeps an indexing run proportional to the documentation rather than to the repository: a vendored blob or a generated dump costs the same to read as to index, and answers nothing either way.
Types ¶
type CloneFetcher ¶
type CloneFetcher struct {
// contains filtered or unexported fields
}
CloneFetcher keeps a shallow clone of each source on disk and brings it up to date by fast-forward, so an unchanged source costs one round trip rather than a clone (spec 0033 D1, D3).
The clones are a cache of public content: the corpus predicate runs before a source is fetched, and Prune removes what stops qualifying.
func NewCloneFetcher ¶
func NewCloneFetcher(dir, token string, opts ...CloneOption) *CloneFetcher
NewCloneFetcher builds a fetcher keeping its clones under dir. Without WithSSH it clones over HTTPS with token.
func (*CloneFetcher) Fetch ¶
func (f *CloneFetcher) Fetch(ctx context.Context, r forge.Repository) (*Snapshot, error)
Fetch brings a repository's clone up to date and reports its HEAD. The tree is read only when the snapshot is.
type CloneOption ¶ added in v0.14.0
type CloneOption func(*CloneFetcher)
CloneOption configures a CloneFetcher.
func WithSSH ¶ added in v0.14.0
func WithSSH(ssh SSH) CloneOption
WithSSH clones over SSH. There is no HTTPS fallback: a source whose SSH URL is missing or names another host fails.
type Discoverer ¶
Discoverer enumerates the corpus and reports a verdict for every candidate.
Declared here rather than taken as *corpus.Discoverer so the indexing cycle is testable against a fixed set of sources, without standing up three forge capabilities to exercise a loop that only consumes their result.
type Fetcher ¶
Fetcher gets a repository's indexable content.
An interface rather than a clone, so the indexing cycle is testable without a network — and so the clone implementation is one replaceable piece rather than something threaded through the whole package. Fetch is called from more than one goroutine at once, for different repositories.
type Indexer ¶
type Indexer struct {
// contains filtered or unexported fields
}
Indexer builds and refreshes the index.
func (*Indexer) Sources ¶
Sources reports every candidate and the verdict reached about it.
Rejections included, because a listing has to explain an absence: an operator asking why infra is not indexed should get the clause that excluded it rather than silence.
type Pruner ¶ added in v0.14.0
Pruner is a Fetcher that keeps something per source, and can discard what belongs to sources no longer qualifying (spec 0033 D4).
type Report ¶
type Report struct {
Sources int
Indexed int
Skipped int
Documents int
Chunks int
// Deferred counts sources a stopped run never reached. They keep their
// previous index. Distinct from Skipped, which means unchanged.
Deferred int
// Stopped says why the run stopped short, empty when it did not.
Stopped string
// Failed names the sources that could not be indexed, and why. A run
// continues past a failure, so this is how an operator learns one happened.
Failed map[string]string
}
Report summarises an indexing run.
type SSH ¶ added in v0.14.0
SSH is the key a fetcher clones with and the host keys it trusts (spec 0033 D2).
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service keeps the index current on a schedule.
It calls the same Refresh the CLI command calls, so the scheduled and manual routes cannot drift apart.
func NewService ¶
func NewService(ix *Indexer, opts ...ServiceOption) *Service
NewService builds a refresher over an indexer.
func (*Service) Live ¶
Live reports whether the process should keep running. It always may.
A refresh failing does not make the service unhealthy: the index it already holds is still answerable, and a restart would not reach the forge either. The loop has no other way to end — it runs until Stop closes it — so there is no unexpected exit for liveness to catch. Refresh health is reported through the counters and `index status`, which is where an operator can act on it.
func (*Service) Ready ¶
Ready reports whether the index is usable.
Staleness is deliberately not a readiness failure. It is the same judgement as a reconnect that lost events: an index that has not refreshed recently is still usable, answering from it with a caveat beats not answering, and taking the bot out of service would not fix a forge nobody can reach. Staleness is reported through `index status` and the refresh counters instead.
func (*Service) StaleCount ¶ added in v0.14.0
StaleCount counts the sources unconfirmed for too many of this service's intervals (spec 0033 D7).
func (*Service) Start ¶
Start begins refreshing.
It returns immediately and refreshes in the background, including the first one. Indexing the corpus takes minutes, and a controller waiting on Start would report the whole daemon as failing to come up while it was working correctly.
func (*Service) Stop ¶
Stop ends the refresh loop and waits for any refresh already running.
Waiting rather than returning immediately: a refresh is mid-write when shutdown arrives, and returning would let the process exit between writing a document and recording the commit it came from, leaving the index claiming to hold content it does not. The wait is bounded by the shutdown deadline in ctx, after which the refresh is cancelled and the transaction rolls back.
Safe unstarted and safe to call twice.
type ServiceOption ¶
type ServiceOption func(*Service)
ServiceOption configures a Service.
func WithInterval ¶
func WithInterval(d time.Duration) ServiceOption
WithInterval sets how often the index refreshes.
type Snapshot ¶
type Snapshot struct {
CommitSHA string
// Read returns path to content, already filtered to what the corpus
// indexes. Separate from the fetch so an unchanged source is never walked
// (spec 0033 D3).
Read func(ctx context.Context) (map[string][]byte, error)
}
Snapshot is a repository at one commit.
type SourceStatus ¶
type SourceStatus struct {
Path string
CommitSHA string
Documents int
Chunks int
LastIndexed time.Time
LastError string
// Refreshed is when the source was last confirmed current.
Refreshed time.Time
// Deferred reports that the last run stopped before reaching it.
Deferred bool
}
SourceStatus is what `index status` reports for one source.