Documentation
¶
Overview ¶
Package index stores the corpus and answers questions against it.
Retrieval sits behind a narrow contract with one implementation over SQLite FTS5. The contract is worth its hour: it lets a vector retriever be measured against the identical path later without either dirtying this one or building it twice, and it keeps the benchmark harness depending on an interface rather than on a database.
Index ¶
- Constants
- func BuildQuery(question string) string
- func ScopeHint(ctx context.Context) []string
- func WithScopeHint(ctx context.Context, prefixes ...string) context.Context
- type Document
- type FTS
- type FTSOption
- type Result
- type Retriever
- type Source
- type TargetOption
- type Targeted
- type Targets
- type Writer
Constants ¶
const ClauseUnlisted = "unlisted"
ClauseUnlisted is the clause recorded for a source the group listing no longer contains. It is not one of the predicate's clauses: nothing evaluated the source, it simply stopped being a candidate.
const DefaultCatalogueTTL = 15 * time.Minute
DefaultCatalogueTTL is how long the module names are trusted before they are re-read, so a newly indexed module is recognised without a restart.
const DefaultScopedSlots = 5
DefaultScopedSlots is how many of a retrieval's passages a named module fills. The rest come from the whole corpus, which is how a question answered only by another module's page still gets its answer (spec 0030 D7). The 5 is a starting point; the rerun recorded in the spec sets it.
const MaxQueryTerms = 32
MaxQueryTerms bounds a query built from a question.
Somebody pasting a stack trace or a whole config file into a channel should get a search, not a database pathology. The bound is generous — real questions are a dozen words — and terms past it are dropped rather than the question being refused, because a long question is still a question.
Variables ¶
This section is empty.
Functions ¶
func BuildQuery ¶
BuildQuery turns a natural question into an FTS5 expression.
A question is not a query. FTS5 MATCH takes its own expression language, and ordinary questions are full of characters that mean something in it: passing user text through unchanged does not merely risk odd results, it fails — "how do I hot-reload config?" errors with "no such column: reload", and an apostrophe is a syntax error. "col:value" reaches the column-filter syntax, which makes the query language addressable from a Discord message.
So nothing from the question survives as syntax. The text is broken into word tokens, each is quoted as a literal term, and the terms are joined with OR. Quoting is what makes an operator word like AND an ordinary search term rather than an instruction.
OR rather than AND because BM25 is doing the ranking: requiring every term would turn one unusual word in a question into no results at all, where scoring lets the documents matching more of the rare terms rise.
Types ¶
type Document ¶
type Document struct {
SourcePath string
RepoPath string
SiteURL string
Title string
Kind corpus.Kind
CommitSHA string
ContentSHA string
Chunks []corpus.Chunk
}
Document is one indexed file and its chunks.
type FTS ¶
type FTS struct {
// contains filtered or unexported fields
}
FTS retrieves with SQLite FTS5 and BM25.
type Result ¶
type Result struct {
Text string
Score float64
Heading string
HeadingPath string
// URL is the citation, section anchor included.
URL string
SourcePath string
RepoPath string
CommitSHA string
IndexedAt time.Time
}
Result is one retrieved passage with everything needed to cite it.
type Retriever ¶
type Retriever interface {
Retrieve(ctx context.Context, question string, limit int) ([]Result, error)
}
Retriever finds passages relevant to a question.
Query in, scored provenance-carrying results out. Deliberately narrow: it is the seam a vector retriever would arrive through, and the one the benchmark measures.
type TargetOption ¶ added in v0.12.0
type TargetOption func(*Targeted)
TargetOption configures a Targeted retriever.
func InCorpus ¶ added in v0.12.0
func InCorpus(prefixes ...string) TargetOption
InCorpus limits the retriever, and every name it recognises, to a server's corpus prefixes (spec 0029 D5).
func WithCatalogueTTL ¶ added in v0.12.0
func WithCatalogueTTL(d time.Duration) TargetOption
WithCatalogueTTL overrides DefaultCatalogueTTL.
func WithScopedSlots ¶ added in v0.12.0
func WithScopedSlots(n int) TargetOption
WithScopedSlots overrides DefaultScopedSlots.
type Targeted ¶ added in v0.12.0
type Targeted struct {
// contains filtered or unexported fields
}
Targeted is a Retriever that recognises the modules a question names and fills most of its slots from them (spec 0030 D7, D8).
func NewTargeted ¶ added in v0.12.0
func NewTargeted(db *store.DB, opts ...TargetOption) *Targeted
NewTargeted builds a targeting retriever over an open database.
type Targets ¶ added in v0.12.0
type Targets struct {
// Scope is the source prefixes the scoped slots are filled from, sorted.
// Empty means the question is retrieved exactly as an untargeted one.
Scope []string
// Excluded is the other-language twin of a module named in one language,
// kept out of every slot.
Excluded []string
}
Targets is what a question was recognised as being about.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer populates the index.
func (*Writer) PutDocument ¶
PutDocument indexes a document, replacing whatever was there before.
The whole document is written in one transaction, and its chunks are deleted before the new ones are inserted. Without the replacement a refresh would accumulate every previous version of a page and happily cite prose that has since been rewritten — a citation that resolves, to text that no longer says what the answer claimed.
func (*Writer) PutSource ¶
PutSource records a source and the verdict that admitted or rejected it.
Rejected sources are kept deliberately: an operator asking why something is not indexed should get the clause that decided it rather than silence.
An unresolved check never replaces a stored verdict: "could not check" is not an answer, and a throttled refresh must not unlist a documented source. The failure goes to last_error, where index status shows it (issue 5), and the next decided verdict clears it.
func (*Writer) RetireUnlisted ¶ added in v0.12.3
RetireUnlisted stops every source missing from listed qualifying, so a renamed, moved or deleted repository leaves the corpus. The listing must be complete; an empty one retires nothing.