Documentation
¶
Overview ¶
Package index stores the corpus and answers questions against it.
Retrieval sits behind a narrow contract with one implementation over SQLite FTS5. The contract is worth its hour: it lets a vector retriever be measured against the identical path later without either dirtying this one or building it twice, and it keeps the benchmark harness depending on an interface rather than on a database.
Index ¶
Constants ¶
const MaxQueryTerms = 32
MaxQueryTerms bounds a query built from a question.
Somebody pasting a stack trace or a whole config file into a channel should get a search, not a database pathology. The bound is generous — real questions are a dozen words — and terms past it are dropped rather than the question being refused, because a long question is still a question.
Variables ¶
This section is empty.
Functions ¶
func BuildQuery ¶
BuildQuery turns a natural question into an FTS5 expression.
A question is not a query. FTS5 MATCH takes its own expression language, and ordinary questions are full of characters that mean something in it: passing user text through unchanged does not merely risk odd results, it fails — "how do I hot-reload config?" errors with "no such column: reload", and an apostrophe is a syntax error. "col:value" reaches the column-filter syntax, which makes the query language addressable from a Discord message.
So nothing from the question survives as syntax. The text is broken into word tokens, each is quoted as a literal term, and the terms are joined with OR. Quoting is what makes an operator word like AND an ordinary search term rather than an instruction.
OR rather than AND because BM25 is doing the ranking: requiring every term would turn one unusual word in a question into no results at all, where scoring lets the documents matching more of the rare terms rise.
Types ¶
type Document ¶
type Document struct {
SourcePath string
RepoPath string
SiteURL string
Title string
Kind corpus.Kind
CommitSHA string
ContentSHA string
Chunks []corpus.Chunk
}
Document is one indexed file and its chunks.
type FTS ¶
type FTS struct {
// contains filtered or unexported fields
}
FTS retrieves with SQLite FTS5 and BM25.
type Result ¶
type Result struct {
Text string
Score float64
Heading string
HeadingPath string
// URL is the citation, section anchor included.
URL string
SourcePath string
RepoPath string
CommitSHA string
IndexedAt time.Time
}
Result is one retrieved passage with everything needed to cite it.
type Retriever ¶
type Retriever interface {
Retrieve(ctx context.Context, question string, limit int) ([]Result, error)
}
Retriever finds passages relevant to a question.
Query in, scored provenance-carrying results out. Deliberately narrow: it is the seam a vector retriever would arrive through, and the one the benchmark measures.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer populates the index.
func (*Writer) PutDocument ¶
PutDocument indexes a document, replacing whatever was there before.
The whole document is written in one transaction, and its chunks are deleted before the new ones are inserted. Without the replacement a refresh would accumulate every previous version of a page and happily cite prose that has since been rewritten — a citation that resolves, to text that no longer says what the answer claimed.
Source Files