Documentation
¶
Overview ¶
Package index stores the corpus and answers questions against it.
Retrieval sits behind a narrow contract with one implementation over SQLite FTS5. The contract is worth its hour: it lets a vector retriever be measured against the identical path later without either dirtying this one or building it twice, and it keeps the benchmark harness depending on an interface rather than on a database.
Index ¶
- Constants
- func BuildQuery(question string, ignore ...string) string
- func EncodeModules(modules map[string]string) (string, error)
- func ProjectOf(sourcePath string) string
- func ScopeHint(ctx context.Context) []string
- func WithScopeHint(ctx context.Context, prefixes ...string) context.Context
- func WordSet(words string) map[string]bool
- type Document
- type FTS
- type FTSOption
- type FeedStore
- type Page
- type PageInfo
- type PageState
- type Readings
- type Requires
- type Result
- type Retriever
- type Source
- type TargetOption
- type Targeted
- type Targets
- type Touch
- type Versions
- type Writer
- func (w *Writer) DropUnpublishedDocs(ctx context.Context) error
- func (w *Writer) Pages(ctx context.Context, source string) (map[string]PageState, error)
- func (w *Writer) PendingRemovals(ctx context.Context, source string) ([]string, error)
- func (w *Writer) PutDocument(ctx context.Context, d Document) error
- func (w *Writer) PutPage(ctx context.Context, p Page) error
- func (w *Writer) PutSite(ctx context.Context, origin string) error
- func (w *Writer) PutSource(ctx context.Context, s Source) error
- func (w *Writer) RecordSiteRun(ctx context.Context, origin string, runErr error) error
- func (w *Writer) RemovePage(ctx context.Context, source, path string) error
- func (w *Writer) RetainDocuments(ctx context.Context, sourcePath string, keep []string, preserve ...corpus.Kind) error
- func (w *Writer) RetireUnconfiguredSites(ctx context.Context, origins []string) error
- func (w *Writer) RetireUnlisted(ctx context.Context, listed []string) error
- func (w *Writer) SweepAbsent(ctx context.Context, source string, listed []string) ([]string, error)
- func (w *Writer) TouchPage(ctx context.Context, source, path string, t Touch) error
Constants ¶
const ( KindRepository = "repository" KindSite = "site" )
Source kinds. Every query about repositories alone filters on kind, so a site is never mistaken for one (spec 0008 D3).
const ( ClauseConfigured = "configured" ClauseUnconfigured = "unconfigured" )
Clauses recorded for a site. The repository predicate never evaluates one.
const AbsencesToRemove = 3
AbsencesToRemove is how many successful listings in a row must omit a page before it leaves the index (spec 0008 Q-3).
const ClauseUnlisted = "unlisted"
ClauseUnlisted is the clause recorded for a source the group listing no longer contains. It is not one of the predicate's clauses: nothing evaluated the source, it simply stopped being a candidate.
const DefaultCatalogueTTL = 15 * time.Minute
DefaultCatalogueTTL is how long the module names are trusted before they are re-read, so a newly indexed module is recognised without a restart.
const DefaultScopedSlots = 5
DefaultScopedSlots is how many of a retrieval's passages a named module fills. The rest come from the whole corpus, which is how a question answered only by another module's page still gets its answer (spec 0030 D7). The 5 is a starting point; the rerun recorded in the spec sets it.
const MaxPassagesPerDocument = 2
MaxPassagesPerDocument bounds how many passages one document contributes to a retrieval. A changelog's sections all match alike and would otherwise take every slot (issue 25).
const MaxPassagesPerFamilySource = 2
MaxPassagesPerFamilySource bounds how many of a family scope's slots one source takes, so 26 near-identical config adapters cannot fill them all (spec 0037 D3).
const MaxQueryTerms = 32
MaxQueryTerms bounds a query built from a question.
Somebody pasting a stack trace or a whole config file into a channel should get a search, not a database pathology. The bound is generous — real questions are a dozen words — and terms past it are dropped rather than the question being refused, because a long question is still a question.
Variables ¶
This section is empty.
Functions ¶
func BuildQuery ¶
BuildQuery turns a natural question into an FTS5 expression.
A question is not a query. FTS5 MATCH takes its own expression language, and ordinary questions are full of characters that mean something in it: passing user text through unchanged does not merely risk odd results, it fails — "how do I hot-reload config?" errors with "no such column: reload", and an apostrophe is a syntax error. "col:value" reaches the column-filter syntax, which makes the query language addressable from a Discord message.
So nothing from the question survives as syntax. The text is broken into word tokens, each is quoted as a literal term, and the terms are joined with OR. Quoting is what makes an operator word like AND an ordinary search term rather than an instruction.
OR rather than AND because BM25 is doing the ranking: requiring every term would turn one unusual word in a question into no results at all, where scoring lets the documents matching more of the rare terms rise.
Function words, and any word in ignore, are left out unless nothing else is left: they match the corpus's FAQ headings, which outweigh bodies, better than the subject matches its own page (issue 25).
func EncodeModules ¶ added in v0.15.0
EncodeModules is the stored form of a release's module requirements.
func ProjectOf ¶ added in v0.14.1
ProjectOf names a source the way people do, without the group: go/comms for phpboyscout/go/comms.
func WithScopeHint ¶ added in v0.12.0
WithScopeHint attaches the corpus prefixes a question's channel is about, such as a project's support forum (spec 0030 D8). A module named in the question takes precedence over it.
Types ¶
type Document ¶
type Document struct {
SourcePath string
RepoPath string
SiteURL string
Title string
Kind corpus.Kind
CommitSHA string
ContentSHA string
Chunks []corpus.Chunk
}
Document is one indexed file and its chunks.
type FTS ¶
type FTS struct {
// contains filtered or unexported fields
}
FTS retrieves with SQLite FTS5 and BM25.
type FTSOption ¶ added in v0.11.0
type FTSOption func(*FTS)
FTSOption configures an FTS retriever.
type FeedStore ¶ added in v0.15.0
type FeedStore struct {
// contains filtered or unexported fields
}
FeedStore keeps the last good estate feed in the database (spec 0034 D8).
func NewFeedStore ¶ added in v0.15.0
NewFeedStore builds a feed store over an open database.
type Page ¶ added in v0.15.0
type Page struct {
SourcePath string
Path string
URL string
Title string
Kind corpus.Kind
Lane string
Published time.Time
site.Validators
ContentSHA string
Chunks []corpus.Chunk
}
Page is one published page and its chunks. Its document's repo_path is the URL path, and its commit_sha the content hash, since a page has no commit (spec 0008 section 5, D2).
type PageInfo ¶ added in v0.15.0
type PageInfo struct {
Lane string `json:"lane,omitempty"`
Published time.Time `json:"published,omitzero"`
Route bool `json:"route,omitempty"`
}
PageInfo is what a published site's page adds to a passage: a post's lane and date (spec 0008 R-KB-14b, R-KB-14g), or that it is a guided topic page. It is zero for a repository.
type PageState ¶ added in v0.15.0
type PageState struct {
Path string
site.Validators
ContentSHA string
Lane string
Published time.Time
Absences int
}
PageState is what the index holds about a page between fetches.
type Readings ¶ added in v0.15.0
type Readings struct {
// Counts is how many passages each reading holds, and Best its best
// score. A reading is a module's path, a family member's being its core's.
Counts map[string]int
Best map[string]float64
// Unresolved maps an ordinary name the question used bare, which targeting
// left alone, to the modules in the corpus carrying it.
Unresolved map[string][]string
// Scope is what the question was targeted at, and TwinScope reports that
// it is one Go and Rust pair named with no language.
Scope []string
TwinScope bool
}
Readings is what clarification judges a question's passages by (spec 0031 D5).
type Requires ¶ added in v0.15.0
Requires is what a source's indexed release records: its tag, and the module versions its go.mod requires (spec 0036 D7).
type Result ¶
type Result struct {
Text string
Score float64
Heading string
HeadingPath string
// URL is the citation, section anchor included.
URL string
SourcePath string
RepoPath string
CommitSHA string
IndexedAt time.Time
PageInfo
}
Result is one retrieved passage with everything needed to cite it.
type Retriever ¶
type Retriever interface {
Retrieve(ctx context.Context, question string, limit int) ([]Result, error)
}
Retriever finds passages relevant to a question.
Query in, scored provenance-carrying results out. Deliberately narrow: it is the seam a vector retriever would arrive through, and the one the benchmark measures.
type Source ¶
type Source struct {
Path string
SiteURL string
RepoURL string
Verdict corpus.Verdict
// Family and Role are the estate feed's, empty for a source in no family.
Family string
Role string
}
Source is a repository the corpus predicate has considered.
type TargetOption ¶ added in v0.12.0
type TargetOption func(*Targeted)
TargetOption configures a Targeted retriever.
func InCorpus ¶ added in v0.12.0
func InCorpus(prefixes ...string) TargetOption
InCorpus limits the retriever, and every name it recognises, to a server's corpus prefixes (spec 0029 D5).
func WithCatalogueTTL ¶ added in v0.12.0
func WithCatalogueTTL(d time.Duration) TargetOption
WithCatalogueTTL overrides DefaultCatalogueTTL.
func WithScopedSlots ¶ added in v0.12.0
func WithScopedSlots(n int) TargetOption
WithScopedSlots overrides DefaultScopedSlots.
type Targeted ¶ added in v0.12.0
type Targeted struct {
// contains filtered or unexported fields
}
Targeted is a Retriever that recognises the modules a question names and fills most of its slots from them (spec 0030 D7, D8).
func NewTargeted ¶ added in v0.12.0
func NewTargeted(db *store.DB, opts ...TargetOption) *Targeted
NewTargeted builds a targeting retriever over an open database.
func (*Targeted) Readings ¶ added in v0.15.0
func (t *Targeted) Readings(ctx context.Context, question string, results []Result) (Readings, error)
Readings groups results by reading and reports the names a question left unresolved.
type Targets ¶ added in v0.12.0
type Targets struct {
// Scope is the source prefixes the scoped slots are filled from, sorted.
// Empty means the question is retrieved exactly as an untargeted one.
Scope []string
// Excluded is the other-language twin of a module named in one language,
// kept out of every slot.
Excluded []string
// Family names the family when Scope is the whole of it (spec 0037 D1).
Family string
}
Targets is what a question was recognised as being about.
type Touch ¶ added in v0.15.0
type Touch struct {
site.Validators
Lane string
Published time.Time
}
Touch is what an unchanged page may still have changed: the validators it was served with, and the lane and date its feed now gives it.
type Versions ¶ added in v0.15.0
type Versions struct {
// contains filtered or unexported fields
}
Versions reads what the index recorded about each source's release.
func NewVersions ¶ added in v0.15.0
NewVersions builds a reader over an open database.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer populates the index.
func (*Writer) DropUnpublishedDocs ¶ added in v0.15.0
DropUnpublishedDocs removes the doc documents of every qualifying source with no site, which have no page to cite (spec 0034 D5, D11). It reads the verdicts already written this run; an unresolved source keeps its previous site, and with it its docs. The indexer's contributes applies the same rule to what it writes.
func (*Writer) Pages ¶ added in v0.15.0
Pages returns the state of every page indexed for source, by path.
func (*Writer) PendingRemovals ¶ added in v0.15.0
PendingRemovals names source's pages that recent listings have omitted, so content leaving the index is visible before it goes (spec 0008 Q-3).
func (*Writer) PutDocument ¶
PutDocument indexes a document, replacing whatever was there before.
The whole document is written in one transaction, and its chunks are deleted before the new ones are inserted. Without the replacement a refresh would accumulate every previous version of a page and happily cite prose that has since been rewritten — a citation that resolves, to text that no longer says what the answer claimed.
func (*Writer) PutPage ¶ added in v0.15.0
PutPage indexes a page, replacing its previous version, and resets its absences.
func (*Writer) PutSource ¶
PutSource records a source and the verdict that admitted or rejected it.
Rejected sources are kept deliberately: an operator asking why something is not indexed should get the clause that decided it rather than silence.
An unresolved check never replaces a stored verdict: "could not check" is not an answer, and a throttled refresh must not unlist a documented source. The failure goes to last_error, where index status shows it (issue 5), and the next decided verdict clears it.
func (*Writer) RecordSiteRun ¶ added in v0.15.0
RecordSiteRun marks a site refreshed, recording the run's error, or clearing it when there was none.
func (*Writer) RemovePage ¶ added in v0.15.0
RemovePage deletes a page and its document now, for a page whose own content rules it out rather than its absence from a listing.
func (*Writer) RetainDocuments ¶ added in v0.15.0
func (w *Writer) RetainDocuments(ctx context.Context, sourcePath string, keep []string, preserve ...corpus.Kind) error
RetainDocuments removes a source's documents whose path is not in keep, leaving documents of the preserved kinds alone. Only a complete snapshot may name keep: a partial one would remove documents that still exist. An empty keep removes them all.
func (*Writer) RetireUnconfiguredSites ¶ added in v0.15.0
RetireUnconfiguredSites stops every site not in origins qualifying, so a site removed from configuration stops being cited. Its pages stay until it is configured again, as a retired repository's do.
func (*Writer) RetireUnlisted ¶ added in v0.12.3
RetireUnlisted stops every source missing from listed qualifying, so a renamed, moved or deleted repository leaves the corpus. The listing must be complete; an empty one retires nothing.
func (*Writer) SweepAbsent ¶ added in v0.15.0
SweepAbsent applies one complete listing: every page in listed has its absences reset, every other page of source gains one, and a page reaching AbsencesToRemove is deleted with its document. Only a listing read in full may be passed here; a partial one would count present pages as absent.