index

package
v0.15.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 4, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package index stores the corpus and answers questions against it.

Retrieval sits behind a narrow contract with one implementation over SQLite FTS5. The contract is worth its hour: it lets a vector retriever be measured against the identical path later without either dirtying this one or building it twice, and it keeps the benchmark harness depending on an interface rather than on a database.

Index

Constants

View Source
const (
	KindRepository = "repository"
	KindSite       = "site"
)

Source kinds. Every query about repositories alone filters on kind, so a site is never mistaken for one (spec 0008 D3).

View Source
const (
	ClauseConfigured   = "configured"
	ClauseUnconfigured = "unconfigured"
)

Clauses recorded for a site. The repository predicate never evaluates one.

View Source
const AbsencesToRemove = 3

AbsencesToRemove is how many successful listings in a row must omit a page before it leaves the index (spec 0008 Q-3).

View Source
const ClauseUnlisted = "unlisted"

ClauseUnlisted is the clause recorded for a source the group listing no longer contains. It is not one of the predicate's clauses: nothing evaluated the source, it simply stopped being a candidate.

View Source
const DefaultCatalogueTTL = 15 * time.Minute

DefaultCatalogueTTL is how long the module names are trusted before they are re-read, so a newly indexed module is recognised without a restart.

View Source
const DefaultScopedSlots = 5

DefaultScopedSlots is how many of a retrieval's passages a named module fills. The rest come from the whole corpus, which is how a question answered only by another module's page still gets its answer (spec 0030 D7). The 5 is a starting point; the rerun recorded in the spec sets it.

View Source
const MaxPassagesPerDocument = 2

MaxPassagesPerDocument bounds how many passages one document contributes to a retrieval. A changelog's sections all match alike and would otherwise take every slot (issue 25).

View Source
const MaxPassagesPerFamilySource = 2

MaxPassagesPerFamilySource bounds how many of a family scope's slots one source takes, so 26 near-identical config adapters cannot fill them all (spec 0037 D3).

View Source
const MaxQueryTerms = 32

MaxQueryTerms bounds a query built from a question.

Somebody pasting a stack trace or a whole config file into a channel should get a search, not a database pathology. The bound is generous — real questions are a dozen words — and terms past it are dropped rather than the question being refused, because a long question is still a question.

Variables

This section is empty.

Functions

func BuildQuery

func BuildQuery(question string, ignore ...string) string

BuildQuery turns a natural question into an FTS5 expression.

A question is not a query. FTS5 MATCH takes its own expression language, and ordinary questions are full of characters that mean something in it: passing user text through unchanged does not merely risk odd results, it fails — "how do I hot-reload config?" errors with "no such column: reload", and an apostrophe is a syntax error. "col:value" reaches the column-filter syntax, which makes the query language addressable from a Discord message.

So nothing from the question survives as syntax. The text is broken into word tokens, each is quoted as a literal term, and the terms are joined with OR. Quoting is what makes an operator word like AND an ordinary search term rather than an instruction.

OR rather than AND because BM25 is doing the ranking: requiring every term would turn one unusual word in a question into no results at all, where scoring lets the documents matching more of the rare terms rise.

Function words, and any word in ignore, are left out unless nothing else is left: they match the corpus's FAQ headings, which outweigh bodies, better than the subject matches its own page (issue 25).

func EncodeModules added in v0.15.0

func EncodeModules(modules map[string]string) (string, error)

EncodeModules is the stored form of a release's module requirements.

func ProjectOf added in v0.14.1

func ProjectOf(sourcePath string) string

ProjectOf names a source the way people do, without the group: go/comms for phpboyscout/go/comms.

func ScopeHint added in v0.12.0

func ScopeHint(ctx context.Context) []string

ScopeHint returns the prefixes WithScopeHint attached, if any.

func WithScopeHint added in v0.12.0

func WithScopeHint(ctx context.Context, prefixes ...string) context.Context

WithScopeHint attaches the corpus prefixes a question's channel is about, such as a project's support forum (spec 0030 D8). A module named in the question takes precedence over it.

func WordSet added in v0.13.4

func WordSet(words string) map[string]bool

WordSet is the set of whitespace-separated words in words.

Types

type Document

type Document struct {
	SourcePath string
	RepoPath   string
	SiteURL    string
	Title      string
	Kind       corpus.Kind
	CommitSHA  string
	ContentSHA string
	Chunks     []corpus.Chunk
}

Document is one indexed file and its chunks.

type FTS

type FTS struct {
	// contains filtered or unexported fields
}

FTS retrieves with SQLite FTS5 and BM25.

func NewFTS

func NewFTS(db *store.DB, opts ...FTSOption) *FTS

NewFTS builds a retriever over an open database.

func (*FTS) Retrieve

func (f *FTS) Retrieve(ctx context.Context, question string, limit int) ([]Result, error)

Retrieve scores the corpus against a question.

The question is rebuilt into an FTS5 expression rather than passed through: see BuildQuery. A question with nothing searchable in it returns no results rather than a query that matches everything.

type FTSOption added in v0.11.0

type FTSOption func(*FTS)

FTSOption configures an FTS retriever.

func Ignoring added in v0.13.4

func Ignoring(words ...string) FTSOption

Ignoring leaves these words out of the query unless nothing else is left.

func Within added in v0.11.0

func Within(prefixes ...string) FTSOption

Within restricts retrieval to the sources under these GitLab path prefixes, a server's slice of the corpus (spec 0029 D5). A prefix covers the source of that path and every source below it.

type FeedStore added in v0.15.0

type FeedStore struct {
	// contains filtered or unexported fields
}

FeedStore keeps the last good estate feed in the database (spec 0034 D8).

func NewFeedStore added in v0.15.0

func NewFeedStore(db *store.DB) *FeedStore

NewFeedStore builds a feed store over an open database.

func (*FeedStore) LoadFeed added in v0.15.0

func (s *FeedStore) LoadFeed(ctx context.Context) (feed.Stored, bool, error)

LoadFeed returns the stored copy, reporting false when there is none.

func (*FeedStore) SaveFeed added in v0.15.0

func (s *FeedStore) SaveFeed(ctx context.Context, f feed.Stored) error

SaveFeed replaces the stored copy.

type Page added in v0.15.0

type Page struct {
	SourcePath string
	Path       string
	URL        string
	Title      string
	Kind       corpus.Kind
	Lane       string
	Published  time.Time
	site.Validators
	ContentSHA string
	Chunks     []corpus.Chunk
}

Page is one published page and its chunks. Its document's repo_path is the URL path, and its commit_sha the content hash, since a page has no commit (spec 0008 section 5, D2).

type PageInfo added in v0.15.0

type PageInfo struct {
	Lane      string    `json:"lane,omitempty"`
	Published time.Time `json:"published,omitzero"`
	Route     bool      `json:"route,omitempty"`
}

PageInfo is what a published site's page adds to a passage: a post's lane and date (spec 0008 R-KB-14b, R-KB-14g), or that it is a guided topic page. It is zero for a repository.

func (PageInfo) Note added in v0.15.0

func (p PageInfo) Note() string

Note is what the page adds to its citation's label: a post's lane and date, or that it is a guide (spec 0008 Q-6). A repository passage adds nothing.

type PageState added in v0.15.0

type PageState struct {
	Path string
	site.Validators
	ContentSHA string
	Lane       string
	Published  time.Time
	Absences   int
}

PageState is what the index holds about a page between fetches.

type Readings added in v0.15.0

type Readings struct {
	// Counts is how many passages each reading holds, and Best its best
	// score. A reading is a module's path, a family member's being its core's.
	Counts map[string]int
	Best   map[string]float64

	// Unresolved maps an ordinary name the question used bare, which targeting
	// left alone, to the modules in the corpus carrying it.
	Unresolved map[string][]string

	// Scope is what the question was targeted at, and TwinScope reports that
	// it is one Go and Rust pair named with no language.
	Scope     []string
	TwinScope bool
}

Readings is what clarification judges a question's passages by (spec 0031 D5).

type Requires added in v0.15.0

type Requires struct {
	Tag     string
	Modules map[string]string
}

Requires is what a source's indexed release records: its tag, and the module versions its go.mod requires (spec 0036 D7).

type Result

type Result struct {
	Text  string
	Score float64

	Heading     string
	HeadingPath string

	// URL is the citation, section anchor included.
	URL string

	SourcePath string
	RepoPath   string
	CommitSHA  string
	IndexedAt  time.Time

	PageInfo
}

Result is one retrieved passage with everything needed to cite it.

type Retriever

type Retriever interface {
	Retrieve(ctx context.Context, question string, limit int) ([]Result, error)
}

Retriever finds passages relevant to a question.

Query in, scored provenance-carrying results out. Deliberately narrow: it is the seam a vector retriever would arrive through, and the one the benchmark measures.

type Source

type Source struct {
	Path    string
	SiteURL string
	RepoURL string
	Verdict corpus.Verdict

	// Family and Role are the estate feed's, empty for a source in no family.
	Family string
	Role   string
}

Source is a repository the corpus predicate has considered.

type TargetOption added in v0.12.0

type TargetOption func(*Targeted)

TargetOption configures a Targeted retriever.

func InCorpus added in v0.12.0

func InCorpus(prefixes ...string) TargetOption

InCorpus limits the retriever, and every name it recognises, to a server's corpus prefixes (spec 0029 D5).

func WithCatalogueTTL added in v0.12.0

func WithCatalogueTTL(d time.Duration) TargetOption

WithCatalogueTTL overrides DefaultCatalogueTTL.

func WithScopedSlots added in v0.12.0

func WithScopedSlots(n int) TargetOption

WithScopedSlots overrides DefaultScopedSlots.

type Targeted added in v0.12.0

type Targeted struct {
	// contains filtered or unexported fields
}

Targeted is a Retriever that recognises the modules a question names and fills most of its slots from them (spec 0030 D7, D8).

func NewTargeted added in v0.12.0

func NewTargeted(db *store.DB, opts ...TargetOption) *Targeted

NewTargeted builds a targeting retriever over an open database.

func (*Targeted) Readings added in v0.15.0

func (t *Targeted) Readings(ctx context.Context, question string, results []Result) (Readings, error)

Readings groups results by reading and reports the names a question left unresolved.

func (*Targeted) Retrieve added in v0.12.0

func (t *Targeted) Retrieve(ctx context.Context, question string, limit int) ([]Result, error)

Retrieve fills up to the scoped slots from the question's targets, then the rest from the server's whole corpus, never repeating a passage and never returning an excluded twin.

func (*Targeted) Targets added in v0.12.0

func (t *Targeted) Targets(ctx context.Context, question string) (Targets, error)

Targets reports what a question is about: the modules it names, or failing that its channel's scope, cut to the server's corpus.

type Targets added in v0.12.0

type Targets struct {
	// Scope is the source prefixes the scoped slots are filled from, sorted.
	// Empty means the question is retrieved exactly as an untargeted one.
	Scope []string
	// Excluded is the other-language twin of a module named in one language,
	// kept out of every slot.
	Excluded []string
	// Family names the family when Scope is the whole of it (spec 0037 D1).
	Family string
}

Targets is what a question was recognised as being about.

type Touch added in v0.15.0

type Touch struct {
	site.Validators
	Lane      string
	Published time.Time
}

Touch is what an unchanged page may still have changed: the validators it was served with, and the lane and date its feed now gives it.

type Versions added in v0.15.0

type Versions struct {
	// contains filtered or unexported fields
}

Versions reads what the index recorded about each source's release.

func NewVersions added in v0.15.0

func NewVersions(db *store.DB) *Versions

NewVersions builds a reader over an open database.

func (*Versions) Requires added in v0.15.0

func (v *Versions) Requires(ctx context.Context, source string) (Requires, bool)

Requires reports what source's release recorded, and false when the index holds nothing for it.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer populates the index.

func NewWriter

func NewWriter(db *store.DB) *Writer

NewWriter builds a writer over an open database.

func (*Writer) DropUnpublishedDocs added in v0.15.0

func (w *Writer) DropUnpublishedDocs(ctx context.Context) error

DropUnpublishedDocs removes the doc documents of every qualifying source with no site, which have no page to cite (spec 0034 D5, D11). It reads the verdicts already written this run; an unresolved source keeps its previous site, and with it its docs. The indexer's contributes applies the same rule to what it writes.

func (*Writer) Pages added in v0.15.0

func (w *Writer) Pages(ctx context.Context, source string) (map[string]PageState, error)

Pages returns the state of every page indexed for source, by path.

func (*Writer) PendingRemovals added in v0.15.0

func (w *Writer) PendingRemovals(ctx context.Context, source string) ([]string, error)

PendingRemovals names source's pages that recent listings have omitted, so content leaving the index is visible before it goes (spec 0008 Q-3).

func (*Writer) PutDocument

func (w *Writer) PutDocument(ctx context.Context, d Document) error

PutDocument indexes a document, replacing whatever was there before.

The whole document is written in one transaction, and its chunks are deleted before the new ones are inserted. Without the replacement a refresh would accumulate every previous version of a page and happily cite prose that has since been rewritten — a citation that resolves, to text that no longer says what the answer claimed.

func (*Writer) PutPage added in v0.15.0

func (w *Writer) PutPage(ctx context.Context, p Page) error

PutPage indexes a page, replacing its previous version, and resets its absences.

func (*Writer) PutSite added in v0.15.0

func (w *Writer) PutSite(ctx context.Context, origin string) error

PutSite records a configured site as a qualifying source.

func (*Writer) PutSource

func (w *Writer) PutSource(ctx context.Context, s Source) error

PutSource records a source and the verdict that admitted or rejected it.

Rejected sources are kept deliberately: an operator asking why something is not indexed should get the clause that decided it rather than silence.

An unresolved check never replaces a stored verdict: "could not check" is not an answer, and a throttled refresh must not unlist a documented source. The failure goes to last_error, where index status shows it (issue 5), and the next decided verdict clears it.

func (*Writer) RecordSiteRun added in v0.15.0

func (w *Writer) RecordSiteRun(ctx context.Context, origin string, runErr error) error

RecordSiteRun marks a site refreshed, recording the run's error, or clearing it when there was none.

func (*Writer) RemovePage added in v0.15.0

func (w *Writer) RemovePage(ctx context.Context, source, path string) error

RemovePage deletes a page and its document now, for a page whose own content rules it out rather than its absence from a listing.

func (*Writer) RetainDocuments added in v0.15.0

func (w *Writer) RetainDocuments(ctx context.Context, sourcePath string, keep []string, preserve ...corpus.Kind) error

RetainDocuments removes a source's documents whose path is not in keep, leaving documents of the preserved kinds alone. Only a complete snapshot may name keep: a partial one would remove documents that still exist. An empty keep removes them all.

func (*Writer) RetireUnconfiguredSites added in v0.15.0

func (w *Writer) RetireUnconfiguredSites(ctx context.Context, origins []string) error

RetireUnconfiguredSites stops every site not in origins qualifying, so a site removed from configuration stops being cited. Its pages stay until it is configured again, as a retired repository's do.

func (*Writer) RetireUnlisted added in v0.12.3

func (w *Writer) RetireUnlisted(ctx context.Context, listed []string) error

RetireUnlisted stops every source missing from listed qualifying, so a renamed, moved or deleted repository leaves the corpus. The listing must be complete; an empty one retires nothing.

func (*Writer) SweepAbsent added in v0.15.0

func (w *Writer) SweepAbsent(ctx context.Context, source string, listed []string) ([]string, error)

SweepAbsent applies one complete listing: every page in listed has its absences reset, every other page of source gains one, and a page reaching AbsencesToRemove is deleted with its document. Only a listing read in full may be passed here; a partial one would count present pages as absent.

func (*Writer) TouchPage added in v0.15.0

func (w *Writer) TouchPage(ctx context.Context, source, path string, t Touch) error

TouchPage records a page whose text is unchanged. A validator the server did not resend keeps its previous value. The lane and date are the feed's, which can move without the text changing (phpbotscout#40).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL