search

type AnalysisLimitError ¶ added in v0.18.0

type AnalysisLimitError struct {
	Kind  AnalysisLimitKind
	Limit int64
	Got   int64
}

AnalysisLimitError reports a deterministic per-document or aggregate index budget failure. Callers should branch on Kind, not Error's prose.

func (e *AnalysisLimitError) Error() string

type AnalysisLimitKind ¶ added in v0.18.0

type AnalysisLimitKind string

AnalysisLimitKind is a stable machine-readable indexing limit name.

const (
	LimitDocumentBytes   AnalysisLimitKind = "document_bytes"
	LimitDocumentTokens  AnalysisLimitKind = "document_tokens"
	LimitDocumentTerms   AnalysisLimitKind = "document_terms"
	LimitLiveTerms       AnalysisLimitKind = "live_terms"
	LimitLivePostings    AnalysisLimitKind = "live_postings"
	LimitPositionEntries AnalysisLimitKind = "position_entries"
)

type AnalysisStats ¶ added in v0.18.0

type AnalysisStats struct {
	ProjectedBytes  int
	Tokens          int
	UniqueTerms     int
	Postings        int
	PositionEntries int
}

AnalysisStats describes the bounded representation produced for one document. ProjectedBytes counts the UTF-8 bytes handed to the analyzer; Postings is the number of distinct (class, term) pairs.

type Analyzer ¶

type Analyzer interface {
	Analyze(text string) []Token
}

Analyzer converts raw text into index terms. Queries use the same Analyze method unless the dynamic implementation also provides QueryAnalyzer for a documented query-only recall channel. Implementations in this package are stateless and safe for concurrent use by multiple goroutines.

func NewAnalyzer(normalizers []Normalizer, tokenizer Tokenizer, filters []TokenFilter) Analyzer

func NewNGramAnalyzer(n int) Analyzer

func NewScriptAwareAnalyzer() Analyzer

type AnalyzerFunc ¶

type AnalyzerFunc func(text string) []Token

AnalyzerFunc adapts an ordinary function to the Analyzer interface.

func (f AnalyzerFunc) Analyze(text string) []Token

type BM25 ¶

type BM25 struct {
	// K1 tunes term-frequency saturation (typical range 1.2–2.0).
	K1 float64
	// B tunes document-length normalization, from 0 (off) to 1 (full).
	B float64
}

BM25 scores a term/document pairing with the Okapi BM25 ranking function, the modern default for keyword relevance. It improves on plain TF-IDF in two ways: term frequency saturates (the 10th occurrence of a word adds far less than the 2nd, tuned by K1), and the score is normalized by document length so a long document does not rank highly just for being wordy (tuned by B). The inverse-document-frequency factor still rewards rare, discriminating terms over common ones.

The zero value is usable but disables length normalization (B == 0); prefer the parameters installed by NewInvertedIndex when scorer is nil, namely K1 = 1.2 and B = 0.75. A non-positive K1 falls back to 1.2, and B is clamped to [0, 1]. BM25 holds no mutable state and is safe for concurrent use.

func (s BM25) Score(stats TermStats) float64

type Bool ¶

type Bool bool

Bool adapts a boolean to Document as "true" or "false".

func (b Bool) String() string

type Budget ¶ added in v0.18.0

type Budget struct {
	MaxQueryTerms       int64
	MaxDictionaryVisits int64
	MaxPostingVisits    int64
	MaxPositionVisits   int64
	MaxExpirationVisits int64
}

Budget caps work performed by one query. Zero means unlimited and is used by compatibility helpers; the server validates and supplies positive limits.

type BudgetExceededError ¶ added in v0.18.0

type BudgetExceededError struct {
	Kind  WorkKind
	Limit int64
}

BudgetExceededError records which stable counter exhausted its limit.

func (e *BudgetExceededError) Error() string

func (e *BudgetExceededError) Unwrap() error

type Bytes ¶

type Bytes []byte

Bytes adapts valid UTF-8 to text. Arbitrary binary bytes produce an empty document instead of accidentally entering the text analyzer.

func (b Bytes) String() string

type CandidateSource ¶ added in v0.18.0

type CandidateSource[S comparable] func(yield func(S) bool)

CandidateSource streams a distinct set of document IDs into a bounded top-k query. A layered store can use it to push a selective secondary-index scope (for example a key-prefix radix) into retrieval instead of scoring the global posting union and filtering afterwards. The source is consumed synchronously while the index read lock is held; it must not call an index write method.

type CanonicalNormalizer ¶ added in v0.18.0

type CanonicalNormalizer struct{}

CanonicalNormalizer converts canonically equivalent spellings to NFC while preserving every combining mark. It is the conservative production choice: composed/decomposed input matches, but Thai tones, Indic viramas/matras, accents, niqqud, and harakat are never erased as a recall shortcut.

func (CanonicalNormalizer) Normalize(text string) string

type CaseFoldNormalizer ¶ added in v0.18.0

type CaseFoldNormalizer struct{}

CaseFoldNormalizer applies Unicode's language-neutral full case fold. Unlike lowercasing, case folding expands equivalences such as German sharp-s to "ss" and maps both Greek sigma forms to sigma. It deliberately does not apply locale-specific mappings (for example Turkish dotted/dotless I).

func (CaseFoldNormalizer) Normalize(text string) string

type ClassWeighted ¶ added in v0.16.0

type ClassWeighted struct {
	// Base scores each pairing before weighting; nil means standard BM25.
	Base Scorer
	// GramWeight multiplies ClassGram matches, clamped to [0, 1].
	GramWeight float64
}

ClassWeighted layers per-class weighting over a base Scorer (#888): a match on the auxiliary gram channel (ClassGram) is multiplied by GramWeight while primary evidence (ClassWord) passes through unchanged. With the dual-class ScriptAwareTokenizer this is what makes a whole-word match dominate the redundant intra-word fragments that only exist to keep infix recall: the fragments still surface a document, but at a fraction of the weight, so "arch" ranks the document containing the word above the ones merely containing the substring.

GramWeight is clamped to [0, 1]; 0 silences the auxiliary channel entirely and 1 restores unweighted scoring. A nil Base falls back to BM25 with the standard parameters, so the zero value scores like the default scorer with the gram channel muted. ClassWeighted holds no mutable state and is safe for concurrent use.

func (s ClassWeighted) Score(stats TermStats) float64

type DiacriticNormalizer ¶

type DiacriticNormalizer struct{}

DiacriticNormalizer folds accents and other nonspacing combining marks so matching is diacritic-insensitive: it decomposes text to Unicode NFD, drops every rune in category Mn (nonspacing mark), then recomposes to NFC. This makes "café" match "cafe", "naïve" match "naive", Greek "Ελλάδα" match "Ελλαδα", and Russian "ёж" match "еж" — across any script that encodes its accents as combining marks (Latin, Greek, Cyrillic, Vietnamese, …). It does not transliterate between scripts and never alters a base letter, so letters that are atomic rather than base-plus-mark — "ß", "ø", "ł" — are left intact.

Because it removes every nonspacing mark, it also strips the optional harakat of Arabic, the niqqud of Hebrew, and the (non-optional) vowel marks of Thai, trading some precision for recall in those scripts. It preserves spacing combining marks (category Mc), so Brahmic base+matra clusters such as Devanagari "भारत" keep their vowels, and the NFC recomposition keeps Hangul syllables and other composed forms stable. It folds independently of case, so its order relative to LowercaseNormalizer does not matter, and it is a no-op for scripts written without combining marks (CJK, Hangul).

func (DiacriticNormalizer) Normalize(text string) string

type Document ¶

type Document interface {
	String() string
}

Document is the indexable payload handed to an Indexer: anything that can serialize itself to the text to be analyzed. It is structurally identical to fmt.Stringer, so any existing Stringer already satisfies it. Implementing String lets a richer record decide which of its fields become searchable (for example concatenating a title and a body) instead of forcing callers to pre-flatten every document into a bare string.

type DocumentField ¶ added in v0.18.0

type DocumentField struct {
	ID   FieldID
	Text string
}

DocumentField is one independently analyzed field instance. Multiple FieldValue entries are allowed (for example JSON string leaves); phrase and proximity never cross from one instance to another.

type Duration ¶

type Duration time.Duration

Duration adapts a duration to Document in its canonical Go form (e.g. "1h30m0s").

func (d Duration) String() string

type EmojiPresentationNormalizer ¶ added in v0.18.0

type EmojiPresentationNormalizer struct{}

EmojiPresentationNormalizer removes the text/emoji presentation selectors U+FE0E and U+FE0F. Presentation is a rendering preference rather than lexical identity, so "❤", "❤︎", and "❤️" share one searchable term. ZWJ and skin-tone modifiers remain significant.

func (EmojiPresentationNormalizer) Normalize(text string) string

type EmptyTokenFilter ¶

type EmptyTokenFilter struct{}

EmptyTokenFilter drops tokens whose term is empty or all whitespace—cheap insurance after custom normalizers or filters.

func (EmptyTokenFilter) Filter(tokens []Token) []Token

type FieldID ¶ added in v0.18.0

type FieldID uint8

FieldID is a stable semantic search field. FieldDefault preserves the single-text Document contract; FieldKey and FieldValue let layered stores keep identifier and content evidence separate without importing their domain types into core.

const (
	FieldDefault FieldID = iota
	FieldKey
	FieldValue
)

type FieldWeighted ¶ added in v0.18.0

type FieldWeighted struct {
	Base        Scorer
	KeyWeight   float64
	ValueWeight float64
}

FieldWeighted layers stable key/value weights over a base scorer. Default text and value evidence use weight 1 unless explicitly overridden.

func (s FieldWeighted) Score(stats TermStats) float64

type FieldedDocument ¶ added in v0.18.0

type FieldedDocument interface {
	Document
	SearchFields() []DocumentField
}

FieldedDocument exposes structured search fields. It remains a Document so existing generic index bounds and diagnostics keep a plain-text fallback, but Prepare uses SearchFields directly when this interface is present.

type Fields ¶ added in v0.18.0

type Fields []DocumentField

Fields is a convenient immutable FieldedDocument for callers that already have projected text instances.

func (f Fields) SearchFields() []DocumentField

func (f Fields) String() string

type Float ¶

type Float float64

Float adapts a floating-point number to Document, using the shortest decimal that round-trips back to the same float64 (e.g. 3.14, 1e+20).

func (f Float) String() string

type IndexHealth ¶ added in v0.18.0

type IndexHealth string

IndexHealth is the search index consistency state.

const (
	IndexHealthy    IndexHealth = "healthy"
	IndexIncomplete IndexHealth = "incomplete"
)

type IndexMemoryStats ¶ added in v0.18.0

type IndexMemoryStats struct {
	Documents              int
	PhysicalDocuments      int
	ExpiredDocuments       int
	ExpirationQueueEntries int
	LiveTerms              int
	RetainedTermSlots      int
	RetainedOrdinals       int
	Postings               int64
	PositionEntries        int64
	EstimatedLiveBytes     int64
	EstimatedRetainedBytes int64
	RebuildCount           uint64
	LastRebuildDuration    time.Duration
	WriteLockAcquisitions  uint64
	ExpirationPurged       uint64
	LastExpirationPurge    time.Duration
	Generation             uint64
	Health                 IndexHealth
}

IndexMemoryStats is an observable snapshot of live and retained index state.

type IndexOption ¶ added in v0.16.0

type IndexOption func(*indexConfig)

IndexOption configures an InvertedIndex at construction time.

func WithAnalysisLimits(limits SearchAnalysisLimits) IndexOption

func WithIndexClock(clock func() time.Time) IndexOption

func WithPositions() IndexOption

func WithProximityWeight(w float64) IndexOption

type Indexer ¶

type Indexer[S comparable, D Document] interface {
	Index(id S, doc D) error
	Delete(id S)
}

Indexer is the write side of a keyword-search engine. Index associates the document identified by id with the searchable terms extracted from doc, and Delete removes a previously indexed document. Indexing the same id again replaces its postings, so Index doubles as update. Index returns a typed AnalysisLimitError when configured document or aggregate budgets reject the final state. How text is analyzed and how postings are stored is left to the implementation. S is the caller's document-identifier type and D is the indexable document type.

type Int ¶

type Int int64

Int adapts a signed integer to Document, rendered in base 10.

func (i Int) String() string

type InvertedIndex ¶

type InvertedIndex[S comparable, D Document] struct {
	// contains filtered or unexported fields
}

InvertedIndex is a generic, in-memory inverted index that maps each analyzed term to the documents whose text contains it. It is the default Indexer + Searcher implementation: indexed documents (any Document, via doc.String()) and raw queries pass through the configured Analyzer. An optional QueryAnalyzer may add query-only evidence; otherwise index-time and query-time terms stay symmetric.

It splits the two responsibilities of relevance search. The index owns matching and the statistics behind it — per-document term frequencies and document lengths — while a pluggable Scorer (BM25 by default) turns those statistics into the ranking. Search returns hits in the stable total order (score descending, caller-provided typed ID comparator ascending).

Terms live in per-class tables (#888): each TokenClass has its own term dictionary and postings. Corpus statistics and posting payloads are further scoped by semantic Document field (#1061), so a dual-channel analyzer (ScriptAwareTokenizer's whole words + auxiliary intra-word grams) never mixes the two channels' document frequencies or lengths, and a class-aware Scorer (ClassWeighted) can weight them apart. A single-channel pipeline simply leaves the other class empty at zero cost.

InvertedIndex is safe for concurrent use by multiple goroutines: reads (Search) take a shared lock and writes (Index, Delete) take an exclusive one, and text analysis runs outside the lock to keep the critical section short.

func NewInvertedIndex[S comparable, D Document](analyzer Analyzer, scorer Scorer, compareID func(S, S) int, opts ...IndexOption) *InvertedIndex[S, D]

func (idx *InvertedIndex[S, D]) Compact()

func (idx *InvertedIndex[S, D]) Delete(id S)

func (idx *InvertedIndex[S, D]) DeleteMany(ids []S)

func (idx *InvertedIndex[S, D]) Health() IndexHealth

func (idx *InvertedIndex[S, D]) Index(id S, doc D) error

func (idx *InvertedIndex[S, D]) IndexManyPrepared(items []PreparedItem[S]) error

func (idx *InvertedIndex[S, D]) IndexManyPreparedValidated(items []PreparedItem[S])

func (idx *InvertedIndex[S, D]) IndexPrepared(id S, prepared PreparedDocument) error

func (idx *InvertedIndex[S, D]) IndexPreparedWithExpiration(id S, prepared PreparedDocument, expiration time.Time) error

func (idx *InvertedIndex[S, D]) IndexWithExpiration(id S, doc D, expiration time.Time) error

func (idx *InvertedIndex[S, D]) MarkIncomplete()

func (idx *InvertedIndex[S, D]) MemoryStats() IndexMemoryStats

func (idx *InvertedIndex[S, D]) Prepare(doc D) (PreparedDocument, AnalysisStats, error)

func (idx *InvertedIndex[S, D]) RebuildDocuments(visit func(yield func(S, D, time.Time) error) error) error

func (idx *InvertedIndex[S, D]) RebuildPrepared(items []PreparedItem[S]) error

func (idx *InvertedIndex[S, D]) Search(query string) []Result[S]

func (idx *InvertedIndex[S, D]) SearchMatch(query string, opts MatchOptions) []Result[S]

func (idx *InvertedIndex[S, D]) SearchMatchTopK(query string, k int, accept func(id S) bool, opts MatchOptions) []Result[S]

func (idx *InvertedIndex[S, D]) SearchMatchTopKCandidatesContextAt(ctx context.Context, query string, k int, accept func(id S) bool, opts MatchOptions, budget Budget, now time.Time, candidates CandidateSource[S]) ([]Result[S], Stats, error)

func (idx *InvertedIndex[S, D]) SearchMatchTopKContext(ctx context.Context, query string, k int, accept func(id S) bool, opts MatchOptions, budget Budget) ([]Result[S], Stats, error)

func (idx *InvertedIndex[S, D]) SearchMatchTopKContextAt(ctx context.Context, query string, k int, accept func(id S) bool, opts MatchOptions, budget Budget, now time.Time) ([]Result[S], Stats, error)

func (idx *InvertedIndex[S, D]) SearchPhrase(query string) []Result[S]

func (idx *InvertedIndex[S, D]) SearchPhraseTopK(query string, k int, accept func(id S) bool) []Result[S]

func (idx *InvertedIndex[S, D]) SearchPhraseTopKCandidatesContextAt(ctx context.Context, query string, k int, accept func(id S) bool, budget Budget, now time.Time, candidates CandidateSource[S]) ([]Result[S], Stats, error)

func (idx *InvertedIndex[S, D]) SearchPhraseTopKContext(ctx context.Context, query string, k int, accept func(id S) bool, budget Budget) ([]Result[S], Stats, error)

func (idx *InvertedIndex[S, D]) SearchPhraseTopKContextAt(ctx context.Context, query string, k int, accept func(id S) bool, budget Budget, now time.Time) ([]Result[S], Stats, error)

func (idx *InvertedIndex[S, D]) SearchTopK(query string, k int, accept func(id S) bool) []Result[S]

func (idx *InvertedIndex[S, D]) Stats() (terms, docs int)

func (idx *InvertedIndex[S, D]) ValidateManyPrepared(items []PreparedItem[S]) error

type JSONStringValueNormalizer ¶ added in v0.12.0

type JSONStringValueNormalizer struct{}

JSONStringValueNormalizer rewrites text that is a JSON object or array into just the text of its string values, dropping every object field name (key) and every non-string scalar (number, boolean, null). Its purpose is to keep a full-text index free of JSON structure: when a vertex stores a serialized document such as {"role":"admin","score":9,"active":true}, only the human content ("admin") becomes searchable — not the field names ("role", "score", "active") and not the numeric or boolean scalars. String values are emitted in object-key sorted order (arrays keep their order) so the output is deterministic and re-indexing is idempotent.

Text that is not a JSON object or array passes through unchanged: a fast path returns the input verbatim when the first non-space byte is neither '{' nor '[', and any string that starts that way but fails to parse as JSON is also returned verbatim (so a value that merely looks like JSON is still indexed as raw text). A JSON object/array carrying no string values normalizes to the empty string.

Unlike the other normalizers in this package, this one is meant to run on a value in isolation — the document projection of a single stored value — not on a composed document such as "key value", because a key concatenated with a JSON blob is not itself valid JSON and the fast path would return it unchanged. Place it at the projection step, not in a shared analyzer chain.

func (JSONStringValueNormalizer) Normalize(text string) string

func (JSONStringValueNormalizer) Values(text string) (values []string, structured bool)

type LengthFilter ¶

type LengthFilter struct {
	Min int
	Max int
}

LengthFilter drops tokens whose length in runes is below Min or above Max. A Max of 0 means no upper bound. It removes noise such as stray single letters.

func (f LengthFilter) Filter(tokens []Token) []Token

type LowercaseFilter ¶

type LowercaseFilter struct{}

LowercaseFilter folds each token to lower case. Use it when tokenization runs before case folding; if you already lowercase with a LowercaseNormalizer it is redundant.

func (LowercaseFilter) Filter(tokens []Token) []Token

type LowercaseNormalizer ¶

type LowercaseNormalizer struct{}

LowercaseNormalizer folds text to lower case with Unicode case mapping, so matching is case-insensitive independent of language.

func (LowercaseNormalizer) Normalize(text string) string

type MatchMode ¶ added in v0.16.0

type MatchMode uint8

MatchMode selects how a multi-term query's word-channel terms combine to decide which documents match. It governs membership only — the Scorer still ranks whatever matches — and counts distinct query terms on the primary (word) channel, so the auxiliary intra-word grams a script-aware analyzer emits are evidence for ranking but never change how many terms a document must satisfy. For a single-channel (word-only) analyzer every match is a word match, so the modes reduce to the textbook boolean semantics.

const (
	// MatchAny keeps every document sharing at least one query term (the
	// OR-union). It is the zero value and the package default, because the
	// graph-seeding path Search feeds wants maximal recall.
	MatchAny MatchMode = iota
	// MatchAll keeps only documents containing every distinct query word term
	// (boolean AND), the precision end of the scale.
	MatchAll
	// MatchMinShould keeps documents containing at least MinShouldMatch distinct
	// query word terms — the tunable middle ground between Any and All.
	MatchMinShould
)

type MatchOptions ¶ added in v0.16.0

type MatchOptions struct {
	// Mode selects the boolean combination of the query's word terms.
	Mode MatchMode
	// MinShouldMatch is the minimum number of distinct query word terms a
	// document must contain when Mode is MatchMinShould. It is clamped to
	// [1, number of distinct query word terms]; other modes ignore it.
	MinShouldMatch int
	// Fuzziness is the maximum edit distance (0, 1, or 2) at which a word-channel
	// query term also matches dictionary terms, so a typo like "serach" still
	// finds "search". 0 (the default) disables fuzzy matching. CJK grams are
	// exempt. Expansion per query term is capped at MaxTermExpansions.
	Fuzziness int
	// PrefixTerms, when set, also matches dictionary terms that extend a
	// word-channel query term, so "lan" finds "lantern" (a prefix query). CJK
	// grams are exempt; expansion is capped at MaxTermExpansions.
	PrefixTerms bool
}

MatchOptions configures document membership for a query. The zero value is MatchAny with no term expansion, so an empty MatchOptions reproduces Search's OR-union exactly.

type NGramTokenizer ¶

type NGramTokenizer struct {
	N int
}

NGramTokenizer slides a fixed-width window of N runes across the whole input and emits one token per position, so "search" with N=3 yields "sea", "ear", "arc", "rch". Because it works on runes (not bytes) and never relies on word boundaries, it is the language-independent way to make substring queries and scripts with no whitespace between words (e.g. CJK) matchable, where the word-splitting tokenizers would otherwise emit one giant token.

The window moves over every rune—whitespace and punctuation included—so a caller who wants grams that do not straddle word boundaries should strip or collapse those runes with a Normalizer first. Every emitted term is exactly N runes long, and an input shorter than N yields no tokens. N < 1 is treated as 1 (unigrams), so the zero value tokenizes into single characters rather than misbehaving.

func (t NGramTokenizer) Tokenize(text string) []Token

type Normalizer ¶

type Normalizer interface {
	Normalize(text string) string
}

Normalizer rewrites raw text before tokenization—for example folding case or collapsing whitespace. Implementations in this package are stateless and therefore safe for concurrent use by multiple goroutines.

type NormalizerFunc ¶

type NormalizerFunc func(text string) string

NormalizerFunc adapts an ordinary function to the Normalizer interface.

func (f NormalizerFunc) Normalize(text string) string

type PreparedDocument ¶ added in v0.11.0

type PreparedDocument struct {
	// contains filtered or unexported fields
}

PreparedDocument carries an already grouped document representation so all analysis, partitioning, sorting, frequency aggregation, and position collection complete before the index write lock is acquired. Produce one with Prepare and apply it with IndexPrepared or IndexManyPrepared. The zero value is a valid empty document: applying it removes the id, matching Index of a document with no analyzable terms.

type PreparedItem ¶ added in v0.11.0

type PreparedItem[S comparable] struct {
	ID         S
	Prepared   PreparedDocument
	Expiration time.Time
}

PreparedItem pairs an id with its PreparedDocument for IndexManyPrepared.

type PunctuationFilter ¶

type PunctuationFilter struct{}

PunctuationFilter strips leading and trailing punctuation and symbol runes from each token and drops tokens left empty, while preserving inner marks (so "hello," becomes "hello" but "node-1" and "u.s.a" are untouched). It is the companion to WhitespaceTokenizer; UnicodeTokenizer already drops punctuation.

func (PunctuationFilter) Filter(tokens []Token) []Token

type PunctuationNormalizer ¶

type PunctuationNormalizer struct{}

PunctuationNormalizer replaces every Unicode punctuation or symbol rune with an ASCII space, turning marks such as '.', ',', '。', and '、' into word boundaries independent of language. It does not merge the spaces it introduces; chain SpaceNormalizer after it to collapse the resulting runs and trim the ends. Unlike PunctuationFilter, which only trims marks from the edges of an already-formed token (so "node-1" stays intact), this splits on every mark, so "node-1" becomes "node 1".

func (PunctuationNormalizer) Normalize(text string) string

type QueryAnalyzer ¶ added in v0.18.0

type QueryAnalyzer interface {
	Analyzer
	AnalyzeQuery(text string) []Token
}

QueryAnalyzer is an optional asymmetric analysis extension. An index always calls Analyze for documents; when the dynamic Analyzer also implements this interface it calls AnalyzeQuery for search input. Match-mode coverage remains defined solely by ClassWord; extra ClassGram terms are ranking evidence.

type Result ¶

type Result[S comparable] struct {
	ID    S
	Score float64
}

Result is a single ranked search hit: the ID of a matching document and the relevance Score the Searcher's Scorer assigned it. Higher scores are more relevant.

type Scorer ¶

type Scorer interface {
	Score(stats TermStats) float64
}

Scorer assigns a relevance weight to a single (query term, document) pairing from the corpus statistics in stats. The Searcher sums a document's per-term scores to rank it, so a Scorer only has to express how one term contributes. Separating this from the index keeps "which documents match" (the index's job) independent of "how relevant each match is" (the Scorer's job), and lets the ranking formula be swapped without touching the postings. A higher score means more relevant; implementations should stay non-negative so that summing across terms is well behaved.

type ScorerFunc ¶

type ScorerFunc func(stats TermStats) float64

ScorerFunc adapts an ordinary function to the Scorer interface.

func (f ScorerFunc) Score(stats TermStats) float64

type ScriptAwareTokenizer ¶ added in v0.16.0

type ScriptAwareTokenizer struct {
	// N is the gram width, for unbounded-script runs and the auxiliary
	// intra-word grams alike. N < 2 is treated as 2.
	N int
}

ScriptAwareTokenizer is the production tokenizer for mixed-script content search (#888). It splits text into script runs and picks the right unit per run, instead of forcing one strategy onto every language the way a pure word or pure n-gram tokenizer does:

• A run of a space-delimited script (Latin, Cyrillic, Greek, digits, …) emits the whole word as a primary token (ClassWord) — the unit Lucene's StandardAnalyzer indexes, and what lets a whole-word match outrank an infix fragment. Words of at least N runes additionally emit every N-rune window as auxiliary tokens (ClassGram), which is what keeps the bigram pipeline's infix recall ("arch" finding "search") and typo tolerance; pair the index with ClassWeighted so this redundant channel stays evidence, not the ranking.
• A CJKBigram run — Unicode Ideographic characters, Hiragana, Katakana, and Hangul, following Lucene's documented CJKBigramFilter script set — emits its N-rune windows as primary tokens, exactly the strategy of Lucene's CJKAnalyzer, because there the gram is the word-level unit. A run shorter than N (a lone ideograph) is emitted whole, so single- character words stay searchable.
• Combining marks continue the preceding lexical run, preserving Thai and Indic identity. Each Unicode symbol is searchable as one primary token; ZWJ-linked symbols form one token and adjacent unjoined symbols do not.
• Every other rune (whitespace and punctuation) delimits runs and is dropped, so no gram ever straddles a word boundary and the WhitespaceFilter step of the plain n-gram pipeline is unnecessary.

Run it after the production normalizers (width, canonical composition, full case fold, emoji presentation, punctuation, space) — NewScriptAwareAnalyzer wires that pipeline. In particular WidthNormalizer must run first so half-width katakana joins the katakana run it belongs to. N < 2 is treated as 2 (bigrams), so the zero value is the production configuration. It holds no state and is safe for concurrent use.

func (t ScriptAwareTokenizer) Tokenize(text string) []Token

func (t ScriptAwareTokenizer) TokenizeBounded(text string, maxTokens int) ([]Token, bool)

type SearchAnalysisLimits ¶ added in v0.18.0

type SearchAnalysisLimits struct {
	MaxDocumentBytes     int
	MaxDocumentTokens    int
	MaxDocumentTerms     int
	MaxLiveTerms         int
	MaxLivePostings      int64
	MaxPositionEntries   int64
	CompactionRatio      float64
	CompactionMinRetired int
}

SearchAnalysisLimits bounds the memory amplification of indexing. A zero field disables that individual limit. CompactionRatio and CompactionMinRetired control reclamation of dictionary/ordinal high-water storage after delete or TTL churn.

type Searcher ¶

type Searcher[S comparable] interface {
	Search(query string) []Result[S]
}

Searcher is the read side of a keyword-search engine. Search turns a raw query string into the matching documents, each paired with a relevance score, ordered from most to least relevant — the ranked seed candidates a caller picks before a wider graph traversal, where the score can double as a seed's initial weight. Documents that share no analyzed term with the query are omitted, and a query with no analyzable terms yields no results. Implementations provide a stable document-ID order for equal scores; InvertedIndex requires a typed ascending ID comparator at construction.

type SizedDocument ¶ added in v0.18.0

type SizedDocument interface {
	Document
	SizeHint() int
}

SizedDocument can expose a cheap upper bound before String performs an expensive projection. The production vertex projection uses it to reject a large JSON/string payload before parsing or projecting its fields.

type SpaceNormalizer ¶

type SpaceNormalizer struct{}

SpaceNormalizer collapses every run of Unicode whitespace to a single ASCII space and trims the ends. It is useful in front of a WhitespaceTokenizer when the input may contain tabs, newlines, or repeated spaces.

func (SpaceNormalizer) Normalize(text string) string

type Stats ¶ added in v0.18.0

type Stats struct {
	QueryBytes        int64
	QueryTokens       int64
	QueryClauses      int64
	QueryTerms        int64
	DictionaryVisits  int64
	ExpansionRetained int64
	PostingVisits     int64
	PositionVisits    int64
	ExpirationVisits  int64
	CandidateVisits   int64
	CandidateSkips    int64
	AnalysisDuration  time.Duration
	ExpansionDuration time.Duration
	SelectionDuration time.Duration
}

Stats captures exact deterministic work plus coarse executor phase timing for one search attempt.

type StopWordFilter ¶

type StopWordFilter struct {
	// contains filtered or unexported fields
}

StopWordFilter removes tokens that appear in a stop-word set. Matching is case-insensitive (terms and stop words are compared lower-cased), so the filter works whether or not case folding ran earlier.

func NewStopWordFilter(words ...string) *StopWordFilter

func (f *StopWordFilter) Filter(tokens []Token) []Token

type SymbolPreservingPunctuationNormalizer ¶ added in v0.18.0

type SymbolPreservingPunctuationNormalizer struct{}

SymbolPreservingPunctuationNormalizer replaces Unicode punctuation with a boundary while retaining symbols as searchable content. Production search uses it so emoji and intentional mathematical/currency symbols do not vanish; callers that want the older punctuation-and-symbol boundary policy can keep using PunctuationNormalizer.

func (SymbolPreservingPunctuationNormalizer) Normalize(text string) string

type TermStats ¶

type TermStats struct {
	// TF is how many times the term occurs in this document (term frequency).
	TF int
	// DF is how many documents in the corpus contain the term (document
	// frequency); it drives the inverse-document-frequency weight.
	DF int
	// N is the number of documents carrying at least one token of the term's
	// class — the corpus size of the term's channel.
	N int
	// DocLen is the length of this document in tokens of the term's class.
	DocLen int
	// AvgLen is the mean DocLen across the class's N documents, used to
	// normalize DocLen so long and short documents compete fairly.
	AvgLen float64
	// Class is the matching channel the term belongs to, so a class-aware
	// Scorer (ClassWeighted) can weight primary and auxiliary evidence apart.
	Class TokenClass
	// Field identifies the semantic document field supplying this evidence.
	Field FieldID
}

TermStats carries the corpus statistics a Scorer needs to weight one (query term, matching document) pairing. The Searcher fills it in from the index while computing a result; a Scorer treats it as read-only input and holds no state of its own, so a single Scorer is safe for concurrent use.

Every count is scoped to the term's token class and semantic field (#888/#1061): DF, N, DocLen, and AvgLen describe only that channel/field. For a single-channel, single-field pipeline that scoping is invisible — the class's corpus is the corpus.

type Text ¶

type Text string

Text adapts a plain string to Document so raw text can be indexed without a bespoke wrapper type: idx.Index(id, Text("some words")).

func (t Text) String() string

type Time ¶

type Time time.Time

Time adapts a timestamp to Document, formatted as RFC3339 with nanosecond precision — the same rendering Vertex timestamps use elsewhere.

func (t Time) String() string

type Token ¶

type Token struct {
	// Term is the analyzed text of this token.
	Term string
	// Class labels the matching channel the token belongs to; the zero value
	// (ClassWord) is the primary channel, so tokenizers that predate classes
	// keep their exact behavior.
	Class TokenClass
}

Token is the unit of analyzed text that flows through the package's analysis pipeline: a Tokenizer emits Tokens, TokenFilters rewrite or drop them, and an Analyzer returns the final slice that the index records and queries against.

type TokenClass ¶ added in v0.16.0

type TokenClass uint8

TokenClass separates a token's matching channel (#888). The inverted index keys terms, corpus statistics (document frequency, document length, corpus size), and postings per class, so the two channels never share BM25 statistics, and a class-aware Scorer (ClassWeighted) can weight them apart. Only the constants below are valid: indexing a token with an undefined class panics, and a query token with an undefined class simply matches nothing.

const (
	// ClassWord is the primary channel: whole words of space-delimited
	// scripts, the n-grams of unbounded (CJK-like) script runs — where the
	// gram is the word-level unit — and every token of a single-channel
	// pipeline such as NewNGramAnalyzer. It is deliberately the zero value.
	ClassWord TokenClass = iota
	// ClassGram is the auxiliary channel: intra-word n-gram fragments emitted
	// alongside the word they came from (ScriptAwareTokenizer) so infix and
	// typo-tolerant matching keeps working. Pair with ClassWeighted so this
	// redundant evidence never outranks a whole-word match.
	ClassGram
)

type TokenFilter ¶

type TokenFilter interface {
	Filter(tokens []Token) []Token
}

TokenFilter post-processes a token stream produced by a Tokenizer—dropping, rewriting, or compacting tokens. An Analyzer applies filters in order. Implementations in this package are stateless and safe for concurrent use, but they may reuse the input slice's backing array, so a caller must not keep using the slice it passes in after the call returns.

type TokenFilterFunc ¶

type TokenFilterFunc func(tokens []Token) []Token

TokenFilterFunc adapts an ordinary function to the TokenFilter interface.

func (f TokenFilterFunc) Filter(tokens []Token) []Token

type Tokenizer ¶

type Tokenizer interface {
	Tokenize(text string) []Token
}

Tokenizer splits text into tokens. Implementations in this package are stateless and therefore safe for concurrent use by multiple goroutines.

type TokenizerFunc ¶

type TokenizerFunc func(text string) []Token

TokenizerFunc adapts an ordinary function to the Tokenizer interface.

func (f TokenizerFunc) Tokenize(text string) []Token

type Uint ¶

type Uint uint64

Uint adapts an unsigned integer to Document, rendered in base 10.

func (u Uint) String() string

type UnicodeTokenizer ¶

type UnicodeTokenizer struct{}

UnicodeTokenizer splits text into maximal runs of letters and digits; every other rune (whitespace, punctuation, symbols) is treated as a delimiter and dropped. Tokenizing and stripping punctuation in a single pass makes it the simplest language-independent choice, so it is the default tokenizer.

func (UnicodeTokenizer) Tokenize(text string) []Token

type WhitespaceFilter ¶

type WhitespaceFilter struct{}

WhitespaceFilter drops every token whose term contains a Unicode whitespace rune. Its purpose is to raise the precision of n-gram search: run an NGramTokenizer over text whose words are already separated by single spaces (normalize first with PunctuationNormalizer + SpaceNormalizer) and the only grams that can contain a space are the ones straddling a word or punctuation boundary, e.g. "search" and "engine" yield the cross-boundary gram "h e" for N=3. Dropping them leaves just the intra-word grams, so a document is indexed under its real word content instead of boundary noise: BM25 document lengths reflect the words present, and a query no longer matches on a gram that merely spans two adjacent words. It is language-independent (any whitespace, any script) and a no-op for scripts written without spaces between words (e.g. CJK, Thai), so it is safe to leave in a multilingual pipeline.

func (WhitespaceFilter) Filter(tokens []Token) []Token

type WhitespaceTokenizer ¶

type WhitespaceTokenizer struct{}

WhitespaceTokenizer splits text on runs of Unicode whitespace, leaving every other rune—including punctuation—attached to its token (so "word." stays "word."). Pair it with a PunctuationFilter when that is not what you want.

func (WhitespaceTokenizer) Tokenize(text string) []Token

type WidthNormalizer ¶

type WidthNormalizer struct{}

WidthNormalizer folds full-width and half-width characters to their normal width with Unicode width folding, so the same word written at either width indexes and queries identically — a frequent need for CJK input. Full-width Latin letters, digits, and punctuation collapse to ASCII ("ＴＯＫＹＯ" → "TOKYO", "２０２４" → "2024", "！" → "!"), and half-width katakana expands to full-width ("ｶﾀｶﾅ" → "カタカナ"). It does not fold case, strip accents, or expand compatibility forms such as ligatures; chain LowercaseNormalizer and DiacriticNormalizer for those. Run it early — before PunctuationNormalizer — so width-folded punctuation is then treated as a word boundary. It is a no-op for text that already uses only normal-width characters, ordinary CJK ideographs ("東京") included.

func (WidthNormalizer) Normalize(text string) string

type WorkKind ¶ added in v0.18.0

type WorkKind string

WorkKind identifies one deterministic search-work counter.

const (
	WorkQueryBytes        WorkKind = "query_bytes"
	WorkQueryTokens       WorkKind = "query_tokens"
	WorkQueryClauses      WorkKind = "query_clauses"
	WorkQueryTerms        WorkKind = "query_terms"
	WorkDictionaryVisits  WorkKind = "dictionary_visits"
	WorkExpansionRetained WorkKind = "expansions_retained"
	WorkPostingVisits     WorkKind = "posting_visits"
	WorkPositionVisits    WorkKind = "position_visits"
	WorkExpirationVisits  WorkKind = "expiration_visits"
	WorkCandidateVisits   WorkKind = "candidate_visits"
	WorkCandidateSkips    WorkKind = "candidate_skips"
)