search

type Analyzer ¶

type Analyzer interface {
	Analyze(text string) []Token
}

Analyzer converts raw text into the sequence of index terms used by both indexing and querying. Running the same Analyzer on documents and on queries is what keeps the two sides symmetric. Implementations in this package are stateless and safe for concurrent use by multiple goroutines.

func NewAnalyzer(normalizers []Normalizer, tokenizer Tokenizer, filters []TokenFilter) Analyzer

func NewNGramAnalyzer(n int) Analyzer

type AnalyzerFunc ¶

type AnalyzerFunc func(text string) []Token

AnalyzerFunc adapts an ordinary function to the Analyzer interface.

func (f AnalyzerFunc) Analyze(text string) []Token

type BM25 ¶

type BM25 struct {
	// K1 tunes term-frequency saturation (typical range 1.2–2.0).
	K1 float64
	// B tunes document-length normalization, from 0 (off) to 1 (full).
	B float64
}

BM25 scores a term/document pairing with the Okapi BM25 ranking function, the modern default for keyword relevance. It improves on plain TF-IDF in two ways: term frequency saturates (the 10th occurrence of a word adds far less than the 2nd, tuned by K1), and the score is normalized by document length so a long document does not rank highly just for being wordy (tuned by B). The inverse-document-frequency factor still rewards rare, discriminating terms over common ones.

The zero value is usable but disables length normalization (B == 0); prefer the parameters installed by NewInvertedIndex when scorer is nil, namely K1 = 1.2 and B = 0.75. A non-positive K1 falls back to 1.2, and B is clamped to [0, 1]. BM25 holds no mutable state and is safe for concurrent use.

func (s BM25) Score(stats TermStats) float64

type Bool ¶

type Bool bool

Bool adapts a boolean to Document as "true" or "false".

func (b Bool) String() string

type Bytes ¶

type Bytes []byte

Bytes adapts a raw byte slice to Document by interpreting it as text, so byte-backed UTF-8 content is indexable just like Text. Non-text bytes yield whatever terms the Analyzer extracts from that byte interpretation.

func (b Bytes) String() string

type DiacriticNormalizer ¶

type DiacriticNormalizer struct{}

DiacriticNormalizer folds accents and other nonspacing combining marks so matching is diacritic-insensitive: it decomposes text to Unicode NFD, drops every rune in category Mn (nonspacing mark), then recomposes to NFC. This makes "café" match "cafe", "naïve" match "naive", Greek "Ελλάδα" match "Ελλαδα", and Russian "ёж" match "еж" — across any script that encodes its accents as combining marks (Latin, Greek, Cyrillic, Vietnamese, …). It does not transliterate between scripts and never alters a base letter, so letters that are atomic rather than base-plus-mark — "ß", "ø", "ł" — are left intact.

Because it removes every nonspacing mark, it also strips the optional harakat of Arabic, the niqqud of Hebrew, and the (non-optional) vowel marks of Thai, trading some precision for recall in those scripts. It preserves spacing combining marks (category Mc), so Brahmic base+matra clusters such as Devanagari "भारत" keep their vowels, and the NFC recomposition keeps Hangul syllables and other composed forms stable. It folds independently of case, so its order relative to LowercaseNormalizer does not matter, and it is a no-op for scripts written without combining marks (CJK, Hangul).

func (DiacriticNormalizer) Normalize(text string) string

type Document ¶

type Document interface {
	String() string
}

Document is the indexable payload handed to an Indexer: anything that can serialize itself to the text to be analyzed. It is structurally identical to fmt.Stringer, so any existing Stringer already satisfies it. Implementing String lets a richer record decide which of its fields become searchable (for example concatenating a title and a body) instead of forcing callers to pre-flatten every document into a bare string.

type Duration ¶

type Duration time.Duration

Duration adapts a duration to Document in its canonical Go form (e.g. "1h30m0s").

func (d Duration) String() string

type EmptyTokenFilter ¶

type EmptyTokenFilter struct{}

EmptyTokenFilter drops tokens whose term is empty or all whitespace—cheap insurance after custom normalizers or filters.

func (EmptyTokenFilter) Filter(tokens []Token) []Token

type Float ¶

type Float float64

Float adapts a floating-point number to Document, using the shortest decimal that round-trips back to the same float64 (e.g. 3.14, 1e+20).

func (f Float) String() string

type Indexer ¶

type Indexer[S comparable, D Document] interface {
	Index(id S, doc D)
	Delete(id S)
}

Indexer is the write side of a keyword-search engine. Index associates the document identified by id with the searchable terms extracted from doc, and Delete removes a previously indexed document. Indexing the same id again replaces its postings, so Index doubles as update. How text is analyzed and how postings are stored is left to the implementation. S is the caller's document-identifier type and D is the indexable document type.

type Int ¶

type Int int64

Int adapts a signed integer to Document, rendered in base 10.

func (i Int) String() string

type InvertedIndex ¶

type InvertedIndex[S comparable, D Document] struct {
	// contains filtered or unexported fields
}

InvertedIndex is a generic, in-memory inverted index that maps each analyzed term to the documents whose text contains it. It is the default Indexer + Searcher implementation: indexed documents (any Document, via doc.String()) and raw queries pass through the same Analyzer, so index-time and query-time terms stay symmetric.

It splits the two responsibilities of relevance search. The index owns matching and the statistics behind it — per-document term frequencies and document lengths — while a pluggable Scorer (BM25 by default) turns those statistics into the ranking. Search returns hits ordered by descending score.

InvertedIndex is safe for concurrent use by multiple goroutines: reads (Search) take a shared lock and writes (Index, Delete) take an exclusive one, and text analysis runs outside the lock to keep the critical section short.

func NewInvertedIndex[S comparable, D Document](analyzer Analyzer, scorer Scorer) *InvertedIndex[S, D]

func (idx *InvertedIndex[S, D]) Delete(id S)

func (idx *InvertedIndex[S, D]) Index(id S, doc D)

func (idx *InvertedIndex[S, D]) Search(query string) []Result[S]

func (idx *InvertedIndex[S, D]) Stats() (terms, docs int)

type LengthFilter ¶

type LengthFilter struct {
	Min int
	Max int
}

LengthFilter drops tokens whose length in runes is below Min or above Max. A Max of 0 means no upper bound. It removes noise such as stray single letters.

func (f LengthFilter) Filter(tokens []Token) []Token

type LowercaseFilter ¶

type LowercaseFilter struct{}

LowercaseFilter folds each token to lower case. Use it when tokenization runs before case folding; if you already lowercase with a LowercaseNormalizer it is redundant.

func (LowercaseFilter) Filter(tokens []Token) []Token

type LowercaseNormalizer ¶

type LowercaseNormalizer struct{}

LowercaseNormalizer folds text to lower case with Unicode case mapping, so matching is case-insensitive independent of language.

func (LowercaseNormalizer) Normalize(text string) string

type NGramTokenizer ¶

type NGramTokenizer struct {
	N int
}

NGramTokenizer slides a fixed-width window of N runes across the whole input and emits one token per position, so "search" with N=3 yields "sea", "ear", "arc", "rch". Because it works on runes (not bytes) and never relies on word boundaries, it is the language-independent way to make substring queries and scripts with no whitespace between words (e.g. CJK) matchable, where the word-splitting tokenizers would otherwise emit one giant token.

The window moves over every rune—whitespace and punctuation included—so a caller who wants grams that do not straddle word boundaries should strip or collapse those runes with a Normalizer first. Every emitted term is exactly N runes long, and an input shorter than N yields no tokens. N < 1 is treated as 1 (unigrams), so the zero value tokenizes into single characters rather than misbehaving.

func (t NGramTokenizer) Tokenize(text string) []Token

type Normalizer ¶

type Normalizer interface {
	Normalize(text string) string
}

Normalizer rewrites raw text before tokenization—for example folding case or collapsing whitespace. Implementations in this package are stateless and therefore safe for concurrent use by multiple goroutines.

type NormalizerFunc ¶

type NormalizerFunc func(text string) string

NormalizerFunc adapts an ordinary function to the Normalizer interface.

func (f NormalizerFunc) Normalize(text string) string

type PunctuationFilter ¶

type PunctuationFilter struct{}

PunctuationFilter strips leading and trailing punctuation and symbol runes from each token and drops tokens left empty, while preserving inner marks (so "hello," becomes "hello" but "node-1" and "u.s.a" are untouched). It is the companion to WhitespaceTokenizer; UnicodeTokenizer already drops punctuation.

func (PunctuationFilter) Filter(tokens []Token) []Token

type PunctuationNormalizer ¶

type PunctuationNormalizer struct{}

PunctuationNormalizer replaces every Unicode punctuation or symbol rune with an ASCII space, turning marks such as '.', ',', '。', and '、' into word boundaries independent of language. It does not merge the spaces it introduces; chain SpaceNormalizer after it to collapse the resulting runs and trim the ends. Unlike PunctuationFilter, which only trims marks from the edges of an already-formed token (so "node-1" stays intact), this splits on every mark, so "node-1" becomes "node 1".

func (PunctuationNormalizer) Normalize(text string) string

type Result ¶

type Result[S comparable] struct {
	ID    S
	Score float64
}

Result is a single ranked search hit: the ID of a matching document and the relevance Score the Searcher's Scorer assigned it. Higher scores are more relevant.

type Scorer ¶

type Scorer interface {
	Score(stats TermStats) float64
}

Scorer assigns a relevance weight to a single (query term, document) pairing from the corpus statistics in stats. The Searcher sums a document's per-term scores to rank it, so a Scorer only has to express how one term contributes. Separating this from the index keeps "which documents match" (the index's job) independent of "how relevant each match is" (the Scorer's job), and lets the ranking formula be swapped without touching the postings. A higher score means more relevant; implementations should stay non-negative so that summing across terms is well behaved.

type ScorerFunc ¶

type ScorerFunc func(stats TermStats) float64

ScorerFunc adapts an ordinary function to the Scorer interface.

func (f ScorerFunc) Score(stats TermStats) float64

type Searcher ¶

type Searcher[S comparable] interface {
	Search(query string) []Result[S]
}

Searcher is the read side of a keyword-search engine. Search turns a raw query string into the matching documents, each paired with a relevance score, ordered from most to least relevant — the ranked seed candidates a caller picks before a wider graph traversal, where the score can double as a seed's initial weight. Documents that share no analyzed term with the query are omitted, and a query with no analyzable terms yields no results. Ties in score have an unspecified relative order.

type SpaceNormalizer ¶

type SpaceNormalizer struct{}

SpaceNormalizer collapses every run of Unicode whitespace to a single ASCII space and trims the ends. It is useful in front of a WhitespaceTokenizer when the input may contain tabs, newlines, or repeated spaces.

func (SpaceNormalizer) Normalize(text string) string

type StopWordFilter ¶

type StopWordFilter struct {
	// contains filtered or unexported fields
}

StopWordFilter removes tokens that appear in a stop-word set. Matching is case-insensitive (terms and stop words are compared lower-cased), so the filter works whether or not case folding ran earlier.

func NewStopWordFilter(words ...string) *StopWordFilter

func (f *StopWordFilter) Filter(tokens []Token) []Token

type TermStats ¶

type TermStats struct {
	// TF is how many times the term occurs in this document (term frequency).
	TF int
	// DF is how many documents in the corpus contain the term (document
	// frequency); it drives the inverse-document-frequency weight.
	DF int
	// N is the total number of documents in the corpus.
	N int
	// DocLen is the length, in tokens, of this document.
	DocLen int
	// AvgLen is the mean document length across the corpus, used to normalize
	// DocLen so long and short documents compete fairly.
	AvgLen float64
}

TermStats carries the corpus statistics a Scorer needs to weight one (query term, matching document) pairing. The Searcher fills it in from the index while computing a result; a Scorer treats it as read-only input and holds no state of its own, so a single Scorer is safe for concurrent use.

type Text ¶

type Text string

Text adapts a plain string to Document so raw text can be indexed without a bespoke wrapper type: idx.Index(id, Text("some words")).

func (t Text) String() string

type Time ¶

type Time time.Time

Time adapts a timestamp to Document, formatted as RFC3339 with nanosecond precision — the same rendering Vertex timestamps use elsewhere.

func (t Time) String() string

type Token ¶

type Token struct {
	// Term is the analyzed text of this token.
	Term string
}

Token is the unit of analyzed text that flows through the package's analysis pipeline: a Tokenizer emits Tokens, TokenFilters rewrite or drop them, and an Analyzer returns the final slice that the index records and queries against.

type TokenFilter ¶

type TokenFilter interface {
	Filter(tokens []Token) []Token
}

TokenFilter post-processes a token stream produced by a Tokenizer—dropping, rewriting, or compacting tokens. An Analyzer applies filters in order. Implementations in this package are stateless and safe for concurrent use, but they may reuse the input slice's backing array, so a caller must not keep using the slice it passes in after the call returns.

type TokenFilterFunc ¶

type TokenFilterFunc func(tokens []Token) []Token

TokenFilterFunc adapts an ordinary function to the TokenFilter interface.

func (f TokenFilterFunc) Filter(tokens []Token) []Token

type Tokenizer ¶

type Tokenizer interface {
	Tokenize(text string) []Token
}

Tokenizer splits text into tokens. Implementations in this package are stateless and therefore safe for concurrent use by multiple goroutines.

type TokenizerFunc ¶

type TokenizerFunc func(text string) []Token

TokenizerFunc adapts an ordinary function to the Tokenizer interface.

func (f TokenizerFunc) Tokenize(text string) []Token

type Uint ¶

type Uint uint64

Uint adapts an unsigned integer to Document, rendered in base 10.

func (u Uint) String() string

type UnicodeTokenizer ¶

type UnicodeTokenizer struct{}

UnicodeTokenizer splits text into maximal runs of letters and digits; every other rune (whitespace, punctuation, symbols) is treated as a delimiter and dropped. Tokenizing and stripping punctuation in a single pass makes it the simplest language-independent choice, so it is the default tokenizer.

func (UnicodeTokenizer) Tokenize(text string) []Token

type WhitespaceFilter ¶

type WhitespaceFilter struct{}

WhitespaceFilter drops every token whose term contains a Unicode whitespace rune. Its purpose is to raise the precision of n-gram search: run an NGramTokenizer over text whose words are already separated by single spaces (normalize first with PunctuationNormalizer + SpaceNormalizer) and the only grams that can contain a space are the ones straddling a word or punctuation boundary, e.g. "search" and "engine" yield the cross-boundary gram "h e" for N=3. Dropping them leaves just the intra-word grams, so a document is indexed under its real word content instead of boundary noise: BM25 document lengths reflect the words present, and a query no longer matches on a gram that merely spans two adjacent words. It is language-independent (any whitespace, any script) and a no-op for scripts written without spaces between words (e.g. CJK, Thai), so it is safe to leave in a multilingual pipeline.

func (WhitespaceFilter) Filter(tokens []Token) []Token

type WhitespaceTokenizer ¶

type WhitespaceTokenizer struct{}

WhitespaceTokenizer splits text on runs of Unicode whitespace, leaving every other rune—including punctuation—attached to its token (so "word." stays "word."). Pair it with a PunctuationFilter when that is not what you want.

func (WhitespaceTokenizer) Tokenize(text string) []Token

type WidthNormalizer ¶

type WidthNormalizer struct{}

WidthNormalizer folds full-width and half-width characters to their normal width with Unicode width folding, so the same word written at either width indexes and queries identically — a frequent need for CJK input. Full-width Latin letters, digits, and punctuation collapse to ASCII ("ＴＯＫＹＯ" → "TOKYO", "２０２４" → "2024", "！" → "!"), and half-width katakana expands to full-width ("ｶﾀｶﾅ" → "カタカナ"). It does not fold case, strip accents, or expand compatibility forms such as ligatures; chain LowercaseNormalizer and DiacriticNormalizer for those. Run it early — before PunctuationNormalizer — so width-folded punctuation is then treated as a word boundary. It is a no-op for text that already uses only normal-width characters, ordinary CJK ideographs ("東京") included.

func (WidthNormalizer) Normalize(text string) string