Documentation
¶
Overview ¶
Package identtoken is the one place search text turns into tokens: Fields cuts free text into identifier-like terms, and Split cuts one identifier into its sub-tokens. It is a leaf utility shared by search indexing, query sanitizing, and both scorers, because two copies of a tokenizer is how a query and the index it searches stop reading text the same way.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Fields ¶ added in v0.13.0
Fields splits free text into identifier-like terms with their original case, so camelCase boundaries survive for sub-token splitting by Split. @intent expose original-case terms; lowercasing happens per consumer. @domainRule only letter, digit, and underscore sequences survive tokenization.
func FieldsLower ¶ added in v0.13.0
FieldsLower splits free text into lowercase identifier-like terms, which is how both indexed documents and the queries aimed at them are read. @intent read a document the same way the query is read.
func Split ¶
Split breaks an identifier into lowercased sub-tokens on separators, camelCase boundaries, and letter/digit transitions ("getUserById" -> get, user, by, id; "HTTPServer" -> http, server; "parseHTML5" -> parse, html, 5). @intent normalize source identifiers into stable search-index tokens without language-specific dependencies.
Types ¶
This section is empty.
Source Files