Documentation
¶
Overview ¶
Package segment tokenises memory text for lexical scoring, with first-class support for CJK (Chinese/Japanese) text.
The graph and file memory runtimes score candidate memories against the chat query by counting token overlaps. The original implementation used strings.Fields, which splits only on Unicode whitespace — CJK text has no inter-word spaces, so a whole Chinese sentence collapsed into a single giant token that never matched a memory body (lexical score 0 for any natural Chinese query). This package routes CJK runs through gse (a pure-Go jieba port with an embedded dictionary) and leaves Latin/numeric runs on strings.Fields, so English/mixed behaviour is unchanged.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func LexicalScore ¶
LexicalScore scores how well body covers the tokens of query, in [0, 1].
It mirrors the historical graphLexicalScore / fileRuntimeScore contract:
- empty query returns 1 (caller treats "no query" as a neutral match);
- if query is a substring of body, returns 1 (fast path, exact phrase);
- otherwise, the fraction of query tokens that appear as substrings of body.
The only behavioural change from the legacy implementation is that Tokens (rather than strings.Fields) defines what a "token" is, so Chinese sentences are split into words and can now match memory bodies.
func Tokens ¶
Tokens splits text into lowercase word tokens for lexical matching.
Latin/numeric runs are split on whitespace (matching the historical strings.Fields behaviour). CJK runs are segmented by gse in search-engine mode (HMM on) which emits both whole words and their sub-words for better recall. Punctuation, whitespace, and empty fragments are dropped.
When the gse dictionary failed to load, Tokens degrades to strings.Fields so search keeps working without CJK word boundaries.
Types ¶
This section is empty.
Source Files