Documentation
¶
Overview ¶
Package chunk splits long memory content into overlapping segments for embedding, so a memory stays searchable past the single-vector budget.
A memory's vector covers only the first EmbedMaxItemChars runes of its content (see internal/embed.Batched): the text beyond that is stored and returned whole but cannot be matched by vector recall. Splitting the content and embedding each piece removes that ceiling instead of moving it.
The rules this package encodes:
- Short content produces NO chunks. A single chunk of a short memory would be a duplicate of its own document vector — a wasted embedder call, a wasted row, and a duplicate hit to merge away. Chunk rows therefore exist only for the memories that are actually being truncated today.
- Cuts prefer a semantic boundary (paragraph, then line, then sentence, then word) over an exact size. memini's corpus is conversational captures and distilled facts, where a fact routinely fits in one sentence, so cutting mid-sentence destroys the very unit being retrieved.
- Chunks overlap, so a fact straddling a boundary survives whole in one of them.
- Everything is counted in runes, matching every other length bound in the codebase.
Pure and deterministic: no I/O, no clock, no config import.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IsWhitespaceOnly ¶
IsWhitespaceOnly reports whether s has no non-space rune. Callers skip embedding such a chunk: it carries no signal and would cost a vector.
Types ¶
type Config ¶
type Config struct {
// Size is the maximum runes in one chunk.
Size int
// Overlap is how many runes of the previous chunk each chunk repeats. It
// must be < Size; Split halves it if it is not, rather than failing to make
// progress.
Overlap int
// MinContent is the content length at or below which Split returns nothing.
// Content this short is already covered whole by its document vector.
MinContent int
// MaxChunks caps the chunks produced from one memory. Past it Split stops
// and reports the truncation through Result.Truncated rather than silently
// covering a prefix — an honest, observable ceiling replacing the silent one.
MaxChunks int
}
Config bounds a split. The zero value is not usable; see DefaultConfig.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig is the built-in split. Sized so a chunk fits any plausible embedder: 1200 runes is roughly 300 tokens, well under the 512-token window of the small local models (BGE, e5) that MEMINI_EMBED_BASE_URL is often pointed at, and far under text-embedding-3-small's 8191. It is also well under the default EmbedMaxItemChars (8000), so a chunk can never itself be truncated — the bug this package exists to fix cannot recur inside the fix.
The 200-rune overlap is one to two sentences: enough that a fact split across a boundary appears whole in the following chunk.
type Result ¶
type Result struct {
// Chunks are the segments to embed, in order. Empty when the content is at
// or under MinContent.
Chunks []string
// Truncated reports that MaxChunks stopped the split before the end of the
// content, so the tail is unchunked and stays unsearchable by chunk recall.
Truncated bool
}
Result is a split's output.
Source Files