Documentation
¶
Overview ¶
Package memindex is the index half of nova-memory: membership in a markdown corpus as a LOOKUP, never a scan.
WHY THIS EXISTS. A mind that keeps its memory as markdown answers "do I already know this?" by re-reading everything it is. Consolidating n new learnings against m existing memories is O(n*m), and m grows every day, so a fixed daily budget buys a shrinking n — and the failure is silent: the self just learns less while every step still looks like working. The requirement this package serves is that the mind's budget per new learning be k receipts, k constant: the index narrows m to k, the mind judges only the survivors, and NOTHING here ever writes the corpus.
THREE DECISIONS A READER SHOULD KNOW, each measured before it was made:
THE INDEX IS REBUILT IN MEMORY EVERY RUN. A full build over the corpus it was written for (548 markdown files, 11,823 paragraph chunks, ~11.5MB) measured 830ms on 2026-08-10 — so there is no persisted index, no staleness surface, and nothing that can drift from the tree. The repo is the store; this is a derivation that stops existing when the process exits. The escalation, if a build ever exceeds a couple of seconds: persist keyed by the tree's commit, then a real FTS engine. Named, not built.
THE NORMALIZER RUNS BEFORE CHUNK TEXT IS STORED, because four grep failure modes were measured on that corpus (2026-07-31 / 2026-08-02): line wraps split phrases, emphasis markers split them, blockquote markers survive a naive unwrap, and function-word variation defeats phrase greps. Strip blockquote and emphasis characters FIRST, then collapse whitespace, then casefold. The order is the fix — collapsing first leaves the blockquote marker sitting inside the recovered phrase.
EVERY RESULT ORDERING IS TOTAL. Go randomizes map iteration, and a hand-rolled retriever that iterates maps while scoring is nondeterministic by default. Ties break by chunk id after score, then by (path, para) after fusion; postings are sorted by chunk id; and the determinism test builds twice and requires identical bytes.
This package does no I/O beyond fs.FS reads handed in by the caller, so every judgment is reachable by a test with no tempdir.
Index ¶
- Constants
- func Excluding(fsys fs.FS, exclude func(string) bool) fs.FS
- func Normalize(s string) string
- func NormalizeNewlines(s string) string
- func Tokenize(s string) []string
- func Truncate(s string, n int) string
- type BM25
- type Channel
- type Chunk
- type Corpus
- type FileHit
- type Finding
- type Scored
- type Trigram
Constants ¶
const ( // MaxFileBytes caps one markdown file, read through readCapped. MaxFileBytes = 8 << 20 // MaxCorpusBytes caps the running byte total of one Build. MaxCorpusBytes = 64 << 20 // MaxVocabulary caps the distinct terms of one Build (len of Corpus.DF). MaxVocabulary = 1_000_000 )
The read bounds. Nothing bounded the corpus read: one planted 40 MB file drove a stats build to 2.06 GB peak RSS (4.8M distinct tokens) and a trigram search to 3.17 GB, on every indexing verb. The corpus this was written for is ~11.5MB over 548 files, so each cap leaves it wide headroom and refuses only what no memory corpus holds (security#76 finding 3).
const MinTerms = 3
MinTerms is the floor below which a paragraph is not worth indexing: one or two tokens is a heading fragment or a separator, and indexing them makes every rare-word query drown in stubs.
const SchemaVersion = "nova-memory/2"
SchemaVersion names the token scheme: tokenizer + normalizer + chunking rules together. It is printed by stats and would key any future persisted index, so a schema change can never silently mix token schemes — preprocessing drift is a failure that corrupts quietly.
Variables ¶
This section is empty.
Functions ¶
func Excluding ¶ added in v1.2.3
Excluding presents a filesystem with .git and caller-excluded paths hidden. Globs, reads and link resolution all see the same tree. Excluding an ancestor hides its descendants, including when a caller names one directly.
func Normalize ¶
Normalize strips blockquote markers and markdown emphasis BEFORE collapsing whitespace, then casefolds. See decision 2 in the package comment: the order is the whole fix.
func NormalizeNewlines ¶ added in v0.8.1
NormalizeNewlines folds CRLF and lone-CR line endings to LF. EVERY parser in this package that looks for "\n" or "\n\n" runs it first, and so must any caller that splits input text the same way — nova-memory's check verb does.
The lone-CR case is here because a fix that only replaces "\r\n" leaves a classic-Mac-style or exporter-written file exactly as broken as an unfixed CRLF one was: the blank-line split never fires, the whole document indexes as one chunk, and its valid frontmatter reports as missing — silently, with a green STATS line. Cross-platform compilation is not cross-platform text behavior.
func Tokenize ¶
Tokenize splits normalized text into terms: alphanumeric-or-apostrophe runs, length >= 2 after trimming apostrophes. No stemming and no stopwords: both were measured unnecessary on the corpus this was built for, and both are levers the eval harness can legitimize later on yours — never taste.
Types ¶
type BM25 ¶
BM25 scores candidate chunks reached through posting lists — the query touches only its own terms' postings, never the whole corpus, which is the lookup-not-scan property expressed as a data structure.
type Channel ¶
Channel is the seam. This version ships bm25 and trigram; a semantic channel (flat exact cosine over local embeddings) would implement the same interface and nothing upstream would change. A channel's Query MUST be deterministic: ties break by chunk id.
type Chunk ¶
type Chunk struct {
File string // slash path relative to the corpus root
Line int // first source line, 1-based
Original string // source paragraph with normalized line endings
Para int // ordinal among the file's indexable paragraphs
Class string // top-level directory, "." for root files — the corpus classifies itself
// Root is the root directory this chunk was indexed from, as the caller
// named it. It is empty for a single-root Build and set by Merge, so a
// receipt spanning several roots can name which one each hit came from.
Root string
Text string
Terms map[string]int
Len int
// Frontmatter, carried as receipt metadata when the file has any. The
// tool surfaces whatever the corpus already writes; it invents nothing
// and requires nothing.
FMName string
FMType string
}
Chunk is one indexed paragraph. Text stays normalized for retrieval; Original and Line locate and quote the source without changing paragraph IDs.
type Corpus ¶
type Corpus struct {
Chunks []Chunk
DF map[string]int
Post map[string][]int32 // term -> chunk ids, ascending
AvgLen float64
Files []string
Bytes int64
ByClass map[string]int // class -> chunk count
}
Corpus is the whole derived index. Build order is deterministic: files sorted by path, chunks in file order.
func Build ¶
Build derives the whole index from a filesystem of markdown. It refuses an empty corpus loudly: a query engine over nothing answers every membership question "no", which is the exact confident-zero failure this tool exists to remove.
exclude, when non-nil, is asked about every directory and every markdown file by slash path relative to the root. It is the caller's scope, stated per run; nothing is excluded by default except .git, which is never a corpus.
func Merge ¶ added in v1.2.3
Merge combines corpuses built over separate roots into one, so a single ranking spans them. Each chunk remembers the root it came from (Root, set from the matching roots entry), so a receipt can name it; within a root the file paths stay root-relative, which is what keeps the class (a top-level directory) meaningful across roots.
The roots slices parallel parts: parts[i] was built over roots[i]. Two roots may hold the same relative path, and two such chunks stay distinct — retrieval keys files by (root, path), never path alone.
type FileHit ¶
type FileHit struct {
File string
Root string // the root this file came from, empty for a single-root build
Class string
FMName string
FMType string
Para int
Line int
Snippet string
// Whole is the hit chunk's paragraph, untruncated. Snippet cuts the same
// paragraph at 120 bytes, so the words a reader searched for can sit just
// past the cut; --whole prints this instead, and the caller bounds it.
Whole string
Fused float64
// Native is the chunk's score in NativeChan, and NativeChan is the first
// NAMED channel that actually surfaced it — not unconditionally the first
// channel named. In a multi-channel run a chunk can reach the fused top-k
// through the second channel alone (bm25 scored it zero, or it fell off
// bm25's deep cutoff on a large corpus); recording only channel 0's score
// printed 0.00 for those, and a reader comparing 0.00 against the
// calibration band concludes the hit is weaker than unrelated control
// text when the number is an artifact of which channel surfaced it.
//
// NativeChan is "" only when no channel scored the chunk, which cannot
// happen for a chunk Retrieve returns; receipts print "-" for both fields
// rather than a fabricated zero if it ever does.
Native float64
NativeChan string
}
FileHit is what a judge sees: the best chunk of a file, with the file's class and frontmatter as receipt metadata, the channel-native score of that chunk for calibration, and the fused score that ordered it.
func Retrieve ¶
Retrieve runs every channel, fuses by reciprocal rank, aggregates per file by best fused chunk, and returns the top k files. With one channel this reduces to that channel's own ranking (order-preserving), so single-channel output is exactly that channel's opinion.
THE UNIT OF THE LIMIT IS FILES, NOT CHUNKS. A fixed per-channel chunk headroom is not headroom at all when one document owns more matching paragraphs than the cap: a 100-paragraph file filled the first 50 slots and every other matching file was truncated away BEFORE aggregation, so a request for the top 2 files returned 1, with nothing on the receipt saying so. Raising the multiplier only moves the fixture that breaks it. So the depth DOUBLES until k distinct files are in hand, the channels have no more candidates to give, or the depth covers the corpus. Deepening is monotone — a chunk's rank within a channel does not change when more chunks are asked for, so the fused scores already computed are the same ones — and the progression is fixed, so results stay deterministic.
type Finding ¶
Finding is one verify observation. Kind is one of "coverage", "backlink", "frontmatter", "wikilink".
func Coverage ¶
Coverage checks one A:B pair — every file matching glob A must be named in at least one file matching glob B, and every relative .md link inside the B files must point at a file that exists. That is the generic form of "every memory file has an index line, and every index line points at a real file"; the globs carry the layout, so the tool assumes none.
"Named" means one of two things, and both are bounded: a relative .md link in a B file that RESOLVES to the A file, or the A file's stem appearing in B as a whole identifier. Substring membership is not membership.
An empty side is an error, never a pass: a coverage check whose A side matched nothing has not verified anything.
func FrontmatterPresent ¶
FrontmatterPresent reports files matching the glob whose frontmatter is missing a name:. exempt holds basename PREFIXES the caller declares are listings rather than entries — an index page is not an entry and would otherwise fire forever, and a gate that fires forever on known-good files trains wave-through.
Nothing is exempt by default: a built-in prefix would be a guess about someone else's layout, so the prefix is the caller's, stated per run.
func Wikilinks ¶
Wikilinks reports every [[stem]] in the corpus that resolves to neither a file stem nor a frontmatter name. The aliased form [[stem|shown text]] and the heading form [[stem#section]] are scanned by their target half — a link whose alias is what the reader sees is still a link, and skipping the two commonest shapes made --links=gate a wall with a hole in it. Both resolution targets matter: a corpus that names files after their frontmatter slug has nothing enforcing it, and a link that reaches the content either way is not broken.
Whether these findings gate is the caller's statement, never a default here: some corpora hold links open on purpose — a [[name]] that matches nothing yet marks something worth writing — and a gate at a high false-positive rate trains a reader to wave findings through.
type Scored ¶
Scored is one channel's opinion of one chunk. Rank is 0-based within the channel's own ordering; Score is channel-native (BM25 is unbounded, Jaccard is [0,1]) and is NEVER compared across channels — fusion is rank-only, because weighted raw-score fusion of an unbounded scale against a bounded one is brittle and query-dependent.
type Trigram ¶
type Trigram struct {
C *Corpus
// contains filtered or unexported fields
}
Trigram ranks by character-3-gram Jaccard similarity between the query and each chunk. It buys robustness to morphology and small rewording that word-token BM25 misses, at the cost of a bounded per-chunk scan — which costs the CPU, not the mind, and the requirement bounds the mind's budget. It is never on unless the caller names it, and the caller should name it only once eval says it helps: on the corpus this was built for, eval measured bm25+trigram WORSE than bm25 alone, which is why it is a channel and not a default.
Source Files