Documentation
¶
Overview ¶
Package memindex is the index half of nova-memory: membership in a markdown corpus as a LOOKUP, never a scan.
WHY THIS EXISTS. A mind that keeps its memory as markdown answers "do I already know this?" by re-reading everything it is. Consolidating n new learnings against m existing memories is O(n*m), and m grows every day, so a fixed daily budget buys a shrinking n — and the failure is silent: the self just learns less while every step still looks like working. The requirement this package serves is that the mind's budget per new learning be k receipts, k constant: the index narrows m to k, the mind judges only the survivors, and NOTHING here ever writes the corpus.
THREE DECISIONS A READER SHOULD KNOW, each measured before it was made:
THE INDEX IS REBUILT IN MEMORY EVERY RUN. A full build over the corpus it was written for (548 markdown files, 11,823 paragraph chunks, ~11.5MB) measured 830ms on 2026-08-10 — so there is no persisted index, no staleness surface, and nothing that can drift from the tree. The repo is the store; this is a derivation that stops existing when the process exits. The escalation, if a build ever exceeds a couple of seconds: persist keyed by the tree's commit, then a real FTS engine. Named, not built.
THE NORMALIZER RUNS BEFORE CHUNK TEXT IS STORED, because four grep failure modes were measured on that corpus (2026-07-31 / 2026-08-02): line wraps split phrases, emphasis markers split them, blockquote markers survive a naive unwrap, and function-word variation defeats phrase greps. Strip blockquote and emphasis characters FIRST, then collapse whitespace, then casefold. The order is the fix — collapsing first leaves the blockquote marker sitting inside the recovered phrase.
EVERY RESULT ORDERING IS TOTAL. Go randomizes map iteration, and a hand-rolled retriever that iterates maps while scoring is nondeterministic by default. Ties break by chunk id after score, then by (path, para) after fusion; postings are sorted by chunk id; and the determinism test builds twice and requires identical bytes.
This package does no I/O beyond fs.FS reads handed in by the caller, so every judgment is reachable by a test with no tempdir.
Index ¶
Constants ¶
const MinTerms = 3
MinTerms is the floor below which a paragraph is not worth indexing: one or two tokens is a heading fragment or a separator, and indexing them makes every rare-word query drown in stubs.
const SchemaVersion = "nova-memory/1"
SchemaVersion names the token space: tokenizer + normalizer + chunking rules together. It is printed by stats and would key any future persisted index, so a schema change can never silently mix token spaces — preprocessing drift is a failure that corrupts quietly.
Variables ¶
This section is empty.
Functions ¶
func Normalize ¶
Normalize strips blockquote markers and markdown emphasis BEFORE collapsing whitespace, then casefolds. See decision 2 in the package comment: the order is the whole fix.
func Tokenize ¶
Tokenize splits normalized text into terms: alphanumeric-or-apostrophe runs, length >= 2 after trimming apostrophes. No stemming and no stopwords: both were measured unnecessary on the corpus this was built for, and both are levers the eval harness can legitimize later on yours — never taste.
func Truncate ¶
Truncate cuts s to at most n bytes on a rune boundary, appending an ellipsis when it cut. Receipts are quoted inside a machine-scannable line, and a mid-rune cut would put invalid UTF-8 into that line — the source this was ported from sliced bytes directly, which is a latent defect on any corpus holding a non-ASCII character near the cut.
Types ¶
type BM25 ¶
BM25 scores candidate chunks reached through posting lists — the query touches only its own terms' postings, never the whole corpus, which is the lookup-not-scan property expressed as a data structure.
type Channel ¶
Channel is the seam. This version ships bm25 and trigram; a semantic channel (flat exact cosine over local embeddings) would implement the same interface and nothing upstream would change. A channel's Query MUST be deterministic: ties break by chunk id.
type Chunk ¶
type Chunk struct {
File string // slash path relative to the corpus root
Para int // ordinal among the file's indexable paragraphs
Class string // top-level directory, "." for root files — the corpus classifies itself
Text string
Terms map[string]int
Len int
// Frontmatter, carried as receipt metadata when the file has any. The
// tool surfaces whatever the corpus already writes; it invents nothing
// and requires nothing.
FMName string
FMType string
}
Chunk is one indexed paragraph. Text is NORMALIZED (see Normalize); the raw bytes are not kept — receipts quote normalized text, and the file:para anchor is the stable address a judge needs to go read the original.
type Corpus ¶
type Corpus struct {
Chunks []Chunk
DF map[string]int
Post map[string][]int32 // term -> chunk ids, ascending
AvgLen float64
Files []string
Bytes int64
ByClass map[string]int // class -> chunk count
}
Corpus is the whole derived index. Build order is deterministic: files sorted by path, chunks in file order.
func Build ¶
Build derives the whole index from a filesystem of markdown. It refuses an empty corpus loudly: a query engine over nothing answers every membership question "no", which is the exact confident-zero failure this tool exists to remove.
exclude, when non-nil, is asked about every directory and every markdown file by slash path relative to the root. It is the caller's scope, stated per run; nothing is excluded by default except .git, which is never a corpus.
type FileHit ¶
type FileHit struct {
File string
Class string
FMName string
FMType string
Para int
Snippet string
Fused float64
// Native is the chunk's score in NativeChan, and NativeChan is the first
// NAMED channel that actually surfaced it — not unconditionally the first
// channel named. In a multi-channel run a chunk can reach the fused top-k
// through the second channel alone (bm25 scored it zero, or it fell off
// bm25's deep cutoff on a large corpus); recording only channel 0's score
// printed 0.00 for those, and a reader comparing 0.00 against the
// calibration band concludes the hit is weaker than unrelated control
// text when the number is an artifact of which channel surfaced it.
//
// NativeChan is "" only when no channel scored the chunk, which cannot
// happen for a chunk Retrieve returns; receipts print "-" for both fields
// rather than a fabricated zero if it ever does.
Native float64
NativeChan string
}
FileHit is what a judge sees: the best chunk of a file, with the file's class and frontmatter as receipt metadata, the channel-native score of that chunk for calibration, and the fused score that ordered it.
type Finding ¶
Finding is one verify observation. Kind is one of "coverage", "backlink", "frontmatter", "wikilink".
func Coverage ¶
Coverage checks one A:B pair — every file matching glob A must be named (by stem) in at least one file matching glob B, and every relative .md link inside the B files must point at a file that exists. That is the generic form of "every memory file has an index line, and every index line points at a real file"; the globs carry the layout, so the tool assumes none.
An empty side is an error, never a pass: a coverage check whose A side matched nothing has not verified anything.
func FrontmatterPresent ¶
FrontmatterPresent reports files matching the glob whose frontmatter is missing a name:. exempt holds basename PREFIXES the caller declares are listings rather than entries — an index page is not an entry and would otherwise fire forever, and a gate that fires forever on known-good files trains wave-through.
Nothing is exempt by default. The tool this was ported from hardcoded one filename prefix from its own corpus, which is a guess about someone else's layout; here the prefix is the caller's, stated per run, the same posture nova-self-talk's --skip took as the condition of its own promotion.
func Wikilinks ¶
Wikilinks reports every [[stem]] in the corpus that resolves to neither a file stem nor a frontmatter name. The aliased form [[stem|shown text]] and the heading form [[stem#section]] are scanned by their target half — a link whose alias is what the reader sees is still a link, and skipping the two commonest shapes made --links=gate a wall with a hole in it. Both resolution targets matter: a corpus that names files after their frontmatter slug has nothing enforcing it, and a link that reaches the content either way is not broken.
Whether these findings gate is the caller's statement, never a default here: some corpora hold links open on purpose — a [[name]] that matches nothing yet marks something worth writing — and a gate at a high false-positive rate trains a reader to wave findings through.
type Scored ¶
Scored is one channel's opinion of one chunk. Rank is 0-based within the channel's own ordering; Score is channel-native (BM25 is unbounded, Jaccard is [0,1]) and is NEVER compared across channels — fusion is rank-only, because weighted raw-score fusion of an unbounded scale against a bounded one is brittle and query-dependent.
type Trigram ¶
type Trigram struct {
C *Corpus
// contains filtered or unexported fields
}
Trigram ranks by character-3-gram Jaccard similarity between the query and each chunk. It buys robustness to morphology and small rewording that word-token BM25 misses, at the cost of a bounded per-chunk scan — which costs the CPU, not the mind, and the requirement bounds the mind's budget. It is never on unless the caller names it, and the caller should name it only once eval says it helps: on the corpus this was built for, eval measured bm25+trigram WORSE than bm25 alone, which is why it is a channel and not a default.
Source Files