bench

memini benchmark harness

A retrieval benchmark: ingest a dataset of memories, then for each question measure how well a system retrieves the gold supporting memories.

The Go eval harnesses in this directory (*_test.go) live behind the bench build tag, so they stay out of the default go test ./... (they need a live embedder and take minutes). Run them with -tags bench, as shown below; a plain go test ./bench/ reports "no test files" by design.

mise run bench                 # offline sample, local embedder
go run ./cmd/bench -k 5        # same, explicit K

Against a real embeddings model and a real dataset:

export MEMINI_EMBED_BASE_URL=http://localhost:8081/v1
export MEMINI_EMBED_MODEL=bge-m3 MEMINI_EMBED_DIMS=1024
# Optional: instruction-tuned asymmetric embedders (Qwen3-Embedding, bge) score
# higher when queries carry a retrieval instruction; documents stay bare.
# Measured on Qwen3-Embedding-8B: +6.0pp R@5 on the LongMemEval vector leg
# (91.2% -> 97.2%), +1.0pp MRR on the fused ranking on both datasets.
export MEMINI_EMBED_QUERY_PREFIX=$'Instruct: Given a user query, retrieve relevant memories that answer it\nQuery:'
go run ./cmd/bench -suite longmemeval -data ./longmemeval_s.json -k 5
go run ./cmd/bench -suite locomo      -data ./locomo.json        -k 5

# Isolate the recency-aware re-ranker against pure RRF on the same candidates,
# using each question's date as "now" (needs a timestamped dataset):
go run ./cmd/bench -suite longmemeval -data ./longmemeval_s.json -rerank -k 5

Full results

Everything this harness measures, in one table — sourced from the committed results/ JSON, all on the same all-MiniLM-L6-v2 (384-d) endpoint. Cells are recall_any@5 / @10 / MRR (%); p50 is in-process recall latency (rerank rows show the added cost). The detailed per-dataset sections below explain the methodology, sweeps, and caveats behind each column.

Questions per dataset: LongMemEval 500 (session granularity), LoCoMo turn-level 1,982 (gold = exact evidence turns), LoCoMo session-level 1,981 (gold = sessions holding those turns). Rerank backends: Qwen3-Reranker-0.6B (cross-encoder) and Qwen3.5-9B (LLM). Reproduce with the per-suite commands in the sections below (-suite longmemeval, locomo, locomo-sessions; add -rerank-url/-llm-rerank for the rerank rows).

Reading it: hybrid never trails either single leg on the saturated session sets (it ties keyword on LoCoMo-session, where keyword's exact-token match is already near-ceiling). On turn-level LoCoMo base recall has real headroom, so the rerank tier earns its keep — the cross-encoder lands +11pp R@5 / +17pp MRR over hybrid at a fraction of the LLM's latency, and the LLM adds a few more points (+15pp / +25pp) if you already run a chat model. Where recall is already at ceiling (both session sets), reranking is a measured no-op.

Results: memini vs other memory systems

All memini numbers below are measured by this harness against a live all-MiniLM-L6-v2 (384-d) endpoint — the same embedding model agentmemory benchmarks with. Competitor numbers are cited from their own publications — we cannot re-run their systems here, and they use different embedding models, readers, and judges. Treat cross-system rows as directional, not a controlled head-to-head. (This mirrors how agentmemory documents its comparison.)

LongMemEval-S — retrieval recall_any@K

Full 500-question LongMemEval-S (~48 sessions/question), same metric agentmemory reports: does any gold session appear in the top-K retrieved? No LLM in the loop — pure retrieval. The run is the full 500 questions with the identical embedding model agentmemory benchmarks with (all-MiniLM-L6-v2, 384-d) for a true apples-to-apples comparison.

Hybrid recall over-fetches a deep candidate pool per leg (max(k*5, 50)) before fusing, so a memory just outside the top-k of both legs can still win — the production Recall path does the same. Fusion is convex score fusion (alpha 0.5, the baked default): each leg's scores are min-max normalized to [0,1] and combined 0.5·vector + 0.5·keyword, keeping score magnitude so a memory a leg ranks far above its runners-up dominates one that is merely middling in both. A negative alpha falls back to Reciprocal Rank Fusion; deep pools then need a steep decay (rrfK=5, not the classic 60), since a flat decay lets both-leg mediocrity outscore single-leg excellence (2/(60+20) > 1/(60+0)). Score fusion gets the same effect from score magnitude directly, and beat RRF on 3 of 4 model×dataset cells (and on MRR in all 4).

On the same model/dataset/metric (full 500 questions), memini hybrid beats agentmemory at R@5 (98.4% vs 95.2%), R@10 (99.4% vs 98.6%), and MRR (92.3% vs 88.2%). memini's keyword leg is +11.4pp over agentmemory's BM25-only (97.6% vs 86.2%) thanks to Porter stemming, and hybrid fusion now beats either leg alone. Relative to fetching only k per leg with the classic rrfK=60, the deep-pool + score fusion is worth +2.0pp R@5 / +1.0pp R@10.

LoCoMo — retrieval recall_any@K

LoCoMo retrieval at dialogue-turn granularity (1,982 questions over 10 long conversations, gold = exact evidence turns among ~590 turns/conversation) — a much harder target than LongMemEval's session granularity, and the regime where flat-decay RRF over deep pools degrades badly.

No published turn-level retrieval baselines exist to compare against (mem0 / Letta report LLM-judged QA accuracy, below). This is the one cell where the default score fusion is edged by RRF (60.1% / 71.0%): when the vector leg is near-noise (MiniLM scores only 41.5% here), giving it an equal-weight normalized vote hurts, whereas RRF's rank-only vote is more robust. Score fusion still wins this cell on MRR and wins outright on every cell with a stronger embedder — so it is the baked default; the benchmark harness can still select RRF (a negative fusion alpha via its flag) for weak-vector experiments. (Ablation: rrfK=60 over the same deep pools scored just 52.8% R@5, below the keyword leg alone — both score fusion and rrfK=5 fix that.)

Pool-depth robustness (-pool-factor / -pool-floor)

Min-max normalization could in principle be fragile to pool depth (the score at the bottom of the pool sets each leg's zero point), so score fusion was swept at per-leg depths 30 / 50 / 80 on both datasets and both embedders (hybrid R@5 / R@10 / MRR):

Quality moves at most ±0.6pp R@5 across a 2.7× depth range — no tail collapse — with the two datasets drifting in opposite directions (deeper pools help session-granularity LongMemEval slightly and hurt turn-granularity LoCoMo slightly), so the default max(k*5, 50) sits at the crossover.

Recency-aware re-ranking (-rerank)

memini re-ranks the fused candidates by a composite of relevance, recency, and importance. The recency weight is deliberately light (0.05): a sweep on LongMemEval-S (knowledge-update + temporal-reasoning, q.Now = question date, sessions timestamped from haystack_dates) shows recency is a net win only as a tie-breaker, and actively harmful when over-weighted.

At 0.05 the re-ranker is +0.5pp R@1 / +0.4pp MRR over pure RRF with no knowledge-update cost, and recall@5 is identical across all weights (the re-rank only reorders within the top results). The steep RRF decay made the composite far more robust to the recency weight than the flat rrfK=60 decay was (where 0.15+ buried correct-but-older memories); the default stays at the conservative 0.05 since the gains beyond it are within noise.

Temporal targeting (temporal0.40)

Recency weighting trades off against itself: raising it helps temporal-reasoning (78.2→81.2% R@1) but hurts knowledge-update (91.0→87.2%), whose answers aren't necessarily recent. Temporal targeting avoids that: when a query names a relative time ("three weeks ago"), it computes target = now − offset and boosts candidates dated near that point, not near now. It only fires on temporal queries, so other categories are unaffected.

Temporal targeting is +1.9pp R@1 overall over the recency default and beats even the heaviest recency weight on temporal-reasoning without the knowledge-update regression — so it ships on in production (temporal boost 0.40, the baked default). The no-LLM regex extractor only catches templated phrasing; an LLM anchor extractor (plugging into the same search.AnchorExtractor interface) can resolve looser references and is the intended with-LLM tier.

Held-out split (-holdout)

To avoid overfitting tuning decisions to the full benchmark, -holdout splits LongMemEval deterministically by load order: every 10th question is held (50/500), the rest are tune (450/500). Sweep parameters on -holdout tune, then report the final number on -holdout held (unseen). Default all runs the full set. Results files are suffixed (longmemeval-held.json) so splits don't overwrite each other.

Measured (memini-hybrid, all-MiniLM-L6-v2 — the parameters were swept on tune, not held):

The held split does not regress against tune, so the tuning choices generalize (no tuned-to-test inflation). Per-category R@5:

Read the per-category numbers off tune (450 questions); it shows the real headroom is single-session-preference (88.9% R@5). On held each category is only 2–13 questions, so its across-the-board 100% is small-sample, not a separate claim of perfection.

Session-doc construction (-session-doc)

LongMemEval sessions are embedded as one document per session; -session-doc controls what text that document contains, to measure the vector leg's sensitivity to document shape:

• full (default) — "role: content" for every turn.
• user-only — only the user turns, no role prefixes. Assistant turns dilute the embedding for user-question recall; this is the shape MemPalace reports 96.6% R@5 vector-only with on the same MiniLM model.
• dated — full prefixed with the session date, giving temporal questions a textual anchor embeddings would otherwise ignore.

Compare the vector row's recall_any@5 across modes (cached embeddings make the sweep cheap); the keyword and hybrid rows shift too but the vector leg is the target.

memini hybrid per-category (all-MiniLM, recall_any@10): multi-session 100%, knowledge-update 100%, single-session-user 98.6%, single-session-assistant 98.2%, temporal-reasoning 97.0%, single-session-preference 96.7%.

Rerank tier — cross-encoder vs LLM (-rerank-url / -llm-rerank)

The read-side rerank reorders the top of the production candidate order. The bench drives either backend through the same comparison (one reranker call per question — use -limit):

# cross-encoder (fast; e.g. Qwen3-Reranker-0.6B via llama-server --rerank):
go run ./cmd/bench -suite locomo -data ./locomo.json -rerank-url http://localhost:8002/v1 -rerank-model qwen3-reranker-0.6b -limit 100 -k 5,10
# LLM reranker (slow; MEMINI_LLM_*):
go run ./cmd/bench -suite locomo -data ./locomo.json -llm-rerank -limit 100 -k 5,10

Measured on all-MiniLM-L6-v2 (cross-encoder = Qwen3-Reranker-0.6B, LLM = Qwen3.5-9B), recall_any@5 / @10 / MRR:

Reranking is a no-op at recall ceiling (session-level) and a big win where recall has headroom (turn-level: +11pp R@5 / +17pp MRR for the cross-encoder, +15pp / +25pp for the LLM). The cross-encoder captures most of the LLM's lift at a fraction of the latency with no chat model — the recommended production rerank (MEMINI_RERANK=<url>); the LLM tier (MEMINI_RERANK=llm) buys the last points if you already run one.

LoCoMo — end-to-end QA accuracy (LLM-judge)

The metric mem0/Letta publish: retrieve → generate an answer → an LLM judges it against the gold answer. memini's number uses a fast instruct reader+judge (Llama-3.3-70B-Instruct); the competitor numbers use their own readers/judges, so this is directional.

Sources: agentmemory COMPARISON.md/LONGMEMEVAL.md; LongMemEval (arXiv 2410.10813); LoCoMo (snap-stanford.github.io/LoCoMo); mem0.ai; letta.com.

LongMemEval-S — end-to-end QA accuracy (LLM-judge)

Full 500-question LongMemEval-S with the production answer path: hybrid retrieval (bge-m3, 1024-d) → chain-of-thought answer prompt → LLM judge. The answer prompt uses chronological context formatting, <mem_thinking> CoT reasoning tags, "most-recent-wins" for knowledge updates, temporal grounding (question date injected), and 1-hop linked-memory expansion. Competitor numbers use GPT-4o as answerer+judge; memini uses Qwen3.6-35b-fast — so this is directional, not a controlled head-to-head.

Per-category breakdown (memini, Qwen3.6-35b-fast, k=10, bge-m3):

The answer-path improvements (chronological context, CoT reasoning, temporal grounding, most-recent-wins, IncludeLinked, agentic tool loop) lifted overall accuracy from 57.6% to 82.8% on the same model and embedder — a +25.2pp gain from prompt engineering and tool-loop enablement alone.

Reproduce:

export MEMINI_EMBED_BASE_URL=https://your-embedder/v1
export MEMINI_EMBED_MODEL=bge-m3 MEMINI_EMBED_DIMS=1024
export MEMINI_LLM_BASE_URL=https://your-llm/v1
export MEMINI_LLM_MODEL=qwen3.6-35b-fast
go run ./cmd/qa -suite longmemeval -data bench/data/longmemeval_s.json \
  -ingest upsert -k 10 -workers 4

Note on comparability: mem0 uses GPT-4o (trillion-parameter) for both answering and judging. memini uses Qwen3.6-35b-fast (35B). The gap to mem0 (80.6% vs 94.4%) is primarily answerer LLM quality — mem0's own data shows a 12pp spread between GPT-5 (91.0%) and GPT-OSS-120B (89.8%) as extraction models, and the answerer matters even more. With a GPT-4o-class answerer, memini's retrieval (R@5=98.4%) and answer prompt are competitive.

Metrics

• Recall@K — fraction of questions whose gold memory appears in the top K.
• MRR — mean reciprocal rank of the first gold hit.
• p50/p95 — recall latency; ingest — total ingest time.

Output is a Markdown table (stdout) plus JSON under bench/results/.

What it compares today

Three memini retrieval strategies over the same ingested store, to show the value of hybrid fusion:

memini-hybrid should never score below either single strategy.

Datasets

• sample — committed at bench/data/sample.json, runs fully offline.
• Normalized schema (-suite file) — {name, items:[{id,content}], questions:[{query,gold:[id]}]}.
• LongMemEval / LoCoMo — loaders map the published JSON shapes to the normalized schema (each session/turn becomes an item; answer/evidence ids become gold). Download the datasets and pass -data.

Recall@K on LongMemEval/LoCoMo is easy to overfit — treat scores as directional.

Recall-quality scoreboard (quality_test.go)

The LongMemEval/LoCoMo suites ingest a single tier and mostly measure recall; the failures memini has actually shipped (spray) were precision failures — recall injecting irrelevant memories. TestRecallQualityScoreboard is the tier-mixed baseline both axes are judged against: durable-fact recall, episodic detail recall, and injection precision on one labeled corpus (near-duplicate durable/episodic pairs, same-template distractor durables, episodic-only topics, and confusable topics that lexically neighbour fact topics). Ranking changes are not done until this scoreboard moves in the right direction on ≥2 embedders.

go test -tags bench ./bench/ -run TestRecallQualityScoreboard -v
# add the cross-encoder rows:
MEMINI_RERANK_URL=http://127.0.0.1:8002/v1 MEMINI_RERANK_MODEL=qwen3-reranker-0.6b \
  go test -tags bench ./bench/ -run TestRecallQualityScoreboard -v

Pre-fix baseline (2026-07, additive quality composite, reserve gate ratio 0.6; reserve=0 reference in parentheses). spray = injected durables on queries with no relevant durable; inj = injected durables over all 40 queries:

What the baseline said: spray was clean, but all injection happened on low-signal detail questions — score anatomy showed durables at fused relevance 0.00 ranked #2–6 because the additive 0.2·quality bonus dwarfed the noise tail — and durable MRR sat at ~.20 on the small embedders because the reserve placed recovered facts at the window bottom. The cross-encoder additionally demotes terse "Decision: …" facts to the bottom of the window (qwen3 MRR .66→.20) while nudging P@3 up.

After the durable-ranking fix

Three coupled changes (2026-07): the composite's quality term is relevance-modulated (rel·(w_r + w_q·q̂) — a zero-relevance durable has nothing to amplify, so tier salience reorders comparable candidates instead of floating off-topic facts into weak windows; single-tier corpora like LongMemEval/LoCoMo have uniform quality and are provably order-invariant); the reserve gate ratio is re-expressed as 0.5 in the new score space (the same effective relevance bar the settled 0.6 imposed under the old composite, which carried a flat +0.2 durable floor); and the gate gains an absolute top-anchor leg (promotion also needs ≥0.4× the window's top hit — the leg that holds when the evictee is noise). Promoted durables now surface directly below the top hit instead of at the window bottom; the top hit is never displaced, so an episodic gold answer cannot be shadowed.

Reading it against the baseline:

• Injections −72 to −91% (50/51/54/45 → 5/9/14/4), and production now adds at most 1 over the reserve=0 reference — the reserve leak is closed. What remains is base relevance (e.g. a BM25 match on a shared token), not tier spam.
• Durable MRR on the small embedders .20 → .50 (promoted facts sit at rank 2, capped by the never-displace-the-top rule); qwen3 .66 → .75. Under the cross-encoder MRR stays ~.20: the reranker reorders the window and demotes terse facts — the known remaining defect, and it lives in the rerank tier, not the composite.
• Durable R@5 matches the pre-fix baseline exactly (100/100/100/70). bge-small's three missing facts are blocked by the evictee leg; ratio 0.4 recovers all three for +1 injection (measured) — a candidate loosening, deliberately not taken to avoid tuning on the eval corpus.
• Episodic recall and spray are untouched (100% / 1.0, 0).

The production window (k=3) and the rerank demotion

Every shipped turn-injection integration (hermes, openclaw, opencode, pi, openwebui) requests limit = recall_limit (default 3) and injects the entire response; the MCP tool defaults to k=10, delivered whole. Nobody truncates below k, so the cross-encoder demoting a promoted fact within the window reorders delivered memories rather than dropping any — a rerank pin/blend was scoped and dropped on that evidence. The prod k=3 scoreboard row measures the real consumer window: membership is identical to k=5 (durable R@3 100/100/100/70, both modes), worst-case rerank MRR is .333 (rank 3 of 3 delivered), and precision is slightly better (inj 4/9/11/4).

A gate-simplification probe (drop the evictee leg, keep only the 0.4 top anchor) was identical on this corpus and recovered bge-small to 100%, but leaked one spray injection on the tier-mix corpus (an off-topic durable at 0.38× the top of a flat chatter window) — the evictee-relative leg is load-bearing exactly there, so the gate keeps both legs.

Multi-hop diagnostics (multihop_test.go, entitydiag_test.go)

LoCoMo category-1 questions need all of ≥2 gold memories ("Where did Caroline move from 4 years ago?" = the moved 4 years ago turn + the home country, Sweden turn). TestMultiHopRetrievalCeiling measures whether that is a retrieval problem and what a second recall would buy (each 2nd-hop regime is the union of two k=10 result sets):

Entity edges don't bridge it (measured, phase stopped)

The planned fix was an associative edge: extract entities at write time (extract.Entities, capitalized spans, no LLM), index entity → memory ids, and let recall pull memories sharing entities with the top hits, relevance- gated like the durable reserve. TestEntityBridgeDiagnostic priced every step of that mechanism before any of it was wired, on two embedders:

• Extraction is precise: on hand-labeled samples, P 1.00 / R 1.00 over 30 LoCoMo turns, P 1.00 / R 0.75 on scoreboard-corpus templates (the misses are lowercase technical names like "pgvector" — out of scope for a no-POS heuristic). Pinned by TestEntitiesPrecisionRecall.
• But the entity graph has no bridging power: under a hub-safe document- frequency cap (≤10% of the conversation) only 1.3–2.6% of failed questions are fully entity-bridgeable. Without the cap it looks like 13%, but every extra link is a speaker-name hub (caroline = 119/419 memories) — expansion through a hub is spray by construction.
• A lowercase concept vocabulary (stemmed noun-chunk proxy for the "noun phrase" tier) links 42–43% of failures in principle — but simulating the actual mechanism (promote ≤3 linked pool candidates into the window under the reserve's two-leg gate) yields no gain or a net loss: 46–49 vs 48 covered on qwen3, 39–41 vs 44 on MiniLM, with ~95% of promotions being noise. The contradiction is structural: a true multi-hop bridge has low query relevance by definition, so any relevance gate strong enough to block spray also blocks the bridges; candidates that clear the gate were already at the window's edge.
• An entity-anchored 2nd hop (augment the query with the top hits' DF-capped terms — the union mechanism above, minus the full-content echo) reaches 19.1% / 22.4% (entities / concepts) on qwen3 and 17.3% / 20.6% on MiniLM — at or below the realistic content hop, never near the oracle.

Conclusion: on this benchmark the multi-hop gap is not an associative-edge problem a no-LLM entity index can close — 59% of missing golds are not even in the 50-deep fused pool, and what connects gold sets is either a hub person or generic lowercase concepts. Entity-aware recall was not wired into the service; extract.Entities stays as a validated building block (a later phase wants entity+attribute+value contradiction triggers, a precision-first use it fits). Re-run with:

go test -tags bench ./bench/ -run 'TestMultiHopRetrievalCeiling|TestEntityBridgeDiagnostic' -v
MEMINI_EMBED_MODEL=text-embedding-all-minilm-l6-v2-embedding MEMINI_EMBED_DIMS=384 \
  go test -tags bench ./bench/ -run 'TestMultiHopRetrievalCeiling|TestEntityBridgeDiagnostic' -v

Contradiction / update handling (contradiction_test.go)

On the no-LLM target a fact update loses to the fact it corrects, and both sit in the delivered window together. corroborateNearestAsync only grows confidence on a restatement; its contradiction mirror did not exist (the only supersede-on-contradiction was LLM-gated), and DurableScore has no recency term — so an entrenched old fact outranks a fresh contradicting write indefinitely. This phase adds a precision-first, LLM-free contradiction detector (internal/contradict, in the tradition of de Marneffe et al., ACL 2008: only surface-detectable value/polarity changes are in scope) and, on a confirmed contradiction, invalidates the stale fact the way Zep/Graphiti's temporal graph does — stamp valid_to, keep the row and its history, never delete.

Detector precision (36 authored quads — base / restatement / update / distinct, pure text, no embedder). The costly error is flagging a restatement as an update (it would downrank a live fact and lose its corroboration):

Similarity-gate routing (per embedder). The write path already pays a top-1 same-tier vector search; the detector runs on that neighbour, gated by the existing writeDedupScore (0.625). Restatement misfires are 0 at every floor on both embedders; at 0.625 distinct misfires are 0 too, and 0.625 sits below corroborateMinScore (0.70) as it must (an update diverges from its base more than a restatement):

Wild false positives on 500 real LongMemEval nearest-neighbour pairs (all distinct by construction, qwen3): 0.00% flagged as update at every floor (4 pairs even clear 0.625; none fire).

The harm, and why confidence alone can't fix it (TestContradictionStaleVsFresh, 12 topics; the old fact entrenched through the real corroboration path — mean effective confidence ~0.42, AccessCount ~4 — then contradicted by a fresh durable write, at the production config):

Confidence shrink only halves the harm: composite rank is 80% relevance / 20% quality (and fresh episodic chatter sets the quality normaliser), so zeroing the stale fact's confidence barely moves its score. Only removing it from the current-state window flips the order — hence the valid_to invalidation (reversible via Restore, still reachable via AsOf), which recall now honours in the default (non-AsOf) path.

Shipped mechanism. contradictNearestAsync (mirror of corroborateNearestAsync, internal/service/service.go) fires on fresh durable writes: top-1 durable neighbour ≥ 0.625, 24h cooldown, extract-anchored detector confirms a value/polarity change, then store.MarkContradicted stamps valid_to, shrinks confidence usage-aware (0.9·seed/usage, so the fresh write outranks it for any AccessCount), and records contradicted_by / contradicted_prev_confidence for audit and reversal. On by default; MEMINI_CONTRADICT_DOWNRANK=false is the kill-switch. Metric: memini_contradict_results_total.

No scoreboard regression — TestRecallQualityScoreboard, both embedders, composite + reranked, at k=5 and prod k=3: durable R@5/MRR, episodic 100%/1.000, spray 0, and injection counts are all identical to the pre-change baseline (the corpus carries no valid_to'd facts, so the new recall filter is inert there). Re-run with:

MEMINI_SWEEP_EMBEDDERS="http://127.0.0.1:8001/v1|text-embedding-qwen3-embedding-0.6b|1024,http://127.0.0.1:8001/v1|text-embedding-all-minilm-l6-v2-embedding|384" \
  go test -tags bench ./bench/ -run 'TestContradiction|TestRecallQualityScoreboard' -v
# add the wild-FP probe (needs the LongMemEval file):
MEMINI_LME_DATA=$PWD/bench/data/longmemeval_s_cleaned.json \
  go test -tags bench ./bench/ -run TestContradictionWildFalsePositives -v

External baselines

bench.System is the extension point. To compare against mem0, Zep/Graphiti, Letta, Cognee, agentmemory, or supermemory, implement System (Name / Ingest / Recall) over each service's API and add it to the run list in cmd/bench. These require the respective services/keys and are intentionally not vendored here.