memory-system

memory-system

Persistent, semantic long-term memory for AI agents, served over the Model Context Protocol.

What it is

memory-system is a self-hostable MCP server that gives AI agents a durable, searchable long-term memory across sessions. Documents are stored as markdown, split into sections, embedded, and made retrievable through hybrid search — vector similarity fused with lexical ranking — backed by PostgreSQL and the pgvector extension. An agent stores what it learns as a self-contained note and recalls it later by meaning, not just keyword.

It ships as a single Go binary (module github.com/eliminyro/memory-system) that serves MCP over HTTP. On store, each section is embedded before any database write, so a provider failure never leaves a half-written document; on search, vector candidates ordered by cosine distance are fused with lexical candidates ranked by Postgres full-text search. A per-doc_type staleness model and an optional near-duplicate guard help keep the corpus healthy; retiring outdated knowledge is agent-driven (supersede, update, or delete), not an automated sweep.

Every request passes through a Zanzibar-style, relationship-based authorization engine — a single decision point backed by relation tuples rather than hard-coded rules. Two independent authentication paths feed that engine: long-lived API keys (ideal for headless agents and CI) and OAuth 2.1 / OIDC (for humans signing in through an upstream identity provider). All data is tenant-scoped, with an optional shared "common" pool for knowledge every tenant can read; a caller's reads aggregate across their home tenant, the common pool, and any tenant they hold a direct grant on, while writes stay scoped to a single tenant.

Features

• Hybrid retrieval — pgvector HNSW cosine search fused with a GIN-indexed tsvector lexical search, scored as 1 - (embedding <=> query_vector).
• Snippet mode — opt-in snippet: true on search_memory returns a short match-centered window (MEMORY_SNIPPET_CHARS, default ~400) of each result instead of full content — cheap triage on search, then get_document for full text. A snippet_centered flag distinguishes a real lexical match from a leading-text fallback (purely-semantic hit). Verbatim (no LLM); withheld results are never snippet-expanded.
• Pluggable embedding providers — ollama, gcp (Vertex AI), openai (any OpenAI-compatible /embeddings endpoint: OpenAI, Azure, vLLM, LM Studio, LocalAI, HuggingFace TEI), aws (Bedrock Titan / Cohere), and a deterministic fake provider for tests.
• Dual authentication, one authz engine — API keys and OAuth 2.1 / OIDC both resolve to a unified Subject and are checked by the same relationship engine.
• Per-tenant isolation — every document, key, and relation tuple is tenant-scoped, with a shared common pool for cross-tenant knowledge.
• Aggregated reads across your tenants — search_memory, list_documents, and get span the caller's readable tenant set (home tenant + common pool + any tenant they hold a direct viewer/member/manager/admin grant on), with each result labeled by its owning tenant and an optional tenant_id filter to narrow to one. Admins are not expanded to every tenant, and writes stay single-tenant.
• Staleness & verification — per-doc_type freshness thresholds as a recall-time signal: off (none), advisory (warn in the response), or hard (withhold stale content until re-verified). The signal never deletes.
• Agent-driven retirement — no automated time-based sweep archives or deletes aged documents. Agents retire knowledge deliberately: supersede an entry (a supersedes edge auto-archives its target), update it in place, or delete it.
• Live global configuration — an admin-only config page at /ui/admin (system admins only) edits runtime globals in the database with no restart: retrieval tuning, rate limits, new-tenant defaults, the near-duplicate threshold, and the cleanup webhook. Environment variables seed the initial values on first migrate; a stored value then wins.
• Duplicate guard — an optional similarity check refuses near-duplicate writes, returning a similar_exists result so agents update instead of cloning. The threshold is a global default (0.85) with an optional per-tenant override.
• Cleanup pipeline — a nightly scanner queues near-duplicate candidates for review and can POST a per-scan JSON summary to a configurable webhook, plus a companion sweep that hard-deletes long-dead (revoked or expired) API keys so key listings stay tidy.
• Admin CLI + web console — a memory-admin binary that operates directly against the database (no server required, ideal for bootstrap), plus a full web console at /ui: browse and edit memories (color-coded per owning tenant), manage tenants (create, rename, delete), members and per-document ACL grants, and API keys (create, rotate, revoke, purge), and run archive imports — every surface gated by the caller's authorization.
• Self-service bootstrap & import — a one-shot first-run setup gated by a token the server generates and logs at boot (no env var to manage), available over HTTP or CLI, plus an async, job-based document-archive import (admin UI, HTTP, or CLI), so a fresh instance needs no direct database access.
• Rich MCP tool surface — a regular tool set for agents and a privileged admin tool set, routed automatically by the caller's authorization.
• Hardened HTTP edge — request-size limits, token-bucket rate limiting, strict CSP and security headers, unauthenticated health/readiness endpoints.

How it compares

Persistent memory for agents is a small, active category. The closest projects are edg-l/engram-mcp (Rust, single-user, local) and softmaxdata/engram (Python, LLM-driven). They solve a related problem with different priorities — this table is about fit, not ranking.

engram-mcp's stated non-goals — no auth, not multi-tenant, no cross-PC sync — are memory-system's core thesis: a governed, multi-tenant server with a shared pool and a single authorization decision point. And unlike an LLM-rewrite store, memory-system keeps a faithful-custody contract: it stores and returns your words, and every governance decision (who may read, is it stale, is it a duplicate) is deterministic and auditable — no model paraphrases your notes on the way in.

Retrieval quality

memory-system ships a reproducible retrieval benchmark on LongMemEval-S (Wu et al., ICLR 2025) — a long-term-memory needle-in-a-haystack set — driving the real search path and reporting session-level recall@k / MRR. Headline (Vertex text-embedding-005, 768-dim; seed 42, N=150):

What it shows, honestly:

• The embedder carries the signal on this workload — hybrid ≡ vector to the digit. RRF fusion here is for score-correctness and robustness, not extra recall.
• Lexical-only collapses (3.3% full-R@5) — paraphrased questions defeat keyword match. That is a LongMemEval property, not a verdict on lexical for real notes/IDs/code that share vocabulary; the arm is kept, not de-weighted, on this evidence alone.
• MMR diversity re-rank helps the hard cases. Tuned to λ=0.5 (the shipped default), it lifts full-recall on multi-evidence questions — full-R@5 88.7% → 91.3% — where a single evidence session isn't enough.
• The asymmetric part: memory-system ships five embedder backends, so it can publish retrieval quality vs. embedder — the axis a single-embedder project structurally can't. The table above is the Vertex flagship row; ollama / OpenAI rows will follow.

Full per-mode, per-question-type tables and the MMR λ sweep live in benchmarks/longmemeval/RESULTS.md; reproduce with the harness in that directory.

Quickstart

The bundled docker-compose.yml brings up the full stack — the memory server, PostgreSQL (pgvector/pgvector:pg17), and Ollama for embeddings — on the API-key path. The server is published on host port 8090 (mapped to the container's 8080).

# 1. Start Postgres, Ollama, and the memory server (published on http://localhost:8090)
docker compose up -d

# 2. Load the embedding model into Ollama (matches OLLAMA_MODEL=nomic-embed-text, 768-dim)
docker compose exec ollama ollama pull nomic-embed-text

Postgres stays on the internal compose network, but the server image also ships the memory-admin binary at /memory-admin. Bootstrap the first tenant and API key by exec-ing it inside the running container — it inherits the container's DATABASE_URL:

# 3. Create a tenant (note the returned id) and issue a key (plaintext shown ONCE)
docker compose exec memory-mcp /memory-admin tenant create --name quickstart
docker compose exec memory-mcp /memory-admin key issue --tenant <TENANT_ID> --label quickstart
# -> key: mmcp_XXXXXXXX...   (copy it now; it is never retrievable again)

memory-admin is also a standalone binary (grab it from a release archive or go build ./cmd/admin) — point it at any reachable Postgres via DATABASE_URL.

Finally, generate an MCP client config and point a client at the server:

docker compose exec memory-mcp /memory-admin setup --url http://localhost:8090 --token mmcp_XXXXXXXX...

That emits a ready-to-paste mcpServers block. The server answers MCP at http://localhost:8090/mcp with Authorization: Bearer mmcp_...; unauthenticated calls receive a 401 JSON challenge. Health lives at /~/health.

Container image

Prebuilt multi-arch images (linux/amd64, linux/arm64) are published on every release to GHCR (canonical) and mirrored to Docker Hub — pull from whichever you prefer:

docker pull ghcr.io/eliminyro/memory-system:latest   # GHCR — no pull rate limits
docker pull eliminyro/memory-system:latest           # Docker Hub mirror

Tags are :vX.Y.Z (semver), :latest, and :<git-sha>. The docker-compose.yml above builds from source for a hackable dev loop; to run a pinned release instead, swap build: . for image: ghcr.io/eliminyro/memory-system:latest.

Configuration

All configuration is via environment variables (parsed by caarlos0/env; see internal/config/config.go). Defaults are chosen so a stock Ollama deploy works out of the box.

Two kinds of variable appear below. Most are boot-only — read once at startup and fixed until a restart (DATABASE_URL, the embedding provider/model/dimension, the OAuth/authlet vars, PUBLIC_BASE_URL, MEMORY_RESET, and the import bounds). The rest, tagged seed, are bootstrap defaults: they seed the DB (instance_config) on first migrate, after which the live value is read from the database and edited without a restart on the admin config page (/ui/admin, system admins only). A stored value always wins over the env default — see Live global configuration.

Setting both MEMORY_MCP_GOOGLE_CLIENT_ID and MEMORY_MCP_GOOGLE_CLIENT_SECRET opts into the OAuth 2.1 / OIDC path; setting one without the other is a fatal boot error. When the OAuth path is enabled, PUBLIC_BASE_URL must be an absolute http(s) origin with no path. Leaving the Google client vars unset keeps /mcp API-key-only.

Warning — signup is PUBLIC by default. With the OAuth path enabled, the first verified login from an identity that has no existing membership auto-provisions a personal tenant for that user (they become its admin). If SIGNUP_ALLOWED_DOMAINS is unset or empty, this is public — anyone your IdP will authenticate (for Google, any Google account) can self-provision. To lock a private instance to your organization, set SIGNUP_ALLOWED_DOMAINS to your domain(s). The gate does not affect already-provisioned users, and the first admin is separately gated by the bootstrap token. See docs/administering.md.

Live global configuration

The seed variables above are only bootstrap defaults. On first migrate each seeds its column in the instance_config singleton; thereafter the live value is read from the database and edited — without a restart — on the admin config page at /ui/admin (system admins only). A stored value always wins over the env default. Most changes apply immediately (retrieval tuning, toggles, new-tenant defaults, self-service, log level, rate limits, request-size cap); CLEANUP_INTERVAL_HOURS takes effect at the next scheduler fire.

Two globals have no env var and are set only on the page:

• Near-duplicate threshold — the global default cosine cutoff for the duplicate guard (0.85), with an optional per-tenant override (unset ⇒ inherit the global default).
• Cleanup webhook URL — the near-duplicate cleanup scan POSTs a JSON summary here on each run; empty disables it. This replaces the previous Telegram notifier.

ADMIN_ALLOWED_EMAILS seeds the config page's admin-emails field: it grants system:memory#admin at startup, so edits there apply on the next restart. Live grant/revoke is in the Tenants admin UI.

Embedding providers

EMBEDDING_DIMENSIONS must match what the model emits — a startup guard freezes the embedding identity (provider + model + dimension) and refuses to run a populated corpus whose stored dimension differs. See docs/embedding-providers.md for per-provider detail, including Azure/vLLM/TEI base URLs and output-dimension truncation.

Administration

Two equivalent administrative surfaces exist:

• memory-admin CLI — operates directly against the database via DATABASE_URL, so it works even with no server running. This is the only way to bootstrap the first tenant and key. Holding DATABASE_URL is equivalent to full control — treat the CLI as an operator tool.
• MCP admin tools — the same operations exposed to an already-privileged MCP client (list_tenants, create_tenant, update_tenant, delete_tenant, create_api_key, list_api_keys, revoke_api_key, rotate_api_key, grant_user, list_users, update_user_role, revoke_user).

Admin status is pure relationship state: a principal is an admin iff it holds the global grant system:memory#admin. ADMIN_ALLOWED_EMAILS only seeds that grant at startup. Keys are issued as opaque mmcp_... tokens, stored only as a SHA-256 hash, and shown in plaintext exactly once. Rotation issues a replacement and can hold the predecessor valid for a grace window for zero-downtime swaps. A small admin web page is served at /ui, gated on bootstrap state: 404 before an admin exists, a static "OAuth not configured" notice once bootstrapped without OAuth wired up, and the real knowledge browser once both hold — it never shows a login or setup form itself.

A fresh instance can bootstrap itself instead of requiring direct database access: on first boot, while no admin exists yet, the server generates a one-time bootstrap token and logs it at WARN (read it with docker logs; it is regenerated every boot until an admin exists and is never persisted or logged again afterward). GET /bootstrap serves a dedicated setup page: paste the logged token and pick a mode. The page always shows OAuth status and offers two modes — MCP tokens (provision the admin Bearer key) and OAuth (additionally seed a founding admin email). POST /bootstrap provisions the first tenant and admin, returning the plaintext admin key exactly once (always, in either mode); the whole /bootstrap surface 404s once an admin exists. When OAuth is configured, the optional founding admin email is granted system:memory#admin (the same authority as the founding key), so the operator signs in at /ui via OAuth with full admin rights. With OAuth enabled the server also auto-registers the /ui OAuth client on boot (from MEMORY_UI_CLIENT_ID, defaulting to memory-ui, and PUBLIC_BASE_URL), so browser login works with no manual database access. (Configuring the OAuth provider from the page itself, persisted to the DB, is planned.) memory-admin bootstrap provisions the same way straight from the CLI with no token needed (it talks to the database directly and is inherently privileged). An operator-only MEMORY_RESET=1 boot flag re-arms bootstrap by clearing the admin key(s) — it never deletes tenant data, and the next boot logs a fresh token. Loading a corpus works the same way: upload an archive from the admin UI, POST /api/admin/import (multipart, a zip, processed asynchronously by a background worker), or run the memory-import CLI over a local directory. See docs/administering.md for the full first-run bootstrap, break-glass reset, document import, user-management, and key-lifecycle runbook.

Connecting a client

memory-admin setup emits the standard mcpServers JSON without writing your token to disk:

memory-admin setup --url https://mem.example.org --token mmcp_your_token_here
# or: MEMORY_URL=... MEMORY_TOKEN=... memory-admin setup

Paste the emitted block into any MCP client that supports HTTP transport.

When OAuth is configured, a static key is optional: an MCP client that supports OAuth can connect with just the server /mcp URL — it discovers the authorization server from the protected-resource metadata, registers dynamically, and completes the PKCE login flow, so there is no token to mint or store. Use a static mmcp_... key as a Bearer token for clients (or CI) that don't do OAuth. The /ui "Connect an MCP client" panel shows both paths. See docs/connecting-clients.md for client examples, and docs/memory-usage-prompt.md for a persona-free, ready-to-paste system-prompt block that teaches an assistant to recall before acting, store durable learnings, and update instead of duplicating.

Architecture

The server is one HTTP listener. /mcp carries the MCP Streamable-HTTP transport; a DualAuth router inspects the bearer token (a three-segment JWT is verified by the OAuth path, anything else falls through to API keys), writes a unified Subject, and the MCP transport calls the authorization engine's Check. Application code lives under internal/ (config, database, repository, service, mcp, authz, auth, authletas, middleware, cleanup, staleness); entry points are cmd/server, cmd/admin, and cmd/import. Data is stored in Postgres with a vector(N) embedding column per section, an HNSW cosine index, a GIN lexical index, and a relation_tuples table for the authorization graph. See docs/architecture.md for the full component map, data-flow walkthroughs, and the dual-auth model.

Building from source

Requires Go (see the version in go.mod).

go build ./...          # build all binaries (cmd/server, cmd/admin, cmd/import)
go test ./...           # unit tests

Integration tests are behind a build tag and need a live PostgreSQL (with pgvector). Point TEST_DATABASE_URL at it and serialize packages so parallel migrations don't race:

export TEST_DATABASE_URL='postgres://memory:memory@localhost:5432/memory?sslmode=disable'
go test -tags=integration -p 1 ./...

Releases are automated: on a green CI run on master, the release workflow computes the next version, tags it, builds and pushes a multi-arch image to ghcr.io/eliminyro/memory-system (mirrored to Docker Hub when its credentials are configured), and runs goreleaser to produce the memory-mcp, memory-import, and memory-admin binaries, archives, checksums, SBOMs, and a GitHub Release.

Status and limits

Honest about what it is and isn't:

• Single-node Postgres. Durability, backups, and availability are Postgres's to provide; there is no built-in sharding or HA — front it with a Postgres HA setup if you need one.
• Embedder identity is fixed to the vector column. Changing the provider, model, or dimension means re-embedding the whole corpus; the server refuses a silent mismatch rather than corrupting search.
• Pick the embedder first. The benchmark shows the embedder is the dominant lever on recall — RRF and MMR are measured refinements on top, not substitutes for a good embedder.
• No LLM in the write path, by design. The server stores your words verbatim and never summarizes, paraphrases, or "corrects" them. That is a deliberate faithful-custody contract — a different product from memory systems that rewrite your notes on ingest.
• History is an audit trail, not rollback. Mutation history is append-only, shared-tenant, and toggle-gated; update_section overwrites — there is no per-edit version restore.
• Opt-in by default. Mutation history is off until an operator enables its global toggle; the duplicate guard and staleness signal are per-tenant opt-in.

References

• RRF (hybrid fusion) — Cormack, Clarke & Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods, SIGIR 2009.
• MMR (diversity re-rank) — Carbonell & Goldstein, The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries, SIGIR 1998.
• LongMemEval (benchmark) — Wu et al., LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, ICLR 2025 · dataset.
• Foundations — pgvector · Model Context Protocol.

Contributing

Contributions are welcome.

• Use Conventional Commits (feat:, fix:, refactor:, chore:, …) — release versioning is derived from commit history.
• Code must be gofmt-clean and pass golangci-lint run.
• Keep tests green: go test ./... and, where relevant, the integration suite above. CI runs gofmt, golangci-lint, unit tests, integration tests, a build, and a boot smoke test on every pull request.

License

Licensed under the GNU Affero General Public License v3.0. See LICENSE.

Note the AGPL's §13 network-use clause: if you run a modified version of this software and let others interact with it over a network, you must offer those users the corresponding source of your modified version. Hosting a fork is fine — publishing its source is the condition.