memory-system

memory-system

Persistent, semantic long-term memory for AI agents, served over the Model Context Protocol.

What it is

memory-system is a self-hostable MCP server that gives AI agents a durable, searchable long-term memory across sessions. Documents are stored as markdown, split into sections, embedded, and made retrievable through hybrid search — vector similarity fused with lexical ranking — backed by PostgreSQL and the pgvector extension. An agent stores what it learns as a self-contained note and recalls it later by meaning, not just keyword.

It ships as a single Go binary (module github.com/eliminyro/memory-system) that serves MCP over HTTP. On store, each section is embedded before any database write, so a provider failure never leaves a half-written document; on search, vector candidates ordered by cosine distance are fused with lexical candidates ranked by Postgres full-text search. A per-doc_type staleness model, an optional near-duplicate guard, and a retention sweep keep the corpus from rotting over time.

Every request passes through a Zanzibar-style, relationship-based authorization engine — a single decision point backed by relation tuples rather than hard-coded rules. Two independent authentication paths feed that engine: long-lived API keys (ideal for headless agents and CI) and OAuth 2.1 / OIDC (for humans signing in through an upstream identity provider). All data is tenant-scoped, with an optional shared "common" pool for knowledge every tenant can read; a caller's reads aggregate across their home tenant, the common pool, and any tenant they hold a direct grant on, while writes stay scoped to a single tenant.

Features

• Hybrid retrieval — pgvector HNSW cosine search fused with a GIN-indexed tsvector lexical search, scored as 1 - (embedding <=> query_vector).
• Pluggable embedding providers — ollama, gcp (Vertex AI), openai (any OpenAI-compatible /embeddings endpoint: OpenAI, Azure, vLLM, LM Studio, LocalAI, HuggingFace TEI), aws (Bedrock Titan / Cohere), and a deterministic fake provider for tests.
• Dual authentication, one authz engine — API keys and OAuth 2.1 / OIDC both resolve to a unified Subject and are checked by the same relationship engine.
• Per-tenant isolation — every document, key, and relation tuple is tenant-scoped, with a shared common pool for cross-tenant knowledge.
• Aggregated reads across your tenants — search_memory, list_documents, and get span the caller's readable tenant set (home tenant + common pool + any tenant they hold a direct viewer/member/manager/admin grant on), with each result labeled by its owning tenant and an optional tenant_id filter to narrow to one. Admins are not expanded to every tenant, and writes stay single-tenant.
• Staleness & verification — per-doc_type freshness thresholds with off, advisory, or hard modes; stale sections can be withheld until re-verified.
• Retention sweep — archives documents left unverified past a configurable multiple of their staleness threshold, then hard-deletes after a grace window (guarded against destructive misconfiguration).
• Duplicate guard — an optional similarity check refuses near-duplicate writes, returning a similar_exists result so agents update instead of cloning.
• Cleanup pipeline — a nightly scanner queues near-duplicate candidates and can post a per-scan summary to Telegram, plus a companion sweep that hard-deletes long-dead (revoked or expired) API keys past a configurable TTL so key listings stay tidy.
• Admin CLI + web console — a memory-admin binary that operates directly against the database (no server required, ideal for bootstrap), plus a full web console at /ui: browse and edit memories (color-coded per owning tenant), manage tenants (create, rename, delete), members and per-document ACL grants, and API keys (create, rotate, revoke, purge), and run archive imports — every surface gated by the caller's authorization.
• Self-service bootstrap & import — a one-shot first-run setup gated by a token the server generates and logs at boot (no env var to manage), available over HTTP or CLI, plus an async, job-based document-archive import (admin UI, HTTP, or CLI), so a fresh instance needs no direct database access.
• Rich MCP tool surface — a regular tool set for agents and a privileged admin tool set, routed automatically by the caller's authorization.
• Hardened HTTP edge — request-size limits, token-bucket rate limiting, strict CSP and security headers, unauthenticated health/readiness endpoints.

Quickstart

The bundled docker-compose.yml brings up the full stack — the memory server, PostgreSQL (pgvector/pgvector:pg17), and Ollama for embeddings — on the API-key path. The server is published on host port 8090 (mapped to the container's 8080).

# 1. Start Postgres, Ollama, and the memory server (published on http://localhost:8090)
docker compose up -d

# 2. Load the embedding model into Ollama (matches OLLAMA_MODEL=nomic-embed-text, 768-dim)
docker compose exec ollama ollama pull nomic-embed-text

Postgres stays on the internal compose network, but the server image also ships the memory-admin binary at /memory-admin. Bootstrap the first tenant and API key by exec-ing it inside the running container — it inherits the container's DATABASE_URL:

# 3. Create a tenant (note the returned id) and issue a key (plaintext shown ONCE)
docker compose exec memory-mcp /memory-admin tenant create --name quickstart
docker compose exec memory-mcp /memory-admin key issue --tenant <TENANT_ID> --label quickstart
# -> key: mmcp_XXXXXXXX...   (copy it now; it is never retrievable again)

memory-admin is also a standalone binary (grab it from a release archive or go build ./cmd/admin) — point it at any reachable Postgres via DATABASE_URL.

Finally, generate an MCP client config and point a client at the server:

docker compose exec memory-mcp /memory-admin setup --url http://localhost:8090 --token mmcp_XXXXXXXX...

That emits a ready-to-paste mcpServers block. The server answers MCP at http://localhost:8090/mcp with Authorization: Bearer mmcp_...; unauthenticated calls receive a 401 JSON challenge. Health lives at /~/health.

Container image

Prebuilt multi-arch images (linux/amd64, linux/arm64) are published on every release to GHCR (canonical) and mirrored to Docker Hub — pull from whichever you prefer:

docker pull ghcr.io/eliminyro/memory-system:latest   # GHCR — no pull rate limits
docker pull eliminyro/memory-system:latest           # Docker Hub mirror

Tags are :vX.Y.Z (semver), :latest, and :<git-sha>. The docker-compose.yml above builds from source for a hackable dev loop; to run a pinned release instead, swap build: . for image: ghcr.io/eliminyro/memory-system:latest.

Configuration

All configuration is via environment variables (parsed by caarlos0/env; see internal/config/config.go). Defaults are chosen so a stock Ollama deploy works out of the box.

Setting both MEMORY_MCP_GOOGLE_CLIENT_ID and MEMORY_MCP_GOOGLE_CLIENT_SECRET opts into the OAuth 2.1 / OIDC path; setting one without the other is a fatal boot error. When the OAuth path is enabled, PUBLIC_BASE_URL must be an absolute http(s) origin with no path. Leaving the Google client vars unset keeps /mcp API-key-only.

Warning — signup is PUBLIC by default. With the OAuth path enabled, the first verified login from an identity that has no existing membership auto-provisions a personal tenant for that user (they become its admin). If SIGNUP_ALLOWED_DOMAINS is unset or empty, this is public — anyone your IdP will authenticate (for Google, any Google account) can self-provision. To lock a private instance to your organization, set SIGNUP_ALLOWED_DOMAINS to your domain(s). The gate does not affect already-provisioned users, and the first admin is separately gated by the bootstrap token. See docs/administering.md.

Embedding providers

EMBEDDING_DIMENSIONS must match what the model emits — a startup guard freezes the embedding identity (provider + model + dimension) and refuses to run a populated corpus whose stored dimension differs. See docs/embedding-providers.md for per-provider detail, including Azure/vLLM/TEI base URLs and output-dimension truncation.

Administration

Two equivalent administrative surfaces exist:

• memory-admin CLI — operates directly against the database via DATABASE_URL, so it works even with no server running. This is the only way to bootstrap the first tenant and key. Holding DATABASE_URL is equivalent to full control — treat the CLI as an operator tool.
• MCP admin tools — the same operations exposed to an already-privileged MCP client (list_tenants, create_tenant, update_tenant, delete_tenant, create_api_key, list_api_keys, revoke_api_key, rotate_api_key, grant_user, list_users, update_user_role, revoke_user).

Admin status is pure relationship state: a principal is an admin iff it holds the global grant system:memory#admin. ADMIN_ALLOWED_EMAILS only seeds that grant at startup. Keys are issued as opaque mmcp_... tokens, stored only as a SHA-256 hash, and shown in plaintext exactly once. Rotation issues a replacement and can hold the predecessor valid for a grace window for zero-downtime swaps. A small admin web page is served at /ui, gated on bootstrap state: 404 before an admin exists, a static "OAuth not configured" notice once bootstrapped without OAuth wired up, and the real knowledge browser once both hold — it never shows a login or setup form itself.

A fresh instance can bootstrap itself instead of requiring direct database access: on first boot, while no admin exists yet, the server generates a one-time bootstrap token and logs it at WARN (read it with docker logs; it is regenerated every boot until an admin exists and is never persisted or logged again afterward). GET /bootstrap serves a dedicated setup page: paste the logged token and pick a mode. The page always shows OAuth status and offers two modes — MCP tokens (provision the admin Bearer key) and OAuth (additionally seed a founding admin email). POST /bootstrap provisions the first tenant and admin, returning the plaintext admin key exactly once (always, in either mode); the whole /bootstrap surface 404s once an admin exists. When OAuth is configured, the optional founding admin email is granted system:memory#admin (the same authority as the founding key), so the operator signs in at /ui via OAuth with full admin rights. With OAuth enabled the server also auto-registers the /ui OAuth client on boot (from MEMORY_UI_CLIENT_ID, defaulting to memory-ui, and PUBLIC_BASE_URL), so browser login works with no manual database access. (Configuring the OAuth provider from the page itself, persisted to the DB, is planned.) memory-admin bootstrap provisions the same way straight from the CLI with no token needed (it talks to the database directly and is inherently privileged). An operator-only MEMORY_RESET=1 boot flag re-arms bootstrap by clearing the admin key(s) — it never deletes tenant data, and the next boot logs a fresh token. Loading a corpus works the same way: upload an archive from the admin UI, POST /api/admin/import (multipart, a zip, processed asynchronously by a background worker), or run the memory-import CLI over a local directory. See docs/administering.md for the full first-run bootstrap, break-glass reset, document import, user-management, and key-lifecycle runbook.

Connecting a client

memory-admin setup emits the standard mcpServers JSON without writing your token to disk:

memory-admin setup --url https://mem.example.org --token mmcp_your_token_here
# or: MEMORY_URL=... MEMORY_TOKEN=... memory-admin setup

Paste the emitted block into any MCP client that supports HTTP transport.

When OAuth is configured, a static key is optional: an MCP client that supports OAuth can connect with just the server /mcp URL — it discovers the authorization server from the protected-resource metadata, registers dynamically, and completes the PKCE login flow, so there is no token to mint or store. Use a static mmcp_... key as a Bearer token for clients (or CI) that don't do OAuth. The /ui "Connect an MCP client" panel shows both paths. See docs/connecting-clients.md for client examples, and docs/memory-usage-prompt.md for a persona-free, ready-to-paste system-prompt block that teaches an assistant to recall before acting, store durable learnings, and update instead of duplicating.

Architecture

The server is one HTTP listener. /mcp carries the MCP Streamable-HTTP transport; a DualAuth router inspects the bearer token (a three-segment JWT is verified by the OAuth path, anything else falls through to API keys), writes a unified Subject, and the MCP transport calls the authorization engine's Check. Application code lives under internal/ (config, database, repository, service, mcp, authz, auth, authletas, middleware, cleanup, retention, staleness); entry points are cmd/server, cmd/admin, and cmd/import. Data is stored in Postgres with a vector(N) embedding column per section, an HNSW cosine index, a GIN lexical index, and a relation_tuples table for the authorization graph. See docs/architecture.md for the full component map, data-flow walkthroughs, and the dual-auth model.

Building from source

Requires Go (see the version in go.mod).

go build ./...          # build all binaries (cmd/server, cmd/admin, cmd/import)
go test ./...           # unit tests

Integration tests are behind a build tag and need a live PostgreSQL (with pgvector). Point TEST_DATABASE_URL at it and serialize packages so parallel migrations don't race:

export TEST_DATABASE_URL='postgres://memory:memory@localhost:5432/memory?sslmode=disable'
go test -tags=integration -p 1 ./...

Releases are automated: on a green CI run on master, the release workflow computes the next version, tags it, builds and pushes a multi-arch image to ghcr.io/eliminyro/memory-system (mirrored to Docker Hub when its credentials are configured), and runs goreleaser to produce the memory-mcp, memory-import, and memory-admin binaries, archives, checksums, SBOMs, and a GitHub Release.

Contributing

Contributions are welcome.

• Use Conventional Commits (feat:, fix:, refactor:, chore:, …) — release versioning is derived from commit history.
• Code must be gofmt-clean and pass golangci-lint run.
• Keep tests green: go test ./... and, where relevant, the integration suite above. CI runs gofmt, golangci-lint, unit tests, integration tests, a build, and a boot smoke test on every pull request.

License

Licensed under the GNU Affero General Public License v3.0. See LICENSE.

Note the AGPL's §13 network-use clause: if you run a modified version of this software and let others interact with it over a network, you must offer those users the corresponding source of your modified version. Hosting a fork is fine — publishing its source is the condition.