README
¶
lore — the auditable knowledge base
A fast, scriptable CLI for grounded, cited, and verifiable retrieval-augmented generation (RAG) over specific sets of documents — built for pipes, scripts, and CI, not GUIs. An answer is only as good as your ability to prove it's grounded, so lore makes faithfulness verification and retrieval evaluation CI-gateable and machine-readable.
$ export LORE_API_KEY=sk-... # any OpenAI-compatible provider (or a local one; see below)
$ lore init notes
$ lore add notes ./docs
$ lore ask notes "how does auth work?"
lore embeds and answers over an OpenAI-compatible API, so it needs a provider
(OpenAI, Azure, Ollama, vLLM, a local server, ...). Configure one before init —
see Quickstart and docs/configuration.md.
Why lore
- Auditable — every answer cites exact chunks;
ask --verifychecks each claim is entailed by what it cites;lore evalgates retrieval and faithfulness in CI. - Fast — single static binary; concurrent ingest; millisecond vector search.
- Safe — invariants enforced in code; no destructive op without explicit intent.
- Automation-first — data to stdout, logs to stderr,
--jsoneverywhere, meaningful exit codes. - Provider-agnostic — any OpenAI-compatible endpoint (OpenAI, Azure, Ollama, vLLM, …).
# prove the answer is grounded — fail CI if any claim isn't supported by its citation
$ lore ask notes "how does auth work?" --verify-strict
# measure retrieval/faithfulness over a question set; non-zero exit below threshold
$ lore eval notes -f questions.jsonl --verify --fail-under recall=0.8 --fail-under support_rate=0.9
Retrieval beyond plain cosine: hybrid BM25⊕vector (--hybrid), metadata
filtering (--where 'author=alice'), cross-encoder rerank (--rerank),
diversity (--mmr, --max-per-source), and recency time-decay ranking
(--recency) so newer documents aren't buried by stale-but-similar ones. See
docs/retrieval.md.
How lore is different. Where RAG usually means a Python framework (LlamaIndex,
LangChain, txtai) or hand-rolled embedding + vector-store calls, lore ships a
portable, citable corpus an agent can query — grounded answers citing exact
chunks, one static binary, and a read-only MCP server. Build the
index once, export it, point an agent at it: no notebook, no service.
Install
Prebuilt binaries (recommended) — every release ships static, cross-compiled binaries for Linux, macOS, and Windows (amd64 + arm64) with checksums. Download the archive for your platform from the latest release, then:
tar -xzf lore_<version>_linux_amd64.tar.gz # Windows ships a .zip
sudo install lore /usr/local/bin/
On macOS the binary is unsigned; if Gatekeeper blocks it, clear the quarantine
flag once with xattr -d com.apple.quarantine lore (or right-click the binary
in Finder and choose Open). On Windows, unzip the archive and add the folder
containing lore.exe to your PATH.
With Go (1.25+): go install github.com/jmurray2011/lore/cmd/lore@latest
From source: git clone https://github.com/jmurray2011/lore && cd lore && go build -o lore ./cmd/lore
Quickstart
# 0. Point lore at an OpenAI-compatible provider. Do this BEFORE `init`: a
# collection is permanently pinned to the embedding model configured when it
# is created. Prefer the env var for the key; other settings go in the config.
export LORE_API_KEY=sk-...
lore config init # optional: write a starter config.toml to edit
lore config path # print the config file lore will read
# Fully local, no key (Ollama):
# export LORE_BASE_URL=http://localhost:11434/v1
# export LORE_EMBED_MODEL=nomic-embed-text LORE_DIMENSIONS=768 LORE_CHAT_MODEL=llama3.1
# create a collection (pinned to your embedding model's space)
lore init notes
# ingest files or directories — idempotent, safe to re-run
lore add notes ./docs report.pdf
lore add notes ./docs --exclude '*(1).pdf' # skip files by glob
cat meeting.md | lore add notes --stdin --name meeting.md # or pipe content in
# re-ingest changed sources later; --prune also drops docs deleted at the source
lore sync notes --prune
lore sync notes --prune --dry-run # preview what --prune would remove
# retrieve the most similar chunks
lore query notes "rotation policy for signing keys" -k 5
lore query notes "rotation policy" --source '*.pdf' # scope to matching docs
# ask a grounded question (retrieval + synthesis with citations)
lore ask notes "what is our key rotation policy?"
lore ask notes "…" --strict # exit 1 if retrieval finds no chunks (empty collection, over-narrow --source/--where)
lore ask notes "…" --verify-strict # exit 5 if any answer claim isn't supported by its citation (faithfulness gate)
# inventory & cleanup
lore ls # collections
lore status notes # one collection's metadata
lore docs notes # documents in a collection
lore cat notes --doc report.md # print a document's chunks (--doc takes a basename, glob, or full URI)
lore rm notes --doc report.md # delete a document (or `lore rm notes` for all)
lore diff old new # document-level diff: added / removed / changed by source
Every command takes --json; human output is colorized on a TTY and plain
elsewhere (--no-color / NO_COLOR=1 forces it off). Query hits and citations
are tagged source#chunk, so every result traces back to its document.
Features
- Grounded, cited answers — every claim traces to the exact chunk it came from. docs →
- Two-stage reranking — a cross-encoder pass, the biggest precision lever in RAG. docs →
- Token-budget retrieval — fill N tokens of context instead of a fixed chunk count. docs →
- Streaming answers — tokens stream on a TTY, buffer cleanly in pipes. docs →
- Cross-collection retrieval — merge several corpora, or semantically diff two. docs →
- Portable & encrypted corpora —
exporta whole indexed corpus to one file; ship orage-encrypt it. A recipient queries it with their own embedder (import --re-embed), or offline with no API key (query --lexical). docs → - Collection diff — see which documents were added, removed, or changed between two collections or a snapshot. docs →
- Read-only MCP server — expose your corpora as grounded, cited tools to any MCP client. docs →
- Structure-aware chunking — heading/paragraph-aware, code-fence-safe, pinned per collection. docs →
- Provider-agnostic — any OpenAI-compatible endpoint, with independent embed/chat/rerank endpoints. docs →
How it works
Documents are extracted, chunked, embedded, and stored with their vectors;
query runs similarity search and ask synthesizes a grounded answer citing the
chunks it used. lore is hexagonal (ports & adapters) with dependencies pointing
inward only, so storage engines and providers swap behind conformance-verified ports.
cli ──► app (use cases) ──► domain
▲
│ implements ports defined in app
adapters (memstore, sqlite, openai, fs, docx, pdf, xlsx)
Adding a storage engine or provider means implementing the relevant ports and passing the conformance suites — no changes to the core.
Configuration
Precedence: flags > env (LORE_*) > config file > defaults; the config file
is TOML at <user-config-dir>/lore/config.toml. Run lore config path to print
the exact location, and lore config init to write a commented starter file
there. Full settings reference — provider, rerank, storage, chunking, cache,
split endpoints, Azure — in docs/configuration.md.
Output and exit codes
stdout carries data (human-readable on a TTY, JSON with --json); logs/errors to stderr.
| Code | Meaning |
|---|---|
| 0 | success |
| 1 | runtime error |
| 2 | usage error |
| 3 | not found |
| 4 | invariant violation (e.g. embedding-space mismatch) |
| 5 | quality gate not met (ask --verify-strict, eval --fail-under) |
Documentation
- Retrieval & answers — diagnostics, reranking, budgets, cross-collection, streaming, chunking, citation auditing.
- Configuration — env/TOML reference, global flags, cache, Azure, split embed/chat endpoints, attachments.
- MCP server — tools, Claude Desktop setup, security posture.
- Portable & encrypted corpora —
export/importand theageencryption threat model. - Document formats — what
lore addingests, and the unsupported/skipped distinction.
Contributing
Contributions are welcome. lore is hexagonal with a strict dependency direction
(cli → app → domain), developed test-first, with a conformance-suite contract
keeping adapters swappable. Before sending a change, make the validate gate pass:
gofmt -l . # must print nothing
go vet ./...
go test -race -count=1 ./...
License
Licensed under the Apache License 2.0.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
lore
command
Command lore is the composition root: the only place that imports adapters.
|
Command lore is the composition root: the only place that imports adapters. |
|
internal
|
|
|
adapters/agecrypt
Package agecrypt is lore's encryption boundary for portable artifacts: it wraps a whole serialized artifact byte stream in an age envelope (and reverses it), so the exported file is encrypted at rest and in transit.
|
Package agecrypt is lore's encryption boundary for portable artifacts: it wraps a whole serialized artifact byte stream in an age envelope (and reverses it), so the exported file is encrypted at rest and in transit. |
|
adapters/cache
Package cache provides a caching decorator around an app.Generator: it serves a previously synthesized answer from an app.AnswerCache when the same question is asked over the same grounding, skipping the LLM round-trip.
|
Package cache provides a caching decorator around an app.Generator: it serves a previously synthesized answer from an app.AnswerCache when the same question is asked over the same grounding, skipping the LLM round-trip. |
|
adapters/docx
Package docx is an Extractor for Word .docx files.
|
Package docx is an Extractor for Word .docx files. |
|
adapters/extract
Package extract turns supported raw content into plain text for chunking.
|
Package extract turns supported raw content into plain text for chunking. |
|
adapters/fs
Package fs is a filesystem Source: it walks a path (file or directory) and yields each regular, non-hidden file as a SourceItem whose content type is detected from the extension.
|
Package fs is a filesystem Source: it walks a path (file or directory) and yields each regular, non-hidden file as a SourceItem whose content type is detected from the extension. |
|
adapters/httpjson
Package httpjson is the shared HTTP-JSON plumbing for lore's provider adapters (the OpenAI-compatible embed/chat client and the Cohere-style reranker): a POST of a JSON body, decode of a 2xx JSON response, bearer or api-key auth, and automatic retry of transient 429/503 responses honoring Retry-After.
|
Package httpjson is the shared HTTP-JSON plumbing for lore's provider adapters (the OpenAI-compatible embed/chat client and the Cohere-style reranker): a POST of a JSON body, decode of a 2xx JSON response, bearer or api-key auth, and automatic retry of transient 429/503 responses honoring Retry-After. |
|
adapters/memstore
Package memstore provides in-memory reference implementations of the persistence ports.
|
Package memstore provides in-memory reference implementations of the persistence ports. |
|
adapters/openai
Package openai adapts the Embedder and Generator ports to any OpenAI-compatible HTTP API (OpenAI, Ollama's /v1, vLLM, LM Studio, OpenRouter).
|
Package openai adapts the Embedder and Generator ports to any OpenAI-compatible HTTP API (OpenAI, Ollama's /v1, vLLM, LM Studio, OpenRouter). |
|
adapters/pdf
Package pdf is an Extractor for PDF files, backed by the pure-Go github.com/ledongthuc/pdf (no cgo, so the static-binary and cross-compile goals hold).
|
Package pdf is an Extractor for PDF files, backed by the pure-Go github.com/ledongthuc/pdf (no cgo, so the static-binary and cross-compile goals hold). |
|
adapters/rerank
Package rerank adapts the app.RerankProvider port to a Cohere-style rerank HTTP API (POST /rerank with {model, query, documents, top_n} → {results: [{index, relevance_score}]}), the de-facto standard that Cohere, Jina, Voyage, and others conform to.
|
Package rerank adapts the app.RerankProvider port to a Cohere-style rerank HTTP API (POST /rerank with {model, query, documents, top_n} → {results: [{index, relevance_score}]}), the de-facto standard that Cohere, Jina, Voyage, and others conform to. |
|
adapters/sqlite
Package sqlite is a persistent backend: one SQLite database file holding all collections, documents, chunks, and vectors.
|
Package sqlite is a persistent backend: one SQLite database file holding all collections, documents, chunks, and vectors. |
|
adapters/tiktoken
Package tiktoken is the TokenCounter adapter: it counts tokens with a pure-Go, offline tiktoken BPE codec (o200k_base — the encoding text-embedding-3 and gpt-4o use).
|
Package tiktoken is the TokenCounter adapter: it counts tokens with a pure-Go, offline tiktoken BPE codec (o200k_base — the encoding text-embedding-3 and gpt-4o use). |
|
adapters/xlsx
Package xlsx is an Extractor for Excel .xlsx files.
|
Package xlsx is an Extractor for Excel .xlsx files. |
|
app
Package app contains the use cases and defines the ports (interfaces) they consume.
|
Package app contains the use cases and defines the ports (interfaces) they consume. |
|
artifact
Package artifact defines lore's portable corpus format: a single, versioned, self-contained byte stream holding one collection — its embedding-space and chunker pins, metadata, documents, chunks, and vectors — so a collection can be exported to a file and reconstructed elsewhere with its invariants intact.
|
Package artifact defines lore's portable corpus format: a single, versioned, self-contained byte stream holding one collection — its embedding-space and chunker pins, metadata, documents, chunks, and vectors — so a collection can be exported to a file and reconstructed elsewhere with its invariants intact. |
|
cli
Package cli is lore's driving adapter: it translates cobra commands into use case calls and renders results.
|
Package cli is lore's driving adapter: it translates cobra commands into use case calls and renders results. |
|
config
Package config loads lore's typed configuration with the precedence flags > env (LORE_*) > file (TOML) > defaults, and builds the slog logger.
|
Package config loads lore's typed configuration with the precedence flags > env (LORE_*) > file (TOML) > defaults, and builds the slog logger. |
|
conformance
Package conformance contains executable contracts for the ports defined in internal/app.
|
Package conformance contains executable contracts for the ports defined in internal/app. |
|
limitio
Package limitio provides bounded reads that fail loudly when an input exceeds a byte cap.
|
Package limitio provides bounded reads that fail loudly when an input exceeds a byte cap. |
|
mcp
Package mcp is lore's MCP (Model Context Protocol) driving adapter: it exposes lore's grounded retrieval and Q&A use cases as MCP tools so any MCP client (Claude Desktop, editors, agent frameworks) can call a private corpus and get back cited answers.
|
Package mcp is lore's MCP (Model Context Protocol) driving adapter: it exposes lore's grounded retrieval and Q&A use cases as MCP tools so any MCP client (Claude Desktop, editors, agent frameworks) can call a private corpus and get back cited answers. |
|
secret
Package secret resolves a secret by running a command and reading its stdout, so a passphrase or key never lands in a flag value, in argv, or on disk.
|
Package secret resolves a secret by running a command and reading its stdout, so a passphrase or key never lands in a flag value, in argv, or on disk. |