lore

lore

A fast, scriptable CLI for grounded, cited retrieval-augmented generation (RAG) over specific sets of documents — built for pipes, scripts, and CI, not GUIs.

$ lore init notes
$ lore add notes ./docs
$ lore ask notes "how does auth work?"

Why lore

• Fast — single static binary; concurrent ingest; millisecond vector search.
• Safe — invariants enforced in code; no destructive op without explicit intent.
• Automation-first — data to stdout, logs to stderr, --json everywhere.
• Provider-agnostic — any OpenAI-compatible endpoint (OpenAI, Azure, Ollama, vLLM, …).

How lore is different. Where RAG usually means a Python framework (LlamaIndex, LangChain, txtai) or hand-rolled embedding + vector-store calls, lore ships a portable, citable corpus an agent can query — grounded answers citing exact chunks, one static binary, and a read-only MCP server. Build the index once, export it, point an agent at it: no notebook, no service.

Install

Prebuilt binaries (recommended) — every release ships static, cross-compiled binaries for Linux, macOS, and Windows (amd64 + arm64) with checksums. Download the archive for your platform from the latest release, then:

tar -xzf lore_<version>_linux_amd64.tar.gz   # Windows ships a .zip
sudo install lore /usr/local/bin/

With Go (1.25+): go install github.com/jmurray2011/lore/cmd/lore@latest

From source: git clone https://github.com/jmurray2011/lore && cd lore && go build -o lore ./cmd/lore

Quickstart

# create a collection (pinned to your embedding model's space)
lore init notes

# ingest files or directories — idempotent, safe to re-run
lore add notes ./docs report.pdf
cat meeting.md | lore add notes --stdin --name meeting.md   # or pipe content in

# re-ingest changed sources later; --prune also drops docs deleted at the source
lore sync notes --prune
lore sync notes --prune --dry-run        # preview what --prune would remove

# retrieve the most similar chunks
lore query notes "rotation policy for signing keys" -k 5
lore query notes "rotation policy" --source '*.pdf'         # scope to matching docs

# ask a grounded question (retrieval + synthesis with citations)
lore ask notes "what is our key rotation policy?"
lore ask notes "unrelated question" --strict   # --strict: ungrounded → hard error (exit 1)

# inventory & cleanup
lore ls                          # collections
lore status notes                # one collection's metadata
lore docs notes                  # documents in a collection
lore cat notes --doc <uri>       # print a document's stored chunks
lore rm  notes --doc <uri>       # delete a document (or `lore rm notes` for all)

Every command takes --json; human output is colorized on a TTY and plain elsewhere (--no-color / NO_COLOR=1 forces it off). Query hits and citations are tagged source#chunk, so every result traces back to its document.

Features

• Grounded, cited answers — every claim traces to the exact chunk it came from. docs →
• Two-stage reranking — a cross-encoder pass, the biggest precision lever in RAG. docs →
• Token-budget retrieval — fill N tokens of context instead of a fixed chunk count. docs →
• Streaming answers — tokens stream on a TTY, buffer cleanly in pipes. docs →
• Cross-collection retrieval — merge several corpora, or semantically diff two. docs →
• Portable & encrypted corpora — export a whole indexed corpus to one file; ship or age-encrypt it. docs →
• Read-only MCP server — expose your corpora as grounded, cited tools to any MCP client. docs →
• Structure-aware chunking — heading/paragraph-aware, code-fence-safe, pinned per collection. docs →
• Provider-agnostic — any OpenAI-compatible endpoint, with independent embed/chat/rerank endpoints. docs →

How it works

Documents are extracted, chunked, embedded, and stored with their vectors; query runs similarity search and ask synthesizes a grounded answer citing the chunks it used. lore is hexagonal (ports & adapters) with dependencies pointing inward only, so storage engines and providers swap behind conformance-verified ports.

cli  ──►  app (use cases)  ──►  domain
              ▲
              │ implements ports defined in app
        adapters (memstore, sqlite, openai, fs, docx, pdf, xlsx)

Adding a storage engine or provider means implementing the relevant ports and passing the conformance suites — no changes to the core.

Configuration

Precedence: flags > env (LORE_*) > config file > defaults; the config file is TOML at <user-config-dir>/lore/config.toml. Full settings reference — provider, rerank, storage, chunking, cache, split endpoints, Azure — in docs/configuration.md.

Output and exit codes

stdout carries data (human-readable on a TTY, JSON with --json); logs/errors to stderr.

Documentation

• Retrieval & answers — diagnostics, reranking, budgets, cross-collection, streaming, chunking, citation auditing.
• Configuration — env/TOML reference, global flags, cache, Azure, split embed/chat endpoints, attachments.
• MCP server — tools, Claude Desktop setup, security posture.
• Portable & encrypted corpora — export/import and the age encryption threat model.
• Document formats — what lore add ingests, and the unsupported/skipped distinction.

Contributing

Contributions are welcome. lore is hexagonal with a strict dependency direction (cli → app → domain), developed test-first, with a conformance-suite contract keeping adapters swappable. Before sending a change, make the validate gate pass:

gofmt -l .        # must print nothing
go vet ./...
go test -race -count=1 ./...

License

Licensed under the Apache License 2.0.