runlore

RunLore

An open-source SRE agent that investigates incidents — and remembers what it learns.

RunLore is an open-source SRE agent that investigates any incident — what changed? what's wrong? — and posts a confidence-scored root cause to chat (Slack, Matrix…). It is read-only by default: it reads your cluster, metrics, logs, and network flows — its only writes go to Git, via reviewed PRs.

What sets it apart: it learns your platform. Every investigation opens a PR in a Git repo you own; a human merges it, building a knowledge base of your incidents and context. The same pattern next time gets an instant answer — no fresh investigation.

Learns your platform · single Go binary · runs in your cluster · on your models.

Note: RunLore is read-only by default — it never mutates your cluster. An autonomy ladder (suggest → approve → auto) is on the roadmap for teams that want to go further.

Who it's for — SRE and platform teams who want their incident knowledge portable and self-hosted (no lock-in, your models, your data), and would rather an agent say "I don't know" than guess. It shines if you run GitOps (Flux/Argo CD) — RunLore turns "what changed?" into an exact Git diff — but GitOps isn't required: every data source is pluggable, and an unset one simply disables its tool.

See it in action

A real RunLore investigation delivered to Slack: confidence-scored root cause, the evidence trail, suggested next steps, open questions for a human, and a link to the pull request it opened in your knowledge base.

How it works

flowchart LR
    A["Incident<br/>any alert · event"] -->|"trigger policy<br/>(prod · critical · ns…)"| B
    subgraph B["🔎 Investigate"]
      direction TB
      W["what changed?<br/>deploys · infra · certs · scaling<br/>(GitOps → exact Git diff)"]
      C["what's wrong?<br/>saturation · network · nodes · deps"]
    end
    B --> R["🎯 Root cause<br/>+ confidence + evidence"]
    R --> D["💬 Chat<br/>(Slack · Matrix…)<br/>findings + suggested fix"]
    R -. learn .-> K[("📚 GitHub PR<br/>draft entry in your KB")]
    K -. instant recall .-> B

1. An event fires — a pluggable source triggers RunLore: an Alertmanager webhook, a GitOps failure event, or any adapter you register.
2. RunLore investigates — it reads your cluster, metrics, logs, and network flows.
3. Findings land in chat — ranked root causes with confidence, the evidence trail, and suggested next steps, delivered through a pluggable notifier (Slack, Matrix, a generic webhook…).
4. A PR opens in your KB repo — RunLore drafts what it found as a knowledge-base entry.
5. A human reviews and merges — after adding resolution context, the PR is merged. That entry is indexed: the same incident next time gets an instant answer, no re-investigation.

📐 Detailed architecture: docs/architecture/runlore-architecture.md — the full component diagram (the flow above is the summary).

📚 The learning loop

flowchart LR
    R["🔎 Retrieve<br/>recall a past answer"] --> C["🧪 Capture<br/>record what happened"]
    C --> U["📝 Curate<br/>write the entry (PR)"]
    U --> P["♻️ Compound<br/>merged note re-indexed"]
    P --> R
    classDef s fill:#eef,stroke:#557,stroke-width:1px,color:#113;
    class R,C,U,P s;

The autonomous alert → RCA → chat loop is a commodity. What isn't: a knowledge base that compounds in a catalog you own. Every merged PR becomes a searchable entry — plain markdown in a Git repo you control, PR-reviewed, with full provenance. Knowledge that consistently resolves incidents gains trust; knowledge that keeps failing decays.

→ How the learning loop works · Reviewing & approving knowledge

🔌 Supported integrations

Every backend is pluggable behind an interface — wire what you run; an unset source just disables its tool. GitOps (Flux / Argo CD) anchors the what-changed spine; everything else is optional and additive. Full setup detail in Data sources.

🚀 Getting started

RunLore runs in your Kubernetes cluster as a single Go binary, deployed via Helm. Before installing, you need:

• Data sources — at least one wired source (each is pluggable, an unset one just disables its tool); for the what-changed anchor, a cluster running Flux or Argo CD, plus optionally Prometheus/VictoriaMetrics, VictoriaLogs, Hubble for richer signals
• An LLM — any OpenAI-compatible endpoint, Anthropic, or Gemini (in-cluster or external)
• A knowledge-base repo — a private GitHub repo + a scoped GitHub App; this is where RunLore commits what it learns
• A notification destination — a pluggable notifier: Slack, Matrix, a generic outgoing webhook, or your own

Wire your credentials into a Kubernetes Secret, point the chart at them via a values.yaml (GitOps engine, LLM endpoint, KB repo, notification), and install:

helm install runlore deploy/helm/runlore -n runlore --create-namespace -f values.yaml

Then point a source at RunLore — for example, route your Alertmanager alerts to http://runlore.runlore.svc:8080/webhook/alertmanager — and it starts investigating immediately.

→ Full getting-started guide — KB repo setup, GitHub App, credentials, complete values.yaml reference, data sources, and verification steps.

Prefer to try it without a cluster first?

# fire mocked Alertmanager alerts through the trigger policy (no cluster)
hack/demo.sh

# verify every feature end-to-end on a throwaway k3d cluster
hack/e2e-k3d.sh

Why RunLore

RunLore is GitOps-engine-agnostic (Flux + Argo CD), metrics-backend-agnostic (VictoriaMetrics + Prometheus), with pluggable logs and CNI-agnostic network signals. Change-aware RCA isn't unique — commercial tools (Komodor, Anyshift) diff changes too (prior art) — so the wedge is the combination the open tools don't have: that signal feeding an open, portable catalog you own (OKF-compatible markdown, not a proprietary store), from an agent that's honest about the sub-50% reality — unresolved is a first-class answer, an adversarial verify pass can only ever lower a finding's confidence, and every claim is checked by a shipped eval harness.

Project status & stability

RunLore is pre-1.0 and under active development — interfaces and config may shift between commits. It's usable today, but "stable" means different things across the surface:

• The supported golden path is eval-tested and stable. That's Flux + VictoriaMetrics / Prometheus + an Anthropic or OpenAI-compatible model + a chat notifier (Slack in the eval) + GitHub for the knowledge base. This is the path the nightly eval and the k3d e2e suite exercise — run it with confidence.
• Argo CD is now end-to-end tested, alongside Flux: the k3d suite reconfigures to the argocd engine and drives an Application Degraded failure through a full investigation.
• Functional but less exercised: Matrix, Gemini, the PagerDuty webhook source, cloud integrations, and the network (Hubble) provider. They work and are unit-tested, but see less real-world mileage — expect rougher edges and please file issues.
• The auto autonomy rung is experimental, frozen, and not recommended on real clusters. The supported posture is read-only → suggest → approve: RunLore reads and recommends, a human reviews and merges. Hands-off auto remains on the roadmap, off by default, and should not be pointed at production.

If you stay on the golden path with a human in the approval loop, you're on the surface we test hardest.

Docs

📐 Design · 📚 Learning loop · ✅ Reviewing knowledge · 🚀 Getting started · 🧪 Worked example · 🔌 Data sources · ⚙️ Configuration · 📊 Observability · 🩺 Troubleshooting · 🔒 Security model · 🛡 LLM security architecture · ⬆️ Upgrade & uninstall · 🧭 Prior art · 📊 Benchmarking models · 🛠 Contributing

License

Apache-2.0.