runlore

RunLore

An open-source SRE agent that investigates incidents — and remembers what it learns.

📖 Read the docs → runlore.io

RunLore is an open-source SRE agent that investigates any incident — what changed? what's wrong? — and posts a confidence-scored root cause to chat (Slack, Matrix…). It is read-only by default: it reads your cluster, metrics, logs, and network flows — its only writes go to Git, via reviewed PRs.

What sets it apart: it learns your platform. Every investigation opens a PR in a Git repo you own; a human merges it, building a knowledge base of your incidents and context. The same pattern next time gets an instant answer — no fresh investigation.

Learns your platform · single Go binary · runs in your cluster · on your models.

The autonomy ladder. Teams that want more than the read-only default can climb suggest → approve: even at the top supported rung RunLore only executes reversible GitOps operations after an explicit human approval — a human stays in the loop at every step (see Project status).

Who it's for — SRE and platform teams who want their incident knowledge portable and self-hosted (no lock-in, your models, your data), and would rather an agent say "I don't know" than guess. It shines if you run GitOps (Flux/Argo CD) — RunLore turns "what changed?" into an exact Git diff (and, with an opt-in source-repo allowlist, into the offending commit inside an image bump) — but GitOps isn't required: every data source is pluggable, and an unset one simply disables its tool.

See it in action

Two sides of the same incident, delivered to your chat (shown here in Slack; Matrix delivers the same findings) — the whole point of the learning loop in one place.

First time — a full investigation. A verdict-first summary: the actionability call (no action / suggested / required / inconclusive), the confidence-scored root cause, the alert metadata and recurrence, top-cause "why", suggested next steps, ruled-out hypotheses and data gaps, and a link to the pull request it opened in your knowledge base. With the Slack notifier's bot token, the full analysis lands as a threaded reply under that summary. The footer shows the real cost — model calls and tokens.

Next time — an instant recall. Once that entry is merged, the same incident — even under a different, generic alert — is answered straight from your knowledge base: no investigation, no new PR, just the known cause, the human-reviewed resolution, and the entry's resolve-rate track record. An order of magnitude cheaper (two model calls vs a full ReAct loop), delivered in seconds.

How it works

flowchart LR
    A["Incident<br/>any alert · event"] -->|"trigger policy<br/>(prod · critical · ns…)"| B
    subgraph B["🔎 Investigate"]
      direction TB
      W["what changed?<br/>deploys · infra · certs · scaling<br/>(GitOps → exact Git diff)"]
      C["what's wrong?<br/>saturation · network · nodes · deps"]
    end
    B --> R["🎯 Root cause<br/>+ confidence + evidence"]
    R --> D["💬 Chat<br/>(Slack · Matrix…)<br/>findings + suggested fix"]
    R -. learn .-> K[("📚 GitHub PR<br/>draft entry in your KB")]
    K -. instant recall .-> B

1. An event fires — a pluggable source triggers RunLore: an Alertmanager webhook, a GitOps failure event, or any adapter you register.
2. RunLore investigates — it reads your cluster, metrics, logs, and network flows.
3. Findings land in chat — ranked root causes with confidence, the evidence trail, and suggested next steps, delivered through a pluggable notifier (Slack, Matrix, a generic webhook…).
4. A PR opens in your KB repo — RunLore drafts what it found as a knowledge-base entry.
5. A human reviews and merges — after adding resolution context, the PR is merged. That entry is indexed: the same incident next time gets an instant answer, no re-investigation.

📐 Detailed architecture: docs/architecture/runlore-architecture.md — the full component diagram (the flow above is the summary).

📚 The learning loop

flowchart LR
    R["🔎 Retrieve<br/>recall a past answer"] --> C["🧪 Capture<br/>record what happened"]
    C --> U["📝 Curate<br/>write the entry (PR)"]
    U --> P["♻️ Compound<br/>merged note re-indexed"]
    P --> R
    classDef s fill:#eef,stroke:#557,stroke-width:1px,color:#113;
    class R,C,U,P s;

The autonomous alert → RCA → chat loop is a commodity. What isn't: a knowledge base that compounds in a catalog you own. Every merged PR becomes a searchable entry — plain markdown in a Git repo you control, PR-reviewed, with full provenance. Knowledge that consistently resolves incidents gains trust; knowledge that keeps failing decays.

An instant recall is never a blind cache hit — three gates stand in front of it: the entry must structurally match the incident (same workload/resource, retrieval score above a floor), it must win by a clear margin over the runner-up entry (ambiguous matches fall through to a full investigation), and its confidence is weighted by its real-world track record — an entry that keeps resolving incidents gains trust, one that keeps failing decays toward re-investigation. Even then, the recalled finding goes through the same adversarial verify pass as a fresh one; the shipped eval suite includes a poisoned-entry scenario proving a bad entry is rejected at recall time.

→ How the learning loop works · Reviewing & approving knowledge

[!NOTE] What about "PR fatigue"?

The question comes up fast: if a team had no time to document incidents yesterday, who reviews AI-drafted PRs tomorrow? That's the bet, and it is a deliberate one — the review is what separates a memory you own from a dump of LLM output.

The volume is bounded by design. A known incident produces no PR at all (it is served from the catalog — recalled findings are never curated); a duplicate is dropped; an incident that already has an open PR gets a comment on it rather than a new one. Only a novel, verified finding above forge.min_confidence (0.75), carrying evidence and a change-ref or a suggested action, ever becomes a PR.

And nothing says you review by hand. Keep an agent in the loop during the diagnosis itself: have it cross-check RunLore's draft against what you found while resolving the incident, and enrich it with your context. You keep the decision — not the line-by-line reading.

Optionally, put an agent on the queue. The kb-steward skill triages open KB PRs from your terminal — quality and duplicate check per PR, a merge / refine / close call with the concrete fix, and a pointer at the volume levers (forge.skip_verdicts, min_confidence, dup_score) when the queue is systematically noisy. It recommends; you merge. Install is two commands, no binary:

/plugin marketplace add Smana/runlore
/plugin install kb-steward@runlore

🔌 Supported integrations

Every backend is pluggable behind an interface — wire what you run; an unset source just disables its tool. GitOps (Flux / Argo CD) anchors the what-changed spine; everything else is optional and additive. Full setup detail in Data sources.

⚡ Try it in one minute — no cluster, no keys

Before you wire up a cluster, see the front of the pipeline for yourself. This runs lore serve locally with a keyless demo config and fires a batch of mocked Alertmanager alerts at it — no Kubernetes, no LLM, no credentials. You only need Go and curl:

hack/demo.sh

It builds the binary, starts the server, POSTs examples/alertmanager-webhook.json to the webhook, and prints the trigger policy deciding which alerts become incidents:

=== trigger-policy decisions ===
msg=incident alert=HarborProbeFailure severity=critical namespace=apps investigate=true  reason="matched trigger policy"
msg=incident alert=HarborProbeFailure severity=critical namespace=apps investigate=false reason="deduplicated (still-firing)"
msg=incident alert=NoisyWarn        severity=warning  namespace=apps investigate=false reason="filtered by trigger policy"
msg=incident alert=StagingCrit      severity=critical namespace=apps investigate=false reason="filtered by trigger policy"
msg=incident alert=Watchdog         severity=critical namespace=apps investigate=false reason="filtered by trigger policy"

That's one alert admitted (critical + prod), the rest correctly deduped, severity-filtered, environment-filtered, and ignore-listed — the exact gate that controls noise and LLM cost in production. The demo stops there: a full investigation (root cause → chat → PR) needs a real cluster, an LLM, and a knowledge base, which is the production install below. To exercise every feature end-to-end on a throwaway cluster, hack/e2e-k3d.sh spins one up with k3d.

🚀 Getting started (production install)

Ready to point it at real incidents? RunLore runs in your Kubernetes cluster as a single Go binary, deployed via Helm. Before installing, you need:

• Data sources — at least one wired source (each is pluggable, an unset one just disables its tool); for the what-changed anchor, a cluster running Flux or Argo CD, plus optionally Prometheus/VictoriaMetrics, VictoriaLogs, Hubble for richer signals
• An LLM — any OpenAI-compatible endpoint, Anthropic, or Gemini (in-cluster or external)
• A knowledge-base repo — a private GitHub repo + a scoped GitHub App; this is where RunLore commits what it learns
• A notification destination — a pluggable notifier: Slack, Matrix, a generic outgoing webhook, or your own

Wire your credentials into a Kubernetes Secret, point the chart at them via a values.yaml (GitOps engine, LLM endpoint, KB repo, notification), and install:

helm install runlore oci://ghcr.io/smana/charts/runlore -n runlore --create-namespace -f values.yaml

The chart is an OCI artifact on GHCR — no git clone, no helm repo add. It is published and cosign-signed on every release; pin a version with --version X.Y.Z. A minimal starting point for values.yaml is deploy/helm/runlore/values-minimal.yaml. Working from a clone (dev alternative): helm install runlore deploy/helm/runlore -n runlore --create-namespace -f values.yaml.

Then point a source at RunLore — for example, route your Alertmanager alerts to http://runlore.runlore.svc:8080/webhook/alertmanager — and it starts investigating immediately.

→ Full getting-started guide — KB repo setup, GitHub App, credentials, complete values.yaml reference, data sources, and verification steps.

Why RunLore

RunLore is GitOps-engine-agnostic (Flux + Argo CD), metrics-backend-agnostic (VictoriaMetrics + Prometheus), with pluggable logs and CNI-agnostic network signals. Change-aware RCA isn't unique — commercial tools (Komodor, Anyshift) diff changes too (prior art). The wedge is the combination the open tools don't have: that signal feeding an open, portable catalog you own — OKF-compatible markdown, not a proprietary store; as far as we know RunLore is the first agent that produces OKF entries from its own investigations — from an agent that's honest about the sub-50% reality:

• unresolved is a first-class answer;
• an adversarial verify pass can only ever lower a finding's confidence, never raise it;
• every claim is checked by a shipped eval harness, and the nightly results are published — pass, fail, model, and cost included.

Project status & stability

RunLore is pre-1.0 and under active development — interfaces and config may shift between commits. It's usable today, but "stable" means different things across the surface:

• The supported golden path is eval-tested and stable. That's Flux + VictoriaMetrics / Prometheus + an Anthropic or OpenAI-compatible model + a chat notifier (Slack in the eval) + GitHub for the knowledge base. This is the path the nightly eval and the k3d e2e suite exercise — run it with confidence.
• Argo CD is now end-to-end tested, alongside Flux — including the approve rung: the k3d suite reconfigures to the argocd engine, drives an Application Degraded failure through a full investigation, then human-approves a pause-auto-sync action that executes reversibly (the prior syncPolicy.automated is preserved for resume). Both engines share the same reversible-only, allowlisted action envelope.
• Functional but less exercised: Matrix, Gemini, the PagerDuty webhook source, cloud integrations, and the network (Hubble) provider. They work and are unit-tested, but see less real-world mileage — expect rougher edges and please file issues.
• The auto autonomy rung is experimental, frozen, and not recommended on real clusters. The supported posture is read-only → suggest → approve: RunLore reads and recommends, a human reviews and merges. Hands-off auto remains on the roadmap, off by default, and should not be pointed at production.

If you stay on the golden path with a human in the approval loop, you're on the surface we test hardest.

Docs

📐 Design · 📚 Learning loop · ✅ Reviewing knowledge · 🧑‍🔧 KB steward skill · 🚀 Getting started · 🧪 Worked example · 🔌 Data sources · ⚙️ Configuration · 🔗 MCP — server & client · 📊 Observability · 🩺 Troubleshooting · 🔒 Security model · 🛡 LLM security architecture · ⬆️ Upgrade & uninstall · 🧭 Prior art · 📊 Benchmarking models · 🧮 Nightly eval scorecard · 🛠 Contributing

License

Apache-2.0.