inference-cache

inference-cache

A vendor-neutral, Kubernetes-native cache-policy control plane for LLM inference.

inference-cache makes routing cache-aware: it tracks which replica already holds a prompt's prefix warm and returns that as a routing hint, so a gateway can reuse KV/prefill instead of recomputing it — cutting time-to-first-token and cost. It orchestrates existing KV-cache technology (LMCache); it is not a new distributed cache and not the data-plane gateway. Guiding principle — "we decide routing; the gateway follows": all routing intelligence lives in the server, and the gateway simply tokenizes → calls LookupRoute → routes to the returned replica → round-robins on NO_HINT.

One request, end to end: an engine serves a prompt and emits KV-cache events → a subscriber sidecar reports them to the server → the in-memory index records (tenant, model, hash_scheme, prefix_hash) → replica → a later LookupRoute returns the ranked replicas holding that prefix warm. The plane moves metadata about where the KV lives — never KV tensors or prompt text. See docs/concepts/ for depth.

Engines and backends

The API separates three choices: spec.runtime selects the inference runtime, spec.type selects its engine-local cache integration, and spec.remoteStorage optionally selects a managed or externally owned remote provider. Supporting another combination is an adapter addition; the core gRPC contract stays stable.

• vLLM + LMCache supports typed PodLocal MP with host-only caching or an optional managed/external Redis L3.
• SGLang + LMCache supports the same typed PodLocal MP and Redis profiles; the engine launch surface remains SGLang-specific.
• SGLang + SGLangHiCache uses SGLang's native host tier and does not accept a remote-storage binding.

See config/samples/ for canonical manifests and docs/design/cachebackend-api.md for the current contract and clearly labeled legacy-compatibility sections retained until Phase 7 of the migration.

What's Inference Cache?

Inference Cache helps LLM serving systems reuse work that has already been computed. It observes which engine replicas hold reusable prompt-prefix KV blocks, maintains a cluster-wide view of that cache locality, and returns ranked routing hints so a gateway can send each request to a replica likely to have the best prefix match.

It also gives operators a Kubernetes-native way to provision shared cache backends, connect them to inference engines, and define cache, tenant, and prompt policies through CRDs. Its cache index stores metadata rather than prompt text or KV tensors, and the cache plane never makes the final routing decision. When no useful hint is available, the gateway follows its normal routing policy: Inference Cache is designed to fail open.

Key Features

• Cache-aware routing hints — ranks engine replicas by reusable prefix locality, including longest-prefix block-chain matches and affinity fallback for requests without an exact match.
• Event-driven cache index — consumes KV-cache events from inference engines and maintains a metadata-only, cluster-wide view of cached prefixes, replicas, and tenants.
• Pluggable engine and backend integration — adapter-based wiring connects supported inference engines to managed or external cache backends without coupling the core API to one implementation.
• Declarative cache policy — Kubernetes CRDs configure backend lifecycle, lookup behavior, tenant isolation and quotas, and expose observed cache state through familiar status and condition surfaces.
• Fail-open request path — returns advisory routing results that gateways may ignore; missing, stale, timed-out, or unusable cache information never prevents the gateway from following its default routing policy.
• Built-in operations surface — provides health and readiness endpoints, Prometheus metrics and alerts, Kubernetes events, functional probes, and the inferencecache doctor diagnostic command.

Repository layout

One operator, split across two control-plane binaries (controller + server) plus the operator CLI and the CRDs.

CRDs — the API

• api/v1alpha1/ — Go types for the six v1alpha1 CRDs (CacheBackend, CachePolicy, CacheTenant, CacheIndex, PromptTemplate, PDTopology) + generated deepcopy
• config/ — generated CRD, RBAC, and sample manifests

inferencecache-controller (cmd/controller) — watches CRDs and provisions cache backends

• cmd/controller/ — controller-runtime manager entrypoint
• internal/controller/ — reconcilers
• internal/adapters/builtin/ — the complete shipping adapter composition used by the controller and webhooks
• pkg/adapters/{backend,runtime}/ — extension contracts; built-in implementations are being migrated under internal/adapters/

inferencecache-server (cmd/server) — gRPC policy server + cache-state index + metrics

• cmd/server/ — gRPC + HTTP server entrypoint
• internal/server/ — gRPC service (LookupRoute, RenderTemplate, …), health, metrics
• proto/ (+ generated stubs) — the gRPC contract
• internal/index/ — cache-state aggregator (CacheIndex)
• pkg/render/ — reserved path for a planned reusable renderer; no stable API yet

kvevent-subscriber (cmd/kvevent-subscriber) — engine-side KV-event ingestion

• cmd/kvevent-subscriber/ — composition and lifecycle entrypoint
• internal/subscriber/ — engine KV-event hook (feeds the index)

Engine client library — narrow pre-tokenized OpenAI-compatible completion client

• pkg/engineclient/ — public EngineClient contract and /v1/completions implementation
• internal/canary/ — repository-owned prefix-cache probe and live canary

inferencecache (cmd/inferencecache) — operator CLI; doctor runs a read-only pre-flight diagnostic

• cmd/inferencecache/ — cobra entrypoint
• internal/cli/doctor/ — diagnostic checks + output formatters (see docs/cli/doctor.md)

Shared — internal/version/, hack/, dockerfiles/, .githooks/

Private cross-binary HTTP DTOs live in internal/controlplaneapi/; pod-binding metadata shared by admission and reconcilers lives in internal/enginebinding/. See Repository boundaries for the dependency rules and staged migration map.

Quick Start

make proto-gen
make build
make test

Run the server locally:

bin/server \
  --grpc-bind-address=:9090 \
  --http-bind-address=:8080 \
  --snapshot-bind-address=:8081 \
  --insecure-disable-auth   # local-dev only; production passes --allowed-controller-sa instead
curl -i http://localhost:8080/healthz   # liveness
curl -i http://localhost:8080/readyz    # readiness
curl -s http://localhost:8080/metrics   # Prometheus metrics (inferencecache_*)
curl -s http://localhost:8081/snapshot  # internal aggregate (auth-gated in production)
curl -i -XPOST http://localhost:8081/policy -d '{"version":3,"policies":[]}'  # controller push (auth-gated in production)
curl -i -XPOST http://localhost:8081/probe \
  -H 'Content-Type: application/json' \
  -d '{"backend":"local/dev","model":"dev-model","hashScheme":"vllm"}'        # functional self-test (auth-gated in production)

The server fails closed by default: omitting both --allowed-controller-sa and --insecure-disable-auth causes it to exit 2 with a stderr message. That keeps an operator who forgets the flag from accidentally shipping unauthenticated /snapshot, /policy, and /probe endpoints on a real cluster.

Cluster Prerequisites

The controller serves admission webhooks — defaulting + validation for CacheBackend, plus a mutating Pod webhook that auto-injects the cache engine configuration (chosen by the matched runtime adapter) into pods labeled to match a CacheBackend's spec.engineSelector — over TLS, so deploying it requires cert-manager v1.0+ in the target cluster. The default install (config/default) provisions a self-signed Issuer plus a Certificate for the webhook serving cert, and relies on cert-manager's cert-manager.io/inject-ca-from annotation to inject the CA bundle into the generated MutatingWebhookConfiguration and ValidatingWebhookConfiguration.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

Install

The default Kustomize overlay brings up both control-plane components:

• inference-cache-controller-manager — the reconciler + admission webhooks.
• inference-cache-server — the gRPC policy server (InferenceCache) and the HTTP /healthz, /readyz, /metrics probe surface, plus a dedicated controller-facing listener carrying /snapshot (controller read), /policy (controller write), and /probe (functional self-test driven by the CacheBackend reconciler), all gated by ServiceAccount bearer auth + a NetworkPolicy. Fronted by a ClusterIP Service inference-cache-server in the inference-cache-system namespace with named ports grpc:9090 (gRPC API), http:8080 (probes / metrics), and snapshot:8081 (controller-only /snapshot + /policy + /probe). The controller's CacheIndex poller scrapes http://inference-cache-server:8081/snapshot, the CachePolicy reconciler POSTs to http://inference-cache-server:8081/policy, and the CacheBackend reconciler POSTs to http://inference-cache-server:8081/probe (default for the --server-probe-url flag — set empty to disable the gate); /snapshot + /probe send the controller-audience projected ServiceAccount token, while /policy sends the write-side policy-audience token. Once both pods are Ready kubectl get cacheindex reports live cluster-wide cache state. FunctionalProbeOK appears on each managed CacheBackend only after it clears the upstream KV-event readiness gate (i.e. real engine pods have published at least one KV event for it) — see docs/design/cachebackend-api.md#functional-probe-gate; it is intentionally not visible on a default install with no engine workload.

Use the digest-pinned manifest attached to the selected GitHub release. The checked-in config/default is a rendering template, not a directly installable bundle; see container image operations for the equivalent source-tree make deploy command.

RELEASE_TAG=vX.Y.Z
kubectl apply -f "inference-cache-${RELEASE_TAG}.yaml"
kubectl -n inference-cache-system wait --for=condition=Available deployment --all --timeout=180s
kubectl get cacheindex cluster-default -o yaml

Monitoring

Both binaries expose Prometheus metrics on their pod's :8080/metrics (prefixed inferencecache_*) — the server binary's series cover the in-memory index and gRPC handlers; the controller binary's series cover the reconcilers (for example, inferencecache_backend_probe_result_total). A default alert bundle for the operational silent-failure patterns this code has hit in production ships under config/observability/ and is not included in config/default — the alerts are opt-in so that installs without prometheus-operator CRDs are not affected by an unknown apiVersion.

For prometheus-operator / kube-prometheus installs:

kubectl apply -k config/observability

This ships FOUR resources: a ServiceMonitor for inference-cache-server:8080/metrics, one PodMonitor for the controller pod's :8080/metrics, one cross-namespace PodMonitor for successfully injected PodLocal LMCache sidecars on their named lmcache-http port, and the PrometheusRule carrying the alerts.

Caveat — Prometheus Operator selectors. All four CRs carry example labels (prometheus: k8s, plus role: alert-rules on the PrometheusRule) that match the upstream kube-prometheus stack (default Prometheus named k8s). The kube-prometheus-stack Helm chart uses a different convention (release: <release-name>, no prometheus: label) — its rule / serviceMonitor / podMonitor selectors do not match what's shipped here. If your Prometheus CR's ruleSelector / serviceMonitorSelector / podMonitorSelector uses a different label set (release: my-prom, etc.), kubectl apply -k succeeds but Prometheus silently ignores the resources. The YAML comments next to each label spell out the introspection command (kubectl get prometheus -A -o jsonpath=...); see docs/observability/alerts.md for the full discussion.

Four of the five Stage 1 alerts (IndexEmpty, LookupRouteDegenerate, LookupRouteHighTimeout, IndexEvictionsSpike) become active as soon as the operator is installed AND the selectors match — they remain quiet on a healthy or idle install (each rule is gated by traffic/ rate/eviction thresholds; see for: + the rate floors in the alert expressions) and only fire when the conditions are met.

The fifth alert needs a vLLM scrape this bundle does NOT ship. LMCacheT2NoHits reads vllm:external_prefix_cache_* from vLLM engine pods directly. The shipped scrape configs do not collect vLLM's own metrics. To make that alert effective, add a separate PodMonitor for your vLLM Deployment (or kubernetes_sd_configs: pod for vanilla Prometheus) so engine /metrics is scraped with both namespace and pod labels attached. See alerts.md "How to enable" for the requirement.

For vanilla Prometheus, ConfigMap mounts, or Helm prometheus.serverFiles, use the flat alerting-rules.yaml. You must also configure scraping yourself for the server, the controller pod, and every injected PodLocal LMCache sidecar. The server's :8080 exposes the index, lookup, and auth series; the controller pod's :8080 exposes the per-stage probe-result counter (inferencecache_backend_probe_result_total); each LMCache sidecar exposes its own lmcache_mp_* series on :8080/metrics — the controller-side alerts (ServerProbeFail today) load against the controller's series, so a server-only scrape leaves them inert.

For multi-install or per-install isolation, use Kubernetes service discovery (kubernetes_sd_configs: pod or endpoints) with relabel_configs: that copies __meta_kubernetes_namespace to namespace — the alerts scope per install by that label. A pair of static DNS scrapes (e.g. inference-cache-server.inference-cache-system.svc.cluster.local:8080 PLUS pod-IP discovery for the controller manager pods, which have no Service in front of them) is acceptable for a single install but loses per-install isolation; do NOT use it if you scrape multiple inference-cache installs into one Prometheus. Without a working scrape on both endpoints, the rules load but fire on nothing (server alerts) or load but never have a series to evaluate (controller alerts like ServerProbeFail).

Per-alert runbooks (causes, triage steps, example PromQL): see docs/observability/alerts.md. For the underlying metric surface, see docs/reference/metrics.md.

Local Development Cluster

Create a kind cluster for controller development:

make dev-cluster

By default this creates or reuses a cluster named inference-cache. You can override the name and node image:

make dev-cluster KIND_CLUSTER=cache-dev KIND_NODE_IMAGE=kindest/node:v1.31.0

Common Targets

• make build: build the controller, server, kvevent-subscriber, and inferencecache binaries
• make test: run unit tests
• make lint: run gofmt and go vet
• make sbom, make sbom-release, make sbom-images, make sbom-registry-images: generate SPDX JSON SBOMs (see docs/operations/sbom.md)
• make verify-minimal-images: verify built images are non-root, shell-free Distroless runtimes
• make ci-lint: run golangci-lint
• make python-lint: lint Python under docs/reference-stack/scripts/ with pinned Ruff
• make verify-prometheus: lint + unit-test the Prometheus alerting rules under config/observability/
• make proto-gen: regenerate protobuf Go code
• make generate: regenerate Kubernetes deepcopy code
• make manifests: regenerate CRD, RBAC, and webhook manifests
• make image-build: build controller, server, and kvevent-subscriber images

Documentation

Docs live under docs/:

• Concepts — docs/concepts/: engine binding, engine overrides, policy tuning, tenant identity/quota, and the CacheIndex cluster aggregate.
• Design — docs/design/: the CacheBackend CRD contract, the InferenceCache gRPC contract, gRPC TLS, policy CRDs, LookupRoute ranking, and KV-event subscriber wiring.
• Reference — docs/reference/: the metric surface and gRPC reason codes.
• Operations & observability — docs/operations/, the alert runbooks in docs/observability/, and the inferencecache doctor preflight CLI.
• Reference stack — docs/reference-stack/: runnable manifests + a GPU runbook for a full vLLM/LMCache stack.
• Quick start — docs/quickstart.md.

Contributor guide: CONTRIBUTING.md (layout, naming rule, push/PR gates).

License

Apache-2.0