Lakerunner

Real-Time Telemetry, Straight From Your S3 Bucket

Lakerunner is an event-driven ingestion engine that turns your S3-compatible object store into a blazing-fast observability backend. It watches for structured telemetry (CSV, Parquet, JSON.gz), transforms it into optimized Apache Parquet, then handles indexing, aggregation, and compaction, all in real time. Paired with our Grafana plugin, your bucket becomes instantly queryable — no black boxes, no vendor lock-in.

Features

• Automatic ingestion from S3 on object creation
• Event-driven Architecture enabled by S3 Object Notifications.
• Native support for OpenTelemetry log/metric proto files
• Seamless ingestion for OTEL Proto, CSV or .json.gz formats.
• Works with DataDog, FluentBit, or your custom source!
• Get started locally in <5 mins with the install script
• docker-compose support coming soon!

Table of Contents

• End User Guide
  • Demo Setup
  • S3 Event Setup
  • Next Steps - Get the CLI and Grafana Plugin!
  • Configuration
• License

Demo setup

Prerequisites

1. Docker Desktop, Rancher Desktop, or equivalent
2. kind, minikube, or equivalent (brew install kind or brew install minikube)
3. kubectl (brew install kubectl)
4. Helm 3.14+ (brew install helm)

Install Lakerunner Demo (Laptop Version)

1. Create a local cluster using kind or minikube

kind create cluster

1. Ensure that your kubectl context is set to the local cluster.

kubectl config use-context kind-kind

1. Download and run the Lakerunner install script.

curl -sSL -o install.sh https://raw.githubusercontent.com/cardinalhq/lakerunner-cli/main/scripts/install.sh
chmod +x install.sh
./install.sh

1. Follow the on-screen prompts until the installation is complete.

(We recommend using the default values for all prompts during a local install for the fastest and most seamless experience.)

Explore Data in Grafana

The Lakerunner install script also installs a local Grafana, bundled with a preconfigured Cardinal Lakerunner Datasource plugin.

Wait for ~5 minutes for the OpenTelemetry Demo apps to generate some sample telemetry data. Then, access Grafana at http://localhost:3000 and login with the default credentials:

• Username: admin
• Password: admin

Navigate to the Explore tab, select the Cardinal datasource, and try running some queries to explore logs and metrics.

1. Docker Desktop, Rancher Desktop, or equivalent
2. kind, minikube, or equivalent (brew install kind or brew install minikube)
3. kubectl (brew install kubectl)
4. Helm 3.14+ (brew install helm)

We will discuss configuration options in the next section

Note: Need help? Join the Lakerunner community on Slack or email us at support@cardinalhq.io.

CardinalHQ offers managed Lakerunner deployments.

• You are ready to explore the data! You can use the grafana plugin that is bundled in the helm chart, and optionally setup the Lakerunner CLI from either the releases page, or with brew

brew tap cardinalhq/lakerunner-cli
brew install lakerunner-cli

You just need to set the LAKERUNNER_QUERY_URL and LAKERUNNER_API_KEY urls in the env, and you should be able to explore logs from the CLI!

For a local demo installation with default values, you can run:

export LAKERUNNER_QUERY_URL="http://localhost:7101"
export LAKERUNNER_API_KEY="test-key"

Production Deployment

We are actively working on a production deployment for Lakerunner and will soon be rolling out CloudFormation templates for AWS, Azure Resource Manager (ARM) Templates for Azure, and Google IaC Templates for GCP. Stay tuned!