e2e

Kafscale E2E Harness

These tests spin up a full cluster (via kind), install the local Helm chart, and hit the console API over a port-forward. They are opt-in because they require Docker/kind/helm on the host and take several minutes.

Prerequisites

1. Docker daemon (Colima, Docker Desktop, etc.)
2. kind, kubectl, and helm binaries on your $PATH
3. Internet access to pull the Bitnami etcd chart (the harness installs a single-node etcd for the operator)

Test Categories and Dependencies

Tests have different dependency requirements. Tests will skip gracefully if their dependencies aren't available.

MinIO Dependency

Tests that require MinIO (produce/consume, durability) will automatically skip if MinIO isn't available:

=== RUN   TestFranzGoProduceConsume
    franz_test.go:42: MinIO not available at http://127.0.0.1:9000; run 'make ensure-minio' first or use 'make test-produce-consume'
--- SKIP: TestFranzGoProduceConsume (0.00s)

To run MinIO-dependent tests:

# Option 1: Use make targets (automatically starts MinIO)
make test-produce-consume

# Option 2: Start MinIO manually, then run tests
make ensure-minio
KAFSCALE_E2E=1 go test -tags=e2e ./test/e2e -run TestFranzGoProduceConsume -v

Running

KAFSCALE_E2E=1 go test -tags=e2e ./test/e2e -v

LFS Go SDK (Kind Cluster)

This test validates the Go SDK against a running Kind cluster (LFS proxy + Kafka + MinIO).

Required environment:

• KAFSCALE_E2E=1
• KAFSCALE_E2E_KIND=1
• KAFSCALE_E2E_BROKER_ADDR (host:port for broker)
• LFS_PROXY_HTTP_URL (full URL), or LFS_PROXY_SERVICE_HOST + LFS_PROXY_HTTP_PORT + LFS_PROXY_HTTP_PATH
• KAFSCALE_LFS_PROXY_S3_BUCKET
• KAFSCALE_LFS_PROXY_S3_REGION
• KAFSCALE_LFS_PROXY_S3_ENDPOINT
• KAFSCALE_LFS_PROXY_S3_ACCESS_KEY
• KAFSCALE_LFS_PROXY_S3_SECRET_KEY
• KAFSCALE_LFS_PROXY_S3_FORCE_PATH_STYLE (optional)

Run:

KAFSCALE_E2E=1 KAFSCALE_E2E_KIND=1 \
KAFSCALE_E2E_BROKER_ADDR=127.0.0.1:9092 \
LFS_PROXY_HTTP_URL=http://127.0.0.1:8080 \
KAFSCALE_LFS_PROXY_S3_BUCKET=kafscale \
KAFSCALE_LFS_PROXY_S3_REGION=us-east-1 \
KAFSCALE_LFS_PROXY_S3_ENDPOINT=http://127.0.0.1:9000 \
KAFSCALE_LFS_PROXY_S3_ACCESS_KEY=minioadmin \
KAFSCALE_LFS_PROXY_S3_SECRET_KEY=minioadmin \
go test -tags=e2e ./test/e2e -run TestLfsSDKKindE2E -v

Note: Running all tests with go test will skip tests whose dependencies aren't available. For complete test coverage, use make test-full.

For local developer workflows, prefer the Makefile targets:

make test-consumer-group          # embedded etcd + in-memory S3
make test-ops-api                 # embedded etcd + in-memory S3
make test-mcp                     # MCP server tests
make test-lfs-proxy-broker        # LFS proxy with fake S3
make test-multi-segment-durability # embedded etcd + MinIO
make test-produce-consume         # MinIO-backed produce/consume suite
make test-full                    # unit tests + local e2e suites

Kind LFS SDK Helper Makefile

The Kind-based SDK test uses lfs-client-sdk/Makefile to orchestrate:

• lfs-demo-up: start the LFS demo stack on Kind (keeps it running)
• pf-start: port-forward broker, LFS proxy HTTP, and MinIO
• test-lfs-sdk-kind: run the Go SDK E2E test

Run all:

make -C lfs-client-sdk run-all

Stop port-forwards:

make -C lfs-client-sdk pf-stop

If you already ran make lfs-demo, ensure port-forwards are up before running the SDK test:

make -C lfs-client-sdk pf-start
make -C lfs-client-sdk test-lfs-sdk-kind

Optional Environment Variables

The harness installs everything into the kafscale-e2e namespace and removes it after the test (unless you reused a cluster).