load/

Load tests

Load/scale harnesses for Dexaflow's control plane. The README calls load tests the remaining Phase 6 gap; this directory is where they land, one experiment at a time.

Each experiment is a small package main that drives the real control-plane code (not a mock) against a real Postgres and prints a table. They boot no cluster and no executor, so they run on a laptop with just Postgres up.

Experiments

Only Experiment 1 is built. The others are documented extension points.

Prerequisites

A migrated Postgres with the default tenant (migration 001 seeds it).

# 1. Postgres (leoflow/leoflow on :5432, database `leoflow`)
docker compose -f docker-compose.dev.yaml up -d postgres

# 2. Apply migrations (installs the migrate CLI first if you don't have it:
#    make install-migrate)
export DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable'
migrate -path migrations -database "$DATABASE_URL" up

Any migrated database works — point DATABASE_URL at it. make test-db (the leoflow_test database) is a fine target too.

Experiment 1: scheduler ceiling

The leader scheduler runs Scheduler.Step once per tick (~1s). Step's dominant read is Store.ActiveRuns, which issues 1 + 2N DB queries (one list, then GetDagVersionByID + ListTaskInstancesByRun per run) and N spec unmarshals, then advances every run serially. Experiment 1 measures how one tick's wall-time grows with the number of active runs N, so we can find where a tick approaches the loop interval (the ceiling).

The harness seeds N synthetic DAGs, each with one run pinned running and a running task instance. Such a run stays in the active set every tick while advance stays a no-op for it (nothing to dispatch, nothing to finalize) — so what is timed is the steady-state read + per-run advance cost. N values are seeded incrementally (ascending), so measuring 50,200,500,1000 seeds 1000 DAGs total, not 1750.

DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable' \
  go run ./test/load/scheduler_ceiling --n 50,200,500,1000 --window 5s

Flags:

Output is a table of N → loop-duration p50/p95/p99/max/mean plus the step-down delta observed during each window. The harness cleans up every DAG it created on exit (best-effort); a fresh migrate up on an empty database is always the clean slate.

Known limitation: dexaflow_scheduler_loop_duration_seconds is not wired

The declared SLI dexaflow_scheduler_loop_duration_seconds is defined in internal/observability/metrics.go but is never Observed inside the scheduler loop (grep confirms: only metrics_test.go observes it). A /metrics scrape today would therefore report an empty histogram for it. Until that observation is wired into Scheduler.tick/Step, this harness times Step directly (the faithful equivalent) and also feeds each measured duration into the real histogram object, so a future scrape — or a one-line follow-up that moves the Observe into the loop — sees the same distribution. dexaflow_scheduler_step_downs_total is scraped from the live registry (it stays 0 here: leadership never churns in the harness). dexaflow_dispatch_queue_depth is a dispatcher gauge and is out of scope for Experiment 1 (no dispatcher is wired).

Adding an experiment

1. Create test/load/<name>/main.go as its own package main.
2. Reuse the storage/scheduler/observability packages the way scheduler_ceiling does — connect with storage.NewPostgres, seed via storage.Repository / storage.SchedulerStore, and read metrics from a prometheus.NewRegistry() you own. Do not modify product code.
3. Keep it cluster-free: prefer seeding rows directly over booting executors. If a signal genuinely needs the full server (e.g. the SSE tailer for Experiment 5), boot dexaflow lite the way test/e2e/lite-*.sh does and scrape its /metrics, and say so in this table.
4. Add a row to the Experiments table above and a section here.
5. It must go build, go vet, and gofmt clean, and actually run.