load/

Load tests

Load/scale harnesses for Dexaflow's control plane. The README calls load tests the remaining Phase 6 gap; this directory is where they land, one experiment at a time.

Each experiment is a small package main that drives the real control-plane code (not a mock) against a real Postgres and prints a table. They boot no cluster and no executor, so they run on a laptop with just Postgres up.

Experiments

Experiments 1 and 6 are built. The others are documented extension points.

Prerequisites

A migrated Postgres with the default tenant (migration 001 seeds it).

# 1. Postgres (leoflow/leoflow on :5432, database `leoflow`)
docker compose -f docker-compose.dev.yaml up -d postgres

# 2. Apply migrations (installs the migrate CLI first if you don't have it:
#    make install-migrate)
export DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable'
migrate -path migrations -database "$DATABASE_URL" up

Any migrated database works — point DATABASE_URL at it. make test-db (the leoflow_test database) is a fine target too.

Experiment 1: scheduler ceiling

The leader scheduler runs Scheduler.Step once per tick (~1s). Step's dominant read is Store.ActiveRuns, which issues 1 + 2N DB queries (one list, then GetDagVersionByID + ListTaskInstancesByRun per run) and N spec unmarshals, then advances every run serially. Experiment 1 measures how one tick's wall-time grows with the number of active runs N, so we can find where a tick approaches the loop interval (the ceiling).

The harness seeds N synthetic DAGs, each with one run pinned running and a running task instance. Such a run stays in the active set every tick while advance stays a no-op for it (nothing to dispatch, nothing to finalize) — so what is timed is the steady-state read + per-run advance cost. N values are seeded incrementally (ascending), so measuring 50,200,500,1000 seeds 1000 DAGs total, not 1750.

DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable' \
  go run ./test/load/scheduler_ceiling --n 50,200,500,1000 --window 5s

Flags:

Output is a table of N → loop-duration p50/p95/p99/max/mean plus the step-down delta observed during each window. The harness cleans up every DAG it created on exit (best-effort); a fresh migrate up on an empty database is always the clean slate.

Known limitation: dexaflow_scheduler_loop_duration_seconds is not wired

The declared SLI dexaflow_scheduler_loop_duration_seconds is defined in internal/observability/metrics.go but is never Observed inside the scheduler loop (grep confirms: only metrics_test.go observes it). A /metrics scrape today would therefore report an empty histogram for it. Until that observation is wired into Scheduler.tick/Step, this harness times Step directly (the faithful equivalent) and also feeds each measured duration into the real histogram object, so a future scrape — or a one-line follow-up that moves the Observe into the loop — sees the same distribution. dexaflow_scheduler_step_downs_total is scraped from the live registry (it stays 0 here: leadership never churns in the harness). dexaflow_dispatch_queue_depth is a dispatcher gauge and is out of scope for Experiment 1 (no dispatcher is wired).

Experiment 6: query plans at scale

Most read paths look fine on a laptop with a few hundred runs and degrade only once dag_runs and task_instances hold millions of rows. Experiment 6 seeds a dedicated tenant (load-query-plans) sized like a busy installation and runs EXPLAIN (ANALYZE, BUFFERS) on the hot queries the UI, the API and the maintenance loops issue.

The SQL comes from internal/storage/queries/*.sql at run time, so it always measures what ships. Every EXPLAIN runs in a transaction that is rolled back, so the mutating case (DeleteDag) leaves the dataset as it was. The first run of each case warms the cache and is discarded.

Seeding the default dataset takes a few minutes and adds about 2 GB, so use a throwaway database. The dataset is kept between runs; --reseed rebuilds it and --drop removes it.

DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable' \
  go run ./test/load/query_plans

Flags:

Output is a Markdown table per case: median, best and worst execution time, the time spent in triggers (FK checks and cascades, already part of execution time) and the tables read with a sequential scan. Recorded results live in BASELINE.md.

Adding an experiment

1. Create test/load/<name>/main.go as its own package main.
2. Reuse the storage/scheduler/observability packages the way scheduler_ceiling does — connect with storage.NewPostgres, seed via storage.Repository / storage.SchedulerStore, and read metrics from a prometheus.NewRegistry() you own. Do not modify product code.
3. Keep it cluster-free: prefer seeding rows directly over booting executors. If a signal genuinely needs the full server (e.g. the SSE tailer for Experiment 5), boot dexaflow lite the way test/e2e/lite-*.sh does and scrape its /metrics, and say so in this table.
4. Add a row to the Experiments table above and a section here.
5. It must go build, go vet, and gofmt clean, and actually run.