load/

Load tests

Load/scale harnesses for Dexaflow's control plane. The README calls load tests the remaining Phase 6 gap; this directory is where they land, one experiment at a time.

Each experiment is a small package main that drives the real control-plane code (not a mock) against a real Postgres and prints a table. They boot no cluster and no executor, so they run on a laptop with just Postgres up.

Experiments

Experiments 1, 6 and 7 are built. The others are documented extension points. Experiment 7 is the exception to "no cluster": it drives the CLI against a running control plane, and its warm-up half needs a cluster with warm pools on.

Prerequisites

A migrated Postgres with the default tenant (migration 001 seeds it).

# 1. Postgres (leoflow/leoflow on :5432, database `leoflow`)
docker compose -f docker-compose.dev.yaml up -d postgres

# 2. Apply migrations (installs the migrate CLI first if you don't have it:
#    make install-migrate)
export DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable'
migrate -path migrations -database "$DATABASE_URL" up

Any migrated database works — point DATABASE_URL at it. make test-db (the leoflow_test database) is a fine target too.

Experiment 1: scheduler ceiling

The leader scheduler runs Scheduler.Step once per tick (~1s). Step's dominant read is Store.ActiveRuns, which issues 1 + 2N DB queries (one list, then GetDagVersionByID + ListTaskInstancesByRun per run) and N spec unmarshals, then advances every run serially. Experiment 1 measures how one tick's wall-time grows with the number of active runs N, so we can find where a tick approaches the loop interval (the ceiling).

The harness seeds N synthetic DAGs, each with one run pinned running and a running task instance. Such a run stays in the active set every tick while advance stays a no-op for it (nothing to dispatch, nothing to finalize) — so what is timed is the steady-state read + per-run advance cost. N values are seeded incrementally (ascending), so measuring 50,200,500,1000 seeds 1000 DAGs total, not 1750.

DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable' \
  go run ./test/load/scheduler_ceiling --n 50,200,500,1000 --window 5s

Flags:

Output is a table of N → loop-duration p50/p95/p99/max/mean plus the step-down delta observed during each window. The harness cleans up every DAG it created on exit (best-effort); a fresh migrate up on an empty database is always the clean slate.

Loop duration: the harness times Step itself

Since 0.5.4 the product observes dexaflow_scheduler_loop_duration_seconds around every leader tick (Scheduler.tick, #1186), so a /metrics scrape of a running leader shows the real distribution. This harness calls Step directly instead of going through the tick loop, so it times each Step itself and feeds the measured duration into the same histogram object; the two measure the same work. dexaflow_scheduler_step_downs_total is scraped from the live registry (it stays 0 here: leadership never churns in the harness). dexaflow_dispatch_queue_depth is a dispatcher gauge and is out of scope for Experiment 1 (no dispatcher is wired).

Experiment 6: query plans at scale

Most read paths look fine on a laptop with a few hundred runs and degrade only once dag_runs and task_instances hold millions of rows. Experiment 6 seeds a dedicated tenant (load-query-plans) sized like a busy installation and runs EXPLAIN (ANALYZE, BUFFERS) on the hot queries the UI, the API and the maintenance loops issue.

The SQL comes from internal/storage/queries/*.sql at run time, so it always measures what ships. Every EXPLAIN runs in a transaction that is rolled back, so the mutating case (DeleteDag) leaves the dataset as it was. The first run of each case warms the cache and is discarded.

Seeding the default dataset takes a few minutes and adds about 2 GB, so use a throwaway database. The dataset is kept between runs; --reseed rebuilds it and --drop removes it.

DATABASE_URL='postgres://leoflow:leoflow@localhost:5432/leoflow?sslmode=disable' \
  go run ./test/load/query_plans

Flags:

Output is a Markdown table per case: median, best and worst execution time, the time spent in triggers (FK checks and cascades, already part of execution time) and the tables read with a sequential scan. Recorded results live in BASELINE.md.

Experiment 7: code-only deploy and warm-up

A deploy that changed only DAG code should not pay for a build: the registry already holds the dependency image, so the CLI appends the code as one layer through the registry API and registers the new digest (ADR 0077). The target is a p50 under 30 seconds (#1706). With warm pools on, the new version's pool starts when it becomes current (ADR 0082), and its pods pull only the code layer, since the node has the dependency layers already (#1707).

deploy_p50 drives the real dexaflow deploy binary. It scaffolds a project with dexaflow init, gives it pinned dependencies and a registry, and deploys it once cold; that deploy builds the dependency image and is not counted. It then changes a comment line of dag.py before each of --n deploys and times each one from the start of the command to its exit (compile, content label, pre-check, code layer push, digest read and register). Every measured deploy must report builder: code-layer; one that took another path did not measure a code-only deploy, and the run fails.

Prerequisites: a built CLI (make build), the parser on PYTHONPATH, uv on PATH (the cold deploy builds the dependency image from wheels without a daemon), a running control plane and a CLI config logged in to it.

make build
bin/dexaflow auth login --server http://localhost:8080 --username admin@dexaflow.local \
  --password admin --config /tmp/p50-cli.yaml
PYTHONPATH=parser go run ./test/load/deploy_p50 --cli bin/dexaflow \
  --config /tmp/p50-cli.yaml --local-registry --n 20

--local-registry serves an in-memory registry on the loopback inside the harness and builds the scaffold on a stand-in runtime base pushed there (it holds only what the dependency layer reads from a base, and runs nothing). It needs no container daemon, no cluster and no access to a public registry, and it gives the lower bound: the CLI and the register, with no network between the CLI and the registry. It also counts the registry requests of each deploy; on a remote registry each of them costs a round trip. A control plane with executor.type=subprocess is enough for it. Measure against a real registry with --registry <prefix> instead, with the registry login a deploy to it needs. --warm refuses --local-registry: a cluster cannot pull from the harness's loopback.

The warm-up half runs on a cluster with warm pools on, such as the k3d setup of test/e2e/deploy-e2e.sh with DEXAFLOW_E2E_WARM_POOLS=on:

PYTHONPATH=parser go run ./test/load/deploy_p50 --cli bin/dexaflow \
  --config /tmp/p50-cli.yaml --registry k3d-registry.localhost:5111 \
  --platform linux/amd64 --warm --n 10

With --warm the project declares min_idle_workers: 1 and deploys with --active, since only an unpaused DAG that declares a warm floor is prewarmed. After each deploy the harness polls the warm worker pods (leoflow.io/warm-worker=true in --namespace) for one whose image is the digest the deploy registered, and records when it appeared and when it became Ready, from the end of the deploy, with the pull time the kubelet reported in its Pulled event. The project builds on the published runtime base, which the pods run.

Flags:

Output is a Markdown table with one row per deploy (time, builder path, registry requests, and with --warm the warm worker's appearance, Ready time and image pull), the p50, p95 and maximum, and a PASS or FAIL line; the exit code is 1 on FAIL. Recorded results live in BASELINE.md.

Adding an experiment

1. Create test/load/<name>/main.go as its own package main.
2. Reuse the storage/scheduler/observability packages the way scheduler_ceiling does — connect with storage.NewPostgres, seed via storage.Repository / storage.SchedulerStore, and read metrics from a prometheus.NewRegistry() you own. Do not modify product code.
3. Keep it cluster-free: prefer seeding rows directly over booting executors. If a signal genuinely needs the full server (e.g. the SSE tailer for Experiment 5), boot dexaflow lite the way test/e2e/lite-*.sh does and scrape its /metrics, and say so in this table.
4. Add a row to the Experiments table above and a section here.
5. It must go build, go vet, and gofmt clean, and actually run.