autoflow/

Autocore Load Test

Stability and scalability test harness for the autocore workflow engine. Deploys worker and generator replicas to GKE against a shared Cloud SQL Postgres instance and Memorystore Redis. Workers run the real autocore engine (shard manager, task processing). Generators drive a weighted mix of Go scenario, Starlark corpus, generated flowgen and adversarial units through the flowtest Driver at a configurable per-replica rate. The two roles scale independently so the submission rate is never bottlenecked by processing capacity.

Prerequisites

• gcloud CLI authenticated with access to the target GCP project
• tofu (OpenTofu) >= 1.6
• kubectl
• docker with buildx (for building and pushing the image)
• pgcli (for database analysis)
• glab

Layout

scripts/loadtests/autoflow/
  cmd/                Go binary entrypoint (autoflow-loadtest)
  metrics/            Metric manifest and capture/report logic (see Measurement)
  modules/loadtest/   Reusable TF module (GKE, Cloud SQL, Redis, obs stack, k8s manifests)
  envs/<name>/        Thin TF root that instantiates the module for one environment
  scripts/            Wrapper scripts (loadtest.sh, analyze.sh, trace.sh, protocol-*.sh)
  manifests/          Workload config template + chaos levels consumed by the TF module
  dashboards/         Grafana dashboards consumed by the TF module (see below)
  Dockerfile          Multi-stage build for the autoflow-loadtest binary

The dashboards are hand-edited JSON. Grafana repairs a broken layout on load rather than rejecting it, so a panel with the wrong gridPos renders somewhere other than where the file puts it and nothing reports it. Run make check-dashboards after editing one; CI runs it too. It wants the panels sorted by grid position, every panel to render where it is placed, and every dashboard to be named in the TF module so it is actually provisioned.

Two envs ship: perf (measurement; pinned workload, chaos off, 960 shards, also what CI gates on under separate TF state) and northstar (24/7 reliability reporting; organic workload, chaos). To add another, copy an env directory and set env_name. Each env has its own TF state and resource names (autoflow-lt-<env>-<rand>) and can be applied and destroyed independently.

Infrastructure Setup

cd scripts/loadtests/autoflow/envs/perf

glab tf init -R gitlab-org/cluster-integration/gitlab-agent autoflow-loadtest-perf
tofu apply

This provisions:

• GKE Standard cluster (zonal, 2x n2d-highcpu-32 nodes)
• Cloud SQL Postgres 17 (Enterprise Plus, C4A Axion)
• Memorystore Redis 7.2
• Artifact Registry repo
• IAM + workload identity bindings
• Docker image build and push
• Observability stack (Prometheus, Tempo, Grafana) with autocore dashboard

The Cloud SQL layer hosts three kinds of logical databases: autocore's central DB (deliberately write-cold coordination data), the workflowN DBs (the hot path; one per instance in the isolated topology), and the ledger DB, where the generator records every driven session and its expected outcome for post-run verification. The ledger is a separate logical database on the central instance so the test observer adds no write load to the write-cold central schema; a dedicated instance is the escalation path if contention shows.

Deploy the Load Test

# Configure kubectl
$(tofu output -raw kubeconfig_command)

# Apply all manifests (ServiceAccount, Secret, ConfigMap, Deployment)
tofu output -raw kubernetes_manifests | kubectl apply -f -

Observability

Access Grafana (pre-configured with Prometheus and Tempo datasources + autocore dashboard):

kubectl port-forward svc/grafana 3000:3000
# Open http://localhost:3000 (anonymous admin, no login needed)

Access Prometheus targets:

kubectl port-forward svc/prometheus 9090:9090
# Open http://localhost:9090/targets

Workloads

This deploys two workloads from the same image:

• autoflow-loadtest-worker (Deployment, --mode=worker) -- runs the real autocore engine (shard manager + workers), idles until terminated.
• autoflow-loadtest-generator (--mode=generate) -- drives the workload mix from manifests/workload-config.yaml through the flowtest Driver: Go scenario units against autocore directly, and Starlark corpus, generated flowgen and adversarial units through the flowcore engine.

Flows can call the compiled-in testkit AutoFlow module (echo, delay, record, counter). record lands a generator-unique side-effect token in the ledger's loadtest_token table with the token as the destination dedup key, and the oracle's side-effect-tokens-exactly-once invariant compares each terminal session's recorded tokens against its expectation, making lost or duplicated side effects visible. counter increments a session-scoped ledger counter and returns its new value on every attempt, which is what a generated poll block waits on. Module actions execute on shard-owning workers, so the worker Deployment carries --ledger-dsn too; a worker without it fails record and counter visibly instead of silently dropping the side effect.

Flows also call the real built-in gitlab and event modules, served from the in-process stub backend (internal/tool/testing/flowtest/stubbackend): gitlab.call_api against an HTTP server that answers any request deterministically, event.emit into a counting publisher. Neither a GitLab nor an events platform is deployed, and a non-2xx status is data rather than a failure, so a flow's outcome stays known.

The generator's shape is selected by the injection_duration TF variable:

• empty (default) -- a 24/7 Deployment (generator_replicas replicas) that produces sessions until scaled down.
• set (e.g. "5m") -- a bounded Job (generator_replicas parallelism/completions) that passes --injection-duration and --wait-for-shards=10m to each replica; the replicas stop producing new sessions after the duration, finish in-flight drive scripts, and exit. This is how CI runs a fixed-length load.

Switching between the two shapes renames the workload kind and kubectl apply does not prune the previous kind, so delete the old Job/Deployment manually (or via loadtest.sh) when flipping.

Scaling

# Scale workers and generators independently
kubectl scale deployment autoflow-loadtest-worker --replicas=16
kubectl scale deployment autoflow-loadtest-generator --replicas=4

# Watch logs by role
kubectl logs -f -l role=worker --max-log-requests=10
kubectl logs -f -l role=generator --max-log-requests=10

# Watch all
kubectl logs -f -l app=autoflow-loadtest --max-log-requests=20

Running a Loadtest

Use scripts/loadtest.sh to kick off a fresh run. It handles cleanup, database reset, image rebuild, and deployment. The --env flag selects the target env (default perf):

cd scripts/loadtests/autoflow
./scripts/loadtest.sh [--env perf] [--skip-build] [--shards-total <n>]

This will:

1. Scale workers and generators to 0 and delete generator jobs
2. Run migrations down + up to reset the database
3. Rebuild and push the Docker image (unless --skip-build)
4. Deploy all manifests
5. Print the start timestamp for use with analyze.sh

--shards-total re-applies the env with a different shard count for one run; see Shard sweeps below for which envs support it.

Post-Loadtest Analysis

cd scripts/loadtests/autoflow
./scripts/analyze.sh --env perf --start '<start-time>' --end '<end-time>' --grafana-port 3000

Verify (oracle invariants)

--mode=verify evaluates the flowtest oracle's invariant registry (internal/tool/testing/flowtest/oracle) against a finished run: the ledger checks reconcile the driver's recorded expectations with the observer's recorded outcomes (allowed terminal states, exactly-once side-effect tokens, no lost sessions, recording consistency), and the history checks read autocore's own databases (gapless history sequences, terminal-event placement, state/history agreement, at most one pending workflow task, empty queue tables post-drain, child shard placement, directory/execution reconciliation).

Verify never constructs the autocore engine and never runs migrations; it only opens read paths. It discovers the workflow databases from the central workflow_database table and opens each one's runtime DSN keyed by its db_id, so it always covers exactly the deployment's databases. The Markdown report always goes to stdout; --verify-report <path> additionally writes the JSON report. Exit code 0 means every invariant passed; violations (or evaluation errors) exit 1. Warnings and skipped invariants never fail a run.

--since/--until (RFC3339) bound verification to the sessions and workflows created in that window, so a shared environment's older runs stay out of scope. When --since is omitted, verify derives it from the earliest ledger session: loadtest.sh resets the ledger with the other databases, so that is the run epoch. Queue emptiness is global by nature and ignores the window.

--prometheus-url (optional) enables the metrics checks: workflow creations must conserve against the terminal transitions Prometheus recorded over the run window (post-drain only), and coverage warnings surface a run that exercised no cold replays, activity retries, or creation dedup hits. When unset, metrics checks are skipped. The checks need a bounded window (an explicit or derived --since) and read metrics up to the evaluation instant rather than --until, so terminals recorded during the drain tail stay in range; on a shared environment, creations injected after --until land in the same range and can distort conservation. verify.sh defaults the URL to the in-cluster Prometheus service (http://prometheus:9090), which the pod reaches directly -- only out-of-cluster tools like analyze.sh need the Grafana datasource proxy.

Run it in-cluster as a one-off pod (the central and ledger DSNs come from the autoflow-loadtest secret, like the generator):

cd scripts/loadtests/autoflow
./scripts/verify.sh --env perf

Or run the binary directly against an env's databases:

ENV_DIR=scripts/loadtests/autoflow/envs/perf
CENTRAL_DSN="$(cd "$ENV_DIR" && tofu output -raw central_dsn)"
LEDGER_DSN="$(cd "$ENV_DIR" && tofu output -raw ledger_dsn)"
go run ./scripts/loadtests/autoflow/cmd --mode=verify \
  --dsn "$CENTRAL_DSN" --ledger-dsn "$LEDGER_DSN"

Pass --post-drain=false when the run is still in flight so liveness checks that assert queue emptiness are skipped instead of misfiring.

CI runs verify after the drain (autoflow:loadtest:02:verify) as a non-blocking step (allow_failure: true) while the checks soak, windowed to the run job's START_TIME/END_TIME; flipping it into a gate is a later phase.

Inspect a Single Trace

cd scripts/loadtests/autoflow
./scripts/trace.sh <trace-id> --grafana-port 3000

Manual Reset (alternative to loadtest.sh)

If you need to reset without rerunning the full script:

ENV_DIR=scripts/loadtests/autoflow/envs/perf

# Scale down
kubectl scale deployment autoflow-loadtest-worker --replicas=0
kubectl scale deployment autoflow-loadtest-generator --replicas=0
kubectl delete job autoflow-loadtest-generator

# Reset databases via migrations
CENTRAL_DSN="$(cd "$ENV_DIR" && tofu output -raw central_dsn)"
LEDGER_DSN="$(cd "$ENV_DIR" && tofu output -raw ledger_dsn)"
mapfile -t WORKFLOW_DSNS < <(cd "$ENV_DIR" && tofu output -json workflow_dsns | jq -r '.[]')
for dsn in "${WORKFLOW_DSNS[@]}"; do
  make run-migrations autocore_workflow down MIGRATION_DSN="$dsn"
done
make run-migrations autocore_central down MIGRATION_DSN="$CENTRAL_DSN"
make run-migrations loadtest_ledger down MIGRATION_DSN="$LEDGER_DSN"
make run-migrations autocore_central up   MIGRATION_DSN="$CENTRAL_DSN"
for dsn in "${WORKFLOW_DSNS[@]}"; do
  make run-migrations autocore_workflow up MIGRATION_DSN="$dsn"
done

# Redeploy
(cd "$ENV_DIR" && tofu output -raw kubernetes_manifests) | kubectl apply -f -

Measurement

This section covers comparing two builds against each other on perf: what is measured, how a point is validated, and what a comparison is allowed to claim. The scripts below read Prometheus through the Grafana datasource proxy (scripts/lib/prom.sh, default http://localhost:${GRAFANA_PORT:-3000}), so run kubectl port-forward svc/grafana 3000:3000 (or set PROM_URL) first. An unreachable Prometheus fails the script hard rather than reading as zero.

Vocabulary

Term	Definition
Point	One protocol run, one measurement: results/<label>.json plus its provenance block. It either passes that protocol's gates or writes invalid: true with invalid_reasons and exits 1.
Arm	The set of points for one build or config. --mode=report reads the base arm before its -- and the tip arm after, each selected by glob out of one flat results/ directory.
Campaign	One sitting with topology, shard count, generator rate, protocol and window all held fixed, collecting points by alternating arms until each has enough of them, and ending in a single --mode=report; scripts/campaign.sh runs one end to end. Nothing records which campaign a point belongs to, so the per-file refusals in --mode=report and the provenance block are all that stand between a pinned value that moved between sittings and a published number.
A/A campaign	A campaign whose two arms are the same build. Every delta it reports is noise by construction, which is what makes the noise floors in manifest.yaml measured rather than inherited.
Noise floor	A metric's noise_floor_pct: its measured repeatability as a percentage, below which a delta is not a finding. Zero means the floor has not been measured, and the check is then skipped rather than defaulted, so an undeclared floor is permissive. Two of the 36 metrics declare one today.
Resolvable finding	A comparison whose delta clears the metric's noise floor, whose arms hold at least two points each, and whose separation (`

Comparing two revisions

./scripts/compare.sh --base origin/master --tip my-branch

That is the whole thing. It refreshes the state backend's short-lived credentials, points kubectl at the cluster, resumes the environment, opens the Grafana port-forward the protocols read Prometheus through, picks the protocol and its flags, runs the campaign and prints a verdict. It will not create the environment, because that is a GKE cluster and three Cloud SQL instances; if perf is not provisioned it prints the apply command and stops.

Three profiles set the protocol and the point count together, so the choice is one word rather than four flags:

Profile	What it runs	Answers
default	drain, 5 points per arm	Capacity. The only protocol that can show a throughput gain.
--quick	matched-rate, 3 points per arm	Cost per unit of work and latency, sooner. Cannot show throughput.
--thorough	drain, 8 points per arm	The same question as the default, with a narrower interval, so smaller effects resolve.

--quick sizes matched-rate's workflow target from the throughput the environment is currently showing, because the protocol's own default is tuned for the measurement topology and would wait tens of minutes on a smaller one.

The campaign id defaults to the two revisions and the date, so re-running the same command after an interruption resumes rather than restarting, and --suspend shuts the environment down when the comparison finishes. --verdict-only --campaign <id> re-reads a finished comparison without touching the cluster.

The verdict says different things depending on what there is to say. With findings, it lists them and then names any whose metric has no measured floor, because those cleared the t-test without anything establishing that they are larger than the rig's own noise; the command to fix that is printed with the revision already filled in. With no findings, it reports the smallest effect the run could have called on its most sensitive metrics, so a null result bounds what could have hidden behind it instead of saying nothing:

  No resolvable difference between the two revisions.

  The smallest effect this run could have called, on its four most
  sensitive metrics:
    commits                                  +/- 1.93%
    workflow_rounds                          +/- 2.48%
    schedule_to_start p99                    +/- 16.36%

It also notices two ways floors go wrong: floors measured more than 30 days ago, and floors that changed after the comparison was computed, which means the verdict was reached against the old ones and the report needs re-judging. Where the floors are current and every finding exceeds its own, it says that too, in one line. A reminder that prints on every run is one nobody reads.

Stored baselines

A comparison measures ten points to answer a question about one revision, and the base arm is always master. baseline.sh publishes a single revision's points to the package registry so that arm can be fetched instead of measured:

./scripts/baseline.sh publish --campaign <single-revision campaign>
./scripts/compare.sh --baseline --tip my-branch

That halves a comparison, and means nobody has to establish their own master baseline before asking a question. The artifact is the raw point files, because the statistics need per-point values, plus the protocol, window, environment shape, revision, capture date and harness ref they are only valid under. The version names the protocol, a hash of the shape and the revision, since a baseline is comparable only against a campaign that shares all three.

A stored-baseline comparison is cross-session, and that is a real weakening. Its arms were captured on separately applied environments days apart, so they carry variation a same-campaign comparison does not, and the floors in manifest.yaml, measured within a single campaign, understate it. The driver labels such a comparison in its output rather than leaving that to memory. The between-session floor is not measured yet; once baselines are published on a schedule, consecutive ones are near-A/A pairs and the number falls out of data already being collected.

Publish and compare in matched-rate. Drain measures capacity, which is the quantity most sensitive to which cluster the scheduler happened to hand out, so a drain baseline compares two environments as much as two builds. compare.sh warns when a baseline is used with any other protocol.

Metric manifest

metrics/manifest.yaml declares the 36 metrics a comparison can draw on, validated against the schema in metrics/manifest.proto. Each metric has a tier: outcome, cost, saturation and latency metrics are the ones a finding quotes; backlog metrics feed the drain protocol below; gate metrics are meant to flag a run as untrustworthy, but today only sessions_skipped is actually checked, by the matched-rate protocol (see The A/A gate below). The other five gate metrics are captured and reported like any other metric, with nothing screening them out of a comparison -- see Known gaps.

--mode=capture resolves every declared metric against Prometheus and refuses to run if a declared metric resolves to no series, rather than reporting a null indistinguishable from the feature being switched off. A histogram is checked through its _count series, because a histogram family exposes only _bucket, _sum and _count and no series carries the bare family name. A histogram whose +Inf bucket holds more than 5 percent of samples has its quantiles marked unreliable instead of reporting a bucket boundary as a latency.

Two manifest flags carve out the metrics that legitimately have no series. structurally_zero marks one this environment cannot populate at all (wal_sync_time, because CloudSQL forbids track_wal_io_timing); it is never queried. absent_means_zero marks a counter incremented only on failure paths (sessions_skipped, invariant_violations, system_failures, workflow_aborts, shard_fenced): an OTel synchronous counter emits nothing until its first Add, so a run in which nothing failed exports no series at all, and requiring one would accept only runs the engine already misbehaved on. Those are still queried and are skipped only by existence validation; an empty result records as zero carrying a note that the series was absent, so a defined zero stays distinguishable from a measured one.

Protocols

Four protocols answer four different questions, all built on --mode=capture. Compare builds on drain. It is the only one that can show a throughput gain.

• drain (scripts/protocol-drain.sh) saturates to fill the backlog to a fixed depth, stops arrivals, waits for the backlog to fall through a second fixed depth, then counts for a fixed window with no arrivals. Answers "what is this build's service capacity." This measures the engine flat out, at its ceiling, which is where an optimization turns into more work done.

• matched-rate (scripts/protocol-matched-rate.sh) pins the generator below saturation and waits for a target cumulative workflow count before capturing, so table size is pinned too. Answers "what does this build cost per unit of work, and at what latency, at a known rate." Comparable against a stored reference without a paired arm, which makes it the cheap routine loop.

  It cannot show a throughput improvement, by construction. Its rate is pinned below capacity, so throughput equals the offered rate no matter how fast the engine is: a measured point had 1897 arrivals/s against 1896 terminals/s. A faster build returns the same throughput and moves only its cost and its latency. Read cost per workflow here, never capacity.

• profile (scripts/protocol-profile.sh) saturates the generator deployment to a fixed backlog depth and holds it there, arrivals included, for the whole capture window, then samples pg_stat_activity wait events (scripts/sql/waits.sql) against every workflow database while still saturated. Answers "what is the system waiting on when it is flat out." At a matched backlog depth it is comparable between builds for latency, pool and wait attribution, because the database is in the same state.

  It is not a capacity measurement, for the same r = -0.99 reason as below. A faster build sitting at the same depth can show the same or a lower rounds/s than a slower one, so read wait attribution here, never throughput; use drain for that. It also cannot simply run inside drain's window: drain's counting window deliberately has no arrivals, so a wait profile sampled there would describe a queue draining with nothing new arriving, not the steady-state operation this exists to characterize.

• aa is not a separate script: run any protocol twice against one unchanged build to measure the instrument's own noise. This is the gate described below.

Saturation belongs inside drain's fill, not in the environment's standing configuration. Holding a rate slightly above capacity sounds like the way to expose a ceiling, and it does not work here: throughput tracks backlog depth at r = -0.99, so exceeding capacity deepens the backlog, which lowers capacity, which deepens it further. There is no stable operating point just above the ceiling. Two runs of the same build once read 11.2 percent apart that way, and the observable of an optimization degrades to "the backlog grows more slowly", which is a derivative of the most confounded quantity on the rig while the tables grow throughout. Drain gets saturation without the feedback loop by removing arrivals from the window. Profile is the deliberate exception: it holds saturation for the whole window because that standing state is exactly what it characterizes, and because it never reports capacity, the feedback loop above does not corrupt its answer.

Because the campaign rate is deliberately sub-saturation, drain scales the generator deployment up to --fill-generators (default 10) for the fill, then to zero for the count, then back to --generators on exit. At the campaign rate alone the standing backlog is a few hundred rows and --fill is unreachable. Profile scales up to --fill-generators the same way, but never back to zero: it stays at --fill-generators for the whole window and returns to --generators only on exit.

All three scripts run preflight.sh first and write one result to results/<label>.json plus its provenance to results/<label>.provenance.json. Profile additionally writes one wait-event sample per workflow database to results/<label>.waits-<n>.txt.

All three also pass their window end to --capture-end, which pins every measurement query to that instant through the API's time parameter. The capture pod is created after the window closes, so without the anchor each increase(...[$WINDOW]) would be evaluated whenever the pod finally reached Prometheus: pod scheduling and image pull alone move that by tens of seconds, which on this rig lands directly in the result and gets attributed to the build. The series-existence probes are deliberately left unanchored, because whether a metric is still exported is a question about the retention window rather than about the measurement window.

All three also record the requested window rather than one derived from the two timestamps that bracket it. date truncates to whole seconds and sleep overshoots, so a derived window is --window or --window plus one depending on sub-second phase, and --mode=report refuses a campaign whose points disagree on window_seconds. Drain still compares its derived elapsed time against the requested one as a separate check.

--mode=report compares two arms of result files already on disk:

autoflow-loadtest --mode=report <base results...> -- <tip results...>

Paths before -- are the base arm, paths after it are the tip arm; either arm can be one or several files. No wrapper script exists for this mode, so this is the only place its invocation is written down. It refuses outright when the files disagree on protocol or on window_seconds: every counter metric is an absolute increase() over the capture window, so a 60-second drain file against a 300-second matched-rate file would report roughly 400 percent deltas with tight standard errors on about twenty metrics. Arms are picked by shell glob out of one flat results/ directory, so that mistake is easy to make. Otherwise it prints one comparison per metric (per quantile or label combination, for a histogram or grouped query) with both arms' point count, mean and standard error, the delta, and whether the delta clears both the metric's noise floor and a Welch t-test against the 95% critical value for the comparison's degrees of freedom to count as a resolvable finding. An arm with a single point carries no repeatability estimate and is refused outright rather than compared.

Campaign driver

scripts/campaign.sh runs a whole campaign: point it at two revisions and it builds both arms, resets once, alternates points and prints the comparison. compare.sh above wraps it and arranges the environment first; reach for the driver directly when you want to control the protocol's own flags.

./scripts/campaign.sh --base origin/master --tip my-branch --points 5

Each arm is built as that revision's engine code plus a fixed harness overlay: the capture binary, the manifest and the protocol scripts all come from --harness-ref (default HEAD) on both arms. Without the overlay an arm would measure itself with its own copy of the instrument, and origin/master could not be an arm at all, having no capture mode. The consequence is that the harness must compile against both revisions; where it does not, the arm build fails loudly.

Arms alternate in balanced pairs (base tip, then tip base) so table growth and environmental drift are shared rather than attributed to whichever arm ran second. The balance matters for anything cumulative: under strict alternation the tip arm is always half a point later in the campaign, which cancels for a metric that varies randomly and not at all for one that only grows. A switch is kubectl set image plus a rollout rather than a rebuild, so a campaign pays for two builds instead of two per point. The arm images are tagged campaign-<id>-base and campaign-<id>-tip, and :latest is pointed at the base arm because the protocols run --mode=capture from it, which makes that tag the instrument rather than a subject. kubectl set image diverges from tofu's rendered manifests, so re-applying them mid-campaign moves both arms onto whatever :latest is.

--pause decides when the driver stops for inspection. on-invalid (the default) prompts only when a point produced no valid measurement, offering retry, skip or abort; after-point prompts after every point; never aborts instead of prompting, which is the mode for an unattended run and the only one that works without a terminal. The pause sits after a point and before the next mutation, because an arm switch restarts the workers and rescales the generators, and because Tempo keeps traces for only 3h.

A campaign is resumable: the same --campaign <id> keeps the valid points already on disk, reuses the arm images already in the registry and takes only what is missing. An invalid point is not kept, so a resume re-measures it. --report-only --campaign <id> re-renders a finished campaign and touches neither the cluster nor the registry.

Two refusals worth knowing. Arms whose migrations differ are refused, because a campaign resets once and the union of both schemas then outlives the arm that introduced it; --allow-migration-drift accepts that deliberately. And the generator must be a Deployment, because a bounded env's Job cannot have its image swapped between arms.

The A/A gate

No comparative claim may be published on this environment until an A/A campaign has run and its measured noise floor is written back into manifest.yaml. Without a known noise floor, "build B is N percent faster" is unfalsifiable: there is nothing to compare N against.

That is not a theoretical worry here. The first A/A at the measurement topology, ten drain points of origin/master against itself, produced two resolvable findings: claim_cycle_duration p90 at -5.78 percent (3.4 SE) and wal_bytes at -13.35 percent (2.6 SE), both clearing the 2.31 critical value at that sample size. Re-judged against the floors that same campaign produced, the first is suppressed outright and the second is flagged as sitting within 1.5x its floor. A 95th-percentile floor is defined to let roughly one split in twenty through, so it bounds false positives rather than eliminating them.

./scripts/calibrate.sh --rev origin/master

That runs a campaign with one revision on both arms and writes the result into metrics/manifest.yaml. It does not read the campaign's own single delta per metric, which would be one sample of the quantity being estimated. It splits the points into two arms every way it can and takes the 95th percentile of the resulting delta distribution, so ten points yield 252 splits rather than one. The same estimate is available on its own through --mode=floor <point files...>, and --from-campaign <id> re-derives floors from points already on disk, which is how to calibrate from an ordinary comparison whose two revisions turned out to be indistinguishable.

A metric's floor is its widest series, because the manifest carries one floor per metric while a comparison reports one row per quantile and label set. Expect some metrics to come back too wide to ever resolve anything; those are diagnostic-only, and knowing which ones is half of what this buys.

Floors are written with a floor_provenance block recording the environment, protocol, window and date they were measured under, because a floor is a property of an instrument in an environment running a workload under a protocol, not of the metric. It expires when any of those move: the environment's shape, the workload, a manifest query, the protocol, the infrastructure underneath, or enough elapsed time that the cloud has drifted. A change to the engine can widen a metric's spread without moving its mean, so a large accepted change is a reason to re-check too. --mode=report reports each comparison against the floor it was judged by, and a campaign whose finding sits within 1.5x its floor says so rather than presenting it as a result: the floor is an estimate with its own spread.

preflight.sh is the precondition gate, run automatically by all three protocol scripts and usable standalone:

./scripts/preflight.sh --env perf [--expect-shards 960] > provenance.json

It asserts, against the live cluster rather than the git tree (manifests are not re-applied between runs, so the two can disagree): chaos is genuinely absent, the deployed workload is pinned (swarm: false, rate min == max), every worker runs one image digest with zero restarts, and Prometheus is reachable. It emits the provenance block (git SHA, image digest, workload config hash, replica counts, the deployed workload pool size, and the per-database shard split used to compute shard_skew_ratio) as JSON on stdout. A nonzero exit means do not measure.

A point can be invalid rather than interesting, and each protocol checks different things: matched-rate marks a point invalid when the backlog moved by more than 1000 rows and by more than 25 percent during the window (not actually matched-rate after all), when sessions_skipped is nonzero (the offered rate was not delivered), or when either sessions_started or workflow_rounds over the window is zero or absent (the window did no measurable work); drain marks a point invalid when the measured elapsed time differs from the requested window, or when workflow_rounds is zero; profile marks a point invalid when workflow_rounds is zero, or when the backlog read at the window start fell short of --depth (a fresh check, independent of the fill loop's own gate, against a scrape that lagged behind it).

The throughput check is the counterpart to drain's and profile's zero-rounds check, and neither of matched-rate's other two gates can replace it. A generator fleet that dies inside the window skips no sessions, so the offered-rate gate stays quiet; and a shallow queue draining to empty stays under the backlog gate's row floor. Without it, a window with almost no arrivals published as valid, carrying a cost-per-workflow ratio computed against almost nothing.

The backlog gate needs both halves. n_live_tup is an estimate that legitimately reads 0 on a shallow queue between autovacuum passes, and a ratio against 0 passed any end depth, so a point that started empty and ended 50,000 deep published as matched-rate. Symmetrically, a small start made the ratio hypersensitive: 5 to 12 rows read as 140 percent growth and invalidated a point that was flat. All three write invalid: true with invalid_reasons and exit 1; an invalid point is discarded, not reported with caveats. None of the three protocols checks any other TIER_GATE metric -- see Known gaps.

At a 3 percent per-point noise floor, resolving a single-digit percent effect takes roughly five points per arm; fewer points push the required separation up toward the wider critical value a small-sample Welch t-test demands. Budget wall-clock time accordingly before promising a same-day answer.

Why fixed-time-offset capture cannot measure capacity here

A fixed-time-offset capture -- start a run, wait a fixed offset, count what happened -- looks like it measures throughput, but under saturation it measures the backlog instead: across 11 runs, throughput tracked backlog depth at r = -0.99, and two runs of the same unchanged build read 11.2 percent apart. The backlog cannot be regressed out of the result either: at a fixed arrival rate, a faster build drains faster and leaves a shallower backlog, so subtracting the backlog's effect subtracts the very signal being measured. This is why the drain protocol removes arrivals entirely instead of trying to correct for them.

Wait-event sampling

protocol-profile.sh runs this automatically, once per workflow database, while the environment is held saturated. To sample by hand instead -- scripts/psql-file.sh runs a local .sql file against a loadtest database from inside the cluster; direct egress to Cloud SQL is blocked from a laptop, and the pooled connection port rejects the session-level statements this needs:

./scripts/psql-file.sh --secret-key WORKFLOW_MIGRATION_DSN_1 --file scripts/sql/waits.sql

The WORKFLOW_MIGRATION_DSN_<n> secret key is required, not the pooled WORKFLOW_DSN_<n>: the sampler runs session-level statements (pg_stat_clear_snapshot()), which Cloud SQL's transaction-mode pooling forbids.

scripts/sql/waits.sql repeatedly snapshots pg_stat_activity to attribute active backend time to wait events and idle-in-transaction backends to their last statement. It is the instrument that measured LWLock:WALWrite at 57 percent of active backend time, the basis for the manifest's active_backend_waits entry.

Shard sweeps

loadtest.sh --shards-total <n> re-applies the environment with a different total shard count for one run:

./scripts/loadtest.sh --env perf --shards-total 480

This only works on an env whose root module declares workflow_shards_total as a variable; today that is perf only (northstar hardcodes the value on the module call instead). tofu silently ignores a TF_VAR_* for a variable the target root module has not declared, so pointing this flag at an unsupported env would apply with no error and no effect; loadtest.sh checks the env's main.tf for the declaration first and fails hard instead.

Operating perf

Treat perf as suspended by default between campaigns: nothing suspends or resumes it automatically (CI's own 03:suspend job is a manual step against its own, separately-suffixed state, not this one), so a session should never assume the Cloud SQL instances or GKE node pool are already running. scripts/resume.sh --env perf is the first step of any campaign session; scripts/suspend.sh --env perf is the last, to avoid paying for idle compute in between. Both are idempotent, and TF state and database contents survive a suspend/resume cycle.

Recovering an interrupted destroy

A full teardown takes long enough to be interrupted, and an interrupted tofu destroy leaves the remote state lock held. Every later command then fails with Error acquiring the state lock, reporting Operation: OperationTypeApply and a lock ID. Break it with the ID from that message and re-run the destroy:

tofu force-unlock <lock-id>
tofu destroy -auto-approve

destroy is resumable: it re-reads state and continues from wherever it stopped.

A destroy can also stop on the VPC with The network resource ... is already being used by ..., and re-running fails identically because the thing holding the network is not in state. Two kinds of object outlive what tofu tracks: the network endpoint groups GKE creates for ingress backends, and the VPC peering Memorystore creates for Redis. Delete them, then destroy again:

gcloud compute network-endpoint-groups list --project <project>
gcloud compute network-endpoint-groups delete <name> --zone <zone> --project <project>
gcloud compute networks peerings list --network <network> --project <project>
gcloud compute networks peerings delete <peering> --network <network> --project <project>
tofu destroy -auto-approve

The first attempt can also fail simply because the GKE cluster is still finishing its own deletion, which does resolve on a re-run. Check what actually holds the network before assuming it is an orphan.

Two Grafana certificates are expected to survive teardown. iap.tf keeps them outside Terraform's destroy lifecycle deliberately, because a managed certificate takes 10 to 60 minutes to provision and leaving it means the next apply does not wait again. A certificate whose name carries a random suffix (autoflow-lt-<env>-<hex>-grafana) is a genuine orphan from an older naming scheme and can be deleted; the unsuffixed autoflow-lt-<env>-grafana should stay.

Known gaps

• TIER_GATE carries no automatic enforcement beyond what's described above: sessions_skipped is checked by matched-rate, and drain's window-length and zero-rounds checks are not tier-based checks at all. The other five gate metrics -- invariant_violations, system_failures, workflow_aborts, shard_fenced, multixact_member_reads -- are captured and can flow straight into --mode=report's output like any other metric, with nothing screening them out; they must be inspected manually before publishing a comparison. invariant_violations and system_failures in particular must be zero for a run to be trustworthy at all: a nonzero value means the engine itself misbehaved during the window, not just that the environment was noisy. A blanket "nonzero TIER_GATE means invalid" rule is not the fix: shard_fenced and multixact_member_reads are routinely nonzero under load (fence-outs and MultiXact SLRU reads happen during normal operation), so treating every gate metric the same way would invalidate every run. Automatic enforcement needs the tier split into must-be-zero metrics (invariant_violations, system_failures, sessions_skipped) and record-only ones first; that split is a design question, not something this documentation pass can resolve.
• The noise_floor_pct values were measured on 2026-08-30 from a ten-point A/A drain campaign at the measurement topology, and floor_provenance records the environment they belong to. Two caveats. They were taken under the strictly alternating arm order this harness has since replaced with a balanced one, so the floors for metrics that drift through a campaign (history_event_rows, multixact_member_reads) absorb a bias the driver no longer introduces and are wider than they need to be. And they are drain floors: matched-rate and profile floors are not the same numbers, and the manifest carries one field per metric, so a matched-rate campaign is judged against drain's floors until the schema carries both.
• Nothing in the baseline flow has run in CI. 05:baseline, 05:compare and the -cmp environment they share are written and linted, never applied. The publish and fetch round trip has been exercised against the real registry from a laptop; the CI jobs' own path, including job-token authentication to the registry, has not.
• What has run against a live environment, and what has not. preflight.sh, protocol-matched-rate.sh, protocol-drain.sh and protocol-profile.sh have each run standalone against a real perf, which is where drain's exhausted-queue guard and preflight's role="worker" filter come from. campaign.sh has driven full campaigns end to end with matched-rate on a small environment and with drain at the measurement topology, the latter being the ten-point A/A that produced the floors. Its profile path has never run under the driver, and neither has the CI pipeline: the perf-ci state, its suffixed resources and the small-tier overrides have been written and reviewed, never applied.
• Comparing two revisions constrains how far apart they can be, and nothing checks it up front. The overlay puts the harness's loadtest command on top of each arm, so it has to compile against that revision's internal packages: measured at 27 commits back from master on 2026-08-24, after which the arm build fails. Separately, every instrument the harness queries has to exist on both arms. Master renamed autocore_shards_per_db to autocore_shards_configured while the other arm still exported the old name, which made that arm unmeasurable until preflight was moved onto a gauge both export. A rename in the manifest's own metrics would do the same to a capture, and the first symptom either way is a failure partway through a campaign that has already paid for points.

Analysis Queries

Queries hit a single workflow DB at a time. With N workflow DBs you'll need to repeat against each DSN (or use analyze.sh for the cross-DB aggregate view). Connect interactively to the first DB:

pgcli "$(tofu output -json workflow_dsns | jq -r '.[0]')"

For non-interactive (scripted) queries, pipe through stdin:

echo "SELECT count(*) FROM workflow_execution;" \
  | pgcli "$(tofu output -json workflow_dsns | jq -r '.[0]')" --less-chatty

State values: 1=running, 2=completed, 3=failed.

Overall Summary

SELECT
    count(*) as total_workflows,
    count(*) FILTER (WHERE workflow_execution_state = 2) as completed,
    count(*) FILTER (WHERE workflow_execution_state = 1) as still_running,
    count(*) FILTER (WHERE workflow_execution_state = 3) as failed
FROM workflow_execution;

Task State Distribution

SELECT state, count(*) as count
FROM task GROUP BY state ORDER BY count DESC;

History Event Types

Event type IDs: 1=WORKFLOW_CREATED, 2=WORKFLOW_COMPLETED, 3=WORKFLOW_FAILED, 4=ACTIVITY_SCHEDULED, 5=ACTIVITY_COMPLETED, 6=ACTIVITY_FAILED, 7=TIMER_STARTED, 8=TIMER_FIRED, 9=SIGNAL_RECEIVED, 10=CANCELLATION_REQUESTED, 11=WORKFLOW_CANCELED

SELECT event_type, count(*) as count
FROM history_event GROUP BY event_type ORDER BY count DESC;

Workflow Creation Throughput (10s buckets)

SELECT
    date_trunc('second', created_at)
        - (EXTRACT(SECOND FROM created_at)::int % 10) * interval '1 second' as bucket,
    count(*) as workflows_created
FROM workflow
GROUP BY 1 ORDER BY 1;

Workflow Completion Throughput (10s buckets)

SELECT
    date_trunc('second', created_at)
        - (EXTRACT(SECOND FROM created_at)::int % 10) * interval '1 second' as bucket,
    count(*) as completions
FROM history_event
WHERE event_type = 2
GROUP BY 1 ORDER BY 1;

Workflow Latency (creation to completion)

SELECT
    percentile_cont(0.50) WITHIN GROUP (ORDER BY dur) as p50,
    percentile_cont(0.90) WITHIN GROUP (ORDER BY dur) as p90,
    percentile_cont(0.99) WITHIN GROUP (ORDER BY dur) as p99,
    min(dur) as min_s,
    max(dur) as max_s,
    avg(dur) as avg_s
FROM (
    SELECT EXTRACT(EPOCH FROM he.created_at - we.created_at) as dur
    FROM history_event he
    JOIN workflow_execution we ON he.shard_id = we.shard_id AND he.workflow_id = we.workflow_id
    WHERE he.event_type = 2
) sub;

Workflow Duration Histogram

SELECT
    CASE
        WHEN dur < 2 THEN '0-2s'
        WHEN dur < 5 THEN '2-5s'
        WHEN dur < 10 THEN '5-10s'
        WHEN dur < 20 THEN '10-20s'
        WHEN dur < 30 THEN '20-30s'
        WHEN dur < 60 THEN '30-60s'
        ELSE '60s+'
    END as bucket,
    count(*) as workflows
FROM (
    SELECT EXTRACT(EPOCH FROM he.created_at - we.created_at) as dur
    FROM history_event he
    JOIN workflow_execution we ON he.shard_id = we.shard_id AND he.workflow_id = we.workflow_id
    WHERE he.event_type = 2
) sub
GROUP BY 1 ORDER BY min(dur);

Overall Throughput

NOTE: uses subqueries to avoid an expensive cross-join on large tables.

SELECT
    round(((SELECT count(*) FROM workflow_execution WHERE workflow_execution_state = 2)::float
    / EXTRACT(EPOCH FROM he_end.created_at - he_start.created_at))::numeric, 1)
    as completed_wf_per_sec
FROM
    (SELECT min(created_at) as created_at FROM history_event WHERE event_type = 1) he_start,
    (SELECT max(created_at) as created_at FROM history_event WHERE event_type = 2) he_end;

Test Duration

NOTE: uses subqueries to avoid an expensive cross-join on large tables.

SELECT
    min(he_start.created_at) as first_workflow,
    max(he_end.created_at) as last_completion,
    round(EXTRACT(EPOCH FROM max(he_end.created_at) - min(he_start.created_at))::numeric, 1)
    as total_seconds
FROM
    (SELECT min(created_at) as created_at FROM history_event WHERE event_type = 1) he_start,
    (SELECT max(created_at) as created_at FROM history_event WHERE event_type = 2) he_end;

Shard Distribution

SELECT shard_id,
    count(*) as workflows,
    count(*) FILTER (WHERE workflow_execution_state = 2) as completed,
    count(*) FILTER (WHERE workflow_execution_state = 1) as running,
    count(*) FILTER (WHERE workflow_execution_state = 3) as failed
FROM workflow_execution
GROUP BY shard_id ORDER BY workflows DESC
LIMIT 20;

Shard Ownership

SELECT owner_id,
    count(*) as shards_owned,
    count(*) FILTER (WHERE lease_expires_at > now()) as alive,
    count(*) FILTER (WHERE lease_expires_at <= now()) as expired
FROM shard_lease
GROUP BY owner_id ORDER BY shards_owned DESC;

Connection and Lock Pressure

-- Active connections by state
SELECT state, count(*) as count
FROM pg_stat_activity
WHERE datname = 'loadtest'
GROUP BY state ORDER BY count DESC;

-- Connection count per client
SELECT application_name, client_addr, count(*) as connections
FROM pg_stat_activity
WHERE datname = 'loadtest'
GROUP BY application_name, client_addr
ORDER BY connections DESC;

-- Lock contention (active waiters)
SELECT wait_event_type, wait_event, count(*) as count
FROM pg_stat_activity
WHERE datname = 'loadtest' AND state = 'active' AND wait_event IS NOT NULL
GROUP BY wait_event_type, wait_event
ORDER BY count DESC;

-- Lock types held
SELECT mode, count(*) as count
FROM pg_locks
WHERE database = (SELECT oid FROM pg_database WHERE datname = 'loadtest')
GROUP BY mode ORDER BY count DESC;

-- Max connections
SELECT setting as max_connections
FROM pg_settings WHERE name = 'max_connections';

Table I/O Stats

SELECT relname, seq_scan, seq_tup_read, idx_scan, idx_tup_fetch,
       n_tup_ins, n_tup_upd, n_tup_del
FROM pg_stat_user_tables
ORDER BY (n_tup_ins + n_tup_upd + n_tup_del) DESC;

Table Sizes

SELECT relname,
    pg_size_pretty(pg_total_relation_size(relid)) as total_size,
    pg_size_pretty(pg_relation_size(relid)) as table_size,
    pg_size_pretty(pg_indexes_size(relid)) as index_size
FROM pg_stat_user_tables
ORDER BY pg_total_relation_size(relid) DESC;

Index Usage

SELECT indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
FROM pg_stat_user_indexes
ORDER BY idx_scan DESC
LIMIT 20;

History Events Per Workflow

SELECT avg(event_count) as avg_events,
       min(event_count) as min_events,
       max(event_count) as max_events
FROM (
    SELECT workflow_id, count(*) as event_count
    FROM history_event GROUP BY workflow_id
) sub;

Planner Behavior (why seq scans?)

Small tables (task_queue, scheduled_task, shard) will always use sequential scans because Postgres correctly determines that reading 1-4 heap pages is cheaper than an index lookup. Verify with:

SELECT relname, relpages, reltuples::bigint as est_rows
FROM pg_class
WHERE relname IN ('task_queue', 'scheduled_task', 'shard',
                  'task', 'workflow_execution', 'history_event')
ORDER BY relpages DESC;

Metrics Analysis (Prometheus/Grafana)

Grafana is available at kubectl port-forward svc/grafana <local-port>:3000. The autocore dashboard shows all panels interactively. For scripted analysis, query Prometheus directly via the Grafana proxy.

All timestamps must be Unix epoch. Convert local time with:

START=$(python3 -c "from datetime import datetime; print(int(datetime(2026,4,15,17,46,0).timestamp()))")
END=$(python3 -c "from datetime import datetime; print(int(datetime(2026,4,15,17,55,0).timestamp()))")

Query pattern

curl -sG 'http://localhost:<grafana-port>/api/datasources/proxy/uid/prometheus/api/v1/query_range' \
  --data-urlencode 'query=<promql>' \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode 'step=30'

NOTE: $__rate_interval is a Grafana variable and does not work in direct API queries. Use a fixed interval like 1m instead.

Useful queries

Worker pool goroutines (active + suspended over time):

sum by (state) (autocore_workflow_goroutines)

Activity tasks in flight:

sum(autocore_activity_tasks_in_flight)

Workflow tasks in flight:

sum(autocore_workflow_tasks_in_flight)

Cold vs warm resume rate:

sum by (resume_type) (rate(autocore_workflow_tasks_claimed_total[1m]))

Workflow completion rate:

sum(rate(autocore_workflow_task_processing_duration_seconds_count{execution_result="completed"}[1m]))

Activity completion rate by name:

sum by (activity_name) (rate(autocore_activity_execution_duration_seconds_count[1m]))

Database connection pool queries

Pool utilization (acquired / max as percentage):

sum(pgxpool_acquired_connections) / sum(pgxpool_max_connections) * 100

Empty acquires/s (callers waiting for a free connection):

sum(rate(pgxpool_empty_acquire_total[1m]))

Canceled acquires/s (operations failed due to pool starvation):

sum(rate(pgxpool_canceled_acquires_total[1m]))

Query duration (p95):

histogram_quantile(0.95, sum by (le) (rate(db_client_operation_duration_seconds_bucket{pgx_operation_type="query"}[1m])))

Connection acquire duration (p95):

histogram_quantile(0.95, sum by (le) (rate(db_client_operation_duration_seconds_bucket{pgx_operation_type="acquire"}[1m])))

Pod resource queries

CPU usage (total cores across all worker pods):

sum(rate(container_cpu_usage_seconds_total{pod=~"autoflow-loadtest-worker-.*",container="worker"}[1m]))

CPU throttling (percentage of CFS periods where workers hit their CPU limit):

sum(rate(container_cpu_cfs_throttled_periods_total{pod=~"autoflow-loadtest-worker-.*",container="worker"}[1m])) / sum(rate(container_cpu_cfs_periods_total{pod=~"autoflow-loadtest-worker-.*",container="worker"}[1m])) * 100

Memory working set (total across all worker pods):

sum(container_memory_working_set_bytes{pod=~"autoflow-loadtest-worker-.*",container="worker"})

Parsing JSON output

Pipe curl output through a python script to format it:

curl -sG '...' | python3 -c "
import sys, json
data = json.load(sys.stdin)
for r in data.get('data', {}).get('result', []):
    labels = r['metric']
    label_str = ' '.join(f'{k}={v}' for k, v in labels.items() if k != '__name__')
    print(f'--- {label_str} ---')
    for ts, val in r['values']:
        from datetime import datetime
        t = datetime.fromtimestamp(ts).strftime('%H:%M:%S')
        print(f'{t}  {val}')
"

Trace Analysis (Tempo)

Search for error traces:

curl -sG 'http://localhost:<grafana-port>/api/datasources/proxy/uid/tempo/api/search' \
  --data-urlencode 'q={resource.service.name="gitlab-kas" && status=error}' \
  --data-urlencode "start=$START" \
  --data-urlencode "end=$END" \
  --data-urlencode 'limit=5'

Fetch a full trace by ID and list all spans with duration:

curl -s 'http://localhost:<grafana-port>/api/datasources/proxy/uid/tempo/api/traces/<trace-id>' \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
spans = []
for batch in data.get('batches', []):
    for scope_span in batch.get('scopeSpans', []):
        for span in scope_span.get('spans', []):
            name = span.get('name', '?')
            start = int(span.get('startTimeUnixNano', 0))
            end = int(span.get('endTimeUnixNano', 0))
            dur_ms = (end - start) / 1_000_000
            status = 'ERROR' if span.get('status', {}).get('code', 0) == 2 else 'ok'
            spans.append((start, name, dur_ms, status))
spans.sort()
for start, name, dur_ms, status in spans:
    if dur_ms >= 1:
        print(f'{name:<60} {dur_ms:>10.1f}ms  {status}')
    else:
        print(f'{name:<60} {dur_ms*1000:>10.1f}us  {status}')
"

CPU Profiling (pprof)

The worker pods expose pprof at /debug/pprof/ on the metrics port. Capture a 30-second CPU profile during load:

kubectl port-forward deploy/autoflow-loadtest-worker 9090:9090
curl -o profile.pb.gz 'http://localhost:9090/debug/pprof/profile?seconds=30'
go tool pprof -http=:8080 profile.pb.gz

Heap profile (current allocations):

curl -o heap.pb.gz 'http://localhost:9090/debug/pprof/heap'
go tool pprof -http=:8080 heap.pb.gz

Goroutine dump (useful for checking suspended goroutine count):

curl -s 'http://localhost:9090/debug/pprof/goroutine?debug=1' | head -5

North-star Environment

The northstar env is a 24/7 environment that continuously generates a configurable mix of diverse autocore workflows (with configurable failure rates) so leadership metrics (cumulative workflows run, success rate, engine health) can be read at any time. It is not a fixed-duration run.

How it differs from perf:

• perf pins the rate (min_rate_per_replica == max_rate_per_replica) and the workload mix (flowgen_swarm = false) and runs with chaos off, so its results are comparable across builds. Its generator runs as a 24/7 Deployment by default; CI overrides injection_duration to run it as a bounded Job that drains and stops.
• northstar lets the per-replica rate wander between min_rate_per_replica and max_rate_per_replica, draws a fresh flowgen mix per replica (flowgen_swarm = true), and runs under chaos. Its generator always runs as a 24/7 Deployment (injection_duration unset), indefinitely.

Every env drives the same workload: a weighted mix picked by the flowtest Driver from the sources enabled in manifests/workload-config.yaml. The scenarios source picks among ~17 Go scenarios (noop, single/inline activities, retries, fanouts, deep selectors, signal consumers, timers, child workflows, deliberate panics, etc.) and runs them against autocore directly; the corpus source picks among curated Starlark units and runs them through the flowcore engine; the adversarial source aims sends the engine must refuse (forged channel tokens, oversized values) at a flow that only sleeps. Signals and cancels are owned by the units themselves via per-unit drive scripts.

The flowgen source synthesizes a fresh AutoFlow unit per session instead of picking from a fixed set: a plan is drawn from a seeded, weighted block grammar (grammar v4: sleeps, timers, testkit echo/delay/record calls, channel receives, loops, select races between a timer and a driver-fed channel, awaitable and detached child workflows running a record-free mini-plan, polls that drive the testkit counter through a CEL predicate until it reaches a drawn value, gitlab_calls that reach the stub backend's REST API over the real gitlab module, and emits that publish a CloudEvent into the stub's counting sink over the real event module; a unit can also draw a cancel op racing its completion or a terminal fail()), and the Starlark script, the drive ops and the expectation all derive from that one plan, so every generated flow is self-validating. Most units expect Completed; a canceled unit expects either Completed or Canceled and a failing one expects Failed. cancel_probability and fail_probability in the workload config gate those two arms. Each unit's seed and generator version land in the ledger's session record, so any session's unit can be re-rendered exactly from (seed, version, config).

Both roles run from the same image, just with different --mode flags:

• autoflow-loadtest-generator (default 4 replicas, --mode=generate) submits workflows through the flowtest Driver.
• autoflow-loadtest-worker (Deployment, default 16 replicas, --mode=worker) runs the real autocore engine and processes the workflows of all sources.

Deploy

cd scripts/loadtests/autoflow/envs/northstar

glab tf init -R gitlab-org/cluster-integration/gitlab-agent autoflow-loadtest-northstar -reconfigure
tofu apply

This env reuses the shared module on isolated-topology infra with workflow_db_count = 2. As with the other envs, tofu apply provisions the infra, builds and pushes the image, and renders the manifests (tofu output -raw kubernetes_manifests | kubectl apply -f -).

Retune the workload

The source mix and per-scenario failure probabilities live in scripts/loadtests/autoflow/manifests/workload-config.yaml, a templatefile the TF module renders. The per-replica rate envelope comes from the min_rate_per_replica / max_rate_per_replica TF variables (defaults 500 and 750):

rate:
  min_per_replica: ${min_rate_per_replica}   # units/sec floor each generator replica targets
  max_per_replica: ${max_rate_per_replica}   # units/sec ceiling; the rate wanders between the two
  period: 10m                                # average time between rate direction changes
sources:
  scenarios:
    weight: 1
    scenarios:
      single_activity:         {weight: 20}
      activity_with_retries:   {weight: 15, fail_prob: 0.25}
      inline_activity_failure: {weight: 5, fail_prob: 0.5}
      # ...
  corpus:
    weight: 1
  adversarial:
    weight: 1
  flowgen:
    weight: 1
    cancel_probability: 0.05 # share of generated units canceled mid-flight
    fail_probability: 0.05   # share of generated units ending in a deliberate fail()

To change the mix:

1. Edit manifests/workload-config.yaml (source and scenario weights, per-scenario fail_prob) or the rate variables in the env's main.tf.

2. tofu apply to update the autoflow-loadtest-workload ConfigMap.

3. Reload the generators (the config is read once at startup):

   kubectl rollout restart deployment autoflow-loadtest-generator
   

This ConfigMap was renamed from an older name; kubectl apply does not prune renamed objects, so an env deployed before the rename still carries the old ConfigMap alongside the new one. Run kubectl get configmap once against such an env and delete the stale entry.

Observability

Grafana is provisioned with three dashboards: autocore.json (engine depth, the same one the load test uses), northstar.json (the leadership panels below) and traces.json (span throughput and latency decomposition). Access it the same way as for the load test:

kubectl port-forward svc/grafana 3000:3000
# Open http://localhost:3000 (anonymous admin, no login needed)

Leadership Metric Queries

Query Prometheus through the Grafana datasource proxy, using the same pattern as Metrics Analysis above:

curl -sG 'http://localhost:<grafana-port>/api/datasources/proxy/uid/prometheus/api/v1/query' \
  --data-urlencode 'query=<promql>'

The workflow round histogram autocore_workflow_task_processing_duration_seconds labels its outcome with execution_result (values yield, completed, failed, canceled, timed_out, system_failed). Its _count series is therefore the source for the cumulative and rate counts below.

Cumulative workflows completed (approximate, for rate/throughput only):

sum(autocore_workflow_task_processing_duration_seconds_count{execution_result="completed"})

The database is authoritative for the official cumulative-since-inception count (the FY "cumulative workflows run" leadership metric). The Prometheus counter above is a per-process counter that resets on every pod restart, so over a 24/7 run a raw sum() undercounts the true lifetime total. Use the Prometheus series for rate and throughput, never for lifetime totals.

For the lifetime total, sum count(*) WHERE workflow_execution_state = 2 (completed; see the State values note under Analysis Queries) across all workflow DBs. The northstar env has 2 workflow DBs, so iterate every DSN from tofu output -json workflow_dsns and add the per-DB counts:

total=0
mapfile -t WORKFLOW_DSNS < <(tofu output -json workflow_dsns | jq -r '.[]')
for dsn in "${WORKFLOW_DSNS[@]}"; do
  n="$(echo "SELECT count(*) FROM workflow_execution WHERE workflow_execution_state = 2;" \
    | pgcli "$dsn" --less-chatty | tail -n +2 | tr -dc '0-9')"
  total=$((total + n))
done
echo "cumulative workflows completed: $total"

Workflow success rate (completed / completed+failed):

sum(autocore_workflow_task_processing_duration_seconds_count{execution_result="completed"})
  / sum(autocore_workflow_task_processing_duration_seconds_count{execution_result=~"completed|failed"})

Terminal throughput by result:

sum by (execution_result) (rate(autocore_workflow_task_processing_duration_seconds_count{execution_result=~"completed|failed|canceled"}[5m]))

Engine-Failure Signals (provisional)

There is no single "engine failure" metric yet. The following are provisional proxy signals; they need refining into a precise, automatable definition before days-without-failure and MTTR can be computed automatically.

Pod restarts (process start-time resets) by role:

sum by (role) (changes(process_start_time_seconds{role=~"worker|generator"}[15m]))

Shard-fencing rate (a spike indicates churn in shard ownership):

sum(rate(autocore_shard_fenced_total[5m]))

Stall: work is queued but nothing is completing. Read both side by side -- a stall is tasks in flight > 0 while completions/s is ~0:

sum(autocore_workflow_tasks_in_flight)
sum(rate(autocore_workflow_task_processing_duration_seconds_count{execution_result="completed"}[5m]))

Until a precise automatable definition exists, MTTR for natural failures is read manually as the elapsed time from one of these signals firing to recovery.

Chaos

Any env can run under configurable chaos so the dashboards show realistic, non-perfect reliability and performance (success-rate dips, latency spikes, pod restarts, drain lag) instead of an idealized steady state. Chaos is built on Chaos Mesh and is off by default; the north-star env is the primary target.

Prerequisites

In addition to the prerequisites above, enabling chaos requires helm and the gke-gcloud-auth-plugin (already needed for kubectl against GKE) on the machine running tofu apply.

Levels

chaos_level is a per-env module variable. Levels increase in intensity:

Level	Character
none (default)	No operator, no experiments. The env is identical to a chaos-free one.
realistic-low	Calm baseline with occasional, mild faults.
realistic-mid	Regular faults and short bounded outages.
realistic-high	Frequent faults, recurring outages, sustained pressure (near-continuous).
unrealistic	Pathological: long blackouts and mass faults, designed to break things.

The low-level per-fault parameters each level maps to live in manifests/chaos-levels.yaml; levels are presets composed from those primitives.

Primitives

Each primitive renders one Chaos Mesh Schedule (labelled chaos.gitlab.io/managed=true) when enabled for the active level.

Primitive	Mechanism	Models
worker_hard_kill	PodChaos pod-kill, gracePeriod: 0	Worker crash (SIGKILL): cold resume, shard fencing
worker_graceful_shutdown	PodChaos pod-kill, gracePeriod > 0	Rollout/eviction (SIGTERM + drain): clean lease handover
db_blip	NetworkChaos partition to Cloud SQL IPs	Loss of database connectivity (network blip / outage)
db_latency	NetworkChaos delay to Cloud SQL IPs	Slow/degraded database link (added latency + jitter)
redis_blip	NetworkChaos partition to the Redis host	Redis unavailability
worker_cpu_stress	StressChaos CPU stressors on workers	CPU pressure / throttling

Enable

Set chaos_level on the module in the env's main.tf (e.g. chaos_level = "realistic-mid"), then apply the infra and the experiments. The experiments are a rendered output (chaos_manifests) applied with kubectl, exactly like kubernetes_manifests:

cd scripts/loadtests/autoflow/envs/northstar
tofu apply                                       # installs the operator, renders chaos_manifests
$(tofu output -raw kubeconfig_command)           # point kubectl at the cluster
tofu output -raw chaos_manifests | kubectl apply -f -

When chaos_level is anything other than none, tofu apply installs the Chaos Mesh operator (pinned chart, GKE/containerd settings, dashboard disabled) into the chaos-mesh namespace via Helm. tofu only renders the experiments; it never applies them. The level is not a Helm trigger, so switching among non-none levels never reinstalls the operator -- only chaos_manifests changes.

All experiments carry the label chaos.gitlab.io/managed=true.

Retune / change level

kubectl apply does not prune, so after editing manifests/chaos-levels.yaml or lowering chaos_level, delete the current experiments before re-applying so dropped ones do not linger (deleting a Schedule recovers any fault it was injecting):

tofu apply
$(tofu output -raw chaos_clear_command)
tofu output -raw chaos_manifests | kubectl apply -f -

Remove

To stop all chaos but keep the operator:

$(tofu output -raw chaos_clear_command)

To remove chaos entirely, set chaos_level = "none" and tofu apply: it clears the managed Schedules (recovering any active fault) and then uninstalls the operator.

Observability

Prometheus scrapes the Chaos Mesh controller-manager (chaos_controller_manager_* metrics), and both the north-star and autocore dashboards have a Chaos activity section (active experiments by kind and phase, event rate, schedule count) at the bottom. Read it alongside the engine panels to line up faults with their effect.

Not covered (yet)

Chaos Mesh has no primitive that scales a Deployment, so the "generators surge 10x" and "workers shrink 50%" faults are intentionally out of scope here and will land later as a small custom scaler keyed off the same chaos_level.

Rebuild and Redeploy

After code changes, rebuild the image and restart pods:

# Rebuild and push (run from an env root)
tofu apply -replace=module.loadtest.null_resource.docker_build_push

# Restart workers (pulls new image)
kubectl rollout restart deployment autoflow-loadtest-worker

# Restart a generator Deployment (unbounded envs)
kubectl rollout restart deployment autoflow-loadtest-generator

# Rerun a bounded generator (Jobs are immutable, must delete first)
kubectl delete job autoflow-loadtest-generator
tofu output -raw kubernetes_manifests | kubectl apply --server-side --force-conflicts -f -

Tear Down

cd scripts/loadtests/autoflow/envs/perf
tofu destroy

Teardown deletes Tempo's PVC before the cluster so its 100Gi PD is reclaimed rather than orphaned.

The managed SSL cert (autoflow-lt-<env>-grafana) is deliberately left behind: it takes 10-60min to provision and the next apply reuses it. It is not an orphan.