README
¶
Real Azure ADX-Mon end-to-end environment
Two independent entry points: create the paid Azure resources once, then
build/deploy/test any local branch as many times as necessary. These scripts do
not use the published latest images or the remotely downloaded quick-start
bundle. Run them from a checkout of the branch you want to test.
Prerequisites
- Python 3.9+, Azure CLI (
az loginandaz extension add -n kusto),kubectl,curl, Git, and network access to Azure. Docker is optional: if its daemon is not running,test.shuses remote ACR Tasks builds. - Permission to create a resource group, AKS, ACR, Kusto, and a Kusto cluster principal assignment in the chosen subscription. The Azure CLI user must be able to query and alter the new Kusto databases.
- Available VM and ADX quota in the region. AKS defaults to one node: prefer D4s_v3, then D4s_v4/v5/v6; if those are restricted or out of quota, choose another available D/E-series VM with at least four cores and 16 GiB (for example, D4ads_v6). Selection checks regional SKU restrictions and both family and regional vCPU quotas. ACR is Basic; ADX prefers L8as_v3, then other storage-optimized sizes, at their minimum supported two-node capacity. ADX is billed for both nodes.
- The resource group must not already exist. ACR and Kusto names are generated from the group name; use a unique group name in each subscription.
One-time setup
tools/dev/azure-test-env/cluster.sh create \
--subscription '<subscription-id>' --region eastus --resource-group adxmon-e2e-jw
tools/dev/azure-test-env/cluster.sh status --resource-group adxmon-e2e-jw
The AKS kubelet identity gets Kusto AllDatabasesAdmin, and AKS gets ACR
pull permission via --attach-acr. Kusto starts with streaming disabled. The
CLI prints the selected sizes before creating anything. Provisioning can take
many minutes; resources keep accruing charges until deleted.
Optional create flags: --aks-sku Standard_D4ads_v6, --kusto-sku.
An explicitly supplied AKS size is still checked for regional restrictions,
minimum memory/cores, and quota, with the specific reason shown if it cannot
be used. Failures preserve the resource group and state record. Re-run the
same create command to resume: it verifies and reuses the recorded AKS
and ACR, then continues with Kusto and any remaining setup. It will not adopt
an existing resource group without a matching state record or resize existing
clusters. For the specific Canada Central example, --aks-sku Standard_D4ads_v6 selects the available four-core fallback.
Repeatable branch test
git switch realtime-ingestion # or any branch under test
tools/dev/azure-test-env/test.sh run --resource-group adxmon-e2e-jw
# Or run just one phase after a fix:
tools/dev/azure-test-env/test.sh baseline --resource-group adxmon-e2e-jw
tools/dev/azure-test-env/test.sh realtime --resource-group adxmon-e2e-jw
tools/dev/azure-test-env/test.sh isolation --resource-group adxmon-e2e-jw
Table isolation phase
isolation tests selective shedding at the collector, rather than depending
on a short-lived ingestor backlog to cross a quota at the instant metrics are
scraped. It enforces a 512-byte budget for Logs.E2eOtlpLogs on the daemonset
collector. Each measured signal is accompanied by a mixed OTLP request
containing eight large noisy records followed by one healthy
Logs.E2eQueuedControl record. The phase requires:
- OTLP partial success reporting at least one, but no more than eight, rejected noisy records per mixed request (the healthy record must not be counted);
- the number of noisy rows arriving in Kusto to equal eight minus the reported rejection count, so accepted and rejected records are both accounted for;
- every same-request healthy control record and all four usual healthy paths to reach Kusto, while the ingestor remains Ready during the requests;
- no unexpected component errors. A transient AKS IMDS 400 at pod startup is recorded separately, and is tolerated only alongside successful readiness and Kusto delivery.
The ordinary noisy path is not polled or reported as missing: small unrelated
records may still arrive as a budget drains. The mixed requests' noisy records
are queried and compared with their partial-success counts. report.md lists
the counts, same-request Kusto rows, readiness, and any backlog gauges.
This phase validates collector-side selective shedding; ingestor peer-scoped
429 and upload fairness remain covered by Go tests, not by this live phase.
To A/B test pod-tail latency using the same newly built image tag, first
run test.sh realtime --resource-group ... --realtime-log-batch-wait-ms 1000
(the former one-second behavior) on a warmed cluster. Repeat realtime with
--realtime-log-batch-wait-ms 100 --reuse-tag <first-run-tag> to avoid a
rebuild. Compare pod_logs p50/p95/max, queued control, scrape latency and
component errors. Use a fresh build, not the old 5e45ace9 image tag.
The default --build-mode auto uses local Docker when its daemon is running;
otherwise it uploads the local checkout as an ACR build context and builds
linux/amd64 images remotely. Choose --build-mode acr to force daemon-free
remote builds, or --build-mode docker to require local Docker. ACR Tasks
builds may incur additional charges. .dockerignore excludes local state,
reports, Git metadata and cache files from both build contexts.
If a run fails after its images were pushed (for example, during Kusto setup),
use --reuse-tag <tag> to skip all three builds on the next attempt. The
runner verifies that each tagged image exists in ACR and writes a new report
directory; omit this option after changing branch code or the workload so
new images are built. The 100 ms tail batching change and bounded scrape
sample lifetime require fresh collector/workload images.
run builds collector, ingestor and the small signal workload from the
current checkout, pushes commit/run-tagged images to the environment's ACR,
deploys the checked-in Kubernetes manifests with those images, then:
- Applies a one-minute Kusto database ingestion batching policy and deploys
collector/ingestor without realtime tables. Sends numbered samples
through pod-tailed JSON logs, OTLP/HTTP protobuf logs, OTLP/HTTP protobuf
metrics, and a one-second annotated Prometheus scrape target. In the test
environment, the Prometheus scrape keeps only
e2e_scraped_sampleso a fresh cluster is not flooded with hundreds of irrelevant metric schemas (default-drop-metricsmust be true for the keep rule to work). Each sample is exposed for two seconds rather than indefinitely. OTLP is sent to a ClusterIP Service selecting only the collector DaemonSet (the node hostPort is not reachable from pods on some AKS networks). The optional singleton collector has a separate hostPath WAL on one-node AKS, avoiding concurrent access to the DaemonSet collector's WAL. - Enables Kusto streaming, configures the same four destination tables on both components, waits for schema and streaming-policy warm-up, and runs fresh numbered samples. A fifth OTLP log table remains queued as a control.
- Polls Kusto concurrently with emission. Each row has a run ID, sequence and
pod-generated millisecond timestamp; the report records first query
visibility from the machine running
test.sh. Each path polls independently, recording its last negative and first positive query times inresult.json. Host/pod clock skew and Kusto query round-trip time are part of the measured latency. Every selected realtime path must become visible in at most 2 seconds and have a positivestreamedoutcome counter delta without unexpected fallback. Missing rows and unexpected ingestor/collector errors fail either phase.
The default is five signals, three seconds apart; adjust with --count N and
--timeout SECONDS. The scraper holds each gauge value until the next signal.
New tables and streaming mappings can take several minutes to become eligible;
both phases wait up to ten minutes for all five test tables to ingest a warm-up
sample before measuring. Warm-up traffic is not counted as measured traffic.
--timeout controls only the subsequent per-phase measurement deadline
(240 seconds by default); --warmup-timeout controls the warm-up limit. If
warm-up times out, inspect the saved ingestor/collector
logs and WAL backlog rather than treating missing tables as latency results.
Results go to .adx-mon-test-env/results/<resource-group>/<commit>-<UTC>/:
environment.json, phase result.json, report.md, collector.log, and
ingestor.log. Individual rows, missing/duplicate counts, per-path median,
p95 and maximum latency, and streaming counter changes are included. Reports
capture all collector/ingestor log lines in each phase (kubectl logs --tail=-1),
not just the ten-line default when using a selector. Results include final row
counts after the queued control is visible without changing the first-visibility
timestamps. Reports
can be regenerated offline:
tools/dev/azure-test-env/test.sh report --results .adx-mon-test-env/results/adxmon-e2e-jw/<run>
Re-running run restores queued-only ADX-Mon configuration for its first
phase, even if an earlier run left the Kusto cluster streaming feature enabled;
turning that cluster feature off can take hours and is unnecessary for a
queued-path test. The test script never recreates or deletes Azure resources.
Scrape parallelism evaluation
Measured before the collector scraped targets concurrently: the test target
was first in the sorted target list, so its scrape-to-send time (208 ms median,
315 ms p95) was dominated by cadence, not ordering. A single fast target cannot
show the benefit of concurrent scraping; the soak slow-target scenario adds
slow targets on the workload's node to measure it.
A/B soak tests
soak.sh compares two builds under the same conditions with thousands of
samples per path, background load and injected faults. Latency is computed in
Kusto for every row as ingestion_time() - emitted_ms, so it does not include
polling or query round trips.
RG=my-adx-mon-test # your resource group
# Optional: record an environment whose state file was lost (never deleted by
# this tool), and scale it so peer transfers run across nodes.
tools/dev/azure-test-env/cluster.sh adopt --subscription '<sub>' --resource-group "$RG"
tools/dev/azure-test-env/cluster.sh scale --resource-group "$RG" --node-count 3
# Build both sides once. Images are tagged by commit; the workload by its sources.
tools/dev/azure-test-env/soak.sh build --resource-group "$RG" \
--ref origin/main --ref my-branch
# origin/main: --tag <a> --workload-tag <wl>
# my-branch: --tag <b> --workload-tag <wl>
tools/dev/azure-test-env/soak.sh soak --resource-group "$RG" --tag <a> --label main --scenario steady
tools/dev/azure-test-env/soak.sh soak --resource-group "$RG" --tag <b> --label branch --scenario steady
tools/dev/azure-test-env/soak.sh compare .adx-mon-test-env/results/"$RG"/soak/{main,branch}-steady-*
Each soak deploys 3 ingestors (--ingestor-replicas), a load Deployment
spread over the nodes (200 remote-write metric tables x 20 series and 5 OTLP
log tables per replica every second), sets ADX queued batching to 10 s
(--batching-seconds), warms every table, then streams --rate signals per
second for --duration seconds on six paths: pod-tail logs, OTLP logs, OTLP
metrics, remote write, the Prometheus scrape, and a queued OTLP control.
Every emitted seq is either delivered or recorded as a failed send by the
workload, so missing counts only data accepted and then lost. The scrape
path is sampled, so it has latency but no missing or duplicate counts.
| Scenario | Fault |
|---|---|
steady |
none |
ingestor-kill |
deletes --fault-pod (default ingestor-1) at --fault-at |
ingestor-rollout |
rollout restart statefulset/ingestor at --fault-at |
ingestor-stall |
SIGSTOPs the ingestor process on its node from a privileged hostPID pod for --fault-duration seconds; TCP still accepts, as with a hung peer |
slow-target |
--slow-targets scrape targets on the workload's node that answer after --slow-delay |
collector-restart |
deletes the workload node's collector every --fault-interval seconds for --fault-duration |
chunked-otlp |
sends OTLP logs with Transfer-Encoding: chunked |
The collector's WAL segment age is set to 5 s (--collector-max-segment-age)
on both sides; its built-in 30 s default otherwise dominates end-to-end
latency. Kusto tokens are requested for the environment's tenant, which may
differ from the Azure CLI's default account. If this machine cannot connect to
the ADX endpoint (for example, when it only accepts Azure networks), add
--kusto-via-aks: Kusto calls are relayed through a TCP pod in AKS with
curl --connect-to, so TLS still terminates at the real host name and its
certificate is verified. Only use it where that routing is permitted.
Pass branch-only settings with --ingestor-arg=--forward-transfers or
--collector-toml 'scrape-concurrency = 4'; older builds may reject unknown
settings. Each run writes result.json, rows.json (every seq's emitted and
ingested time), report.md and the component logs. The report shows per-path
p50/p95/p99/p99.9/max, missing, duplicates and failed sends, the same split
before, during and after the fault, error-like counter increases from every
pod's /metrics, log errors grouped by message, kubectl top CPU and memory,
ADX ingestion failures and .create-merge command counts. Run each side at
least twice, alternating, because ADX ingestion latency varies between runs.
Offline tests for the harness: python3 -m unittest discover -s tools/dev/azure-test-env -p 'test_*.py'
and go test ./tools/dev/azure-test-env/workload.
Cleanup
tools/dev/azure-test-env/cluster.sh delete --resource-group adxmon-e2e-jw --yes
This deletes the entire recorded resource group, including AKS, ACR, Kusto
and their data. It does not delete local reports. Inspect status first.