racer-loadgen

racer-loadgen

Generate deterministic blobs and repeatedly consume them through Gantry's registry API, Racer's client Unix domain socket (UDS), or racer-object's S3 HTTP endpoint. All backends use the same catalog selection, worker admission, pacing, deadlines, integrity checking, diagnostics, and metrics. Payloads are generated on demand, not retained in a load-generator content cache.

Workloads and backends

S3 HTTP benchmark and synthetic origin

S3 mode performs an unsigned, path-style HTTP GET for every operation, not an SDK shortcut. Use a racer-object sidecar as the endpoint to measure the object adapter, or point the same client directly at the origin for a baseline.

Origin-only process (no Garage provisioning or uploads required):

racer-loadgen --backend=s3 --s3-origin --listen=:8080 --concurrency=0 \
  --bucket=benchmark --object-count=128 --object-bytes=67108864 \
  --seed=benchmark-v1 --startup-timeout=4m --metrics-listen=:9090

Client process in a pod with a racer-object sidecar:

racer-loadgen --backend=s3 --endpoint=http://127.0.0.1:8080 \
  --bucket=benchmark --object-count=128 --object-bytes=67108864 \
  --seed=benchmark-v1 --concurrency=8 --profile=shuffle \
  --pull-timeout=2m --startup-timeout=4m --metrics-listen=:9090

S3 mode defaults to bucket benchmark, 128 objects of 64 MiB, and endpoint http://127.0.0.1:8080. Objects have keys object-000000 through object-000127 for this example. Count is bounded to 1-512; size must be positive. Bytes use the existing deterministic raw-blob generator, without OCI metadata or tar framing. Each process hashes its whole catalog at startup using bounded scratch space, not an in-memory object cache. The example hashes 8 GiB on each pod before readiness; allow CPU and a sufficient startup probe budget. Very small payloads may collide; duplicate-content catalogs are rejected.

Keep seed, count, size, and bucket identical on origin and client pods. Opt-in integrity verification compares downloaded SHA-256 and size against the local deterministic catalog, independently of the upstream's ETag format. The default --verify=false skips hashing during reads, but still checks body size and does not skip startup catalog hashing. An external S3 store can be used only if these exact keys and payloads are populated separately and unsigned reads are allowed; loadgen neither uploads objects nor signs requests.

Recommended DaemonSet split:

1. Back the Racer cache with an Azure Blob container named like --bucket (default benchmark) that holds the exact catalog keys and payloads. Set the ClusterCache spec.origin.azure.containerURL to it; the Racer dataplane reads the container directly. Loadgen does not upload objects.
2. Run another loadgen DaemonSet with a racer-object sidecar container: sidecar --cache=racer-object --listen=127.0.0.1:8080. Mount the cache's client socket directory only into the sidecar. The loadgen container needs no socket mounts or credentials. Without --s3-origin, S3 loadgen starts only its metrics/probe listener, leaving port 8080 free for the sidecar. Do not expose the sidecar with a Service or host networking.
3. For a direct-origin comparison, run origin-only loadgen containers with --s3-origin --concurrency=0 and use the same client flags, but change --endpoint to the origin Service URL (for example, http://racer-s3-origin:8080). Run comparisons separately with identical concurrency, catalog, profile, verification, and resource limits. Label the jobs direct and racer-object in deployment metadata, not object labels. A Service-routed baseline and a node-local origin can have different network paths; report that topology rather than attributing all differences to caching.

The synthetic origin implements object HEAD/GET, one closed/open/suffix byte range, strong quoted ETags, If-Match/If-None-Match (including wildcard conditions), and S3 XML errors. It is a direct S3 baseline only: Racer reads Azure Blob, not S3. The racer-object sidecar path is tested against a synthetic Racer SDK origin. ETags are quoted sha256:<hex>, not MD5. This is an unauthenticated, read-only test fixture, not a general S3 server: no listing, writes, versioning, date conditions, If-Range, or multipart reads. Keep it on a trusted benchmark network; it accepts signed requests without validating signatures. x-id=GetObject and x-id=HeadObject are supported SDK query plumbing; version reads fail rather than silently returning current data.

Existing --profile=shuffle|zipf, --zipf-exponent, --concurrency-file, node caps, --interval, --retry-delay, --duration, and --diagnose-integrity work unchanged. One operation is one full object GET; per-blob concurrency is fixed to one. S3 mode rejects OCI sizing/transport flags instead of silently ignoring them; use --endpoint, not --target, and configure the upstream container on the ClusterCache, not loadgen.

Metrics retain their existing names and the kind="blob" label. Successful operations appear in racer_loadgen_pulls_total{result="success"}; verified goodput is rate(racer_loadgen_verified_bytes_total[1m]), in bytes/s, and failures are classified in racer_loadgen_pull_failures_total. Use request/pull duration histograms for latency and racer_loadgen_origin_bytes_total on origin pods for generated body bytes. No per-key metric labels are added. /readyz indicates catalog readiness, not sidecar/dataplane health, and performs no warmup GET. The process can exit normally after --duration even if reads failed: evaluate the success, failure, and byte counters, not just its exit code. A finite run stops the metrics listener at completion, so scrape during the run.

Verification consumes CPU, and synthetic origins spend CPU regenerating data; neither result represents persistent-storage performance. Restarting with the same seed does not make the cache cold. Unlike the digest-keyed UDS mode, S3 names stay stable across seed changes: use a new ClusterCache for an isolated cold run to avoid metadata-TTL staleness. The local adapter integration test uses a noncaching SDK test daemon and does not establish real cache-hit behavior, cluster compatibility, or throughput. Size the sidecar and dataplane separately; the loadgen's streaming buffer does not account for their page storage.

Gantry and direct SDK modes

For independent raw blobs through Gantry:

racer-loadgen --backend=gantry --target=http://127.0.0.1:5000 \
  --namespace=loadgen.invalid --catalog-blobs=128 --blob-bytes=67108864 \
  --concurrency=8 --seed=benchmark-v1

Configure Gantry's upstream for loadgen.invalid to reach the synthetic HTTP origin on --listen (default :8080). To measure the HTTP origin without Gantry, point --target at that origin instead.

For the same blobs directly through Racer, without Gantry:

racer-loadgen --backend=uds --cache=racer-loadgen \
  --catalog-blobs=128 --blob-bytes=67108864 \
  --concurrency=8 --seed=benchmark-v1

Direct mode uses pkg/racersdk against /run/racer/racer-loadgen/client/socket and serves the synthetic origin at /run/racer/racer-loadgen/origin/socket. It starts no HTTP registry and ignores --target, --namespace, and --listen. Metrics and probes still use --metrics-listen (default :9090). An operational Racer dataplane and cache configuration are prerequisites; loadgen does not start the dataplane.

Unit tests use pkg/racersdk/racersdktest.NewClient with a noncaching local daemon and real SDK origin validation over temporary Unix sockets. The helper registers cleanup with the test. These tests are not evidence of real Racer compatibility or performance.

Use a dedicated cache, not a cache whose origin socket Gantry already owns. The racer creates the origin directory owned by UID/GID 65532 with mode 0700, and the loadgen image runs as that user. Loadgen creates no directories and changes no permissions: it fails at startup if the origin directory is missing, is not owned by its effective UID, or is group/world writable. The SDK provides exclusive ownership, stale-socket recovery, and shutdown cleanup. See the standalone Kubernetes example.

Every node eligible to supply origin data must have a running origin with the same seed and catalog parameters. There is no upload step: Racer retrieves synthetic data from the origin on demand. Blob keys are the SHA-256 digest bytes; ETags are the quoted sha256:<hex> digest. Initial reads are unpinned, like Gantry's ordinary full-object reads. Direct mode also checks returned metadata against the expected digest and size.

Catalog and operation units

• --catalog-blobs=N creates 1-32768 independent raw objects. Each is exactly --blob-bytes bytes (default 64 MiB), without tar framing or jitter. One worker operation reads one blob. Very small objects can produce duplicate content; startup rejects duplicate catalogs rather than reporting a false distinct count.
• --catalog-workers=N uses 1-64 startup hashing workers (default 1). It requires --catalog-blobs; OCI and S3 invocations reject the explicit flag. Each worker reuses a 128 KiB buffer. Catalog metadata grows with object count, not payload size. All workers stop before a failed or canceled build returns. Worker count does not change bytes, SHA-256 keys, or catalog rank. Completed object and byte counts are logged at most once per five seconds as objects finish, plus a final completion log. For example, --catalog-blobs=16384 --blob-bytes=67108864 --catalog-workers=32 creates a 1 TiB working set with 4 MiB of hashing scratch space. Every origin hashes the full working set before readiness. Size the startup deadline and CPU budget accordingly. Choose catalog size and Zipf skew from measured cache hit rates; a larger catalog alone does not guarantee a balanced cache mix.
• Without --catalog-blobs, loadgen generates an OCI image catalog. Each catalog entry becomes a blob batch: ordered manifest and config, then concurrent layer blobs. This workload also works directly over UDS.
• --catalog-images, --layers, --layer-bytes, and --jitter keep their OCI meanings and cannot be explicitly combined with --catalog-blobs. The OCI catalog limit is 512 images; the S3 limit is 512 objects. --blob-bytes requires --catalog-blobs.
• --concurrency counts admitted operations, not individual requests. --blob-concurrency bounds parallel blobs within a batch (default 4). Independent single-blob operations always use one request.
• --profile=shuffle traverses the catalog in shuffled passes; --profile=zipf samples by stable catalog rank with --zipf-exponent controlling skew. --profile=sweep shares one in-order cursor across all workers in the process, so each pass reads every entry once. Use it to measure cold reads without the repeats that independent per-worker shuffles produce.
• Live --concurrency-file and node caps apply to the same admission loop for both backends. Reducing concurrency drains admitted operations under their original deadline. Zero pauses reads while keeping the origin available.

--pull-timeout covers a complete operation. --interval and --retry-delay control per-worker pacing. --duration bounds the load phase. --startup-timeout bounds catalog hashing and origin startup; UDS origin readiness also has a 10-second bound. /readyz means the catalog and origin are ready, not that the dataplane has successfully acquired data. The UDS readiness probe talks directly to the origin and does not warm measured objects in Racer.

Large blobs and coordinated starts

These flags help when many nodes load one very large blob, such as 1 TB:

• --trusted-blob-digests=sha256:... (comma-separated, one per catalog index, starting at 0) skips startup hashing for those blobs. Use digests computed earlier for the same seed, index, and size. A wrong digest is not detected at startup.
• --start-at=<unix seconds> waits until that time after --start-delay, so pods that become ready at different times start together.
• --page-credits=N (UDS only, 0-64) sets how many pages Racer may send ahead of each read. The dataplane's fetch window for a read is the smaller of this and RACER_RANGE_WINDOW_PAGES; 0 keeps the SDK default of 2.
• --range-bytes=N (UDS only, requires --verify=false unless --delivery=host-read) reads each blob as byte ranges pinned to one ETag. --range-parallel ranges run at once (raise --blob-concurrency to at least the same value, since it sizes the SDK connection pool). A range that fails mid-transfer resumes at its first unread byte, up to --range-retries times, and counts in racer_loadgen_range_resumes_total. --range-rotate starts each pod at a range derived from its host name, so nodes do not all request the same pages at the same moment. --range-cohorts=K instead splits pods into K cohorts (by the trailing number of --node-name when it has one); a cohort starts at the same range, and cohorts start evenly spaced across the blob.
• --delivery=host-read (UDS only; default stream) has the dataplane write each blob straight into a memfd with positioned io_uring writes instead of sending it over the socket (POST /v1/objects/<key>/host-read, see the Racer reference). The memfd is passed with SCM_RIGHTS on the same client socket, so it works across pods that share the socket's hostPath and needs no shared uid. Each concurrent pull borrows a memfd from a pool; memfds are grown with ftruncate and fallocate to the largest blob seen and reused, so their pages stay allocated and are charged to the loadgen's memory cgroup, not the dataplane's. Budget about concurrency * blob-concurrency * blob size of shared memory. Received bytes count per page as the dataplane confirms them. With --range-bytes, each range is written at its own offset, so --verify=true hashes the memfd through a read-only mapping. Whole-object reads skip Stat and size the window from the catalog descriptor. --diagnose-integrity is not supported.

Memory and measurement

By default, reads drain with io.Copy to an open /dev/null file reused for the puller's lifetime. Direct UDS reads preserve the SDK object's WriteTo method, allowing the zero-copy splice path rather than copying whole objects into Go memory. --verify=true uses a 32 KiB read buffer to hash each object, so bytes are copied into Go memory. Protocol, socket, origin, and runtime allocations still contribute to RSS; there is no fixed page-storage budget per read.

Failure log lines carry a cause attribute that names the error class (unexpected_eof, unavailable, deadline_exceeded, and so on) without the error text, which can contain URLs or credentials.

Metrics:

• racer_loadgen_pulls_total, pull duration, and in-flight metrics describe operations: one raw blob or one complete OCI batch, according to workload.
• Request metrics describe individual blob acquisitions. The bounded kind label is blob for raw objects, or manifest, config, and layer for OCI.
• racer_loadgen_verified_bytes_total credits only completely successful, verified operations. Failed batches and the default --verify=false receive no credit; enable --verify when measuring verified goodput.
• racer_loadgen_received_bytes_total counts bytes delivered to the consumer, including failed operations. Direct UDS drains credit bytes every second while they run (from the SDK object's Progress), and the rest when io.Copy returns, including on failure. This is not wire traffic; SDK framing is excluded.
• UDS origin metrics count callback operations and bytes generated for the SDK, not HTTP socket writes. They are not network throughput counters.

Image-oriented dashboards should not be interpreted as image-pull measurements for raw-blob jobs. Keep backend/workload jobs distinguishable through deployment labels. A warmed read does not guarantee permanent cache residency; changing the seed changes content identities, whereas restarting loadgen with the same seed does not make Racer's cache cold.

--verify=false is the default. Use --verify to hash on the copy path; --diagnose-integrity also requires --verify and uses the same deterministic oracle on both backends. Tests cover the shared SDK and Gantry adapters, including a noncaching SDK fake and temporary UDS readiness servers; they do not establish real Racer cache-hit behavior or throughput.