Documentation
¶
Overview ¶
Package skew generates the synthetic hot-key benchmark fixture for the adaptive skew-aware shuffle A/B (docs/design/skew-aware-shuffle.md, Phase 3). It is the TestSkewSplitParity fixture shape scaled to sizes that engage the mechanism at PRODUCTION thresholds — no lowered knobs.
Shape: skew_events (probe side, one hot join key carrying HotPct% of all rows) joins skew_dims (build side, uniform keys). The sizes are dialed to the mechanism's engagement envelope on a 3-worker cluster:
- skew_dims manifest bytes must exceed the cluster broadcast threshold (broadcastThresholdFromCluster, capped at 200 MiB) so the planner emits a hash-partitioned join instead of broadcast+probe-split. The wide `pad` column exists only for this: queries never project it, so the shuffled build stays a few tens of MB — far under the skew split's build-replication bound. Wide dimension table, narrow projection: the common shape in the security workloads this mechanism targets.
- The hot group's probe bytes must exceed skewSplitMinGroupBytes (256 MiB) while the uniform groups stay below it, so exactly one group splits. Suite queries aggregate over events.v (count(DISTINCT e.v)), which forces the ~256-byte payload through the shuffle — projection pruning would otherwise reduce the probe to its 8-byte key and no realistic row count would cross the threshold.
Generation is deterministic for a given Config (seeded rand, fixed chunk boundaries), so two harness runs — the flag-off and flag-on A/B arms — see byte-identical logical rows and row-level parity checks are valid.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var Queries = map[string]string{
"skew_join_agg": `SELECT d.name, count(*) AS cnt, count(DISTINCT e.v) AS distinct_payloads
FROM skew_events e JOIN skew_dims d ON e.k = d.k
GROUP BY d.name ORDER BY d.name`,
"skew_left_join": `SELECT count(*) AS total_rows, count(d.name) AS matched_rows, count(DISTINCT e.v) AS distinct_payloads
FROM skew_events e LEFT JOIN skew_dims d ON e.k = d.k`,
}
Queries is the fixed suite, keyed by name. Both queries force events.v through the shuffle (see package comment) and produce small deterministic result sets so cross-arm parity can compare full row checksums.
skew_join_agg: inner join + aggregate over the build side's low-cardinality category. The hot key's category count and per-category distinct-payload counts surface any row loss or duplication under the split. The count(DISTINCT e.v) partial state also makes task memory proportional to task probe bytes — the memory-relief signal.
skew_left_join: unmatched probe rows (events keys outside dims' key range) must survive a replicated-build split exactly once each; total_rows and distinct_payloads equal the generated events row count.
var QueryOrder = []string{"skew_join_agg", "skew_left_join"}
QueryOrder is the canonical execution order of the suite.
var Tables = map[string]parquet.Schema{ "skew_events": {Columns: []parquet.Column{ {Name: "k", Type: parquet.TypeInt64}, {Name: "v", Type: parquet.TypeString}, }}, "skew_dims": {Columns: []parquet.Column{ {Name: "k", Type: parquet.TypeInt64}, {Name: "name", Type: parquet.TypeString}, {Name: "pad", Type: parquet.TypeString}, }}, }
Tables maps table name → schema, for catalog registration and S3 discovery.
Functions ¶
func GenerateChunked ¶
GenerateChunked streams the fixture through emit in deterministic chunk order: all skew_dims chunks, then all skew_events chunks. emit receives at most ChunkRows rows per call, mirroring tpch.GenerateChunked's contract.
Types ¶
type Config ¶
type Config struct {
EventsRows int // total probe rows
DimsRows int // total build rows; dims keys are [0, DimsRows)
HotKey int64 // the hot events key; must be < DimsRows so it matches
HotPct int // percent of events rows on HotKey (0-100)
KeySpace int64 // non-hot events keys are uniform in [0, KeySpace); keys >= DimsRows miss
NameCard int // distinct dims.name values (result cardinality of skew_join_agg)
PadBytes int // random payload width for events.v and dims.pad
ChunkRows int // rows per emitted parquet chunk
Seed int64
}
Config parameterizes the generator. All sizes are row counts; byte sizes follow from PadBytes (~PadBytes+16 per events row).
func DefaultDeployConfig ¶
func DefaultDeployConfig() Config
DefaultDeployConfig is the SF10-class fixture: events ≈ 8.5 GB with a ≈ 7.6 GB hot partition, dims identical to the local config. Uniform groups ≈ 220 MB stay under the 256 MiB floor; only the hot group splits.
func DefaultLocalConfig ¶
func DefaultLocalConfig() Config
DefaultLocalConfig engages the skew split at production thresholds on a 3-worker local harness cluster: dims ≈ 330 MB parquet (over the 200 MiB broadcast cap), events ≈ 1.7 GB with 90% on the hot key — hot group ≈ 1.6 GB probe (splits at k=3), uniform groups ≈ 60 MB (stay unsplit).
Source Files