Documentation
¶
Overview ¶
Package typematrix is the fixture and query corpus for the type-coverage gates: one table carrying all 22 column types, and a generated corpus that pushes every type through every consumer that can retain, re-key, re-order or re-encode a value.
Why it exists. Every differential corpus in this repo is built on three storage types — Int32, Float64 and String (TPC-H schema.go; ClickBench adds Int64 and Date). So an entire class of wrong-answer defects is structurally invisible: no gate can see a bug that needs a BYTES, IPv4, UUID, DECIMAL or nested column to fire. That is not hypothetical. (*Vector).GetValue's TypeBytes arm returns a slice ALIASING the column arena, so MIN_BY over a BYTES column answers with whatever the pool wrote into those bytes next — a silent wrong answer that shipped through TPC-H, ClickBench, the DuckDB fingerprint corpus, the PostgreSQL oracle, the two-path suite and the shape fuzzer, because not one of them has a top-level BYTES column.
The corpus is GENERATED from a column table rather than hand-written, so a 23rd type is covered by adding one row to Columns() instead of by remembering to write ten queries.
This package holds no assertions. Three gates consume it, each supplying its own reference:
wadjet.TestTypeMatrixBatchReuse — poisoned pool vs clean pool wadjet.TestTypeMatrixOptimizationInvariance — each kill switch off vs on coordinator.TestTypeMatrixTwoPath — stage DAG vs single process
Index ¶
- Constants
- func ColumnNames() []string
- func Data(n int) []map[string]any
- func DimData() []map[string]any
- func DimSchema() parquet.Schema
- func NestedData(n int) []map[string]any
- func NestedDeclaredData(n int) []map[string]any
- func NestedDeclaredSchema() parquet.Schema
- func NestedSchema() parquet.Schema
- func Schema() parquet.Schema
- type Col
- type Pin
- type Query
Constants ¶
const ( NestedDeclared = "typemx_ndecl" NestedDeclaredRows = 600 )
NestedDeclared is the fixture table name, and NestedDeclaredRows its size — past 2×DefaultBatchSize is unnecessary here (the batch-reuse question is the main matrix's), but several parquet row groups is not.
const ( // Table carries the 18 flat types. Nested carries the four container // types. They are SEPARATE tables because readBatchDirect // (internal/planner/physical/util.go:151) decides between the columnar // decoder and the row fallback on the WHOLE table schema, not on the // columns a query projects: one ARRAY column anywhere in a table forces // every query on it onto the row reader, which mints unpooled batches. // Merging the two would silently disable batch reuse — and therefore the // batch-reuse gate — for all 22 types at once. Table = "typemx" Nested = "typemx_nested" Dim = "typemx_dim" )
Table is the fixture table name, Dim the small dimension table the join entries build against.
const Groups = 7
Groups is the number of distinct non-NULL group keys.
const RowGroup = 1100
RowGroup is the parquet row-group size the loaders should use. Deliberately not a multiple of the batch size, so batch boundaries and row-group boundaries fall in different places.
const Rows = 5000
Rows is the fixture row count. Past 2×DefaultBatchSize so the scan recycles pooled batches several times over: a retention defect that only fires on the second batch is the whole point of this corpus.
Variables ¶
This section is empty.
Functions ¶
func ColumnNames ¶
func ColumnNames() []string
ColumnNames returns the type-column names in corpus order, for reporting.
func Data ¶
Data builds the flat table's rows deterministically. Values are derived from the row index so the expected answer of any query is recomputable in Go, and so the fixture is identical in every process that loads it.
func DimData ¶
DimData is one row per group key plus one unmatched key, so the outer-join entries have an unmatched side to null-pad.
func NestedData ¶
NestedData builds the nested table's rows, over the same row indices, so a join between the two tables on id lines up.
func NestedDeclaredData ¶ added in v0.18.3
NestedDeclaredData puts row i's value in every position at once, so the flat column and every container leaf beside it must agree.
Two NULL states on different strides: every fifth row NULLs the containers themselves and every seventh gives a PRESENT container a NULL leaf. The flat columns NULL on the same rows, so the anchor holds in both — a reader that turned a NULL into "" fails the comparison exactly as one that turned a value into "" does.
func NestedDeclaredSchema ¶ added in v0.18.3
NestedDeclaredSchema writes IPv6, UUID and CIDR as a top-level column AND as a ROW field, a nested ROW field, an ARRAY element and a MAP value.
func NestedSchema ¶
NestedSchema is the nested table's schema: the same id and group key plus the four container types.
Types ¶
type Col ¶
type Col struct {
Name string
Type parquet.TypeID
// Flat marks a scalar type: one that can be a GROUP BY key, a sort key,
// a join key and a DISTINCT value. ARRAY/ROW/MAP/VECTOR are not.
Flat bool
// Wide marks a column with enough distinct values that an equi-self-join
// on it stays roughly 1:1 instead of exploding into a cross product.
Wide bool
// Ordered marks a type MIN/MAX and ORDER BY are meaningful over. Every
// Flat type qualifies today; the field exists so a future opaque scalar
// can opt out without opting out of grouping.
Ordered bool
}
Col describes one typed column and what the generator may do with it.
func Columns ¶
func Columns() []Col
Columns is the type matrix: one column per wadjet type, all 22, plus the one SHAPE that takes a different reader from the type it shares.
Each is nullable and each carries a NULL every so often, at a stride coprime with both the batch size and the row-group size so nulls land at interior AND boundary positions — the offsets-on-NULL corruption class only fires at a boundary.
c_rownest is a second ROW, and it is here because a ROW's field types decide which READER answers for it: a ROW of primitive leaves is addressed by leaf path in the native columnar decoder, while a ROW whose fields are themselves containers has no such path and routes to the row reader (scan.HasUnsupportedColumnarTypes, #448). One entry in this list is the difference between the two paths being compared and one of them never being exercised — the columnar arm read such a field back as all-NULL for as long as no corpus column had the shape.
type Pin ¶
type Pin struct {
// Issue is the tracking issue, e.g. "#391".
Issue string
// Reason says what diverges and why it is not fixed here.
Reason string
// GatedBy names the gate that enforces the "this bug is fixed, delete the
// pin" half of the ratchet for this issue, when THIS gate cannot.
//
// Some defects are nondeterministic by nature: #391 answers with whatever
// the allocator happened to write over a freed arena, so an arm that does
// not force the reuse sees it only sometimes. Demanding a divergence every
// run from such an arm makes the suite flap; dropping the demand silently
// would let a fixed bug keep its exemption forever. Naming the arm that
// DOES force it keeps the ratchet exact and keeps it in one place. A pin
// with GatedBy set is exempt from this arm's must-still-diverge check and
// from nothing else — the "matches no corpus entry" check still applies.
GatedBy string
}
Pin records one gate's known divergence on one corpus entry.
type Query ¶
type Query struct {
// Name identifies the entry; it is the subtest name.
Name string
SQL string
// Mode is how strictly two arms' results are compared.
Mode oracle.CmpMode
// Col names the type column the entry targets, "" for the entries that
// target none. Used only for reporting which types a run exercised.
Col string
// contains filtered or unexported fields
}
Query is one corpus entry.
func Corpus ¶
func Corpus() []Query
Corpus generates the query corpus: per type-column templates plus the entries that need no particular column.
Ordering is deterministic (Columns() order, then template order) so a failing entry name is stable across runs and across processes.
func NestedDeclaredCorpus ¶ added in v0.18.3
func NestedDeclaredCorpus() []Query
NestedDeclaredCorpus is the shape corpus: every way a query can reach one of these values — the whole container, a field path into it, an element, and the consumers that RE-KEY or RE-ORDER it (GROUP BY, ORDER BY, WHERE).
Two shapes are deliberately absent, and neither is this fixture's to fix:
- `GROUP BY rw.f_ipv6` and `SELECT DISTINCT rw.f_ipv6` fail on BOTH paths with "GROUP BY key is not a column of its input". A field PATH cannot be a grouping key today (#568's family) whatever type the field has.
- `m['k']` answers NULL for EVERY map, MAP(STRING,INT64) included: the engine materializes a MAP as an ARRAY of key/value ROWs, so element_at's string-key arm never sees a Go map to look the key up in.
- a QUALIFIED field path (`a.rw.f_uuid`) does not parse — "syntax error at or near \".\"" — so the join entries key on the flat column and CARRY the container as payload instead.
Source Files