Documentation
¶
Overview ¶
Package dialect is the part of building a model that differs per warehouse.
ONE INTERFACE WITH A CONFORMANCE SUITE, written while there was a single implementation. That order is the whole point: a second dialect added against an interface nobody tests agrees with the first wherever the author happened to look and differs everywhere else, and the difference is found by a consumer. Here it is found by `go test`.
What is NOT in here is as deliberate as what is. There is no "quote an identifier", because this package refuses a name that would need quoting instead -- see the Postgres dialect's Build for why. There is no "begin a transaction", because BigQuery does not put DDL in one and an interface that lies about that would be worse than two honest orderings.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NewRows ¶ added in v0.2.0
NewRows wraps a model's SQL in the filter that makes a build incremental.
THE WATERMARK IS A SUBQUERY AND NEVER A VALUE, which is the single most load-bearing line in this file. BigQuery hands its own TIMESTAMP back over the REST API as `"1767484800.0"` -- epoch seconds, as a JSON string -- and feeding that back in is refused outright: `Could not cast literal "1767484800.0" to type TIMESTAMP`, measured 2026-10-08. pgx, for the same column type, returns a time.Time. So a design that read MAX(w) in Go and formatted it into the next statement would be two different bugs in two warehouses, and the quiet one is the dangerous one.
AN EMPTY TARGET IS THE OTHER TRAP. MAX over no rows is NULL, and `w > NULL` is NULL, which admits nothing -- so a model whose first build landed an empty table would stay empty forever while every build reported success. The IS NULL arm is what stops that, and the conformance suite has a case for it because nothing else would ever notice.
Types ¶
type Conn ¶
type Conn interface {
// Exec runs one statement. DDL, so nothing is returned.
Exec(ctx context.Context, statement string) error
// Scalar runs a query of one row and one column. A query matching no row
// returns nil, nil -- that is "nothing is there", which is an ANSWER on
// this path and not a failure.
Scalar(ctx context.Context, query string) (any, error)
// Target names a relation the way the CATALOG does:
// `bigquery://project/dataset/table`, `postgres://database/schema/table`.
//
// ON Conn AND NOT ON Dialect, because only a live connection knows the
// first segment: the project for BigQuery, the database for Postgres,
// and the latter is inside a DSN this package never sees in one piece.
//
// It is what puts a model on `/data`. The engine takes it on a `landed`
// line and refuses anything that does not match its own rule -- counting
// the refusal rather than failing the step, deliberately, because a
// target the engine repaired would be a target the engine inferred. A
// wrong one here is therefore a model that builds, reports success and
// never appears, which is why the conformance suite asserts its shape.
Target(ref string) string
Close(ctx context.Context) error
}
Conn is the little a warehouse has to do.
Two methods, and they are the two the build loop needs: run a statement, and read one value back. A driver's own richer surface stays behind its dialect, where it belongs -- the loop must not learn what a pgx.Rows is.
type Dialect ¶
type Dialect interface {
// Name is what `--dialect` is given, and it is also the name the
// reference extractor is given: a project cannot read its SQL as one
// dialect and build it as another.
Name() string
// Open connects. The DSN comes from an environment variable named in the
// command line, never from a file -- a connection string carries a
// password and a project directory is in git.
Open(ctx context.Context, dsn string) (Conn, error)
// EnsureSchema is run before the models of that schema, and has to be
// safe to run when the schema is already there.
EnsureSchema(schema string) []string
// KindOf is a query whose single value is "table", "view" or NULL.
KindOf(ref string) string
// Literal is a Go string as this warehouse's SQL spells it, quotes
// included.
//
// ON THE INTERFACE BECAUSE IT MEASURABLY DIFFERS, and for no other
// reason -- with one backslash in the statement, on 2026-10-08:
//
// postgres SELECT LENGTH('a\bc') -> 4 a backslash is a character
// bigquery SELECT LENGTH('a\bc') -> 3 \b is a backspace
//
// A value a consumer wrote in an `accepted_values` test therefore means
// two different things depending on where the project runs, and the
// check built from it would pass against data it should refuse.
Literal(s string) string
// ColumnsOf is a query whose single value is the relation's column
// names, in ordinal order, comma-separated.
//
// ONE VALUE AND NOT A ROW SET, so Conn keeps its two methods. `MERGE ...
// WHEN MATCHED THEN UPDATE SET` has to name every column -- BigQuery has
// no `SET *`, measured 2026-10-08 -- so the list has to be asked for, and
// `STRING_AGG(column_name, ',' ORDER BY ordinal_position)` is what both
// warehouses answer it with. A third Conn method for one caller would
// have been the thing this package's own comment argues against.
ColumnsOf(ref string) string
// Build is every statement that turns the model into what its header
// says, in order, given what is in the warehouse NOW.
Build(m model.Model, st State) ([]string, error)
}
Dialect is one warehouse's half of building a model.
type Disposable ¶ added in v0.2.0
type Disposable interface {
// ExpireSchema is the statement that makes everything in this schema
// expire. Run by a test right after EnsureSchema, and by nothing else.
ExpireSchema(schema string) string
}
Disposable is implemented by a dialect whose warehouse can be told to throw away what a TEST leaves behind.
NOT ON Dialect, and that separation is the whole point. EnsureSchema is production -- run.go calls it for every schema a consumer's models land in -- so an expiry reachable from there would be a build that quietly deletes a customer's tables in a day. There is no sentence that makes that acceptable, so the capability lives where the production path cannot reach it by accident.
It exists because a test CAN leak. The conformance suite and the runner's incremental test each create a dataset of their own, beside the client's real ones, and drop it in t.Cleanup -- which does not run after a panic, a SIGKILL or a laptop that sleeps. BigQuery has no dataset expiry at all, only `defaultTableExpirationMs` for the tables inside, so what this buys is a leak that EMPTIES rather than a leak that disappears. That residual is real and is the reason this is one small interface and not a cleanup framework.
Postgres does not implement it: its test warehouse is a container, and `docker compose down` is the expiry.
type Kind ¶
type Kind string
Kind is what a relation IS in the warehouse right now.
Build is given it, because "create or replace" is three different statements depending on the answer and every dialect has to ask.
func KindFrom ¶
KindFrom reads what Scalar returned from a KindOf query.
Here rather than in each dialect because the three answers are the same three everywhere, and a driver that hands back []byte where another hands back string is not a dialect difference -- it is a driver difference, and turning it into a Kind twice is how the two drift.
type Merge ¶ added in v0.2.0
type Merge struct {
// On is `t.k = s.k AND t.j = s.j`.
On string
// Set is `c = s.c, d = s.d`, the columns that are NOT keys.
//
// Empty when every column is a key. A MERGE then has nothing to update
// and the WHEN MATCHED clause is omitted entirely -- `SET` with no
// assignment is a syntax error in both warehouses, and "the row is
// already exactly this row" is not a failure.
Set string
// Columns and Values are the INSERT's two halves, in the target's own
// ordinal order.
Columns string
Values string
}
Merge is the parts of a MERGE that are the same in every warehouse.
HERE AND NOT IN EACH DIALECT, because none of it is a dialect difference: a join on the key columns, a SET of the ones that are not keys, and an INSERT naming the target's columns. What DOES differ -- how the source is deduplicated -- is not in here, and that is the whole split.
type State ¶ added in v0.2.0
type State struct {
// Current is what the relation IS right now.
Current Kind
// FullRefresh is `--full-refresh`: build it as if nothing were there.
FullRefresh bool
// Columns is the TARGET's column names in ordinal order, from ColumnsOf.
//
// Read only when it is going to be used -- an incremental model that is
// adding to an existing table -- and nil otherwise. A list read on every
// build would be a query per model for an answer three materialisations
// out of four never look at.
Columns []string
}
State is what the warehouse holds for this model right now, and what the caller asked for.
A STRUCT AND NOT A THIRD ARGUMENT, which is the precedent #42 set for CheckRow: the questions a build has to answer grow, and a signature that grows with them is one every implementation and every test has to be edited for, one argument at a time, with the compiler unable to tell a swapped pair apart.
func StateOf ¶ added in v0.2.0
func StateOf(ctx context.Context, d Dialect, conn Conn, ref string, incremental, fullRefresh bool) (State, error)
StateOf asks the warehouse the questions a build needs answered.
ONE PLACE, because two would drift and the drift would be invisible: the conformance harness mimics the build loop on purpose -- "every test here also exercises KindOf" -- and a harness that assembled the State its own way would pass a suite about itself. Both call this.
THE COLUMNS ARE READ ONLY WHEN THEY WILL BE USED. Three materialisations out of four never look at them, and a query per model for an answer nobody reads is a round trip per model per build.
func (State) Rebuild ¶ added in v0.2.0
Rebuild says this build creates the relation rather than adding to it.
DERIVED AND NEVER STORED. Three inputs decide it, and a fourth field holding the answer would be a second place for it to be wrong -- which is exactly how a `--full-refresh` that set the flag and forgot the field would merge into the table it was told to replace.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package bigquery builds models on Google BigQuery.
|
Package bigquery builds models on Google BigQuery. |
|
Package dialecttest is the suite every dialect has to pass.
|
Package dialecttest is the suite every dialect has to pass. |
|
leakprobe
command
Command leakprobe is the other half of `scripts/leak-check.sh`: a harness that never cleans up, and the questions you ask BigQuery afterwards.
|
Command leakprobe is the other half of `scripts/leak-check.sh`: a harness that never cleans up, and the questions you ask BigQuery afterwards. |
|
Package postgres builds models on PostgreSQL.
|
Package postgres builds models on PostgreSQL. |