Documentation
¶
Overview ¶
Package wadjet provides the public embeddable API for Wadjet.
Index ¶
- func BuildUpdatedRows(ctx context.Context, b *batch.RecordBatch, matched []int64, ...) (rows []map[string]any, err error)
- func CheckDMLQualifier(target plansql.DMLTarget) error
- func ConvertValue(s string, typ parquet.TypeID) (any, error)
- func ConvertValueForColumn(s string, col parquet.Column) (any, error)
- func MatchDMLRows(ctx context.Context, b *batch.RecordBatch, predicate DMLPredicate, ...) (matched []int64, err error)
- type ColumnMeta
- type Config
- type DB
- func (db *DB) Catalog() *catalog.Catalog
- func (db *DB) Close()
- func (db *DB) CreateTable(ctx context.Context, name string, schema parquet.Schema, ...) error
- func (db *DB) DropTable(ctx context.Context, name string) error
- func (db *DB) Execute(ctx context.Context, sql string) (res *ExecResult, err error)
- func (db *DB) ListTables(ctx context.Context) ([]string, error)
- func (db *DB) NewIngester(tableName string, schema parquet.Schema, partitionKeys []string, ...) *ingest.Ingester
- func (db *DB) Query(ctx context.Context, sql string) (res *QueryResult, err error)
- func (db *DB) SetAuthProvider(p *auth.Provider)
- func (db *DB) Store() objstore.Store
- type DMLAssignment
- type DMLPredicate
- type ExecResult
- type QueryResult
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildUpdatedRows ¶ added in v0.18.5
func BuildUpdatedRows(ctx context.Context, b *batch.RecordBatch, matched []int64, assigns []DMLAssignment) (rows []map[string]any, err error)
BuildUpdatedRows boxes b's matched rows with the assignments applied.
It carries the same panic boundary MatchDMLRows does and for the same reason: a SET expression is evaluated by the same engine as a WHERE, so `SET n = 1/0` raises a FatalEvalPanic that must become 22012 on the statement rather than a dead connection (ADR-0019, #677). One deferred call per file, nothing per row.
A value the target column cannot hold is refused HERE, before any delete marker is committed — the rule #647 established for literals, applied to computed values too.
func CheckDMLQualifier ¶ added in v0.18.5
CheckDMLQualifier accepts the schema/catalog qualifier of a DML relation when it names this server's own schema, and refuses any other.
`DELETE FROM public.orders` is what a PostgreSQL client writes by default, and the DML parser used to read only the first identifier — so the statement addressed a table named "public". This is the SELECT path's rule (logical/builder.go, buildScan) applied to the DML doors; PostgreSQL 17 answers an unknown qualifier with 42P01.
func ConvertValue ¶
convertValue converts a string value from the parser to the appropriate Go type based on the target column's type. ConvertValue converts a string value to the appropriate Go type for a given Parquet type.
func ConvertValueForColumn ¶ added in v0.18.5
ConvertValueForColumn converts a literal's text against a column's FULL declaration rather than its TypeID alone, so a value the column cannot hold is refused HERE — before the caller commits anything destructive.
ConvertValue is handed a TypeID and nothing else, so a DECIMAL literal passes through it as text and is first judged at the parquet leaf, where the declared (p, s) is known. That was harmless while nothing could refuse it, and became DATA LOSS the moment something could: executeUpdate wrote a file's delete markers and ingested afterwards, so `UPDATE u SET d = 99999999999999999999.99` answered 22003 with the matched rows already deleted, and three such failures emptied a three-row table (#647 review). A value conversion that can fail must run before the first irreversible step, and this is where the declaration to check against lives.
The value is VALIDATED, not rewritten: the box returned is the box the writer receives, so what is checked here is what is stored. Returning the resolved Decimal128 instead would change the box a DECIMAL partition key is formatted from, and the check costs one parse per literal, not per row.
func MatchDMLRows ¶ added in v0.18.5
func MatchDMLRows(ctx context.Context, b *batch.RecordBatch, predicate DMLPredicate, deleted map[int64]bool) (matched []int64, err error)
MatchDMLRows returns the indices of b's rows the statement matches, and is the ONLY place a DMLPredicate is allowed to be called.
deleted is the set of row positions in THIS file that a delete marker has already removed (catalog.DeletedRowsByFile), and passing it is not optional — a nil map means "this file has no markers", not "do not check". The parameter exists rather than a second entry point precisely because the defect was that every DML match scan simply did not look: an UPDATE matched rows its own earlier UPDATEs had superseded, re-ingested them beside the live copy and marked the source file again, so re-updating one row produced 1, then 2, then 4 rows — silently, on plain INT64 columns (#674). The SELECT path has applied this filter all along (scan.Scanner), which is why the row COUNT a client saw was wrong and the DML's own view was internally consistent.
Expression evaluation has no error return (ADR-0019): the one class of condition that cannot answer with a value and must not answer with NULL — a division by zero, an invalid cast — raises a panic carrying a FatalEvalPanic, and a driver converts it back into an error with PostgreSQL's SQLSTATE. Every DML match scan called Eval with NO such boundary, so `DELETE FROM t WHERE 1/0 = 1` over HTTP returned a transport EOF and a goroutine dump instead of 22012 — net/http's own recover, which drops the connection (#677). The embedded and pgwire doors survived only because DB.Execute's own boundary caught it several frames up, and the error a caller got there named the statement rather than the predicate.
The boundary is per FILE SCAN, not per row: one deferred call for a whole batch, no per-row cost. It owns nothing — no lock, no channel, no reservation — so discharging its obligations (ADR-0019 §2a) is exactly returning the error.
Types ¶
type ColumnMeta ¶
type ColumnMeta struct {
Name string
TypeName string // Wadjet type name (e.g., "INT64", "STRING")
TypeID parquet.TypeID // Wadjet type ID
Nullable bool
Precision int // DECIMAL: declared max digits
Scale int // DECIMAL: declared digits after the point
// WireUnconstrained is true for a DECIMAL column produced by an
// aggregate function (MIN/MAX/MIN_BY/MAX_BY/SUM/AVG and any other
// DECIMAL-producing aggregate): Precision/Scale above still carry the
// real declaration for callers that want it, but PostgreSQL's own wire
// protocol reports typmod -1 ("unconstrained numeric") for any such
// column — verified against live postgres:17-alpine's \gdesc, which
// keeps a real typmod only for a BARE column reference. pgTypeMod
// treats this the same as Precision <= 0 (FIX 2, #457/#458 fold-in).
WireUnconstrained bool
}
ColumnMeta describes a result column's type information.
Precision and Scale are the DECLARATION, not the value: a bare TypeID is not a type for a DECIMAL, and the pgwire layer needs them to fill RowDescription's type modifier — PostgreSQL packs a numeric's precision and scale there, and it is where a JDBC or ODBC client reads ResultSetMetaData.getPrecision()/getScale() from. Sending the constant -1 declares an unconstrained numeric, so a tool that sizes a display column or round-trips DDL from a result set got it wrong for every DECIMAL(p,s) column (#454). They are zero for every other type.
type Config ¶
type Config struct {
Store objstore.Store
Bucket string
Logger *slog.Logger
MetaKV catalog.MetaKV // optional: NATS KV for production, nil = in-memory
MemoryBudget int64 // per-query memory budget in bytes (0 = unlimited)
SpillDir string // directory for spill-to-disk files (empty = os temp dir)
AuthProvider *auth.Provider // optional: enables ABAC enforcement at query level
// SortMergeJoinBytes routes inner equi-joins whose sides BOTH exceed this
// estimated size through the sort-merge join instead of the hash join
// (docs/design/sort-merge-join.md). 0 = disabled (default).
SortMergeJoinBytes int64
// LateMaterialization emits inner/left hash-join output as view
// (dictionary) columns with the gather deferred to first touch
// (docs/design/late-materialization.md). Off by default.
LateMaterialization bool
// BushyJoinReorder lets the cost-based join reorder emit bushy plans
// when strictly cheaper than every left-deep order
// (docs/design/bushy-join-cbo.md). PROCESS-WIDE: the logical optimizer
// has no per-query config surface, so Open stores this into a package
// flag shared by every DB in the process. Off by default.
BushyJoinReorder bool
// EnableAlerts turns on the CREATE ALERT scheduler in embedded mode.
// When true, Open() creates a Scheduler that evaluates alerts on cadence.
EnableAlerts bool
}
Config holds configuration for creating a DB instance.
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is the main entry point for embedded usage of Wadjet.
func (*DB) Close ¶
func (db *DB) Close()
Close shuts down any background goroutines started by Open (e.g. alert scheduler). It is safe to call Close multiple times.
func (*DB) CreateTable ¶
func (db *DB) CreateTable(ctx context.Context, name string, schema parquet.Schema, partitionKeys []string) error
CreateTable creates a new table with the given schema and partition keys.
This is one of the RESERVED-NAMESPACE doors. A column whose name is in a hidden-slot family (`__win_N`, `__sortkey_N`, …) is refused here, 42939, because this is where the name is being CREATED. Reading such a column is never refused — a table that already has one stays readable, and the planner renumbers its own slot instead (physical.renameCollidingSlots).
func (*DB) ListTables ¶
ListTables returns all table names.
func (*DB) NewIngester ¶
func (db *DB) NewIngester(tableName string, schema parquet.Schema, partitionKeys []string, cfg ingest.Config) *ingest.Ingester
NewIngester creates a micro-batch ingester for the given table.
The ingest door of the reserved namespace: an Ingester's schema CREATES the table when it does not exist, so a slot-family column name is refused here the way CreateTable refuses it. The error is deferred to the first Ingest call because this constructor returns no error — see Ingester.Ingest.
func (*DB) SetAuthProvider ¶
SetAuthProvider sets the auth provider for ABAC enforcement. This allows wiring auth after DB creation (e.g., when the provider depends on config reload).
type DMLAssignment ¶ added in v0.18.5
type DMLAssignment struct {
Column string
// contains filtered or unexported fields
}
DMLAssignment is one resolved `SET column = ...`: the target column's full declaration, plus EITHER a constant or a compiled expression.
func ResolveDMLSetClauses ¶ added in v0.18.5
func ResolveDMLSetClauses(clauses []plansql.SetClause, target plansql.DMLTarget, schema []parquet.Column) ([]DMLAssignment, error)
ResolveDMLSetClauses resolves an UPDATE's SET list against the table's schema, before anything executes.
Two defects met here (#678). `UPDATE t SET nosuchcol = 1` reported "UPDATE 1": the assignment was dropped into a map nothing read and the matched rows were rewritten unchanged, where PostgreSQL raises 42703. And the value was read ONLY as a literal, through a converter whose STRING arm cannot fail — so `SET s = UPPER(s)` stored the seven characters "UPPER(s)" into the column. PostgreSQL evaluates it, and so does this now.
Whether a SET value is a literal is decided from its PARSE, not from whether a conversion succeeded, because for a STRING column the conversion always succeeds and the literal path always won. A `*plansql.Lit` takes the constant path — which is what keeps #647's declaration checks (ConvertValueForColumn's DECIMAL precision, the temporal accept-sets) running on the values that have them; anything else is compiled and evaluated per row against the file's own batch, which carries the table's declared types.
A SET VALUE resolves against the same relation name the WHERE does, so `UPDATE pr AS a SET n = a.n + 1` reads a.n and `SET n = pr.n` under that alias is 42P01 — PostgreSQL's answer for both (#686).
type DMLPredicate ¶ added in v0.18.5
type DMLPredicate func(*batch.RecordBatch, int) bool
DMLPredicate is a compiled DML WHERE clause: it answers, per row of a scanned file, whether the statement matches it. A nil DMLPredicate matches every row, which is what a DML statement with no WHERE means.
func BuildDMLPredicate ¶ added in v0.18.5
BuildDMLPredicate compiles a DML WHERE clause against a table's schema. An empty clause compiles to nil — "every row".
The SCHEMA is a parameter, not an optional extra, because the DML doors do not go through the planner and so had no name-resolution step at all: `UPDATE t SET n = 1 WHERE nosuchcol = 1` compiled fine, evaluated to NULL on every row and reported "UPDATE 0", where PostgreSQL raises 42703 (#678). Every column the clause names is resolved here, before anything executes.
It is exported because the HTTP DML executors are a second copy of the embedded ones and had a third and fourth copy of this compile step. A predicate is only half of the contract, though; MatchDMLRows is the other half and is the one that must be used to RUN it.
THE EMPTY-PREDICATE BACKSTOP. A nil predicate is the widest answer this function can give — every row of the table — so "the statement had no WHERE" and "the parser dropped the statement's WHERE" must not look the same here. They did, and the second one emptied tables: a DELETE with an aliased table returned an empty WhereSQL and deleted everything (#686). The check below is not about that spelling, which the parser now reads; it makes the CLASS unreachable, so the next clause any parser path fails to carry fails the STATEMENT instead of widening it (ADR-0019, correctness-fix protocol item 8: loud beats plausible).
type ExecResult ¶
ExecResult contains the result of a DML operation (INSERT/UPDATE/DELETE).
type QueryResult ¶
type QueryResult struct {
Columns []string
ColumnMetas []ColumnMeta // typed column metadata (may be nil for introspection queries)
// Rows is the result keyed by column NAME, and it is a convenience: a
// result may legally carry two columns of the same name (PostgreSQL
// answers `SELECT abs(a), abs(b)` with two columns called `abs`, and
// #513 made this engine agree), and a map cannot hold both — the LAST
// one wins and the earlier value is not represented. Columns still lists
// every column, so len(Rows[i]) < len(Columns) is how a caller detects
// it. Read RowValues when the values matter.
Rows []map[string]any
// RowValues is the same result POSITIONALLY, cells aligned with Columns,
// and it is populated ONLY when Rows would lose a value — that is, when
// two output columns share a name. nil means the names are unique and
// Rows is exact. Nothing that transports values (the pgwire DataRow
// path) may read Rows without consulting this first.
RowValues [][]any
Plan string
}
QueryResult contains the result of a SQL query.
func (*QueryResult) Cells ¶ added in v0.18.3
func (r *QueryResult) Cells(i int) []any
Cells returns row i positionally, whether or not the result needed RowValues: from RowValues when duplicate column names made the map lossy, and otherwise by looking each column up in Rows, which is exact there. Returns nil when i is out of range.
Source Files