Documentation
¶
Overview ¶
Package wadjet provides the public embeddable API for Wadjet.
Index ¶
- func BuildUpdatedRows(ctx context.Context, b *batch.RecordBatch, matched []int64, ...) (rows []map[string]any, err error)
- func CheckDMLQualifier(target plansql.DMLTarget) error
- func CommandTag(command string, rows int64) string
- func ConvertTextForColumn(s string, col parquet.Column) (any, error)
- func ConvertValue(s string, typ parquet.TypeID) (any, error)
- func ConvertValueForColumn(s string, col parquet.Column) (any, error)
- func MatchDMLRows(ctx context.Context, b *batch.RecordBatch, predicate DMLPredicate, ...) (matched []int64, err error)
- type ColumnMeta
- type Config
- type DB
- func (db *DB) Catalog() *catalog.Catalog
- func (db *DB) Close()
- func (db *DB) CreateTable(ctx context.Context, name string, schema parquet.Schema, ...) error
- func (db *DB) DropTable(ctx context.Context, name string) error
- func (db *DB) Execute(ctx context.Context, sql string) (*ExecResult, error)
- func (db *DB) ExecuteParsed(ctx context.Context, parsed *plansql.ParsedQuery) (res *ExecResult, err error)
- func (db *DB) ListTables(ctx context.Context) ([]string, error)
- func (db *DB) NewIngester(tableName string, schema parquet.Schema, partitionKeys []string, ...) *ingest.Ingester
- func (db *DB) Query(ctx context.Context, sql string) (res *QueryResult, err error)
- func (db *DB) ReadDataFile(ctx context.Context, filePath string, schema []parquet.Column) (*batch.RecordBatch, error)
- func (db *DB) SetAuthProvider(p *auth.Provider)
- func (db *DB) SetQueryLimits(global *config.QueryLimits, perRole map[string]*config.QueryLimits)
- func (db *DB) Store() objstore.Store
- type DMLAssignment
- type DMLPredicate
- type ExecResult
- type QueryResult
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildUpdatedRows ¶ added in v0.18.5
func BuildUpdatedRows(ctx context.Context, b *batch.RecordBatch, matched []int64, assigns []DMLAssignment) (rows []map[string]any, err error)
BuildUpdatedRows boxes b's matched rows with the assignments applied.
It carries the same panic boundary MatchDMLRows does and for the same reason: a SET expression is evaluated by the same engine as a WHERE, so `SET n = 1/0` raises a FatalEvalPanic that must become 22012 on the statement rather than a dead connection (ADR-0019, #677). One deferred call per file, nothing per row.
A value the target column cannot hold is refused HERE, before any delete marker is committed — the rule #647 established for literals, applied to computed values too.
func CheckDMLQualifier ¶ added in v0.18.5
CheckDMLQualifier accepts the schema/catalog qualifier of a DML relation when it names this server's own schema, and refuses any other.
`DELETE FROM public.orders` is what a PostgreSQL client writes by default, and the DML parser used to read only the first identifier — so the statement addressed a table named "public". This is the SELECT path's rule (logical/builder.go, buildScan) applied to the DML doors; PostgreSQL 17 answers an unknown qualifier with 42P01.
func CommandTag ¶ added in v0.18.20
CommandTag is Tag for a caller holding the verb and the count separately.
PostgreSQL's INSERT tag carries an oid field that has been fixed at 0 since 12; every other verb is `VERB <rows>`. Measured, not remembered.
func ConvertTextForColumn ¶ added in v0.18.20
ConvertTextForColumn converts RAW TEXT — not a SQL literal — to the box a column stores.
The difference from ConvertValueForColumn is the two rules that belong to LITERAL text and to nothing else: the word `null` is the SQL keyword, and a leading and trailing apostrophe are quoting. A COPY field is neither. It used to go through the literal converter, so a COPY field spelled `NULL` became a SQL NULL — even though COPY's own NULL marker is `\N` and is handled before this — and a field whose text happened to begin and end with an apostrophe silently lost both (#690's third site).
func ConvertValue ¶
convertValue converts a string value from the parser to the appropriate Go type based on the target column's type. ConvertValue converts a string value to the appropriate Go type for a given Parquet type.
func ConvertValueForColumn ¶ added in v0.18.5
ConvertValueForColumn converts a literal's text against a column's FULL declaration rather than its TypeID alone, so a value the column cannot hold is refused HERE — before the caller commits anything destructive.
ConvertValue is handed a TypeID and nothing else, so a DECIMAL literal passes through it as text and is first judged at the parquet leaf, where the declared (p, s) is known. That was harmless while nothing could refuse it, and became DATA LOSS the moment something could: executeUpdate wrote a file's delete markers and ingested afterwards, so `UPDATE u SET d = 99999999999999999999.99` answered 22003 with the matched rows already deleted, and three such failures emptied a three-row table (#647 review). A value conversion that can fail must run before the first irreversible step, and this is where the declaration to check against lives.
The value is VALIDATED, not rewritten: the box returned is the box the writer receives, so what is checked here is what is stored. Returning the resolved Decimal128 instead would change the box a DECIMAL partition key is formatted from, and the check costs one parse per literal, not per row.
func MatchDMLRows ¶ added in v0.18.5
func MatchDMLRows(ctx context.Context, b *batch.RecordBatch, predicate DMLPredicate, deleted map[int64]bool) (matched []int64, err error)
MatchDMLRows returns the indices of b's rows the statement matches, and is the ONLY place a DMLPredicate is allowed to be called.
deleted is the set of row positions in THIS file that a delete marker has already removed (catalog.DeletedRowsByFile), and passing it is not optional — a nil map means "this file has no markers", not "do not check". The parameter exists rather than a second entry point precisely because the defect was that every DML match scan simply did not look: an UPDATE matched rows its own earlier UPDATEs had superseded, re-ingested them beside the live copy and marked the source file again, so re-updating one row produced 1, then 2, then 4 rows — silently, on plain INT64 columns (#674). The SELECT path has applied this filter all along (scan.Scanner), which is why the row COUNT a client saw was wrong and the DML's own view was internally consistent.
Expression evaluation has no error return (ADR-0019): the one class of condition that cannot answer with a value and must not answer with NULL — a division by zero, an invalid cast — raises a panic carrying a FatalEvalPanic, and a driver converts it back into an error with PostgreSQL's SQLSTATE. Every DML match scan called Eval with NO such boundary, so `DELETE FROM t WHERE 1/0 = 1` over HTTP returned a transport EOF and a goroutine dump instead of 22012 — net/http's own recover, which drops the connection (#677). The embedded and pgwire doors survived only because DB.Execute's own boundary caught it several frames up, and the error a caller got there named the statement rather than the predicate.
The boundary is per FILE SCAN, not per row: one deferred call for a whole batch, no per-row cost. It owns nothing — no lock, no channel, no reservation — so discharging its obligations (ADR-0019 §2a) is exactly returning the error.
Types ¶
type ColumnMeta ¶
type ColumnMeta struct {
Name string
TypeName string // Wadjet type name (e.g., "INT64", "STRING")
TypeID parquet.TypeID // Wadjet type ID
Nullable bool
Precision int // DECIMAL: declared max digits
Scale int // DECIMAL: declared digits after the point
// WireUnconstrained is true for a DECIMAL column produced by an
// aggregate function (MIN/MAX/MIN_BY/MAX_BY/SUM/AVG and any other
// DECIMAL-producing aggregate): Precision/Scale above still carry the
// real declaration for callers that want it, but PostgreSQL's own wire
// protocol reports typmod -1 ("unconstrained numeric") for any such
// column — verified against live postgres:17-alpine's \gdesc, which
// keeps a real typmod only for a BARE column reference. pgTypeMod
// treats this the same as Precision <= 0 (FIX 2, #457/#458 fold-in).
WireUnconstrained bool
}
ColumnMeta describes a result column's type information.
Precision and Scale are the DECLARATION, not the value: a bare TypeID is not a type for a DECIMAL, and the pgwire layer needs them to fill RowDescription's type modifier — PostgreSQL packs a numeric's precision and scale there, and it is where a JDBC or ODBC client reads ResultSetMetaData.getPrecision()/getScale() from. Sending the constant -1 declares an unconstrained numeric, so a tool that sizes a display column or round-trips DDL from a result set got it wrong for every DECIMAL(p,s) column (#454). They are zero for every other type.
type Config ¶
type Config struct {
Store objstore.Store
Bucket string
Logger *slog.Logger
MetaKV catalog.MetaKV // optional: NATS KV for production, nil = in-memory
MemoryBudget int64 // per-query memory budget in bytes (0 = unlimited)
SpillDir string // directory for spill-to-disk files (empty = os temp dir)
AuthProvider *auth.Provider // optional: enables ABAC enforcement at query level
// SortMergeJoinBytes routes inner equi-joins whose sides BOTH exceed this
// estimated size through the sort-merge join instead of the hash join
// (docs/design/sort-merge-join.md). 0 = disabled (default).
SortMergeJoinBytes int64
// LateMaterialization emits inner/left hash-join output as view
// (dictionary) columns with the gather deferred to first touch
// (docs/design/late-materialization.md). Off by default.
LateMaterialization bool
// BushyJoinReorder lets the cost-based join reorder emit bushy plans
// when strictly cheaper than every left-deep order
// (docs/design/bushy-join-cbo.md). PROCESS-WIDE: the logical optimizer
// has no per-query config surface, so Open stores this into a package
// flag shared by every DB in the process. Off by default.
BushyJoinReorder bool
// EnableAlerts turns on the CREATE ALERT scheduler in embedded mode.
// When true, Open() creates a Scheduler that evaluates alerts on cadence.
EnableAlerts bool
// QueryLimits / RoleLimits are the cost guard (docs/security.md,
// "Query Cost Estimation and Guards"): the global limits, and the
// per-role overrides keyed by role name (an entry present with a nil
// value means that role is unlimited).
//
// The embedded planner is the LAST entry point that could reach a scan
// without them. Every other one — HTTP, gRPC, and the pgwire statements
// the coordinator answers — goes through Coordinator.ExecuteSQL, which
// carries the same limits; but pgwire falls back to this DB for any
// statement its routing gate declines (a leading comment, TABLE, VALUES)
// and for every statement when a provider is present but disabled, so a
// guard that stopped at the coordinator left the PostgreSQL wire — the
// protocol the BI clients use — unbounded (#803).
QueryLimits *config.QueryLimits
RoleLimits map[string]*config.QueryLimits
}
Config holds configuration for creating a DB instance.
type DB ¶
type DB struct {
// contains filtered or unexported fields
}
DB is the main entry point for embedded usage of Wadjet.
func Attach ¶ added in v0.18.20
Attach wraps an ALREADY-INITIALIZED catalog as a DB, without creating a second catalog or re-running Init.
It exists so a process that already owns a *catalog.Catalog — the HTTP API server in coordinator mode — reaches DML through the SAME implementation the embedded and pgwire doors use, instead of keeping its own copy of the executors. internal/server carried an independent INSERT/UPDATE/DELETE that had drifted into the same defects as this package's (#815): an unresolved INSERT column list (#814), the marker/manifest window (#691), the literal kind discarded in SET (#690) — and no MERGE at all. A fix written once has to be reachable from both doors, and this is that reach.
The returned DB owns no background goroutines (Open's alert scheduler is not started), so it needs no Close; the caller keeps ownership of the catalog.
func Open ¶
Open creates and initializes a new Wadjet database.
Example ¶
ExampleOpen is the README's "Start embedded, scale distributed" snippet, compiled. It has no Output comment on purpose: `go test` builds an example without one but never runs it, which is all this needs to be — a compile gate so the three lines a reader copies cannot drift from the API the way the pre-2026-09-02 README example did (it named a Config field and a constructor arity that no longer existed).
package main
import (
"context"
"github.com/derekmwright/wadjet/internal/storage/objstore"
"github.com/derekmwright/wadjet/wadjet"
)
func main() {
ctx := context.Background()
store, _ := objstore.NewFileStore("/var/lib/wadjet") // or NewMinIOStore(...) for S3
db, _ := wadjet.Open(ctx, wadjet.Config{Store: store, Bucket: "analytics"})
res, _ := db.Query(ctx, "SELECT src_ip, SUM(bytes_in) FROM flow_logs GROUP BY 1")
_ = res
defer db.Close()
}
Output:
func (*DB) Close ¶
func (db *DB) Close()
Close shuts down any background goroutines started by Open (e.g. alert scheduler). It is safe to call Close multiple times.
func (*DB) CreateTable ¶
func (db *DB) CreateTable(ctx context.Context, name string, schema parquet.Schema, partitionKeys []string) error
CreateTable creates a new table with the given schema and partition keys.
This is one of the RESERVED-NAMESPACE doors. A column whose name is in a hidden-slot family (`__win_N`, `__sortkey_N`, …) is refused here, 42939, because this is where the name is being CREATED. Reading such a column is never refused — a table that already has one stays readable, and the planner renumbers its own slot instead (physical.renameCollidingSlots).
func (*DB) Execute ¶
Execute runs a DML statement (INSERT/UPDATE/DELETE/MERGE) and returns the result.
func (*DB) ExecuteParsed ¶ added in v0.18.20
func (db *DB) ExecuteParsed(ctx context.Context, parsed *plansql.ParsedQuery) (res *ExecResult, err error)
ExecuteParsed runs an already-parsed DML statement.
It is the ONE DML entry point (#815): the embedded door, the pgwire door and the HTTP API server all reach the executors below through it, so a fix is written once and every door carries the same table state, command tag and SQLSTATE. It is exported for the callers that have already parsed — the HTTP handler routes on the statement type before dispatching — and for the tests that drive a synthesized statement no text can spell.
func (*DB) ListTables ¶
ListTables returns all table names.
func (*DB) NewIngester ¶
func (db *DB) NewIngester(tableName string, schema parquet.Schema, partitionKeys []string, cfg ingest.Config) *ingest.Ingester
NewIngester creates a micro-batch ingester for the given table.
The ingest door of the reserved namespace: an Ingester's schema CREATES the table when it does not exist, so a slot-family column name is refused here the way CreateTable refuses it. The error is deferred to the first Ingest call because this constructor returns no error — see Ingester.Ingest.
func (*DB) ReadDataFile ¶ added in v0.18.20
func (db *DB) ReadDataFile(ctx context.Context, filePath string, schema []parquet.Column) (*batch.RecordBatch, error)
readParquetFile downloads and decodes a Parquet file into a RecordBatch. ReadDataFile reads one of a table's data files as a columnar batch, through the exact path the DML executors read it.
Exported for the gates that assert what a statement left in a specific FILE rather than what a query returns: delete markers are metadata, so a file's surviving rows can only be seen by reading the file and applying them (#815 folded the HTTP door's own copy of this reader into this one).
func (*DB) SetAuthProvider ¶
SetAuthProvider sets the auth provider for ABAC enforcement. This allows wiring auth after DB creation (e.g., when the provider depends on config reload).
func (*DB) SetQueryLimits ¶ added in v0.18.15
func (db *DB) SetQueryLimits(global *config.QueryLimits, perRole map[string]*config.QueryLimits)
SetQueryLimits installs the cost guard after Open, for callers that build the DB before they have read the config (the `serve` command opens the pgwire DB alongside the coordinator). Call before serving traffic, the same contract as Coordinator.SetQueryLimits.
type DMLAssignment ¶ added in v0.18.5
type DMLAssignment struct {
Column string
// contains filtered or unexported fields
}
DMLAssignment is one resolved `SET column = ...`: the target column's full declaration, plus EITHER a constant or a compiled expression.
func ResolveDMLSetClauses ¶ added in v0.18.5
func ResolveDMLSetClauses(clauses []plansql.SetClause, target plansql.DMLTarget, schema []parquet.Column) ([]DMLAssignment, error)
ResolveDMLSetClauses resolves an UPDATE's SET list against the table's schema, before anything executes.
Two defects met here (#678). `UPDATE t SET nosuchcol = 1` reported "UPDATE 1": the assignment was dropped into a map nothing read and the matched rows were rewritten unchanged, where PostgreSQL raises 42703. And the value was read ONLY as a literal, through a converter whose STRING arm cannot fail — so `SET s = UPPER(s)` stored the seven characters "UPPER(s)" into the column. PostgreSQL evaluates it, and so does this now.
Whether a SET value is a literal is decided from its PARSE, not from whether a conversion succeeded, because for a STRING column the conversion always succeeds and the literal path always won. A `*plansql.Lit` takes the constant path — which is what keeps #647's declaration checks (ConvertValueForColumn's DECIMAL precision, the temporal accept-sets) running on the values that have them; anything else is compiled and evaluated per row against the file's own batch, which carries the table's declared types.
A SET VALUE resolves against the same relation name the WHERE does, so `UPDATE pr AS a SET n = a.n + 1` reads a.n and `SET n = pr.n` under that alias is 42P01 — PostgreSQL's answer for both (#686).
type DMLPredicate ¶ added in v0.18.5
type DMLPredicate func(*batch.RecordBatch, int) bool
DMLPredicate is a compiled DML WHERE clause: it answers, per row of a scanned file, whether the statement matches it. A nil DMLPredicate matches every row, which is what a DML statement with no WHERE means.
func BuildDMLPredicate ¶ added in v0.18.5
BuildDMLPredicate compiles a DML WHERE clause against a table's schema. An empty clause compiles to nil — "every row".
The SCHEMA is a parameter, not an optional extra, because the DML doors do not go through the planner and so had no name-resolution step at all: `UPDATE t SET n = 1 WHERE nosuchcol = 1` compiled fine, evaluated to NULL on every row and reported "UPDATE 0", where PostgreSQL raises 42703 (#678). Every column the clause names is resolved here, before anything executes.
It is exported because MatchDMLRows is the other half of the contract and is the one that must be used to RUN a predicate. (The HTTP door is no longer a second caller: since #815 it reaches the executors through DB.ExecuteParsed like everything else.)
DEFERRED — #688: A SUBQUERY IN A DML PREDICATE.
`DELETE … WHERE id IN (SELECT …)`, `NOT IN (SELECT …)`, a scalar subquery and a correlated EXISTS are all 0A000 here, and the reason is structural rather than local: THIS FUNCTION IS NOT A PLANNER. It parses, resolves the column names against the target's schema, and calls expr.Compile with a NIL runner. Every planner-resident guarantee is therefore absent on this door — the subquery runner, the decorrelation rewrites, the plan-time literal refusals (#721's had to be re-implemented here for exactly that reason) and node-coverage validation.
The STRUCTURAL fix is to plan a DML predicate through the normal planner, so that a DML WHERE is a WHERE. The BOUNDED one — handing expr.CompileWithRunner a db.Query closure — is three lines, closes `IN (SELECT …)`, `NOT IN (SELECT …)` and the scalar subquery, and leaves CORRELATED EXISTS refused. Correlated EXISTS is the shape #688's own body names first, so rule 11 of docs/design/correctness-fix-protocol.md forbids shipping it: a fix bounded by a model that leaves the issue's headline shape pinned is a surface repair standing where a structural one belongs, and the disposition is to DEFER with the mechanism written down. This is that writing-down; the five #688 entries in the DML census (internal/server/pgwire/dml_census_test.go) are its pin, each carrying PostgreSQL 17's answer beside the refusal.
THE EMPTY-PREDICATE BACKSTOP. A nil predicate is the widest answer this function can give — every row of the table — so "the statement had no WHERE" and "the parser dropped the statement's WHERE" must not look the same here. They did, and the second one emptied tables: a DELETE with an aliased table returned an empty WhereSQL and deleted everything (#686). The check below is not about that spelling, which the parser now reads; it makes the CLASS unreachable, so the next clause any parser path fails to carry fails the STATEMENT instead of widening it (ADR-0019, correctness-fix protocol item 8: loud beats plausible).
type ExecResult ¶
ExecResult contains the result of a DML operation (INSERT/UPDATE/DELETE).
func (*ExecResult) Tag ¶ added in v0.18.20
func (r *ExecResult) Tag() string
Tag renders this result the way PostgreSQL's CommandComplete does.
It is a method on the result rather than a private helper in one door because the doors DISAGREED: pgwire special-cased INSERT to PostgreSQL's three-field `INSERT <oid> <rows>` form and the HTTP door rendered `fmt.Sprintf("%s %d", …)`, so the same statement was `INSERT 0 3` over the wire and `INSERT 3` over REST — while docs/api-reference.md claimed the tag does not depend on the door (review B8). One renderer is the only way that claim can be true.
type QueryResult ¶
type QueryResult struct {
Columns []string
ColumnMetas []ColumnMeta // typed column metadata (may be nil for introspection queries)
// Rows is the result keyed by column NAME, and it is a convenience: a
// result may legally carry two columns of the same name (PostgreSQL
// answers `SELECT abs(a), abs(b)` with two columns called `abs`, and
// #513 made this engine agree), and a map cannot hold both — the LAST
// one wins and the earlier value is not represented. Columns still lists
// every column, so len(Rows[i]) < len(Columns) is how a caller detects
// it. Read RowValues when the values matter.
Rows []map[string]any
// RowValues is the same result POSITIONALLY, cells aligned with Columns,
// and it is populated ONLY when Rows would lose a value — that is, when
// two output columns share a name. nil means the names are unique and
// Rows is exact. Nothing that transports values (the pgwire DataRow
// path) may read Rows without consulting this first.
RowValues [][]any
Plan string
}
QueryResult contains the result of a SQL query.
func (*QueryResult) Cells ¶ added in v0.18.3
func (r *QueryResult) Cells(i int) []any
Cells returns row i positionally, whether or not the result needed RowValues: from RowValues when duplicate column names made the map lossy, and otherwise by looking each column up in Rows, which is exact there. Returns nil when i is out of range.
Source Files