Documentation
¶
Index ¶
- Constants
- func BatchBytes(b *dataplane.Batch) int64
- func ConcatRecords(alloc memory.Allocator, parts []arrow.RecordBatch) (arrow.RecordBatch, error)
- func CountOps(b *dataplane.Batch) (upserts, deletes int)
- func EmptyBatch(b *dataplane.Batch, mode dataplane.WriteMode) *dataplane.Batch
- func EncodeKey(record arrow.RecordBatch, row int, pkIdxs []int, pkCols []string) ([]byte, error)
- func LastRowPos(b *dataplane.Batch) string
- func SelectRows(b *dataplane.Batch, idx []int32, pos string, mode dataplane.WriteMode, ...) (*dataplane.Batch, error)
- type Batch
Constants ¶
const ( OpInsert = 0 OpUpdate = 1 OpDelete = 2 )
Op values matching the wire schema __op column (CR-021).
Variables ¶
This section is empty.
Functions ¶
func BatchBytes ¶ added in v0.4.0
BatchBytes is a batch's in-memory size: the length of every buffer of every column, child arrays included. Rows carry payloads, so a row count alone is no bound on memory: batching limits check this as well (#437).
func ConcatRecords ¶ added in v0.5.0
func ConcatRecords(alloc memory.Allocator, parts []arrow.RecordBatch) (arrow.RecordBatch, error)
ConcatRecords concatenates Arrow records that share a schema (the chunk encoder's per-column concatenation). The caller owns the result.
func CountOps ¶ added in v0.5.0
CountOps returns the number of upsert and delete rows in a batch by scanning its __op column. Used by OnCommit observers for bookkeeping.
func EmptyBatch ¶ added in v0.5.0
EmptyBatch returns a 0-row batch with b's schema, carrying b's Seq and watermark. A staged cycle must still deliver a descriptor when its sub-batch produced no data — e.g. an append-mode delete with no before image: the coordinator expects exactly one delivery per (table, seq), and a missing one leaves the cycle open, blocking every cycle behind it in the table's send order.
func EncodeKey ¶
EncodeKey encodes a composite key as type-tagged binary payload concat — used by collapse to avoid hash collisions (CR-069 §3.2).
pkIdxs are the pre-resolved data-column indices of the PK columns (resolved ONCE by the caller — M-9); pkCols ride along for error messages only. Null in any PK column is an error here — EncodeKey is the single authority for key validity (M-9).
Each field is [type-byte][payload…]: fixed-width payloads for numerics/bool/date/time/timestamp, length- prefixed for string/binary/decimal, raw fixed width for FixedSizeBinary. Different types cannot collide (distinct tags); equal values of the same type always produce equal keys (NaN/±0 compare by bit pattern — M-11).
INVARIANT: keys are EPHEMERAL — they live for the duration of one Collapse call and never persist or cross batches. If keys ever need to persist or be compared across batches, this design must be revisited (type widening across batches, e.g. int32→int64, would change keys). See website/docs/architecture/encode-key.md.
func LastRowPos ¶ added in v0.5.0
LastRowPos returns the __pos of the batch's last row (its commit coordinate), or "" for an empty batch.
func SelectRows ¶ added in v0.5.0
func SelectRows(b *dataplane.Batch, idx []int32, pos string, mode dataplane.WriteMode, snapState string, snapPending []uint32) (*dataplane.Batch, error)
SelectRows returns a new owned batch holding the rows at the given indices, with the given mode and position. The input is NOT released.
Types ¶
type Batch ¶
Batch is the seam type. It lives in the public dataplane package so that source, sink, and plugin contracts can reference it without violating the internal/ boundary. The alias means internal/dataplane and dataplane.Batch are the same type — zero conversion cost.
func Collapse ¶
func Collapse(ctx context.Context, alloc memory.Allocator, batch *Batch, pkCols []string) (upserts, deletes *Batch, err error)
Collapse deduplicates by PK, keeping last-occurrence per group in first-appearance emission order (CR-069 §3.2).
The encoding MUST be exact: length-prefixed binary concat of encoded key columns. Naive concat ("ab"+"c" == "a"+"bc") collides.
Null in a PK column is a source contract violation → error.
func ConcatBatches ¶ added in v0.5.0
ConcatBatches concatenates batches that must share a schema, skipping empty ones. It returns nil when nothing carries rows. The returned batch owns its record (a single input is Retained, never aliased to the caller's pointer).
func MergeBatches ¶ added in v0.5.0
MergeBatches concatenates two batches with the same schema, preserving a, b's metadata (the result is append mode, like the collapse's upserts + deletes merge). Either side may be nil or empty. The returned batch owns its record; the caller still owns and releases a and b.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package dataplanetest holds the batch fixtures the engine's tests build their inputs from.
|
Package dataplanetest holds the batch fixtures the engine's tests build their inputs from. |