Documentation
¶
Overview ¶
Package arrowbatch owns Arrow row batches, builders, schema mapping, and in-memory integrity checks.
Index ¶
- Constants
- Variables
- func Bytes(rows arrow.RecordBatch) int64
- func CRC(rows arrow.RecordBatch, ops Operations) uint32
- func Field(f rowmodel.Field) arrow.Field
- func LogicalOf(f arrow.Field) rowmodel.LogicalType
- func NativeOf(f arrow.Field) string
- func Schema(rs rowmodel.Schema) *arrow.Schema
- func Type(f rowmodel.Field) arrow.DataType
- type Batch
- func (b *Batch) Bytes() int64
- func (b *Batch) IntegrityCRC() uint32
- func (b *Batch) NumRows() int
- func (b *Batch) Op(i int) rowmodel.Operation
- func (b *Batch) Operations() Operations
- func (b *Batch) Release()
- func (b *Batch) Retain() *Batch
- func (b *Batch) Rows() arrow.RecordBatch
- func (b *Batch) SeqString() string
- func (b *Batch) WithRows(rows arrow.RecordBatch) *Batch
- type Builder
- func (b *Builder) Bool(v bool)
- func (b *Builder) Bytes(v []byte)
- func (b *Builder) Close() error
- func (b *Builder) Closed() bool
- func (b *Builder) Date(days int32)
- func (b *Builder) Decimal(v decimal128.Num)
- func (b *Builder) Drain(meta rowmodel.Meta) error
- func (b *Builder) EndRow(meta rowmodel.Meta) error
- func (b *Builder) Float32(v float32)
- func (b *Builder) Float64(v float64)
- func (b *Builder) Flush() error
- func (b *Builder) Int16(v int16)
- func (b *Builder) Int32(v int32)
- func (b *Builder) Int64(v int64)
- func (b *Builder) Null()
- func (b *Builder) RequestFlush()
- func (b *Builder) Schema() *arrow.Schema
- func (b *Builder) String(v string)
- func (b *Builder) StringBytes(v []byte)
- func (b *Builder) Time(micros int64)
- func (b *Builder) Timestamp(micros int64)
- type Inlet
- type Operations
- type Options
- type Receiver
- type RowWriter
Constants ¶
const ( MetaLogical = "filament.logical" MetaNative = "filament.native" )
Field metadata keys carrying the source-side type of each column, so a sink can map back from the Arrow storage type (uuid, json and unbounded numeric all travel as utf8) and a same-engine sink can reuse the native spelling.
Variables ¶
var ErrClosed = errors.New("arrowbatch: builder closed")
ErrClosed reports use of a closed builder.
Functions ¶
func Bytes ¶
func Bytes(rows arrow.RecordBatch) int64
Bytes returns the memory a record batch's column buffers occupy.
func CRC ¶
func CRC(rows arrow.RecordBatch, ops Operations) uint32
CRC returns the CRC32C over a batch's rows and ops. It hashes each column's buffers as laid out (validity bitmap, offsets, values, over the batch's own row range), each prefixed with its length, and the row and column counts, so regrouping rows or columns changes the result. The pipeline writer computes it before Apply and the sink recomputes it over the same batch for the receipt; it guards the batch in memory, not a re-encoding of it.
func LogicalOf ¶
func LogicalOf(f arrow.Field) rowmodel.LogicalType
LogicalOf returns the logical type a field was built from.
Types ¶
type Batch ¶
type Batch struct {
Resource string
Part int
Seq uint64
Cursor *rowmodel.CheckpointData
Drained bool
// Control is a borrowed, immutable stream marker, independent of Drained.
Control *rowmodel.Control
Last rowmodel.Meta
// contains filtered or unexported fields
}
Batch owns one Arrow record batch and its operation vector. A newly created Batch has one reference. Rows and Operations are borrowed views valid only while the caller owns a reference to Batch.
Batch must be passed by pointer. On successful handoff, ownership transfers to the receiver; callers retain ownership when a handoff returns an error.
func NewBatch ¶
func NewBatch(rows arrow.RecordBatch, ops Operations) *Batch
NewBatch takes ownership of rows and ops. Ops must not be mutated afterward.
func NewControl ¶ added in v0.6.0
NewControl validates and clones a rowless stream control. The returned batch owns its position bytes and follows the same handoff rules as a data batch.
func (*Batch) IntegrityCRC ¶
IntegrityCRC computes the in-memory integrity checksum for b.
func (*Batch) Operations ¶
func (b *Batch) Operations() Operations
Operations returns a borrowed, immutable operation vector. Nil means inserts.
func (*Batch) Release ¶
func (b *Batch) Release()
Release drops an ownership reference and releases Arrow memory exactly once.
func (*Batch) Retain ¶
Retain adds an ownership reference. Retaining an already released batch is a programming error and panics deterministically instead of reviving freed Arrow memory.
func (*Batch) Rows ¶
func (b *Batch) Rows() arrow.RecordBatch
Rows returns a borrowed Arrow record batch, valid until the caller releases its reference to Batch.
func (*Batch) WithRows ¶ added in v0.21.0
func (b *Batch) WithRows(rows arrow.RecordBatch) *Batch
WithRows returns a new owned batch carrying rows in place of b's, sharing b's operations and metadata. It takes ownership of rows; b is untouched and still needs its own Release. Row count must match so operations stay aligned.
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder is the filament.RowWriter for one (resource, part): typed column builders in schema order that become an Arrow record batch when a chunk fills. It belongs to the one goroutine that appends to it; the only thing another goroutine may do is RequestFlush, which EndRow honors once the row is closed. Errors are sticky: after the first, appends are no-ops and every method returns it.
func NewBuilder ¶
NewBuilder returns a builder for schema that hands owned batches to recv.
func (*Builder) Close ¶
Close releases the column builders. It is idempotent and discards any unflushed or partial row; callers explicitly Flush or Drain before closing when those rows are valid.
func (*Builder) Decimal ¶
func (b *Builder) Decimal(v decimal128.Num)
Decimal appends a decimal128 value at the column's scale.
func (*Builder) Drain ¶
Drain flushes the open chunk and marks the part complete: meta carries a change stream's final position, and the receiver gets the part's total rows.
func (*Builder) EndRow ¶
EndRow closes the row with its meta and flushes the chunk once it reaches MaxRows or MaxBytes, or a flush was requested. It returns the builder's sticky error, including a row short of columns.
func (*Builder) Flush ¶
Flush hands the open chunk to the receiver, if it has any rows. A source with an idle stream calls it so buffered rows do not wait for the next one.
func (*Builder) RequestFlush ¶
func (b *Builder) RequestFlush()
RequestFlush asks the appending goroutine to flush at the next EndRow. Safe from any goroutine; the pipeline's flush timer uses it.
func (*Builder) StringBytes ¶
StringBytes appends a utf8 value held as bytes.
type Inlet ¶
type Inlet interface {
Builder(resource string, part int, schema rowmodel.Schema) (RowWriter, error)
}
Inlet is the source-facing entry point for opening typed row writers.
type Operations ¶
type Operations struct {
// contains filtered or unexported fields
}
Operations is an immutable row-operation vector. An empty vector means every row is an insert. Its backing slice is never exposed, preventing sinks from mutating batch semantics through a borrowed reference.
func NewOperations ¶
func NewOperations(values []rowmodel.Operation) Operations
NewOperations copies values into an immutable operation vector.
func (Operations) At ¶
func (o Operations) At(i int) rowmodel.Operation
At returns the explicitly stored operation at i.
func (Operations) Clone ¶
func (o Operations) Clone() []rowmodel.Operation
Clone returns a caller-owned copy for storage outside the batch lifetime.
func (Operations) Len ¶
func (o Operations) Len() int
Len returns the number of explicitly stored operations. Zero means inserts.
type Options ¶
type Options struct {
MaxRows int // rows per chunk
MaxBytes int64 // approximate value bytes per chunk; 0 = unbounded
}
Options bounds the chunks a Builder emits.
type Receiver ¶
Receiver takes the owned batches a Builder produces. Chunk runs on the appending goroutine and may block to apply backpressure. On success ownership transfers to Receiver; on error the Builder retains ownership and releases the batch. Drained reports completion of the part after rows total rows.
type RowWriter ¶
type RowWriter interface {
Null()
Bool(bool)
Int16(int16)
Int32(int32)
Int64(int64)
Float32(float32)
Float64(float64)
Decimal(decimal128.Num)
String(string)
StringBytes([]byte)
Bytes([]byte)
Date(int32)
Time(int64)
Timestamp(int64)
EndRow(rowmodel.Meta) error
Flush() error
Drain(rowmodel.Meta) error
Close() error
}
RowWriter appends one schema-ordered row at a time.
Source Files