Documentation
¶
Overview ¶
Package record defines streamio's canonical, format-neutral row: the intermediate every cross-format conversion goes through between a decoder and an encoder, and what a custom pipeline stage reads and writes.
A Record is a flat []Field of tagged-union Values, not a map[string]any, so decoding a scalar column costs no heap allocation or interface boxing. KindMap and KindList are the two exceptions that nest — a Parquet-style key/value column group, and an ordered JSON array, respectively; everything else a decoder can't express in these Kinds is an error rather than an invented representation.
Lifetime ¶
A Record and everything it references (Value.Str, Value.Map, Value.List, and the field names' backing storage) belongs to the decoder that produced it and is reused for the next batch. Code that receives one — a RowFilter, a RecordTransform, a RecordConsumer — may read it freely during the call but must copy anything it wants to keep (string(v.Str), append([]byte(nil), v.Str...)). A stage may replace a field's Value or truncate the Record it is handed, but must not retain a slice into the Record after returning.
Stability ¶
This package is experimental: it is the first public commitment to the record model, and its API may change in a v0 release if building real stages shows it needs to.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Arena ¶
type Arena struct {
// contains filtered or unexported fields
}
Arena is batch-lived scratch memory a pipeline stage writes its output values into.
A stage that replaces a field's bytes must put the new bytes somewhere that stays valid until the batch is encoded. A single reused buffer does not: the next record's value would overwrite the previous record's. The pool hands every worker one Arena and resets it at the start of each batch, so everything copied into it lives exactly as long as the batch does.
type Kind ¶
type Kind uint8
Kind is the physical type a Value carries. It selects which of Value's fixed fields holds the data; every other field is unspecified and must not be read.
const ( // KindNull is a null/absent value. No Value field carries data. KindNull Kind = iota // KindBool reads Value.Bool. KindBool // KindInt64 reads Value.I64. KindInt64 // KindFloat64 reads Value.F64. KindFloat64 // KindBytes reads Value.Str, holding either a UTF-8 string or raw bytes. KindBytes // KindMap reads Value.Map, a nested Record holding one key/value column group's entries. KindMap // KindList reads Value.List, an ordered slice of nested Values holding a JSON array's elements. KindList )
type Record ¶
type Record []Field
Record is one row: a reusable []Field buffer filled via Reset and Append rather than allocated fresh per row.
func (Record) Append ¶
Append adds a named field to r and returns the grown Record, exactly as append would.
type Semantic ¶
type Semantic uint8
Semantic tags a Value with what its physical Kind actually represents, when that isn't deducible from Kind alone — an Int64 that is really a day count or an epoch timestamp, say.
Decoders only classify, never format: a date decodes to its raw day count, not a rendered string, so encoders stay free to render the same value differently and decoding stays allocation-free.
const ( // SemanticNone means the Kind fully describes the value; no reinterpretation applies. SemanticNone Semantic = iota // SemanticDate marks a KindInt64 holding a day count since the Unix epoch. SemanticDate // SemanticTimestampMillis marks a KindInt64 holding an epoch timestamp in milliseconds. SemanticTimestampMillis // SemanticTimestampMicros marks a KindInt64 holding an epoch timestamp in microseconds. SemanticTimestampMicros // SemanticTimestampNanos marks a KindInt64 holding an epoch timestamp in nanoseconds. SemanticTimestampNanos // SemanticUnsigned marks a KindInt64 whose bits are to be read as a uint64. SemanticUnsigned // SemanticFloat32 marks a KindFloat64 widened from a 32-bit float, so an encoder renders it at // its original precision rather than float64's. SemanticFloat32 )
type Value ¶
type Value struct {
// Str is valid for KindBytes.
Str []byte
// Map is valid for KindMap.
Map Record
// List is valid for KindList, holding its elements in their original order.
List []Value
// I64 is valid for KindInt64.
I64 int64
// F64 is valid for KindFloat64 (including SemanticFloat32-tagged values).
F64 float64
// Kind selects which of the other fields is live.
Kind Kind
// Semantic refines how Kind's value is to be interpreted.
Semantic Semantic
// Bool is valid for KindBool.
Bool bool
}
Value is a tagged union rather than an any, so a scalar field costs no allocation or heap escape. Kind selects which field is live; Semantic refines how to interpret it.
Str aliases the decoder's own buffer and is only valid until the owning Record is reset or refilled; copy it to retain it past that.
func Bytes ¶
Bytes returns a Value referencing b without copying it; see Value's doc for the lifetime this implies.
func Float32 ¶
Float32 returns a Value holding v widened to a float64 and tagged SemanticFloat32, so an encoder renders it at 32-bit precision rather than at the float64 precision the widening implies.
func Int64 ¶
Int64 returns a Value holding v, tagged with the given semantic (SemanticNone when v is a plain signed integer).
func List ¶
List returns a Value holding elements as an ordered nested sequence, referencing them without copying — the same lifetime Bytes implies, so a decoder is free to hand over scratch it refills for the next row.
func Map ¶
Map returns a Value holding entries as a nested key/value group, referencing them without copying — the same lifetime Bytes implies, so a decoder is free to hand over scratch it refills for the next row.
func (Value) AppendText ¶
AppendText appends v's canonical text to dst, rendered exactly as the converters write it, and reports whether v had a text to append.
ok is false for a null, a zero-length string, and a KindMap or KindList (which have no scalar text): such a value is "empty" for hashing, comparison and profiling purposes. -0 renders as 0 so one number is one text, and a date or timestamp renders as ISO-8601 by its Semantic.
Source Files