encoding

package
v0.22.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 14, 2026 License: MIT Imports: 6 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// MaxColumnValueBytes is the largest single value that can reach a chunk:
	// the WAL truncates every field to 1 MiB before it is sealed. (The larger
	// MaxValueBytes in strlen.go bounds the *framing*, not what the pipeline
	// actually admits.)
	MaxColumnValueBytes = 1 << 20
	// MaxVarintBytes is the widest encoding of one varint/zigzag-varint value.
	MaxVarintBytes = 10
	// MaxLenFramingBytes covers the length prefix stored alongside a
	// variable-length value.
	MaxLenFramingBytes = 8
)

Per-value ceilings used to derive a column's decoded-size bound.

View Source
const MaxBlockBytes = 2 << 30 // 2 GiB

MaxBlockBytes is the largest uncompressed block the codec supports. It is the absolute decoder ceiling *and* the encoder's contract: anything larger than this seals fine but can never be read back, so Compress reports a violation loudly instead of producing a write-only block. A chunk holds up to 50k rows, so this leaves ~40KB per row for the biggest single-block column (message).

It is deliberately *not* the ceiling any single column decode runs under — see DecodeCeiling. Bounding only the compressed bytes against the file size let a corrupt or crafted 2 MB column expand toward 2 GiB inside a search goroutine; every column decode now runs under a ceiling derived from its own value count.

View Source
const MaxDictEntries = 65535

MaxDictEntries is the largest dictionary DictMarshal can represent: the serialized entry count is a uint16, so 65535 entries (indices 0..65534) is the ceiling. Beyond it the count would wrap and the whole column would decode to garbage, so DictEncode stops growing the dictionary instead.

View Source
const (

	// MaxValueBytes bounds a single length-prefixed value. It exists so a corrupt
	// length can never be used for an unbounded allocation; decoders additionally
	// check the length against the bytes actually available.
	MaxValueBytes = 1 << 30 // 1 GiB
)

Variables

This section is empty.

Functions

func AppendLenBytes added in v0.22.0

func AppendLenBytes(buf, b []byte) []byte

AppendLenBytes appends b with a LenExtended length prefix.

func AppendLenString added in v0.22.0

func AppendLenString(buf []byte, s string) []byte

AppendLenString appends s with a LenExtended length prefix.

func BitpackDecode

func BitpackDecode(data []byte) ([]uint8, error)

BitpackDecode unpacks N-bit packed indices.

func BitpackEncode

func BitpackEncode(indices []uint8, bitsPerValue int) []byte

BitpackEncode packs uint8 indices into N bits per value. bitsPerValue must be 1-8. Returns: [1 byte bitsPerValue] [4 bytes count] [packed bits]

func BitsNeeded

func BitsNeeded(n int) int

BitsNeeded returns the minimum number of bits to represent n distinct values.

func Compress

func Compress(src []byte) []byte

Compress compresses data using zstd.

func DecodeCeiling added in v0.22.0

func DecodeCeiling(count, perValueBytes int) int

DecodeCeiling returns the largest decoded size a column of count values can legitimately reach, given a per-value maximum. Fixed-width columns (timestamps, durations, dictionary indices) end up with a tight bound; only the genuinely variable-length columns keep a large one, and even those are capped at MaxBlockBytes.

func Decompress

func Decompress(src []byte) ([]byte, error)

Decompress decompresses zstd data under the absolute MaxBlockBytes ceiling. Callers that know how many values the block holds should use DecompressBounded with DecodeCeiling instead.

func DecompressBounded added in v0.22.0

func DecompressBounded(src []byte, maxDecoded int) ([]byte, error)

DecompressBounded decompresses zstd data, refusing to expand it past maxDecoded bytes. The declared frame content size is checked first (cheap, and it stops the allocation before it happens); frames that don't declare one are still hard-bounded by the decoder's own memory ceiling.

func DeltaDecode

func DeltaDecode(data []byte, count int) ([]int64, error)

DeltaDecode decodes base + zigzag-varint deltas + zstd back to int64 values.

func DeltaEncode

func DeltaEncode(values []int64) []byte

DeltaEncode encodes a slice of int64 values as base + zigzag-varint deltas + zstd. Returns: [8 bytes base] [zstd(zigzag-varint deltas)]

func DictLookup

func DictLookup(d *DictEncoded) []string

DictLookup resolves dictionary indices back to strings.

func DictMarshal

func DictMarshal(d *DictEncoded) []byte

DictMarshal serializes a DictEncoded to bytes + zstd. Format: [2 bytes dict_len] [dict entries: 2-byte len + string]... [zstd(uint16 indices)]

func PutUvarint

func PutUvarint(dst []byte, v uint64) []byte

PutUvarint appends a uvarint-encoded uint64 to dst and returns the extended slice.

func PutVarint

func PutVarint(dst []byte, v int64) []byte

PutVarint appends a varint-encoded int64 to dst and returns the extended slice.

func SparseBool

func SparseBool(values []*bool) []byte

SparseBool encodes a slice of *bool where nil = null. Non-null values stored as a second bitmap.

func SparseBytes

func SparseBytes(values [][]byte) []byte

SparseBytes encodes a slice of []byte where nil = null. Used for body column (already compressed blobs).

func SparseInt64

func SparseInt64(values []*int64) []byte

SparseInt64 encodes a slice of *int64 where nil = null. Uses varint encoding for non-null values.

func SparseStrings

func SparseStrings(values []string) []byte

SparseStrings encodes a slice of strings where empty string = null. Format: [4 bytes non_null_count] [bitmap bytes] [zstd(length-prefixed non-null strings)]

func UnsparseBool

func UnsparseBool(data []byte, count int) ([]*bool, error)

UnsparseBool decodes sparse bool column.

func UnsparseBytes

func UnsparseBytes(data []byte, count int) ([][]byte, error)

UnsparseBytes decodes sparse byte slice column.

func UnsparseInt64

func UnsparseInt64(data []byte, count int) ([]*int64, error)

UnsparseInt64 decodes sparse int64 column.

func UnsparseStrings

func UnsparseStrings(data []byte, count int, f LenFormat) ([]string, error)

UnsparseStrings decodes sparse string column using the given length framing (LenUint16 for chunk v1 files, LenExtended for v2+).

func ZigZagDecode

func ZigZagDecode(v uint64) int64

ZigZagDecode converts a zigzag-encoded uint64 back to int64.

func ZigZagEncode

func ZigZagEncode(v int64) uint64

ZigZagEncode converts a signed int64 to an unsigned uint64 using zigzag encoding. This makes small negative numbers small unsigned numbers (good for varint).

func ZstdBlockDecodeInt64

func ZstdBlockDecodeInt64(data []byte, count int) ([]int64, error)

ZstdBlockDecodeInt64 decodes zstd + raw LE int64 values.

func ZstdBlockDecodeStrings

func ZstdBlockDecodeStrings(data []byte, count int, f LenFormat) ([]string, error)

ZstdBlockDecodeStrings decodes zstd + length-prefixed strings using the given length framing (LenUint16 for chunk v1 files, LenExtended for v2+).

func ZstdBlockEncodeInt64

func ZstdBlockEncodeInt64(values []int64) []byte

ZstdBlockEncodeInt64 encodes a slice of int64 as raw LE bytes + zstd. Used for ts column (not delta-encoded, may not be monotonic).

func ZstdBlockEncodeStrings

func ZstdBlockEncodeStrings(values []string) []byte

ZstdBlockEncodeStrings encodes a slice of strings as length-prefixed + zstd.

Types

type DictEncoded

type DictEncoded struct {
	Dict    []string // ordered dictionary: index → string
	Indices []uint16 // per-entry dictionary index
}

DictEncoded holds the result of dictionary encoding.

func DictEncode

func DictEncode(values []string) *DictEncoded

DictEncode builds a dictionary from string values and returns indices. Null/empty strings are stored as index 0 with dict[0] = "".

func DictEncodeLimit added in v0.22.0

func DictEncodeLimit(values []string, limit int) *DictEncoded

DictEncodeLimit is DictEncode with an explicit cap on dictionary size. Once the cap is reached, further distinct values map to index 0 (the empty string) and the dictionary stops growing — lossy, but bounded and never misframed. Callers whose index width is narrower than uint16 (bitpacked columns) pass a smaller limit.

func DictUnmarshal

func DictUnmarshal(data []byte, count int, f LenFormat) (*DictEncoded, error)

DictUnmarshal deserializes DictEncoded from bytes using the given length framing (LenUint16 for chunk v1 files, LenExtended for v2+).

type LenFormat added in v0.22.0

type LenFormat uint8

LenFormat selects the on-disk framing of a length-prefixed value.

const (
	// LenUint16 is the legacy framing: a bare uint16 length. Values longer than
	// maxUint16Len bytes cannot be represented and are rejected at encode time.
	LenUint16 LenFormat = 1
	// LenExtended is the current framing: a uint16 length where extendedLenMarker
	// escapes to a following uint32 length.
	LenExtended LenFormat = 2
)

func (LenFormat) ReadLen added in v0.22.0

func (f LenFormat) ReadLen(data []byte, off int) (int, int, error)

ReadLen reads a length prefix in the given framing and returns the length and the offset of the first value byte.

func (LenFormat) ReadString added in v0.22.0

func (f LenFormat) ReadString(data []byte, off int) (string, int, error)

ReadString reads a length-prefixed value and copies it into a string.

func (LenFormat) ReadValue added in v0.22.0

func (f LenFormat) ReadValue(data []byte, off int) ([]byte, int, error)

ReadValue reads a length-prefixed value and returns it as a sub-slice of data (no copy) plus the offset just past it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL