Documentation
¶
Index ¶
- Constants
- func AppendLenBytes(buf, b []byte) []byte
- func AppendLenString(buf []byte, s string) []byte
- func BitpackDecode(data []byte) ([]uint8, error)
- func BitpackEncode(indices []uint8, bitsPerValue int) []byte
- func BitsNeeded(n int) int
- func Compress(src []byte) []byte
- func DecodeCeiling(count, perValueBytes int) int
- func Decompress(src []byte) ([]byte, error)
- func DecompressBounded(src []byte, maxDecoded int) ([]byte, error)
- func DeltaDecode(data []byte, count int) ([]int64, error)
- func DeltaEncode(values []int64) []byte
- func DictLookup(d *DictEncoded) []string
- func DictMarshal(d *DictEncoded) []byte
- func PutUvarint(dst []byte, v uint64) []byte
- func PutVarint(dst []byte, v int64) []byte
- func SparseBool(values []*bool) []byte
- func SparseBytes(values [][]byte) []byte
- func SparseInt64(values []*int64) []byte
- func SparseStrings(values []string) []byte
- func UnsparseBool(data []byte, count int) ([]*bool, error)
- func UnsparseBytes(data []byte, count int) ([][]byte, error)
- func UnsparseInt64(data []byte, count int) ([]*int64, error)
- func UnsparseStrings(data []byte, count int, f LenFormat) ([]string, error)
- func ZigZagDecode(v uint64) int64
- func ZigZagEncode(v int64) uint64
- func ZstdBlockDecodeInt64(data []byte, count int) ([]int64, error)
- func ZstdBlockDecodeStrings(data []byte, count int, f LenFormat) ([]string, error)
- func ZstdBlockEncodeInt64(values []int64) []byte
- func ZstdBlockEncodeStrings(values []string) []byte
- type DictEncoded
- type LenFormat
Constants ¶
const ( // MaxColumnValueBytes is the largest single value that can reach a chunk: // the WAL truncates every field to 1 MiB before it is sealed. (The larger // MaxValueBytes in strlen.go bounds the *framing*, not what the pipeline // actually admits.) MaxColumnValueBytes = 1 << 20 // MaxVarintBytes is the widest encoding of one varint/zigzag-varint value. MaxVarintBytes = 10 // MaxLenFramingBytes covers the length prefix stored alongside a // variable-length value. MaxLenFramingBytes = 8 )
Per-value ceilings used to derive a column's decoded-size bound.
const MaxBlockBytes = 2 << 30 // 2 GiB
MaxBlockBytes is the largest uncompressed block the codec supports. It is the absolute decoder ceiling *and* the encoder's contract: anything larger than this seals fine but can never be read back, so Compress reports a violation loudly instead of producing a write-only block. A chunk holds up to 50k rows, so this leaves ~40KB per row for the biggest single-block column (message).
It is deliberately *not* the ceiling any single column decode runs under — see DecodeCeiling. Bounding only the compressed bytes against the file size let a corrupt or crafted 2 MB column expand toward 2 GiB inside a search goroutine; every column decode now runs under a ceiling derived from its own value count.
const MaxDictEntries = 65535
MaxDictEntries is the largest dictionary DictMarshal can represent: the serialized entry count is a uint16, so 65535 entries (indices 0..65534) is the ceiling. Beyond it the count would wrap and the whole column would decode to garbage, so DictEncode stops growing the dictionary instead.
const ( // MaxValueBytes bounds a single length-prefixed value. It exists so a corrupt // length can never be used for an unbounded allocation; decoders additionally // check the length against the bytes actually available. MaxValueBytes = 1 << 30 // 1 GiB )
Variables ¶
This section is empty.
Functions ¶
func AppendLenBytes ¶ added in v0.22.0
AppendLenBytes appends b with a LenExtended length prefix.
func AppendLenString ¶ added in v0.22.0
AppendLenString appends s with a LenExtended length prefix.
func BitpackDecode ¶
BitpackDecode unpacks N-bit packed indices.
func BitpackEncode ¶
BitpackEncode packs uint8 indices into N bits per value. bitsPerValue must be 1-8. Returns: [1 byte bitsPerValue] [4 bytes count] [packed bits]
func BitsNeeded ¶
BitsNeeded returns the minimum number of bits to represent n distinct values.
func DecodeCeiling ¶ added in v0.22.0
DecodeCeiling returns the largest decoded size a column of count values can legitimately reach, given a per-value maximum. Fixed-width columns (timestamps, durations, dictionary indices) end up with a tight bound; only the genuinely variable-length columns keep a large one, and even those are capped at MaxBlockBytes.
func Decompress ¶
Decompress decompresses zstd data under the absolute MaxBlockBytes ceiling. Callers that know how many values the block holds should use DecompressBounded with DecodeCeiling instead.
func DecompressBounded ¶ added in v0.22.0
DecompressBounded decompresses zstd data, refusing to expand it past maxDecoded bytes. The declared frame content size is checked first (cheap, and it stops the allocation before it happens); frames that don't declare one are still hard-bounded by the decoder's own memory ceiling.
func DeltaDecode ¶
DeltaDecode decodes base + zigzag-varint deltas + zstd back to int64 values.
func DeltaEncode ¶
DeltaEncode encodes a slice of int64 values as base + zigzag-varint deltas + zstd. Returns: [8 bytes base] [zstd(zigzag-varint deltas)]
func DictLookup ¶
func DictLookup(d *DictEncoded) []string
DictLookup resolves dictionary indices back to strings.
func DictMarshal ¶
func DictMarshal(d *DictEncoded) []byte
DictMarshal serializes a DictEncoded to bytes + zstd. Format: [2 bytes dict_len] [dict entries: 2-byte len + string]... [zstd(uint16 indices)]
func PutUvarint ¶
PutUvarint appends a uvarint-encoded uint64 to dst and returns the extended slice.
func SparseBool ¶
SparseBool encodes a slice of *bool where nil = null. Non-null values stored as a second bitmap.
func SparseBytes ¶
SparseBytes encodes a slice of []byte where nil = null. Used for body column (already compressed blobs).
func SparseInt64 ¶
SparseInt64 encodes a slice of *int64 where nil = null. Uses varint encoding for non-null values.
func SparseStrings ¶
SparseStrings encodes a slice of strings where empty string = null. Format: [4 bytes non_null_count] [bitmap bytes] [zstd(length-prefixed non-null strings)]
func UnsparseBool ¶
UnsparseBool decodes sparse bool column.
func UnsparseBytes ¶
UnsparseBytes decodes sparse byte slice column.
func UnsparseInt64 ¶
UnsparseInt64 decodes sparse int64 column.
func UnsparseStrings ¶
UnsparseStrings decodes sparse string column using the given length framing (LenUint16 for chunk v1 files, LenExtended for v2+).
func ZigZagDecode ¶
ZigZagDecode converts a zigzag-encoded uint64 back to int64.
func ZigZagEncode ¶
ZigZagEncode converts a signed int64 to an unsigned uint64 using zigzag encoding. This makes small negative numbers small unsigned numbers (good for varint).
func ZstdBlockDecodeInt64 ¶
ZstdBlockDecodeInt64 decodes zstd + raw LE int64 values.
func ZstdBlockDecodeStrings ¶
ZstdBlockDecodeStrings decodes zstd + length-prefixed strings using the given length framing (LenUint16 for chunk v1 files, LenExtended for v2+).
func ZstdBlockEncodeInt64 ¶
ZstdBlockEncodeInt64 encodes a slice of int64 as raw LE bytes + zstd. Used for ts column (not delta-encoded, may not be monotonic).
func ZstdBlockEncodeStrings ¶
ZstdBlockEncodeStrings encodes a slice of strings as length-prefixed + zstd.
Types ¶
type DictEncoded ¶
type DictEncoded struct {
Dict []string // ordered dictionary: index → string
Indices []uint16 // per-entry dictionary index
}
DictEncoded holds the result of dictionary encoding.
func DictEncode ¶
func DictEncode(values []string) *DictEncoded
DictEncode builds a dictionary from string values and returns indices. Null/empty strings are stored as index 0 with dict[0] = "".
func DictEncodeLimit ¶ added in v0.22.0
func DictEncodeLimit(values []string, limit int) *DictEncoded
DictEncodeLimit is DictEncode with an explicit cap on dictionary size. Once the cap is reached, further distinct values map to index 0 (the empty string) and the dictionary stops growing — lossy, but bounded and never misframed. Callers whose index width is narrower than uint16 (bitpacked columns) pass a smaller limit.
func DictUnmarshal ¶
func DictUnmarshal(data []byte, count int, f LenFormat) (*DictEncoded, error)
DictUnmarshal deserializes DictEncoded from bytes using the given length framing (LenUint16 for chunk v1 files, LenExtended for v2+).
type LenFormat ¶ added in v0.22.0
type LenFormat uint8
LenFormat selects the on-disk framing of a length-prefixed value.
const ( // LenUint16 is the legacy framing: a bare uint16 length. Values longer than // maxUint16Len bytes cannot be represented and are rejected at encode time. LenUint16 LenFormat = 1 // LenExtended is the current framing: a uint16 length where extendedLenMarker // escapes to a following uint32 length. LenExtended LenFormat = 2 )
func (LenFormat) ReadLen ¶ added in v0.22.0
ReadLen reads a length prefix in the given framing and returns the length and the offset of the first value byte.
func (LenFormat) ReadString ¶ added in v0.22.0
ReadString reads a length-prefixed value and copies it into a string.