encoding

package
v1.0.0-alpha.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 16, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package encoding handles the binary .pulse file format: reading, writing, and schema management.

Index

Constants

View Source 
const (
	// FormatVersionV1 is the original layout: header + field descriptors
	// + inline dictionaries + fixed-stride records, with nothing between
	// the last descriptor and the first record.
	FormatVersionV1 byte = 0x01
	// FormatVersionV2 appends a length-prefixed schema extension block
	// after the field descriptors (see [WriteSchema]). Readers of a v2
	// file always know exactly where the record region starts.
	FormatVersionV2 byte = 0x02
)

Format versions a .pulse header may declare in byte 8.

The version a writer emits is a FUNCTION OF SCHEMA CONTENT (Schema.RequiredFormatVersion), never a global flag: a schema that uses no 0x02 feature is written as 0x01, byte-identical to every file written before 0x02 existed. Old cohorts stay readable forever — the reader accepts every version in the supported set and parses the schema block by the version it read (ReadSchema).

A binary that predates a version refuses it at ReadHeader with a coded ENCODING_INVALID naming the version, before any schema byte is interpreted, so a newer file never misparses silently in an older binary.

View Source 
const (
	MinVectorDim = 2
	MaxVectorDim = 1024
)

MinVectorDim and MaxVectorDim bound a vector field's dim, inclusive.

View Source 
const CanonicalDateTimeLayout = "2006-01-02T15:04:05Z"

CanonicalDateTimeLayout is the single text form FormatDateTime emits for a `datetime` field: an ISO-8601 / RFC 3339 calendar-date + wall-clock-time literal with a literal `Z` suffix, second resolution, no fractional part. It is the canonical round-trip form — a value exported through it re-parses under DateTimeFormats[0] to the exact same on-wire uint64, which is what makes the text-oriented adapters (CSV / TSV / NDJSON / JSON) lossless for `datetime` without any format-native datetime column type.

View Source 
const FormatVersion byte = FormatVersionV1

FormatVersion is the BASELINE .pulse format version: the version WriteHeader emits and the one every schema that uses no 0x02 feature is written at. It is not "the newest version this binary understands" — that is FormatVersionV2.

View Source 
const GroupIndexWidth = 4

GroupIndexWidth is the on-wire width of an indexed group's per-row index. The descriptor carries the width explicitly; this binary accepts only this value for GroupKindIndexed and 0 for GroupKindConstant.

View Source 
const HeaderSize = 9

HeaderSize is the total byte size of the file header (magic + version).

View Source 
const MaxDecimalPrecision = 38

MaxDecimalPrecision is the upper bound on decimal128 precision.

View Source 
const MaxDescriptionBytes = 1000

MaxDescriptionBytes is the maximum allowed byte length for a field description.

View Source 
const MinDecimalScale = 4

MinDecimalScale is the floor on division result scale (matches Arrow).

View Source 
const SecondsPerDay = temporal.SecondsPerDay

SecondsPerDay is the conversion factor between the two temporal on-wire representations Pulse persists: a `datetime` cell holds whole epoch SECONDS, a `date` cell holds whole epoch DAYS. Every conversion between them goes through DateTimeToDay rather than open-coding this constant at the call site.

It forwards to internal/temporal, the single home of epoch-day math.

View Source 
const SetMaskBits = SetMaskWords * 64

SetMaskBits is the hard ceiling on set membership across every set rung: SetMaskWords × 64 = 256. The ceiling is what lets SetMask be a fixed array instead of a slice — see the SetMask doc comment.

View Source 
const SetMaskWords = 4

SetMaskWords is the number of uint64 words backing a SetMask.

Variables

View Source 
var DateFormats = []string{
	"2006-01-02",
	"01/02/2006",
	"2006-01-02T15:04:05Z",
	"2006-01-02T15:04:05",
	"2006/01/02",
	"02-Jan-2006",
}

DateFormats enumerates the date literal layouts ParseDate accepts, in priority order — the first layout that parses a given literal wins. This is the authoritative list for turning a date literal into the persisted on-wire epoch-day representation; internal/io/import.go's column-type inference (internal/io/infer.go's own, independently-maintained dateFormats) is a lighter, best-effort "does this column look date-shaped" probe and intentionally stays separate from this conversion authority.

View Source 
var DateTimeFormats = []string{
	"2006-01-02T15:04:05Z07:00",
	"2006-01-02T15:04:05",
	"2006-01-02T15:04Z07:00",
	"2006-01-02T15:04",
	"2006-01-02 15:04:05",
	"2006-01-02 15:04",
}

DateTimeFormats enumerates the datetime literal layouts ParseDateTime accepts, in priority order — the first layout that parses a given literal wins. Same shape and same first-match-wins contract as DateFormats, but a deliberately stricter list:

  • The slash forms DateFormats tolerates ("01/02/2006") are NOT accepted. `03/04/2024` is 4 March in most of the world and 3 April in the US; a datetime column silently swapping day and month is exactly the quiet degradation this type must not do. An ambiguous literal fails loudly with ENCODING_INVALID instead.
  • A literal carrying no time-of-day is NOT accepted. That is a `date`, and internal/io/infer.go's column classification depends on this rejection to keep a date-only column typed FieldTypeDate rather than widening it to FieldTypeDateTime.

Go's time.Parse accepts a fractional-second field immediately after the seconds field even when the layout omits one, so "2024-03-04T10:11:12.5Z" parses under the first layout with no dedicated entry. Sub-second precision is then truncated by the seconds-resolution on-wire representation (see ParseDateTime).

View Source 
var MagicBytes = [8]byte{'P', 'U', 'L', 'S', 'E', 0x00, 0x00, 0x00}

MagicBytes identifies a .pulse file. 8 bytes: "PULSE\x00\x00\x00"

View Source 
var MaxGroupEntries uint64 = math.MaxUint32 + 1

MaxGroupEntries is the entry-count ceiling of an indexed group: the u32 index space. A variable (not a const) only so tests can shrink it.

View Source 
var ZstdMagic = [4]byte{0x28, 0xB5, 0x2F, 0xFD}

ZstdMagic is the zstd frame magic (0xFD2FB528 little-endian, RFC 8878) that opens a `.pulse.zst` transfer artifact. It is NOT a cohort layout and there is no third magic-byte variant: Pulse never reads a compressed cohort. ReadHeader recognises these four bytes only so it can refuse with PULSE_COHORT_COMPRESSED ("decompress first") instead of a generic ENCODING_INVALID.

Functions

func BitmapIsNull added in v0.9.0 

func BitmapIsNull(bitmap []byte, i int) bool

BitmapIsNull reports whether field at index i is marked null in bitmap. Bit ordering: field index i → byte i/8, bit i%8 (LSB-first within each byte). 1 = null, 0 = present.

func BitmapSetNull added in v0.9.0 

func BitmapSetNull(bitmap []byte, i int)

BitmapSetNull marks the field at index i as null in bitmap. The caller must have pre-allocated bitmap to at least ceil((i+1)/8) bytes.

func CompressedCohortError added in v0.39.0 

func CompressedCohortError() *errors.CodedError

CompressedCohortError is the ONE coded refusal for a cohort path that holds a zstd transfer artifact rather than a `.pulse` cohort.

func DateDays 

func DateDays(raw uint64) int32

DateDays interprets a raw `date` word — as returned by ReadFieldValue or ParseDate — as its signed epoch-day count. The on-wire date word is two's-complement int32 days since 1970-01-01, so 1969-12-31 is -1 and 1900-01-01 is -25567. Zero-extending the word instead (uint64(raw)) turns every pre-1970 date into a day near 4.29e9; always read a date through this helper.

func DateTimeSeconds 

func DateTimeSeconds(raw uint64) int64

DateTimeSeconds interprets a raw `datetime` word — as returned by ReadFieldValue or ParseDateTime — as its signed epoch-second count. The on-wire datetime word is two's-complement int64 seconds since 1970-01-01T00:00:00Z, so 1969-12-31T23:59:59Z is -1.

func DateTimeToDay added in v0.30.0 

func DateTimeToDay(sec int64) int64

DateTimeToDay truncates an on-wire `datetime` value (epoch seconds, naive UTC) to the epoch-day integer a `date` value carries. This is THE day-truncation semantic the date-family operators (GROUP_DATE, GROUP_DATE_RANGES, FILTER_DATE_RANGES) apply when their Field names a `datetime` column: the time-of-day component is discarded so the row buckets by the calendar day that contains its instant.

Truncation, never rounding — 2024-03-04T23:59:59Z is 2024-03-04, not 2024-03-05. And truncation toward the PAST, not toward zero: Go's integer division truncates toward zero, which would map every pre-1970 instant inside the first day before the epoch onto day 0 (1970-01-01) instead of day -1 (1969-12-31). The floor correction keeps the calendar day honest across the sign boundary.

The int64 parameter is deliberate: ParseDateTime widens a negative second count into uint64 two's-complement for storage, so callers converting a stored value re-narrow with int64(raw) first — exactly the reinterpretation FormatDateTime performs.

The implementation lives in internal/temporal; this is a forwarder.

func EncodeDecimal128 added in v0.2.0 

func EncodeDecimal128(d Decimal128) [16]byte

EncodeDecimal128 serializes a Decimal128 as 16 bytes of two's-complement little-endian integer.

func FormatDateTime added in v0.30.0 

func FormatDateTime(raw uint64) string

FormatDateTime renders an on-wire `datetime` value (epoch seconds as uint64) back into CanonicalDateTimeLayout. It is the exact inverse of ParseDateTime: FormatDateTime(ParseDateTime(s)) re-parses to the same uint64 for every s that ParseDateTime accepts.

The int64 conversion is deliberate and reverses the uint64 widening ParseDateTime performs, so pre-1970 instants render as their real calendar time rather than as a year-292-billion overflow.

func IsSupportedFormatVersion added in v0.39.0 

func IsSupportedFormatVersion(v byte) bool

IsSupportedFormatVersion reports whether v is a .pulse format version this binary can read.

func IsZstdMagic added in v0.39.0 

func IsZstdMagic(b []byte) bool

IsZstdMagic reports whether b begins with ZstdMagic.

func ParseDate added in v0.27.0 

func ParseDate(raw string) (uint32, error)

ParseDate parses raw against DateFormats (first match wins) and returns the on-wire representation the .pulse `date` field type carries: whole days since the Unix epoch as a two's-complement int32 word (a pre-1970 day is negative; read it back with DateDays) — exactly the conversion internal/io/import.go's row importer performs when converting a date column cell (`temporal.TimeToDay`). This function is the single source of truth both the importer (internal/io/import.go's convertValue) and the point-lookup key resolver (processing.ResolveLookupKeyBytes) call, so a date literal parsed at import time and the same literal parsed at lookup time always resolve to the identical on-wire uint32 — no separate epoch-math implementation to drift out of sync.

Returns an ENCODING_INVALID coded error when raw matches none of DateFormats.

func ParseDateTime added in v0.30.0 

func ParseDateTime(raw string) (uint64, error)

ParseDateTime parses raw against DateTimeFormats (first match wins) and returns the on-wire representation the .pulse `datetime` field type carries: whole SECONDS since the Unix epoch, as uint64.

Note the contrast with ParseDate, which returns epoch DAYS as a uint32. The two are not interchangeable and must never be swapped: a `date` cell holds days, a `datetime` cell holds seconds.

Timezone policy is naive UTC — the format stores an instant and no zone. A layout with no offset is read as UTC (time.Parse's own default), and a layout carrying an explicit offset is normalised to the same instant, with the offset itself discarded. Round-tripping "2024-03-04T10:11:12+02:00" therefore yields "2024-03-04T08:11:12Z": the instant is preserved exactly, the local-wall-clock presentation is not.

Precision is seconds; any fractional-second component in raw is truncated toward the epoch by time.Time.Unix.

Pre-1970 instants are representable: the negative int64 second count is reinterpreted as uint64 two's-complement, which FormatDateTime reverses exactly, so the round trip is lossless across the whole int64 second range.

Returns an ENCODING_INVALID coded error when raw matches none of DateTimeFormats.

func PutSetMask added in v0.38.0 

func PutSetMask(dst []byte, ft FieldType, m SetMask) error

PutSetMask writes m into dst as the on-wire payload of a wide set field. Exactly ft.ByteSize() bytes are written, starting at dst[0], in the little-endian word order documented on SetMask; dst may be longer (a record buffer sliced at the field offset) and nothing past the field width is touched. A dst shorter than the field width, a non-wide field type, or a mask wider than the rung's capacity is an error and leaves dst untouched.

func PutVector 

func PutVector(dst []byte, ft FieldType, dim int, v []float64) error

PutVector writes v into dst as the on-wire payload of a vector field of type ft and the given dim: exactly VectorElemSize × dim bytes from dst[0], little-endian, nothing past them touched. A nil v writes the NULL placeholder — a zeroed full stride (the null itself rides the record's bitmap bit). A non-vector ft is ENCODING_TYPE_MISMATCH; a dim out of range, len(v) other than dim, a dst shorter than the stride, or a finite value a vec_f32 element cannot hold (it would silently become ±Inf) is ENCODING_INVALID, and dst is left untouched. NaN and ±Inf are stored verbatim.

func ReadBit 

func ReadBit(r io.Reader, bitPos uint) (bool, error)

ReadBit reads a single bit from the byte at the current position in r. bitPos is 0-7 within that byte.

func ReadBitmap added in v0.9.0 

func ReadBitmap(r io.Reader, byteLen int) ([]byte, error)

ReadBitmap reads a per-record null bitmap of the given byte length and returns the raw bytes. byteLen MUST be Schema.BitmapByteSize(); callers invoke this only when the schema has at least one nullable field.

func ReadDescription 

func ReadDescription(r io.Reader) (string, error)

ReadDescription reads a field description from r. Format: u16 length + utf8 bytes.

func ReadFieldValue 

func ReadFieldValue(r io.Reader, ft FieldType) (uint64, error)

ReadFieldValue reads a single field value from r, returning raw bits as uint64. For bit-packed types (U4, PackedBool), use ReadBit/ReadNibble instead. decimal128, the wide set rungs (set_u128, set_u256) and the vector types (vec_f32, vec_f64 — use ReadVector) exceed the uint64 return and are rejected with ENCODING_TYPE_MISMATCH.

func ReadHeader 

func ReadHeader(r io.Reader) (byte, error)

ReadHeader reads and validates the .pulse file header from r and returns the format version it declares. The version must be handed to ReadSchema: the schema block's layout depends on it, and record data begins immediately after the schema block with no terminator, so parsing a v2 schema as v1 would misplace every record.

A version outside SupportedFormatVersions is ENCODING_INVALID with the offending byte under details["version"]. A zstd transfer artifact (ZstdMagic) is PULSE_COHORT_COMPRESSED (CompressedCohortError): error classification only — nothing is ever decompressed here.

func ReadNibble 

func ReadNibble(r io.Reader, high bool) (uint8, error)

ReadNibble reads a 4-bit value from a byte. If high is true, it reads bits 4-7; otherwise bits 0-3.

func ReadVector 

func ReadVector(r io.Reader, ft FieldType, dim int, dst []float64) ([]float64, error)

ReadVector reads a vector field's payload from the record stream into dst[:0] (see VectorFromBytes). Null state rides the per-record null bitmap, so a null vector reads as its zeroed placeholder; the caller consults the bitmap.

func SupportedFormatVersions added in v0.39.0 

func SupportedFormatVersions() []byte

SupportedFormatVersions returns the accepted version set, ascending.

func VectorFromBytes 

func VectorFromBytes(src []byte, ft FieldType, dim int, dst []float64) ([]float64, error)

VectorFromBytes decodes the on-wire payload of a vector field of type ft and the given dim from the front of src (anything past the stride is ignored), appending the dim elements to dst[:0] and returning it — pass a reused dst to decode without allocating. vec_f32 elements widen to float64 exactly. Errors are those of PutVector, plus a src shorter than the stride.

func WriteBit 

func WriteBit(w io.Writer, bitPos uint, val bool) error

WriteBit writes a byte with a single bit set or cleared at bitPos.

func WriteBitmap added in v0.9.0 

func WriteBitmap(w io.Writer, bitmap []byte) error

WriteBitmap writes a per-record null bitmap. The slice MUST be exactly Schema.BitmapByteSize() bytes; this helper is intentionally thin to keep the writer responsible for allocating and populating the buffer.

func WriteDecimal128 added in v0.2.0 

func WriteDecimal128(w io.Writer, d Decimal128) error

WriteDecimal128 writes a Decimal128 to the .pulse record stream.

func WriteDescription 

func WriteDescription(w io.Writer, desc string) error

WriteDescription writes a field description to w. Format: u16 length + utf8 bytes. Returns PULSE_IMPORT_DESCRIPTION_TOO_LONG if the UTF-8 byte length exceeds MaxDescriptionBytes.

func WriteFieldValue 

func WriteFieldValue(w io.Writer, ft FieldType, val uint64) error

WriteFieldValue writes a single field value (as raw bits in uint64) to w. For bit-packed types (U4, PackedBool), use WriteBit/WriteNibble instead. For decimal128 (16-byte type), use the dedicated WriteDecimal128 helper; this function will reject those types with ENCODING_TYPE_MISMATCH. The wide set rungs (set_u128, set_u256) are rejected the same way: their bitmask is wider than the uint64 this API speaks — and so are the vector types (vec_f32, vec_f64): use WriteVector.

func WriteHeader 

func WriteHeader(w io.Writer) error

WriteHeader writes a .pulse file header declaring the baseline FormatVersion (0x01). It is the raw-byte primitive for an ungrouped cohort; Pulse's own writers derive the version from the schema's content (a grouped schema needs 0x02) so the header cannot disagree with the schema block that follows.

func WriteNibble 

func WriteNibble(w io.Writer, high bool, val uint8) error

WriteNibble writes a 4-bit value into a byte. If high is true, it writes to bits 4-7; otherwise bits 0-3. The other nibble is zero.

func WriteSchema 

func WriteSchema(w io.Writer, s *Schema) error

WriteSchema serializes s's schema block to w at the baseline 0x01 layout, to follow a WriteHeader. A schema whose content requires a newer version is refused rather than written in a layout its header would contradict; Pulse's own writers emit a grouped schema with a matching 0x02 header.

The 0x01 schema block: 

u16 field_count
per field:
  u8 type
  u8 nullable (0 or 1)
  u16 name_length + utf8 name
  u32 byte_offset
  u8 bit_position
  u16 csv_column_idx
  u16 description_length + utf8 description
  (if decimal128) u8 precision + u8 scale
  (if categorical) dictionary block

A 0x02 schema block is the 0x01 block followed by the schema extension block (see group_wire.go for the payload): 

u64 extension_length
extension_length bytes of extension payload (tagged sections)

The length prefix is what makes the record region's start derivable without understanding the payload.

func WriteSetMask added in v0.38.0 

func WriteSetMask(w io.Writer, ft FieldType, m SetMask) error

WriteSetMask writes a wide set field's payload to the record stream. It is the streaming mirror of PutSetMask and the API the uint64 WriteFieldValue points at when it refuses set_u128 / set_u256.

func WriteVector 

func WriteVector(w io.Writer, ft FieldType, dim int, v []float64) error

WriteVector writes a vector field's payload to the record stream — the streaming mirror of PutVector, and the API the uint64 WriteFieldValue points at when it refuses a vector type. The dim is explicit (it is schema metadata, Schema.VectorDim); a nil v writes the null placeholder.

Types

type Decimal128 added in v0.2.0 

type Decimal128 struct {
	// contains filtered or unexported fields
}

Decimal128 is a fixed-point decimal value with up to 38 digits of precision. Logically it is a signed 128-bit two's-complement integer mantissa, paired with a per-field scale that places the implicit decimal point. Internally it is carried as *big.Int (always copied, never aliased) bounded to the decimal128 representable range [-(10^38 - 1), 10^38 - 1].

func DecodeDecimal128 added in v0.2.0 

func DecodeDecimal128(buf [16]byte) Decimal128

DecodeDecimal128 deserializes 16 bytes of two's-complement little-endian integer into a Decimal128. Null state is carried by the per-record bitmap (see encoding.Schema.HasBitmap), so this function does not flag null values.

func NewDecimal128FromBigInt added in v0.2.0 

func NewDecimal128FromBigInt(m *big.Int) (Decimal128, error)

NewDecimal128FromBigInt builds a Decimal128 from a big.Int mantissa. Returns PULSE_DECIMAL_OVERFLOW if |m| >= 10^38.

func NewDecimal128FromInt added in v0.2.0 

func NewDecimal128FromInt(i int64) Decimal128

NewDecimal128FromInt builds a Decimal128 with the given integer mantissa.

func ParseDecimal128 added in v0.2.0 

func ParseDecimal128(s string) (Decimal128, uint8, error)

ParseDecimal128 parses a strict decimal string into a (Decimal128, scale) pair. The scale is inferred from the input — exactly the count of digits after the decimal point. Accepts an optional single leading sign char. Rejects whitespace, thousand separators, scientific notation, currency symbols.

func ReadDecimal128 added in v0.2.0 

func ReadDecimal128(r io.Reader) (Decimal128, error)

ReadDecimal128 reads 16 bytes of decimal128 from r and decodes them. Null state is carried by the per-record bitmap, not by the payload bytes, so this function has no null channel.

func ZeroDecimal128 added in v0.2.0 

func ZeroDecimal128() Decimal128

ZeroDecimal128 returns a Decimal128 with mantissa 0.

func (Decimal128) Add added in v0.2.0 

func (d Decimal128) Add(o Decimal128) (Decimal128, error)

Add returns d + o, treating both at the same scale. The result has the same scale; the caller is responsible for SQL:2016 precision propagation at the schema level.

func (Decimal128) Cmp added in v0.2.0 

func (d Decimal128) Cmp(o Decimal128) int

Cmp compares two Decimal128 values with the same scale. Comparing values at different scales requires the caller to align scales first.

func (Decimal128) Div added in v0.2.0 

func (d Decimal128) Div(o Decimal128, s1, s2, resultScale uint8) (Decimal128, error)

Div divides d by o using banker's rounding to produce a result at scale `resultScale` given the operands at scales s1 and s2. Returns PULSE_DECIMAL_DIVIDE_BY_ZERO when o is zero.

func (Decimal128) FitsPrecision added in v0.2.0 

func (d Decimal128) FitsPrecision(precision uint8) bool

FitsPrecision reports whether the mantissa fits in `precision` digits.

func (Decimal128) Float64 added in v0.2.0 

func (d Decimal128) Float64(scale uint8) float64

Float64 returns the decimal as a float64 at the given scale. Lossy for values that exceed float64 precision; callers preserving precision should keep using Decimal128 directly.

func (Decimal128) Mantissa added in v0.2.0 

func (d Decimal128) Mantissa() *big.Int

Mantissa returns a copy of the unscaled integer mantissa.

func (Decimal128) Mul added in v0.2.0 

func (d Decimal128) Mul(o Decimal128) (Decimal128, error)

Mul returns d * o. The product mantissa is the product of mantissas; callers are responsible for tracking scale propagation (s1 + s2).

func (Decimal128) Rescale added in v0.2.0 

func (d Decimal128) Rescale(sourceScale, targetScale uint8) (Decimal128, error)

Rescale converts d at sourceScale to targetScale using banker's rounding. Returns PULSE_DECIMAL_OVERFLOW if the rescaled mantissa exceeds 10^38-1.

func (Decimal128) Sign added in v0.2.0 

func (d Decimal128) Sign() int

Sign returns -1, 0, or +1 depending on the mantissa sign.

func (Decimal128) Sqrt added in v0.2.0 

func (d Decimal128) Sqrt(sourceScale, targetScale uint8) (Decimal128, error)

Sqrt returns floor-banker-rounded sqrt(d) at the target scale, given the source value at sourceScale. The result is computed entirely in decimal arithmetic via big.Int.Sqrt and a half-to-even rounding step on the residual. Returns PULSE_DECIMAL_OVERFLOW if intermediate state cannot fit in the integer representation; returns PROCESSING_RUNTIME for negative inputs (sqrt of a negative decimal is undefined).

func (Decimal128) String added in v0.2.0 

func (d Decimal128) String(scale uint8) string

String renders the decimal at the given scale. Trailing zeros are preserved; the leading sign is included only for negative values.

func (Decimal128) Sub added in v0.2.0 

func (d Decimal128) Sub(o Decimal128) (Decimal128, error)

Sub returns d - o.

type Dictionary 

type Dictionary struct {
	// contains filtered or unexported fields
}

Dictionary maps string values to sequential uint32 IDs. It is used for categorical field types to encode string categories as compact integer IDs in the binary record format.

func NewDictionary 

func NewDictionary() *Dictionary

NewDictionary creates an empty dictionary.

func (*Dictionary) Add 

func (d *Dictionary) Add(s string) (uint32, error)

Add inserts a string into the dictionary, returning its ID. If the string already exists, the existing ID is returned. There is no capacity limit with Add; use AddWithLimit to enforce one.

func (*Dictionary) AddWithLimit 

func (d *Dictionary) AddWithLimit(s string, maxEntries uint32) (uint32, error)

AddWithLimit inserts a string, enforcing a maximum entry count. Returns PULSE_IMPORT_CATEGORICAL_OVERFLOW if the dictionary is full.

func (*Dictionary) Count 

func (d *Dictionary) Count() int

Count returns the number of entries in the dictionary.

func (*Dictionary) IDFor 

func (d *Dictionary) IDFor(s string) (uint32, bool)

IDFor looks up the ID for a string. Returns the ID and true if found, or 0 and false otherwise.

func (*Dictionary) ReadFrom 

func (d *Dictionary) ReadFrom(r io.Reader) (int64, error)

ReadFrom deserializes a dictionary from r, replacing current contents. Format: u32 count + (u16 strlen + utf8 bytes) x count

Performance: a single byte buffer is grown to hold all string payloads across the dictionary, instead of allocating one []byte per entry. Each resolved string is copied once via the standard string([]byte) conversion (no unsafe), so we drop one allocation per entry while preserving the safety contract that callers can mutate the underlying buffer afterwards without affecting the stored strings.

func (*Dictionary) Resolve 

func (d *Dictionary) Resolve(id uint32) string

Resolve returns the string for a given ID. Returns "" if the ID is out of range.

func (*Dictionary) Values 

func (d *Dictionary) Values() []string

Values returns a copy of all dictionary values in insertion order.

func (*Dictionary) WriteTo 

func (d *Dictionary) WriteTo(w io.Writer) (int64, error)

WriteTo serializes the dictionary to w. Format: u32 count + (u16 strlen + utf8 bytes) x count

type Field 

type Field struct {
	Name         string      `json:"name"`
	Type         FieldType   `json:"type"`
	Nullable     bool        `json:"nullable"` // true ⇒ field participates in per-record null bitmap
	ByteOffset   int         `json:"byte_offset"`
	BitPosition  int         `json:"bit_position"`
	CsvColumnIdx int         `json:"csv_column_idx"`
	Description  string      `json:"description"` // empty = synthesized at inspect time
	Dictionary   *Dictionary `json:"-"`           // non-nil only for categorical types

	// Precision is the decimal128 precision (1-38). Meaningful only when
	// Type is FieldTypeDecimal128.
	Precision uint8 `json:"precision,omitempty"`
	// Scale is the decimal128 scale (0-Precision). Meaningful only when
	// Type is FieldTypeDecimal128.
	Scale uint8 `json:"scale,omitempty"`
}

Field describes a single column in a .pulse schema.

The json tags are the report shape import / export / convert / shard `--json` output and a library json.Marshal render: snake_case keys, the type as its string name ("f64", never the type byte — see FieldType.MarshalText) and, in place of the dictionary's values, its entry count as dictionary_size (Field.MarshalJSON). The `.pulse` schema block is binary and never passes through this shape.

func (Field) MarshalJSON 

func (f Field) MarshalJSON() ([]byte, error)

MarshalJSON renders the field's report shape: its tagged keys plus dictionary_size — the entry count of a dictionary-bearing field, omitted for every other field. The values themselves stay off the wire (a categorical_u32 dictionary can run to millions of entries); `pulse inspect` lists them, truncated by default.

type FieldType 

type FieldType byte

FieldType identifies the data type stored in a schema field. 

const (
	FieldTypeU8             FieldType = iota // 0
	FieldTypeU16                             // 1
	FieldTypeU32                             // 2
	FieldTypeU64                             // 3
	FieldTypeF32                             // 4
	FieldTypeF64                             // 5
	FieldTypeU4                              // 6  (4-bit, bit-packed)
	FieldTypeDate                            // 7
	FieldTypePackedBool                      // 8  (1-bit, bit-packed)
	FieldTypeCategoricalU8                   // 9
	FieldTypeCategoricalU16                  // 10
	FieldTypeCategoricalU32                  // 11
	FieldTypeDecimal128                      // 12
	FieldTypeSetU8                           // 13 (bitmask over shared dict, ≤8 members)
	FieldTypeSetU16                          // 14 (≤16 members)
	FieldTypeSetU32                          // 15 (≤32 members)
	FieldTypeSetU64                          // 16 (≤64 members)
	FieldTypeDateTime                        // 17 (epoch seconds, 8-byte u64)
	FieldTypeSetU128                         // 18 (16-byte bitmask, ≤128 members)
	FieldTypeSetU256                         // 19 (32-byte bitmask, ≤256 members)
	FieldTypeVecF32                          // 20 (fixed-dim float32 vector; width = 4 × dim)
	FieldTypeVecF64                          // 21 (fixed-dim float64 vector; width = 8 × dim)

)

All field types supported by the .pulse format. Nullability is orthogonal to type — any field can be marked nullable via encoding.Field.Nullable and the per-record null bitmap carries the actual null state.

func ParseFieldType added in v0.15.0 

func ParseFieldType(name string) (FieldType, bool)

ParseFieldType inverts FieldType.String, returning the typed value and ok=true when name matches a registered FieldType. Caller paths (managed-imports sidecar parsing, MCP schema validation, ad-hoc admin tooling) use this to round-trip an externally supplied type name through the codec layer without relying on stringly typed switches in every place.

func (FieldType) ByteSize 

func (ft FieldType) ByteSize() int

ByteSize returns the number of bytes this field type occupies in a record. Bit-packed types (U4, PackedBool) share bytes with adjacent fields and return 0 here; their on-wire stride is handled by Schema.RecordByteSize. The vector types (vec_f32, vec_f64) also return 0: their width is VectorElemSize × the field's dim, a property of the field within its schema (Schema.FieldWidth), never of the type byte alone.

func (FieldType) HasDictionary added in v0.15.0 

func (ft FieldType) HasDictionary() bool

HasDictionary reports whether the field type carries an inline dictionary block in the schema (categorical_* or set_*).

func (FieldType) IsBitPacked added in v0.9.0 

func (ft FieldType) IsBitPacked() bool

IsBitPacked reports whether the field type shares its on-wire byte with adjacent fields via bit packing (U4, PackedBool). Used by stride math and the schema layout pass to advance the bit cursor instead of the byte cursor.

func (FieldType) IsCategorical 

func (ft FieldType) IsCategorical() bool

IsCategorical reports whether the field type is one of the categorical types.

func (FieldType) IsDecimal added in v0.2.0 

func (ft FieldType) IsDecimal() bool

IsDecimal reports whether the field type is decimal128.

func (FieldType) IsKnown added in v0.2.0 

func (ft FieldType) IsKnown() bool

IsKnown reports whether the byte value corresponds to a registered type. Used by the schema reader to reject files written by a future binary version that introduces unknown type bytes.

func (FieldType) IsNumeric added in v0.7.2 

func (ft FieldType) IsNumeric() bool

IsNumeric reports whether the field type is a strict scalar number: the unsigned-integer family (u8/u16/u32/u64), the float family (f32/f64), and decimal128. Date, datetime, and bit-packed integer encodings are excluded — see IsNumericForAnalytics for the analytics-layer predicate.

func (FieldType) IsNumericForAnalytics added in v0.7.2 

func (ft FieldType) IsNumericForAnalytics() bool

IsNumericForAnalytics reports whether the field type carries a meaningful scalar value for numeric analytics (regression, sum/avg/stddev/min/max/ variance aggregators). The set is broader than IsNumeric: bit-packed integer encodings (u4, packed_bool) plus date (epoch days) and datetime (epoch seconds) are included because their stored representation is an ordinal / cardinal number the analytics layer can average, sum, or regress without an explicit ATTR_FORMULA cast.

Null exclusion is the reader's responsibility: the per-record null bitmap marks any field index as null at decode time so the downstream Record.NumericValue contract (returns ok=false on null) keeps the aggregation denominator honest.

func (FieldType) IsSet added in v0.15.0 

func (ft FieldType) IsSet() bool

IsSet reports whether the field type is a bitmask-over-dictionary set (multi-select) type. Set fields share the categorical dictionary block shape but the on-wire payload is a fixed-width bitmask whose bit i corresponds to dictionary entry i.

The narrow rungs (set_u8..set_u64) fit a uint64 and ride the scalar value API; the wide rungs (set_u128, set_u256) are 16- and 32-byte little-endian bitmasks that do NOT fit a uint64 and are rejected by Read/WriteFieldValue — see IsWideSet.

func (FieldType) IsVector 

func (ft FieldType) IsVector() bool

IsVector reports whether the field type is a fixed-dim numeric vector (vec_f32, vec_f64). A vector is not a scalar: IsNumeric and IsNumericForAnalytics are false for it, the uint64 value API refuses it (use ReadVector / WriteVector), and its dim and metadata live in the schema's VECTORS section (Schema.Vectors).

func (FieldType) IsWideSet added in v0.38.0 

func (ft FieldType) IsWideSet() bool

IsWideSet reports whether the field type is a set rung whose bitmask is too wide for the uint64 scalar value API (set_u128, set_u256). Callers that hand a set value through Read/WriteFieldValue must branch on this: the uint64 path would silently truncate the mask to its low 64 bits, dropping selections without any error.

func (FieldType) MarshalText 

func (ft FieldType) MarshalText() ([]byte, error)

MarshalText renders the type by its canonical name ("f64", "categorical_u8", ...) so every JSON surface — import / export / convert / shard reports, a library json.Marshal of a Schema — shows the name, never the schema-block type byte. The binary schema block is untouched: it writes the byte through WriteSchema.

func (FieldType) MaxCategoricalEntries 

func (ft FieldType) MaxCategoricalEntries() uint32

MaxCategoricalEntries returns the maximum dictionary size for a categorical type. Returns 0 for non-categorical types.

func (FieldType) MaxDictEntries added in v0.15.0 

func (ft FieldType) MaxDictEntries() uint32

MaxDictEntries returns the capacity of the inline dictionary block for dictionary-bearing types. Categoricals return MaxCategoricalEntries; sets return MaxSetEntries (= bitmask width). Non-dictionary types return 0.

func (FieldType) MaxSetEntries added in v0.15.0 

func (ft FieldType) MaxSetEntries() uint32

MaxSetEntries returns the maximum dictionary size (= bitmask capacity) for a set type. Returns 0 for non-set types.

func (FieldType) String 

func (ft FieldType) String() string

String returns a human-readable name for the field type.

func (*FieldType) UnmarshalText 

func (ft *FieldType) UnmarshalText(b []byte) error

UnmarshalText inverts MarshalText through ParseFieldType; an unknown name is an error, never a silent u8.

func (FieldType) VectorElemSize 

func (ft FieldType) VectorElemSize() int

VectorElemSize returns the on-wire bytes of ONE element of a vector type: 4 for vec_f32, 8 for vec_f64, 0 for every non-vector type. A vector field's width is VectorElemSize × its dim (Schema.FieldWidth).

type Group added in v0.39.0 

type Group struct {
	Kind GroupKind `json:"kind"`
	// Members are in strictly ascending Field order.
	Members []GroupMember `json:"members"`
	// Entries is the flat dictionary: EntryCount × EntryWidth bytes.
	// Entry e occupies Entries[e*w : (e+1)*w]. Never strings: a
	// categorical member's bytes are its own dictionary ID, and each
	// member field keeps its own Field.Dictionary unchanged.
	Entries []byte `json:"-"`
}

Group is one parent-group descriptor plus its dictionary. Its JSON report shape is {kind, members}: the raw entry bytes never ride JSON (an import report's GroupReport carries the entry count and widths).

type GroupKind added in v0.39.0 

type GroupKind uint8

GroupKind is the per-group kind byte of the group descriptor. 

const (
	// GroupKindIndexed is a dictionary of N entries addressed by a u32
	// index carried in every physical row.
	GroupKindIndexed GroupKind = 0
	// GroupKindConstant is a dictionary of exactly ONE entry and no
	// per-row index (index width 0): the members hold the same value —
	// or the same null — on every row. It is how a global-constant field
	// is elided from the row.
	GroupKindConstant GroupKind = 1
)

func (GroupKind) MarshalText 

func (k GroupKind) MarshalText() ([]byte, error)

MarshalText renders the kind by name ("indexed" / "constant") so a schema's JSON report never shows the descriptor byte.

func (GroupKind) String added in v0.39.0 

func (k GroupKind) String() string

String names the kind as reports print it: "indexed" or "constant".

func (*GroupKind) UnmarshalText 

func (k *GroupKind) UnmarshalText(b []byte) error

UnmarshalText inverts MarshalText; any other name is an error.

type GroupMember added in v0.39.0 

type GroupMember struct {
	// Field is the member's position in Schema.Fields.
	Field int `json:"field"`
	// Key marks the member as part of the group's declared key: the
	// members whose values determine the tuple. Informational for
	// decode (every member is read from the entry either way); an
	// encoder validates that non-key members are functionally dependent
	// on the key, and a future precompute may rely on key members
	// being a bijection with the entry index.
	Key bool `json:"key"`
}

GroupMember names one member of a group by its LOGICAL field index.

type RecordLocator added in v0.27.0 

type RecordLocator struct {
	// Schema is the parsed schema this locator's geometry is derived
	// from.
	Schema *Schema

	// RecordRegionStart is the absolute byte offset of the first record,
	// i.e. the number of bytes the header + schema prefix consumed.
	RecordRegionStart int64

	// Stride is Schema.RecordByteSize() — the fixed on-wire byte width
	// of one record (including the trailing null bitmap, if any).
	Stride int64

	// TotalRecords is the number of complete records the payload holds,
	// derived from the payload's total byte length. Zero for an empty
	// cohort or a cohort shorter than one full record stride.
	TotalRecords uint64
}

RecordLocator captures the fixed geometry a single-file .pulse payload needs for O(1) index-addressed record reads: the byte offset where the record region begins (immediately after the header + schema prefix), the fixed per-record stride, and the total record count for bounds checking. Construct one per opened cohort via NewRecordLocator and reuse it across many point lookups — the offset math mirrors internal/service/parallel_decode.go's per-worker segment slicing (recordRegionStart + i*stride), just applied to a single record instead of a worker's contiguous range.

func NewRecordLocator added in v0.27.0 

func NewRecordLocator(r *bytes.Reader, schema *Schema) (*RecordLocator, error)

NewRecordLocator derives record-region geometry for a single-file .pulse payload from r, an io.ReadSeeker over the raw payload bytes — most commonly a *bytes.Reader wrapping a mmap'd byte slice, the same path the streaming iterator and the parallel decode context use. r's cursor must be at absolute position 0 (the very first header byte) when this is called; NewRecordLocator reads (and discards, beyond measuring their length) the header and schema, leaving r positioned at the first record byte as a side effect. Callers that need r re-wound for further use should re-seek to loc.RecordRegionStart or construct a fresh reader over the same bytes.

schema is the already-parsed schema for this cohort (e.g. from pulse.Open) — NewRecordLocator does not re-validate its field contents against the freshly-read schema bytes; it reads the schema block purely to measure how many bytes the header+schema prefix consumes. A mismatched schema silently produces a locator with wrong geometry, so callers must pass the schema that was actually parsed from this same payload.

A *bytes.Reader is required (rather than a generic io.ReadSeeker) so this function can read Size() — the reader's fixed, read-independent total length — to derive TotalRecords without a second file stat.

func (*RecordLocator) Offset added in v0.27.0 

func (loc *RecordLocator) Offset(i uint64) int64

Offset returns the absolute byte offset of record i within the payload this locator was built from. It does not bounds-check — callers must check i < TotalRecords before seeking.

type Schema 

type Schema struct {
	// Fields is the LOGICAL schema: every field, in original order. A
	// field index anywhere in the API is a position in this slice.
	Fields []Field `json:"fields"`
	// Groups are the parent-group descriptors (format 0x02, see
	// group.go). Empty for every 0x01 cohort. When non-empty, member
	// fields are stored in the groups' dictionaries rather than in the
	// row, and RecordByteSize / BitmapByteSize / HasBitmap describe the
	// PHYSICAL (reduced) row; Logical() is the ungrouped view.
	Groups []Group `json:"groups,omitempty"`
	// Vectors describes every vec_f32 / vec_f64 field — its dim and
	// optional kind, scale range and element labels (format 0x02, the
	// VECTORS section; see vector.go). Exactly one entry per vec field;
	// empty for a vector-free cohort.
	Vectors []VectorField `json:"vectors,omitempty"`
}

Schema holds all field descriptors for a .pulse file.

func ReadSchema 

func ReadSchema(r io.Reader, v byte) (*Schema, error)

ReadSchema deserializes a schema block written at format version v — the version ReadHeader returned for the same stream. On return r is positioned at the first record byte. An unsupported v is ENCODING_INVALID; unknown field-type bytes fail loud here, at parse time, for every version.

func (*Schema) BitmapByteSize added in v0.9.0 

func (s *Schema) BitmapByteSize() int

BitmapByteSize returns the number of bytes the null bitmap occupies per record, or 0 when no field is nullable. With groups it is the NARROWED bitmap: ceil(row_field_count/8), bit j = the j-th row field (fields in no group, logical order).

func (*Schema) Categorical 

func (s *Schema) Categorical(name string) (*Dictionary, bool)

Categorical returns the dictionary for a named categorical field. Returns nil, false if the field is not found or is not categorical.

func (*Schema) Field 

func (s *Schema) Field(name string) *Field

Field returns a pointer to the named field, or nil if not found.

func (*Schema) FieldGroup added in v0.39.0 

func (s *Schema) FieldGroup(fi int) (g, k int, ok bool)

FieldGroup reports whether logical field fi is a group member and, if so, which group and member slot. It is the entry point for code that wants to work per dictionary entry instead of per row (filter precompute): the member's value in entry e is GroupMemberBytes(g, k, GroupEntry(g, e)).

func (*Schema) FieldWidth 

func (s *Schema) FieldWidth(i int) int

FieldWidth returns the on-wire bytes LOGICAL field i occupies — in an ungrouped row and, for a parent-group member, in its group entry: one whole byte for a bit-packed type (u4, packed_bool — FieldType.ByteSize reports 0 for those), VectorElemSize × dim for a vector type (the dim from Schema.Vectors; 0 while the field has no entry, which ValidateVectors refuses on every write and read), FieldType.ByteSize for every other known type, and 0 for an unknown type byte or an i outside Fields.

It is the ONE field-aware width. Every stride, offset and skip — this package's RecordByteSize and group geometry, the decode plans, the readers, import, widen, synth — is a sum of FieldWidth, never of a type-only width: a field's width is a property of the field within its schema, not of its type byte alone, and a site that sizes a field from its type alone silently misaligns every field after it the day the two differ. TestNoTypeOnlyWidthInStridePaths keeps it that way.

func (*Schema) GroupEntry added in v0.39.0 

func (s *Schema) GroupEntry(g, e int) []byte

GroupEntry returns entry e of group g (a subslice of Entries — do not mutate it).

func (*Schema) GroupEntryCount added in v0.39.0 

func (s *Schema) GroupEntryCount(g int) int

GroupEntryCount returns the number of entries in group g's dictionary.

func (*Schema) GroupEntryWidth added in v0.39.0 

func (s *Schema) GroupEntryWidth(g int) int

GroupEntryWidth returns the byte width of one entry of group g: the members' on-wire widths plus the member null bitmap when any member is nullable.

func (*Schema) GroupIndexOffset added in v0.39.0 

func (s *Schema) GroupIndexOffset(g int) int

GroupIndexOffset returns the physical byte offset of group g's u32 index within a physical row, or -1 for a constant group (no per-row index).

func (*Schema) GroupMemberBytes added in v0.39.0 

func (s *Schema) GroupMemberBytes(g, k int, entry []byte) []byte

GroupMemberBytes returns member k's on-wire bytes within entry — the same bytes the member occupies in a logical row.

func (*Schema) GroupMemberIsNull added in v0.39.0 

func (s *Schema) GroupMemberIsNull(g, k int, entry []byte) bool

GroupMemberIsNull reports whether member k is null in entry.

func (*Schema) GroupMemberRowBytes added in v0.39.0 

func (s *Schema) GroupMemberRowBytes(g int) int

GroupMemberRowBytes returns the on-wire bytes group g's members occupy in a LOGICAL row, null bits excluded: what each row stops carrying when the group is formed, before an indexed group's GroupIndexWidth is added back. With GroupEntryCount and GroupEntryWidth it is everything a viability check needs.

func (*Schema) HasBitmap added in v0.9.0 

func (s *Schema) HasBitmap() bool

HasBitmap reports whether the record carries a per-record null bitmap: whether any field stored IN THE ROW is nullable. Without groups that is any field; with groups, member fields' null bits ride their dictionary entry, so only the row fields count — a cohort whose every nullable field is a group member has no per-row bitmap.

func (*Schema) HasGroups added in v0.39.0 

func (s *Schema) HasGroups() bool

HasGroups reports whether the schema declares any parent group — i.e. whether its physical row differs from its logical row.

func (*Schema) HasVectors 

func (s *Schema) HasVectors() bool

HasVectors reports whether the schema declares vector-field metadata (a VECTORS section). Every vec field must be covered, so a valid schema has vectors exactly when it has a vec field.

func (*Schema) Logical added in v0.39.0 

func (s *Schema) Logical() *Schema

Logical returns the schema's ungrouped view: the same Fields and Vectors (both slices SHARED, so *Field pointers, field indices and vector dims agree), with no group declared. Its RecordByteSize / BitmapByteSize / HasBitmap describe the logical row. For a schema without groups it returns s.

func (*Schema) RecordByteSize added in v0.8.4 

func (s *Schema) RecordByteSize() int

RecordByteSize returns the on-wire stride of one record under this schema: the sum of FieldWidth over the fields (a bit-packed field still consumes one whole byte). When the schema declares at least one nullable field, the trailing null bitmap of ceil(field_count/8) bytes is appended to every record and included in the stride.

With groups it is the PHYSICAL stride: one GroupIndexWidth index per indexed group, plus the row fields, plus the narrowed bitmap. It is still fixed and a pure function of the schema.

func (*Schema) RecordCountForPayload added in v0.34.0 

func (s *Schema) RecordCountForPayload(payloadBytes int64) (count, trailing int64, ok bool)

RecordCountForPayload derives how many whole records occupy a payload of payloadBytes under this schema, together with the leftover bytes that do not complete a record.

It is the ONE derivation behind every "how many records does this cohort hold" answer for a single-file cohort — internal/service.CountRecords (the header-fast facade path and the parallel-decode eligibility gate) and internal/descriptor.Inspect (the header-only reporting path) both call it. They lived as two independent floor divisions until they were lifted here; identical arithmetic written twice is one edit away from two different record counts over the same bytes, and nothing on either wire says which arm produced the number a caller is holding.

count is always the FLOOR. trailing is payloadBytes % stride and is non-zero only for a truncated tail — a cohort whose last record was half-written. The two arms deliberately differ in what they do with it, and only in that: Inspect has an envelope and raises an ENCODING_INVALID warning naming the leftover bytes, while CountRecords returns (uint64, error) with no warning channel and stays silent, because it is a counter feeding an eligibility gate rather than a diagnostic and a truncated tail must not make a cohort that still processes fail to count. The count itself is identical on both arms by construction.

ok is false when the stride is not positive (a field-less schema has no records to count) or payloadBytes is negative (the file is shorter than its own header + schema); the caller reports no count rather than a fabricated zero.

func (*Schema) RequiredFormatVersion added in v0.39.0 

func (s *Schema) RequiredFormatVersion() byte

RequiredFormatVersion returns the .pulse format version this schema's content needs on the wire. It is the single place a writer's version is chosen — never a global flag — so a schema that uses no 0x02 feature is always written at 0x01, byte-identical to a pre-0x02 file.

A schema requires 0x02 exactly when it declares a parent group or a vector field (whose dim rides the VECTORS section).

func (*Schema) SetField added in v0.15.0 

func (s *Schema) SetField(name string) (*Dictionary, bool)

SetField returns the dictionary for a named set-typed field. Returns nil, false if the field is not found or is not a set type.

func (*Schema) ValidateGroups added in v0.39.0 

func (s *Schema) ValidateGroups() error

ValidateGroups checks the group descriptors against the schema's fields. A schema with no groups is always valid. Rules:

  • every group has a known kind and at least one member;
  • members are in range, strictly ascending, and no field belongs to two groups (groups are independent, never nested);
  • Entries is a whole number of entries; an indexed group has at most MaxGroupEntries entries, a constant group exactly one;
  • the physical stride is positive (a cohort whose every field is constant-elided would have a zero stride and an underivable record count).

func (*Schema) ValidateVectors 

func (s *Schema) ValidateVectors() error

ValidateVectors checks Schema.Vectors against Schema.Fields: each entry names an in-range vec field, no field twice, dim in MinVectorDim..MaxVectorDim, a known kind, a finite scale range with ScaleMin < ScaleMax when HasScale, nil or exactly Dim labels (each valid UTF-8, at most 65,535 bytes), and every vec field covered. Every fault is ENCODING_INVALID with details naming the field. Both the writer and the reader run it, so neither can produce or accept a vec field whose width is unknown.

func (*Schema) Vector 

func (s *Schema) Vector(i int) (VectorField, bool)

Vector returns the metadata of logical field i when it is a described vector field. The Labels slice is shared with the schema — do not mutate it.

func (*Schema) VectorDim 

func (s *Schema) VectorDim(i int) int

VectorDim returns logical field i's dim, or 0 when it is not a described vector field.

type SetMask added in v0.38.0 

type SetMask struct {
	// contains filtered or unexported fields
}

SetMask is the single in-memory representation of a set field's membership bitmask, for every rung from set_u8 through set_u256. Bit i corresponds to dictionary entry i; an all-zero mask is a valid "no selection" and is distinct from null (set nulls ride the per-record null bitmap only, with no in-band sentinel).

Word order — normative wire contract

words[0] holds bits 0–63 (the LOW 64 bits), words[1] bits 64–127, words[2] bits 128–191, words[3] bits 192–255. On the wire the words are laid down in that same order — LITTLE-ENDIAN word order — with each word itself written little-endian. The payload of a wide set field is therefore exactly the little-endian byte image of the 128- or 256-bit integer, and the low 8 bytes of a set_u128 are byte-identical to the set_u64 payload holding the same selections. Reverse either the word order or the intra-word byte order and every cohort written since the wide rungs landed decodes as a different, plausible-looking selection.

Value semantics

SetMask is a fixed [SetMaskWords]uint64 VALUE. It never allocates, is copied by plain assignment, and can be retained across records without a defensive copy — no method aliases the receiver's backing array and every mutator returns a new value rather than writing through a pointer. That is a deliberate consequence of the 256-bit ceiling and is why the wide-set surface carries no reuse contract.

func ReadSetMask added in v0.38.0 

func ReadSetMask(r io.Reader, ft FieldType) (SetMask, error)

ReadSetMask reads a wide set field's payload from the record stream. Null state rides the per-record null bitmap, not the payload bytes, so this function has no null channel — an all-zero mask is an empty selection, not a null.

func SetMaskFromBytes added in v0.38.0 

func SetMaskFromBytes(ft FieldType, src []byte) (SetMask, error)

SetMaskFromBytes decodes the on-wire payload of a wide set field. It reads exactly ft.ByteSize() bytes from the front of src and ignores anything past that, so a caller may hand it a record buffer sliced at the field offset without a neighbouring column bleeding in. Words the rung does not carry stay zero.

func SetMaskFromUint64 added in v0.38.0 

func SetMaskFromUint64(low uint64) SetMask

SetMaskFromUint64 lifts a narrow-rung mask (set_u8..set_u64, all of which store their bitmask in a uint64) into the shared wide type. The value lands in words[0]; the higher words stay zero.

func SetMaskFromWords added in v0.38.0 

func SetMaskFromWords(words [SetMaskWords]uint64) SetMask

SetMaskFromWords builds a SetMask from raw words in the documented order: words[0] is the low 64 bits.

func (SetMask) Bits added in v0.38.0 

func (m SetMask) Bits() iter.Seq[int]

Bits returns an iterator over the set bit indices in ASCENDING order, for `for bit := range mask.Bits()`. Stopping early stops the walk.

func (SetMask) Equal added in v0.38.0 

func (m SetMask) Equal(o SetMask) bool

Equal reports whether two masks carry exactly the same bits.

func (SetMask) FitsFieldType added in v0.38.0 

func (m SetMask) FitsFieldType(ft FieldType) bool

FitsFieldType reports whether every set bit is within the capacity of the given set field type (ft.MaxSetEntries()). A non-set type has capacity 0, so only an empty mask fits one.

func (SetMask) Has added in v0.38.0 

func (m SetMask) Has(i int) bool

Has reports whether bit i is set. Out-of-range indices (negative, or at or above SetMaskBits) report false rather than panicking — a corrupt or mid-remap payload must not take the decoder down.

func (SetMask) HighestBit added in v0.38.0 

func (m SetMask) HighestBit() int

HighestBit returns the index of the highest set bit, or -1 when the mask is empty. Used by the write path to check a mask against the declared rung's capacity.

func (SetMask) Intersect added in v0.38.0 

func (m SetMask) Intersect(o SetMask) SetMask

Intersect returns the bitwise AND of m and o. Neither operand is modified.

func (SetMask) IsEmpty added in v0.38.0 

func (m SetMask) IsEmpty() bool

IsEmpty reports whether no bit is set. An empty mask is a valid "no selection", NOT a null.

func (SetMask) Labels added in v0.38.0 

func (m SetMask) Labels(dict *Dictionary) []string

Labels resolves the set bits to dictionary labels in ascending bit order. The walk is bounded by dict.Count(), NOT by the mask width or the field type's capacity: a bit beyond the dictionary is a corrupt or mid-remap payload and is skipped, never resolved and never fatal. A nil or empty dictionary yields an empty, non-nil slice.

func (SetMask) NextBit added in v0.38.0 

func (m SetMask) NextBit(from int) (int, bool)

NextBit returns the lowest set bit at index >= from, and ok=false when there is none. It is the allocation-free primitive Bits rides on, and the form to use on a hot path: 

for bit, ok := m.NextBit(0); ok; bit, ok = m.NextBit(bit + 1) { … }

func (SetMask) PopCount added in v0.38.0 

func (m SetMask) PopCount() int

PopCount returns the number of set bits — the size of the selection.

func (SetMask) Uint64 added in v0.38.0 

func (m SetMask) Uint64() (uint64, bool)

Uint64 returns the low 64 bits and reports whether the whole mask fits in them. ok=false means the caller is about to truncate: the returned value is the low word, but bits at or above 64 exist and would be lost. Every narrowing hand-off to the uint64 value API must check ok.

func (SetMask) Union added in v0.38.0 

func (m SetMask) Union(o SetMask) SetMask

Union returns the bitwise OR of m and o. Neither operand is modified.

func (SetMask) WithBit added in v0.38.0 

func (m SetMask) WithBit(i int) SetMask

WithBit returns a copy of m with bit i set. An out-of-range i is a no-op. The receiver is untouched.

func (SetMask) WithoutBit added in v0.38.0 

func (m SetMask) WithoutBit(i int) SetMask

WithoutBit returns a copy of m with bit i cleared. An out-of-range i is a no-op. The receiver is untouched.

func (SetMask) Words added in v0.38.0 

func (m SetMask) Words() [SetMaskWords]uint64

Words returns a copy of the backing words. Mutating the result cannot reach the receiver — arrays are values in Go, and that is the point.

type SetWidthHeadroom added in v0.38.0 

type SetWidthHeadroom struct {
	Field string `json:"field"`
	// Type is the rung the field currently declares.
	Type string `json:"type"`
	// Used is the number of dictionary entries in the canonical schema.
	Used int `json:"used"`
	// Capacity is the rung's bitmask width.
	Capacity int `json:"capacity"`
	// Headroom is Capacity - Used: how many more distinct members the
	// archive can absorb before the next `shard add` widens it.
	Headroom int `json:"headroom"`
	// NextType is the rung a widen would promote to, or "" when the
	// field is already at the widest rung — at which point Headroom is
	// the hard remainder and an overflow is fatal, not widenable.
	NextType string `json:"next_type,omitempty"`
}

SetWidthHeadroom reports how much of one set field's bitmask capacity the canonical dictionary has consumed, and which rung it would be widened to next.

It exists so an impending archive-wide widen is FORESEEABLE. The widen itself is correct but expensive (every record of every shard is re-laid-out), and without this a caller learns the field was one entry from its ceiling only by paying for the rewrite.

type VectorField 

type VectorField struct {
	// FieldIndex is the field's LOGICAL position in Schema.Fields.
	FieldIndex int `json:"field_index"`
	// Dim is the element count, MinVectorDim..MaxVectorDim.
	Dim int `json:"dim"`
	// Kind is what the values mean; VectorKindNone when undeclared.
	Kind VectorKind `json:"kind,omitempty"`
	// ScaleMin and ScaleMax are the declared value range, meaningful
	// only when HasScale; ScaleMin < ScaleMax, both finite.
	ScaleMin float64 `json:"scale_min"`
	ScaleMax float64 `json:"scale_max"`
	HasScale bool    `json:"has_scale"`
	// Labels names each element, in element order: nil (no labels) or
	// exactly Dim entries.
	Labels []string `json:"labels,omitempty"`
}

VectorField is the schema metadata of one vec_f32 / vec_f64 field: its dim and its optional kind, scale range and element labels.

type VectorKind 

type VectorKind uint8

VectorKind is what a vector's values mean — the same vocabulary as a request's virtual-vector kind (measure, scale, composition, binary). VectorKindNone is "no kind declared"; it is the zero value and is never written as a kind byte (the has_kind flag is clear instead). 

const (
	VectorKindNone        VectorKind = 0
	VectorKindMeasure     VectorKind = 1
	VectorKindScale       VectorKind = 2
	VectorKindComposition VectorKind = 3
	VectorKindBinary      VectorKind = 4
)

The vector kinds. The numeric value IS the on-wire kind byte, so the order is frozen; append only.

func ParseVectorKind 

func ParseVectorKind(name string) (VectorKind, bool)

ParseVectorKind inverts VectorKind.String for the four declared kinds; ok is false for anything else (the empty string included).

func (VectorKind) IsKnown 

func (k VectorKind) IsKnown() bool

IsKnown reports whether k is VectorKindNone or a declared kind.

func (VectorKind) MarshalText 

func (k VectorKind) MarshalText() ([]byte, error)

MarshalText renders the kind by name ("measure", ...; "" for VectorKindNone) so a schema's JSON report never shows the kind byte.

func (VectorKind) String 

func (k VectorKind) String() string

String returns the kind's name ("measure", "scale", "composition", "binary"), "" for VectorKindNone and "unknown(N)" for an unregistered byte.

func (*VectorKind) UnmarshalText 

func (k *VectorKind) UnmarshalText(b []byte) error

UnmarshalText inverts MarshalText: "" is VectorKindNone, a declared kind name its kind; anything else is an error.

type WidenReport added in v0.38.0 

type WidenReport struct {
	// Field is the name of the widened set field.
	Field string
	// From is the set rung the field carried before the rewrite.
	From FieldType
	// To is the set rung it carries after.
	To FieldType
	// Records is the number of records re-laid-out.
	Records int64
	// StrideBefore is the source record stride in bytes, bitmap included.
	StrideBefore int
	// StrideAfter is the rewritten record stride in bytes, bitmap included.
	StrideAfter int
}

WidenReport describes a completed widen. It is the value both callers of the engine (shard auto-widen and the `pulse widen` leaf) report from, so it carries the strides as well as the identities: the stride delta is the only observable difference between a widen that re-laid-out every record and one that did nothing.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.