encoding

package
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 7, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package encoding handles the binary .pulse file format: reading, writing, and schema management.

Index

Constants

View Source 
const FingerprintSize = sha256.Size // 32

FingerprintSize is the byte length of the embedded .pulse content-hash fingerprint (a raw SHA-256 digest).

View Source 
const FormatVersion byte = 0x01

FormatVersion is the current .pulse format version.

View Source 
const HeaderSize = 9

HeaderSize is the total byte size of the file header (magic + version).

View Source 
const IndexFormatVersion byte = 0x03

IndexFormatVersion is the current point-lookup sidecar index format version. Independent of encoding.FormatVersion (the .pulse envelope version) — this is a separate, standalone sidecar format that versions on its own schedule.

v3 (current) adds a fixed-width bucket-offset table immediately before the bucket-data region, and moves the source-stat snapshot (size + mtime, introduced in v2) earlier — right after the key-spec block, before the bucket count — so ReadIndexMeta can read every non-bucket field with one short linear pass and never touch bucket data. The offset table is what makes ReadBucketByKey an O(1) seek: a reader computes bucketIndex = BucketIndex(key, bucketCount), seeks to offsetTableStart + bucketIndex*8, reads that one u64, seeks to bucketDataStart + offset, and parses only that bucket's self-delimited entry_count-prefixed data — never the whole table, never the whole file. See WriteIndex's format comment for the full v3 layout and IndexMeta's doc comment for the seek anchors.

v1/v2 sidecars are no longer readable: ReadIndexHeader rejects any version byte other than the current one with ENCODING_INVALID, forcing an explicit `pulse index build` rebuild rather than a silent partial read. There is no in-place migration path — sidecars are cheap, deterministic rebuild artifacts, never hand-authored or long-lived across binary versions.

View Source 
const IndexHeaderSize = 9

IndexHeaderSize is the total byte size of the sidecar index header (magic + version), mirroring HeaderSize for the .pulse envelope.

View Source 
const MaxDecimalPrecision = 38

MaxDecimalPrecision is the upper bound on decimal128 precision.

View Source 
const MaxDescriptionBytes = 1000

MaxDescriptionBytes is the maximum allowed byte length for a field description.

View Source 
const MinDecimalScale = 4

MinDecimalScale is the floor on division result scale (matches Arrow).

View Source 
const ReservedSchemaName = "_schema.pulse"

ReservedSchemaName is the canonical entry name inside a Pulse shard archive that carries the cohort's canonical schema, dictionaries, and sharding metadata (aggregate record count, shard count). The name is reserved — inserting a shard with this basename raises PULSE_SHARD_RESERVED_NAME at write time. Read-side enumeration of shards EXCLUDES this entry.

Variables

View Source 
var DateFormats = []string{
	"2006-01-02",
	"01/02/2006",
	"2006-01-02T15:04:05Z",
	"2006-01-02T15:04:05",
	"2006/01/02",
	"02-Jan-2006",
}

DateFormats enumerates the date literal layouts ParseDate accepts, in priority order — the first layout that parses a given literal wins. This is the authoritative list for turning a date literal into the persisted on-wire epoch-day representation; io/import.go's column-type inference (io/infer.go's own, independently-maintained dateFormats) is a lighter, best-effort "does this column look date-shaped" probe and intentionally stays separate from this conversion authority.

View Source 
var IndexMagicBytes = [8]byte{'P', 'U', 'L', 'S', 'E', 'I', 'D', 'X'}

IndexMagicBytes identifies a Pulse point-lookup sidecar index file. 8 bytes: "PULSEIDX". Distinct from MagicBytes ("PULSE\x00\x00\x00") so magic-byte dispatch never confuses a sidecar index with a .pulse cohort file — the two formats are never read through the same code path, but sharing the discriminant style keeps every Pulse binary artifact self-identifying the same way.

View Source 
var MagicBytes = [8]byte{'P', 'U', 'L', 'S', 'E', 0x00, 0x00, 0x00}

MagicBytes identifies a .pulse file. 8 bytes: "PULSE\x00\x00\x00"

View Source 
var SchemaDocMagic = [4]byte{'S', 'H', 'R', 'D'}

SchemaDocMagic is the four-byte marker that introduces the sharding metadata extension appended to a `_schema.pulse` entry after the standard schema block. Readers detect the marker and parse the extension; absence means the metadata is unavailable (the canonical fallback is to peek each shard's header for record counts).

View Source 
var ZipMagic = [4]byte{'P', 'K', 0x03, 0x04}

ZipMagic is the standard PKZip local-file-header magic. Pulse shard archives begin with these four bytes when written as zip containers; single-file .pulse cohorts begin with MagicBytes ("PULSE\x00\x00\x00"). Pulse.Open dispatches on the first four bytes to pick the read path.

Functions

func BitmapIsNull added in v0.9.0 

func BitmapIsNull(bitmap []byte, i int) bool

BitmapIsNull reports whether field at index i is marked null in bitmap. Bit ordering: field index i → byte i/8, bit i%8 (LSB-first within each byte). 1 = null, 0 = present.

func BitmapSetNull added in v0.9.0 

func BitmapSetNull(bitmap []byte, i int)

BitmapSetNull marks the field at index i as null in bitmap. The caller must have pre-allocated bitmap to at least ceil((i+1)/8) bytes.

func BucketIndex added in v0.27.0 

func BucketIndex(key []byte, bucketCount uint32) uint32

BucketIndex maps a key's hash onto a bucket slot in a table sized bucketCount. bucketCount == 0 always resolves to 0 — callers must guard the empty-index case themselves via len(Index.Buckets) before indexing.

func EncodeDecimal128 added in v0.2.0 

func EncodeDecimal128(d Decimal128) [16]byte

EncodeDecimal128 serializes a Decimal128 as 16 bytes of two's-complement little-endian integer.

func HashKey added in v0.27.0 

func HashKey(key []byte) uint64

HashKey returns the 64-bit FNV-1a digest of raw key bytes. Builders and lookup callers both route through this (and BucketIndex) so the two sides never diverge on hash choice.

func IsArchive added in v0.8.0 

func IsArchive(r io.ReaderAt, size int64) (bool, error)

IsArchive reports whether the first four bytes of r identify a zip container. It does no central-directory parsing — magic-byte check only — so it is cheap enough to call on every Open. Errors are returned for I/O failures; a short file returns (false, nil).

func ParseDate added in v0.27.0 

func ParseDate(raw string) (uint32, error)

ParseDate parses raw against DateFormats (first match wins) and returns the on-wire representation the .pulse `date` field type carries: whole days since the Unix epoch, narrowed to uint32 — exactly the conversion io/import.go's row importer performs when converting a date column cell (`t.Unix() / 86400`). This function is the single source of truth both the importer (io/import.go's convertValue) and the point-lookup key resolver (processing.ResolveLookupKeyBytes) call, so a date literal parsed at import time and the same literal parsed at lookup time always resolve to the identical on-wire uint32 — no separate epoch-math implementation to drift out of sync.

Returns an ENCODING_INVALID coded error when raw matches none of DateFormats.

func PromoteAdd added in v0.2.0 

func PromoteAdd(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteAdd returns the (precision, scale) of a SUM/SUB result given two operand types per SQL:2016 / Arrow Decimal128 rules: 

(p1, s1) ± (p2, s2) => (max(p1-s1, p2-s2) + max(s1, s2) + 1, max(s1, s2))

The result precision is clamped at MaxDecimalPrecision; clamping callers must check ClampedPrecision and emit PULSE_DECIMAL_OVERFLOW when overflow surfaces at runtime.

func PromoteDiv added in v0.2.0 

func PromoteDiv(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteDiv returns the (precision, scale) of a DIV result. 

(p1, s1) ÷ (p2, s2) => (p1 + s2 + 1, max(s1+s2, MIN_SCALE))

func PromoteMul added in v0.2.0 

func PromoteMul(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteMul returns the (precision, scale) of a MUL result. 

(p1, s1) × (p2, s2) => (p1 + p2, s1 + s2)

func ReadBit 

func ReadBit(r io.Reader, bitPos uint) (bool, error)

ReadBit reads a single bit from the byte at the current position in r. bitPos is 0-7 within that byte.

func ReadBitmap added in v0.9.0 

func ReadBitmap(r io.Reader, byteLen int) ([]byte, error)

ReadBitmap reads a per-record null bitmap of the given byte length and returns the raw bytes. byteLen MUST be Schema.BitmapByteSize(); callers invoke this only when the schema has at least one nullable field.

func ReadDescription 

func ReadDescription(r io.Reader) (string, error)

ReadDescription reads a field description from r. Format: u16 length + utf8 bytes.

func ReadFieldValue 

func ReadFieldValue(r io.Reader, ft FieldType) (uint64, error)

ReadFieldValue reads a single field value from r, returning raw bits as uint64. For bit-packed types (U4, PackedBool), use ReadBit/ReadNibble instead.

func ReadHeader 

func ReadHeader(r io.Reader) error

ReadHeader reads and validates the .pulse file header from r.

func ReadIndexHeader added in v0.27.0 

func ReadIndexHeader(r io.Reader) error

ReadIndexHeader reads and validates the sidecar index header from r. Returns ENCODING_INVALID on a truncated header, wrong magic, or an unsupported version byte — mirroring ReadHeader's contract for the .pulse envelope.

func ReadNibble 

func ReadNibble(r io.Reader, high bool) (uint8, error)

ReadNibble reads a 4-bit value from a byte. If high is true, it reads bits 4-7; otherwise bits 0-3.

func RewriteShardCategoricals added in v0.8.0 

func RewriteShardCategoricals(shardBytes []byte, targetSchema *Schema, remap map[int]DictRemap) ([]byte, error)

RewriteShardCategoricals re-emits a single-file .pulse shard with its schema block replaced by targetSchema and its records' categorical indices translated according to remap. The on-wire byte layout of every non-categorical field is preserved exactly.

targetSchema must be structurally identical to the shard's own schema (field count, names, types, byte offsets, bit positions) — only the per-categorical dictionaries differ.

When remap is empty (every incoming dict was a prefix of canonical), the function still re-emits the shard with targetSchema so the shard's standalone dictionary matches the archive's canonical (callers reading the shard via the `archive.pulse#shard.pulse` anchor see the union dictionary). Record bytes are copied verbatim.

Returns PULSE_SHARD_HEADER_INVALID if the input bytes are not a valid single-file Pulse cohort, and PULSE_SHARD_SCHEMA_MISMATCH if targetSchema is structurally incompatible with the shard's own schema.

func SidecarIndexPath added in v0.27.0 

func SidecarIndexPath(cohortPath string, keyFields []string) string

SidecarIndexPath derives the deterministic on-disk path for a point-lookup sidecar index built against cohortPath's key column set: "<cohortPath>.<keyhash>.idx". keyhash is the 16-hex-digit FNV-1a digest (HashKey) of the ordered key column names, joined with a NUL byte separator that cannot appear inside a schema field name — so ["ab", "c"] and ["a", "bc"] hash to distinct paths rather than colliding on naive concatenation.

Key column order is significant: SidecarIndexPath(p, []string{"a", "b"}) and SidecarIndexPath(p, []string{"b", "a"}) derive different, stable paths, matching the composite-key contract ("key column order is significant") the format's Keys spec carries. Single-key builds (this story) pass a one-element slice; the derivation is already N-column-general so E2-S1's composite-key work and E4-S1's CLI / E1-S4's lookup service need no path-derivation change.

Deterministic and pure — no filesystem access, no ordering beyond what the caller supplies in keyFields. Callers own key-order normalization (or intentional non-normalization) before calling.

func ValidatePrecisionScale added in v0.2.0 

func ValidatePrecisionScale(precision, scale uint8) error

ValidatePrecisionScale reports whether (precision, scale) form a legal decimal128 type spec (1 ≤ precision ≤ 38, 0 ≤ scale ≤ precision).

func WriteBit 

func WriteBit(w io.Writer, bitPos uint, val bool) error

WriteBit writes a byte with a single bit set or cleared at bitPos.

func WriteBitmap added in v0.9.0 

func WriteBitmap(w io.Writer, bitmap []byte) error

WriteBitmap writes a per-record null bitmap. The slice MUST be exactly Schema.BitmapByteSize() bytes; this helper is intentionally thin to keep the writer responsible for allocating and populating the buffer.

func WriteDecimal128 added in v0.2.0 

func WriteDecimal128(w io.Writer, d Decimal128) error

WriteDecimal128 writes a Decimal128 to the .pulse record stream.

func WriteDescription 

func WriteDescription(w io.Writer, desc string) error

WriteDescription writes a field description to w. Format: u16 length + utf8 bytes. Returns PULSE_IMPORT_DESCRIPTION_TOO_LONG if the UTF-8 byte length exceeds MaxDescriptionBytes.

func WriteFieldValue 

func WriteFieldValue(w io.Writer, ft FieldType, val uint64) error

WriteFieldValue writes a single field value (as raw bits in uint64) to w. For bit-packed types (U4, PackedBool), use WriteBit/WriteNibble instead. For decimal128 (16-byte type), use the dedicated WriteDecimal128 helper; this function will reject those types with ENCODING_TYPE_MISMATCH.

func WriteHeader 

func WriteHeader(w io.Writer) error

WriteHeader writes the .pulse file header to w.

func WriteIndex added in v0.27.0 

func WriteIndex(w io.Writer, idx *Index) error

WriteIndex serializes idx to w.

Format (v3): 

9-byte header: magic "PULSEIDX" + version 0x03
32-byte fingerprint: raw SHA-256 digest
key-spec block:
  u16 key_count
  per key: u16 name_len + utf8 name, u8 type
source-stat snapshot:
  u64 source_size
  i64 source_mod_time_unix_nano
u32 bucket_count
bucket-offset table (fixed-width, bucket_count entries):
  per bucket: u64 byte_offset — relative to the start of the
    bucket-data region (i.e. relative to the byte immediately
    following this table). offsets[0] is always 0.
bucket-data region (bucket_count self-delimited blocks, in order):
  per bucket:
    u32 entry_count
    per entry:
      u16 key_len + raw key bytes
      u32 row_id_count
      per row_id: u64 (little-endian)

The offset table's fixed width is what lets a reader (ReadBucketByKey) seek directly to entry bucketIndex without parsing any other bucket: offsetTableStart + bucketIndex*8 locates the one relevant offset, and bucketDataStart + offsets[bucketIndex] locates that bucket's self-delimited data. See IndexMeta's doc comment for the anchor definitions.

func WriteIndexFile added in v0.27.0 

func WriteIndexFile(fsys afero.Fs, path string, idx *Index) error

WriteIndexFile serializes idx and writes it to path on fsys, creating or truncating the file as needed. All sidecar index I/O goes through afero.Fs so callers get the same hermetic-test story as the rest of Pulse's file surface (fs.NewMemMap() in tests, fs.Default() / a caller-injected afero.Fs in production).

func WriteIndexHeader added in v0.27.0 

func WriteIndexHeader(w io.Writer) error

WriteIndexHeader writes the sidecar index header (magic + version) to w.

func WriteNibble 

func WriteNibble(w io.Writer, high bool, val uint8) error

WriteNibble writes a 4-bit value into a byte. If high is true, it writes to bits 4-7; otherwise bits 0-3. The other nibble is zero.

func WriteSchema 

func WriteSchema(w io.Writer, s *Schema) error

WriteSchema serializes a schema to w. Format: 

u16 field_count
per field:
  u8 type
  u8 nullable (0 or 1)
  u16 name_length + utf8 name
  u32 byte_offset
  u8 bit_position
  u16 csv_column_idx
  u16 description_length + utf8 description
  (if decimal128) u8 precision + u8 scale
  (if categorical) dictionary block

func WriteSchemaDoc added in v0.8.0 

func WriteSchemaDoc(w io.Writer, schema *Schema, agg uint64, shardCount uint16) error

WriteSchemaDoc emits a `_schema.pulse` payload: standard Pulse header, standard schema block, then the sharding metadata extension (magic "SHRD" + u64 aggregate_record_count + u16 shard_count, little-endian). The result is a valid single-file Pulse cohort with zero records, so legacy readers that don't know about the SHRD extension can still parse the header + schema and stop at the (empty) record region — the extension bytes live where records would otherwise begin and a header-only reader (`Inspect`) ignores them.

Types

type Archive added in v0.8.0 

type Archive struct {
	// contains filtered or unexported fields
}

Archive is a parsed Pulse shard archive: zip central-directory cache plus the underlying ReaderAt. Callers Open or OpenAt entries by name. The zero value is not usable; construct via OpenArchive.

func OpenArchive added in v0.8.0 

func OpenArchive(r io.ReaderAt, size int64) (*Archive, error)

OpenArchive parses the zip central directory at the tail of r and returns an Archive ready for entry enumeration. The ReaderAt is retained — the caller is responsible for its lifetime. Returns PULSE_ARCHIVE_MAGIC_INVALID when r does not start with the zip magic, and PULSE_ARCHIVE_CORRUPT when the EOCD or central directory cannot be parsed.

func (*Archive) Entries added in v0.8.0 

func (a *Archive) Entries() []ArchiveEntry

Entries returns every entry in central-directory order (equals shard insertion order for Pulse-written archives). The slice is a fresh copy; callers may mutate freely.

func (*Archive) IsStored added in v0.8.0 

func (a *Archive) IsStored(name string) bool

IsStored reports whether the named entry uses store-only (Method 0) compression. Pulse-written shards are always store-only; this helper is exposed for callers that want to confirm the contract before using OpenAt on a section reader.

func (*Archive) Open added in v0.8.0 

func (a *Archive) Open(name string) (io.ReadCloser, error)

Open returns an io.ReadCloser over the named entry's payload. Returns PULSE_SHARD_MISSING when no entry of that name exists in the central directory. Suitable for streaming reads; the closer must be drained or closed when done.

func (*Archive) OpenAt added in v0.8.0 

func (a *Archive) OpenAt(name string) (io.SectionReader, error)

OpenAt returns an io.SectionReader covering the named entry's stored payload. Only safe for store-only (Method 0) entries — Pulse always writes Method 0, but third-party archives that pass through this path may carry deflated entries; the caller should verify the entry's Method (see Archive.IsStored) before consuming the section reader. Returns PULSE_SHARD_MISSING for unknown names.

func (*Archive) PeekShardHeader added in v0.8.0 

func (a *Archive) PeekShardHeader(name string) error

PeekShardHeader reads the first encoding.HeaderSize bytes of the named entry and validates them as a single-file Pulse header (magic + format version). Returns PULSE_SHARD_HEADER_INVALID on mismatch so callers can surface a structured error without parsing the full schema. Used when peeking shard metadata on first read.

func (*Archive) PeekShardRecordCount added in v0.8.0 

func (a *Archive) PeekShardRecordCount(name string) (int64, error)

PeekShardRecordCount opens the named entry, reads its Pulse header and schema block, validates both, and computes the record count from the remaining bytes divided by the schema's per-record size. Used by `Pulse.Open` to populate `ShardEntry.RecordCount` from the per-shard headers (the source of truth — `_schema.pulse`'s AggregateRecordCount is only a sanity check).

Returns PULSE_SHARD_HEADER_INVALID when the shard's bytes are not a valid single-file Pulse cohort (bad magic, unsupported version, truncated header or schema).

type ArchiveEntry added in v0.8.0 

type ArchiveEntry struct {
	// Name is the zip entry name (basename inside the archive). For
	// flat archives this is the shard filename, e.g. "20190101.pulse".
	Name string

	// Size is the uncompressed payload size in bytes.
	Size int64

	// Offset is the byte offset of the payload within the archive.
	Offset int64
}

ArchiveEntry is a single payload inside a Pulse shard archive. Offset is the byte offset of the entry's compressed/stored payload within the archive (the equivalent of zip.File.DataOffset), NOT the offset of the local file header. Pulse stores entries uncompressed (Method 0), so Offset is a direct pointer into the shard's bytes.

type CohesionWarning added in v0.8.0 

type CohesionWarning struct {
	Code    string
	Message string
	Details map[string]any
}

CohesionWarning is a structured non-fatal divergence emitted by the schema-cohesion validators. Its shape mirrors the descriptor envelope's {code, message, details} entries so callers (typically `service/shard add` and `pulse shard verify`) can forward warnings through the standard --json output without reshaping.

encoding/ stays free of descriptor/ imports — this is the local equivalent.

func ValidateStructuralCohesion added in v0.8.0 

func ValidateStructuralCohesion(canonical, incoming *Schema) ([]CohesionWarning, error)

ValidateStructuralCohesion verifies that incoming's structural schema is byte-equal to canonical's. The check is field-count strict and per-field strict on:

  • Name
  • Type byte
  • ByteOffset
  • BitPosition
  • Categorical-width identity (a categorical_u8 cannot become a categorical_u16 across shards — the width is fixed at folder creation)

Descriptions are advisory and may diverge across shards; per-field divergence emits a PULSE_SHARD_DESCRIPTION_DIVERGENCE warning but does NOT fail validation. Any other mismatch returns a coded PULSE_SHARD_SCHEMA_MISMATCH error.

This validator does NOT compare dictionaries — dictionary cohesion is governed by the append-only prefix rule (see ValidateDictPrefixRule).

type Decimal128 added in v0.2.0 

type Decimal128 struct {
	// contains filtered or unexported fields
}

Decimal128 is a fixed-point decimal value with up to 38 digits of precision. Logically it is a signed 128-bit two's-complement integer mantissa, paired with a per-field scale that places the implicit decimal point. Internally it is carried as *big.Int (always copied, never aliased) bounded to the decimal128 representable range [-(10^38 - 1), 10^38 - 1].

func DecodeDecimal128 added in v0.2.0 

func DecodeDecimal128(buf [16]byte) Decimal128

DecodeDecimal128 deserializes 16 bytes of two's-complement little-endian integer into a Decimal128. Null state is carried by the per-record bitmap (see encoding.Schema.HasBitmap), so this function does not flag null values.

func NewDecimal128FromBigInt added in v0.2.0 

func NewDecimal128FromBigInt(m *big.Int) (Decimal128, error)

NewDecimal128FromBigInt builds a Decimal128 from a big.Int mantissa. Returns PULSE_DECIMAL_OVERFLOW if |m| >= 10^38.

func NewDecimal128FromInt added in v0.2.0 

func NewDecimal128FromInt(i int64) Decimal128

NewDecimal128FromInt builds a Decimal128 with the given integer mantissa.

func ParseDecimal128 added in v0.2.0 

func ParseDecimal128(s string) (Decimal128, uint8, error)

ParseDecimal128 parses a strict decimal string into a (Decimal128, scale) pair. The scale is inferred from the input — exactly the count of digits after the decimal point. Accepts an optional single leading sign char. Rejects whitespace, thousand separators, scientific notation, currency symbols.

func ReadDecimal128 added in v0.2.0 

func ReadDecimal128(r io.Reader) (Decimal128, error)

ReadDecimal128 reads 16 bytes of decimal128 from r and decodes them. Null state is carried by the per-record bitmap, not by the payload bytes, so this function has no null channel.

func ZeroDecimal128 added in v0.2.0 

func ZeroDecimal128() Decimal128

ZeroDecimal128 returns a Decimal128 with mantissa 0.

func (Decimal128) Add added in v0.2.0 

func (d Decimal128) Add(o Decimal128) (Decimal128, error)

Add returns d + o, treating both at the same scale. The result has the same scale; the caller is responsible for SQL:2016 precision propagation at the schema level.

func (Decimal128) Cmp added in v0.2.0 

func (d Decimal128) Cmp(o Decimal128) int

Cmp compares two Decimal128 values with the same scale. Comparing values at different scales requires the caller to align scales first.

func (Decimal128) Div added in v0.2.0 

func (d Decimal128) Div(o Decimal128, s1, s2, resultScale uint8) (Decimal128, error)

Div divides d by o using banker's rounding to produce a result at scale `resultScale` given the operands at scales s1 and s2. Returns PULSE_DECIMAL_DIVIDE_BY_ZERO when o is zero.

func (Decimal128) FitsPrecision added in v0.2.0 

func (d Decimal128) FitsPrecision(precision uint8) bool

FitsPrecision reports whether the mantissa fits in `precision` digits.

func (Decimal128) Float64 added in v0.2.0 

func (d Decimal128) Float64(scale uint8) float64

Float64 returns the decimal as a float64 at the given scale. Lossy for values that exceed float64 precision; callers preserving precision should keep using Decimal128 directly.

func (Decimal128) Mantissa added in v0.2.0 

func (d Decimal128) Mantissa() *big.Int

Mantissa returns a copy of the unscaled integer mantissa.

func (Decimal128) Mul added in v0.2.0 

func (d Decimal128) Mul(o Decimal128) (Decimal128, error)

Mul returns d * o. The product mantissa is the product of mantissas; callers are responsible for tracking scale propagation (s1 + s2).

func (Decimal128) Rescale added in v0.2.0 

func (d Decimal128) Rescale(sourceScale, targetScale uint8) (Decimal128, error)

Rescale converts d at sourceScale to targetScale using banker's rounding. Returns PULSE_DECIMAL_OVERFLOW if the rescaled mantissa exceeds 10^38-1.

func (Decimal128) Sign added in v0.2.0 

func (d Decimal128) Sign() int

Sign returns -1, 0, or +1 depending on the mantissa sign.

func (Decimal128) Sqrt added in v0.2.0 

func (d Decimal128) Sqrt(sourceScale, targetScale uint8) (Decimal128, error)

Sqrt returns floor-banker-rounded sqrt(d) at the target scale, given the source value at sourceScale. The result is computed entirely in decimal arithmetic via big.Int.Sqrt and a half-to-even rounding step on the residual. Returns PULSE_DECIMAL_OVERFLOW if intermediate state cannot fit in the integer representation; returns PROCESSING_RUNTIME for negative inputs (sqrt of a negative decimal is undefined).

func (Decimal128) String added in v0.2.0 

func (d Decimal128) String(scale uint8) string

String renders the decimal at the given scale. Trailing zeros are preserved; the leading sign is included only for negative values.

func (Decimal128) Sub added in v0.2.0 

func (d Decimal128) Sub(o Decimal128) (Decimal128, error)

Sub returns d - o.

type DecodeFields added in v0.17.0 

type DecodeFields struct {
	Fields []*Field
}

DecodeFields runs the existing per-field decoder loop for the listed fields together. The Fields slice is in schema order; the caller walks it exactly as the unprojected decoder would, so bit-packed neighbours (which on-wire each consume one byte via ReadBit / ReadNibble) stay correctly aligned.

Fields is non-empty for any segment the builder emits.

type DecodePlan added in v0.17.0 

type DecodePlan struct {
	Segments []Segment
}

DecodePlan is a pre-computed walk of one record's on-wire bytes that elides ranges the caller will not consume. It is produced by Schema.BuildDecodePlan given a retained-field set, and consumed by the streaming iterator so unprojected byte ranges turn into a single advance-cursor step instead of a per-field typed read.

A DecodePlan is a pure function of (Schema, retained-set). It carries no file content, no offsets into a particular reader, and no mutable state. Two calls with the same inputs return byte-equivalent Segments in identical order.

type DictRemap added in v0.8.0 

type DictRemap = map[uint32]uint32

DictRemap is the per-field index translation produced by MergeDictUnion. The key is the incoming dictionary index; the value is the corresponding canonical (union) dictionary index. Fields whose incoming dictionary is already a prefix of the canonical (union) dictionary are absent from the returned map — callers can skip remapping their record bytes.

type Dictionary 

type Dictionary struct {
	// contains filtered or unexported fields
}

Dictionary maps string values to sequential uint32 IDs. It is used for categorical field types to encode string categories as compact integer IDs in the binary record format.

func NewDictionary 

func NewDictionary() *Dictionary

NewDictionary creates an empty dictionary.

func (*Dictionary) Add 

func (d *Dictionary) Add(s string) (uint32, error)

Add inserts a string into the dictionary, returning its ID. If the string already exists, the existing ID is returned. There is no capacity limit with Add; use AddWithLimit to enforce one.

func (*Dictionary) AddWithLimit 

func (d *Dictionary) AddWithLimit(s string, maxEntries uint32) (uint32, error)

AddWithLimit inserts a string, enforcing a maximum entry count. Returns PULSE_IMPORT_CATEGORICAL_OVERFLOW if the dictionary is full.

func (*Dictionary) Count 

func (d *Dictionary) Count() int

Count returns the number of entries in the dictionary.

func (*Dictionary) IDFor 

func (d *Dictionary) IDFor(s string) (uint32, bool)

IDFor looks up the ID for a string. Returns the ID and true if found, or 0 and false otherwise.

func (*Dictionary) ReadFrom 

func (d *Dictionary) ReadFrom(r io.Reader) (int64, error)

ReadFrom deserializes a dictionary from r, replacing current contents. Format: u32 count + (u16 strlen + utf8 bytes) x count

Performance: a single byte buffer is grown to hold all string payloads across the dictionary, instead of allocating one []byte per entry. Each resolved string is copied once via the standard string([]byte) conversion (no unsafe), so we drop one allocation per entry while preserving the safety contract that callers can mutate the underlying buffer afterwards without affecting the stored strings.

func (*Dictionary) Resolve 

func (d *Dictionary) Resolve(id uint32) string

Resolve returns the string for a given ID. Returns "" if the ID is out of range.

func (*Dictionary) Values 

func (d *Dictionary) Values() []string

Values returns a copy of all dictionary values in insertion order.

func (*Dictionary) WriteTo 

func (d *Dictionary) WriteTo(w io.Writer) (int64, error)

WriteTo serializes the dictionary to w. Format: u32 count + (u16 strlen + utf8 bytes) x count

type Field 

type Field struct {
	Name         string
	Type         FieldType
	Nullable     bool // true ⇒ field participates in per-record null bitmap
	ByteOffset   int
	BitPosition  int
	CsvColumnIdx int
	Description  string      // empty = synthesized at inspect time
	Dictionary   *Dictionary // non-nil only for categorical types

	// Precision is the decimal128 precision (1-38). Meaningful only when
	// Type is FieldTypeDecimal128.
	Precision uint8
	// Scale is the decimal128 scale (0-Precision). Meaningful only when
	// Type is FieldTypeDecimal128.
	Scale uint8
}

Field describes a single column in a .pulse schema.

type FieldFilter added in v0.8.0 

type FieldFilter func(name string) bool

FieldFilter returns true for field names whose values should be written into the caller's maps. Used by ReadRecordWithWideProjected to skip map writes for fields the request doesn't read. A nil FieldFilter is equivalent to "keep every field."

type FieldType 

type FieldType byte

FieldType identifies the data type stored in a schema field. 

const (
	FieldTypeU8             FieldType = iota // 0
	FieldTypeU16                             // 1
	FieldTypeU32                             // 2
	FieldTypeU64                             // 3
	FieldTypeF32                             // 4
	FieldTypeF64                             // 5
	FieldTypeU4                              // 6  (4-bit, bit-packed)
	FieldTypeDate                            // 7
	FieldTypePackedBool                      // 8  (1-bit, bit-packed)
	FieldTypeCategoricalU8                   // 9
	FieldTypeCategoricalU16                  // 10
	FieldTypeCategoricalU32                  // 11
	FieldTypeDecimal128                      // 12
	FieldTypeSetU8                           // 13 (bitmask over shared dict, ≤8 members)
	FieldTypeSetU16                          // 14 (≤16 members)
	FieldTypeSetU32                          // 15 (≤32 members)
	FieldTypeSetU64                          // 16 (≤64 members)

)

All field types supported by the .pulse format. Nullability is orthogonal to type — any field can be marked nullable via encoding.Field.Nullable and the per-record null bitmap carries the actual null state.

func ParseFieldType added in v0.15.0 

func ParseFieldType(name string) (FieldType, bool)

ParseFieldType inverts FieldType.String, returning the typed value and ok=true when name matches a registered FieldType. Caller paths (managed-imports sidecar parsing, MCP schema validation, ad-hoc admin tooling) use this to round-trip an externally supplied type name through the codec layer without relying on stringly typed switches in every place.

func (FieldType) ByteSize 

func (ft FieldType) ByteSize() int

ByteSize returns the number of bytes this field type occupies in a record. Bit-packed types (U4, PackedBool) share bytes with adjacent fields and return 0 here; their on-wire stride is handled by Schema.RecordByteSize.

func (FieldType) HasDictionary added in v0.15.0 

func (ft FieldType) HasDictionary() bool

HasDictionary reports whether the field type carries an inline dictionary block in the schema (categorical_* or set_*).

func (FieldType) IsBitPacked added in v0.9.0 

func (ft FieldType) IsBitPacked() bool

IsBitPacked reports whether the field type shares its on-wire byte with adjacent fields via bit packing (U4, PackedBool). Used by stride math and the schema layout pass to advance the bit cursor instead of the byte cursor.

func (FieldType) IsCategorical 

func (ft FieldType) IsCategorical() bool

IsCategorical reports whether the field type is one of the categorical types.

func (FieldType) IsDecimal added in v0.2.0 

func (ft FieldType) IsDecimal() bool

IsDecimal reports whether the field type is decimal128.

func (FieldType) IsKnown added in v0.2.0 

func (ft FieldType) IsKnown() bool

IsKnown reports whether the byte value corresponds to a registered type. Used by the schema reader to reject files written by a future binary version that introduces unknown type bytes.

func (FieldType) IsNumeric added in v0.7.2 

func (ft FieldType) IsNumeric() bool

IsNumeric reports whether the field type is a strict scalar number: the unsigned-integer family (u8/u16/u32/u64), the float family (f32/f64), and decimal128. Date and bit-packed integer encodings are excluded — see IsNumericForAnalytics for the analytics-layer predicate.

func (FieldType) IsNumericForAnalytics added in v0.7.2 

func (ft FieldType) IsNumericForAnalytics() bool

IsNumericForAnalytics reports whether the field type carries a meaningful scalar value for numeric analytics (regression, sum/avg/stddev/min/max/ variance aggregators). The set is broader than IsNumeric: bit-packed integer encodings (u4, packed_bool) and date are included because their stored representation is an ordinal / cardinal number the analytics layer can average, sum, or regress without an explicit ATTR_FORMULA cast.

Null exclusion is the reader's responsibility: the per-record null bitmap marks any field index as null at decode time so the downstream Record.NumericValue contract (returns ok=false on null) keeps the aggregation denominator honest.

func (FieldType) IsSet added in v0.15.0 

func (ft FieldType) IsSet() bool

IsSet reports whether the field type is a bitmask-over-dictionary set (multi-select) type. Set fields share the categorical dictionary block shape but the on-wire payload is a fixed-width unsigned integer whose bit i corresponds to dictionary entry i.

func (FieldType) MaxCategoricalEntries 

func (ft FieldType) MaxCategoricalEntries() uint32

MaxCategoricalEntries returns the maximum dictionary size for a categorical type. Returns 0 for non-categorical types.

func (FieldType) MaxDictEntries added in v0.15.0 

func (ft FieldType) MaxDictEntries() uint32

MaxDictEntries returns the capacity of the inline dictionary block for dictionary-bearing types. Categoricals return MaxCategoricalEntries; sets return MaxSetEntries (= bitmask width). Non-dictionary types return 0.

func (FieldType) MaxSetEntries added in v0.15.0 

func (ft FieldType) MaxSetEntries() uint32

MaxSetEntries returns the maximum dictionary size (= bitmask capacity) for a set type. Returns 0 for non-set types.

func (FieldType) String 

func (ft FieldType) String() string

String returns a human-readable name for the field type.

type Fingerprint added in v0.27.0 

type Fingerprint [FingerprintSize]byte

Fingerprint is a raw SHA-256 content-hash digest of the source .pulse file an Index was built from. A later story (index build) computes one from the live cohort's bytes at build time; the lookup path recomputes it and compares against the embedded value to detect a stale index without decoding the cohort's records.

func ComputeFingerprint added in v0.27.0 

func ComputeFingerprint(r io.Reader) (Fingerprint, error)

ComputeFingerprint hashes the full contents of r with SHA-256. Callers pass the raw bytes of the source .pulse file (header + schema + records) — identical content always yields an identical Fingerprint, regardless of platform or filesystem.

type Index added in v0.27.0 

type Index struct {
	Fingerprint Fingerprint
	Keys        []IndexKeySpec
	Buckets     []IndexBucket

	// SourceSize is the byte length of the source .pulse file, as
	// reported by the filesystem at build time. Paired with
	// SourceModTime to give a freshness check (Service.VerifyIndex) a
	// cheap fast-path that avoids re-hashing the whole cohort: if
	// either value no longer matches the current file's stat, the
	// cohort has definitely changed and the index is definitely stale
	// — no need to pay for a full content hash to know that. A
	// matching size+mtime pair is NOT by itself sufficient proof of
	// freshness (mtime resolution and pathological same-size rewrites
	// both make silent false negatives possible), so a match always
	// falls through to a full Fingerprint recompute for a conclusive
	// answer. See Service.Lookup / Service.VerifyIndex.
	SourceSize uint64

	// SourceModTime is the source .pulse file's modification time, as
	// Unix nanoseconds, at build time. See SourceSize's doc comment for
	// the fast-path contract this pairs with.
	SourceModTime int64
}

Index is the full in-memory representation of a point-lookup sidecar index: the .pulse Fingerprint it was built from, the ordered key spec, the fixed-size hash-bucket table (len(Buckets) is the table's bucket count), and a source-stat snapshot (SourceSize / SourceModTime) taken at build time. Encoding/decoding here is pure codec — populating an Index from a live cohort and serving lookups against one are later stories' responsibility; this package stays free of service/processing imports.

func ReadIndex added in v0.27.0 

func ReadIndex(r io.Reader) (*Index, error)

ReadIndex deserializes a sidecar Index from r. Returns ENCODING_INVALID on a truncated/corrupt header, an unknown key FieldType byte, or a truncated body. A full read — every bucket's every entry — for callers that genuinely need the whole index (e.g. Service.ListIndexes' distinct-key / indexed-record summary). Callers that only need metadata should prefer ReadIndexMeta; callers that only need one bucket should prefer ReadBucketByKey.

func ReadIndexFile added in v0.27.0 

func ReadIndexFile(fsys afero.Fs, path string) (*Index, error)

ReadIndexFile opens path on fsys and deserializes a sidecar Index.

type IndexBucket added in v0.27.0 

type IndexBucket struct {
	Entries []IndexEntry
}

IndexBucket is one slot of the on-disk hash table. A well-distributed build populates each bucket with zero or one entries on average; more than one entry means a hash collision on BucketIndex, resolved by the reader scanning Entries for a byte-equal Key.

func ReadBucketByKey added in v0.27.0 

func ReadBucketByKey(rs io.ReadSeeker, meta *IndexMeta, key []byte) (*IndexBucket, error)

ReadBucketByKey seeks directly to and parses exactly ONE bucket of a v3 sidecar index — the bucket key hashes to via BucketIndex(key, meta.BucketCount) — never reading the bucket-offset table's other entries or any other bucket's data. meta must have been produced by ReadIndexMeta (or ReadIndexMetaFile) against the SAME sidecar rs reads from; rs's absolute byte offset 0 must correspond to the start of that sidecar (a freshly opened file handle, or one explicitly Seek'd back to 0), since meta.OffsetTableStart is an absolute file offset.

Returns an empty *IndexBucket (no error) when meta.BucketCount == 0 (an empty index has no buckets to seek into — mirrors the read side's existing empty-index handling). Returns ENCODING_INVALID on a seek failure, a truncated/corrupt bucket-offset entry, an offset that decodes to a value larger than a signed 64-bit byte offset can represent, or a truncated/corrupt bucket-data block at the resolved position.

type IndexEntry added in v0.27.0 

type IndexEntry struct {
	Key    []byte
	RowIDs []uint64
}

IndexEntry is one hash-bucket slot's payload: the exact on-wire key bytes (the concatenation of every key field's on-wire representation, in key-spec order) plus every row-id sharing that key. RowIDs holds more than one value whenever the source cohort has duplicate key values — the sidecar format is multimap-capable from day one so a later epic's multiplicity work needs no format change.

type IndexKeySpec added in v0.27.0 

type IndexKeySpec struct {
	Name string
	Type FieldType
}

IndexKeySpec describes one ordered key column carried in the sidecar index's key-spec block: the schema field name and its on-wire FieldType. Composite keys (more than one IndexKeySpec) concatenate every key field's on-wire byte representation, in key-spec order, to form a single IndexEntry.Key.

type IndexMeta added in v0.27.0 

type IndexMeta struct {
	Fingerprint Fingerprint
	Keys        []IndexKeySpec

	SourceSize    uint64
	SourceModTime int64

	// BucketCount is the sidecar's hash-table bucket count — the same
	// value BucketIndex(key, bucketCount) needs to resolve a lookup key
	// to a bucket slot.
	BucketCount uint32

	// OffsetTableStart is the absolute byte offset, from the start of
	// the sidecar file, where the fixed-width bucket-offset table
	// begins. See the type doc comment for the full seek-math contract.
	OffsetTableStart int64
}

IndexMeta is the cheap, bucket-data-free subset of a sidecar index: everything ReadIndexMeta can read with one short linear pass — header, fingerprint, key-spec, source-stat snapshot, and bucket count — plus the two byte-offset anchors a seekable reader needs to jump straight to one bucket's data without reading the offset table or any other bucket:

  • OffsetTableStart: the absolute byte offset (from the start of the sidecar) where the fixed-width bucket-offset table begins. Entry i of that table lives at OffsetTableStart + i*8.
  • BucketDataStart (a derived method, not a stored field): the absolute byte offset where the bucket-data region begins, immediately after the offset table (OffsetTableStart + bucketCount*8). Every bucket-offset table entry is a byte offset RELATIVE to this anchor, so bucket i's absolute data position is BucketDataStart() + offsets[i].

Staleness checks (Service.VerifyIndex) and arity checks (matching a lookup request's key-component count against the sidecar's own key-spec) only ever need this struct — never bucket data.

func ReadIndexMeta added in v0.27.0 

func ReadIndexMeta(r io.Reader) (*IndexMeta, error)

ReadIndexMeta deserializes only the bucket-data-free prefix of a sidecar index from r: header, fingerprint, key-spec, source-stat snapshot, and bucket count — never touching the bucket-offset table or any bucket data. r must start at byte 0 of the sidecar (the same assumption ReadIndex and ReadIndexHeader already make) so the returned IndexMeta.OffsetTableStart — computed analytically from the exact byte lengths of the sections just read, not by seeking — is a correct absolute file offset.

This is the cheap path Service.VerifyIndex's staleness check and a lookup request's key-arity check both want: neither needs a single bucket, let alone the whole table.

func ReadIndexMetaFile added in v0.27.0 

func ReadIndexMetaFile(fsys afero.Fs, path string) (*IndexMeta, error)

ReadIndexMetaFile opens path on fsys and deserializes only the bucket-data-free IndexMeta prefix — the afero-backed convenience counterpart to ReadIndexMeta, mirroring ReadIndexFile's relationship to ReadIndex. Callers that only need staleness/arity metadata (e.g. Service.VerifyIndex) should prefer this over ReadIndexFile.

func (*IndexMeta) BucketDataStart added in v0.27.0 

func (m *IndexMeta) BucketDataStart() int64

BucketDataStart returns the absolute byte offset, from the start of the sidecar file, where the bucket-data region begins — immediately after the fixed-width bucket-offset table. Every bucket-offset table entry is a byte offset relative to this anchor.

type RecordLocator added in v0.27.0 

type RecordLocator struct {
	// Schema is the parsed schema this locator's geometry is derived
	// from. Callers pass it straight through to ReadRecordWithWidePlan
	// / ReadRecordWithWideProjected via NewRecordReader.
	Schema *Schema

	// RecordRegionStart is the absolute byte offset of the first record,
	// i.e. the number of bytes the header + schema prefix consumed.
	RecordRegionStart int64

	// Stride is Schema.RecordByteSize() — the fixed on-wire byte width
	// of one record (including the trailing null bitmap, if any).
	Stride int64

	// TotalRecords is the number of complete records the payload holds,
	// derived from the payload's total byte length. Zero for an empty
	// cohort or a cohort shorter than one full record stride.
	TotalRecords uint64
}

RecordLocator captures the fixed geometry a single-file .pulse payload needs for O(1) index-addressed record reads: the byte offset where the record region begins (immediately after the header + schema prefix), the fixed per-record stride, and the total record count for bounds checking. Construct one per opened cohort via NewRecordLocator and reuse it across many point lookups — the offset math mirrors service/parallel_decode.go's per-worker segment slicing (recordRegionStart + i*stride), just applied to a single record instead of a worker's contiguous range.

func NewRecordLocator added in v0.27.0 

func NewRecordLocator(r *bytes.Reader, schema *Schema) (*RecordLocator, error)

NewRecordLocator derives record-region geometry for a single-file .pulse payload from r, an io.ReadSeeker over the raw payload bytes — most commonly a *bytes.Reader wrapping a mmap'd byte slice, the same path the streaming iterator and the parallel decode context use. r's cursor must be at absolute position 0 (the very first header byte) when this is called; NewRecordLocator reads (and discards, beyond measuring their length) the header and schema, leaving r positioned at the first record byte as a side effect. Callers that need r re-wound for further use should re-seek to loc.RecordRegionStart or construct a fresh reader over the same bytes.

schema is the already-parsed schema for this cohort (e.g. from pulse.Open) — NewRecordLocator does not re-validate its field contents against the freshly-read schema bytes; it reads the schema block purely to measure how many bytes the header+schema prefix consumes. A mismatched schema silently produces a locator with wrong geometry, so callers must pass the schema that was actually parsed from this same payload.

A *bytes.Reader is required (rather than a generic io.ReadSeeker) so this function can read Size() — the reader's fixed, read-independent total length — to derive TotalRecords without a second file stat.

func (*RecordLocator) Offset added in v0.27.0 

func (loc *RecordLocator) Offset(i uint64) int64

Offset returns the absolute byte offset of record i within the payload this locator was built from. It does not bounds-check — callers that need a guarded lookup should use ReadRecordAt, which validates i < TotalRecords before ever seeking.

func (*RecordLocator) ReadRecordAt added in v0.27.0 

func (loc *RecordLocator) ReadRecordAt(
	r io.ReadSeeker,
	i uint64,
	values map[string]float64,
	nulls map[string]bool,
	wide map[string]any,
	keep FieldFilter,
	plan *DecodePlan,
) error

ReadRecordAt decodes the record at index i directly: it seeks r to loc.Offset(i) and decodes from there, without iterating any preceding record. r must be an io.ReadSeeker positioned over the same underlying bytes the locator was built from (the same *bytes.Reader passed to NewRecordLocator, or a fresh reader over the same byte slice) — the seek is absolute, so r's current cursor position on entry is irrelevant.

plan is optional: pass a *DecodePlan (from Schema.BuildDecodePlan) to decode only the columns the plan retains — the O(1) projection variant required by point-lookup callers that only need a handful of return columns out of a wide schema. Pass nil to decode every field via the existing full-record path. keep mirrors the FieldFilter contract used throughout this package (see ReadRecordWithWidePlan / ReadRecordWithWideProjected) — pass the same filter used to build plan's retained set so map writes for incidental group members (a bit-packed neighbour or a bitmap-adjacent nullable field) stay suppressed; pass nil to keep every field the plan/schema visits.

Returns a coded ENCODING_INVALID error — never a panic, never an out-of-bounds read — when i >= loc.TotalRecords.

type RecordReader 

type RecordReader struct {
	// contains filtered or unexported fields
}

RecordReader reads records one at a time from a binary stream. It reads directly from the io.Reader without buffering the entire file.

func NewRecordReader 

func NewRecordReader(r io.Reader, schema *Schema) *RecordReader

NewRecordReader creates a RecordReader. The reader must be positioned immediately after the header and schema (i.e., at the first record byte).

func (*RecordReader) ReadRecord 

func (rr *RecordReader) ReadRecord(values map[string]float64, nulls map[string]bool) error

ReadRecord reads a single record from the stream, populating the values and nulls maps. Returns io.EOF when no more records are available.

The caller provides pre-allocated maps to avoid per-record allocation. Maps are cleared at the start of each call.

Reuse contract: the maps are owned by the caller. ReadRecord does not retain references to them after returning. If the caller plans to reuse the same maps across calls (the typical pattern), they must consume the populated values BEFORE invoking ReadRecord again, because the next call clears and repopulates the maps in-place. If the caller needs to retain the values past the next call (e.g., collecting Records into a slice for later aggregation), it must pass distinct map instances per record OR copy the contents out before the next ReadRecord call.

To populate typed wide values for fields whose representation does not fit in float64 (decimal128), call ReadRecordWithWide instead and pass a third map.

func (*RecordReader) ReadRecordReused added in v0.2.0 

func (rr *RecordReader) ReadRecordReused(rec ReusableRecord) error

ReadRecordReused reads one record into an existing ReusableRecord, reusing the record's internal maps. Returns io.EOF when the underlying reader is exhausted.

Hot path semantics:

  • Caller MUST consume the populated rec before the next call.
  • The whole record stride (Schema.RecordByteSize(), including the trailing null bitmap) is read into a reusable per-RecordReader buffer with a SINGLE io.ReadFull, then every field is decoded from a running cursor subslice of that buffer — no per-field read, no copy. This eliminates the per-field io.ReadFull that dominates wide-schema decode.
  • Fields are walked in schema order with a running byte cursor, NOT by Field.ByteOffset (bit-packed layout is cursor-driven; stored offsets are unreliable). Each bit-packed field (u4, packed_bool) occupies a whole on-wire byte, matching ReadBit/ReadNibble semantics.
  • A short read at end-of-stream surfaces io.EOF exactly as before via mapEOF; a partial trailing record surfaces as io.EOF as well (io.ReadFull maps a nonzero short read to io.ErrUnexpectedEOF, which mapEOF normalizes to io.EOF).

func (*RecordReader) ReadRecordReusedWithPlan added in v0.26.0 

func (rr *RecordReader) ReadRecordReusedWithPlan(rec ReusableRecord, keep FieldFilter, plan *DecodePlan) error

ReadRecordReusedWithPlan reads one record into an existing ReusableRecord by walking a precomputed DecodePlan, decoding only the fields the plan's DecodeFields segments carry and seeking past the SkipBytes ranges the caller will not consume. It is the plan-aware sibling of ReadRecordReused: same reuse contract (the record's null/wide maps are cleared once via ClearForRow, then populated in place), same io.EOF surfacing, but projection-honoring under reuse.

plan == nil falls back to ReadRecordReused (full-decode reuse path), so an iterator that never installed a plan is byte-identical to today.

keep governs record writes within DecodeFields segments, mirroring ReadRecordWithWidePlan / decodeFieldGroup / decodeBitmap exactly:

  • Bit-packed members (u4, packed_bool) of a retained group are ALWAYS consumed for cursor alignment (1 whole on-wire byte each), but their record writes are suppressed when keep rejects them.
  • The trailing bitmap segment reads the bitmap once and surfaces nulls only for nullable fields the caller retains.

Side effects match ReadRecordReused field-for-field:

  • null → SetNullField(name) + SetNumeric(name, 0);
  • decimal128 → SetNumeric(mean-ish scalar) + SetWideField(Decimal128);
  • set_* → SetNumeric(float64 echo) + SetWideField(uint64 mask).

Unlike ReadRecordReused, this path does NOT read the whole record stride: each DecodeFields group reads only its own on-wire bytes into the reusable buffer with a single io.ReadFull, and SkipBytes segments advance the reader past unread ranges with a single Seek (io.CopyN fallback). On a wide schema with a small retained set that is the projection win under reuse.

func (*RecordReader) ReadRecordWithWide added in v0.2.0 

func (rr *RecordReader) ReadRecordWithWide(values map[string]float64, nulls map[string]bool, wide map[string]any) error

ReadRecordWithWide reads a record and populates a wide map with typed values for decimal128 fields. The wide map may be nil to skip wide population.

func (*RecordReader) ReadRecordWithWidePlan added in v0.17.0 

func (rr *RecordReader) ReadRecordWithWidePlan(
	values map[string]float64,
	nulls map[string]bool,
	wide map[string]any,
	keep FieldFilter,
	plan *DecodePlan,
) error

ReadRecordWithWidePlan reads one record by walking a precomputed DecodePlan instead of iterating every schema field. SkipBytes segments advance the underlying reader by N bytes with a single seek (or io.CopyN fallback); DecodeFields segments run the same per-field decode loop the unprojected reader uses, but only for the fields the segment carries.

keep governs map writes within DecodeFields segments. The plan builder emits bit-packed groups together (every member of a group whose retained subset is non-empty), so bit-packed members that the caller did NOT request are still decoded — their on-wire bytes must be consumed to keep the cursor aligned — but their map writes are suppressed by keep. Similarly the trailing bitmap segment carries every nullable field so the iterator can read the bitmap once and surface nulls; keep filters those map writes.

keep == nil and plan == nil both fall back to the existing full-decode behaviour. Passing keep == nil with a non-nil plan widens every DecodeFields segment to "keep all members" (useful for golden parity tests).

Behaviour invariants matched against the per-field readRecord loop:

  • Caller-supplied maps are cleared in place before population.
  • Decimal128 fields populate wide[name] with the typed Decimal128 value (only when wide != nil and the field is retained).
  • Set-typed fields populate wide[name] with the raw uint64 mask (only when wide != nil and the field is retained).
  • Null surfacing clears values[name] back to 0 and deletes wide[name] for nullable fields the bitmap marks null AND the caller retains.

func (*RecordReader) ReadRecordWithWideProjected added in v0.8.0 

func (rr *RecordReader) ReadRecordWithWideProjected(values map[string]float64, nulls map[string]bool, wide map[string]any, keep FieldFilter) error

ReadRecordWithWideProjected reads a record but only writes the fields for which keep(name) returns true into the caller's maps. Bytes for excluded fields are still consumed from the underlying reader so byte offsets stay aligned — projection saves map allocations, not decode work.

keep == nil falls back to the full-decode path.

type ReusableRecord added in v0.2.0 

type ReusableRecord interface {
	SetNumeric(name string, value float64)
	SetNullField(name string)
	SetWideField(name string, value any)
	ClearForRow()
}

ReusableRecord is the subset of *processing.Record needed by the reuse fast path. Declared here as an interface so encoding/ does not depend on processing/. Implementations (processing.Record) MUST clear their own null/wide maps before this call returns successfully; the reader populates them in place but only on fields where the value applies.

type Schema 

type Schema struct {
	Fields []Field
}

Schema holds all field descriptors for a .pulse file.

func MergeDictUnion added in v0.8.0 

func MergeDictUnion(canonical, incoming *Schema) (*Schema, map[int]DictRemap, error)

MergeDictUnion is the relaxed dictionary cohesion check. For each categorical field it computes the union of canonical's and incoming's dictionaries: canonical entries first in their existing order, then any new entries from incoming in their order. Returns the extended canonical schema (deep copy when an extension is adopted; identical pointer when not) and a per-field-index map of DictRemap entries describing how to translate incoming record indices into canonical indices.

The structural validator (ValidateStructuralCohesion) must run FIRST — this validator assumes field positions, names, and categorical widths already match.

Errors:

  • PULSE_SHARD_DICT_WIDTH_OVERFLOW when the union would exceed the field's categorical width capacity.

MergeDictUnion is the default cohesion mode at insert time (CreateShardArchive / AddShard). The stricter prefix-only rule (ValidateDictPrefixRule) is retained for callers that want to surface divergence as an error instead of merging — used by VerifyShardArchive when checking archives written before the union-merge default landed and by embedders that prefer to align dictionaries upstream.

func ReadSchema 

func ReadSchema(r io.Reader) (*Schema, error)

ReadSchema deserializes a schema from r.

func ValidateDictPrefixRule added in v0.8.0 

func ValidateDictPrefixRule(canonical, incoming *Schema) (*Schema, error)

ValidateDictPrefixRule enforces the append-only prefix rule for every categorical field shared by canonical and incoming. For each `categorical_*` field:

  • If incoming's dictionary is a prefix of canonical's, accept the shard as-is; no canonical change. (Older shard that never saw newer dictionary values.)
  • If canonical's dictionary is a prefix of incoming's, the canonical schema adopts the extension. The returned extendedCanonical carries the merged dictionaries and the caller is responsible for rewriting `_schema.pulse` to publish them before the new shard is placed (crash-safety ordering documented in placeholder §3.2).
  • Neither is a prefix of the other → PULSE_SHARD_DICT_DIVERGENCE.
  • An extension that would exceed the field's width capacity (256 for u8, 65 536 for u16, 2^32 for u32) → PULSE_SHARD_DICT_WIDTH_OVERFLOW.

The structural validator (ValidateStructuralCohesion) must run FIRST — this validator assumes field positions, names, and categorical widths already match.

The returned extendedCanonical is a deep copy when an extension is adopted, and is identically the canonical input otherwise. Callers can compare pointers to detect whether a rewrite is required.

func (*Schema) BitmapByteSize added in v0.9.0 

func (s *Schema) BitmapByteSize() int

BitmapByteSize returns the number of bytes the null bitmap occupies per record, or 0 when no field is nullable.

func (*Schema) BuildDecodePlan added in v0.17.0 

func (s *Schema) BuildDecodePlan(retained []string) (*DecodePlan, error)

BuildDecodePlan produces a deterministic DecodePlan for this schema given the set of field names the caller intends to consume.

Rules:

  • Contiguous unprojected, non-bit-packed fields coalesce into a single SkipBytes whose N is the sum of those fields' ByteSize().
  • Bit-packed neighbours (any contiguous run of FieldType.IsBitPacked() fields) form one group. The group becomes one DecodeFields segment if any member is retained; otherwise it becomes one SkipBytes whose N equals the group's on-wire byte size — which mirrors Schema.RecordByteSize: 1 byte per bit-packed field. (BitPosition is a per-field annotation; ReadBit / ReadNibble each consume a full byte today, so a run of K bit-packed fields occupies K bytes on-wire.)
  • Trailing null bitmap: if Schema.HasBitmap() and any nullable field is retained, append a DecodeFields segment carrying every nullable field in schema order (the iterator decodes the bitmap, then walks this list to surface null flags). If HasBitmap() but no nullable field is retained, append a SkipBytes{N: Schema.BitmapByteSize()}. If !HasBitmap(), no bitmap segment is emitted.
  • Empty retained set: the plan is a single SkipBytes covering the full record stride (RecordByteSize, which already includes the bitmap if any).
  • Full retained set (every schema field present in retained): no SkipBytes segments — the output is equivalent to the existing per-field walk.

retained is consulted as a set (order does not matter); duplicates and names not present in the schema are silently ignored. The function never returns an error today, but the signature carries one for forward-compatibility with future shape validation (e.g. once extension-driven retained sets carry wildcard tokens).

func (*Schema) Categorical 

func (s *Schema) Categorical(name string) (*Dictionary, bool)

Categorical returns the dictionary for a named categorical field. Returns nil, false if the field is not found or is not categorical.

func (*Schema) Field 

func (s *Schema) Field(name string) *Field

Field returns a pointer to the named field, or nil if not found.

func (*Schema) HasBitmap added in v0.9.0 

func (s *Schema) HasBitmap() bool

HasBitmap reports whether any field in the schema is marked nullable. When true, every record carries a trailing null bitmap of ceil(field_count/8) bytes after the payload; when false, records have no bitmap (legacy fixed-stride path, zero overhead).

func (*Schema) RecordByteSize added in v0.8.4 

func (s *Schema) RecordByteSize() int

RecordByteSize returns the on-wire stride of one record under this schema. Bit-packed fields (U4, PackedBool) report ByteSize()==0 but the wire format still consumes one whole byte per such field. When the schema declares at least one nullable field, the trailing null bitmap of ceil(field_count/8) bytes is appended to every record and included in the stride.

func (*Schema) SetField added in v0.15.0 

func (s *Schema) SetField(name string) (*Dictionary, bool)

SetField returns the dictionary for a named set-typed field. Returns nil, false if the field is not found or is not a set type.

type SchemaDoc added in v0.8.0 

type SchemaDoc struct {
	Schema               *Schema
	AggregateRecordCount uint64
	ShardCount           uint16
}

SchemaDoc carries the contents of a `_schema.pulse` entry: the canonical Schema plus the sharding metadata extension. AggregateRecordCount is the cached sum of every shard's record count at the time the archive was last written; ShardCount mirrors the zip central directory's entry count (excluding the reserved schema entry).

Both metadata fields are a SANITY CHECK, not the source of truth. Per the design contract, the per-shard headers are authoritative — callers populate live `ShardEntry.RecordCount` values by peeking each shard's header. AggregateRecordCount is exposed so cheap `pulse inspect` calls can report an approximate total without opening every shard.

func ReadSchemaDoc added in v0.8.0 

func ReadSchemaDoc(r io.Reader) (*SchemaDoc, error)

ReadSchemaDoc parses a `_schema.pulse` payload. It always reads the Pulse header + standard schema block. The sharding metadata extension (magic "SHRD" + agg + shard count) is read when present and silently skipped when absent — older `_schema.pulse` artifacts or single-file cohorts addressed by mistake still parse cleanly, returning AggregateRecordCount=0 and ShardCount=0.

type Segment added in v0.17.0 

type Segment interface {
	// contains filtered or unexported methods
}

Segment is the sealed sum type of decode steps emitted by Schema.BuildDecodePlan. The unexported marker method blocks out-of-package implementations so the iterator's switch is exhaustive.

type SkipBytes added in v0.17.0 

type SkipBytes struct {
	N int
}

SkipBytes advances the underlying reader by N bytes without decoding. Coalesces a run of contiguous unprojected, non-bit-packed fields, or an entire bit-packed group when no member is retained, or the trailing null bitmap when no nullable field is retained. N > 0 always.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.