encoding

package
v0.39.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 2, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package encoding handles the binary .pulse file format: reading, writing, and schema management.

Index

Constants

View Source 
const (
	// GroupVerdictAdmitted: the group is written and passed both floors.
	GroupVerdictAdmitted = "admitted"
	// GroupVerdictLowRatio: the group is written, with a
	// PULSE_DEDUP_LOW_RATIO warning.
	GroupVerdictLowRatio = "low_ratio"
	// GroupVerdictDroppedTooNarrow: the group was not formed; its
	// members stay row fields (PULSE_GROUP_TOO_NARROW).
	GroupVerdictDroppedTooNarrow = "dropped_too_narrow"
)

Group viability verdicts, as reported per group.

View Source 
const (
	// FormatVersionV1 is the original layout: header + field descriptors
	// + inline dictionaries + fixed-stride records, with nothing between
	// the last descriptor and the first record.
	FormatVersionV1 byte = 0x01
	// FormatVersionV2 appends a length-prefixed schema extension block
	// after the field descriptors (see [WriteSchema]). Readers of a v2
	// file always know exactly where the record region starts.
	FormatVersionV2 byte = 0x02
)

Format versions a .pulse header may declare in byte 8.

The version a writer emits is a FUNCTION OF SCHEMA CONTENT (Schema.RequiredFormatVersion), never a global flag: a schema that uses no 0x02 feature is written as 0x01, byte-identical to every file written before 0x02 existed. Old cohorts stay readable forever — the reader accepts every version in the supported set and parses the schema block by the version it read (ReadSchema).

A binary that predates a version refuses it at ReadHeader with a coded ENCODING_INVALID naming the version, before any schema byte is interpreted, so a newer file never misparses silently in an older binary.

View Source 
const CanonicalDateTimeLayout = "2006-01-02T15:04:05Z"

CanonicalDateTimeLayout is the single text form FormatDateTime emits for a `datetime` field: an ISO-8601 / RFC 3339 calendar-date + wall-clock-time literal with a literal `Z` suffix, second resolution, no fractional part. It is the canonical round-trip form — a value exported through it re-parses under DateTimeFormats[0] to the exact same on-wire uint64, which is what makes the text-oriented adapters (CSV / TSV / NDJSON / JSON) lossless for `datetime` without any format-native datetime column type.

View Source 
const DefaultDedupRatioFloor = 2.0

DefaultDedupRatioFloor is the ratio floor DedupGate applies when none is configured: 2 rows per distinct tuple.

Why 2. For a group whose members are wide against the 4-byte index, the resident dictionary is (member bytes per row) × rows ÷ ratio, so the ratio is the factor by which the parent block shrinks: at 2 the dictionary still holds half the bytes the rows used to carry, and below it most of the parent data becomes resident memory while the saving tends to zero (a ratio of 1.25 keeps 80% of it resident for a 20% cut). A floor of 2 therefore reads "each stored tuple is shared by at least two rows on average" and flags exactly the case the planning measurements called out — the same declaration measured 2.73x on one cohort (passes) and 1.24x on another (warns). It is a judgement, not a law: it is overridable (DedupGate.RatioFloor, CLI --dedup-ratio-floor) and only ever warns. The grows-the-file check is separate and applies at any floor, because a narrow-ish group (members only slightly wider than the index) can grow the file at a ratio well above 2 — its break-even ratio is reported alongside.

View Source 
const FingerprintSize = sha256.Size // 32

FingerprintSize is the byte length of the embedded .pulse content-hash fingerprint (a raw SHA-256 digest).

View Source 
const FormatVersion byte = FormatVersionV1

FormatVersion is the BASELINE .pulse format version: the version WriteHeader emits and the one every schema that uses no 0x02 feature is written at. It is not "the newest version this binary understands" — that is MaxFormatVersion.

View Source 
const GroupIndexWidth = 4

GroupIndexWidth is the on-wire width of an indexed group's per-row index. The descriptor carries the width explicitly; this binary accepts only this value for GroupKindIndexed and 0 for GroupKindConstant.

View Source 
const HeaderSize = 9

HeaderSize is the total byte size of the file header (magic + version).

View Source 
const IndexFormatVersion byte = 0x03

IndexFormatVersion is the current point-lookup sidecar index format version. Independent of encoding.FormatVersion (the .pulse envelope version) — this is a separate, standalone sidecar format that versions on its own schedule.

v3 (current) adds a fixed-width bucket-offset table immediately before the bucket-data region, and moves the source-stat snapshot (size + mtime, introduced in v2) earlier — right after the key-spec block, before the bucket count — so ReadIndexMeta can read every non-bucket field with one short linear pass and never touch bucket data. The offset table is what makes ReadBucketByKey an O(1) seek: a reader computes bucketIndex = BucketIndex(key, bucketCount), seeks to offsetTableStart + bucketIndex*8, reads that one u64, seeks to bucketDataStart + offset, and parses only that bucket's self-delimited entry_count-prefixed data — never the whole table, never the whole file. See WriteIndex's format comment for the full v3 layout and IndexMeta's doc comment for the seek anchors.

v1/v2 sidecars are no longer readable: ReadIndexHeader rejects any version byte other than the current one with ENCODING_INVALID, forcing an explicit `pulse index build` rebuild rather than a silent partial read. There is no in-place migration path — sidecars are cheap, deterministic rebuild artifacts, never hand-authored or long-lived across binary versions.

View Source 
const IndexHeaderSize = 9

IndexHeaderSize is the total byte size of the sidecar index header (magic + version), mirroring HeaderSize for the .pulse envelope.

View Source 
const IndexManifestFormatVersion = "1"

IndexManifestFormatVersion is the manifest document's own version, independent of both the .pulse FormatVersion and the sidecar IndexFormatVersion: the manifest is a JSON catalog, not a binary layout, and versioning it separately means adding a field to it can never force a cohort or index rebuild. An unrecognised value is REFUSED, for the same reason a foreign kind is.

View Source 
const IndexManifestKind = "pulse.index-manifest"

IndexManifestKind is the manifest document's self-identifying kind. A document carrying any other kind is REFUSED rather than read as an empty manifest — silently reporting "this cohort has no indexes" over a file that is simply something else is exactly the failure the manifest exists to remove.

View Source 
const IndexManifestSuffix = ".indexes.json"

IndexManifestSuffix is appended to a cohort's filename to derive its sidecar index MANIFEST — "cohort.pulse.indexes.json". It follows the same "suffix appended to the cohort filename" convention imports.SidecarSuffix (".meta.json") and spss.SidecarSuffix (".spss.json") already use, and is deliberately distinct from both.

It exists because SidecarIndexPath derives an index's filename from a HASH OF ITS KEY TUPLE, which makes a sidecar impossible to open without already knowing the thing a caller is trying to learn. The key tuple is recorded INSIDE the sidecar (IndexMeta.Keys), so on a local disk the escape is to glob "<cohort>.*.idx" and read each match's meta — but OBJECT STORAGE CANNOT LIST A DIRECTORY, so a caller reading cohorts out of a bucket has neither the hash nor a listing and cannot discover an index at all. Guessing is a permutation search, because key ORDER is significant end to end.

This manifest is the keyless, deterministically-named answer: one predictable path, reachable with a single GET, carrying every sidecar's ordered key tuple. Service.ListIndexes reads it first and falls back to the directory listing only when it is absent, so a corpus written by a Pulse that predates it keeps working unchanged.

View Source 
const MaxDecimalPrecision = 38

MaxDecimalPrecision is the upper bound on decimal128 precision.

View Source 
const MaxDescriptionBytes = 1000

MaxDescriptionBytes is the maximum allowed byte length for a field description.

View Source 
const MaxFormatVersion byte = FormatVersionV2

MaxFormatVersion is the newest .pulse format version this binary reads and writes.

View Source 
const MinDecimalScale = 4

MinDecimalScale is the floor on division result scale (matches Arrow).

View Source 
const MinElisionRows = 2

MinElisionRows is the smallest cohort constant elision considers. Over a single row every field is trivially "constant", which says nothing about the data; over zero rows a constant group has no value to hold and cannot be written.

View Source 
const ReservedSchemaName = "_schema.pulse"

ReservedSchemaName is the canonical entry name inside a Pulse shard archive that carries the cohort's canonical schema, dictionaries, and sharding metadata (aggregate record count, shard count). The name is reserved — inserting a shard with this basename raises PULSE_SHARD_RESERVED_NAME at write time. Read-side enumeration of shards EXCLUDES this entry.

View Source 
const SecondsPerDay = 86400

SecondsPerDay is the conversion factor between the two temporal on-wire representations Pulse persists: a `datetime` cell holds whole epoch SECONDS, a `date` cell holds whole epoch DAYS. Every conversion between them goes through DateTimeToDay rather than open-coding this constant at the call site.

View Source 
const SetMaskBits = SetMaskWords * 64

SetMaskBits is the hard ceiling on set membership across every set rung: SetMaskWords × 64 = 256. The ceiling is what lets SetMask be a fixed array instead of a slice — see the SetMask doc comment.

View Source 
const SetMaskWords = 4

SetMaskWords is the number of uint64 words backing a SetMask.

Variables

View Source 
var DateFormats = []string{
	"2006-01-02",
	"01/02/2006",
	"2006-01-02T15:04:05Z",
	"2006-01-02T15:04:05",
	"2006/01/02",
	"02-Jan-2006",
}

DateFormats enumerates the date literal layouts ParseDate accepts, in priority order — the first layout that parses a given literal wins. This is the authoritative list for turning a date literal into the persisted on-wire epoch-day representation; io/import.go's column-type inference (io/infer.go's own, independently-maintained dateFormats) is a lighter, best-effort "does this column look date-shaped" probe and intentionally stays separate from this conversion authority.

View Source 
var DateTimeFormats = []string{
	"2006-01-02T15:04:05Z07:00",
	"2006-01-02T15:04:05",
	"2006-01-02T15:04Z07:00",
	"2006-01-02T15:04",
	"2006-01-02 15:04:05",
	"2006-01-02 15:04",
}

DateTimeFormats enumerates the datetime literal layouts ParseDateTime accepts, in priority order — the first layout that parses a given literal wins. Same shape and same first-match-wins contract as DateFormats, but a deliberately stricter list:

  • The slash forms DateFormats tolerates ("01/02/2006") are NOT accepted. `03/04/2024` is 4 March in most of the world and 3 April in the US; a datetime column silently swapping day and month is exactly the quiet degradation this type must not do. An ambiguous literal fails loudly with ENCODING_INVALID instead.
  • A literal carrying no time-of-day is NOT accepted. That is a `date`, and io/infer.go's column classification depends on this rejection to keep a date-only column typed FieldTypeDate rather than widening it to FieldTypeDateTime.

Go's time.Parse accepts a fractional-second field immediately after the seconds field even when the layout omits one, so "2024-03-04T10:11:12.5Z" parses under the first layout with no dedicated entry. Sub-second precision is then truncated by the seconds-resolution on-wire representation (see ParseDateTime).

View Source 
var IndexMagicBytes = [8]byte{'P', 'U', 'L', 'S', 'E', 'I', 'D', 'X'}

IndexMagicBytes identifies a Pulse point-lookup sidecar index file. 8 bytes: "PULSEIDX". Distinct from MagicBytes ("PULSE\x00\x00\x00") so magic-byte dispatch never confuses a sidecar index with a .pulse cohort file — the two formats are never read through the same code path, but sharing the discriminant style keeps every Pulse binary artifact self-identifying the same way.

View Source 
var MagicBytes = [8]byte{'P', 'U', 'L', 'S', 'E', 0x00, 0x00, 0x00}

MagicBytes identifies a .pulse file. 8 bytes: "PULSE\x00\x00\x00"

View Source 
var MaxGroupEntries uint64 = math.MaxUint32 + 1

MaxGroupEntries is the entry-count ceiling of an indexed group: the u32 index space. A variable (not a const) only so tests can shrink it.

View Source 
var SchemaDocMagic = [4]byte{'S', 'H', 'R', 'D'}

SchemaDocMagic is the four-byte marker that introduces the sharding metadata extension appended to a `_schema.pulse` entry after the standard schema block. Readers detect the marker and parse the extension; absence means the metadata is unavailable (the canonical fallback is to peek each shard's header for record counts).

View Source 
var ZipMagic = [4]byte{'P', 'K', 0x03, 0x04}

ZipMagic is the standard PKZip local-file-header magic. Pulse shard archives begin with these four bytes when written as zip containers; single-file .pulse cohorts begin with MagicBytes ("PULSE\x00\x00\x00"). Pulse.Open dispatches on the first four bytes to pick the read path.

View Source 
var ZstdMagic = [4]byte{0x28, 0xB5, 0x2F, 0xFD}

ZstdMagic is the zstd frame magic (0xFD2FB528 little-endian, RFC 8878) that opens a `.pulse.zst` transfer artifact. It is NOT a cohort layout and there is no third magic-byte variant: Pulse never reads a compressed cohort. ReadHeader recognises these four bytes only so it can refuse with PULSE_COHORT_COMPRESSED ("decompress first") instead of a generic ENCODING_INVALID.

Functions

func BitmapIsNull added in v0.9.0 

func BitmapIsNull(bitmap []byte, i int) bool

BitmapIsNull reports whether field at index i is marked null in bitmap. Bit ordering: field index i → byte i/8, bit i%8 (LSB-first within each byte). 1 = null, 0 = present.

func BitmapSetNull added in v0.9.0 

func BitmapSetNull(bitmap []byte, i int)

BitmapSetNull marks the field at index i as null in bitmap. The caller must have pre-allocated bitmap to at least ceil((i+1)/8) bytes.

func BucketIndex added in v0.27.0 

func BucketIndex(key []byte, bucketCount uint32) uint32

BucketIndex maps a key's hash onto a bucket slot in a table sized bucketCount. bucketCount == 0 always resolves to 0 — callers must guard the empty-index case themselves via len(Index.Buckets) before indexing.

func CheckSetWiden added in v0.38.0 

func CheckSetWiden(from, to FieldType) error

CheckSetWiden reports whether a set field of type from may be widened to type to, returning the refusal as a coded error otherwise. It is the whole admission policy, exported so a caller can pre-flight a widen (a CLI predict, a shard-admin decision) without touching bytes.

Every refusal is ENCODING_TYPE_MISMATCH: each one is a caller asking for a type transition the format does not define, which is exactly what that code names. This effort introduces no new error codes.

func CompressedCohortError added in v0.39.0 

func CompressedCohortError() *errors.CodedError

CompressedCohortError is the ONE coded refusal for a cohort path that holds a zstd transfer artifact rather than a `.pulse` cohort.

func DateTimeToDay added in v0.30.0 

func DateTimeToDay(sec int64) int64

DateTimeToDay truncates an on-wire `datetime` value (epoch seconds, naive UTC) to the epoch-day integer a `date` value carries. This is THE day-truncation semantic the date-family operators (GROUP_DATE, GROUP_DATE_RANGES, FILTER_DATE_RANGES) apply when their Field names a `datetime` column: the time-of-day component is discarded so the row buckets by the calendar day that contains its instant.

Truncation, never rounding — 2024-03-04T23:59:59Z is 2024-03-04, not 2024-03-05. And truncation toward the PAST, not toward zero: Go's integer division truncates toward zero, which would map every pre-1970 instant inside the first day before the epoch onto day 0 (1970-01-01) instead of day -1 (1969-12-31). The floor correction below keeps the calendar day honest across the sign boundary.

The int64 parameter is deliberate: ParseDateTime widens a negative second count into uint64 two's-complement for storage, so callers converting a stored value re-narrow with int64(raw) first — exactly the reinterpretation FormatDateTime performs.

func EncodeDecimal128 added in v0.2.0 

func EncodeDecimal128(d Decimal128) [16]byte

EncodeDecimal128 serializes a Decimal128 as 16 bytes of two's-complement little-endian integer.

func ForEachRecord added in v0.39.0 

func ForEachRecord(r io.Reader, stride int, fn func(rec []byte) error) (int64, error)

ForEachRecord reads fixed-width records of stride bytes from r and calls fn with each one, returning how many it read. The slice fn receives is reused between calls. A clean end of input between two records ends the walk; input ending MID-record is ENCODING_INVALID (details trailing_bytes, records_read) rather than a silently dropped tail. Every whole-cohort record pass of the dedup family — encoding, constant detection, candidate detection — walks through here, so the three cannot disagree about where a cohort ends.

func FormatDateTime added in v0.30.0 

func FormatDateTime(raw uint64) string

FormatDateTime renders an on-wire `datetime` value (epoch seconds as uint64) back into CanonicalDateTimeLayout. It is the exact inverse of ParseDateTime: FormatDateTime(ParseDateTime(s)) re-parses to the same uint64 for every s that ParseDateTime accepts.

The int64 conversion is deliberate and reverses the uint64 widening ParseDateTime performs, so pre-1970 instants render as their real calendar time rather than as a year-292-billion overflow.

func HashKey added in v0.27.0 

func HashKey(key []byte) uint64

HashKey returns the 64-bit FNV-1a digest of raw key bytes. Builders and lookup callers both route through this (and BucketIndex) so the two sides never diverge on hash choice.

func IndexManifestPath added in v0.35.0 

func IndexManifestPath(cohortPath string) string

IndexManifestPath derives the deterministic, KEYLESS on-disk path of the sidecar index manifest for cohortPath. Pure — no filesystem access. Unlike SidecarIndexPath there is nothing to know in advance: that is the whole point.

func IsArchive added in v0.8.0 

func IsArchive(r io.ReaderAt, size int64) (bool, error)

IsArchive reports whether the first four bytes of r identify a zip container. It does no central-directory parsing — magic-byte check only — so it is cheap enough to call on every Open. Errors are returned for I/O failures; a short file returns (false, nil).

func IsSupportedFormatVersion added in v0.39.0 

func IsSupportedFormatVersion(v byte) bool

IsSupportedFormatVersion reports whether v is a .pulse format version this binary can read.

func IsZstdMagic added in v0.39.0 

func IsZstdMagic(b []byte) bool

IsZstdMagic reports whether b begins with ZstdMagic.

func ParseDate added in v0.27.0 

func ParseDate(raw string) (uint32, error)

ParseDate parses raw against DateFormats (first match wins) and returns the on-wire representation the .pulse `date` field type carries: whole days since the Unix epoch, narrowed to uint32 — exactly the conversion io/import.go's row importer performs when converting a date column cell (`t.Unix() / 86400`). This function is the single source of truth both the importer (io/import.go's convertValue) and the point-lookup key resolver (processing.ResolveLookupKeyBytes) call, so a date literal parsed at import time and the same literal parsed at lookup time always resolve to the identical on-wire uint32 — no separate epoch-math implementation to drift out of sync.

Returns an ENCODING_INVALID coded error when raw matches none of DateFormats.

func ParseDateTime added in v0.30.0 

func ParseDateTime(raw string) (uint64, error)

ParseDateTime parses raw against DateTimeFormats (first match wins) and returns the on-wire representation the .pulse `datetime` field type carries: whole SECONDS since the Unix epoch, as uint64.

Note the contrast with ParseDate, which returns epoch DAYS as a uint32. The two are not interchangeable and must never be swapped: a `date` cell holds days, a `datetime` cell holds seconds.

Timezone policy is naive UTC — the format stores an instant and no zone. A layout with no offset is read as UTC (time.Parse's own default), and a layout carrying an explicit offset is normalised to the same instant, with the offset itself discarded. Round-tripping "2024-03-04T10:11:12+02:00" therefore yields "2024-03-04T08:11:12Z": the instant is preserved exactly, the local-wall-clock presentation is not.

Precision is seconds; any fractional-second component in raw is truncated toward the epoch by time.Time.Unix.

Pre-1970 instants are representable: the negative int64 second count is reinterpreted as uint64 two's-complement, which FormatDateTime reverses exactly, so the round trip is lossless across the whole int64 second range.

Returns an ENCODING_INVALID coded error when raw matches none of DateTimeFormats.

func PromoteAdd added in v0.2.0 

func PromoteAdd(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteAdd returns the (precision, scale) of a SUM/SUB result given two operand types per SQL:2016 / Arrow Decimal128 rules: 

(p1, s1) ± (p2, s2) => (max(p1-s1, p2-s2) + max(s1, s2) + 1, max(s1, s2))

The result precision is clamped at MaxDecimalPrecision; clamping callers must check ClampedPrecision and emit PULSE_DECIMAL_OVERFLOW when overflow surfaces at runtime.

func PromoteDiv added in v0.2.0 

func PromoteDiv(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteDiv returns the (precision, scale) of a DIV result. 

(p1, s1) ÷ (p2, s2) => (p1 + s2 + 1, max(s1+s2, MIN_SCALE))

func PromoteMul added in v0.2.0 

func PromoteMul(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteMul returns the (precision, scale) of a MUL result. 

(p1, s1) × (p2, s2) => (p1 + p2, s1 + s2)

func PutSetMask added in v0.38.0 

func PutSetMask(dst []byte, ft FieldType, m SetMask) error

PutSetMask writes m into dst as the on-wire payload of a wide set field. Exactly ft.ByteSize() bytes are written, starting at dst[0], in the little-endian word order documented on SetMask; dst may be longer (a record buffer sliced at the field offset) and nothing past the field width is touched. A dst shorter than the field width, a non-wide field type, or a mask wider than the rung's capacity is an error and leaves dst untouched.

func ReadBit 

func ReadBit(r io.Reader, bitPos uint) (bool, error)

ReadBit reads a single bit from the byte at the current position in r. bitPos is 0-7 within that byte.

func ReadBitmap added in v0.9.0 

func ReadBitmap(r io.Reader, byteLen int) ([]byte, error)

ReadBitmap reads a per-record null bitmap of the given byte length and returns the raw bytes. byteLen MUST be Schema.BitmapByteSize(); callers invoke this only when the schema has at least one nullable field.

func ReadDescription 

func ReadDescription(r io.Reader) (string, error)

ReadDescription reads a field description from r. Format: u16 length + utf8 bytes.

func ReadFieldValue 

func ReadFieldValue(r io.Reader, ft FieldType) (uint64, error)

ReadFieldValue reads a single field value from r, returning raw bits as uint64. For bit-packed types (U4, PackedBool), use ReadBit/ReadNibble instead. decimal128 and the wide set rungs (set_u128, set_u256) exceed the uint64 return and are rejected with ENCODING_TYPE_MISMATCH.

func ReadHeader 

func ReadHeader(r io.Reader) (byte, error)

ReadHeader reads and validates the .pulse file header from r and returns the format version it declares. The version must be handed to ReadSchema: the schema block's layout depends on it, and record data begins immediately after the schema block with no terminator, so parsing a v2 schema as v1 would misplace every record.

A version outside SupportedFormatVersions is ENCODING_INVALID with the offending byte under details["version"]. A zstd transfer artifact (ZstdMagic) is PULSE_COHORT_COMPRESSED (CompressedCohortError): error classification only — nothing is ever decompressed here.

func ReadIndexHeader added in v0.27.0 

func ReadIndexHeader(r io.Reader) error

ReadIndexHeader reads and validates the sidecar index header from r. Returns ENCODING_INVALID on a truncated header, wrong magic, or an unsupported version byte — mirroring ReadHeader's contract for the .pulse envelope.

func ReadNibble 

func ReadNibble(r io.Reader, high bool) (uint8, error)

ReadNibble reads a 4-bit value from a byte. If high is true, it reads bits 4-7; otherwise bits 0-3.

func RefuseGroups added in v0.39.0 

func RefuseGroups(s *Schema, operation string, code errors.Code) error

RefuseGroups returns a coded error when s declares parent groups and the caller's operation rewrites or reinterprets physical row bytes without understanding them yet (shard-archive assembly, set-field widening, categorical byte remap). It keeps such a path loud rather than silently misreading a grouped row by its logical offsets. nil for an ungrouped schema.

func RewriteShardCategoricals added in v0.8.0 

func RewriteShardCategoricals(shardBytes []byte, targetSchema *Schema, remap map[int]DictRemap) ([]byte, error)

RewriteShardCategoricals re-emits a single-file .pulse shard with its schema block replaced by targetSchema and its records' categorical indices translated according to remap. The on-wire byte layout of every non-categorical field is preserved exactly.

targetSchema must be structurally identical to the shard's own schema (field count, names, types, byte offsets, bit positions) — only the per-categorical dictionaries differ.

When remap is empty (every incoming dict was a prefix of canonical), the function still re-emits the shard with targetSchema so the shard's standalone dictionary matches the archive's canonical (callers reading the shard via the `archive.pulse#shard.pulse` anchor see the union dictionary). Record bytes are copied verbatim.

Returns PULSE_SHARD_HEADER_INVALID if the input bytes are not a valid single-file Pulse cohort, and PULSE_SHARD_SCHEMA_MISMATCH if targetSchema is structurally incompatible with the shard's own schema.

func RewriteShardCategoricalsKeepNulls added in v0.39.0 

func RewriteShardCategoricalsKeepNulls(shardBytes []byte, targetSchema *Schema, remap map[int]DictRemap) ([]byte, error)

RewriteShardCategoricalsKeepNulls is RewriteShardCategoricals except that a field which is NULL in a record keeps its placeholder bytes instead of having them remapped. A null's bytes carry no value, but they are part of a parent-group ENTRY's identity (an entry is compared byte for byte): remapping a null categorical member's placeholder would make a shard's (region = null) tuple a different entry from the archive's identical one — a duplicate entry in an unkeyed group, and a spurious PULSE_GROUP_MEMBER_NOT_CONSTANT on a keyed group whose non-key member is null. The shard-archive group path remaps the LOGICAL (flattened) form of a shard through this variant before re-encoding it against the canonical dictionaries; the ungrouped archive path keeps RewriteShardCategoricals' byte-for-byte behaviour.

func SidecarIndexPath added in v0.27.0 

func SidecarIndexPath(cohortPath string, keyFields []string) string

SidecarIndexPath derives the deterministic on-disk path for a point-lookup sidecar index built against cohortPath's key column set: "<cohortPath>.<keyhash>.idx". keyhash is the 16-hex-digit FNV-1a digest (HashKey) of the ordered key column names, joined with a NUL byte separator that cannot appear inside a schema field name — so ["ab", "c"] and ["a", "bc"] hash to distinct paths rather than colliding on naive concatenation.

Key column order is significant: SidecarIndexPath(p, []string{"a", "b"}) and SidecarIndexPath(p, []string{"b", "a"}) derive different, stable paths, matching the composite-key contract ("key column order is significant") the format's Keys spec carries. Single-key builds (this story) pass a one-element slice; the derivation is already N-column-general so E2-S1's composite-key work and E4-S1's CLI / E1-S4's lookup service need no path-derivation change.

Deterministic and pure — no filesystem access, no ordering beyond what the caller supplies in keyFields. Callers own key-order normalization (or intentional non-normalization) before calling.

func SupportedFormatVersions added in v0.39.0 

func SupportedFormatVersions() []byte

SupportedFormatVersions returns the accepted version set, ascending.

func ValidateGroupCohesion added in v0.39.0 

func ValidateGroupCohesion(canonical, incoming *Schema) error

ValidateGroupCohesion is the parent-group dimension of structural cohesion: canonical and incoming must declare the identical group layout — the same group count and, per group, the same kind, members and key flags. It is what makes the physical stride of every shard equal the canonical stride, so a 0x01 shard in a 0x02 archive (or the reverse, or two different groupings) is PULSE_SHARD_SCHEMA_MISMATCH. Group DICTIONARIES are not compared here; they follow the prefix rule (ValidateDictPrefixRule). ValidateStructuralCohesion calls it, so `pulse shard verify` refuses a genuinely mixed-layout archive.

func ValidatePrecisionScale added in v0.2.0 

func ValidatePrecisionScale(precision, scale uint8) error

ValidatePrecisionScale reports whether (precision, scale) form a legal decimal128 type spec (1 ≤ precision ≤ 38, 0 ≤ scale ≤ precision).

func WriteBit 

func WriteBit(w io.Writer, bitPos uint, val bool) error

WriteBit writes a byte with a single bit set or cleared at bitPos.

func WriteBitmap added in v0.9.0 

func WriteBitmap(w io.Writer, bitmap []byte) error

WriteBitmap writes a per-record null bitmap. The slice MUST be exactly Schema.BitmapByteSize() bytes; this helper is intentionally thin to keep the writer responsible for allocating and populating the buffer.

func WriteDecimal128 added in v0.2.0 

func WriteDecimal128(w io.Writer, d Decimal128) error

WriteDecimal128 writes a Decimal128 to the .pulse record stream.

func WriteDescription 

func WriteDescription(w io.Writer, desc string) error

WriteDescription writes a field description to w. Format: u16 length + utf8 bytes. Returns PULSE_IMPORT_DESCRIPTION_TOO_LONG if the UTF-8 byte length exceeds MaxDescriptionBytes.

func WriteFieldValue 

func WriteFieldValue(w io.Writer, ft FieldType, val uint64) error

WriteFieldValue writes a single field value (as raw bits in uint64) to w. For bit-packed types (U4, PackedBool), use WriteBit/WriteNibble instead. For decimal128 (16-byte type), use the dedicated WriteDecimal128 helper; this function will reject those types with ENCODING_TYPE_MISMATCH. The wide set rungs (set_u128, set_u256) are rejected the same way: their bitmask is wider than the uint64 this API speaks.

func WriteHeader 

func WriteHeader(w io.Writer) error

WriteHeader writes a .pulse file header declaring the baseline FormatVersion (0x01). A writer that already holds the schema should prefer WritePreamble, which derives the version from the schema's content and cannot disagree with the schema block that follows.

func WriteIndex added in v0.27.0 

func WriteIndex(w io.Writer, idx *Index) error

WriteIndex serializes idx to w.

Format (v3): 

9-byte header: magic "PULSEIDX" + version 0x03
32-byte fingerprint: raw SHA-256 digest
key-spec block:
  u16 key_count
  per key: u16 name_len + utf8 name, u8 type
source-stat snapshot:
  u64 source_size
  i64 source_mod_time_unix_nano
u32 bucket_count
bucket-offset table (fixed-width, bucket_count entries):
  per bucket: u64 byte_offset — relative to the start of the
    bucket-data region (i.e. relative to the byte immediately
    following this table). offsets[0] is always 0.
bucket-data region (bucket_count self-delimited blocks, in order):
  per bucket:
    u32 entry_count
    per entry:
      u16 key_len + raw key bytes
      u32 row_id_count
      per row_id: u64 (little-endian)

The offset table's fixed width is what lets a reader (ReadBucketByKey) seek directly to entry bucketIndex without parsing any other bucket: offsetTableStart + bucketIndex*8 locates the one relevant offset, and bucketDataStart + offsets[bucketIndex] locates that bucket's self-delimited data. See IndexMeta's doc comment for the anchor definitions.

func WriteIndexFile added in v0.27.0 

func WriteIndexFile(fsys afero.Fs, path string, idx *Index) error

WriteIndexFile serializes idx and writes it to path on fsys, creating or truncating the file as needed. All sidecar index I/O goes through afero.Fs so callers get the same hermetic-test story as the rest of Pulse's file surface (fs.NewMemMap() in tests, fs.Default() / a caller-injected afero.Fs in production).

func WriteIndexHeader added in v0.27.0 

func WriteIndexHeader(w io.Writer) error

WriteIndexHeader writes the sidecar index header (magic + version) to w.

func WriteIndexManifest added in v0.35.0 

func WriteIndexManifest(fsys afero.Fs, cohortPath string, m *IndexManifest) error

WriteIndexManifest serialises m to cohortPath's derived manifest path through fsys (never raw os), indented because the document's second job is to be read by a human debugging a corpus. Entries are sorted first, so the same set of indexes always produces the same bytes.

func WriteNibble 

func WriteNibble(w io.Writer, high bool, val uint8) error

WriteNibble writes a 4-bit value into a byte. If high is true, it writes to bits 4-7; otherwise bits 0-3. The other nibble is zero.

func WritePreamble added in v0.39.0 

func WritePreamble(w io.Writer, s *Schema) error

WritePreamble writes the header and schema block for s at the version its content requires (Schema.RequiredFormatVersion). A schema that uses no 0x02 feature produces bytes identical to WriteHeader + WriteSchema.

func WriteSchema 

func WriteSchema(w io.Writer, s *Schema) error

WriteSchema serializes s's schema block to w at the baseline 0x01 layout, to follow a WriteHeader. A schema whose content requires a newer version is refused rather than written in a layout its header would contradict — write it with WritePreamble, which emits a matching header.

The 0x01 schema block: 

u16 field_count
per field:
  u8 type
  u8 nullable (0 or 1)
  u16 name_length + utf8 name
  u32 byte_offset
  u8 bit_position
  u16 csv_column_idx
  u16 description_length + utf8 description
  (if decimal128) u8 precision + u8 scale
  (if categorical) dictionary block

A 0x02 schema block is the 0x01 block followed by the schema extension block (see group_wire.go for the payload): 

u64 extension_length
extension_length bytes of extension payload (tagged sections)

The length prefix is what makes the record region's start derivable without understanding the payload.

func WriteSchemaDoc added in v0.8.0 

func WriteSchemaDoc(w io.Writer, schema *Schema, agg uint64, shardCount uint16) error

WriteSchemaDoc emits a `_schema.pulse` payload: standard Pulse header, standard schema block, then the sharding metadata extension (magic "SHRD" + u64 aggregate_record_count + u16 shard_count, little-endian). The result is a valid single-file Pulse cohort with zero records, so legacy readers that don't know about the SHRD extension can still parse the header + schema and stop at the (empty) record region — the extension bytes live where records would otherwise begin and a header-only reader (`Inspect`) ignores them.

func WriteSetMask added in v0.38.0 

func WriteSetMask(w io.Writer, ft FieldType, m SetMask) error

WriteSetMask writes a wide set field's payload to the record stream. It is the streaming mirror of PutSetMask and the API the uint64 WriteFieldValue points at when it refuses set_u128 / set_u256.

Types

type Archive added in v0.8.0 

type Archive struct {
	// contains filtered or unexported fields
}

Archive is a parsed Pulse shard archive: zip central-directory cache plus the underlying ReaderAt. Callers Open or OpenAt entries by name. The zero value is not usable; construct via OpenArchive.

func OpenArchive added in v0.8.0 

func OpenArchive(r io.ReaderAt, size int64) (*Archive, error)

OpenArchive parses the zip central directory at the tail of r and returns an Archive ready for entry enumeration. The ReaderAt is retained — the caller is responsible for its lifetime. Returns PULSE_COHORT_COMPRESSED when r is a zstd transfer artifact (every archive entry point — shard verify / compact / remove / add / list / extract — routes through here, so none of them misreports "decompress first" as a bad zip magic), PULSE_ARCHIVE_MAGIC_INVALID when r does not start with the zip magic, and PULSE_ARCHIVE_CORRUPT when the EOCD or central directory cannot be parsed.

func (*Archive) Entries added in v0.8.0 

func (a *Archive) Entries() []ArchiveEntry

Entries returns every entry in central-directory order (equals shard insertion order for Pulse-written archives). The slice is a fresh copy; callers may mutate freely.

func (*Archive) IsStored added in v0.8.0 

func (a *Archive) IsStored(name string) bool

IsStored reports whether the named entry uses store-only (Method 0) compression. Pulse-written shards are always store-only; this helper is exposed for callers that want to confirm the contract before using OpenAt on a section reader.

func (*Archive) Open added in v0.8.0 

func (a *Archive) Open(name string) (io.ReadCloser, error)

Open returns an io.ReadCloser over the named entry's payload. Returns PULSE_SHARD_MISSING when no entry of that name exists in the central directory. Suitable for streaming reads; the closer must be drained or closed when done.

func (*Archive) OpenAt added in v0.8.0 

func (a *Archive) OpenAt(name string) (io.SectionReader, error)

OpenAt returns an io.SectionReader covering the named entry's stored payload. Only safe for store-only (Method 0) entries — Pulse always writes Method 0, but third-party archives that pass through this path may carry deflated entries; the caller should verify the entry's Method (see Archive.IsStored) before consuming the section reader. Returns PULSE_SHARD_MISSING for unknown names.

func (*Archive) PeekShardHeader added in v0.8.0 

func (a *Archive) PeekShardHeader(name string) error

PeekShardHeader reads the first encoding.HeaderSize bytes of the named entry and validates them as a single-file Pulse header (magic + format version). Returns PULSE_SHARD_HEADER_INVALID on mismatch so callers can surface a structured error without parsing the full schema. Used when peeking shard metadata on first read.

func (*Archive) PeekShardRecordCount added in v0.8.0 

func (a *Archive) PeekShardRecordCount(name string) (int64, error)

PeekShardRecordCount opens the named entry, reads its Pulse header and schema block, validates both, and computes the record count from the remaining bytes divided by the schema's per-record size. Used by `Pulse.Open` to populate `ShardEntry.RecordCount` from the per-shard headers (the source of truth — `_schema.pulse`'s AggregateRecordCount is only a sanity check).

Returns PULSE_SHARD_HEADER_INVALID when the shard's bytes are not a valid single-file Pulse cohort (bad magic, unsupported version, truncated header or schema).

type ArchiveEntry added in v0.8.0 

type ArchiveEntry struct {
	// Name is the zip entry name (basename inside the archive). For
	// flat archives this is the shard filename, e.g. "20190101.pulse".
	Name string

	// Size is the uncompressed payload size in bytes.
	Size int64

	// Offset is the byte offset of the payload within the archive.
	Offset int64
}

ArchiveEntry is a single payload inside a Pulse shard archive. Offset is the byte offset of the entry's compressed/stored payload within the archive (the equivalent of zip.File.DataOffset), NOT the offset of the local file header. Pulse stores entries uncompressed (Method 0), so Offset is a direct pointer into the shard's bytes.

type CohesionWarning added in v0.8.0 

type CohesionWarning struct {
	Code    string
	Message string
	Details map[string]any
}

CohesionWarning is a structured non-fatal divergence emitted by the schema-cohesion validators. Its shape mirrors the descriptor envelope's {code, message, details} entries so callers (typically `service/shard add` and `pulse shard verify`) can forward warnings through the standard --json output without reshaping.

encoding/ stays free of descriptor/ imports — this is the local equivalent.

func ValidateStructuralCohesion added in v0.8.0 

func ValidateStructuralCohesion(canonical, incoming *Schema) ([]CohesionWarning, error)

ValidateStructuralCohesion verifies that incoming's structural schema is byte-equal to canonical's. The check is field-count strict and per-field strict on:

  • Name
  • Type byte
  • ByteOffset
  • BitPosition
  • Categorical-width identity (a categorical_u8 cannot become a categorical_u16 across shards — the width is fixed at folder creation)

Descriptions are advisory and may diverge across shards; per-field divergence emits a PULSE_SHARD_DESCRIPTION_DIVERGENCE warning but does NOT fail validation. Any other mismatch returns a coded PULSE_SHARD_SCHEMA_MISMATCH error.

This validator does NOT compare dictionaries — dictionary cohesion is governed by the append-only prefix rule (see ValidateDictPrefixRule).

type ConstantDetector added in v0.39.0 

type ConstantDetector struct {
	// contains filtered or unexported fields
}

ConstantDetector tracks, over a stream of logical rows of an ungrouped schema, which fields still hold the value of the first row. Two cells are the same value iff their on-wire bytes are equal AND their null bits are equal: a field that is null on every row is a (null) constant, a field that is one value except for some nulls is NOT constant. The comparison is conservative on null cells — two nulls whose placeholder bytes differ count as different, which can only forgo an elision, never make one wrong.

func NewConstantDetector added in v0.39.0 

func NewConstantDetector(flat *Schema) (*ConstantDetector, error)

NewConstantDetector prepares a detector for logical rows of flat, an ungrouped schema (pass a grouped schema's Logical() view).

func (*ConstantDetector) ConstantFields added in v0.39.0 

func (d *ConstantDetector) ConstantFields() []int

ConstantFields returns the logical indices of the fields that held one value across every observed row, ascending. Nil before any row: over zero rows no field has a value to be constant at.

func (*ConstantDetector) FirstRow added in v0.39.0 

func (d *ConstantDetector) FirstRow() []byte

FirstRow returns a copy-owned view of the first observed row (nil before any row). It is the value every still-constant field holds.

func (*ConstantDetector) IsNullConstant added in v0.39.0 

func (d *ConstantDetector) IsNullConstant(fi int) bool

IsNullConstant reports whether constant field fi is constant at NULL (as opposed to constant at a value). Meaningful only for a field ConstantFields returned.

func (*ConstantDetector) Observe added in v0.39.0 

func (d *ConstantDetector) Observe(lrow []byte) error

Observe folds one logical row into the detector. A row of the wrong width is ENCODING_INVALID.

func (*ConstantDetector) Rows added in v0.39.0 

func (d *ConstantDetector) Rows() int64

Rows is the number of rows observed.

func (*ConstantDetector) Stride added in v0.39.0 

func (d *ConstantDetector) Stride() int

Stride is the logical row width Observe expects.

type ConstantPlan added in v0.39.0 

type ConstantPlan struct {
	// Spec is the constant group to encode, or nil when nothing is
	// elided (see Skipped).
	Spec *GroupSpec
	// Fields names the elided fields in logical order (Spec.Members).
	Fields []string
	// Retained names a constant field deliberately kept in the row
	// because eliding every field would leave a zero-byte stride, from
	// which no record count can be derived. Empty otherwise.
	Retained string
	// BytesSaved is the net file-size reduction: the per-row bytes
	// removed times the row count, minus the schema-block growth. Zero
	// when nothing is elided.
	BytesSaved int64
	// Skipped says why nothing was elided ("" when Spec is set):
	// "too_few_rows", "no_constant_fields" or "not_smaller".
	Skipped string
}

ConstantPlan is PlanConstantElision's decision.

func PlanConstantElision added in v0.39.0 

func PlanConstantElision(d *ConstantDetector, reserved []string) (*ConstantPlan, error)

PlanConstantElision decides, from a detector that has observed EVERY row of the cohort, which fields to elide as one constant group. reserved names fields already claimed by another group (a field belongs to at most one); they are never elided. The rules:

  • fewer than MinElisionRows rows: nothing is elided;
  • when the candidates would leave no field in the row (every field constant), the LOWEST-index candidate stays in the row, so the physical stride stays positive;
  • the elision must make the file strictly smaller: the per-row bytes removed times the row count must exceed the schema block's growth (the constant group's descriptor and entry, and the extension block framing). A small cohort whose constants cost more to declare than they save is left alone.

It never changes a value: the plan's group is encoded by GroupEncoder, which refuses a constant member that changes on any row.

func PlanConstantElisionFor added in v0.39.0 

func PlanConstantElisionFor(flat *Schema, constant []int, rows int64, reserved []string) (*ConstantPlan, error)

PlanConstantElisionFor is PlanConstantElision for a caller that decided constancy itself over EVERY row of a cohort whose final schema is flat: constant lists the constant fields' logical indices (ascending) and rows the row count. The rules and the arithmetic are PlanConstantElision's — only the row the saving is sized on is a zeroed stand-in, which cannot change a byte count because entries are fixed-width. Import predict uses it: it sees every row but not in the final schema's layout (a later null can still promote a field).

type Decimal128 added in v0.2.0 

type Decimal128 struct {
	// contains filtered or unexported fields
}

Decimal128 is a fixed-point decimal value with up to 38 digits of precision. Logically it is a signed 128-bit two's-complement integer mantissa, paired with a per-field scale that places the implicit decimal point. Internally it is carried as *big.Int (always copied, never aliased) bounded to the decimal128 representable range [-(10^38 - 1), 10^38 - 1].

func DecodeDecimal128 added in v0.2.0 

func DecodeDecimal128(buf [16]byte) Decimal128

DecodeDecimal128 deserializes 16 bytes of two's-complement little-endian integer into a Decimal128. Null state is carried by the per-record bitmap (see encoding.Schema.HasBitmap), so this function does not flag null values.

func NewDecimal128FromBigInt added in v0.2.0 

func NewDecimal128FromBigInt(m *big.Int) (Decimal128, error)

NewDecimal128FromBigInt builds a Decimal128 from a big.Int mantissa. Returns PULSE_DECIMAL_OVERFLOW if |m| >= 10^38.

func NewDecimal128FromInt added in v0.2.0 

func NewDecimal128FromInt(i int64) Decimal128

NewDecimal128FromInt builds a Decimal128 with the given integer mantissa.

func ParseDecimal128 added in v0.2.0 

func ParseDecimal128(s string) (Decimal128, uint8, error)

ParseDecimal128 parses a strict decimal string into a (Decimal128, scale) pair. The scale is inferred from the input — exactly the count of digits after the decimal point. Accepts an optional single leading sign char. Rejects whitespace, thousand separators, scientific notation, currency symbols.

func ReadDecimal128 added in v0.2.0 

func ReadDecimal128(r io.Reader) (Decimal128, error)

ReadDecimal128 reads 16 bytes of decimal128 from r and decodes them. Null state is carried by the per-record bitmap, not by the payload bytes, so this function has no null channel.

func ZeroDecimal128 added in v0.2.0 

func ZeroDecimal128() Decimal128

ZeroDecimal128 returns a Decimal128 with mantissa 0.

func (Decimal128) Add added in v0.2.0 

func (d Decimal128) Add(o Decimal128) (Decimal128, error)

Add returns d + o, treating both at the same scale. The result has the same scale; the caller is responsible for SQL:2016 precision propagation at the schema level.

func (Decimal128) Cmp added in v0.2.0 

func (d Decimal128) Cmp(o Decimal128) int

Cmp compares two Decimal128 values with the same scale. Comparing values at different scales requires the caller to align scales first.

func (Decimal128) Div added in v0.2.0 

func (d Decimal128) Div(o Decimal128, s1, s2, resultScale uint8) (Decimal128, error)

Div divides d by o using banker's rounding to produce a result at scale `resultScale` given the operands at scales s1 and s2. Returns PULSE_DECIMAL_DIVIDE_BY_ZERO when o is zero.

func (Decimal128) FitsPrecision added in v0.2.0 

func (d Decimal128) FitsPrecision(precision uint8) bool

FitsPrecision reports whether the mantissa fits in `precision` digits.

func (Decimal128) Float64 added in v0.2.0 

func (d Decimal128) Float64(scale uint8) float64

Float64 returns the decimal as a float64 at the given scale. Lossy for values that exceed float64 precision; callers preserving precision should keep using Decimal128 directly.

func (Decimal128) Mantissa added in v0.2.0 

func (d Decimal128) Mantissa() *big.Int

Mantissa returns a copy of the unscaled integer mantissa.

func (Decimal128) Mul added in v0.2.0 

func (d Decimal128) Mul(o Decimal128) (Decimal128, error)

Mul returns d * o. The product mantissa is the product of mantissas; callers are responsible for tracking scale propagation (s1 + s2).

func (Decimal128) Rescale added in v0.2.0 

func (d Decimal128) Rescale(sourceScale, targetScale uint8) (Decimal128, error)

Rescale converts d at sourceScale to targetScale using banker's rounding. Returns PULSE_DECIMAL_OVERFLOW if the rescaled mantissa exceeds 10^38-1.

func (Decimal128) Sign added in v0.2.0 

func (d Decimal128) Sign() int

Sign returns -1, 0, or +1 depending on the mantissa sign.

func (Decimal128) Sqrt added in v0.2.0 

func (d Decimal128) Sqrt(sourceScale, targetScale uint8) (Decimal128, error)

Sqrt returns floor-banker-rounded sqrt(d) at the target scale, given the source value at sourceScale. The result is computed entirely in decimal arithmetic via big.Int.Sqrt and a half-to-even rounding step on the residual. Returns PULSE_DECIMAL_OVERFLOW if intermediate state cannot fit in the integer representation; returns PROCESSING_RUNTIME for negative inputs (sqrt of a negative decimal is undefined).

func (Decimal128) String added in v0.2.0 

func (d Decimal128) String(scale uint8) string

String renders the decimal at the given scale. Trailing zeros are preserved; the leading sign is included only for negative values.

func (Decimal128) Sub added in v0.2.0 

func (d Decimal128) Sub(o Decimal128) (Decimal128, error)

Sub returns d - o.

type DecodeFields added in v0.17.0 

type DecodeFields struct {
	Fields  []*Field
	Indices []int
}

DecodeFields runs the existing per-field decoder loop for the listed fields together. The Fields slice is in schema order; the caller walks it exactly as the unprojected decoder would, so bit-packed neighbours (which on-wire each consume one byte via ReadBit / ReadNibble) stay correctly aligned.

Fields is non-empty for any segment the builder emits.

Indices is parallel to Fields: Indices[k] is the position of Fields[k] in Schema.Fields, recorded by BuildDecodePlan's own schema walk. It is what lets the reuse decoder key an IndexedReusableRecord write by schema position across SkipBytes gaps without a name lookup. A hand-built segment that omits it (len(Indices) != len(Fields)) still decodes: the reader recovers the positions on a slow path.

type DecodePlan added in v0.17.0 

type DecodePlan struct {
	Segments []Segment
}

DecodePlan is a pre-computed walk of one record's on-wire bytes that elides ranges the caller will not consume. It is produced by Schema.BuildDecodePlan given a retained-field set, and consumed by the streaming iterator so unprojected byte ranges turn into a single advance-cursor step instead of a per-field typed read.

A DecodePlan is a pure function of (Schema, retained-set). It carries no file content, no offsets into a particular reader, and no mutable state. Two calls with the same inputs return byte-equivalent Segments in identical order.

type DedupGate added in v0.39.0 

type DedupGate struct {
	// RatioFloor is the rows-per-tuple floor below which a group draws
	// PULSE_DEDUP_LOW_RATIO. Zero, negative or NaN selects
	// DefaultDedupRatioFloor. A floor in (0, 1] disables the
	// floor — every ratio is ≥ 1 — leaving only the grows-the-file
	// check.
	RatioFloor float64
	// Strict returns the first finding as a fatal error with the same
	// code instead of a warning.
	Strict bool
}

DedupGate is the per-group viability policy. The zero value is the default: DefaultDedupRatioFloor, findings as warnings.

func (DedupGate) AssessRatios added in v0.39.0 

func (gt DedupGate) AssessRatios(grouped *Schema, specs []GroupSpec, rows int64) ([]GroupViability, []*errors.CodedError, error)

AssessRatios applies the ratio floor to the first len(specs) groups of grouped, a schema whose dictionaries hold rows records (the schema the encoder produced). specs are the admitted specs in group order — used only for labels. It returns one GroupViability per spec and one PULSE_DEDUP_LOW_RATIO warning per finding; under Strict the first finding is returned as an error instead. With rows == 0 nothing is measured and nothing is flagged. Constant groups are reported admitted, unjudged.

func (DedupGate) Floor added in v0.39.0 

func (gt DedupGate) Floor() float64

Floor returns the effective ratio floor.

func (DedupGate) ScreenWidths added in v0.39.0 

func (gt DedupGate) ScreenWidths(flat *Schema, specs []GroupSpec) ([]GroupSpec, []GroupViability, []*errors.CodedError, error)

ScreenWidths applies the width floor to specs over flat, an ungrouped schema, before any row is read. It returns the specs that survive (in order, each with Ordinal set to its 1-based position in specs unless already set, so labels do not shift when an earlier group is dropped), one GroupViability per INPUT spec (dropped ones carry their widths; admitted ones are completed by AssessRatios), and one PULSE_GROUP_TOO_NARROW warning per dropped group. Under Strict the first narrow group is returned as an error instead. Constant specs pass through unjudged. An unknown member name is PULSE_GROUP_FIELD_UNKNOWN (NewGroupEncoder is the full declaration check; callers normally run it first).

type DictRemap added in v0.8.0 

type DictRemap = map[uint32]uint32

DictRemap is the per-field index translation produced by MergeDictUnion. The key is the incoming dictionary index; the value is the corresponding canonical (union) dictionary index. Fields whose incoming dictionary is already a prefix of the canonical (union) dictionary are absent from the returned map — callers can skip remapping their record bytes.

type Dictionary 

type Dictionary struct {
	// contains filtered or unexported fields
}

Dictionary maps string values to sequential uint32 IDs. It is used for categorical field types to encode string categories as compact integer IDs in the binary record format.

func NewDictionary 

func NewDictionary() *Dictionary

NewDictionary creates an empty dictionary.

func (*Dictionary) Add 

func (d *Dictionary) Add(s string) (uint32, error)

Add inserts a string into the dictionary, returning its ID. If the string already exists, the existing ID is returned. There is no capacity limit with Add; use AddWithLimit to enforce one.

func (*Dictionary) AddWithLimit 

func (d *Dictionary) AddWithLimit(s string, maxEntries uint32) (uint32, error)

AddWithLimit inserts a string, enforcing a maximum entry count. Returns PULSE_IMPORT_CATEGORICAL_OVERFLOW if the dictionary is full.

func (*Dictionary) Count 

func (d *Dictionary) Count() int

Count returns the number of entries in the dictionary.

func (*Dictionary) IDFor 

func (d *Dictionary) IDFor(s string) (uint32, bool)

IDFor looks up the ID for a string. Returns the ID and true if found, or 0 and false otherwise.

func (*Dictionary) ReadFrom 

func (d *Dictionary) ReadFrom(r io.Reader) (int64, error)

ReadFrom deserializes a dictionary from r, replacing current contents. Format: u32 count + (u16 strlen + utf8 bytes) x count

Performance: a single byte buffer is grown to hold all string payloads across the dictionary, instead of allocating one []byte per entry. Each resolved string is copied once via the standard string([]byte) conversion (no unsafe), so we drop one allocation per entry while preserving the safety contract that callers can mutate the underlying buffer afterwards without affecting the stored strings.

func (*Dictionary) Resolve 

func (d *Dictionary) Resolve(id uint32) string

Resolve returns the string for a given ID. Returns "" if the ID is out of range.

func (*Dictionary) Values 

func (d *Dictionary) Values() []string

Values returns a copy of all dictionary values in insertion order.

func (*Dictionary) WriteTo 

func (d *Dictionary) WriteTo(w io.Writer) (int64, error)

WriteTo serializes the dictionary to w. Format: u32 count + (u16 strlen + utf8 bytes) x count

type Field 

type Field struct {
	Name         string
	Type         FieldType
	Nullable     bool // true ⇒ field participates in per-record null bitmap
	ByteOffset   int
	BitPosition  int
	CsvColumnIdx int
	Description  string      // empty = synthesized at inspect time
	Dictionary   *Dictionary // non-nil only for categorical types

	// Precision is the decimal128 precision (1-38). Meaningful only when
	// Type is FieldTypeDecimal128.
	Precision uint8
	// Scale is the decimal128 scale (0-Precision). Meaningful only when
	// Type is FieldTypeDecimal128.
	Scale uint8
}

Field describes a single column in a .pulse schema.

type FieldFilter added in v0.8.0 

type FieldFilter func(name string) bool

FieldFilter returns true for field names whose values should be written into the caller's maps. Used by ReadRecordWithWideProjected to skip map writes for fields the request doesn't read. A nil FieldFilter is equivalent to "keep every field."

type FieldType 

type FieldType byte

FieldType identifies the data type stored in a schema field. 

const (
	FieldTypeU8             FieldType = iota // 0
	FieldTypeU16                             // 1
	FieldTypeU32                             // 2
	FieldTypeU64                             // 3
	FieldTypeF32                             // 4
	FieldTypeF64                             // 5
	FieldTypeU4                              // 6  (4-bit, bit-packed)
	FieldTypeDate                            // 7
	FieldTypePackedBool                      // 8  (1-bit, bit-packed)
	FieldTypeCategoricalU8                   // 9
	FieldTypeCategoricalU16                  // 10
	FieldTypeCategoricalU32                  // 11
	FieldTypeDecimal128                      // 12
	FieldTypeSetU8                           // 13 (bitmask over shared dict, ≤8 members)
	FieldTypeSetU16                          // 14 (≤16 members)
	FieldTypeSetU32                          // 15 (≤32 members)
	FieldTypeSetU64                          // 16 (≤64 members)
	FieldTypeDateTime                        // 17 (epoch seconds, 8-byte u64)
	FieldTypeSetU128                         // 18 (16-byte bitmask, ≤128 members)
	FieldTypeSetU256                         // 19 (32-byte bitmask, ≤256 members)

)

All field types supported by the .pulse format. Nullability is orthogonal to type — any field can be marked nullable via encoding.Field.Nullable and the per-record null bitmap carries the actual null state.

func ParseFieldType added in v0.15.0 

func ParseFieldType(name string) (FieldType, bool)

ParseFieldType inverts FieldType.String, returning the typed value and ok=true when name matches a registered FieldType. Caller paths (managed-imports sidecar parsing, MCP schema validation, ad-hoc admin tooling) use this to round-trip an externally supplied type name through the codec layer without relying on stringly typed switches in every place.

func SetLadder added in v0.38.0 

func SetLadder() []FieldType

SetLadder returns the set-width ladder narrowest rung first. The returned slice is a fresh copy; callers may mutate it freely.

func SetTypeFor added in v0.38.0 

func SetTypeFor(elements int) (FieldType, bool)

SetTypeFor returns the narrowest set_* type with a bit for each of `elements` elements, and reports false when no rung is wide enough (or when `elements` is not positive — a set with no elements has no type, and that is a caller error rather than a width verdict).

func WidestSetType added in v0.38.0 

func WidestSetType() FieldType

WidestSetType returns the top rung of the ladder — the widest set_* type Pulse has. Callers that must NAME the ceiling in a diagnostic use this rather than writing "set_u256" into a string, so the message cannot outlive the type it names.

func (FieldType) ByteSize 

func (ft FieldType) ByteSize() int

ByteSize returns the number of bytes this field type occupies in a record. Bit-packed types (U4, PackedBool) share bytes with adjacent fields and return 0 here; their on-wire stride is handled by Schema.RecordByteSize.

func (FieldType) HasDictionary added in v0.15.0 

func (ft FieldType) HasDictionary() bool

HasDictionary reports whether the field type carries an inline dictionary block in the schema (categorical_* or set_*).

func (FieldType) IsBitPacked added in v0.9.0 

func (ft FieldType) IsBitPacked() bool

IsBitPacked reports whether the field type shares its on-wire byte with adjacent fields via bit packing (U4, PackedBool). Used by stride math and the schema layout pass to advance the bit cursor instead of the byte cursor.

func (FieldType) IsCategorical 

func (ft FieldType) IsCategorical() bool

IsCategorical reports whether the field type is one of the categorical types.

func (FieldType) IsDecimal added in v0.2.0 

func (ft FieldType) IsDecimal() bool

IsDecimal reports whether the field type is decimal128.

func (FieldType) IsKnown added in v0.2.0 

func (ft FieldType) IsKnown() bool

IsKnown reports whether the byte value corresponds to a registered type. Used by the schema reader to reject files written by a future binary version that introduces unknown type bytes.

func (FieldType) IsNumeric added in v0.7.2 

func (ft FieldType) IsNumeric() bool

IsNumeric reports whether the field type is a strict scalar number: the unsigned-integer family (u8/u16/u32/u64), the float family (f32/f64), and decimal128. Date, datetime, and bit-packed integer encodings are excluded — see IsNumericForAnalytics for the analytics-layer predicate.

func (FieldType) IsNumericForAnalytics added in v0.7.2 

func (ft FieldType) IsNumericForAnalytics() bool

IsNumericForAnalytics reports whether the field type carries a meaningful scalar value for numeric analytics (regression, sum/avg/stddev/min/max/ variance aggregators). The set is broader than IsNumeric: bit-packed integer encodings (u4, packed_bool) plus date (epoch days) and datetime (epoch seconds) are included because their stored representation is an ordinal / cardinal number the analytics layer can average, sum, or regress without an explicit ATTR_FORMULA cast.

Null exclusion is the reader's responsibility: the per-record null bitmap marks any field index as null at decode time so the downstream Record.NumericValue contract (returns ok=false on null) keeps the aggregation denominator honest.

func (FieldType) IsSet added in v0.15.0 

func (ft FieldType) IsSet() bool

IsSet reports whether the field type is a bitmask-over-dictionary set (multi-select) type. Set fields share the categorical dictionary block shape but the on-wire payload is a fixed-width bitmask whose bit i corresponds to dictionary entry i.

The narrow rungs (set_u8..set_u64) fit a uint64 and ride the scalar value API; the wide rungs (set_u128, set_u256) are 16- and 32-byte little-endian bitmasks that do NOT fit a uint64 and are rejected by Read/WriteFieldValue — see IsWideSet.

func (FieldType) IsWideSet added in v0.38.0 

func (ft FieldType) IsWideSet() bool

IsWideSet reports whether the field type is a set rung whose bitmask is too wide for the uint64 scalar value API (set_u128, set_u256). Callers that hand a set value through Read/WriteFieldValue must branch on this: the uint64 path would silently truncate the mask to its low 64 bits, dropping selections without any error.

func (FieldType) MaxCategoricalEntries 

func (ft FieldType) MaxCategoricalEntries() uint32

MaxCategoricalEntries returns the maximum dictionary size for a categorical type. Returns 0 for non-categorical types.

func (FieldType) MaxDictEntries added in v0.15.0 

func (ft FieldType) MaxDictEntries() uint32

MaxDictEntries returns the capacity of the inline dictionary block for dictionary-bearing types. Categoricals return MaxCategoricalEntries; sets return MaxSetEntries (= bitmask width). Non-dictionary types return 0.

func (FieldType) MaxSetEntries added in v0.15.0 

func (ft FieldType) MaxSetEntries() uint32

MaxSetEntries returns the maximum dictionary size (= bitmask capacity) for a set type. Returns 0 for non-set types.

func (FieldType) String 

func (ft FieldType) String() string

String returns a human-readable name for the field type.

type Fingerprint added in v0.27.0 

type Fingerprint [FingerprintSize]byte

Fingerprint is a raw SHA-256 content-hash digest of the source .pulse file an Index was built from. A later story (index build) computes one from the live cohort's bytes at build time; the lookup path recomputes it and compares against the embedded value to detect a stale index without decoding the cohort's records.

func ComputeFingerprint added in v0.27.0 

func ComputeFingerprint(r io.Reader) (Fingerprint, error)

ComputeFingerprint hashes the full contents of r with SHA-256. Callers pass the raw bytes of the source .pulse file (header + schema + records) — identical content always yields an identical Fingerprint, regardless of platform or filesystem.

type Group added in v0.39.0 

type Group struct {
	Kind GroupKind
	// Members are in strictly ascending Field order.
	Members []GroupMember
	// Entries is the flat dictionary: EntryCount × EntryWidth bytes.
	// Entry e occupies Entries[e*w : (e+1)*w]. Never strings: a
	// categorical member's bytes are its own dictionary ID, and each
	// member field keeps its own Field.Dictionary unchanged.
	Entries []byte
}

Group is one parent-group descriptor plus its dictionary.

func CloneGroups added in v0.39.0 

func CloneGroups(groups []Group) []Group

CloneGroups is cloneGroups for callers outside the package that build a schema copy of their own (the shard-archive canonical seed).

type GroupEncoder added in v0.39.0 

type GroupEncoder struct {
	// contains filtered or unexported fields
}

GroupEncoder turns logical rows (the 0x01 row of an ungrouped schema) into physical rows of the grouped schema, building each group's dictionary as it goes: a row's tuple is looked up, or appended as a new entry. It is the one encode path shared by import and retro-dedup; the dictionary must precede the records on the wire, so callers spool the physical rows and write the preamble (Schema) last.

func NewGroupEncoder added in v0.39.0 

func NewGroupEncoder(flat *Schema, specs []GroupSpec) (*GroupEncoder, error)

NewGroupEncoder prepares an encoder that groups flat (a schema without groups — pass a grouped schema's Logical() view) by specs. Field names are resolved here: an unknown name is PULSE_GROUP_FIELD_UNKNOWN, a field named by two groups PULSE_GROUP_FIELD_CONFLICT (naming both groups by Label), and an empty group, a field named twice in one group or a key naming a non-member PULSE_GROUP_DECLARATION_INVALID. A shape the format cannot hold (e.g. a zero stride) is ENCODING_INVALID.

func (*GroupEncoder) EncodeRow added in v0.39.0 

func (e *GroupEncoder) EncodeRow(dst, lrow []byte) ([]byte, error)

EncodeRow appends the physical row for logical row lrow to dst.

Failures carry details naming the group (index and Label), the offending field and the row index (0-based, counted by this encoder): a constant group whose members change, or a keyed group whose non-key member disagrees with the entry its key already selected, is PULSE_GROUP_MEMBER_NOT_CONSTANT; a dictionary that would outgrow MaxGroupEntries is PULSE_GROUP_ENTRIES_EXHAUSTED (never a wrap). A logical row of the wrong width is ENCODING_INVALID.

func (*GroupEncoder) EncodeStream added in v0.39.0 

func (e *GroupEncoder) EncodeStream(spool io.Writer, lr io.Reader) (int64, error)

EncodeStream encodes every logical row read from lr (a logical record stream of the encoder's input schema, e.g. from NewLogicalStream) and writes the physical rows to spool, in order. It returns the number of rows encoded. The dictionaries precede the records on the wire, so spool is NOT the cohort: once the stream is drained the caller writes WritePreamble(Schema()) and then the spooled bytes. Holding the spool is the caller's choice — memory (DedupCohort) or a temp file (io.DedupJob, whose memory stays O(dictionaries) whatever the cohort size). A failing row leaves spool holding only the rows before it.

func (*GroupEncoder) LogicalStride added in v0.39.0 

func (e *GroupEncoder) LogicalStride() int

LogicalStride is the width of the logical rows EncodeRow takes.

func (*GroupEncoder) PhysicalStride added in v0.39.0 

func (e *GroupEncoder) PhysicalStride() int

PhysicalStride is the grouped schema's RecordByteSize.

func (*GroupEncoder) Rows added in v0.39.0 

func (e *GroupEncoder) Rows() int64

Rows is the number of rows encoded so far.

func (*GroupEncoder) Schema added in v0.39.0 

func (e *GroupEncoder) Schema() *Schema

Schema returns the grouped schema with every dictionary built so far. Its Entries alias the encoder's storage: take it after the last row.

func (*GroupEncoder) Seed added in v0.39.0 

func (e *GroupEncoder) Seed(seed *Schema) error

Seed pre-loads every group dictionary with seed's entries, in order, before any row is encoded: a row whose tuple (or key) is already there gets that entry's index, and a new tuple is appended AFTER them. It is how a shard joins an archive — the archive's canonical dictionary is the seed, so every entry the archive's stored shards reference keeps its index (the union is canonical-first, the prefix rule every shard relies on) and only the arriving shard's rows are renumbered.

seed must declare the encoder's exact group layout — same group count, kinds, members and key flags — with the same entry widths; anything else is ENCODING_INVALID (a layout reconciliation is the caller's job, before the encoder is built). Seeding goes through the same key rule EncodeRow uses, so a keyed group whose seed repeats a key is refused as a corrupt dictionary, and the u32 ceiling applies to the seeded count exactly as to an appended one (PULSE_GROUP_ENTRIES_EXHAUSTED). Seed must be called before the first EncodeRow.

type GroupEntryDecoder added in v0.39.0 

type GroupEntryDecoder struct {
	// contains filtered or unexported fields
}

GroupEntryDecoder writes a chosen subset of one group's members from any entry into a record, with the member geometry resolved once. It is DecodeGroupEntry restricted to the members a per-entry consumer (filter precompute) actually reads, at O(chosen members) per entry instead of O(all members²): DecodeGroupEntry re-derives each member's offset and the entry geometry per member, which dominates on a wide parent block.

func (*GroupEntryDecoder) Decode added in v0.39.0 

func (d *GroupEntryDecoder) Decode(e int, sink IndexedReusableRecord) error

Decode writes the chosen members of entry e into sink with exactly DecodeGroupEntry's side effects for those members.

type GroupIndexHeadroom added in v0.39.0 

type GroupIndexHeadroom struct {
	Group int    `json:"group"`
	Label string `json:"label"`
	// Kind is "indexed" or "constant".
	Kind string `json:"kind"`
	// Entries is the canonical dictionary's entry count.
	Entries int `json:"entries"`
	// Capacity is the entry ceiling: the u32 index space for an indexed
	// group, exactly 1 for a constant group.
	Capacity uint64 `json:"capacity"`
	// Headroom is Capacity - Entries: how many more distinct tuples the
	// archive can absorb. A constant group always reports 0 — the next
	// distinct value promotes it to an indexed group across the archive.
	Headroom uint64 `json:"headroom"`
}

GroupIndexHeadroom reports how much of one parent group's index space the canonical dictionary has consumed. It is the group counterpart of SetWidthHeadroom and exists for the same reason: a union past the u32 index space is fatal (PULSE_SHARD_DICT_WIDTH_OVERFLOW) with nowhere to widen to, and a constant group that an arriving shard disagrees with is promoted to an indexed group by an ARCHIVE-WIDE rewrite — both are foreseeable from here rather than discovered by paying for them.

func GroupIndexHeadroomFor added in v0.39.0 

func GroupIndexHeadroomFor(s *Schema) []GroupIndexHeadroom

GroupIndexHeadroomFor returns one GroupIndexHeadroom per parent group of s, in group order; nil for a schema without groups.

type GroupIndexRecord added in v0.39.0 

type GroupIndexRecord interface {
	SetGroupIndices(schema *Schema, idx []uint32)
}

GroupIndexRecord is an optional extension of IndexedReusableRecord for grouped (0x02) schemas. After the reuse decoders (ReadRecordReused / ReadRecordReusedWithPlan) finish a grouped row, they hand the record the row's dictionary entry per group: idx[g] is the entry of Schema.Groups[g] every member of g carries on this row (0 for a constant group), and schema is the reader's schema — the one whose group dictionaries the entries address. idx is the reader's own buffer — an implementation copies it and must not retain the slice. A nil idx means "no index for this row" (the row was skipped whole by a plan that decodes nothing); an implementation then reports no index, and it must also report none when schema is not the schema it resolves group dictionaries against, since an entry index is meaningless against any other dictionary.

It is how a per-dictionary-entry precompute (filter precompute) learns which entry a row holds without encoding/ importing processing/ and without the precompute reaching into reader internals. An ungrouped schema never calls it; neither do the map decoders.

type GroupKind added in v0.39.0 

type GroupKind uint8

GroupKind is the per-group kind byte of the group descriptor. 

const (
	// GroupKindIndexed is a dictionary of N entries addressed by a u32
	// index carried in every physical row.
	GroupKindIndexed GroupKind = 0
	// GroupKindConstant is a dictionary of exactly ONE entry and no
	// per-row index (index width 0): the members hold the same value —
	// or the same null — on every row. It is how a global-constant field
	// is elided from the row.
	GroupKindConstant GroupKind = 1
)

func (GroupKind) String added in v0.39.0 

func (k GroupKind) String() string

String names the kind as reports print it: "indexed" or "constant".

type GroupMember added in v0.39.0 

type GroupMember struct {
	// Field is the member's position in Schema.Fields.
	Field int
	// Key marks the member as part of the group's declared key: the
	// members whose values determine the tuple. Informational for
	// decode (every member is read from the entry either way); an
	// encoder validates that non-key members are functionally dependent
	// on the key, and a future precompute may rely on key members
	// being a bijection with the entry index.
	Key bool
}

GroupMember names one member of a group by its LOGICAL field index.

type GroupSpec added in v0.39.0 

type GroupSpec struct {
	// Kind is GroupKindIndexed (a dictionary + a u32 per row) or
	// GroupKindConstant (one entry, nothing per row).
	Kind GroupKind
	// Members names the member fields, in any order.
	Members []string
	// Key optionally names the subset of Members whose values determine
	// the tuple. When set, the encoder REFUSES a row whose non-key
	// members disagree with an earlier row carrying the same key — a
	// declaration that is not actually a parent group is an error, not
	// a silently larger dictionary. Empty means every member is key.
	Key []string
	// Ordinal, when > 0, is the 1-based position Label reports instead
	// of the spec's position in the list it was passed in. A caller
	// that drops a declared group before encoding (the viability gate)
	// sets it so the surviving groups keep the names the user declared
	// them under. Never written to the file.
	Ordinal int
}

GroupSpec declares one parent group by member field NAME.

func GroupSpecsOf added in v0.39.0 

func GroupSpecsOf(s *Schema) []GroupSpec

GroupSpecsOf returns the name-level declaration of every group of s, in group order — the specs a GroupEncoder needs to reproduce s's exact group layout (members, kinds, key flags) over s.Logical().

func (GroupSpec) Label added in v0.39.0 

func (sp GroupSpec) Label(g int) string

Label is the display name error details and reports use for spec g (0-based position in the spec list). The format has no group-name slot, so a group is named by its 1-based declaration position and its key (or, with no key, its members): "group 2 [key: order_id]".

type GroupViability added in v0.39.0 

type GroupViability struct {
	// Group is the 0-based position in the spec list given to the gate.
	Group int `json:"group"`
	// Label names the group ("group 2 [key: order_id]").
	Label string `json:"label"`
	// Verdict is one of the GroupVerdict* constants.
	Verdict string `json:"verdict"`
	// Reason is the coded finding behind a non-admitted verdict
	// (PULSE_GROUP_TOO_NARROW or PULSE_DEDUP_LOW_RATIO); empty when
	// admitted.
	Reason string `json:"reason,omitempty"`
	// Rows is the record count the ratio was measured over (0 for a
	// group dropped before the row pass).
	Rows int64 `json:"rows"`
	// EntryCount is the number of distinct tuples (0 when dropped).
	EntryCount int `json:"entry_count"`
	// EntryWidth is the bytes per dictionary entry.
	EntryWidth int `json:"entry_width"`
	// MemberRowBytes is the bytes the members occupy in a flat row.
	MemberRowBytes int `json:"member_row_bytes"`
	// IndexWidth is the bytes of the per-row index replacing them.
	IndexWidth int `json:"index_width"`
	// DictionaryBytes is EntryCount × EntryWidth: what the group holds
	// RESIDENT in memory for as long as the cohort is open.
	DictionaryBytes int64 `json:"dictionary_bytes"`
	// DescriptorBytes is the group's descriptor in the schema block,
	// dictionary excluded.
	DescriptorBytes int `json:"descriptor_bytes"`
	// Ratio is Rows ÷ EntryCount (0 when not measured).
	Ratio float64 `json:"ratio"`
	// BreakEvenRatio is the ratio at which the dictionary exactly eats
	// the per-row saving: EntryWidth ÷ (MemberRowBytes − IndexWidth).
	// 0 when the group is too narrow to ever break even.
	BreakEvenRatio float64 `json:"break_even_ratio"`
	// RatioFloor is the floor the ratio was judged against.
	RatioFloor float64 `json:"ratio_floor"`
	// UndedupedBytes is Rows × MemberRowBytes: the members stored flat.
	UndedupedBytes int64 `json:"undeduped_bytes"`
	// DedupedBytes is Rows × IndexWidth + DictionaryBytes +
	// DescriptorBytes: the same information grouped.
	DedupedBytes int64 `json:"deduped_bytes"`
	// ByteDelta is DedupedBytes − UndedupedBytes: negative is a saving,
	// zero or positive means grouping made the file no smaller.
	ByteDelta int64 `json:"byte_delta"`
}

GroupViability is one group's gate verdict and the numbers behind it. Every byte figure is exact for the group in isolation; the per-row null bitmap is left out (moving a nullable member out of the row can shrink it by at most one byte per row, so ByteDelta can only understate a saving, never invent one).

func AssessGroup added in v0.39.0 

func AssessGroup(grouped *Schema, g int, rows int64, floor float64) GroupViability

AssessGroup measures group g of grouped over rows records against floor, without judging it (Verdict is admitted, Label empty). Any caller that reports ratio figures — import predict, retro-dedup — derives them here so the arithmetic exists once.

func (GroupViability) Finding added in v0.39.0 

func (v GroupViability) Finding() (low, grows bool)

Finding applies the ratio-floor rule to an assessed indexed group: low is true when the ratio is below RatioFloor or the group makes the file no smaller (grows). It is the ONE judgement AssessRatios and a read-only report (inspect) share, so the two cannot disagree about which group is low-ratio. A constant group, or one measured over zero rows, is never low.

func (GroupViability) Viable added in v0.39.0 

func (v GroupViability) Viable() bool

Viable reports whether the group was written without a finding.

type Index added in v0.27.0 

type Index struct {
	Fingerprint Fingerprint
	Keys        []IndexKeySpec
	Buckets     []IndexBucket

	// SourceSize is the byte length of the source .pulse file, as
	// reported by the filesystem at build time. Paired with
	// SourceModTime to give a freshness check (Service.VerifyIndex) a
	// cheap fast-path that avoids re-hashing the whole cohort: if
	// either value no longer matches the current file's stat, the
	// cohort has definitely changed and the index is definitely stale
	// — no need to pay for a full content hash to know that. A
	// matching size+mtime pair is NOT by itself sufficient proof of
	// freshness (mtime resolution and pathological same-size rewrites
	// both make silent false negatives possible), so a match always
	// falls through to a full Fingerprint recompute for a conclusive
	// answer. See Service.Lookup / Service.VerifyIndex.
	SourceSize uint64

	// SourceModTime is the source .pulse file's modification time, as
	// Unix nanoseconds, at build time. See SourceSize's doc comment for
	// the fast-path contract this pairs with.
	SourceModTime int64
}

Index is the full in-memory representation of a point-lookup sidecar index: the .pulse Fingerprint it was built from, the ordered key spec, the fixed-size hash-bucket table (len(Buckets) is the table's bucket count), and a source-stat snapshot (SourceSize / SourceModTime) taken at build time. Encoding/decoding here is pure codec — populating an Index from a live cohort and serving lookups against one are later stories' responsibility; this package stays free of service/processing imports.

func ReadIndex added in v0.27.0 

func ReadIndex(r io.Reader) (*Index, error)

ReadIndex deserializes a sidecar Index from r. Returns ENCODING_INVALID on a truncated/corrupt header, an unknown key FieldType byte, or a truncated body. A full read — every bucket's every entry — for callers that genuinely need the whole index (e.g. Service.ListIndexes' distinct-key / indexed-record summary). Callers that only need metadata should prefer ReadIndexMeta; callers that only need one bucket should prefer ReadBucketByKey.

func ReadIndexFile added in v0.27.0 

func ReadIndexFile(fsys afero.Fs, path string) (*Index, error)

ReadIndexFile opens path on fsys and deserializes a sidecar Index.

type IndexBucket added in v0.27.0 

type IndexBucket struct {
	Entries []IndexEntry
}

IndexBucket is one slot of the on-disk hash table. A well-distributed build populates each bucket with zero or one entries on average; more than one entry means a hash collision on BucketIndex, resolved by the reader scanning Entries for a byte-equal Key.

func ReadBucketByKey added in v0.27.0 

func ReadBucketByKey(rs io.ReadSeeker, meta *IndexMeta, key []byte) (*IndexBucket, error)

ReadBucketByKey seeks directly to and parses exactly ONE bucket of a v3 sidecar index — the bucket key hashes to via BucketIndex(key, meta.BucketCount) — never reading the bucket-offset table's other entries or any other bucket's data. meta must have been produced by ReadIndexMeta (or ReadIndexMetaFile) against the SAME sidecar rs reads from; rs's absolute byte offset 0 must correspond to the start of that sidecar (a freshly opened file handle, or one explicitly Seek'd back to 0), since meta.OffsetTableStart is an absolute file offset.

Returns an empty *IndexBucket (no error) when meta.BucketCount == 0 (an empty index has no buckets to seek into — mirrors the read side's existing empty-index handling). Returns ENCODING_INVALID on a seek failure, a truncated/corrupt bucket-offset entry, an offset that decodes to a value larger than a signed 64-bit byte offset can represent, or a truncated/corrupt bucket-data block at the resolved position.

type IndexEntry added in v0.27.0 

type IndexEntry struct {
	Key    []byte
	RowIDs []uint64
}

IndexEntry is one hash-bucket slot's payload: the exact on-wire key bytes (the concatenation of every key field's on-wire representation, in key-spec order) plus every row-id sharing that key. RowIDs holds more than one value whenever the source cohort has duplicate key values — the sidecar format is multimap-capable from day one so a later epic's multiplicity work needs no format change.

type IndexKeySpec added in v0.27.0 

type IndexKeySpec struct {
	Name string
	Type FieldType
}

IndexKeySpec describes one ordered key column carried in the sidecar index's key-spec block: the schema field name and its on-wire FieldType. Composite keys (more than one IndexKeySpec) concatenate every key field's on-wire byte representation, in key-spec order, to form a single IndexEntry.Key.

type IndexManifest added in v0.35.0 

type IndexManifest struct {
	FormatVersion string               `json:"format_version"`
	Kind          string               `json:"kind"`
	Cohort        string               `json:"cohort"`
	Indexes       []IndexManifestEntry `json:"indexes"`
}

IndexManifest is the catalog document itself. Cohort carries the cohort's BASENAME for the same location-independence reason IndexManifestEntry.IndexPath does.

func NewIndexManifest added in v0.35.0 

func NewIndexManifest(cohortPath string) *IndexManifest

NewIndexManifest returns an empty, well-formed manifest for cohortPath.

func ReadIndexManifest added in v0.35.0 

func ReadIndexManifest(fsys afero.Fs, cohortPath string) (*IndexManifest, bool, error)

ReadIndexManifest loads the sidecar index manifest for cohortPath. The boolean reports PRESENCE: false with a nil error means no manifest exists, which is not an error — a cohort whose indexes were built by a Pulse that predates the manifest legitimately has none, and Service.ListIndexes falls back to the directory listing there.

A manifest that EXISTS but cannot be trusted is refused with PULSE_INDEX_MANIFEST_INVALID naming the path: malformed JSON, a foreign Kind, an unrecognised FormatVersion, or an entry with no index_path. The alternative — degrading to the directory listing — would succeed on a local disk and answer "no indexes" on a bucket, which is the same silent, backend-dependent divergence this whole mechanism exists to remove.

func (*IndexManifest) Remove added in v0.35.0 

func (m *IndexManifest) Remove(indexBase string) bool

Remove drops the entry for the given sidecar basename, reporting whether one was present.

func (*IndexManifest) Upsert added in v0.35.0 

func (m *IndexManifest) Upsert(entry IndexManifestEntry)

Upsert records entry, replacing any earlier entry for the same sidecar basename, and keeps Indexes sorted by basename so the document is byte-stable across rebuilds regardless of the order indexes were built in.

type IndexManifestEntry added in v0.35.0 

type IndexManifestEntry struct {
	IndexPath      string             `json:"index_path"`
	KeyHash        string             `json:"key_hash"`
	Keys           []IndexManifestKey `json:"keys"`
	DistinctKeys   int                `json:"distinct_keys"`
	IndexedRecords int                `json:"indexed_records"`
}

IndexManifestEntry describes one sidecar index. IndexPath is the sidecar's BASENAME, not a full path, so the document survives the cohort tree being moved, copied or served under a different prefix — a manifest holding absolute paths would be wrong the first time a corpus was synced anywhere.

DistinctKeys / IndexedRecords are recorded at build time so a listing served from this manifest needs no read of any sidecar. They are facts about a deterministic rebuild artifact (BuildIndex over unchanged content produces a byte-identical sidecar), not a freshness claim: freshness lives in each sidecar's own Fingerprint and source-stat snapshot, and nothing here is consulted for it.

func IndexManifestEntryFor added in v0.35.0 

func IndexManifestEntryFor(indexPath string, idx *Index) IndexManifestEntry

IndexManifestEntryFor builds the manifest entry describing idx, the index just written to indexPath. Counts come from the index's own bucket table rather than from a re-scan of the cohort.

func (IndexManifestEntry) KeyNames added in v0.35.0 

func (e IndexManifestEntry) KeyNames() []string

KeyNames returns the entry's ordered key column names — the tuple a caller passes to Lookup / BuildIndex / VerifyIndex, in the order those calls require.

type IndexManifestKey added in v0.35.0 

type IndexManifestKey struct {
	Name string `json:"name"`
	Type string `json:"type"`
}

IndexManifestKey is one key column in a manifest entry's ordered key tuple. Type carries FieldType.String() — the manifest reports the type as well as the name because a caller composing a lookup key has to render its value as a literal, and the per-type literal rules (a categorical's dictionary entry, a datetime's parse form) differ.

type IndexMeta added in v0.27.0 

type IndexMeta struct {
	Fingerprint Fingerprint
	Keys        []IndexKeySpec

	SourceSize    uint64
	SourceModTime int64

	// BucketCount is the sidecar's hash-table bucket count — the same
	// value BucketIndex(key, bucketCount) needs to resolve a lookup key
	// to a bucket slot.
	BucketCount uint32

	// OffsetTableStart is the absolute byte offset, from the start of
	// the sidecar file, where the fixed-width bucket-offset table
	// begins. See the type doc comment for the full seek-math contract.
	OffsetTableStart int64
}

IndexMeta is the cheap, bucket-data-free subset of a sidecar index: everything ReadIndexMeta can read with one short linear pass — header, fingerprint, key-spec, source-stat snapshot, and bucket count — plus the two byte-offset anchors a seekable reader needs to jump straight to one bucket's data without reading the offset table or any other bucket:

  • OffsetTableStart: the absolute byte offset (from the start of the sidecar) where the fixed-width bucket-offset table begins. Entry i of that table lives at OffsetTableStart + i*8.
  • BucketDataStart (a derived method, not a stored field): the absolute byte offset where the bucket-data region begins, immediately after the offset table (OffsetTableStart + bucketCount*8). Every bucket-offset table entry is a byte offset RELATIVE to this anchor, so bucket i's absolute data position is BucketDataStart() + offsets[i].

Staleness checks (Service.VerifyIndex) and arity checks (matching a lookup request's key-component count against the sidecar's own key-spec) only ever need this struct — never bucket data.

func ReadIndexMeta added in v0.27.0 

func ReadIndexMeta(r io.Reader) (*IndexMeta, error)

ReadIndexMeta deserializes only the bucket-data-free prefix of a sidecar index from r: header, fingerprint, key-spec, source-stat snapshot, and bucket count — never touching the bucket-offset table or any bucket data. r must start at byte 0 of the sidecar (the same assumption ReadIndex and ReadIndexHeader already make) so the returned IndexMeta.OffsetTableStart — computed analytically from the exact byte lengths of the sections just read, not by seeking — is a correct absolute file offset.

This is the cheap path Service.VerifyIndex's staleness check and a lookup request's key-arity check both want: neither needs a single bucket, let alone the whole table.

func ReadIndexMetaFile added in v0.27.0 

func ReadIndexMetaFile(fsys afero.Fs, path string) (*IndexMeta, error)

ReadIndexMetaFile opens path on fsys and deserializes only the bucket-data-free IndexMeta prefix — the afero-backed convenience counterpart to ReadIndexMeta, mirroring ReadIndexFile's relationship to ReadIndex. Callers that only need staleness/arity metadata (e.g. Service.VerifyIndex) should prefer this over ReadIndexFile.

func (*IndexMeta) BucketDataStart added in v0.27.0 

func (m *IndexMeta) BucketDataStart() int64

BucketDataStart returns the absolute byte offset, from the start of the sidecar file, where the bucket-data region begins — immediately after the fixed-width bucket-offset table. Every bucket-offset table entry is a byte offset relative to this anchor.

type IndexedReusableRecord added in v0.39.0 

type IndexedReusableRecord interface {
	SetNumericAt(idx int, value float64)
	SetNullFieldAt(idx int)
	SetWideFieldAt(idx int, value any)
	ClearForRow()
}

IndexedReusableRecord is the index-keyed sibling of ReusableRecord. It exists for the same reason ReusableRecord does — it lets the decoder populate a *processing.Record without encoding/ importing processing/ — but its writes are keyed by the field's POSITION in the reader's schema (the index into Schema.Fields) rather than by name, so a positional record implementation can store the value with a slice index instead of a map hash.

The position is the decoder's own schema-walk counter, never a name→index lookup: ReadRecordReused counts it as it walks Schema.Fields, ReadRecordReusedWithPlan carries it in DecodeFields.Indices (filled by BuildDecodePlan's walk), and the bitmap decoder uses the bitmap bit index, which IS the field position. An implementation therefore MUST be built over the same schema (or a structurally identical one — same field order) the RecordReader was constructed with; the index is meaningless against any other.

Side-effect contract is identical to ReusableRecord, field for field: ClearForRow once per row; a null surfaces as SetNullFieldAt(i) followed by SetNumericAt(i, 0); decimal128 and set_* fields write both the numeric echo and the typed wide value.

When a record implements both interfaces the index-keyed methods win and the name-keyed ones are never called by the reuse decoders. A record that also implements TypedSetRecord receives set masks through it instead of SetWideFieldAt, and one that implements RunSkipRecord gets BeginRunRow in place of ClearForRow and only its changed fields rewritten (reader_runskip.go).

type RecordLocator added in v0.27.0 

type RecordLocator struct {
	// Schema is the parsed schema this locator's geometry is derived
	// from. Callers pass it straight through to ReadRecordWithWidePlan
	// / ReadRecordWithWideProjected via NewRecordReader.
	Schema *Schema

	// RecordRegionStart is the absolute byte offset of the first record,
	// i.e. the number of bytes the header + schema prefix consumed.
	RecordRegionStart int64

	// Stride is Schema.RecordByteSize() — the fixed on-wire byte width
	// of one record (including the trailing null bitmap, if any).
	Stride int64

	// TotalRecords is the number of complete records the payload holds,
	// derived from the payload's total byte length. Zero for an empty
	// cohort or a cohort shorter than one full record stride.
	TotalRecords uint64
}

RecordLocator captures the fixed geometry a single-file .pulse payload needs for O(1) index-addressed record reads: the byte offset where the record region begins (immediately after the header + schema prefix), the fixed per-record stride, and the total record count for bounds checking. Construct one per opened cohort via NewRecordLocator and reuse it across many point lookups — the offset math mirrors service/parallel_decode.go's per-worker segment slicing (recordRegionStart + i*stride), just applied to a single record instead of a worker's contiguous range.

func NewRecordLocator added in v0.27.0 

func NewRecordLocator(r *bytes.Reader, schema *Schema) (*RecordLocator, error)

NewRecordLocator derives record-region geometry for a single-file .pulse payload from r, an io.ReadSeeker over the raw payload bytes — most commonly a *bytes.Reader wrapping a mmap'd byte slice, the same path the streaming iterator and the parallel decode context use. r's cursor must be at absolute position 0 (the very first header byte) when this is called; NewRecordLocator reads (and discards, beyond measuring their length) the header and schema, leaving r positioned at the first record byte as a side effect. Callers that need r re-wound for further use should re-seek to loc.RecordRegionStart or construct a fresh reader over the same bytes.

schema is the already-parsed schema for this cohort (e.g. from pulse.Open) — NewRecordLocator does not re-validate its field contents against the freshly-read schema bytes; it reads the schema block purely to measure how many bytes the header+schema prefix consumes. A mismatched schema silently produces a locator with wrong geometry, so callers must pass the schema that was actually parsed from this same payload.

A *bytes.Reader is required (rather than a generic io.ReadSeeker) so this function can read Size() — the reader's fixed, read-independent total length — to derive TotalRecords without a second file stat.

func (*RecordLocator) Offset added in v0.27.0 

func (loc *RecordLocator) Offset(i uint64) int64

Offset returns the absolute byte offset of record i within the payload this locator was built from. It does not bounds-check — callers that need a guarded lookup should use ReadRecordAt, which validates i < TotalRecords before ever seeking.

func (*RecordLocator) ReadRecordAt added in v0.27.0 

func (loc *RecordLocator) ReadRecordAt(
	r io.ReadSeeker,
	i uint64,
	values map[string]float64,
	nulls map[string]bool,
	wide map[string]any,
	keep FieldFilter,
	plan *DecodePlan,
) error

ReadRecordAt decodes the record at index i directly: it seeks r to loc.Offset(i) and decodes from there, without iterating any preceding record. r must be an io.ReadSeeker positioned over the same underlying bytes the locator was built from (the same *bytes.Reader passed to NewRecordLocator, or a fresh reader over the same byte slice) — the seek is absolute, so r's current cursor position on entry is irrelevant.

plan is optional: pass a *DecodePlan (from Schema.BuildDecodePlan) to decode only the columns the plan retains — the O(1) projection variant required by point-lookup callers that only need a handful of return columns out of a wide schema. Pass nil to decode every field via the existing full-record path. keep mirrors the FieldFilter contract used throughout this package (see ReadRecordWithWidePlan / ReadRecordWithWideProjected) — pass the same filter used to build plan's retained set so map writes for incidental group members (a bit-packed neighbour or a bitmap-adjacent nullable field) stay suppressed; pass nil to keep every field the plan/schema visits.

Returns a coded ENCODING_INVALID error — never a panic, never an out-of-bounds read — when i >= loc.TotalRecords.

type RecordReader 

type RecordReader struct {
	// contains filtered or unexported fields
}

RecordReader reads records one at a time from a binary stream. It reads directly from the io.Reader without buffering the entire file.

func NewRecordReader 

func NewRecordReader(r io.Reader, schema *Schema) *RecordReader

NewRecordReader creates a RecordReader. The reader must be positioned immediately after the header and schema (i.e., at the first record byte).

For a grouped (0x02) schema the reader decodes the LOGICAL record: same field indices, same values, same nulls as the ungrouped twin. The map decoders read r through the logical stream (every physical row expanded into the exact row the twin stores); the reuse decoders (ReadRecordReused*, run-skip included) decode the physical row directly and never expand it (group_decode.go). Exactly one physical row is consumed per record on every path, so r's position after a record is the end of that physical record.

func (*RecordReader) GroupIndex added in v0.39.0 

func (rr *RecordReader) GroupIndex(g int) (uint32, bool)

GroupIndex reports the dictionary entry index group g (a position in the grouped schema's Groups) resolved to on the record just decoded: every member of g carries that entry's values. ok is false for an ungrouped schema, before the first record, or when the last record was skipped whole by a decode plan. It is the row -> entry map a per-entry precompute (filter precompute) tests instead of decoding the members.

func (*RecordReader) ReadRecord 

func (rr *RecordReader) ReadRecord(values map[string]float64, nulls map[string]bool) error

ReadRecord reads a single record from the stream, populating the values and nulls maps. Returns io.EOF when no more records are available.

The caller provides pre-allocated maps to avoid per-record allocation. Maps are cleared at the start of each call.

Reuse contract: the maps are owned by the caller. ReadRecord does not retain references to them after returning. If the caller plans to reuse the same maps across calls (the typical pattern), they must consume the populated values BEFORE invoking ReadRecord again, because the next call clears and repopulates the maps in-place. If the caller needs to retain the values past the next call (e.g., collecting Records into a slice for later aggregation), it must pass distinct map instances per record OR copy the contents out before the next ReadRecord call.

To populate typed wide values for fields whose representation does not fit in float64 (decimal128), call ReadRecordWithWide instead and pass a third map.

func (*RecordReader) ReadRecordReused added in v0.2.0 

func (rr *RecordReader) ReadRecordReused(rec ReusableRecord) error

ReadRecordReused reads one record into an existing ReusableRecord, reusing the record's internal maps. Returns io.EOF when the underlying reader is exhausted.

Hot path semantics:

  • Caller MUST consume the populated rec before the next call.
  • The whole record stride (Schema.RecordByteSize(), including the trailing null bitmap) is read into a reusable per-RecordReader buffer with a SINGLE io.ReadFull, then every field is decoded from a running cursor subslice of that buffer — no per-field read, no copy. This eliminates the per-field io.ReadFull that dominates wide-schema decode.
  • Fields are walked in schema order with a running byte cursor, NOT by Field.ByteOffset (bit-packed layout is cursor-driven; stored offsets are unreliable). Each bit-packed field (u4, packed_bool) occupies a whole on-wire byte, matching ReadBit/ReadNibble semantics.
  • A short read at end-of-stream surfaces io.EOF exactly as before via mapEOF; a partial trailing record surfaces as io.EOF as well (io.ReadFull maps a nonzero short read to io.ErrUnexpectedEOF, which mapEOF normalizes to io.EOF).
  • A record implementing RunSkipRecord is decoded by readStrideRunSkip: same result, but fields whose bytes repeat the previous row are not rewritten.

func (*RecordReader) ReadRecordReusedWithPlan added in v0.26.0 

func (rr *RecordReader) ReadRecordReusedWithPlan(rec ReusableRecord, keep FieldFilter, plan *DecodePlan) error

ReadRecordReusedWithPlan reads one record into an existing ReusableRecord by walking a precomputed DecodePlan, decoding only the fields the plan's DecodeFields segments carry and seeking past the SkipBytes ranges the caller will not consume. It is the plan-aware sibling of ReadRecordReused: same reuse contract (the record's null/wide maps are cleared once via ClearForRow, then populated in place), same io.EOF surfacing, but projection-honoring under reuse.

plan == nil falls back to ReadRecordReused (full-decode reuse path), so an iterator that never installed a plan is byte-identical to today.

keep governs record writes within DecodeFields segments, mirroring ReadRecordWithWidePlan / decodeFieldGroup / decodeBitmap exactly:

  • Bit-packed members (u4, packed_bool) of a retained group are ALWAYS consumed for cursor alignment (1 whole on-wire byte each), but their record writes are suppressed when keep rejects them.
  • The trailing bitmap segment reads the bitmap once and surfaces nulls only for nullable fields the caller retains.

Side effects match ReadRecordReused field-for-field:

  • null → SetNullField(name) + SetNumeric(name, 0);
  • decimal128 → SetNumeric(mean-ish scalar) + SetWideField(Decimal128);
  • set_u8..set_u64 → SetNumeric(float64 echo) + SetWideField(uint64 mask); set_u128 / set_u256 → SetNumeric(low-64-bit echo) + SetWideField(SetMask).

Unlike ReadRecordReused, this path does NOT read the whole record stride: each DecodeFields group reads only its own on-wire bytes into the reusable buffer with a single io.ReadFull, and SkipBytes segments advance the reader past unread ranges with a single Seek (io.CopyN fallback). On a wide schema with a small retained set that is the projection win under reuse.

func (*RecordReader) ReadRecordWithWide added in v0.2.0 

func (rr *RecordReader) ReadRecordWithWide(values map[string]float64, nulls map[string]bool, wide map[string]any) error

ReadRecordWithWide reads a record and populates a wide map with typed values for decimal128 fields. The wide map may be nil to skip wide population.

func (*RecordReader) ReadRecordWithWidePlan added in v0.17.0 

func (rr *RecordReader) ReadRecordWithWidePlan(
	values map[string]float64,
	nulls map[string]bool,
	wide map[string]any,
	keep FieldFilter,
	plan *DecodePlan,
) error

ReadRecordWithWidePlan reads one record by walking a precomputed DecodePlan instead of iterating every schema field. SkipBytes segments advance the underlying reader by N bytes with a single seek (or io.CopyN fallback); DecodeFields segments run the same per-field decode loop the unprojected reader uses, but only for the fields the segment carries.

keep governs map writes within DecodeFields segments. The plan builder emits bit-packed groups together (every member of a group whose retained subset is non-empty), so bit-packed members that the caller did NOT request are still decoded — their on-wire bytes must be consumed to keep the cursor aligned — but their map writes are suppressed by keep. Similarly the trailing bitmap segment carries every nullable field so the iterator can read the bitmap once and surface nulls; keep filters those map writes.

keep == nil and plan == nil both fall back to the existing full-decode behaviour. Passing keep == nil with a non-nil plan widens every DecodeFields segment to "keep all members" (useful for golden parity tests).

Behaviour invariants matched against the per-field readRecord loop:

  • Caller-supplied maps are cleared in place before population.
  • Decimal128 fields populate wide[name] with the typed Decimal128 value (only when wide != nil and the field is retained).
  • Narrow set fields (set_u8..set_u64) populate wide[name] with the raw uint64 mask; the wide rungs (set_u128 / set_u256) populate it with a SetMask. Both only when wide != nil and the field is retained.
  • Null surfacing clears values[name] back to 0 and deletes wide[name] for nullable fields the bitmap marks null AND the caller retains.

func (*RecordReader) ReadRecordWithWideProjected added in v0.8.0 

func (rr *RecordReader) ReadRecordWithWideProjected(values map[string]float64, nulls map[string]bool, wide map[string]any, keep FieldFilter) error

ReadRecordWithWideProjected reads a record but only writes the fields for which keep(name) returns true into the caller's maps. Bytes for excluded fields are still consumed from the underlying reader so byte offsets stay aligned — projection saves map allocations, not decode work.

keep == nil falls back to the full-decode path.

type ReusableRecord added in v0.2.0 

type ReusableRecord interface {
	SetNumeric(name string, value float64)
	SetNullField(name string)
	SetWideField(name string, value any)
	ClearForRow()
}

ReusableRecord is the subset of *processing.Record needed by the reuse fast path. Declared here as an interface so encoding/ does not depend on processing/. Implementations (processing.Record) MUST clear their own null/wide maps before this call returns successfully; the reader populates them in place but only on fields where the value applies.

ReusableRecord is the NAME-keyed contract: every write pays a hash of the field name. It is retained as a compatibility shim — an implementation of only this interface still decodes correctly, routed through nameKeyedShim — but a record that also implements IndexedReusableRecord is driven through the index-keyed sibling instead. The reader methods still take a ReusableRecord so an existing implementation keeps compiling unchanged.

type RowExpander added in v0.39.0 

type RowExpander struct {
	// contains filtered or unexported fields
}

RowExpander turns physical rows of a grouped schema into logical rows: the exact 0x01 row the ungrouped twin stores. Not safe for concurrent use (it caches the last entry spliced per group).

func NewRowExpander added in v0.39.0 

func NewRowExpander(s *Schema) (*RowExpander, error)

NewRowExpander compiles the expansion for a grouped schema.

func (*RowExpander) Expand added in v0.39.0 

func (x *RowExpander) Expand(dst, phys []byte) ([]byte, error)

Expand writes the logical row for physical row phys into dst (grown to LogicalStride) and returns it. An index past its group's dictionary is ENCODING_INVALID. When dst is the same buffer as the previous call, a group whose index did not change is not re-spliced — its member bytes are already in place.

func (*RowExpander) GroupIndex added in v0.39.0 

func (x *RowExpander) GroupIndex(g int) uint32

GroupIndex returns the entry index group g resolved to in the most recent successful Expand (0 for a constant group).

func (*RowExpander) LogicalStride added in v0.39.0 

func (x *RowExpander) LogicalStride() int

LogicalStride is the logical row width: Logical().RecordByteSize().

func (*RowExpander) PhysicalStride added in v0.39.0 

func (x *RowExpander) PhysicalStride() int

PhysicalStride is the grouped schema's RecordByteSize.

type RunSkipRecord added in v0.39.0 

type RunSkipRecord interface {
	IndexedReusableRecord
	// BeginRunRow starts a row for the reader identified by token
	// (never 0). See the type documentation for the keep / return
	// contract.
	BeginRunRow(token uint64, keep bool) bool
	// ClearNullAt clears field idx's null mark, leaving its value and
	// wide value as they are.
	ClearNullAt(idx int)
}

RunSkipRecord is an optional extension of IndexedReusableRecord that lets the reuse decoders skip repopulating fields whose on-wire bytes did not change since the previous row.

On a cohort sorted by a parent key most of a row's fields repeat the previous row byte for byte (a denormalised join holds the whole parent block constant across the parent's child rows). Comparing a field's 1–32 on-wire bytes against the previous row's is far cheaper than decoding them and writing the record, so when the record still holds exactly what the decoder wrote for the previous row, only the fields that changed are rewritten. The optimisation is self-activating: no declaration, no metadata, no format change.

The contract has two halves, one per side:

  • The READER guarantees the comparison baseline: it keeps the previous row's bytes, and passes keep=true to BeginRunRow only when those bytes were decoded, successfully and by this same decode shape (full stride, or the same *DecodePlan), into this same record.
  • The RECORD guarantees its state is still what the decoder left: BeginRunRow returns true only when keep is true, token names the reader that last began a row on it, and nothing but that reader's index-keyed writes has touched a schema field since. Any other mutation of a schema field (Set, SetNull, SetWide, a name-keyed decode write, ClearForRow) must make the next BeginRunRow return false. It must also return false for any storage in which two schema positions share state (a duplicated field name), because a skipped position could then carry its sibling's value.

BeginRunRow replaces ClearForRow on this path. When it returns false the record must have done a full ClearForRow and the decoder writes every field exactly as ReadRecordReused always has. When it returns true the record clears only per-row state the decoder never writes (off-schema null / wide marks, memoised views) and the decoder writes just the changed fields:

  • a field whose value bytes AND null bit are unchanged is not written at all;
  • a field that is non-null and changed (value bytes differ, or it was null on the previous row) gets its full value write — numeric, plus the typed wide value for decimal128 / set_*;
  • a field that turned null gets SetNullFieldAt + SetNumericAt(0), exactly the full path's null signal; one that turned non-null gets ClearNullAt. A field null on both rows is left alone.

Because every write is a function of the field's own bytes and null bit, the record after a partial rewrite is observably identical to a full decode of the same row.

type Schema 

type Schema struct {
	// Fields is the LOGICAL schema: every field, in original order. A
	// field index anywhere in the API is a position in this slice.
	Fields []Field
	// Groups are the parent-group descriptors (format 0x02, see
	// group.go). Empty for every 0x01 cohort. When non-empty, member
	// fields are stored in the groups' dictionaries rather than in the
	// row, and RecordByteSize / BitmapByteSize / HasBitmap describe the
	// PHYSICAL (reduced) row; Logical() is the ungrouped view.
	Groups []Group
}

Schema holds all field descriptors for a .pulse file.

func DedupCohort added in v0.39.0 

func DedupCohort(dst io.Writer, src io.Reader, specs []GroupSpec) (*Schema, int64, error)

DedupCohort reads a whole cohort (header, schema, records — any version, grouped or not) from src and writes its grouped equivalent, grouped by specs, to dst. The source is read through its LOGICAL stream, so a grouped source is regrouped from scratch. The physical rows are spooled in memory because the dictionaries precede the records on the wire — it is the small-cohort form of the one encode path (GroupEncoder.EncodeStream) that io.DedupJob drives with a temp-file spool. A truncated trailing record is ENCODING_INVALID rather than silently dropped. Returns the written schema and the record count.

func FlattenCohortBytes added in v0.39.0 

func FlattenCohortBytes(cohort []byte) ([]byte, *Schema, error)

FlattenCohortBytes returns the ungrouped (0x01) twin of a single-file cohort: the logical schema's preamble followed by every logical row, in order. A cohort without groups is returned unchanged (the same slice). The logical rows are exactly what every decoder yields, so the twin decodes identically; for a cohort deduped from a 0x01 original it reproduces that original byte-for-byte.

A payload ending mid-record is ENCODING_INVALID rather than a silently dropped tail (the walk is ForEachRecord's).

func MergeDictUnion added in v0.8.0 

func MergeDictUnion(canonical, incoming *Schema) (*Schema, map[int]DictRemap, error)

MergeDictUnion is the relaxed dictionary cohesion check. For each categorical field it computes the union of canonical's and incoming's dictionaries: canonical entries first in their existing order, then any new entries from incoming in their order. Returns the extended canonical schema (deep copy when an extension is adopted; identical pointer when not) and a per-field-index map of DictRemap entries describing how to translate incoming record indices into canonical indices.

The structural validator (ValidateStructuralCohesion) must run FIRST — this validator assumes field positions, names, and categorical widths already match.

Errors:

  • PULSE_SHARD_DICT_WIDTH_OVERFLOW when the union would exceed the field's categorical width capacity.

MergeDictUnion is the default cohesion mode at insert time (CreateShardArchive / AddShard). The stricter prefix-only rule (ValidateDictPrefixRule) is retained for callers that want to surface divergence as an error instead of merging — used by VerifyShardArchive when checking archives written before the union-merge default landed and by embedders that prefer to align dictionaries upstream.

func NewLogicalStream added in v0.39.0 

func NewLogicalStream(r io.Reader, s *Schema) (io.Reader, *Schema, error)

NewLogicalStream returns a reader of the logical record stream of s over r (positioned at the first physical record), and the schema that describes that stream (s.Logical()). Byte-level record consumers — a decoder of their own, an exporter — read through it and walk the returned schema exactly as they walk a 0x01 cohort. For a schema without groups it returns (r, s) unchanged.

func ReadPreamble added in v0.39.0 

func ReadPreamble(r io.Reader) (*Schema, byte, error)

ReadPreamble reads the header and the version-matched schema block, leaving r positioned at the first record byte. It is the one-call form of ReadHeader + ReadSchema for callers that do not need to wrap the two failures differently.

func ReadSchema 

func ReadSchema(r io.Reader, v byte) (*Schema, error)

ReadSchema deserializes a schema block written at format version v — the version ReadHeader returned for the same stream. On return r is positioned at the first record byte. An unsupported v is ENCODING_INVALID; unknown field-type bytes fail loud here, at parse time, for every version.

func ValidateDictPrefixRule added in v0.8.0 

func ValidateDictPrefixRule(canonical, incoming *Schema) (*Schema, error)

ValidateDictPrefixRule enforces the append-only prefix rule for every categorical field shared by canonical and incoming. For each `categorical_*` field:

  • If incoming's dictionary is a prefix of canonical's, accept the shard as-is; no canonical change. (Older shard that never saw newer dictionary values.)
  • If canonical's dictionary is a prefix of incoming's, the canonical schema adopts the extension. The returned extendedCanonical carries the merged dictionaries and the caller is responsible for rewriting `_schema.pulse` to publish them before the new shard is placed (crash-safety ordering documented in placeholder §3.2).
  • Neither is a prefix of the other → PULSE_SHARD_DICT_DIVERGENCE.
  • An extension that would exceed the field's width capacity (256 for u8, 65 536 for u16, 2^32 for u32) → PULSE_SHARD_DICT_WIDTH_OVERFLOW.

The structural validator (ValidateStructuralCohesion) must run FIRST — this validator assumes field positions, names, and categorical widths already match.

The returned extendedCanonical is a deep copy when an extension is adopted, and is identically the canonical input otherwise. Callers can compare pointers to detect whether a rewrite is required.

func WidenSchemaSetField added in v0.38.0 

func WidenSchemaSetField(s *Schema, field string, target FieldType) (*Schema, error)

WidenSchemaSetField returns a copy of s in which the named set field carries the wider rung target, with every field's ByteOffset recomputed for the new stride.

It is the SCHEMA half of a widen, split out because the shard auto-widen path has to publish a widened canonical `_schema.pulse` that has no records of its own alongside shard payloads widened by WidenSetFieldBytes. Both go through this one function, so the canonical schema block and every shard's schema block cannot end up describing different layouts — a divergence that would pass ValidateStructuralCohesion nowhere and decode as garbage everywhere.

Names, nullability, descriptions, CSV indices, decimal metadata and the inline dictionary pointer ride across by value, so bit i still means dictionary entry i. The input schema is never modified.

func (*Schema) BitmapByteSize added in v0.9.0 

func (s *Schema) BitmapByteSize() int

BitmapByteSize returns the number of bytes the null bitmap occupies per record, or 0 when no field is nullable. With groups it is the NARROWED bitmap: ceil(row_field_count/8), bit j = the j-th row field (fields in no group, logical order).

func (*Schema) BuildDecodePlan added in v0.17.0 

func (s *Schema) BuildDecodePlan(retained []string) (*DecodePlan, error)

BuildDecodePlan produces a deterministic DecodePlan for this schema given the set of field names the caller intends to consume.

Rules:

  • Contiguous unprojected, non-bit-packed fields coalesce into a single SkipBytes whose N is the sum of those fields' ByteSize().
  • Bit-packed neighbours (any contiguous run of FieldType.IsBitPacked() fields) form one group. The group becomes one DecodeFields segment if any member is retained; otherwise it becomes one SkipBytes whose N equals the group's on-wire byte size — which mirrors Schema.RecordByteSize: 1 byte per bit-packed field. (BitPosition is a per-field annotation; ReadBit / ReadNibble each consume a full byte today, so a run of K bit-packed fields occupies K bytes on-wire.)
  • Trailing null bitmap: if Schema.HasBitmap() and any nullable field is retained, append a DecodeFields segment carrying every nullable field in schema order (the iterator decodes the bitmap, then walks this list to surface null flags). If HasBitmap() but no nullable field is retained, append a SkipBytes{N: Schema.BitmapByteSize()}. If !HasBitmap(), no bitmap segment is emitted.
  • Empty retained set: the plan is a single SkipBytes covering the full record stride (RecordByteSize, which already includes the bitmap if any).
  • Full retained set (every schema field present in retained): no SkipBytes segments — the output is equivalent to the existing per-field walk.

retained is consulted as a set (order does not matter); duplicates and names not present in the schema are silently ignored. The function never returns an error today, but the signature carries one for forward-compatibility with future shape validation (e.g. once extension-driven retained sets carry wildcard tokens).

A grouped (0x02) schema's plan walks its LOGICAL row — the stream a RecordReader over that schema decodes — so it is the plan of Logical(): SkipBytes widths are logical widths, and the empty plan covers the logical stride.

func (*Schema) Categorical 

func (s *Schema) Categorical(name string) (*Dictionary, bool)

Categorical returns the dictionary for a named categorical field. Returns nil, false if the field is not found or is not categorical.

func (*Schema) DecodeGroupEntry added in v0.39.0 

func (s *Schema) DecodeGroupEntry(g, e int, sink IndexedReusableRecord) error

DecodeGroupEntry writes entry e of group g into sink at the members' LOGICAL field indices, with exactly the side effects a full row decode has for those fields (value + typed wide value; a null member gets SetNullFieldAt + SetNumericAt(0)). It does not call ClearForRow and touches no non-member field. It is the per-entry evaluation hook: a predicate over member fields can be evaluated once per entry through a record built over this schema.

func (*Schema) Field 

func (s *Schema) Field(name string) *Field

Field returns a pointer to the named field, or nil if not found.

func (*Schema) FieldGroup added in v0.39.0 

func (s *Schema) FieldGroup(fi int) (g, k int, ok bool)

FieldGroup reports whether logical field fi is a group member and, if so, which group and member slot. It is the entry point for code that wants to work per dictionary entry instead of per row (filter precompute): the member's value in entry e is GroupMemberBytes(g, k, GroupEntry(g, e)).

func (*Schema) GroupEntry added in v0.39.0 

func (s *Schema) GroupEntry(g, e int) []byte

GroupEntry returns entry e of group g (a subslice of Entries — do not mutate it).

func (*Schema) GroupEntryCount added in v0.39.0 

func (s *Schema) GroupEntryCount(g int) int

GroupEntryCount returns the number of entries in group g's dictionary.

func (*Schema) GroupEntryWidth added in v0.39.0 

func (s *Schema) GroupEntryWidth(g int) int

GroupEntryWidth returns the byte width of one entry of group g: the members' on-wire widths plus the member null bitmap when any member is nullable.

func (*Schema) GroupIndexOffset added in v0.39.0 

func (s *Schema) GroupIndexOffset(g int) int

GroupIndexOffset returns the physical byte offset of group g's u32 index within a physical row, or -1 for a constant group (no per-row index).

func (*Schema) GroupMemberBytes added in v0.39.0 

func (s *Schema) GroupMemberBytes(g, k int, entry []byte) []byte

GroupMemberBytes returns member k's on-wire bytes within entry — the same bytes the member occupies in a logical row.

func (*Schema) GroupMemberIsNull added in v0.39.0 

func (s *Schema) GroupMemberIsNull(g, k int, entry []byte) bool

GroupMemberIsNull reports whether member k is null in entry.

func (*Schema) GroupMemberRowBytes added in v0.39.0 

func (s *Schema) GroupMemberRowBytes(g int) int

GroupMemberRowBytes returns the on-wire bytes group g's members occupy in a LOGICAL row, null bits excluded: what each row stops carrying when the group is formed, before an indexed group's GroupIndexWidth is added back. With GroupEntryCount and GroupEntryWidth it is everything a viability check needs.

func (*Schema) GroupSpecOf added in v0.39.0 

func (s *Schema) GroupSpecOf(g int) GroupSpec

GroupSpecOf reconstructs the name-level declaration of group g from the descriptor, for display: Members in member (logical field) order, and Key only when the key is a strict subset of the members — the encoder marks every member key when none was declared, so an all-key group reads back as "no key", exactly as the declaration that produced it most likely did. The format carries no group name and no declaration ordinal, so the spec's Label numbers the group by its position in THIS file: a group dropped by the viability gate at import shifts later groups down by one relative to their declared names.

func (*Schema) HasBitmap added in v0.9.0 

func (s *Schema) HasBitmap() bool

HasBitmap reports whether the record carries a per-record null bitmap: whether any field stored IN THE ROW is nullable. Without groups that is any field; with groups, member fields' null bits ride their dictionary entry, so only the row fields count — a cohort whose every nullable field is a group member has no per-row bitmap.

func (*Schema) HasGroups added in v0.39.0 

func (s *Schema) HasGroups() bool

HasGroups reports whether the schema declares any parent group — i.e. whether its physical row differs from its logical row.

func (*Schema) Logical added in v0.39.0 

func (s *Schema) Logical() *Schema

Logical returns the schema's ungrouped view: the same Fields (the slice is SHARED, so *Field pointers and field indices agree), with no group declared. Its RecordByteSize / BitmapByteSize / HasBitmap describe the logical row. For a schema without groups it returns s.

func (*Schema) NewGroupEntryDecoder added in v0.39.0 

func (s *Schema) NewGroupEntryDecoder(g int, fields []int) (*GroupEntryDecoder, error)

NewGroupEntryDecoder compiles a decoder for the members of group g at the given LOGICAL field indices (each must be a member of g).

func (*Schema) RecordByteSize added in v0.8.4 

func (s *Schema) RecordByteSize() int

RecordByteSize returns the on-wire stride of one record under this schema. Bit-packed fields (U4, PackedBool) report ByteSize()==0 but the wire format still consumes one whole byte per such field. When the schema declares at least one nullable field, the trailing null bitmap of ceil(field_count/8) bytes is appended to every record and included in the stride.

With groups it is the PHYSICAL stride: one GroupIndexWidth index per indexed group, plus the row fields, plus the narrowed bitmap. It is still fixed and a pure function of the schema.

func (*Schema) RecordCountForPayload added in v0.34.0 

func (s *Schema) RecordCountForPayload(payloadBytes int64) (count, trailing int64, ok bool)

RecordCountForPayload derives how many whole records occupy a payload of payloadBytes under this schema, together with the leftover bytes that do not complete a record.

It is the ONE derivation behind every "how many records does this cohort hold" answer for a single-file cohort — service.CountRecords (the header-fast facade path and the parallel-decode eligibility gate) and descriptor.Inspect (the header-only reporting path) both call it. They lived as two independent floor divisions until they were lifted here; identical arithmetic written twice is one edit away from two different record counts over the same bytes, and nothing on either wire says which arm produced the number a caller is holding.

count is always the FLOOR. trailing is payloadBytes % stride and is non-zero only for a truncated tail — a cohort whose last record was half-written. The two arms deliberately differ in what they do with it, and only in that: Inspect has an envelope and raises an ENCODING_INVALID warning naming the leftover bytes, while CountRecords returns (uint64, error) with no warning channel and stays silent, because it is a counter feeding an eligibility gate rather than a diagnostic and a truncated tail must not make a cohort that still processes fail to count. The count itself is identical on both arms by construction.

ok is false when the stride is not positive (a field-less schema has no records to count) or payloadBytes is negative (the file is shorter than its own header + schema); the caller reports no count rather than a fabricated zero.

func (*Schema) RequiredFormatVersion added in v0.39.0 

func (s *Schema) RequiredFormatVersion() byte

RequiredFormatVersion returns the .pulse format version this schema's content needs on the wire. It is the single place a writer's version is chosen — never a global flag — so a schema that uses no 0x02 feature is always written at 0x01, byte-identical to a pre-0x02 file.

A schema requires 0x02 exactly when it declares a parent group.

func (*Schema) SetField added in v0.15.0 

func (s *Schema) SetField(name string) (*Dictionary, bool)

SetField returns the dictionary for a named set-typed field. Returns nil, false if the field is not found or is not a set type.

func (*Schema) ValidateGroups added in v0.39.0 

func (s *Schema) ValidateGroups() error

ValidateGroups checks the group descriptors against the schema's fields. A schema with no groups is always valid. Rules:

  • every group has a known kind and at least one member;
  • members are in range, strictly ascending, and no field belongs to two groups (groups are independent, never nested);
  • Entries is a whole number of entries; an indexed group has at most MaxGroupEntries entries, a constant group exactly one;
  • the physical stride is positive (a cohort whose every field is constant-elided would have a zero stride and an underivable record count).

type SchemaDoc added in v0.8.0 

type SchemaDoc struct {
	Schema               *Schema
	AggregateRecordCount uint64
	ShardCount           uint16
}

SchemaDoc carries the contents of a `_schema.pulse` entry: the canonical Schema plus the sharding metadata extension. AggregateRecordCount is the cached sum of every shard's record count at the time the archive was last written; ShardCount mirrors the zip central directory's entry count (excluding the reserved schema entry).

Both metadata fields are a SANITY CHECK, not the source of truth. Per the design contract, the per-shard headers are authoritative — callers populate live `ShardEntry.RecordCount` values by peeking each shard's header. AggregateRecordCount is exposed so cheap `pulse inspect` calls can report an approximate total without opening every shard.

func ReadSchemaDoc added in v0.8.0 

func ReadSchemaDoc(r io.Reader) (*SchemaDoc, error)

ReadSchemaDoc parses a `_schema.pulse` payload. It always reads the Pulse header + standard schema block. The sharding metadata extension (magic "SHRD" + agg + shard count) is read when present and silently skipped when absent — older `_schema.pulse` artifacts or single-file cohorts addressed by mistake still parse cleanly, returning AggregateRecordCount=0 and ShardCount=0.

type Segment added in v0.17.0 

type Segment interface {
	// contains filtered or unexported methods
}

Segment is the sealed sum type of decode steps emitted by Schema.BuildDecodePlan. The unexported marker method blocks out-of-package implementations so the iterator's switch is exhaustive.

type SetMask added in v0.38.0 

type SetMask struct {
	// contains filtered or unexported fields
}

SetMask is the single in-memory representation of a set field's membership bitmask, for every rung from set_u8 through set_u256. Bit i corresponds to dictionary entry i; an all-zero mask is a valid "no selection" and is distinct from null (set nulls ride the per-record null bitmap only, with no in-band sentinel).

Word order — normative wire contract

words[0] holds bits 0–63 (the LOW 64 bits), words[1] bits 64–127, words[2] bits 128–191, words[3] bits 192–255. On the wire the words are laid down in that same order — LITTLE-ENDIAN word order — with each word itself written little-endian. The payload of a wide set field is therefore exactly the little-endian byte image of the 128- or 256-bit integer, and the low 8 bytes of a set_u128 are byte-identical to the set_u64 payload holding the same selections. Reverse either the word order or the intra-word byte order and every cohort written since the wide rungs landed decodes as a different, plausible-looking selection.

Value semantics

SetMask is a fixed [SetMaskWords]uint64 VALUE. It never allocates, is copied by plain assignment, and can be retained across records without a defensive copy — no method aliases the receiver's backing array and every mutator returns a new value rather than writing through a pointer. That is a deliberate consequence of the 256-bit ceiling and is why the wide-set surface carries no reuse contract.

func ReadSetMask added in v0.38.0 

func ReadSetMask(r io.Reader, ft FieldType) (SetMask, error)

ReadSetMask reads a wide set field's payload from the record stream. Null state rides the per-record null bitmap, not the payload bytes, so this function has no null channel — an all-zero mask is an empty selection, not a null.

func SetMaskFromBytes added in v0.38.0 

func SetMaskFromBytes(ft FieldType, src []byte) (SetMask, error)

SetMaskFromBytes decodes the on-wire payload of a wide set field. It reads exactly ft.ByteSize() bytes from the front of src and ignores anything past that, so a caller may hand it a record buffer sliced at the field offset without a neighbouring column bleeding in. Words the rung does not carry stay zero.

func SetMaskFromUint64 added in v0.38.0 

func SetMaskFromUint64(low uint64) SetMask

SetMaskFromUint64 lifts a narrow-rung mask (set_u8..set_u64, all of which store their bitmask in a uint64) into the shared wide type. The value lands in words[0]; the higher words stay zero.

func SetMaskFromWords added in v0.38.0 

func SetMaskFromWords(words [SetMaskWords]uint64) SetMask

SetMaskFromWords builds a SetMask from raw words in the documented order: words[0] is the low 64 bits.

func (SetMask) Bits added in v0.38.0 

func (m SetMask) Bits() iter.Seq[int]

Bits returns an iterator over the set bit indices in ASCENDING order, for `for bit := range mask.Bits()`. Stopping early stops the walk.

func (SetMask) Equal added in v0.38.0 

func (m SetMask) Equal(o SetMask) bool

Equal reports whether two masks carry exactly the same bits.

func (SetMask) FitsFieldType added in v0.38.0 

func (m SetMask) FitsFieldType(ft FieldType) bool

FitsFieldType reports whether every set bit is within the capacity of the given set field type (ft.MaxSetEntries()). A non-set type has capacity 0, so only an empty mask fits one.

func (SetMask) Has added in v0.38.0 

func (m SetMask) Has(i int) bool

Has reports whether bit i is set. Out-of-range indices (negative, or at or above SetMaskBits) report false rather than panicking — a corrupt or mid-remap payload must not take the decoder down.

func (SetMask) HighestBit added in v0.38.0 

func (m SetMask) HighestBit() int

HighestBit returns the index of the highest set bit, or -1 when the mask is empty. Used by the write path to check a mask against the declared rung's capacity.

func (SetMask) Intersect added in v0.38.0 

func (m SetMask) Intersect(o SetMask) SetMask

Intersect returns the bitwise AND of m and o. Neither operand is modified.

func (SetMask) IsEmpty added in v0.38.0 

func (m SetMask) IsEmpty() bool

IsEmpty reports whether no bit is set. An empty mask is a valid "no selection", NOT a null.

func (SetMask) Labels added in v0.38.0 

func (m SetMask) Labels(dict *Dictionary) []string

Labels resolves the set bits to dictionary labels in ascending bit order. The walk is bounded by dict.Count(), NOT by the mask width or the field type's capacity: a bit beyond the dictionary is a corrupt or mid-remap payload and is skipped, never resolved and never fatal. A nil or empty dictionary yields an empty, non-nil slice.

func (SetMask) NextBit added in v0.38.0 

func (m SetMask) NextBit(from int) (int, bool)

NextBit returns the lowest set bit at index >= from, and ok=false when there is none. It is the allocation-free primitive Bits rides on, and the form to use on a hot path: 

for bit, ok := m.NextBit(0); ok; bit, ok = m.NextBit(bit + 1) { … }

func (SetMask) PopCount added in v0.38.0 

func (m SetMask) PopCount() int

PopCount returns the number of set bits — the size of the selection.

func (SetMask) Uint64 added in v0.38.0 

func (m SetMask) Uint64() (uint64, bool)

Uint64 returns the low 64 bits and reports whether the whole mask fits in them. ok=false means the caller is about to truncate: the returned value is the low word, but bits at or above 64 exist and would be lost. Every narrowing hand-off to the uint64 value API must check ok.

func (SetMask) Union added in v0.38.0 

func (m SetMask) Union(o SetMask) SetMask

Union returns the bitwise OR of m and o. Neither operand is modified.

func (SetMask) WithBit added in v0.38.0 

func (m SetMask) WithBit(i int) SetMask

WithBit returns a copy of m with bit i set. An out-of-range i is a no-op. The receiver is untouched.

func (SetMask) WithoutBit added in v0.38.0 

func (m SetMask) WithoutBit(i int) SetMask

WithoutBit returns a copy of m with bit i cleared. An out-of-range i is a no-op. The receiver is untouched.

func (SetMask) Words added in v0.38.0 

func (m SetMask) Words() [SetMaskWords]uint64

Words returns a copy of the backing words. Mutating the result cannot reach the receiver — arrays are values in Go, and that is the point.

type SetWidenPlan added in v0.38.0 

type SetWidenPlan struct {
	// FieldIndex is the field's position in both schemas (structural
	// cohesion has already established they are the same position).
	FieldIndex int
	// Field is the field name, for diagnostics.
	Field string
	// From is the rung the CANONICAL schema declares. From == To means
	// the archive itself does not move — only the incoming shard is
	// promoted to meet it.
	From FieldType
	// IncomingFrom is the rung the ARRIVING shard declares. It may be
	// narrower than From (the shard is promoted to canonical), wider
	// than From (the archive is promoted to the shard), or equal.
	//
	// The two sides are recorded separately because the executor widens
	// them separately: a payload already at To must not be handed to
	// WidenSetFieldBytes, which refuses a non-widening target, and a
	// caller reading the warning needs to tell an archive-wide rewrite
	// from a one-shard re-stride.
	IncomingFrom FieldType
	// To is the rung both sides carry after the plan is applied: the
	// widest of From, IncomingFrom and the narrowest rung holding the
	// merged dictionary. It is never narrower than either input rung —
	// narrowing would drop every selection above the target's ceiling.
	To FieldType
	// UnionEntries is the size of the merged dictionary the two schemas
	// imply.
	UnionEntries int
}

SetWidenPlan describes the rung promotion one set field needs before a shard can join an archive: the merged dictionary the two schemas imply no longer fits the rung both currently declare.

func PlanSetWidening added in v0.38.0 

func PlanSetWidening(canonical, incoming *Schema) ([]SetWidenPlan, error)

PlanSetWidening reports which set fields must be promoted to a wider rung before canonical and incoming can be dictionary-union-merged, in field order.

It answers the question MergeDictUnion cannot: MergeDictUnion sees a union that exceeds the declared bitmask width and can only refuse (PULSE_SHARD_DICT_WIDTH_OVERFLOW), because widening is an archive-wide byte rewrite and nothing at the schema layer may perform one. Running this FIRST lets the caller widen the whole archive and then merge into a rung that fits.

TWO forces produce a plan, and they compose:

  1. The merged DICTIONARY outgrows the declared bitmask, the original case.
  2. The two schemas DECLARE DIFFERENT RUNGS for the same set column. A re-import of a new period infers the rung its own data needs, so an archive seeded at set_u64 legitimately meets a set_u128 shard — and refusing that would force a full re-import of every prior period to reach a layout a mechanical re-stride already produces.

Because of (2) this function must run BEFORE ValidateStructuralCohesion rather than after it: cohesion is strict on the type byte and on every ByteOffset a rung change shifts, so a divergent rung is fatal there. The contract is that the caller applies the returned plans and THEN runs cohesion over the reconciled schemas, which stays strict on everything else. Only the set-rung dimension relaxes.

The target rung is never narrower than either declared rung: To is the widest of canonical's rung, incoming's rung and the narrowest rung holding the union. Narrowing would drop every selection above the target's ceiling, and it would do so SILENTLY — the record still decodes, to a smaller and entirely plausible selection.

An empty plan and a nil error is the ordinary case: the rungs agree and every merged dictionary already fits.

Errors:

  • PULSE_SHARD_DICT_WIDTH_OVERFLOW when the union exceeds the WIDEST rung. There is nowhere left to widen to, so this stays fatal — it is the one set-width overflow auto-widen does not remove.
  • PULSE_SHARD_SCHEMA_MISMATCH on nil schemas, differing field counts, or a set column meeting a non-set column of the same position. Other structural divergence is ValidateStructuralCohesion's to report; run it over the reconciled schemas.

Categorical fields are ignored entirely: a categorical's width is fixed at folder creation by contract, and its capacity (256 / 65 536 / 2^32) is not a bitmask a record rewrite can widen the same way.

type SetWidthHeadroom added in v0.38.0 

type SetWidthHeadroom struct {
	Field string `json:"field"`
	// Type is the rung the field currently declares.
	Type string `json:"type"`
	// Used is the number of dictionary entries in the canonical schema.
	Used int `json:"used"`
	// Capacity is the rung's bitmask width.
	Capacity int `json:"capacity"`
	// Headroom is Capacity - Used: how many more distinct members the
	// archive can absorb before the next `shard add` widens it.
	Headroom int `json:"headroom"`
	// NextType is the rung a widen would promote to, or "" when the
	// field is already at the widest rung — at which point Headroom is
	// the hard remainder and an overflow is fatal, not widenable.
	NextType string `json:"next_type,omitempty"`
}

SetWidthHeadroom reports how much of one set field's bitmask capacity the canonical dictionary has consumed, and which rung it would be widened to next.

It exists so an impending archive-wide widen is FORESEEABLE. The widen itself is correct but expensive (every record of every shard is re-laid-out), and without this a caller learns the field was one entry from its ceiling only by paying for the rewrite.

func SetWidthHeadroomFor added in v0.38.0 

func SetWidthHeadroomFor(s *Schema) []SetWidthHeadroom

SetWidthHeadroomFor returns one SetWidthHeadroom per set field in s, in field order. A schema with no set fields returns nil.

type SkipBytes added in v0.17.0 

type SkipBytes struct {
	N int
}

SkipBytes advances the underlying reader by N bytes without decoding. Coalesces a run of contiguous unprojected, non-bit-packed fields, or an entire bit-packed group when no member is retained, or the trailing null bitmap when no nullable field is retained. N > 0 always.

type TypedSetRecord added in v0.39.0 

type TypedSetRecord interface {
	SetNarrowSetAt(idx int, mask uint64)
	SetWideSetAt(idx int, m SetMask)
}

TypedSetRecord is an optional extension of IndexedReusableRecord that takes a set field's mask with its concrete type instead of through `any`. Boxing a uint64 mask (256 and above) or an encoding.SetMask into an interface costs one heap allocation per set field per row; a record implementing this interface stores the mask in typed storage and the reuse decoders never box it. Decimal128 needs no typed twin: it is a single-pointer struct, which Go stores in an interface without allocating.

The decoder checks for this interface once per decode call (a row on the full-stride path, a group on the plan path) and, when it is present, calls SetNarrowSetAt for set_u8..set_u64 and SetWideSetAt for set_u128 / set_u256 IN PLACE OF SetWideFieldAt. The value, and the SetNumericAt echo written before it, are identical to the boxed path.

type WidenReport added in v0.38.0 

type WidenReport struct {
	// Field is the name of the widened set field.
	Field string
	// From is the set rung the field carried before the rewrite.
	From FieldType
	// To is the set rung it carries after.
	To FieldType
	// Records is the number of records re-laid-out.
	Records int64
	// StrideBefore is the source record stride in bytes, bitmap included.
	StrideBefore int
	// StrideAfter is the rewritten record stride in bytes, bitmap included.
	StrideAfter int
}

WidenReport describes a completed widen. It is the value both callers of the engine (shard auto-widen and the `pulse widen` leaf) report from, so it carries the strides as well as the identities: the stride delta is the only observable difference between a widen that re-laid-out every record and one that did nothing.

func WidenSetFieldBytes added in v0.38.0 

func WidenSetFieldBytes(cohort []byte, field string, target FieldType) ([]byte, *WidenReport, error)

WidenSetFieldBytes rewrites a single-file .pulse cohort held in memory so the named set field is stored at the wider rung target, returning the new cohort bytes. The input slice is never modified.

It is the entry point for callers that already hold a cohort as bytes — a shard payload lifted out of a Zip64 archive, most of all. Callers holding a path want WidenSetFieldFile, which adds the atomic temp/fsync/rename dance around the same engine.

func WidenSetFieldFile added in v0.38.0 

func WidenSetFieldFile(fsys afero.Fs, path, field string, target FieldType) (*WidenReport, error)

WidenSetFieldFile rewrites the cohort at path in place, widening the named set field to target.

The rewrite is ATOMIC, and that is a correctness requirement rather than a nicety: widening changes the field's stride, so every record is re-laid-out and every field after the widened one moves. A half- written cohort is not a cohort that fails to open — it is one that opens and decodes every row past the failure point as garbage, with the schema block confidently describing a layout the tail does not have. So the new cohort is built into a temp file beside the original, fsynced, and only then renamed over it. Any failure before the rename removes the temp file and leaves the original byte-identical.

Sidecars are NOT updated: a widened cohort changes length, so the point-lookup index (`cohort.pulse.<keyhash>.idx`) and the SPSS metadata sidecar both invalidate themselves through their own size+mtime fingerprints on the next read. Rebuilding them is the caller's business.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.