encoding

func BucketIndex ¶

func BucketIndex(key []byte, bucketCount uint32) uint32

BucketIndex maps a key's hash onto a bucket slot in a table sized bucketCount. bucketCount == 0 always resolves to 0 — callers must guard the empty-index case themselves via len(Index.Buckets) before indexing.

func CheckSetWiden ¶

func CheckSetWiden(from, to encoding.FieldType) error

CheckSetWiden reports whether a set field of type from may be widened to type to, returning the refusal as a coded error otherwise. It is the whole admission policy, exported so a caller can pre-flight a widen (a CLI predict, a shard-admin decision) without touching bytes.

Every refusal is ENCODING_TYPE_MISMATCH: each one is a caller asking for a type transition the format does not define, which is exactly what that code names. This effort introduces no new error codes.

func CloneGroups ¶

func CloneGroups(groups []encoding.Group) []encoding.Group

CloneGroups is cloneGroups for callers outside the package that build a schema copy of their own (the shard-archive canonical seed).

func CloneVectors ¶

func CloneVectors(vs []encoding.VectorField) []encoding.VectorField

CloneVectors deep-copies vector-field metadata (labels included), so a schema clone keeps every vec field's dim — a clone that dropped it would size each vec field at 0 bytes and misalign every later field. nil for none.

func DecodeGroupEntry ¶

func DecodeGroupEntry(s *encoding.Schema, g, e int, sink IndexedReusableRecord) error

DecodeGroupEntry writes entry e of group g into sink at the members' LOGICAL field indices, with exactly the side effects a full row decode has for those fields (value + typed wide value; a null member gets SetNullFieldAt + SetNumericAt(0)). It does not call ClearForRow and touches no non-member field. It is the per-entry evaluation hook: a predicate over member fields can be evaluated once per entry through a record built over this schema.

func DedupCohort ¶

func DedupCohort(dst io.Writer, src io.Reader, specs []GroupSpec) (*encoding.Schema, int64, error)

DedupCohort reads a whole cohort (header, schema, records — any version, grouped or not) from src and writes its grouped equivalent, grouped by specs, to dst. The source is read through its LOGICAL stream, so a grouped source is regrouped from scratch. The physical rows are spooled in memory because the dictionaries precede the records on the wire — it is the small-cohort form of the one encode path (GroupEncoder.EncodeStream) that io.DedupJob drives with a temp-file spool. A truncated trailing record is ENCODING_INVALID rather than silently dropped. Returns the written schema and the record count.

func FlattenCohortBytes ¶

func FlattenCohortBytes(cohort []byte) ([]byte, *encoding.Schema, error)

FlattenCohortBytes returns the ungrouped (0x01) twin of a single-file cohort: the logical schema's preamble followed by every logical row, in order. A cohort without groups is returned unchanged (the same slice). The logical rows are exactly what every decoder yields, so the twin decodes identically; for a cohort deduped from a 0x01 original it reproduces that original byte-for-byte.

A payload ending mid-record is ENCODING_INVALID rather than a silently dropped tail (the walk is ForEachRecord's).

func ForEachRecord ¶

func ForEachRecord(r io.Reader, stride int, fn func(rec []byte) error) (int64, error)

ForEachRecord reads fixed-width records of stride bytes from r and calls fn with each one, returning how many it read. The slice fn receives is reused between calls. A clean end of input between two records ends the walk; input ending MID-record is ENCODING_INVALID (details trailing_bytes, records_read) rather than a silently dropped tail. Every whole-cohort record pass of the dedup family — encoding, constant detection, candidate detection — walks through here, so the three cannot disagree about where a cohort ends.

func HashKey ¶

func HashKey(key []byte) uint64

HashKey returns the 64-bit FNV-1a digest of raw key bytes. Builders and lookup callers both route through this (and BucketIndex) so the two sides never diverge on hash choice.

func IndexManifestPath ¶

func IndexManifestPath(cohortPath string) string

IndexManifestPath derives the deterministic, KEYLESS on-disk path of the sidecar index manifest for cohortPath. Pure — no filesystem access. Unlike SidecarIndexPath there is nothing to know in advance: that is the whole point.

func IsArchive ¶

func IsArchive(r io.ReaderAt, size int64) (bool, error)

IsArchive reports whether the first four bytes of r identify a zip container. It does no central-directory parsing — magic-byte check only — so it is cheap enough to call on every Open. Errors are returned for I/O failures; a short file returns (false, nil).

func IsLowQualityDescription ¶

func IsLowQualityDescription(desc string) bool

IsLowQualityDescription reports whether a field description is empty, shorter than 10 bytes once trimmed, or one of the generic placeholders — the PULSE_FIELD_DESCRIPTION_LOW_QUALITY rule. It is the one predicate predict's description check and the cohort builder's schema check share, so the two cannot disagree on which fields draw the finding.

func JoinKeyTypesCompatible ¶

func JoinKeyTypesCompatible(a, b encoding.FieldType) bool

JoinKeyTypesCompatible reports whether two schema types can be compared as equi-join keys after normalisation. The conservative v1 rule: a set_* column never (not even against an identical rung — see JoinKeySetRejection), nor a vector column (JoinKeyVectorRejection); identical types match; categorical types of any width match each other (dict strings normalise to text); the unsigned-int / float / date family matches within itself. decimal128 matches only decimal128 (precision differences matter).

func JoinKeysRefusal ¶

func JoinKeysRefusal(left, right *encoding.Schema, spec *types.JoinSpec) error

JoinKeysRefusal is the first of JoinKeysRefusals, or nil — the refusal the runtime returns.

func JoinKeysRefusals ¶

func JoinKeysRefusals(left, right *encoding.Schema, spec *types.JoinSpec) []*errors.CodedError

JoinKeysRefusals is the ONE join-key rule. It judges a JoinSpec's kind and OnPairs against the left and right cohort schemas and returns every refusal, in the order the runtime meets them:

• no OnPair at all — PULSE_JOIN_KEYS_EMPTY (no details);
• a kind other than inner — PULSE_JOIN_KIND_NOT_IMPLEMENTED {kind};
• per OnPair i, in order: a blank side — PULSE_JOIN_KEYS_EMPTY {index}; a LeftField the left schema lacks, then a RightField the right schema lacks — PULSE_JOIN_FIELD_UNKNOWN {field, index}; key types that cannot be compared — PULSE_JOIN_TYPE_MISMATCH {left_field, left_type, right_field, right_type, index}, plus reason "set_key" with JoinKeySetRejection for a set_* key, reason "vector_key" with JoinKeyVectorRejection for a vec_f32 / vec_f64 key.

The runtime (processing.NewHashJoinIterator, and the service before it materialises the right side) refuses with the first entry; predict, ValidateJoin and the Compose / chain validators report them — so a caller sees the same code, message and details from either side. A nil spec or schema yields nil: there is nothing to judge.

func JoinedSchema ¶

func JoinedSchema(left, right *encoding.Schema, as string) (*encoding.Schema, error)

JoinedSchema synthesises the schema a single inner hash join produces: left fields unchanged, then every right field renamed with the `as` prefix (when non-empty). A right field whose final name collides with an earlier field is PULSE_JOIN_FIELD_COLLISION with {field, as} details.

It is the ONE join-schema rule: the runtime (processing.JoinedSchema, which the hash-join iterator builds its records over) and predict (internal/descriptor, which may not import the engine) both call it, so zone resolution, smart defaults and field validation see the same field set and the same field types on both sides. The result has no on-wire byte layout; it exists for field lookups over in-memory joined records.

func LayoutFields ¶

func LayoutFields(s *encoding.Schema) int

LayoutFields is the canonical record layout: it assigns every field's ByteOffset (the running sum of Schema.FieldWidth over the fields before it) and BitPosition (always 0 — each bit-packed field owns its own byte, never sharing it with a neighbour) in place, and returns the payload width (the stride minus any trailing null bitmap).

The stored offset is a cache of this derivation — readers are cursor-driven — but it is written into the schema block and compared by shard cohesion and the shard dictionary rewrite, so every path that builds a schema (inference, explicit schema files, the cohort builder, adapters that read a typed source schema, synth, width promotion) lays it out HERE. A second hand-rolled sum that summed ByteSize() gave every field after a bit-packed one an offset one byte short.

It takes the whole schema, not a field slice: a field's width is the field-aware Schema.FieldWidth, never a function of its type byte alone (TestNoTypeOnlyWidthInStridePaths).

func MergeDictUnion ¶

func MergeDictUnion(canonical, incoming *encoding.Schema) (*encoding.Schema, map[int]DictRemap, error)

MergeDictUnion is the relaxed dictionary cohesion check. For each categorical field it computes the union of canonical's and incoming's dictionaries: canonical entries first in their existing order, then any new entries from incoming in their order. Returns the extended canonical schema (deep copy when an extension is adopted; identical pointer when not) and a per-field-index map of DictRemap entries describing how to translate incoming record indices into canonical indices.

The structural validator (ValidateStructuralCohesion) must run FIRST — this validator assumes field positions, names, and categorical widths already match.

Errors:

• PULSE_SHARD_DICT_WIDTH_OVERFLOW when the union would exceed the field's categorical width capacity.

MergeDictUnion is the default cohesion mode at insert time (CreateShardArchive / AddShard). The stricter prefix-only rule (ValidateDictPrefixRule) is retained for callers that want to surface divergence as an error instead of merging — used by VerifyShardArchive when checking archives written before the union-merge default landed and by embedders that prefer to align dictionaries upstream.

func NewLogicalStream ¶

func NewLogicalStream(r io.Reader, s *encoding.Schema) (io.Reader, *encoding.Schema, error)

NewLogicalStream returns a reader of the logical record stream of s over r (positioned at the first physical record), and the schema that describes that stream (s.Logical()). Byte-level record consumers — a decoder of their own, an exporter — read through it and walk the returned schema exactly as they walk a 0x01 cohort. For a schema without groups it returns (r, s) unchanged.

func PromoteAdd ¶

func PromoteAdd(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteAdd returns the (precision, scale) of a SUM/SUB result given two operand types per SQL:2016 / Arrow Decimal128 rules:

(p1, s1) ± (p2, s2) => (max(p1-s1, p2-s2) + max(s1, s2) + 1, max(s1, s2))

The result precision is clamped at MaxDecimalPrecision; clamping callers must check ClampedPrecision and emit PULSE_DECIMAL_OVERFLOW when overflow surfaces at runtime.

func PromoteDiv ¶

func PromoteDiv(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteDiv returns the (precision, scale) of a DIV result.

(p1, s1) ÷ (p2, s2) => (p1 + s2 + 1, max(s1+s2, MIN_SCALE))

func PromoteMul ¶

func PromoteMul(p1, s1, p2, s2 uint8) (uint8, uint8)

PromoteMul returns the (precision, scale) of a MUL result.

(p1, s1) × (p2, s2) => (p1 + p2, s1 + s2)

func ReadIndexHeader ¶

func ReadIndexHeader(r io.Reader) error

ReadIndexHeader reads and validates the sidecar index header from r. Returns ENCODING_INVALID on a truncated header, wrong magic, or an unsupported version byte — mirroring ReadHeader's contract for the .pulse envelope.

func ReadPreamble ¶

func ReadPreamble(r io.Reader) (*encoding.Schema, byte, error)

ReadPreamble reads the header and the version-matched schema block, leaving r positioned at the first record byte. It is the one-call form of [ReadHeader] + [ReadSchema] for callers that do not need to wrap the two failures differently.

func ReadRecordAt ¶

func ReadRecordAt(loc *encoding.RecordLocator,
	r io.ReadSeeker,
	i uint64,
	values map[string]float64,
	nulls map[string]bool,
	wide map[string]any,
	keep FieldFilter,
	plan *DecodePlan,
) error

ReadRecordAt decodes the record at index i directly: it seeks r to loc.Offset(i) and decodes from there, without iterating any preceding record. r must be an io.ReadSeeker positioned over the same underlying bytes the locator was built from (the same *bytes.Reader passed to NewRecordLocator, or a fresh reader over the same byte slice) — the seek is absolute, so r's current cursor position on entry is irrelevant.

plan is optional: pass a *DecodePlan (from BuildDecodePlan(schema, retained)) to decode only the columns the plan retains — the O(1) projection variant required by point-lookup callers that only need a handful of return columns out of a wide schema. Pass nil to decode every field via the existing full-record path. keep mirrors the FieldFilter contract used throughout this package (see ReadRecordWithWidePlan / ReadRecordWithWideProjected) — pass the same filter used to build plan's retained set so map writes for incidental group members (a bit-packed neighbour or a bitmap-adjacent nullable field) stay suppressed; pass nil to keep every field the plan/schema visits.

Returns a coded ENCODING_INVALID error — never a panic, never an out-of-bounds read — when i >= loc.TotalRecords.

func ReadRecordAtRaw ¶

func ReadRecordAtRaw(loc *encoding.RecordLocator,
	r io.ReadSeeker,
	i uint64,
	values map[string]float64,
	nulls map[string]bool,
	wide map[string]any,
	raw map[string]uint64,
) error

ReadRecordAtRaw is ReadRecordAt's full-record exact variant: it decodes record i into values / nulls / wide exactly as ReadRecordAt with a nil plan and a nil keep does, and ALSO writes the exact on-wire word of every field read through ReadFieldValue into raw (integers u8..u64, f32/f64 bit patterns, date, datetime, categorical dictionary IDs and the narrow set masks). Bit-packed fields (u4, packed_bool) are exact in values; decimal128 and the wide set rungs are exact in wide. A null field is absent from raw and wide.

It is the decode path for an index-addressed reader that must hand back the stored value rather than the float64 echo (u64 above 2^53, datetime beyond 2^53 seconds). A grouped (0x02) locator decodes the logical record, identical to the ungrouped twin.

func ReadSchemaLayout ¶

func ReadSchemaLayout(r io.Reader, v byte) (*encoding.Schema, error)

ReadSchemaLayout reads a schema block at version v without its dictionaries or group entry tables (nil Field.Dictionary and Group.Entries) — enough for the record stride, at a fraction of a full decode's allocation. r is left at the first record byte. Used wherever a shard's own schema block is only read past: the record-count peek (Archive.PeekShardRecordCount) and the archive scan (internal/service shardIter.openShard / processOneShard), which decodes every row through the canonical schema, the lookup locator (openRecordLocator), the shard-admin count (recordCountFromBytes) and the shard-archive filter (skipShardPreamble).

func RefuseGroups ¶

func RefuseGroups(s *encoding.Schema, operation string, code errors.Code) error

RefuseGroups returns a coded error when s declares parent groups and the caller's operation rewrites or reinterprets physical row bytes without understanding them yet (shard-archive assembly, set-field widening, categorical byte remap). It keeps such a path loud rather than silently misreading a grouped row by its logical offsets. nil for an ungrouped schema.

func RewriteShardCategoricals ¶

func RewriteShardCategoricals(shardBytes []byte, targetSchema *encoding.Schema, remap map[int]DictRemap) ([]byte, error)

RewriteShardCategoricals re-emits a single-file .pulse shard with its schema block replaced by targetSchema and its records' categorical indices translated according to remap. The on-wire byte layout of every non-categorical field is preserved exactly.

targetSchema must be structurally identical to the shard's own schema (field count, names, types, byte offsets, bit positions) — only the per-categorical dictionaries differ. The record bytes are addressed by the type-derived layout the decoder walks (widenFieldOffsets / LayoutFields), not by the stored offsets.

When remap is empty (every incoming dict was a prefix of canonical), the function still re-emits the shard with targetSchema so the shard's standalone dictionary matches the archive's canonical (callers reading the shard via the `archive.pulse#shard.pulse` anchor see the union dictionary). Record bytes are copied verbatim.

Returns PULSE_SHARD_HEADER_INVALID if the input bytes are not a valid single-file Pulse cohort, and PULSE_SHARD_SCHEMA_MISMATCH if targetSchema is structurally incompatible with the shard's own schema.

func RewriteShardCategoricalsKeepNulls ¶

func RewriteShardCategoricalsKeepNulls(shardBytes []byte, targetSchema *encoding.Schema, remap map[int]DictRemap) ([]byte, error)

RewriteShardCategoricalsKeepNulls is RewriteShardCategoricals except that a field which is NULL in a record keeps its placeholder bytes instead of having them remapped. A null's bytes carry no value, but they are part of a parent-group ENTRY's identity (an entry is compared byte for byte): remapping a null categorical member's placeholder would make a shard's (region = null) tuple a different entry from the archive's identical one — a duplicate entry in an unkeyed group, and a spurious PULSE_GROUP_MEMBER_NOT_CONSTANT on a keyed group whose non-key member is null. The shard-archive group path remaps the LOGICAL (flattened) form of a shard through this variant before re-encoding it against the canonical dictionaries; the ungrouped archive path keeps RewriteShardCategoricals' byte-for-byte behaviour.

func SetLadder ¶

func SetLadder() []encoding.FieldType

SetLadder returns the set-width ladder narrowest rung first. The returned slice is a fresh copy; callers may mutate it freely.

func SetTypeFor ¶

func SetTypeFor(elements int) (encoding.FieldType, bool)

SetTypeFor returns the narrowest set_* type with a bit for each of `elements` elements, and reports false when no rung is wide enough (or when `elements` is not positive — a set with no elements has no type, and that is a caller error rather than a width verdict).

func SetWidthHeadroomFor ¶

func SetWidthHeadroomFor(s *encoding.Schema) []encoding.SetWidthHeadroom

SetWidthHeadroomFor returns one SetWidthHeadroom per set field in s, in field order. A schema with no set fields returns nil.

func SidecarIndexPath ¶

func SidecarIndexPath(cohortPath string, keyFields []string) string

SidecarIndexPath derives the deterministic on-disk path for a point-lookup sidecar index built against cohortPath's key column set: "<cohortPath>.<keyhash>.idx". keyhash is the 16-hex-digit FNV-1a digest (HashKey) of the ordered key column names, joined with a NUL byte separator that cannot appear inside a schema field name — so ["ab", "c"] and ["a", "bc"] hash to distinct paths rather than colliding on naive concatenation.

Key column order is significant: SidecarIndexPath(p, []string{"a", "b"}) and SidecarIndexPath(p, []string{"b", "a"}) derive different, stable paths, matching the composite-key contract ("key column order is significant") the format's Keys spec carries. Single-key builds (this story) pass a one-element slice; the derivation is already N-column-general so E2-S1's composite-key work and E4-S1's CLI / E1-S4's lookup service need no path-derivation change.

Deterministic and pure — no filesystem access, no ordering beyond what the caller supplies in keyFields. Callers own key-order normalization (or intentional non-normalization) before calling.

func ValidateDictPrefixRule ¶

func ValidateDictPrefixRule(canonical, incoming *encoding.Schema) (*encoding.Schema, error)

ValidateDictPrefixRule enforces the append-only prefix rule for every categorical field shared by canonical and incoming. For each `categorical_*` field:

• If incoming's dictionary is a prefix of canonical's, accept the shard as-is; no canonical change. (Older shard that never saw newer dictionary values.)
• If canonical's dictionary is a prefix of incoming's, the canonical schema adopts the extension. The returned extendedCanonical carries the merged dictionaries and the caller is responsible for rewriting `_schema.pulse` to publish them before the new shard is placed (crash-safety ordering documented in placeholder §3.2).
• Neither is a prefix of the other → PULSE_SHARD_DICT_DIVERGENCE.
• An extension that would exceed the field's width capacity (256 for u8, 65 536 for u16, 2^32 for u32) → PULSE_SHARD_DICT_WIDTH_OVERFLOW.

The structural validator (ValidateStructuralCohesion) must run FIRST — this validator assumes field positions, names, and categorical widths already match.

The returned extendedCanonical is a deep copy when an extension is adopted, and is identically the canonical input otherwise. Callers can compare pointers to detect whether a rewrite is required.

func ValidateGroupCohesion ¶

func ValidateGroupCohesion(canonical, incoming *encoding.Schema) error

ValidateGroupCohesion is the parent-group dimension of structural cohesion: canonical and incoming must declare the identical group layout — the same group count and, per group, the same kind, members and key flags. It is what makes the physical stride of every shard equal the canonical stride, so a 0x01 shard in a 0x02 archive (or the reverse, or two different groupings) is PULSE_SHARD_SCHEMA_MISMATCH. Group DICTIONARIES are not compared here; they follow the prefix rule (ValidateDictPrefixRule). ValidateStructuralCohesion calls it, so `pulse shard verify` refuses a genuinely mixed-layout archive.

func ValidatePrecisionScale ¶

func ValidatePrecisionScale(precision, scale uint8) error

ValidatePrecisionScale reports whether (precision, scale) form a legal decimal128 type spec (1 ≤ precision ≤ 38, 0 ≤ scale ≤ precision).

func VectorFromSpan ¶

func VectorFromSpan(ft encoding.FieldType, sub []byte, dst []float64) ([]float64, error)

VectorFromSpan decodes a vector field's whole on-wire span — exactly its Schema.FieldWidth bytes — appending the elements to dst[:0]. The dim is the span's element count; a span that is not a whole multiple of the element size, or whose count is outside MinVectorDim .. MaxVectorDim, is ENCODING_INVALID (encoding.VectorFromBytes).

func WidenSchemaSetField ¶

func WidenSchemaSetField(s *encoding.Schema, field string, target encoding.FieldType) (*encoding.Schema, error)

WidenSchemaSetField returns a copy of s in which the named set field carries the wider rung target, with every field's ByteOffset recomputed for the new stride.

It is the SCHEMA half of a widen, split out because the shard auto-widen path has to publish a widened canonical `_schema.pulse` that has no records of its own alongside shard payloads widened by WidenSetFieldBytes. Both go through this one function, so the canonical schema block and every shard's schema block cannot end up describing different layouts — a divergence that would pass ValidateStructuralCohesion nowhere and decode as garbage everywhere.

Names, nullability, descriptions, CSV indices, decimal metadata and the inline dictionary pointer ride across by value, so bit i still means dictionary entry i. The input schema is never modified.

func WidenSetFieldBytes ¶

func WidenSetFieldBytes(cohort []byte, field string, target encoding.FieldType) ([]byte, *encoding.WidenReport, error)

WidenSetFieldBytes rewrites a single-file .pulse cohort held in memory so the named set field is stored at the wider rung target, returning the new cohort bytes. The input slice is never modified.

It is the entry point for callers that already hold a cohort as bytes — a shard payload lifted out of a Zip64 archive, most of all. Callers holding a path want WidenSetFieldFile, which adds the atomic temp/fsync/rename dance around the same engine.

func WidenSetFieldFile ¶

func WidenSetFieldFile(fsys afero.Fs, path, field string, target encoding.FieldType) (*encoding.WidenReport, error)

WidenSetFieldFile rewrites the cohort at path in place, widening the named set field to target.

The rewrite is ATOMIC, and that is a correctness requirement rather than a nicety: widening changes the field's stride, so every record is re-laid-out and every field after the widened one moves. A half- written cohort is not a cohort that fails to open — it is one that opens and decodes every row past the failure point as garbage, with the schema block confidently describing a layout the tail does not have. So the new cohort is built into a temp file beside the original, fsynced, and only then renamed over it. Any failure before the rename removes the temp file and leaves the original byte-identical.

Sidecars are NOT updated: a widened cohort changes length, so the point-lookup index (`cohort.pulse.<keyhash>.idx`) and the SPSS metadata sidecar both invalidate themselves through their own size+mtime fingerprints on the next read. Rebuilding them is the caller's business.

func WidestSetType ¶

func WidestSetType() encoding.FieldType

WidestSetType returns the top rung of the ladder — the widest set_* type Pulse has. Callers that must NAME the ceiling in a diagnostic use this rather than writing "set_u256" into a string, so the message cannot outlive the type it names.

func WriteIndex ¶

func WriteIndex(w io.Writer, idx *Index) error

WriteIndex serializes idx to w.

Format (v3):

9-byte header: magic "PULSEIDX" + version 0x03
32-byte fingerprint: raw SHA-256 digest
key-spec block:
  u16 key_count
  per key: u16 name_len + utf8 name, u8 type
source-stat snapshot:
  u64 source_size
  i64 source_mod_time_unix_nano
u32 bucket_count
bucket-offset table (fixed-width, bucket_count entries):
  per bucket: u64 byte_offset — relative to the start of the
    bucket-data region (i.e. relative to the byte immediately
    following this table). offsets[0] is always 0.
bucket-data region (bucket_count self-delimited blocks, in order):
  per bucket:
    u32 entry_count
    per entry:
      u16 key_len + raw key bytes
      u32 row_id_count
      per row_id: u64 (little-endian)

The offset table's fixed width is what lets a reader (ReadBucketByKey) seek directly to entry bucketIndex without parsing any other bucket: offsetTableStart + bucketIndex*8 locates the one relevant offset, and bucketDataStart + offsets[bucketIndex] locates that bucket's self-delimited data. See IndexMeta's doc comment for the anchor definitions.

func WriteIndexFile ¶

func WriteIndexFile(fsys afero.Fs, path string, idx *Index) error

WriteIndexFile serializes idx and writes it to path on fsys, creating or truncating the file as needed. All sidecar index I/O goes through afero.Fs so callers get the same hermetic-test story as the rest of Pulse's file surface (fs.NewMemMap() in tests, fs.Default() / a caller-injected afero.Fs in production).

func WriteIndexHeader ¶

func WriteIndexHeader(w io.Writer) error

WriteIndexHeader writes the sidecar index header (magic + version) to w.

func WriteIndexManifest ¶

func WriteIndexManifest(fsys afero.Fs, cohortPath string, m *IndexManifest) error

WriteIndexManifest serialises m to cohortPath's derived manifest path through fsys (never raw os), indented because the document's second job is to be read by a human debugging a corpus. Entries are sorted first, so the same set of indexes always produces the same bytes.

func WritePreamble ¶

func WritePreamble(w io.Writer, s *encoding.Schema) error

WritePreamble writes the header and schema block for s at the version its content requires ([Schema.RequiredFormatVersion]). A schema that uses no 0x02 feature produces bytes identical to [WriteHeader] + [WriteSchema].

func WriteSchemaDoc ¶

func WriteSchemaDoc(w io.Writer, schema *encoding.Schema, agg uint64, shardCount uint16) error

WriteSchemaDoc emits a `_schema.pulse` payload: standard Pulse header, standard schema block, then the sharding metadata extension (magic "SHRD" + u64 aggregate_record_count + u16 shard_count, little-endian). The result is a valid single-file Pulse cohort with zero records, so legacy readers that don't know about the SHRD extension can still parse the header + schema and stop at the (empty) record region — the extension bytes live where records would otherwise begin and a header-only reader (`Inspect`) ignores them.