processing

package
v0.25.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 10, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package processing provides the single dynamic processing engine for Pulse. It implements aggregators, attributes, filterers, and groupers that operate on record iterators backed by .pulse encoded data.

welfordBucket carries the streaming Welford-Pébaÿ recurrence (mean + M2 + count) reused across statistical aggregators and tests.

Helpers for components-source consumption by the four stat-test parity overlay handlers — OVERLAY_T_CELL / OVERLAY_Z_CELL (MATRIX arm) and OVERLAY_T_VS_REF / OVERLAY_Z_VS_REF (SERIES arm).

Handlers read `{mean, variance, n}` from the universal Components surface — the canonical source of stat-test inputs for the parity overlays.

  • MATRIX arm: extractCellComponentsTriple pulls the triple from Response.Components.Crosstab.CellComponents[r][c]. Maps without all three keys fall through to (0, 0, 0, false) so the scalar + Params fallback path engages.

  • SERIES arm: encodeSeriesRowAnyMap / buildSeriesRowLookupAnyMap mirror the row-encoding identity rule of encodeSeriesRowAny but detect a `map[string]any` value column carrying `{mean, variance, n}` keys. Same canonical key rendering — scalar-only rows produce byte-identical keys so the additive contract is preserved.

Raw-cell lookup helper for COMPOSE overlay handlers. The four parity overlays (OVERLAY_T_CELL / OVERLAY_Z_CELL / OVERLAY_T_VS_REF / OVERLAY_Z_VS_REF) read `(mean, variance, n)` from Response.Components.Crosstab.CellComponents[r][c] (MATRIX arm) or the row-shape components map (SERIES arm); the components-source equivalents live in welford_components_extract.go.

`buildMatrixCellRawLookup` remains because the MATRIX-arm handlers still need to verify a reference cell is `Present` at the matching (row_key, col_key) tuple before falling through to the scalar + Params additive contract. The lookup carries the raw cell payload untouched — the handler reads scalar fallback values via `scalarFromCell` only when CellComponents is absent.

Index

Constants

This section is empty.

Variables

View Source 
var ErrGrouperKeyNull = stderrors.New("processing: grouper key null")

ErrGrouperKeyNull is the sentinel error returned by StreamableGrouper.KeyFor when the record's grouped field is null, missing, or otherwise has no defined bucket key. Callers in the fused crosstab path (and any future per-record bucket router) check it via errors.Is to decide whether to skip the record.

The sentinel is distinct from an empty-string key — set-typed groupers (GROUP_SET_VALUE with an empty mask) legitimately emit "" as a valid bucket key, so an in-band string sentinel is impossible.

Exported so embedder-supplied StreamableGrouper extensions can return the same null signal the built-in groupers do; the fused crosstab router accepts the sentinel from any source.

Functions

func AggregateDecimalField added in v0.2.0 

func AggregateDecimalField(agg types.AggregationType, records []*Record, field string, scale uint8) (decimalAggResult, error)

AggregateDecimalField runs the decimal-aware aggregator for `agg` over the given records, reading the wide map for `field`. Used by the orchestrator when the field type is decimal128 or nullable_decimal128.

func AggregationLabel added in v0.10.0 

func AggregationLabel(agg *types.Aggregation) string

AggregationLabel returns the output column name for an aggregation, matching the labelling used by the streaming and buffered paths in Processor.Process. Empty label falls back to "<TYPE>_<Field>".

func ApplyChainOverlays added in v0.19.0 

func ApplyChainOverlays(specs []*types.ChainOverlaySpec, stages []*types.Response, stageNames []string) ([]*types.OverlayLayer, []types.OverlayWarning, error)

ApplyChainOverlays executes every spec in specs against the chain of finalised stage responses and returns one OverlayLayer per spec in matching order (FR-A2: stable spec-order layer emission), plus a flat warning slice. Returns (nil, nil, nil) when specs is empty so the service-side ProcessChain post-stage-loop call site can call ApplyChainOverlays unconditionally — mirrors the MATRIX ApplyOverlays / SERIES ApplyOverlaysSeries / FACET ApplyOverlaysFacet short-circuit contract.

`stages` is the ordered list of per-stage *Response objects in the same order as the originating `ChainRequest.Stages` slice; `stageNames` is the parallel ordered list of `ChainStage.Name` values (empty string where the stage had no Name) — the resolver uses it to build the `Name → Index` lookup table for `StageRef.Name` resolution.

Resolution policy:

  • StageRef.Index (non-nil) resolves directly; out-of-range fires a coded error (PULSE_OVERLAY_TARGET_UNKNOWN for the Target arm, PULSE_OVERLAY_REFERENCE_UNKNOWN for the Ref arm).
  • StageRef.Name (non-empty) resolves against the namedIndex lookup; absent name fires the same coded error.
  • Target with both slots empty defaults to `len(stages) - 1` (the latest stage). Ref with both slots empty fires the coded error immediately — every spec MUST name a baseline.

Defense in depth: the descriptor.ValidateOverlays gate will reject bad kinds at predict time, so a missing dispatch entry should never reach the runtime in practice; nonetheless ApplyChainOverlays guards against an unknown kind and returns a CodedError whose details carry errors.PULSE_OVERLAY_KIND_UNKNOWN so the orchestrator surfaces the same failure mode predict would have flagged.

Shape divergence: the per-kind CHAIN handlers (applyIndexVsStage / applyDeltaVsStage) emit PULSE_OVERLAY_CHAIN_STAGE_SHAPE_DIVERGENT when the resolved target stage and reference stage host shapes disagree. The dispatcher itself does not gate on shape — the per-kind handlers own the shape-divergence defence so each kind picks its own fallback payload (the canonical rule today is "empty payload that inherits the target stage's shape").

func ApplyComposeOverlays added in v0.19.0 

func ApplyComposeOverlays(specs []types.ComposeOverlaySpec, responses []*types.Response, labels []string) ([]types.OverlayLayer, []types.OverlayWarning, error)

ApplyComposeOverlays executes every spec in specs against the already-finalised per-slot responses and returns one OverlayLayer per spec in matching order, plus a flat warning slice. Returns (nil, nil, nil) when specs is empty so the service-side Compose / ComposeParallel post-slot-barrier call site can call ApplyComposeOverlays unconditionally — mirrors the MATRIX / SERIES / FACET / CHAIN short-circuit contract.

`responses` is the ordered list of per-slot *Response objects in the same order as the originating `ComposedRequest.Requests` slice; nil entries correspond to slots that failed when `FailFast=false`. `labels` is the parallel ordered list of per-slot `Request.Label` values (after applyComposeLabelDefaults has filled empty slots with `request_<index+1>`) — the resolver uses it to build the `Label → Index` lookup table for `Reference` / `Targets` resolution.

Resolution policy:

  • Reference is REQUIRED on every spec. Empty Reference (or a Reference that does not resolve to any label in `labels`) fires a coded error carrying errors.PULSE_OVERLAY_REFERENCE_UNKNOWN via processing.LookupReference.
  • Targets is REQUIRED to have at least one entry. Empty Targets fires PULSE_OVERLAY_TARGET_UNKNOWN.
  • A target label that does not resolve to any label in `labels` OR resolves to a nil slot (failed under FailFast=false) fires PULSE_OVERLAY_TARGET_UNKNOWN via processing.LookupTarget.

Defense in depth: the descriptor.ValidateComposedRequest gate rejects bad references / unknown kinds at predict time, so a missing dispatch entry should rarely reach the runtime in practice; nonetheless ApplyComposeOverlays guards against missing slot labels and returns a CodedError whose details carry the appropriate canonical code so the orchestrator surfaces the same failure mode predict would have flagged.

Stub fallback: kinds without a registered handler route through `applyComposeStub`, which emits a zero-payload OverlayLayer that inherits the reference slot's host shape. The stub emits no warnings and no per-coordinate values.

func ApplyOverlays added in v0.19.0 

func ApplyOverlays(specs []types.OverlaySpec, host *CrosstabHostView) ([]types.OverlayLayer, []types.OverlayWarning, error)

ApplyOverlays executes every spec in specs against the host view and returns one OverlayLayer per spec in matching order, plus a flat warning slice. Returns (nil, nil, nil) when specs is empty.

Per PRD §6: buffered execution — each handler reads the already materialised host MatrixPayload, never re-scans records. Cost is O(cells × layers) where cells = RowCount × ColumnCount.

Defense in depth: the descriptor.ValidateOverlays gate rejects bad kinds at predict time, so a missing dispatch entry should never reach the runtime in practice; nonetheless ApplyOverlays guards against an unknown kind and returns a CodedError whose details carry errors.PULSE_OVERLAY_KIND_UNKNOWN so the orchestrator surfaces the same failure mode that predict would have flagged.

host may be nil when specs is empty; ApplyOverlays short-circuits. When specs is non-empty but host is nil the call fails fast — every crosstab overlay family expects a MATRIX-shaped host.

Level / Within belt-and-suspenders gate: before dispatching each spec ApplyOverlays runs `validateOverlayLevelWithinRuntime` to surface PULSE_OVERLAY_LEVEL_OUT_OF_RANGE for out-of-range Level / Within values against the host's RowAxisDepth / ColumnAxisDepth. The same condition is caught at predict time (descriptor.ValidateOverlays); the runtime gate is defensive in case the predict step was bypassed (programmatic Process callers).

Extension visibility: callers that need embedder-registered ExprFunctions (OVERLAY_FORMULA) to participate in the per-cell expression environment should use `ApplyOverlaysWithExtensions` instead — this no-extension entry point preserves backward compatibility for in-package tests + non-FORMULA call sites.

func ApplyOverlaysFacet added in v0.19.0 

func ApplyOverlaysFacet(specs []types.OverlaySpec, host *types.FacetField, pop *FacetPopulationView) ([]types.OverlayLayer, []types.OverlayWarning, error)

ApplyOverlaysFacet executes every spec in specs against the FACET host view and returns one OverlayLayer per spec in matching order (FR-A2: stable spec-order layer emission), plus a flat warning slice. Returns (nil, nil, nil) when specs is empty so the service-side FacetSchema finalize call site can call ApplyOverlaysFacet unconditionally — mirrors the MATRIX ApplyOverlays and SERIES ApplyOverlaysSeries short-circuit contract.

Host policy:

  • When specs is empty, host AND pop may be nil. The dispatch short-circuits before touching either.
  • When specs is non-empty, host may still be nil OR carry an empty discrete / numeric payload — the dispatch passes the view straight through to the handler. The per-kind handler decides how to surface an empty host (typically an empty Entries slice without warning; INDEX_VS_POP does exactly that).
  • When specs is non-empty, pop SHOULD be the result of a successful `ResolveFacetPopulation` call — a nil pop indicates the caller bypassed the resolver. Handlers fail closed with a coded PROCESSING_INTERNAL error on a nil pop; the resolver itself surfaces `PULSE_OVERLAY_REF_UNKNOWN` for the unknown- field case BEFORE this dispatch is reached.

Defense in depth: the descriptor.ValidateOverlays gate rejects bad kinds at predict time, so a missing dispatch entry should never reach the runtime in practice; nonetheless ApplyOverlaysFacet guards against an unknown kind and returns a CodedError whose details carry errors.PULSE_OVERLAY_KIND_UNKNOWN so the orchestrator surfaces the same failure mode predict would have flagged. The Level / Within belt-and-suspenders runtime gate the MATRIX path runs at `validateOverlayLevelWithinRuntime` does NOT fire here — FACET kinds in v1 do not engage prefix-bucket denominators, and the per-kind handlers add their own gates when their math requires it.

func ApplyOverlaysSeries added in v0.19.0 

func ApplyOverlaysSeries(specs []types.OverlaySpec, host *SeriesHostView) ([]types.OverlayLayer, []types.OverlayWarning, error)

ApplyOverlaysSeries executes every spec in specs against the SERIES host view and returns one OverlayLayer per spec in matching order (FR-A2: stable spec-order layer emission), plus a flat warning slice. Returns (nil, nil, nil) when specs is empty so the grouped Process orchestrator can call ApplyOverlaysSeries unconditionally — mirrors the MATRIX ApplyOverlays short-circuit contract.

Host policy:

  • When specs is empty, host may be nil. The dispatch short-circuits before touching the host.
  • When specs is non-empty, host may still be nil OR carry an empty group-key list — the dispatch passes the view straight through to the handler. Handlers see the empty group-key case the same way they see "every group has no value" and emit an empty Entries slice. The fold never panics on an empty host.
  • Loose alignment (FR-A2): when a handler's payload covers a strict subset of host group keys, the dispatch does NOT re-order or drop host keys. The handler is responsible for emitting one SeriesEntry per host group ordinal — entries for groups outside the handler's covered subset carry a zero-value Summary.

Defense in depth: the descriptor.ValidateOverlays gate rejects bad kinds at predict time, so a missing dispatch entry should never reach the runtime in practice; nonetheless ApplyOverlaysSeries guards against an unknown kind and returns a CodedError whose details carry errors.PULSE_OVERLAY_KIND_UNKNOWN so the orchestrator surfaces the same failure mode predict would have flagged. The Level / Within belt-and-suspenders runtime gate the MATRIX path runs at validateOverlayLevelWithinRuntime does NOT fire here — SERIES kinds do not engage prefix-bucket denominators, and the per-kind handlers add their own gates when their math requires it.

func ApplyOverlaysWithExtensions added in v0.19.0 

func ApplyOverlaysWithExtensions(specs []types.OverlaySpec, host *CrosstabHostView, exts *ExtensionRegistry) ([]types.OverlayLayer, []types.OverlayWarning, error)

ApplyOverlaysWithExtensions is the registry-aware sibling of ApplyOverlays. Identical contract to ApplyOverlays except that the supplied `*ExtensionRegistry` (when non-nil) participates in the per-spec FORMULA compile step — embedder-registered ExprFunctions become reachable from `OVERLAY_FORMULA` Params["formula"] expressions via the same `ExprOptions()` slice that ATTR_FORMULA / FILTER_EXPRESSION already consume. Non-FORMULA kinds ignore the registry; the per-kind runtime handler signature is intentionally NOT widened.

The buffered crosstab exit (`applyOverlaysToResponse`) routes every Request-host overlay fold through this entry point and forwards the Processor's live `*ExtensionRegistry` so a request that lists `OVERLAY_FORMULA` alongside any other kind picks up the registry's custom functions at compile time. The nil-registry no-extension behaviour is preserved when the caller does not have a registry on hand (tests, isolated handler exercises).

func AxisKeyPrefix added in v0.19.0 

func AxisKeyPrefix(key types.AxisKey, depth int) string

AxisKeyPrefix returns the first `depth+1` elements of an AxisKey joined as a composite key string using the canonical crosstabAxisKeySep separator. Mirrors the truncateCompositeKey shape the buffered crosstab orchestrator uses to address partial-margin buckets at depth `depth`.

`depth` is the level index (0-based) — the same value EffectiveAxisDepth returns. A depth >= len(key)-1 returns the full leaf composite key (no truncation happens). A negative depth treats the key as a single full-leaf composite (defensive guard so a pre-clamp value never produces a panic; callers should always run EffectiveAxisDepth first).

Used by the overlay handler dispatch to map leaf RowKeys[i] / ColumnKeys[j] tuples to the partial-prefix bucket the cell belongs to, so per-row / per-column denominators land on the same partial grouping the buffered crosstab `normalize_level` / `normalize_within` path would have used.

The composite key separator is internal to the processing package (crosstabAxisKeySep). Callers should not parse the returned string — use it only as a map key to compare prefix-bucket membership.

func CanChainRequest added in v0.10.0 

func CanChainRequest(req *types.Request, schema *encoding.Schema) bool

CanChainRequest reports whether a request is eligible to participate in a ProcessChain stage. A request qualifies iff:

  • It passes the existing mergeable gate (CanMergeRequest), i.e. its online state can be merged across partitions of the input.
  • Every aggregator emits a single scalar value per output row (FREQUENCY emits a map; MODE emits a string — both are excluded from chain output until a richer downstream-schema synth lands).

The chain executor calls this before each stage. A failing stage surfaces PULSE_CHAIN_NOT_MERGEABLE with the stage index in details.

func CanFuseCrosstab added in v0.17.0 

func CanFuseCrosstab(req *types.Request, schema *encoding.Schema, ext *ExtensionRegistry) (bool, string)

CanFuseCrosstab reports whether a crosstab request is eligible for the fused in-decode execution path that materializes per-cell state while iterating records — bypassing the full buffered Row materializa- tion the standard processCrosstab path uses today. Pure predicate: no side effects, no execution.

Naming mirrors CanMergeRequest / CanStreamRequest / CanChainRequest. The returned reason string is short and operator-specific so callers (dispatch in service/crosstab.go, predict surfaces in a follow-up) can surface a human-readable explanation without re-deriving the rule.

Eligibility = ALL of:

  • req.Crosstab != nil — nothing to fuse otherwise.
  • The cell aggregator is mergeable per AggregationType.Mergeable(). The fused path folds per-cell online state row-by-row; non- mergeable aggregators (median/percentile/zscore/skewness/kurtosis) need a finalize-time sorted view that the fused walk cannot provide.
  • The cell aggregator's MarginReducibility is MarginSummable or MarginMeanReducible. MarginRecompute aggregators force a re-scan of raw rows for margin derivation, which defeats the fused path by construction. The two non-recompute classes overlap exactly with the set of mergeable, scalar, cell-derived aggregators.
  • Every grouper on req.Crosstab.Rows ∪ req.Crosstab.Columns is constructable and the resulting instance implements StreamableGrouper (i.e. exposes a per-record KeyFor). The static types.GroupType.Streamable() table is too narrow here — it tracks Process-level streamability rather than per-record key derivation (GROUP_DATE is non-streamable at the Process layer but does implement StreamableGrouper.KeyFor and is therefore fusable). We consult the actual interface via a factory-probe to widen the gate while still rejecting truly key-non-derivable groupers (GROUP_QUANTILE).
  • No req.Features — every FEAT_* operator forces a buffered pre-filter pass that the fused path skips.
  • No req.Attributes of type ATTR_FORMULA with a non-empty Expression. Expression-runtime field extraction is conservative; #59 bail rules treat it as a forced widen, which the fused path can't honour while keeping per-field decode bounds tight.
  • No req.Filterers of type FILTER_EXPRESSION (same reason).
  • No req.Tests and no req.PostTests. Tier-1 row tests and tier-2 post-tests fold over the buffered row set after aggregation; the fused path doesn't buffer.
  • No extension-bound operator anywhere in the request without a registered FieldInputs hook. The fused path's projection bound is built from NeededFields; an opaque extension operator would widen the projection to "every field", which collapses the fused path's decode-cost advantage and is treated as ineligible here.
  • No mergeable-but-decimal aggregation target on the cell. Decimal- typed fields aggregate via AggregateDecimalField (the wide decimal path); Pulse forces buffered for those today and the fused gate mirrors that constraint.

Returns (true, "") for an eligible request. Returns (false, reason) for an ineligible one — the reason is intentionally short ("non- mergeable cell aggregator (AGG_MEDIAN)", "stat tests force buffered", "non-streamable grouper on column axis (GROUP_QUANTILE)", "ATTR_FORMULA bail", etc.).

The gate is a pure predicate. It does NOT modify req or schema, and it does NOT touch the orchestrator (RunCrosstab / processCrosstab). service/crosstab.go wires the dispatch around the result of this call.

func CanMergeRequest added in v0.8.0 

func CanMergeRequest(req *types.Request, schema *encoding.Schema) bool

CanMergeRequest reports whether a request's online state is mergeable across input partitions — the gate that the per-shard parallel reducer in service/shard_reduce.go consults before fanning out work across a worker pool. Returns true iff every aggregator, grouper, and filterer is mergeable AND the request contains no windows, no features, no regressions, no tests, no two-pass attributes, and no decimal-typed aggregation targets. A nil/empty aggregation list returns false (nothing to merge).

Mergeable is a strict subset of Streamable. Custom extension operators are conservatively treated as non-mergeable (the merge surface is not yet exposed on the extension registration struct).

func CanStreamRequest added in v0.2.0 

func CanStreamRequest(req *types.Request, schema *encoding.Schema) bool

CanStreamRequest is the exported parity hook used by tests in descriptor/ (which cannot import processing) to confirm that a PredictResult.Streamable value matches the runtime gate. Application code should rely on PredictResult.Streamable, not this helper.

func ChainOutputSchema added in v0.10.0 

func ChainOutputSchema(req *types.Request) (*encoding.Schema, error)

ChainOutputSchema synthesises the schema produced by a mergeable chain stage. The returned schema is purely in-memory — no byte layout — and is intended for downstream stages constructed via SliceIterator over RecordsFromChainRows.

Layout:

  • One categorical_u32 field per Group entry (named grp.Field), with an empty Dictionary that the caller populates as rows are materialized.
  • One f64 field per Aggregation entry (named via AggregationLabel).

Row-local attributes (FORMULA, DATE_PART) feed aggregators but do not surface in the streaming/grouped output today, so they do not appear in the chain schema. ByteOffset / BitPosition are zero across the board because the schema is never serialised.

func ComposeDefaultLabel added in v0.19.0 

func ComposeDefaultLabel(index int) string

ComposeDefaultLabel is the exported sibling of the package-internal composeDefaultLabel helper. The descriptor-side compose validator (descriptor.ValidateCompose) re-derives the same default slot label rule (`request_<i+1>`, 1-based) so a predict-time reference / target lookup matches the runtime's view of the slot labels; the per-helper sync test in descriptor/compose_test.go (TestComposeDescriptorDefaultLabel_MatchesProcessingHelper) pins the two implementations in lockstep so a future tweak on either side fails loud.

func CompositeAxisKey added in v0.12.0 

func CompositeAxisKey(parts []string) string

CompositeAxisKey joins a tuple of per-grouper key components into a single composite key string. Mirrors the composite-key shape used by PartitionByAxis so callers can index its Records map directly. The separator is internal — callers should not parse the output.

func EffectiveAxisDepth added in v0.19.0 

func EffectiveAxisDepth(axisDepth, level int) int

EffectiveAxisDepth clamps a caller-supplied depth (typically `OverlaySpec.Level`) into the valid `[0, axisDepth)` range and returns the effective leaf-or-prefix index the handler should use to truncate composite axis keys. Mirrors the CrosstabSpec NormalizeLevelOrLeaf clamp semantics (types/crosstab.go) so an overlay with Level=L and a crosstab with normalize_level=L compute against the same partial-axis depth.

Behaviour summary:

  • axisDepth <= 0 returns 0 (degenerate axis — no levels available; the caller must guard against this case independently).
  • level < 0 returns the leaf depth (axisDepth - 1). The on-wire omitempty zero-default makes Level=0 the "leaf" sentinel from the caller's perspective so this branch is the canonical "default to leaf" path.
  • level >= axisDepth is the predict-time PULSE_OVERLAY_LEVEL_OUT_OF_RANGE case — the runtime entry still gracefully clamps to the leaf index so a misconfigured caller that survives predict surfaces a degenerate-but-shaped overlay rather than a panic. The descriptor.ValidateOverlays predict gate is the authoritative rejection surface.

The leaf depth is `axisDepth - 1` (zero-based level index). For a 2-deep row axis (`Rows=[brand, region]`, axisDepth=2) the valid levels are {0, 1}; Level=0 truncates to the `brand` prefix and Level=1 hits the leaf `(brand, region)` tuple.

func EnableReuse added in v0.2.0 

func EnableReuse(iter RecordIterator)

EnableReuse opts the iterator into per-row Record reuse if it implements ReusableIterator. Safe no-op for iterators that do not support reuse (e.g. SliceIterator, whose records already exist as independent values).

func FormatJoinKindError added in v0.10.0 

func FormatJoinKindError(kind string) string

FormatJoinKindError surfaces a unified "unsupported kind" message for callers that want to print the rejection reason themselves. Unused today; reserved for the upcoming outer/left/anti landings.

func IsDecimalAggregationSupported added in v0.2.0 

func IsDecimalAggregationSupported(agg types.AggregationType) bool

IsDecimalAggregationSupported reports whether agg has a decimal-aware implementation.

func JoinedSchema added in v0.10.0 

func JoinedSchema(left, right *encoding.Schema, spec *types.JoinSpec) (*encoding.Schema, error)

JoinedSchema synthesises the schema produced by a single inner hash-join attached to a Request. Left fields appear unchanged; right fields are prefixed with spec.As (when set) and validated for collisions against the left side. The returned schema has no on-wire byte layout — like ChainOutputSchema, it exists to satisfy downstream operator lookups against an in-memory SliceIterator.

Returns PULSE_JOIN_FIELD_COLLISION when a non-prefixed right field shares a name with a left field. Callers can set spec.As to disambiguate.

func KindRequiresMatrix added in v0.19.0 

func KindRequiresMatrix(kind types.OverlayKind) bool

KindRequiresMatrix is the exported sibling of the package-internal kindRequiresMatrix predicate. The descriptor-side compose validator (descriptor.ValidateCompose) needs to read the catalog without dragging in processing's full overlay machinery, but the per-helper sync test (TestKindRequiresMatrixCompose_MatchesProcessing in descriptor/compose_test.go) pins the two surfaces in lockstep so a new matrix-required kind cannot land here without an accompanying descriptor-side row.

Body intentionally duplicated against the internal predicate so the exported surface can be lifted in isolation when the catalog grows; the package-internal call sites continue to read the lowercase form.

func LookupReference added in v0.19.0 

func LookupReference(byLabel map[string]*types.Response, label string, specIdx int) (*types.Response, error)

LookupReference resolves a `ComposeOverlaySpec.Reference` label against the byLabel map produced by ResolveComposeSlots. Returns a coded error carrying PULSE_OVERLAY_REFERENCE_UNKNOWN when the label is empty, absent from the map, or resolves to a nil *Response (the FailFast=false failed-slot hole).

The handler-side wrapping pattern: every per-kind COMPOSE handler calls LookupReference once at the top of its body and returns the coded error verbatim. Keeping the canonical-code dispatch on this single helper means handlers never re-code the "which arm" branching twice; the call site reads as a one-line guard.

`specIdx` is the originating ComposeOverlaySpec index in `ComposedRequest.Overlays`. The MCP fix-up surface uses the index to point the caller at the failing spec in their request body.

func LookupTarget added in v0.19.0 

func LookupTarget(byLabel map[string]*types.Response, label string, specIdx, targetIdx int) (*types.Response, error)

LookupTarget resolves a single `ComposeOverlaySpec.Targets[j]` label against the byLabel map produced by ResolveComposeSlots. Returns a coded error carrying PULSE_OVERLAY_TARGET_UNKNOWN when the label is empty, absent from the map, or resolves to a nil *Response (the FailFast=false failed-slot hole).

`specIdx` is the originating ComposeOverlaySpec index in `ComposedRequest.Overlays`; `targetIdx` is the index inside the spec's `Targets` slice. Both echo into the Details so the MCP fix-up surface can address the offending target by its (spec, position) coordinate.

func OppositeAxisPrefixDepth added in v0.19.0 

func OppositeAxisPrefixDepth(oppositeDepth, within int) int

OppositeAxisPrefixDepth returns the depth at which the OPPOSITE axis (the cross-axis the overlay does NOT centerpoint-lock to) should be fixed when the spec's Within slot engages the cross-axis-partition path. Mirrors SameAxisPrefixDepth's counting model — `Within=1` fixes the parent-prefix of the opposite axis (drops the last grouper); `Within=oppositeDepth-1` fixes the root-most prefix.

Returns 0 (= no cross-axis fixing) when Within <= 0, so the zero- default code path produces leaf-margin denominators byte-identical to the pre-Level/Within handler output (the SHARE_OF_ROW denominator stays "row margin sums across all columns").

The non-zero return is a prefix LENGTH (count of opposite-axis groupers retained), NOT a zero-based depth index — callers pass `length-1` to AxisKeyPrefix.

Out-of-range (Within >= oppositeDepth) is caught by the runtime validateOverlayLevelWithinRuntime gate; defensive clamp here keeps the helper panic-free.

func OverlayLevelEnabled added in v0.19.0 

func OverlayLevelEnabled(spec *types.OverlaySpec) bool

OverlayLevelEnabled reports whether a non-zero Level / Within slot on an OverlaySpec engages the prefix-axis denominator path. Mirrors the "zero-is-off" sentinel the OverlaySpec.Level / Within JSON tags already encode (`omitempty` omits the zero value on the wire). Used by the share / index / delta / zscore handler dispatch to skip the per-cell prefix-bucket precompute when the caller is on the legacy path — preserves the overlay-handler byte-identity guarantee that Default Level=0, Within=0 MUST be byte-equivalent to the no-Level/no-Within path.

Returns true when either slot is positive (non-zero). Negative slots (predict gate would reject; runtime gate would also reject) are treated as "not enabled" so the runtime handler fails closed on the gate before reaching the prefix-bucket code path.

func ResolveBaselineIndex added in v0.19.0 

func ResolveBaselineIndex(host *SeriesHostView, ref *types.OverlayBaselineIndexRef) (float64, error)

ResolveBaselineIndex resolves the OverlayBaselineIndexRef.Position arm against the host's ordered SERIES axis. Returns the baseline metric value the host produced for the group at that ordinal plus a nil error on success, or a PULSE_OVERLAY_REF_UNKNOWN-coded CodedError on failure.

Two rejection arms:

  • Negative Position (`ref.Position < 0`): the resolver returns `(0, CodedError)` with code `PULSE_OVERLAY_REF_UNKNOWN` and details `{baseline_index: <n>, series_length: <len>}`. A 0-based ordinal cannot resolve to a negative slot — the caller likely constructed the ref with an uninitialised default that wandered negative.

  • Position outside the series (`ref.Position >= host.GroupCount()`): same code and details shape. The resolver does NOT defend against a missing host group at that ordinal (the orchestrator skipped the group); the host's own resolver surfaces `(0, false)` at `host.ValueAt(i)` and the caller (the per-kind handler) decides whether absent-baseline emits a layer-level PULSE_OVERLAY_REF_ZERO or a per-entry NaN-statistic shape. The resolver's job is structural — it ensures the requested ordinal falls inside the host's declared range and lets the host's present-flag surface the metric.

Defense-in-depth returns:

  • `host == nil`: returns `(0, CodedError)` with the same PULSE_OVERLAY_REF_UNKNOWN code and `{baseline_index: <n>, series_length: 0}` details so the caller does not crash on a nil-host fold (the dispatch should never feed a nil host but the resolver stays robust).
  • `ref == nil`: returns `(0, nil)` — the caller's responsibility is to consult `ref` before invoking the resolver. The function does not raise on the empty case so an overlay handler can short-circuit cleanly when no baseline ref was supplied (other ref families take over).

Deterministic + order-stable: the function calls `host.ValueAt(i)` once and returns whatever the host produced. Same series + same Position → byte-equal return on every invocation. The resolver does NOT alter the host or fold any state.

Forward-compat (Row / Column arms): the MATRIX-host baseline-cell shape (`ref.Row` + `ref.Column`) is reserved for a future Crosstab overlay family — `ResolveBaselineIndex` ONLY consumes `ref.Position` and ignores the Row / Column slots. The MATRIX-arm resolver will land alongside the future "vs designated cell" kind (per PRD §4) and live in its own file.

func ResolveComposeSlots added in v0.19.0 

func ResolveComposeSlots(req *types.ComposedRequest, responses []*types.Response) (map[string]*types.Response, map[string]int, error)

ResolveComposeSlots builds the slot-name → *types.Response lookup the COMPOSE-host overlay dispatch consumes. Walks `req.Requests` in index order and produces two parallel maps keyed by the final (post-auto-default) slot label:

  • byLabel maps each final label to the *types.Response at the same slot index. The value MAY be nil — under FailFast=false the orchestrator passes a partial responses slice with nil holes for failed slots, and the resolver preserves those holes so the downstream LookupReference / LookupTarget helpers can surface a coded error (rather than silently skipping the slot).

  • byIndex maps each final label to the originating slot index in `req.Requests`. Handlers carry this through to warning Details so MCP fix-ups can address the offending slot by its ComposedRequest.Requests[i] position.

Final label rule (mirrors applyComposeLabelDefaults):

  • Non-empty `req.Requests[i].Label` → used verbatim.
  • Empty `req.Requests[i].Label` → `request_<i+1>` (1-based).
  • Nil *Request slot → skipped. The slot index is NOT registered in either map; downstream Reference / Target lookups against a nil slot surface PULSE_OVERLAY_REFERENCE_UNKNOWN / PULSE_OVERLAY_TARGET_UNKNOWN at handler time (the same code path an unknown caller-supplied label takes).

Returns a coded error when:

  • len(req.Requests) != len(responses) — orchestrator-side invariant violation. PROCESSING_INTERNAL.

  • Two non-nil slots resolve to the same final label after auto-default. PULSE_COMPOSE_LABEL_COLLISION. The applyComposeLabelDefaults normaliser rejects this at the pre-barrier validate, so this branch is the defense in depth for callers that reach the resolver without going through the normaliser (descriptor predict, MCP, tests).

Returns (nil, nil, nil) when req is nil or req.Requests is empty — matches the chassis short-circuit shape so callers that compute the lookup unconditionally still see a clean zero-allocation path.

func SameAxisPrefixDepth added in v0.19.0 

func SameAxisPrefixDepth(axisDepth, level int) int

SameAxisPrefixDepth returns the depth at which the SAME axis (the one the overlay is centerpoint-locked to) should be truncated when the spec's Level slot engages the prefix-denominator path. Counts from the leaf — `Level=1` truncates to the parent prefix (drops the last grouper); `Level=axisDepth-1` truncates to the root-most prefix (keeps only the first grouper). The returned depth is the prefix LENGTH (count of groupers retained), NOT a zero-based depth index — callers pass this directly to AxisKeyPrefix's `depth` argument as `length-1`.

Returns axisDepth (= no truncation, full leaf) when Level <= 0, so the zero-default code path produces leaf-margin denominators byte- identical to the pre-Level/Within handler output.

Out-of-range (Level >= axisDepth) is caught by the runtime validateOverlayLevelWithinRuntime gate before this function is reached; the helper clamps defensively to the [1, axisDepth] range to keep a misconfigured caller out of a panic path.

func StreamabilityKey added in v0.7.0 

func StreamabilityKey(category, name string) string

StreamabilityKey is the canonical map key for ExtensionRegistry.Streamable. Format: "<category>|<name>". The category strings are the lowercase singular forms used in error details and the manifest: aggregator, attribute, filterer, grouper, window, feature, test.

func WelfordStdDev added in v0.19.0 

func WelfordStdDev(values []float64) float64

WelfordStdDev computes the population standard deviation of values using the single-pass Welford recurrence over central moment M2: 

mean_n = mean_{n-1} + (x_n - mean_{n-1}) / n
M2_n   = M2_{n-1}   + (x_n - mean_{n-1}) * (x_n - mean_n)

and finalises sd = sqrt(M2 / n).

Population (divide by N) — not sample (divide by N-1). Overlay margins describe the FULL population of cells contributing to a margin slice (every row cell within the row-margin slice, every column cell within the column-margin slice, every matrix cell within the grand-margin slice). A z-score against the full population uses the population sd; the sample sd would imply we are inferring a wider population from a sample, which is not the overlay's intent.

Returns 0 for len(values) == 0 to keep callers branch-free. Returns 0 for len(values) == 1 because a single-observation slice has undefined variance — callers (e.g. ZSCORE_VS_MARGIN) treat the zero return identically to a zero-denominator slice and surface the matching PULSE_OVERLAY_REF_ZERO warning.

Why this single function rather than the per-bucket welfordBucket machinery in test_t.go / test_z.go / test_anova_welch.go: those pre-existing helpers are coupled to per-key state across the streaming Process path (groups map, insertion order, sample-vs- population variance toggles). The overlay handler walks a closed margin slice synchronously and needs only a single number out; a purpose-built helper keeps the surface narrow and the call sites readable without re-deriving the recurrence inline. Future overlays that need streaming Welford may either reuse this helper against an accumulated slice or be lifted alongside the existing per-bucket state — both options remain open.

Types

type Aggregator 

type Aggregator interface {
	// Aggregate computes the aggregation over the given records for the named field.
	Aggregate(records []*Record, field string) (float64, error)
}

Aggregator computes a single aggregate value from a set of records.

type AggregatorFactory 

type AggregatorFactory func(agg *types.Aggregation, schema *encoding.Schema) (Aggregator, error)

AggregatorFactory creates an Aggregator from a type specification.

type AttributeComputer 

type AttributeComputer interface {
	// Compute calculates derived values for all records, returning a value per record.
	// The returned slice has one entry per record. A nil second return means no null handling.
	Compute(records []*Record, field string) ([]float64, error)
}

AttributeComputer computes derived attribute values for each record.

type AttributeFactory 

type AttributeFactory func(attr *types.Attribute, schema *encoding.Schema) (AttributeComputer, error)

AttributeFactory creates an AttributeComputer from a type specification.

type BitsetSet added in v0.8.3 

type BitsetSet struct {
	// contains filtered or unexported fields
}

BitsetSet stores membership as a packed bit per dictionary id.

Memory: dictSize bits, rounded up to a uint64 word. A categorical_u32 dict at its 4 294 967 295 cap would be ~512 MiB, but in practice categorical dicts run much smaller — a 65 536-entry categorical_u16 is 8 KiB.

func (*BitsetSet) Add added in v0.8.3 

func (b *BitsetSet) Add(id uint32)

Add marks id as present. Idempotent.

func (*BitsetSet) Contains added in v0.8.3 

func (b *BitsetSet) Contains(id uint32) bool

Contains reports whether id is set. Out-of-range ids return false without panicking — the loader can build a bitset sized to the dictionary, but records read against a later, larger dictionary on a shard archive could in principle present an id beyond the bitset.

func (*BitsetSet) Kind added in v0.8.3 

func (b *BitsetSet) Kind() string

func (*BitsetSet) Len added in v0.8.3 

func (b *BitsetSet) Len() int

type CrosstabAxisPartition added in v0.12.0 

type CrosstabAxisPartition struct {
	// Keys is the sorted list of composite-key strings (one per axis
	// tuple). compositeAxisKey(Tuples[i]) == Keys[i].
	Keys []string
	// Tuples is the parallel list of axis tuples, one per Key. Each
	// tuple has len(axis) entries — one per axis grouper, in axis
	// order. Categorical / numeric / date keys are kept as the grouper
	// emitted them (always strings via Grouper.Group's contract).
	Tuples []types.AxisKey
	// Records maps composite key → record bucket for that axis tuple.
	Records map[string][]*Record
}

CrosstabAxisPartition is the result of recursively partitioning a record set by an ordered list of groupers (rows or columns).

type CrosstabHostView added in v0.19.0 

type CrosstabHostView struct {
	// contains filtered or unexported fields
}

CrosstabHostView is the read-only window an overlay handler uses to fold a derived projection over a fully-materialised MatrixPayload. Exposing margins through a view (rather than recomputing) is the PRD §6 buffered contract: "O(cells × layers) per PRD §6 — handlers never re-scan records".

Construction: NewCrosstabHostView(payload). The view holds a pointer to the underlying payload; lifetime is the caller's responsibility and the view does not mutate the payload.

All margin lookups are byte-equal to the matching slot on the underlying MatrixPayload — handlers read what the buffered crosstab orchestrator already computed and emitted, never re-deriving the math. RowKeys / ColumnKeys / Cells dimensions echo the payload verbatim so the handler addresses cells by integer index pairs.

func NewCrosstabHostView added in v0.19.0 

func NewCrosstabHostView(payload *types.MatrixPayload) *CrosstabHostView

NewCrosstabHostView wraps a MatrixPayload as a host view. payload must be non-nil; the view does not copy the payload, callers must not mutate it during overlay execution. The components block is left nil — use NewCrosstabHostViewWithComponents for overlays that read per-cell counters / Welford triples.

func NewCrosstabHostViewWithComponents added in v0.24.0 

func NewCrosstabHostViewWithComponents(payload *types.MatrixPayload, comps *types.CrosstabComponents) *CrosstabHostView

NewCrosstabHostViewWithComponents wraps a MatrixPayload together with its Response.Components.Crosstab block. comps may be nil (components disabled); component-reading handlers surface PULSE_OVERLAY_COMPONENTS_REQUIRED in that case rather than dividing by a zero sample size. The view does not copy either pointer; callers must not mutate them during overlay execution.

func (*CrosstabHostView) CellAt added in v0.19.0 

func (h *CrosstabHostView) CellAt(rowIdx, colIdx int) (float64, bool)

CellAt returns the scalar form of the host cell at (rowIdx, colIdx) plus a present flag mirroring MatrixCell.Present. Returns (0, false) when the indices are out of range or the cell is structurally missing. Map-valued cells (RichAggregator outputs) fall through as (0, false) because the index-vs-margin handler is scalar-only — the validator already rejects map-valued cell aggregators paired with overlays that need division (the PULSE_CROSSTAB_NORMALIZE_MAP_VALUED gate covers the normalize=row case; INDEX_VS_MARGIN inherits the same shape).

func (*CrosstabHostView) CellComponentFloat added in v0.24.0 

func (h *CrosstabHostView) CellComponentFloat(rowIdx, colIdx int, key string) (float64, bool)

CellComponentFloat reads a numeric key from CellComponents[rowIdx][colIdx]. Returns (0, false) when components are absent or the slot/key is missing or non-numeric.

func (*CrosstabHostView) CellN added in v0.24.0 

func (h *CrosstabHostView) CellN(rowIdx, colIdx int) (int, bool)

CellN reads the universal-floor "n" counter from CellComponents[r][c].

func (*CrosstabHostView) CellWeightSum added in v0.24.0 

func (h *CrosstabHostView) CellWeightSum(rowIdx, colIdx int) (float64, bool)

CellWeightSum reads the "sum_weights" key from CellComponents[r][c] (emitted by AGG_WEIGHTED_MEAN). Used as the prop-Z sample-size leg when the cell aggregator is weighted so n matches the weighted denominator.

func (*CrosstabHostView) ColumnAxisDepth added in v0.19.0 

func (h *CrosstabHostView) ColumnAxisDepth() int

ColumnAxisDepth returns the per-column tuple length (the count of column groupers the host crosstab declared). Reads `len(ColumnKeys[0])` when the host has at least one column; zero otherwise. Used by the overlay Level / Within handlers to clamp configured depth values into the valid `[0, ColumnAxisDepth())` range via EffectiveAxisDepth (see processing/crosstab_normalize.go).

func (*CrosstabHostView) ColumnCount added in v0.19.0 

func (h *CrosstabHostView) ColumnCount() int

ColumnCount returns the number of column tuples on the host. Zero when the view is nil or the underlying payload is nil.

func (*CrosstabHostView) ColumnMarginN added in v0.24.0 

func (h *CrosstabHostView) ColumnMarginN(colIdx int) (int, bool)

ColumnMarginN reads the per-column margin record count from CrosstabComponents.ColumnMarginCounts.

func (*CrosstabHostView) ColumnSlabN added in v0.24.0 

func (h *CrosstabHostView) ColumnSlabN(rowIdx, colIdx, prefix int) (int, bool)

ColumnSlabN sums CellCounts at (rowIdx, c') over every column c' whose ColumnKey agrees with colIdx on the first `prefix` dim positions — the fixed-prefix subgroup that is the denominator under normalize=row when NormalizeWithin = prefix-1 (Pulse-W convention). Returns (0, false) when components/CellCounts are absent; (sum, true) otherwise (sum may be 0).

func (*CrosstabHostView) Components added in v0.24.0 

func (h *CrosstabHostView) Components() *types.CrosstabComponents

Components returns the underlying CrosstabComponents pointer (nil when components were disabled). Read-only.

func (*CrosstabHostView) HasComponents added in v0.24.0 

func (h *CrosstabHostView) HasComponents() bool

HasComponents reports whether the host carries a components block. Component-reading handlers gate on this once up front and bail with PULSE_OVERLAY_COMPONENTS_REQUIRED when false.

func (*CrosstabHostView) HasWelfordCells added in v0.24.0 

func (h *CrosstabHostView) HasWelfordCells() bool

HasWelfordCells reports whether at least one present cell carries a full {mean, variance, n} triple. The Welford-input handlers gate on this so a matrix whose cell aggregator is not AGG_WELFORD fails fast with a shape error instead of skipping every pair.

func (*CrosstabHostView) MarginFor added in v0.19.0 

func (h *CrosstabHostView) MarginFor(axis types.MarginAxis, rowIdx, colIdx int) (float64, bool)

MarginFor resolves the margin slot named by axis at the given cell coordinates and returns (value, present). RowIdx is ignored when axis = column or grand; colIdx is ignored when axis = row or grand. An unknown axis returns (0, false); the validator rejects bad axes at predict time so this branch should not fire in practice (defense in depth per CLAUDE.md "Predict / Inspect contracts").

Layout:

  • MarginAxisRow → payload.RowMargins[rowIdx]
  • MarginAxisColumn → payload.ColumnMargins[colIdx]
  • MarginAxisGrand → payload.GrandTotal

Missing or absent slots (no margin computed because the crosstab spec did not request and did not need them) return (0, false) so the handler can surface the matching PULSE_OVERLAY_REF_ZERO warning rather than silently divide by zero.

func (*CrosstabHostView) Payload added in v0.19.0 

func (h *CrosstabHostView) Payload() *types.MatrixPayload

Payload returns the underlying MatrixPayload pointer so handlers can read RowKeys / ColumnKeys headers when shaping their output payload. Read-only — handlers must not mutate the returned payload.

func (*CrosstabHostView) RowAxisDepth added in v0.19.0 

func (h *CrosstabHostView) RowAxisDepth() int

RowAxisDepth returns the per-row tuple length (the count of row groupers the host crosstab declared). Reads `len(RowKeys[0])` when the host has at least one row; zero otherwise. Used by the overlay Level / Within handlers to clamp configured depth values into the valid `[0, RowAxisDepth())` range via EffectiveAxisDepth (see processing/crosstab_normalize.go).

func (*CrosstabHostView) RowCount added in v0.19.0 

func (h *CrosstabHostView) RowCount() int

RowCount returns the number of row tuples on the host. Zero when the view is nil or the underlying payload is nil.

func (*CrosstabHostView) RowMarginN added in v0.24.0 

func (h *CrosstabHostView) RowMarginN(rowIdx int) (int, bool)

RowMarginN reads the per-row margin record count from CrosstabComponents.RowMarginCounts.

func (*CrosstabHostView) RowSlabN added in v0.24.0 

func (h *CrosstabHostView) RowSlabN(rowIdx, colIdx, prefix int) (int, bool)

RowSlabN is the row-axis analog of ColumnSlabN: sums CellCounts at (r', colIdx) over rows r' whose RowKey matches rowIdx's on the first `prefix` dim positions. Used by row-axis pairwise specs with n_within.

func (*CrosstabHostView) WelfordTriple added in v0.24.0 

func (h *CrosstabHostView) WelfordTriple(rowIdx, colIdx int) (mean, variance float64, n int, ok bool)

WelfordTriple reads {mean, variance, n} from CellComponents[r][c]. Used by OVERLAY_PAIRWISE_WELCH_T and OVERLAY_PAIRWISE_TWO_MEANS_Z. Returns ok=false unless all three keys are present and numeric.

type CrosstabValidationError added in v0.12.0 

type CrosstabValidationError = errors.CodedError

CrosstabValidationError represents a structural failure surfaced by pre-execution validation in RunCrosstab. The error wraps a CodedError so callers see the same PULSE_CROSSTAB_* code regardless of entry point.

type ExprFunction added in v0.7.0 

type ExprFunction struct {
	Name string
	Fn   any
}

ExprFunction is the runtime-side mirror of pulse.ExprFunction. The processor injects every entry into the expr-lang environment when compiling ATTR_FORMULA / FILTER_EXPRESSION expressions.

type ExtensionAware added in v0.7.0 

type ExtensionAware interface {
	SetExtensions(r *ExtensionRegistry)
}

ExtensionAware is the optional interface that AttributeComputer / FiltererBuilder instances implement when they want the Processor to inject the live ExtensionRegistry after construction. The formula attribute and expression filterer use this hook to expose embedder-registered ExprFunctions and LookupTables to the expr environment.

type ExtensionRegistry added in v0.7.0 

type ExtensionRegistry struct {
	Aggregators map[types.AggregationType]AggregatorFactory
	Attributes  map[types.AttributeType]AttributeFactory
	Filterers   map[types.FiltererType]FiltererFactory
	Groupers    map[types.GroupType]GrouperFactory
	Windows     map[types.WindowType]window.WindowFactory
	Features    map[types.FeatureType]feature.Factory
	RowTests    map[types.TestType]RowTestFactory
	PostTests   map[types.TestType]PostTestFactory

	// Streamable is the per-(category, name) override consulted by
	// IsStreamable. Built-in entries are not stored here; the
	// fallback path consults the per-type Streamable() method.
	Streamable map[string]bool

	// ExprFunctions are merged into the runtime expression environment
	// used by ATTR_FORMULA and FILTER_EXPRESSION. Each entry is
	// callable from a request expression under its declared Name. The
	// shape mirrors pulse.ExprFunction (pulse package can't be
	// imported here without a cycle) — the public pulse.ExprFunction
	// is the canonical embedder-facing surface.
	ExprFunctions []ExprFunction

	// LookupTables are exposed to the runtime expression environment
	// via the built-in lookup() function. Mirrors pulse.LookupTable.
	LookupTables map[string]LookupTable

	// LabelTables are the string-valued ID→label maps consumed by the
	// output-time label resolver. Mirrors pulse.LabelTable. Indexed
	// by the user-facing table name.
	LabelTables map[string]LabelTable

	// FieldInputs is the per-(category, name) field-introspection
	// callback consulted by the buffered-projection extractor
	// (NeededFields). When the registry contains an entry for an
	// extension operator the projection extractor calls the callback
	// with the operator's raw Params; otherwise the extractor widens
	// the projection to "every field" so the runtime stays correct
	// for embedders that haven't opted in.
	FieldInputs map[string]FieldInputsFunc
}

ExtensionRegistry holds per-Service overlays for every operator category that supports the public extension API. A nil receiver is the no-extension case — all Lookup methods fall through to the built-in package-level registries.

The overlay maps are read-only after pulse.New populates them; the runtime never mutates them. Two Service instances with different Extensions inputs hold distinct ExtensionRegistry values, so a process can host more than one Pulse with disjoint extension sets.

func (*ExtensionRegistry) CustomAggregatorNames added in v0.7.0 

func (r *ExtensionRegistry) CustomAggregatorNames() []types.AggregationType

CustomAggregatorNames returns the overlay-only aggregator names in sorted-ish order (map iteration; callers sort if needed). Used by manifest emission to list extension-shipped operators separately from built-ins.

func (*ExtensionRegistry) CustomAttributeNames added in v0.7.0 

func (r *ExtensionRegistry) CustomAttributeNames() []types.AttributeType

func (*ExtensionRegistry) CustomFeatureNames added in v0.7.0 

func (r *ExtensionRegistry) CustomFeatureNames() []types.FeatureType

func (*ExtensionRegistry) CustomFiltererNames added in v0.7.0 

func (r *ExtensionRegistry) CustomFiltererNames() []types.FiltererType

func (*ExtensionRegistry) CustomGrouperNames added in v0.7.0 

func (r *ExtensionRegistry) CustomGrouperNames() []types.GroupType

func (*ExtensionRegistry) CustomPostTestNames added in v0.7.0 

func (r *ExtensionRegistry) CustomPostTestNames() []types.TestType

CustomPostTestNames returns the tier-2 overlay-only test names.

func (*ExtensionRegistry) CustomRowTestNames added in v0.7.0 

func (r *ExtensionRegistry) CustomRowTestNames() []types.TestType

CustomRowTestNames returns the tier-1 overlay-only test names.

func (*ExtensionRegistry) CustomWindowNames added in v0.7.0 

func (r *ExtensionRegistry) CustomWindowNames() []types.WindowType

func (*ExtensionRegistry) ExprOptions added in v0.7.0 

func (r *ExtensionRegistry) ExprOptions() []expr.Option

ExprOptions returns the expr-lang Option slice an ExtensionRegistry contributes to ATTR_FORMULA / FILTER_EXPRESSION compilation. A nil receiver still returns the built-in set helpers; non-nil receivers layer their custom ExprFunctions and (when LookupTables are registered) the `lookup` builtin on top.

Contributions:

  • set helpers (`contains`, `has_any`, `has_all`, `has_none`, `popcount`, `set_union`, `set_intersect`, `set_diff`, `set_xor`) are always registered. Record.AllValues() resolves set fields to []string of labels, so the helpers consume []string operands.
  • One expr.Function per ExprFunctions entry, registered under its declared Name.
  • The built-in `lookup(table, keys...)` function when at least one LookupTable is registered.

Expressions in cohorts with no lookup tables registered still reference `lookup` legally — the function returns PULSE_LOOKUP_TABLE_UNKNOWN at evaluation time. The compile-time typecheck catches static misuse.

func (*ExtensionRegistry) FeatureFactories added in v0.7.0 

func (r *ExtensionRegistry) FeatureFactories() map[types.FeatureType]feature.Factory

FeatureFactories returns the overlay map for feature operators. Nil-receiver-safe.

func (*ExtensionRegistry) FieldInputsFor added in v0.8.0 

func (r *ExtensionRegistry) FieldInputsFor(category, name string, raw json.RawMessage) ([]string, bool)

FieldInputsFor consults the FieldInputs overlay for the operator identified by (category, name). Returns (inputs, true) when the registration supplied a callback, ([], true) when no extra fields are read, or (nil, false) when the operator is custom but has no registered callback — caller should treat that as "can't introspect" and widen the projection.

Built-in operators are not stored in this map; callers should only reach FieldInputsFor for extension-resolved operators.

func (*ExtensionRegistry) HasAggregator added in v0.7.0 

func (r *ExtensionRegistry) HasAggregator(t types.AggregationType) bool

HasAggregator reports whether name resolves either via overlay or built-in registry. Same shape for the remaining categories.

func (*ExtensionRegistry) HasAttribute added in v0.7.0 

func (r *ExtensionRegistry) HasAttribute(t types.AttributeType) bool

func (*ExtensionRegistry) HasFeature added in v0.7.0 

func (r *ExtensionRegistry) HasFeature(t types.FeatureType) bool

func (*ExtensionRegistry) HasFilterer added in v0.7.0 

func (r *ExtensionRegistry) HasFilterer(t types.FiltererType) bool

func (*ExtensionRegistry) HasGrouper added in v0.7.0 

func (r *ExtensionRegistry) HasGrouper(t types.GroupType) bool

func (*ExtensionRegistry) HasPostTest added in v0.7.0 

func (r *ExtensionRegistry) HasPostTest(t types.TestType) bool

func (*ExtensionRegistry) HasRowTest added in v0.7.0 

func (r *ExtensionRegistry) HasRowTest(t types.TestType) bool

func (*ExtensionRegistry) HasWindow added in v0.7.0 

func (r *ExtensionRegistry) HasWindow(t types.WindowType) bool

func (*ExtensionRegistry) IsStreamable added in v0.7.0 

func (r *ExtensionRegistry) IsStreamable(category, name string) bool

IsStreamable consults the Streamable overlay first, then the built-in per-type Streamable() method. Unknown categories return false. The cross-check between this and the runtime path lives in the per-category integration phases.

func (*ExtensionRegistry) LookupAggregator added in v0.7.0 

func (r *ExtensionRegistry) LookupAggregator(t types.AggregationType) (AggregatorFactory, bool)

LookupAggregator returns the factory for an aggregator type. The overlay map wins over the built-in registry. The boolean second return is the standard "found" signal.

func (*ExtensionRegistry) LookupAttribute added in v0.7.0 

func (r *ExtensionRegistry) LookupAttribute(t types.AttributeType) (AttributeFactory, bool)

LookupAttribute returns the factory for an attribute type. Overlay wins.

func (*ExtensionRegistry) LookupFeature added in v0.7.0 

func (r *ExtensionRegistry) LookupFeature(t types.FeatureType) (feature.Factory, bool)

LookupFeature returns the factory for a feature type. Overlay wins. Falls through to feature.Lookup for built-ins.

func (*ExtensionRegistry) LookupFilterer added in v0.7.0 

func (r *ExtensionRegistry) LookupFilterer(t types.FiltererType) (FiltererFactory, bool)

LookupFilterer returns the factory for a filterer type. Overlay wins.

func (*ExtensionRegistry) LookupGrouper added in v0.7.0 

func (r *ExtensionRegistry) LookupGrouper(t types.GroupType) (GrouperFactory, bool)

LookupGrouper returns the factory for a grouper type. Overlay wins.

func (*ExtensionRegistry) LookupPostTest added in v0.7.0 

func (r *ExtensionRegistry) LookupPostTest(t types.TestType) (PostTestFactory, bool)

LookupPostTest returns the tier-2 post-test factory for a test type. Overlay wins.

func (*ExtensionRegistry) LookupRowTest added in v0.7.0 

func (r *ExtensionRegistry) LookupRowTest(t types.TestType) (RowTestFactory, bool)

LookupRowTest returns the tier-1 row-test factory for a test type. Overlay wins.

func (*ExtensionRegistry) LookupWindow added in v0.7.0 

func (r *ExtensionRegistry) LookupWindow(t types.WindowType) (window.WindowFactory, bool)

LookupWindow returns the factory for a window type. Overlay wins. Falls through to window.Lookup for built-ins.

func (*ExtensionRegistry) WindowFactories added in v0.7.0 

func (r *ExtensionRegistry) WindowFactories() map[types.WindowType]window.WindowFactory

WindowFactories returns the overlay map for window operators, or nil when the registry has no entries. Safe to call on a nil receiver — the runtime treats nil as "no overlay" and falls through to the built-in window registry.

type FacetPopulationView added in v0.19.0 

type FacetPopulationView struct {
	// contains filtered or unexported fields
}

FacetPopulationView is the read-only window a Facet overlay handler uses to read population statistics for a single overlay field. The view holds a pointer to the underlying FacetResult and a resolved *FacetField cached at construction so per-call lookups are constant-time map walks rather than repeated map probes.

Construction: ResolveFacetPopulation(result, field). The view holds a pointer to the underlying FacetResult; lifetime is the caller's responsibility and the view does not mutate the payload.

All lookups read what FacetSchema already computed and emitted — the view never re-derives counts, never re-runs Welford, never recomputes percentiles. The four handlers read what is already on the wire.

func ResolveFacetPopulation added in v0.19.0 

func ResolveFacetPopulation(result *types.FacetResult, field string) (*FacetPopulationView, error)

ResolveFacetPopulation resolves a population reference against a finalized FacetResult and returns `(view, err)`. The view exposes per-kind lookups (categorical fast path + numeric streaming-state path) so the four Facet overlay handlers can consume one consistent surface.

Two rejection arms — both return `(nil, CodedError)` with code `PULSE_OVERLAY_REF_UNKNOWN`:

  • `result == nil` OR `result.Fields == nil`: the host did not produce a FacetResult the resolver can consume. Details carry `{field, available_fields: []}`.

  • `field` is absent from `result.Fields`: the overlay named a population field that was not part of the FacetRequest.Fields slice (the canonical "unknown reference field" arm for this kind). Details carry `{field, available_fields: [<sorted field names>]}`.

Defense-in-depth returns:

  • `field == ""`: same code with `{field: ""}` Details so the caller does not crash on a missing reference. The descriptor validator should reject empty field names at predict time; the runtime resolver stays robust.

  • `result.Fields[field] == nil`: same shape as the absent-field arm (a nil entry under a present key is structurally equivalent to a missing key for the resolver's purposes).

Deterministic + order-stable: the resolver does no folding, no I/O — the same FacetResult + same field name produces the same view + the same coded-error shape on every invocation. The view itself is a pointer-equal wrapper — handlers may compare views via pointer identity safely.

func (*FacetPopulationView) DiscreteCount added in v0.19.0 

func (v *FacetPopulationView) DiscreteCount(value string) (int64, bool)

DiscreteCount returns the per-value count for a single value in the categorical population. Returns `(count, true)` when the value resolves; `(0, false)` when the view is nil / non-discrete / missing payload, OR when the value is not present in the discrete payload. Distinct from DiscreteFrequency in that the count is returned without dividing by TotalRecords — handlers that need the raw observed-vs- expected contingency (CHISQ_VS_POP) read this surface; handlers that want a normalized index (INDEX_VS_POP) read DiscreteFrequency.

func (*FacetPopulationView) DiscreteCounts added in v0.19.0 

func (v *FacetPopulationView) DiscreteCounts() ([]types.FacetValueCount, bool)

DiscreteCounts returns the per-value `(value, count)` tuple slice from the underlying FacetField.Discrete.Values when the view resolves a categorical / boolean field. Returns `(nil, false)` when the view is nil, when the resolved field is non-discrete, or when the discrete payload is missing.

The slice is the canonical population contingency the per-value overlay kinds read from (CHISQ_VS_POP folds the observed cohort distribution against this slice; INDEX_VS_POP divides per-value observed by per-value population). Sort order matches `FacetField.Discrete.Values` (descending by count, ties ascending by value string) — handlers can iterate in payload order without re-sorting.

Read-only: handlers must not mutate the returned slice. The view returns the underlying slice pointer (no defensive copy) to keep the hot path allocation-free; if a handler needs to mutate it must clone first.

func (*FacetPopulationView) DiscreteFrequency added in v0.19.0 

func (v *FacetPopulationView) DiscreteFrequency(value string) (float64, bool)

DiscreteFrequency returns the per-value frequency for a single value in the categorical population — `count / total_records` where `total_records` is the view's `TotalRecords()`. Returns `(frequency, true)` when the value resolves and the denominator is non-zero. Returns `(0, false)` when:

  • the view is nil OR the resolved field is non-discrete OR the discrete payload is missing;
  • the value is not present in the discrete payload (legitimate "value never observed in the population" — the caller surfaces a handler-defined zero / NaN / warning depending on kind);
  • `TotalRecords()` is zero (the population is empty — handlers surface a layer-level zero-denominator warning the same way the existing SERIES grand-total kinds do).

Lookup is O(n) over the discrete payload; n is the truncated top-K slice length (capped per FacetRequest.DiscreteTopK or by the dictionary cardinality otherwise — typically small). The handlers can cache a value→count map externally if their per-cell hot path warrants it; this resolver stays allocation-free for the common single-lookup case.

func (*FacetPopulationView) DiscreteFrequencyStdev added in v0.19.0 

func (v *FacetPopulationView) DiscreteFrequencyStdev() (float64, bool)

DiscreteFrequencyStdev returns the population standard deviation of the per-category frequencies (count / TotalRecords) over the resolved discrete payload. Returns `(sd, true)` when the resolved field is discrete and the denominator is non-zero; `(0, false)` otherwise (view is nil, non-discrete, missing payload, or empty counts slice).

This is the per-category frequency-SD that the Facet-host z-score overlay (OVERLAY_ZSCORE_VS_POP) reads as `sd_pop`. The implementation walks the already-resolved `DiscreteCounts()` slice (the per-value (value, count) tuples FacetSchema folded into the payload) and uses the canonical Welford-Pébaÿ recurrence over those frequencies — one pass, allocation-free, byte-equal to the recurrence used by `WelfordStdDev` (`processing/welford.go`) for any other population-SD computation. POPULATION SD (divide by N, not N-1) is the correct convention because the per-category frequency set IS the whole population being summarised — mirrors the `OVERLAY_ZSCORE_VS_TOTAL` variance-choice rationale.

Single-entry payload (N=1): the recurrence is well-defined and returns `(0, true)` — every frequency equals the single observed value's frequency so the variance is exactly zero. Callers (the ZSCORE_VS_POP handler) then route every entry through the PULSE_OVERLAY_REF_ZERO arm. Distinct from the unknown-shape case `(0, false)` which means the view itself is not discrete-enabled.

Read-only: the function does not mutate the underlying payload. O(n) over the discrete payload; n is the truncated top-K slice length (capped per FacetRequest.DiscreteTopK or by the dictionary cardinality otherwise — typically small).

Additive accessor on the resolver — keeps the "no second pass over records" contract intact by reading only already-folded population state.

func (*FacetPopulationView) FacetField added in v0.19.0 

func (v *FacetPopulationView) FacetField() *types.FacetField

FacetField returns the underlying *types.FacetField the resolver matched. Read-only — handlers must not mutate the returned struct. Nil-safe: returns nil when the view is nil.

func (*FacetPopulationView) Field added in v0.19.0 

func (v *FacetPopulationView) Field() string

Field returns the field name the view resolves against. Read-only.

func (*FacetPopulationView) IsDiscrete added in v0.19.0 

func (v *FacetPopulationView) IsDiscrete() bool

IsDiscrete reports whether the view resolves a categorical (or boolean) field — handlers branch on this to enter the `DiscreteCounts` / `DiscreteFrequency` fast path. False when the view is nil or the resolved field is non-discrete.

func (*FacetPopulationView) IsNumeric added in v0.19.0 

func (v *FacetPopulationView) IsNumeric() bool

IsNumeric reports whether the view resolves a numeric field — handlers branch on this to enter the `NumericSummary` / `NumericPercentiles` / `NumericHistogram` streaming-state path. False when the view is nil or the resolved field is non-numeric.

func (*FacetPopulationView) Kind added in v0.19.0 

func (v *FacetPopulationView) Kind() string

Kind returns the underlying FacetField's Kind discriminator (`"discrete"` for the categorical fast path, `"numeric"` for the streaming Welford path). Empty string when the view is nil. Handlers branch on this string to dispatch the per-kind lookup surface.

func (*FacetPopulationView) NullCount added in v0.19.0 

func (v *FacetPopulationView) NullCount() int64

NullCount returns the count of filtered records where the resolved field was null. Mirrors `FacetField.NullCount` on the underlying payload. Handlers use this to compute the non-null denominator for per-value frequency math (`TotalRecords - NullCount = non-null records`). Zero when the view is nil or the resolved field is nil.

func (*FacetPopulationView) NumericHistogram added in v0.19.0 

func (v *FacetPopulationView) NumericHistogram() (*types.FacetHistogram, bool)

NumericHistogram returns the fixed-width histogram populated by `FacetSchema` when `FacetRequest.IncludeHistogram` was true. Returns `(nil, false)` when the view is nil, non-numeric, has no numeric payload, or the histogram is absent (the caller did not request a histogram — handlers that need one MUST request it on the FacetRequest; the resolver does not synthesise).

The returned `*types.FacetHistogram` carries `Min`, `Max`, and `Bins` (a slice of bucket counts). KS_VS_POP can use the histogram as a coarse population CDF approximation when percentiles are not available; INDEX_VS_POP can use it for per-bucket frequency math.

Read-only: handlers must not mutate the returned struct.

func (*FacetPopulationView) NumericPercentiles added in v0.19.0 

func (v *FacetPopulationView) NumericPercentiles() (map[string]float64, bool)

NumericPercentiles returns the percentile map populated by `FacetSchema` when `FacetRequest.NumericPercentiles` was non-empty. Returns `(nil, false)` when the view is nil, non-numeric, has no numeric payload, or the percentile map is empty (the caller did not request percentiles — handlers that need them MUST request them on the FacetRequest; the resolver does not synthesise).

Map keys are percentile labels (`"p50"`, `"p95"`, ...); values are the linearly-interpolated percentile points. The map is the exact surface `FacetSchema` populated via `linearPercentile`. KS_VS_POP uses this to build the population CDF baseline.

Read-only: handlers must not mutate the returned map.

func (*FacetPopulationView) NumericSummary added in v0.19.0 

func (v *FacetPopulationView) NumericSummary() (*types.FacetNumeric, bool)

NumericSummary returns the streaming Welford summary (`*types.FacetNumeric`) when the view resolves a numeric field. Returns `(nil, false)` when the view is nil, when the resolved field is non-numeric, or when the numeric payload is missing.

The handlers read `Mean`, `StdDev`, `Sum`, `Count`, `Min`, `Max` directly off the returned struct — these are the exact slots `FacetSchema` populated during the streaming pass. No second pass; no recomputation. ZSCORE_VS_POP reads `Mean + StdDev`; KS_VS_POP reads the percentile / histogram slots.

Read-only: handlers must not mutate the returned struct.

func (*FacetPopulationView) TotalRecords added in v0.19.0 

func (v *FacetPopulationView) TotalRecords() int64

TotalRecords returns the host FacetResult's `FilteredRecords` count. This is the canonical denominator for index-vs-pop-style ratios on the categorical fast path — every per-value count is divided by the post-filter record total to obtain a population frequency. Zero when the view is nil or the underlying FacetResult is nil.

Distinct from `FacetResult.TotalRecords` (the unfiltered cohort record count) — the overlay family compares against the same post-filter denominator the per-value counts were folded against, so using `FilteredRecords` keeps the per-value-frequency contract internally consistent.

type FieldInputsFunc added in v0.8.0 

type FieldInputsFunc func(raw json.RawMessage) []string

FieldInputsFunc reports the additional source-field names an extension operator reads beyond the spec's explicit Field/Field2/ PartitionBy/etc. references. raw carries the operator's Params block (may be nil for filterers). Return value is consumed by the buffered-projection extractor; nil/empty means "no extra fields."

type FieldSet added in v0.8.0 

type FieldSet map[string]struct{}

FieldSet is an order-insignificant set of schema field names that must be decoded into each Record during the buffered Process path. The sentinel "*" entry means "no projection — keep every field" and short-circuits Has() to true; callers that produced a wide set should fall back to the full-decode path rather than try to project.

func NeededFields added in v0.8.0 

func NeededFields(req *types.Request, schema *encoding.Schema, ext *ExtensionRegistry) FieldSet

NeededFields walks req against schema and returns the set of source field names that the buffered or streaming path must decode into each Record so every requested operator output computes correctly.

When extraction can't prove completeness (malformed expression, extension operator without a FieldInputs hook) the returned set is widened and callers should fall back to the full-decode path.

The ext argument may be nil — in that case any custom operator in req that resolves only via overlay would still produce a wide set because we'd have no introspection. Built-in operators are fully introspectable here.

func NewFieldSet added in v0.8.0 

func NewFieldSet(hint int) FieldSet

NewFieldSet returns an empty FieldSet sized for hint members.

func (FieldSet) Add added in v0.8.0 

func (s FieldSet) Add(name string)

Add inserts name into the set. Empty names are ignored.

func (FieldSet) Has added in v0.8.0 

func (s FieldSet) Has(name string) bool

Has reports whether name is in the set. Returns true for every name when the set has been widened.

func (FieldSet) IsWide added in v0.8.0 

func (s FieldSet) IsWide() bool

IsWide reports whether the set has been widened to cover every field.

func (FieldSet) Len added in v0.8.0 

func (s FieldSet) Len() int

Len returns the number of explicitly listed fields. Returns the schema field count when the set is wide (caller-supplied schema resolves the count; pass len(schema.Fields) at the call site).

func (FieldSet) Widen added in v0.8.0 

func (s FieldSet) Widen()

Widen marks the set as covering every field. After this call Has() returns true for any name. Used when extraction can't be proven complete — for example a malformed expression or an extension operator without an introspection hook.

type FilterFunc 

type FilterFunc func(record *Record) (bool, error)

FilterFunc evaluates whether a record passes a filter.

func BuildFilters added in v0.7.0 

func BuildFilters(filterers []*types.Filterer, schema *encoding.Schema, exts *ExtensionRegistry) ([]FilterFunc, error)

BuildFilters compiles a slice of types.Filterer into runtime FilterFuncs against the given schema. Mirrors the per-Processor helper so non-Processor consumers (FacetSchema, Sample variants) can reuse the same factory + extension semantics. Returns nil when filterers is empty.

func BuildMemberSetPredicate added in v0.8.3 

func BuildMemberSetPredicate(set MemberSet, schema *encoding.Schema, fieldName string) (FilterFunc, error)

BuildMemberSetPredicate returns a per-row FilterFunc that reports whether the record's value for fieldName is a member of set. The concrete predicate is selected once at construction by type-switching on the set + schema field type — the returned closure carries no interface dispatch on the hot path.

Float fields are rejected, matching LoadMemberSetFromReader. Unknown field names return SERVICE_VALIDATION.

type FiltererBuilder 

type FiltererBuilder interface {
	// Build creates a filter function from the filter specification.
	Build(filter *types.Filterer, schema *encoding.Schema) (FilterFunc, error)
}

FiltererBuilder constructs a FilterFunc from a filter specification.

type FiltererFactory 

type FiltererFactory func() FiltererBuilder

FiltererFactory creates a FiltererBuilder from a type specification.

type FormulaContext added in v0.19.0 

type FormulaContext struct {
	// Shape selects the per-shape binder arm. One of OverlayShapeMatrix
	// / OverlayShapeSeries / OverlayShapeScalar; unknown values produce
	// an empty env.
	Shape types.OverlayShape

	// MatrixHost is the host view carrying the iteration's host
	// MatrixPayload. The matrix binder reads CellAt + MarginFor against
	// it. May be nil when Shape != OverlayShapeMatrix; the matrix
	// binder no-ops on a nil host.
	MatrixHost *CrosstabHostView

	// RowSDs carries the precomputed per-row standard deviation. Length
	// matches MatrixHost.RowCount(); element i is the Welford SD across
	// every PRESENT column in row i. Population variance (N-divisor),
	// matching the existing OVERLAY_ZSCORE_VS_MARGIN SD derivation.
	// Zero-length when the formula does not reference `sd_row` (the
	// per-host orchestrator short-circuits the recurrence).
	RowSDs []float64

	// ColumnSDs carries the precomputed per-column standard deviation.
	// Length matches MatrixHost.ColumnCount(); element j is the
	// Welford SD across every PRESENT row in column j.
	ColumnSDs []float64

	// GrandSD carries the precomputed whole-matrix standard deviation
	// across every PRESENT cell. Always populated when the formula
	// references `sd_grand`; zero otherwise.
	GrandSD float64

	// RefMatrix carries the Compose reference slot's matrix payload
	// values when the host is a Compose-matrix and the per-spec
	// Reference resolves to a matrix-shaped slot. Indexed
	// `[rowIdx][colIdx]`; parallel to MatrixHost's RowKeys × ColumnKeys
	// grid. Nil when the spec is not Compose OR the Reference slot is
	// not matrix-shaped.
	RefMatrix        [][]float64
	RefMatrixPresent [][]bool

	// SeriesHost is the host view carrying the iteration's host
	// SeriesPayload (the grouped Process orchestrator's per-group
	// metric). The series binder reads ValueAt + GroupKey against it.
	// May be nil when Shape != OverlayShapeSeries.
	SeriesHost *SeriesHostView

	// SeriesTotal carries the precomputed grand total across every
	// PRESENT entry in SeriesHost. Always populated when the formula
	// references `total`; zero otherwise.
	SeriesTotal float64

	// BaselineValue carries the precomputed `Params["baseline_position"]`
	// lookup against SeriesHost. Set only when the per-spec
	// `Params["baseline_position"]` is present AND the formula
	// references `baseline`; the predict gate makes this consistent.
	BaselineValue        float64
	BaselineValuePresent bool

	// RefSeries carries the Compose reference slot's series payload
	// values when the host is a Compose-series and the per-spec
	// Reference resolves to a series-shaped slot. Parallel to
	// SeriesHost's GroupKeys slice.
	RefSeries        []float64
	RefSeriesPresent []bool

	// ScalarValue carries the host's scalar payload value when the
	// host is scalar-shaped. `ScalarPresent` is the present flag.
	ScalarValue   float64
	ScalarPresent bool

	// RefScalar carries the Compose reference slot's scalar payload
	// value when the host is a Compose-scalar and the per-spec
	// Reference resolves to a scalar-shaped slot.
	RefScalar        float64
	RefScalarPresent bool

	// Slots carries the per-spec Compose slot bindings in slot order.
	// Each entry's Label + per-shape payload feeds the dotted
	// `slot.<label>.<field>` namespace per research note § 2.4. The
	// per-shape binder writes the same nested-map shape the prototype
	// env declared at compile time so expr-lang's compile-time env
	// validator accepts the dotted identifiers.
	Slots []formulaSlotBinding
}

FormulaContext bundles every shape-specific input the per-shape binders consume to populate the FORMULA env. Built once per overlay spec at handler entry by the per-host orchestrator and passed straight to `bindFormulaEnv`. The shape field selects which arm of the union the umbrella binder dispatches to; the un-matching arms are ignored.

Reference values (RefMatrix / RefSeries / RefScalar) are populated when the per-spec OverlayRef carries a Compose Reference slot whose host result shape matches the iteration host shape; predict-time shape-match enforcement (`checkSlotShapeAndSchema` for COMPOSE Reference slots) keeps the per-shape ref payload consistent with the iteration shape so the binder doesn't have to validate the pairing — it just reads whichever arm matches `Shape`.

SDs (RowSDs / ColumnSDs / GrandSD) are precomputed by the per-shape orchestrator BEFORE the cell loop runs — research note § 2.1 SD derivation paragraph: SDs are computed once per axis (Welford-Pébaÿ over the matching cell strip) and cached for the strip's iterations. The binder reads the precomputed slot.

BaselinePosition opt-in: when the formula references `baseline` and the per-spec `Params["baseline_position"]` is set, the per-host orchestrator resolves the baseline value once and plumbs it through `BaselineValue` / `BaselineValuePresent`. The predict gate rejects formulas referencing `baseline` when the param is absent so the runtime never sees `BaselineValuePresent == false` paired with a `baseline` identifier.

type FusedCrosstabState added in v0.17.0 

type FusedCrosstabState struct {
	// contains filtered or unexported fields
}

FusedCrosstabState is the per-record streaming accumulator used by the fused crosstab execution path. It composes a row-axis grouper chain, a column-axis grouper chain, and per-cell streaming aggregator state so the orchestrator can decode the cohort exactly once and route every filter-passing record into the correct cell / row-margin / column- margin / grand-margin / cross-axis-margin accumulator with no record buffering.

Output contract: Finalize() returns a *types.Response byte-equal to the buffered RunCrosstab output for any (spec, schema, ext, record stream) the CanFuseCrosstab gate accepts — same MatrixPayload / long-shape Data row order, same normalization and divide-by-zero policy, same margin recomputation rule. The orchestrator is responsible for dispatching to the fused path only when CanFuseCrosstab holds; the defensive guards in this file echo that gate so a mis-routed request fails fast with a typed CodedError instead of producing a diverging result.

Hot-path representation: rather than addressing cell and margin accumulators through a string-keyed map per record, the state interns each axis composite key into a per-axis integer index the first time it is observed and stores accumulators in a 2D slice indexed by (rowIdx, colIdx). Per-record lookup drops from two hash probes (a strings.Join allocation plus a map[crosstabCellKey] lookup) to two map[string]int probes (one per axis) plus pure slice indexing into cells/rowMargins/colMargins.

Memory cost is O(cells + row margins + col margins + cross margins) rather than O(records): each cell holds one OnlineAggregator instance (the same one the streaming Process path drives), no record bucket. Significantly less heap pressure on wide cohorts where the buffered RunCrosstab materialises every filter-passing record.

func NewFusedCrosstabState added in v0.17.0 

func NewFusedCrosstabState(spec *types.CrosstabSpec, schema *encoding.Schema, ext *ExtensionRegistry) (*FusedCrosstabState, error)

NewFusedCrosstabState constructs a FusedCrosstabState. Validates the spec against the gate's structural assumptions (every axis grouper must implement StreamableGrouper, cell aggregator must be online) and pre-builds the per-axis grouper instances so the per-record hot path can compute composite keys with a single method call per axis entry.

req-level slots that the gate excludes (Tests / Features / Regressions / Windows / Joins / two-pass attributes / Groups / top-level Aggregations) are NOT validated here — the caller is responsible for dispatching only when CanFuseCrosstab(req) returned true. AssertCanFuse is the defensive sibling that callers can use to fail fast when their dispatch shortcut may have missed an edge case.

func (*FusedCrosstabState) AddTotalRow added in v0.17.0 

func (s *FusedCrosstabState) AddTotalRow()

AddTotalRow advances the unfiltered row counter. Called once per physical record the orchestrator decodes from the cohort regardless of whether the record passes the filter chain. Separate from Update so the orchestrator can advance the total even on filtered-out records.

func (*FusedCrosstabState) AssertCanFuse added in v0.17.0 

func (s *FusedCrosstabState) AssertCanFuse(req *types.Request) error

AssertCanFuse is the defensive companion check the orchestrator calls before driving the state with records. The static gate (CanFuseCrosstab) is the load-bearing check; this method echoes the most failure-prone subset so a stale gate that drifts past a newly added request slot still fails fast with a typed error rather than producing a divergent fused result.

Returns nil when the request is structurally compatible with the fused path. Returns a typed CodedError otherwise.

func (*FusedCrosstabState) FilteredRows added in v0.17.0 

func (s *FusedCrosstabState) FilteredRows() int64

FilteredRows returns the count of records that reached Update. The orchestrator's metadata block reads from this. Exposed publicly for the same reason as TotalRows.

func (*FusedCrosstabState) Finalize added in v0.17.0 

func (s *FusedCrosstabState) Finalize() (*types.Response, error)

Finalize closes the streaming accumulators and emits the *types.Response in the same shape RunCrosstab produces — MatrixPayload for shape=matrix, long-form Response.Data with margin tags for shape=long. Normalization (mode + normalize_level + normalize_within) is applied here using the same denominator + divide-by-zero rules the buffered path uses.

Safe to call exactly once after the last Update. Subsequent calls observe drained accumulator state and produce undefined output.

func (*FusedCrosstabState) SetTotalRows added in v0.17.0 

func (s *FusedCrosstabState) SetTotalRows(n int64)

SetTotalRows lets callers set the counter directly when they have already counted (e.g. CountRecords header-fast path) or when they process rows in bulk outside the Update loop. Either AddTotalRow or SetTotalRows is enough; both interfaces are provided so the orchestrator can pick whichever is more convenient.

func (*FusedCrosstabState) TotalRows added in v0.17.0 

func (s *FusedCrosstabState) TotalRows() int64

TotalRows returns the total-row counter for use after Finalize. The orchestrator's metadata block reads from this. Exposed publicly so tests can assert the bookkeeping without inspecting unexported state.

func (*FusedCrosstabState) Update added in v0.17.0 

func (s *FusedCrosstabState) Update(rec *Record) error

Update folds a single filter-passing record into the cell, margin, and cross-margin accumulators it touches. The caller is responsible for running filters / row-local attributes / features / etc. before passing the record in — by the time Update sees a record it is already a filter-passing observation in the same sense the buffered RunCrosstab path treats its filtered slice. Records skipped by the caller's filter contribute to totalRows only via the totalRows counter the caller separately advances.

Axis-key nullity is tracked independently per axis. A record whose row-axis composite key is non-null AND col-axis composite key is non-null lands a cell update at the (rowIdx, colIdx) intersection. A record whose row-axis key is non-null but col-axis key is null (or vice versa) still contributes to the appropriate axis margin (rowMargin via the row key, colMargin via the col key), matching the buffered RunCrosstab path: there `PartitionByAxis(spec.Rows, filtered)` builds the row partition over all filtered records with valid row keys regardless of column key nullity (and vice versa). The grand margin counts every filter-passing record regardless of any axis nullity. Partial-depth (normalize_level) margins follow the same per-axis gate: row partial-margin updates whenever the row axis key is non-null. Cross-axis (normalize_within) margins update only when BOTH the row prefix at crossRowDepth AND the column prefix at crossColDepth are non-null — mirroring the buffered joined-partition behaviour at processing/crosstab.go::crossActive (lines ~410-448), where the partition only contains records whose prefix groupers all produce non-null keys.

type Grouper 

type Grouper interface {
	// Group partitions the records by the specified field, returning a map of group key to records.
	Group(records []*Record, field string) (map[string][]*Record, error)
}

Grouper partitions records into named groups.

type GrouperFactory 

type GrouperFactory func(grp *types.Group, schema *encoding.Schema) (Grouper, error)

GrouperFactory creates a Grouper from a type specification.

type HashJoinIterator added in v0.10.0 

type HashJoinIterator struct {
	// contains filtered or unexported fields
}

HashJoinIterator wraps a left-side iterator and yields joined records on each Next() call. The right side is materialised into a hashmap keyed by the composite (LeftField → string) tuple. Inner join only — non-matching left rows are dropped silently.

Memory: O(right_rows × per_record_state). The orchestrator picks the smaller side as the build side; in v1 this is always the caller-provided "right" path. A future iteration adds a CountRecords pre-pass to swap sides automatically.

func NewHashJoinIterator added in v0.10.0 

func NewHashJoinIterator(left RecordIterator, right []*Record, leftSchema, rightSchema *encoding.Schema, spec *types.JoinSpec) (*HashJoinIterator, *encoding.Schema, error)

NewHashJoinIterator builds the right-side hash table eagerly from the supplied right-iterator, then returns an iterator that walks the left side. Each left row's composite key is looked up; matched pairs are emitted via Next() in (left, right[0]), (left, right[1]) order.

func (*HashJoinIterator) Next added in v0.10.0 

func (h *HashJoinIterator) Next() bool

Next advances to the next joined record. Returns false when the left side is exhausted and the per-left-row match buffer is empty.

func (*HashJoinIterator) Record added in v0.10.0 

func (h *HashJoinIterator) Record() *Record

Record returns the current joined record. Combines the left iterator's current record with the matched right record indexed by cursor-1 (Next() already advanced past it).

func (*HashJoinIterator) Reset added in v0.10.0 

func (h *HashJoinIterator) Reset()

Reset rewinds the left iterator and clears per-row state. Right- hash retention is intentional: build-once + scan-many.

type IncludeOrdered added in v0.25.0 

type IncludeOrdered interface {
	Grouper
	// IncludeFilter returns the grouper's ordered include filter, or nil
	// when no Include list was supplied (accept-all).
	IncludeFilter() *includeFilter
}

IncludeOrdered is the optional sibling of Grouper for groupers that carry a Group.Include inclusion list and can surface it as an ordered filter. GROUP_CATEGORY / GROUP_SET_VALUE / GROUP_SET_PER_ELEMENT implement it; every other built-in grouper (and any extension grouper that does not opt in) is treated as accept-all by includeFilterOf.

type LabelResolver added in v0.10.1 

type LabelResolver struct {
	// contains filtered or unexported fields
}

LabelResolver translates raw categorical values to display labels per a request's LabelBinding slice. It is constructed once per Request via BuildLabelResolver and consumed at every output site (Sample rows, FacetField counts, AGG_FREQUENCY result keys, grouped result keys, Export cells).

Resolver methods are not safe for concurrent use. Each Request owns its own resolver; shard-parallel reducers either build one resolver per partial and merge warnings at the end, or serialise the output stage.

Replace-mode collision behaviour: when two distinct source values map to the same label string, the output disambiguates with the source value in parentheses ("United States (US)"). Rows-backed tables get a pre-pass so both colliding sources render symmetrically; function-driven tables detect collisions online, so the first source seen renders cleanly and the second renders disambiguated (asymmetric). The PULSE_LABEL_COLLISION warning names every involved source value so callers can audit.

func BuildLabelResolver added in v0.10.1 

func BuildLabelResolver(bindings []*types.LabelBinding, registry *ExtensionRegistry) (*LabelResolver, error)

BuildLabelResolver constructs a LabelResolver from the request bindings and the runtime extension registry. Returns (nil, nil) when bindings is empty so callers can guard cheaply with a single nil check.

Returns PULSE_LABEL_TABLE_UNKNOWN if a binding references a table absent from the registry. Other binding-shape issues (unknown field, non-categorical field, augment collision) are validated upstream by descriptor.ValidateLabels — this constructor trusts schema-level checks.

func (*LabelResolver) Apply added in v0.10.1 

func (r *LabelResolver) Apply(field, raw string) (out string, sibling string, ok bool)

Apply translates a raw categorical value through the binding for the named field. Behaviour by mode:

  • Replace: returns (label, sibling="", ok=true) on hit; on miss returns (raw, "", false) and records a miss for the warning summary. The label may be disambiguated as "label (raw)" when two source values collide on the same label.
  • Augment: returns (raw, label, ok=true) on hit; on miss returns (raw, "", false) and records a miss.

Returns (raw, "", false) when the resolver carries no binding for the field. Callers should typically gate with Has before calling.

func (*LabelResolver) AugmentField added in v0.10.1 

func (r *LabelResolver) AugmentField(field string) string

AugmentField returns the augment-mode sibling column name for the named field, or empty string when the field has no augment binding.

func (*LabelResolver) FieldsWithAugment added in v0.10.1 

func (r *LabelResolver) FieldsWithAugment() []string

FieldsWithAugment returns the field names that should emit a "<field>_label" sibling column. Stable in iteration order of the underlying map (used only for output schema overlay, where stable ordering is enforced by caller).

func (*LabelResolver) Has added in v0.10.1 

func (r *LabelResolver) Has(field string) bool

Has reports whether the resolver carries any binding for the named field. Cheap and nil-safe.

func (*LabelResolver) Mode added in v0.10.1 

func (r *LabelResolver) Mode(field string) types.LabelMode

Mode returns the binding mode for the named field. Returns the zero LabelMode when no binding exists; callers should gate with Has.

func (*LabelResolver) Warnings added in v0.10.1 

func (r *LabelResolver) Warnings() []ResolverWarning

Warnings flushes the per-binding collision and miss summaries into envelope-ready warning records. Safe to call once at the end of a run; subsequent calls return the same set.

type LabelTable added in v0.10.1 

type LabelTable struct {
	Rows   map[string]string
	Lookup func(key string) (string, bool, error)
}

LabelTable is the runtime-side mirror of pulse.LabelTable. The label resolver consults this map when a request supplies a LabelBinding referencing the table by name; exactly one of Rows or Lookup is populated per table.

type LoadMemberSetResult added in v0.8.3 

type LoadMemberSetResult struct {
	Set             MemberSet
	Lines           int
	NotInDictionary int
	Invalid         int
}

LoadMemberSetResult is the per-load report returned by LoadMemberSetFromReader. Lines is total non-blank lines processed; NotInDictionary counts categorical lookups whose value was absent from the field's dictionary (those keys can never match a record so they're dropped); Invalid counts numeric lines that failed to parse on the integer path. Callers can decide whether to surface each counter as a warning or hard error.

func LoadMemberSetFromReader added in v0.8.3 

func LoadMemberSetFromReader(r io.Reader, schema *encoding.Schema, fieldName string) (LoadMemberSetResult, error)

LoadMemberSetFromReader reads newline-delimited values from r and builds the best MemberSet impl for the named field on schema. Lines are trimmed of surrounding whitespace (covers CRLF). A UTF-8 BOM is stripped from the very first line. Blank lines are skipped silently — they don't count against Lines.

Dispatch by field type:

  • categorical_u{8,16,32}: parse line as string, resolve via dict.IDFor → BitsetSet. Lines not in the dictionary are counted in NotInDictionary and dropped — they can never match a record.
  • u8/u16/u32/u64, nullable_u4/u8/u16, nullable_bool, packed_bool, date: parse as uint64 → Uint64Set. ParseUint failures are counted in Invalid and dropped.
  • decimal128 / nullable_decimal128: parse as string → StringSet. The cohort filter compares the literal text against the decimal128 wide value's String() form; exact match required.

Float fields are rejected with SERVICE_VALIDATION. Unknown field names return SERVICE_VALIDATION.

type LookupTable added in v0.7.0 

type LookupTable struct {
	Rows   map[string]float64
	Lookup func(keys ...string) (float64, bool, error)
}

LookupTable is the runtime-side mirror of pulse.LookupTable. The expr environment's lookup() built-in consults this map; exactly one of Rows or Lookup is populated per table.

type MemberSet added in v0.8.3 

type MemberSet interface {
	Len() int
	Kind() string
}

MemberSet is a read-only set of field values used by the cohort filter include-from path to test record membership. Three concrete impls back this interface, picked by field type at load time so the per-row lookup runs on the fastest path available:

  • *BitsetSet — categorical fields. One bit per dictionary entry. Lookup is a single word load + bit test, no hashing, no compare.
  • *Uint64Set — integer / date / bit-packed integer fields. Backed by map[uint64]struct{}; avoids the per-row string allocation a string-keyed map would force.
  • *StringSet — fallback (decimal128, any non-categorical string surface). Backed by map[string]struct{}.

Float fields (f32/f64) are rejected at load time. Exact-equality membership on floats is a footgun; the caller is expected to use the expression filter for numeric ranges.

The interface itself carries only metadata; the per-row hot path goes through the concrete type via the predicate built by BuildMemberSetPredicate, which type-switches once at construction and returns a closure with no interface dispatch in the loop.

type MergeableAggregator added in v0.8.0 

type MergeableAggregator interface {
	OnlineAggregator
	MergeOnline(other OnlineAggregator) error
}

MergeableAggregator is the optional sibling of OnlineAggregator for aggregators whose running state is associative+commutative (or associative under a parallel-friendly recurrence). Implementations fold another instance's state into the receiver without rescanning any rows, enabling per-shard parallel execution: each worker computes its partial state from its shard, the orchestrator merges partials in shard insertion order, then Finalize() emits the aggregate.

Implementations MUST be safe to call MergeOnline repeatedly. The receiver absorbs other's state; other is left in an unspecified state and should not be reused. Returning a non-nil error indicates the two instances were not constructed from compatible specs (a programming bug — the orchestrator only merges aggregators constructed from the same Aggregation spec).

MergeOnline preserves the mathematical contract of Finalize:

  • COUNT / SUM / NULL_COUNT: sum the counters.
  • MIN / MAX: pick the extremum, accounting for the "seen" flag.
  • MEAN (Welford): combine (n, mean, M2) via the Chan-Welford parallel formula so the merged mean equals the single-pass mean to within ULP on well-conditioned inputs.
  • FREQUENCY: union the per-value count maps; Finalize then picks the max as it does in the serial path.

Aggregators whose state is fundamentally non-mergeable (percentile/median/zscore — they require a sorted view) MUST NOT implement this interface; the parallel orchestrator falls through to the serial shard iterator for such requests.

Components contract under merge. MergeOnline composes per-chunk state with another mergeable aggregator's state. After MergeOnline returns, the receiver's scalar (Aggregate/Finalize) output AND any MetaAggregator.Components() output reflect the merged state — there is no separate MergeComponents call. Implementations MUST keep internal component-state mirrors consistent with the value- state under MergeOnline. The orchestrator path is MergeOnline...; Finalize(); Components() — so per-aggregator freeze() helpers stamped from Finalize automatically cover the merged emission. Operators with ComponentsMergeability=Partial may stage merge at terminal flush rather than per-chunk; consult types.ComponentsMergeability.

type MetaAggregator added in v0.20.0 

type MetaAggregator interface {
	Aggregator
	// Components returns the per-operator components map keyed by the
	// snake_case names declared in the aggregator's
	// descriptor.ComponentSchema. Returning (nil, nil) means "no
	// operator-specific keys; caller fills the universal floor
	// unconditionally". A non-nil error aborts the request with the
	// same error-routing contract as Aggregate / Finalize.
	Components() (map[string]any, error)
}

MetaAggregator is the optional sibling of Aggregator for aggregators that emit a per-operator "components" map alongside their scalar (or rich) value. The map carries the named constituent-parts metadata declared in the aggregator's descriptor.ComponentSchema — Welford triples (mean, variance, sum_squares, m2), frequency tables, percentile sort sizes, distinct-mask popcounts, and so on.

Components is called exactly once after Aggregate or Finalize. The orchestrator routes the result into Response.Components.Aggregations (or the Crosstab cell-components matrix when the aggregator runs inside a crosstab cell). Implementations MUST be safe to call Components once per result; calling it multiple times or before the terminating Aggregate / Finalize is a programming error and may return stale state.

Universal floor contract: the orchestrator ALWAYS populates the universal floor keys ({"n", "n_null"}) on Response.Components from per-record bookkeeping. Components() returns ONLY the operator- specific keys (mean, variance, mode, range_min, dict_size, …) — do NOT re-emit n / n_null from the operator. Returning (nil, nil) is the canonical signal for "no operator-specific keys; caller still fills the universal floor unconditionally" (e.g. AGG_COUNT, where the universal floor IS the entire component payload).

Streaming semantics follow the descriptor.ComponentsMergeability declared at registration time:

  • Mergeable: components fold across chunks via the same MergeOnline path as the scalar value (Welford, sums, counts).
  • Partial: map / set merges that work but cost allocation (frequency tables, distinct masks); orchestrator may stage the merge at terminal flush.
  • None: components emitted only on terminal buffered flush (median / percentile — need a sorted view of the full input); streaming chunks omit them and predict declares the slot as buffered-components-only.

Subsequent stories add the per-aggregator implementations under compile-time assertions of the form: 

var _ processing.MetaAggregator = (*welfordAgg)(nil)

which keep the wiring grep-discoverable and catch interface drift at build time. No runtime wiring is added by the interface itself — the descriptor.ComponentSchema declared in capabilities_*.go is the matching half consumed by predict / manifest.

type MetaFilterer added in v0.20.0 

type MetaFilterer interface {
	// Components returns the per-filterer components map keyed by the
	// snake_case names declared in the filterer's
	// descriptor.ComponentSchema. Returning (nil, nil) means "no
	// operator-specific keys; caller fills the universal floor
	// unconditionally" — the canonical v1 contract for every built-in
	// filterer. A non-nil error aborts the request with the same
	// error-routing contract as Build / FilterFunc.
	Components() (map[string]any, error)
}

MetaFilterer is the optional sibling of FiltererBuilder for filterers that emit a per-filterer "components" map alongside their per-record predicate. The map carries the named constituent-parts metadata declared in the filterer's descriptor.ComponentSchema.

In v1 the components surface is uniform across every registered filterer — the orchestrator emits the universal floor {n_in, n_out, n_null_input} from the filter pass's record-walker counters. No built-in filterer implements Components(): the interface exists for extension-author parity with MetaAggregator / MetaGrouper and to leave room for future per-filter specifics (e.g. n_below / n_above for FILTER_RANGE; per-value n for FILTER_INCLUDE / FILTER_EXCLUDE). The orchestrator's universal- floor pass is always authoritative for the three floor keys; an embedder-supplied Components() returns ONLY the operator-specific extras.

Components is called exactly once after the filter pass terminates (the iterator is exhausted). The orchestrator routes the result into Response.Components.Filterers (one entry per Request.Filterers slot, in declared order). Implementations MUST be safe to call Components once per result; calling it multiple times or before the pass completes is a programming error and may return stale state.

Streaming merge semantics follow descriptor.ComponentsMergeability declared at registration time. Filterer components in v1 are Mergeable — counters fold trivially across chunks via integer addition, so the streaming path emits per-chunk deltas without staging. Future per-filter specifics may downgrade individual entries to Partial / None; the orchestrator follows the same per-operator dispatch the aggregator side uses.

Subsequent stories (no built-in implementations in v1) may add per-filterer sentinels of the form 

var _ processing.MetaFilterer = (*rangeFilterer)(nil)

to keep the wiring grep-discoverable and catch interface drift at build time, mirroring the MetaAggregator / MetaGrouper pattern. No runtime wiring is added by the interface itself — the descriptor.ComponentSchema declared in capabilities_filterers.go is the matching half consumed by predict / manifest.

type MetaGrouper added in v0.20.0 

type MetaGrouper interface {
	// Components returns the per-grouper components map keyed by the
	// snake_case names declared in the grouper's
	// descriptor.ComponentSchema. Returning (nil, nil) means "no
	// operator-specific keys; caller fills the universal floor
	// unconditionally". A non-nil error aborts the request with the
	// same error-routing contract as Group / KeyFor / KeyForRow /
	// KeysForRow.
	Components() (map[string]any, error)
}

MetaGrouper is the optional sibling of Grouper / StreamingGrouper / StreamableGrouper / MultiKeyStreamingGrouper for groupers that emit a per-operator "components" map alongside their partitioning output. The map carries the named constituent-parts metadata declared in the grouper's descriptor.ComponentSchema — bucket arrays, edge values, dictionary mappings, range_min / range_max, granularity, etc.

Components is called exactly once after the grouper's terminal pass:

  • For buffered groupers (Grouper.Group), after Group returns.
  • For streaming groupers (StreamingGrouper / StreamableGrouper), after the per-record KeyForRow / KeyFor sequence terminates and the iterator is exhausted.
  • For multi-key streaming groupers (MultiKeyStreamingGrouper, e.g. GROUP_SET_PER_ELEMENT), after the per-record KeysForRow sequence terminates. Per-bucket counts in the emitted map reflect emission count (one increment per emitted key), NOT row count — a single row may contribute to multiple buckets, and the orchestrator's TotalN reflects the row count.

The orchestrator routes the result into Response.Components.Groupers (one entry per Request.Groups slot, in declared order). Implementations MUST be safe to call Components once per result; calling it multiple times or before the terminating pass is a programming error and may return stale state.

Universal floor contract: the orchestrator ALWAYS populates the universal floor keys (TotalN, NNull on types.GrouperComponents) from the post-filter record walker. Components() returns ONLY the operator-specific keys (buckets, edges, dict_size, granularity, …) — do NOT re-emit total_n / n_null from the operator. Returning (nil, nil) is the canonical signal for "no operator-specific keys; caller still fills the universal floor unconditionally".

Streaming merge semantics follow the descriptor.ComponentsMergeability declared at registration time, mirroring the MetaAggregator contract: mergeable groupers (bucket counts) fold across chunks; partial groupers (dict-bearing) stage merges at terminal flush; non-mergeable groupers (rare for groupers; QUANTILE-class needs the full input) emit components only on terminal buffered flush.

Per-grouper implementations register under compile-time assertions of the form: 

var _ processing.MetaGrouper = (*categoryGrouper)(nil)
var _ processing.MetaGrouper = (*rangeGrouper)(nil)
var _ processing.MetaGrouper = (*roundedGrouper)(nil)
var _ processing.MetaGrouper = (*quantileGrouper)(nil)
var _ processing.MetaGrouper = (*dateGrouper)(nil)
...

which keep the wiring grep-discoverable and catch interface drift at build time. No runtime wiring is added by the interface itself — the descriptor.ComponentSchema declared in capabilities_groupers.go is the matching half consumed by predict / manifest.

type MultiKeyStreamingGrouper added in v0.15.0 

type MultiKeyStreamingGrouper interface {
	KeysForRow(record *Record, field string) (keys []string, ok bool, err error)
}

MultiKeyStreamingGrouper is the optional sibling of StreamingGrouper for groupers that fan a single record into multiple per-key buckets — GROUP_SET_PER_ELEMENT is the canonical implementer: a row whose set field selected N labels contributes to N buckets, one per label.

The streaming orchestrator prefers this interface when present and drives UpdateRow once per resulting key; groupers that implement only StreamingGrouper continue to follow the single-key path.

Implementations MUST be safe to call repeatedly; ok=false signals the row should be skipped (e.g. null or empty set); ok=true means at least one key was returned. An empty key slice with ok=true is not a valid response — return ok=false instead.

type OnlineAggregator added in v0.2.0 

type OnlineAggregator interface {
	// UpdateRow folds a single record into the running state. The field
	// argument is the field the aggregator operates on; for COUNT this
	// can be ignored (COUNT counts rows, not values).
	UpdateRow(record *Record, field string) error
	// Finalize returns the final aggregated value and resets internal
	// state. It must be safe to call exactly once after streaming.
	Finalize() (float64, error)
}

OnlineAggregator is the optional sibling of Aggregator that supports single-pass streaming computation without materializing the full record set. Aggregators that can produce their result via O(1) (or O(unique)) state per row implement this interface so the orchestrator can stream the iterator directly when every aggregation in a request is online.

Implementations MUST handle null values internally: callers invoke UpdateRow once per record (after filters), and the aggregator decides whether the row contributes (e.g., COUNT counts every row, SUM skips nulls on its target field).

Finalize is called once after the iterator is exhausted; it returns the aggregated value. Implementations MUST be safe to call Finalize without any UpdateRow calls (i.e., empty input case).

type PostTest added in v0.3.0 

type PostTest interface {
	Run(rows []map[string]any) (*types.TestResult, error)
}

PostTest is a tier-2 statistical test consumed after the window stage on the materialized result row set. Run is called once over the post- pipeline data and returns a TestResult; tier-2 tests are always buffered.

type PostTestFactory added in v0.3.0 

type PostTestFactory func(spec *types.Test, schema *encoding.Schema) (PostTest, error)

PostTestFactory creates a PostTest from a specification.

type ProcessPath added in v0.2.0 

type ProcessPath int

ProcessPath identifies which execution path Process took. The streaming path runs aggregations in a single pass over the iterator without materializing the full record set. The buffered path collects every record into a slice first. This is exposed primarily for tests and benchmarks; production callers do not need it. 

const (
	// PathUnknown is the zero value; set before the first Process call.
	PathUnknown ProcessPath = iota
	// PathBuffered is the materialize-then-aggregate path. It is
	// always correct and is the fallback whenever streaming would be
	// unsafe (groups, attributes, non-online aggregators, expression
	// filters that need the full set, etc.).
	PathBuffered
	// PathStreaming runs aggregations in a single pass over the iterator,
	// folding each record into the running state of every aggregator.
	// Selected only when every aggregation is an OnlineAggregator and
	// the request has no groups, no attributes, and only row-level
	// filters.
	PathStreaming
)

func (ProcessPath) String added in v0.2.0 

func (p ProcessPath) String() string

type Processor 

type Processor struct {
	// contains filtered or unexported fields
}

Processor is the single dynamic processing engine for Pulse. It handles filtering, attribute computation, grouping, and aggregation over record iterators backed by .pulse encoded data.

func NewProcessor 

func NewProcessor(schema *encoding.Schema) *Processor

NewProcessor creates a new Processor for the given schema. The resulting Processor uses only Pulse-shipped operator factories; embedder extensions land via NewProcessorWithExtensions.

func NewProcessorWithExtensions added in v0.7.0 

func NewProcessorWithExtensions(schema *encoding.Schema, exts *ExtensionRegistry) *Processor

NewProcessorWithExtensions creates a Processor whose operator lookups consult exts before falling through to the built-in registries. Passing nil is equivalent to NewProcessor.

func (*Processor) DisableComponents added in v0.22.2 

func (p *Processor) DisableComponents() bool

DisableComponents reports the current setting. Exposed so the crosstab / fused-crosstab dispatch paths (which build their own emission helper) can consult it.

func (*Processor) LastPath added in v0.2.0 

func (p *Processor) LastPath() ProcessPath

LastPath returns the ProcessPath taken by the most recent Process call on this processor instance. Returns PathUnknown before any call. Used by tests to verify that the orchestrator selected the streaming path for online-only requests; not part of the stable API contract.

func (*Processor) PartitionByAxis added in v0.12.0 

func (p *Processor) PartitionByAxis(axis []*types.Group, records []*Record) (*CrosstabAxisPartition, error)

PartitionByAxis recursively partitions records by an ordered list of groupers, producing one bucket per axis tuple. The returned partition is sorted by composite key for deterministic output ordering across runs. Empty axis returns a single global bucket keyed by the empty string, so callers can write a single uniform reshape loop.

Exported for crosstab and any embedder that needs deterministic multi-grouper partitioning without going through the full Process pipeline.

func (*Processor) Process 

func (p *Processor) Process(ctx context.Context, req *types.Request, iter RecordIterator) (*types.Response, error)

Process executes a single request against the record iterator.

Selects between two execution strategies:

  1. Streaming: when every aggregation supports OnlineAggregator and the request has no groups, no attributes, the iterator is consumed in one pass. Filters are applied per row before each aggregator folds the row into its running state. Memory is O(distinct values) for FREQUENCY/MODE/DISTINCT_COUNT and O(1) for everything else.

  2. Buffered: the fallback path. Every record is collected into a slice first, then filters, attributes, grouping, and aggregations run over the materialized set. Memory is O(rows). Always correct.

Output is identical between paths to float64 precision on well-conditioned inputs (variance/stddev/skewness/kurtosis use Welford-Pébaÿ recurrences in the streaming path).

func (*Processor) ProcessComposed 

func (p *Processor) ProcessComposed(ctx context.Context, composed *types.ComposedRequest, records []*Record) ([]*types.Response, error)

ProcessComposed executes multiple requests against a shared record set.

func (*Processor) RunCrosstab added in v0.12.0 

func (p *Processor) RunCrosstab(_ context.Context, req *types.Request, records []*Record) (*types.Response, error)

RunCrosstab executes a Request.Crosstab against a pre-drained record set. It runs the standard pre-aggregate pipeline (features → filters → attributes), partitions filtered records by the configured row and column axes, computes per-cell aggregates, recomputes the requested margins, applies the configured normalization, and emits either a MatrixPayload (shape=matrix) or long-form rows (shape=long).

Margin computation is a recompute-only path in v1: every margin is derived from the raw filter-passing rows (not the cell values), so AGG_MEDIAN / AGG_STDDEV / AGG_PERCENTILE produce statistically correct margins. See AggregationType.MarginReducibility for the classification surfaced in the manifest.

Streamability: this method always materializes the full filter- passing record set. shape=long without margins and without normalization is the only streamable case, and the orchestrator short-circuits to the standard grouped process path before reaching here (handled in service/crosstab.go).

func (*Processor) RunCrosstabFused added in v0.17.0 

func (p *Processor) RunCrosstabFused(_ context.Context, req *types.Request, iter RecordIterator) (*types.Response, error)

RunCrosstabFused executes a Request.Crosstab against a streaming record iterator using the fused accumulator path. The caller is responsible for dispatching through CanFuseCrosstab before invoking this method; AssertCanFuse echoes the gate's exclusions so a mis- routed request fails fast with a typed CodedError rather than producing a divergent fused result.

Pipeline mirrors the buffered RunCrosstab pre-aggregate order — there are no features and no two-pass attributes by gate construction, so the surviving stages reduce to filters → row-local attributes → per-cell fold. Both helpers run on the same per-record cursor (filters first, then attributes) so a label injected by an attribute is visible to the cell aggregator on the next iteration but never participates in the filter chain — same ordering the buffered processCrosstab path establishes via applyFilters then applyAttributes.

Total-row counting advances on every iterator record regardless of whether the filter chain passes; filtered-row counting advances inside FusedCrosstabState.Update via state.filteredRows. The two counts feed Response.Metadata.TotalRows / FilteredRows respectively.

Iterator reuse is opted in via EnableReuse — the fused state never retains a record reference past the Update call, so per-row Record reuse is safe and shaves the per-record allocation cost on wide cohorts.

Errors surface the standard CodedError shapes. A gate drift that allows a non-streamable grouper / non-online cell aggregator to reach here surfaces as PROCESSING_INTERNAL via NewFusedCrosstabState or AssertCanFuse.

func (*Processor) SetDisableComponents added in v0.22.2 

func (p *Processor) SetDisableComponents(disabled bool)

SetDisableComponents toggles Response.Components emission for this processor instance. When true, every attach helper (aggregation, grouper, filterer, run, crosstab) early-returns before constructing any per-operator components map — the MetaAggregator.Components / MetaGrouper.Components calls are skipped entirely. Service-layer callers compute the effective decision (per-request override against the engine default) and flip this flag before invoking Process. See pulse.Options.DisableComponents and types.Request.DisableComponents for the full contract.

type Record 

type Record struct {
	// contains filtered or unexported fields
}

Record represents a single data row with field accessors. It provides both numeric and string access for processing operations.

func NewRecord 

func NewRecord(schema *encoding.Schema, values map[string]float64) *Record

NewRecord creates a record with the given schema and field values.

func NewRecordWithNulls 

func NewRecordWithNulls(schema *encoding.Schema, values map[string]float64, nulls map[string]bool) *Record

NewRecordWithNulls creates a record with explicit null tracking.

func NewRecordWithWide added in v0.2.0 

func NewRecordWithWide(schema *encoding.Schema, values map[string]float64, nulls map[string]bool, wide map[string]any) *Record

NewRecordWithWide creates a record with typed wide values for fields that do not fit in float64 (decimal128).

func NewReusableRecord added in v0.2.0 

func NewReusableRecord(schema *encoding.Schema) *Record

NewReusableRecord constructs a Record whose internal maps are sized for the given schema and intended to be reused across many ReadRecordReused calls. Returns a Record that callers must NOT retain past the next iteration step.

func RecordsFromChainRows added in v0.10.0 

func RecordsFromChainRows(rows []map[string]any, schema *encoding.Schema) ([]*Record, error)

RecordsFromChainRows materializes records for the next chain stage from a Response.Data slice + the synthesised schema. Categorical fields land in the dictionary lazily as new keys arrive. Numeric cells are converted to float64; missing / nil cells become nulls.

The returned records share the schema reference; the schema's dictionaries are mutated as new keys are observed.

func (*Record) AllValues 

func (r *Record) AllValues() map[string]any

AllValues returns all field values as a map (for expression evaluation).

The returned map is cached on the Record after the first call and reused on subsequent calls; callers MUST NOT mutate it. If a caller mutates the underlying values map directly (e.g., the processor injecting computed attributes), it must call invalidateAllValuesCache to discard the cache.

func (*Record) ClearForRow added in v0.2.0 

func (r *Record) ClearForRow()

ClearForRow implements encoding.ReusableRecord. Resets per-row state so the next ReadRecordReused call starts from a clean slate while keeping the underlying maps allocated.

values is left intact because every field is overwritten on every row. nulls and wide are cleared because their entries are sparse.

func (*Record) IsNull added in v0.20.0 

func (r *Record) IsNull(name string) bool

IsNull reports whether the named field is null on this record. A field is null when explicitly marked via SetNull / SetNullField (the on-wire bitmap signal during decode), or when the field is absent from both the values map and the wide map. Used by the orchestrator's filter pass to track the n_null_input universal-floor counter per FiltererComponents slot without coupling each filterer to a particular null-detection path.

FILTER_EXPRESSION carries no Field and callers MUST skip the IsNull check for that filterer kind — expression filters do not have a single source field whose null state would be meaningful.

func (*Record) NumericValue 

func (r *Record) NumericValue(name string) (float64, bool)

NumericValue returns the numeric value for the named field. Returns the value and true if present and non-null, or 0 and false if null or missing.

func (*Record) Schema 

func (r *Record) Schema() *encoding.Schema

Schema returns the record's schema.

func (*Record) Set added in v0.2.0 

func (r *Record) Set(name string, value float64)

Set assigns a numeric value to the named field on this record. It clears any prior null marker and invalidates the AllValues cache. Used by pre-filter feature operators to inject derived columns into the record stream so downstream stages (filters, attributes, groupers, aggregators) can reference them by label.

func (*Record) SetLabels added in v0.15.0 

func (r *Record) SetLabels(name string) ([]string, bool)

SetLabels decodes a set-typed field's bitmask into a slice of resolved dictionary labels in bit order (i.e. dictionary insertion order, which is also the bit-assignment order). Returns (nil, false) when the field is null, missing, not a set type, or has no dictionary. Empty mask returns ([]string{}, true) — the field is present but the respondent picked nothing.

func (*Record) SetNull added in v0.2.0 

func (r *Record) SetNull(name string)

SetNull marks the named field as null. Used by feature operators to propagate input nulls into the derived column.

func (*Record) SetNullField added in v0.2.0 

func (r *Record) SetNullField(name string)

SetNullField implements encoding.ReusableRecord. Marks a field as null in the reuse path; does not invalidate the AllValues cache (reuse path resets the cache once per row via ClearForRow).

func (*Record) SetNumeric added in v0.2.0 

func (r *Record) SetNumeric(name string, value float64)

SetNumeric implements encoding.ReusableRecord. Assigns a numeric value without touching the null marker or invalidating the AllValues cache. Intended only for the streaming reuse path that calls ClearForRow before each row.

func (*Record) SetValue added in v0.15.0 

func (r *Record) SetValue(name string) (uint64, bool)

SetValue returns the raw bitmask for a set-typed field. Bit i is set when dictionary entry i is selected. Returns (0, false) when the field is null, missing, or not a set type.

The mask is sourced from the typed wide map so bit-level precision is preserved across all four width tiers (set_u8/u16/u32/u64) — callers that need exact-bit semantics MUST NOT read set fields via NumericValue because the float64 echo loses high bits for set_u64.

func (*Record) SetWide added in v0.2.0 

func (r *Record) SetWide(name string, v any)

SetWide assigns a typed wide value to a field. Used by readers and feature operators that produce non-float values (decimal128).

func (*Record) SetWideField added in v0.2.0 

func (r *Record) SetWideField(name string, v any)

SetWideField implements encoding.ReusableRecord. Stores a typed wide value (decimal128) without invalidating the AllValues cache.

func (*Record) StringValue 

func (r *Record) StringValue(name string) (string, bool)

StringValue returns the resolved string value for categorical fields. For non-categorical fields, returns the empty string and false.

func (*Record) WideValue added in v0.2.0 

func (r *Record) WideValue(name string) (any, bool)

WideValue returns the typed wide value for the named field, if present. Wide values are populated for decimal128 fields.

type RecordIterator 

type RecordIterator interface {
	// Next advances to the next record. Returns false when exhausted.
	Next() bool
	// Record returns the current record. Only valid after Next returns true.
	Record() *Record
	// Reset resets the iterator to the beginning.
	Reset()
}

RecordIterator provides sequential access to records.

type ResolverWarning added in v0.10.1 

type ResolverWarning struct {
	Code    errors.Code
	Message string
	Details map[string]any
}

ResolverWarning captures a single resolver-side diagnostic. The caller folds these into the response envelope after Process / Sample / Facet / Export completes.

type ReusableIterator added in v0.2.0 

type ReusableIterator interface {
	SetReuse(bool)
}

ReusableIterator is an optional interface implemented by iterators that can return the same Record pointer across Next() calls, refreshing its values/nulls/wide maps in place. Streaming consumers that consume each record inline (no slice retention) can opt in to drop the per-row map allocations.

Callers MUST consume each record before invoking Next() again — the next call will overwrite the Record's contents.

type RichAggregator added in v0.15.0 

type RichAggregator interface {
	Aggregator
	Rich() (any, error)
}

RichAggregator is the optional sibling of Aggregator for aggregators whose natural output is not a scalar float64 — set-typed aggregators emit []string (resolved labels) or map[string]int (per-label counts).

Callers invoke Aggregate or Finalize first (legacy float64 contract; implementations return a sensible scalar fallback such as popcount or distinct-mask-count), then check whether the aggregator also satisfies RichAggregator. When it does, Rich returns the typed value and the orchestrator routes it into Response.Data rows or Crosstab cells instead of the float64.

Implementations MUST be safe to call Rich exactly once after Aggregate/Finalize. Returning (nil, nil) signals "use the float64 path" — useful when an aggregator implements the interface for type reasons but produced no rich value for a given input.

type RowLocalAttribute added in v0.2.0 

type RowLocalAttribute interface {
	// Row computes this attribute's value for a single record and field.
	// For a pure RowLocalAttribute, no PrePass call is needed; for a
	// TwoPassAttribute, Row must be called only after Finalize.
	Row(record *Record, field string) (float64, error)
}

RowLocalAttribute is the optional sibling of AttributeComputer for attributes whose value depends only on the current row (no first-pass population stats needed). Streaming paths drive RowLocalAttribute.Row inline instead of buffering the full record set.

FORMULA and DATE_PART implement this interface; ZSCORE / TSCORE / NORMALIZED implement TwoPassAttribute (a superset). PERCENTILE does NOT implement either — it needs a sorted view of every value, which forces the buffered path.

type RowTest added in v0.3.0 

type RowTest interface {
	UpdateRow(record *Record) error
	Finalize() (*types.TestResult, error)
}

RowTest is a tier-1 statistical test consumed during the row scan alongside online aggregators. Implementations fold each filter-passing record into running state via UpdateRow and produce a TestResult via Finalize after the iterator is exhausted.

Per-test state is per-instance; callers construct a fresh instance per Process call via a RowTestFactory.

UpdateRow MUST be safe to call zero times (Finalize handles the empty input case). Implementations decide whether a row contributes; null values are typically skipped via the Record's NumericValue / StringValue helpers.

type RowTestFactory added in v0.3.0 

type RowTestFactory func(spec *types.Test, schema *encoding.Schema) (RowTest, error)

RowTestFactory creates a RowTest from a specification.

type RunCountersInput added in v0.20.0 

type RunCountersInput struct {
	// TotalRecords is the total number of records decoded from the
	// cohort BEFORE the filter pass (mirrors Response.Metadata.TotalRows
	// on every Pulse execution path).
	TotalRecords int64

	// FilteredRecords is the post-all-filters record count. When no
	// filterers are declared this equals TotalRecords by definition;
	// when filterers are declared this is the last filterer's nOut (the
	// records admitted to the downstream aggregation / grouping stage).
	FilteredRecords int64

	// NullRecords counts records whose primary aggregation field was
	// null on the post-filter record set. See package comment for the
	// "primary field" definition.
	NullRecords int64

	// ShardCount is the number of shards merged when the cohort is a
	// shard archive. Zero for single-file cohorts.
	ShardCount int

	// PartialCohortReason carries a free-form diagnostic explaining why
	// the run consumed less than the full cohort. Empty when the run
	// completed fully.
	PartialCohortReason string
}

RunCountersInput is the orchestrator-side input shape carrying every counter the run accumulated by terminal flush. The helper materialises a types.RunComponents from this shape and attaches it to the response. All four processing/processor.go exit points (processStreaming / processStreamingGrouped / processStreamingTwoPass / processRecords) compute these values inline and pass them in; the service-layer shard-parallel / parallel-buffered exits compute them via the merged shardPartial state and pass them in via the same shape.

type SeriesHostView added in v0.19.0 

type SeriesHostView struct {
	// contains filtered or unexported fields
}

SeriesHostView is the read-only window a SERIES overlay handler uses to fold a derived projection over a grouped-Process result. Exposing group keys + a value resolver through a view (rather than re-scanning records) is the analogue of the buffered CrosstabHostView contract for the SERIES family: handlers consume the orchestrator's already-computed per-group metric and never re-scan the source cohort.

Construction: NewSeriesHostView(keys, resolver). The view holds the caller's slice pointer to the ordered group-key list and a callable resolver that turns an index into (value, present). The caller retains ownership of both; the view does not mutate.

SERIES alignment contract (FR-A2): handler-emitted SeriesPayload entries align with the host's GroupKeys list element-for-element. The dispatch never reorders, never drops, and never expands the key list; loose alignment (overlay payload as a subset of host keys) is the per-handler responsibility — the dispatch passes the full host key list down so handlers can decide whether each entry surfaces a real value or carries a zero-value/no-Statistic Summary.

Per-group nullity policy: a host that did not produce a record for group i (the orchestrator's group iterator skipped the group, the filter elided every row in the bucket, etc.) surfaces `(0, false)` from `ValueAt(i)`. Handlers emit a SeriesEntry whose Summary has no Statistic / PValue populated — the renderer-facing zero-value Summary is the documented "present but empty" shape (mirrors the MATRIX path's absent-cell handling: the entry slot stays in the parallel slice so downstream consumers can index by group ordinal).

func NewSeriesHostView added in v0.19.0 

func NewSeriesHostView(groupKeys []types.AxisKey, resolver func(i int) (float64, bool)) *SeriesHostView

NewSeriesHostView wraps an ordered list of group keys plus a value resolver as a SERIES host view. groupKeys may be empty (handlers receive an empty SeriesPayload); resolver may be nil for hosts that expose only the structural shape (e.g. a future axis-only host whose overlay handlers read a Ref-driven side channel). When resolver is nil ValueAt always returns (0, false).

The view holds the caller's groupKeys slice header by value; mutating the slice's backing array after construction yields undefined fold behavior. The dispatch contract documents that callers (the grouped Process orchestrator) treat the slice as read-only between the host build and ApplyOverlaysSeries return.

func NewSeriesHostViewWithFields added in v0.19.0 

func NewSeriesHostViewWithFields(groupKeys []types.AxisKey, resolver func(i int) (float64, bool), groupFields []string) *SeriesHostView

NewSeriesHostViewWithFields wraps an ordered list of group keys, a resolver, and the grouper-field list the AxisKey tuples align to. The grouper-field list is the source-of-truth for the sibling resolver — `groupFields[i]` names the field index `i` of every AxisKey corresponds to. Length of groupFields SHOULD match the width of every AxisKey tuple (the orchestrator's grouper list); a shorter list is tolerated and the resolver simply cannot resolve fields outside the declared range. Empty groupFields keeps the resolver in its scan-every-element fallback. The helper exists alongside NewSeriesHostView so legacy handlers (which never consult the field list) keep compiling against the simpler two-argument constructor.

func NewSeriesHostViewWithGrouper added in v0.19.0 

func NewSeriesHostViewWithGrouper(groupKeys []types.AxisKey, resolver func(i int) (float64, bool), groupFields []string, grouperKinds []types.GroupType) *SeriesHostView

NewSeriesHostViewWithGrouper wraps an ordered list of group keys, a resolver, the grouper-field list, and the grouper-kind list the AxisKey tuples are produced from. The grouper-kind list is the source-of-truth for host-shape-introspecting overlay handlers — the `OVERLAY_YOY` handler reads `grouperKinds[0]` to enforce its "host grouper must be GROUP_DATE" requirement at runtime.

Length of `grouperKinds` SHOULD match `groupFields` (the orchestrator's grouper list); a shorter list is tolerated and host-shape-introspecting handlers simply cannot resolve kinds outside the declared range. Empty `grouperKinds` keeps host-shape introspection in its deferred fallback (the handler defers to the predict gate's earlier rejection). The helper exists alongside `NewSeriesHostView` and `NewSeriesHostViewWithFields` so legacy handlers (which never consult the grouper-kind list) keep compiling against the simpler constructors.

func (*SeriesHostView) GroupCount added in v0.19.0 

func (h *SeriesHostView) GroupCount() int

GroupCount returns the number of group tuples on the host. Zero when the view is nil or the underlying slice is empty.

func (*SeriesHostView) GroupFields added in v0.19.0 

func (h *SeriesHostView) GroupFields() []string

GroupFields returns the grouper Fields the host's AxisKey tuples align to element-for-element. Returns nil when the view is nil OR when the view was built via NewSeriesHostView (the field list is optional). The sibling resolver consults this slot to map a `Ref.Sibling.Field` to the matching axis-key element index; when the slot is empty the resolver falls back to scanning every axis-key element for a value match. Callers consume the returned slice read-only — the slice header points at the host's backing array.

func (*SeriesHostView) GroupKey added in v0.19.0 

func (h *SeriesHostView) GroupKey(i int) (types.AxisKey, bool)

GroupKey returns the composite group-key tuple at index i. Returns (nil, false) when i is out of range. Callers consume the returned AxisKey read-only — the slice header points at the host's backing array and mutating it would corrupt the dispatch's parallel-slice alignment with downstream handler emissions.

func (*SeriesHostView) GroupKeys added in v0.19.0 

func (h *SeriesHostView) GroupKeys() []types.AxisKey

GroupKeys returns the full ordered group-key slice. Callers consume the slice read-only — the slice header points at the host's backing array. The helper exists so a handler that needs the full key list (e.g. to copy keys into the SeriesPayload for FR-A2 alignment) can avoid the per-index GroupKey loop. Returns nil when the view is nil.

func (*SeriesHostView) GrouperKinds added in v0.19.0 

func (h *SeriesHostView) GrouperKinds() []types.GroupType

GrouperKinds returns the grouper Types the host's AxisKey tuples are produced from, element-for-element with `GroupFields()`. Returns nil when the view is nil OR when the view was built via `NewSeriesHostView` / `NewSeriesHostViewWithFields` (the slot is optional). The `OVERLAY_YOY` handler consults `GrouperKinds[0]` to enforce its "host grouper must be GROUP_DATE" requirement at runtime. When the slot is empty the YoY handler defers to the predict gate's earlier rejection (predict already surfaces `PULSE_OVERLAY_REF_INCOMPATIBLE_WITH_SHAPE` for non-DATE hosts via `descriptor.validateOverlayYoY`). Callers consume the returned slice read-only — the slice header points at the host's backing array.

func (*SeriesHostView) ValueAt added in v0.19.0 

func (h *SeriesHostView) ValueAt(i int) (float64, bool)

ValueAt returns the host's metric value for the group at index i plus a present flag mirroring CrosstabHostView.CellAt. Returns (0, false) when the index is out of range, the view is nil, the resolver is nil, or the resolver itself reports the group as absent. A handler should treat (0, false) as "no observation" — emit a SeriesEntry whose Summary leaves Statistic / PValue unset rather than computing against a sentinel value.

type SliceIterator 

type SliceIterator struct {
	// contains filtered or unexported fields
}

SliceIterator implements RecordIterator over a slice of records.

func NewSliceIterator 

func NewSliceIterator(records []*Record) *SliceIterator

NewSliceIterator creates an iterator over the given records.

func (*SliceIterator) Next 

func (it *SliceIterator) Next() bool

Next advances to the next record.

func (*SliceIterator) Record 

func (it *SliceIterator) Record() *Record

Record returns the current record.

func (*SliceIterator) Reset 

func (it *SliceIterator) Reset()

Reset resets the iterator to the beginning.

type StreamableGrouper added in v0.17.0 

type StreamableGrouper interface {
	Grouper
	// KeyFor returns the bucket key for record on the grouper's bound
	// field. Returns (key, nil) on success. Returns ("", ErrGrouperKeyNull)
	// when the record's field is null, missing, or otherwise has no
	// defined bucket (e.g. an empty mask on SET_VALUE remains a valid
	// key of ""; only an absent/null mask triggers the sentinel).
	KeyFor(record *Record) (string, error)
}

StreamableGrouper is the field-bound sibling of StreamingGrouper used by the fused crosstab path. The grouper instance carries its target field (extracted from the *types.Group at factory time), so the per-record hot path can compute a bucket key with a single method call and no extra argument plumbing — important for tight decode loops that route each record into the correct cell accumulator.

Semantics mirror StreamingGrouper.KeyForRow without the explicit field argument and without the ok bool. Null / missing values are signalled by returning ErrGrouperKeyNull instead of a non-nil (key, nil) pair — empty-string is a legitimate bucket key for set groupers, so an in-band string sentinel is not safe.

Streamable groupers (CATEGORY, RANGE, ROUNDED, DATE, SET_VALUE) all implement this interface; non-streamable groupers (QUANTILE) and multi-key streaming groupers (SET_PER_ELEMENT) do NOT — callers must use interface assertion to discriminate.

type StreamingGrouper added in v0.2.0 

type StreamingGrouper interface {
	// KeyForRow returns the group key string for record's value of field.
	// ok=false signals the row should be skipped (e.g. null value);
	// ok=true means the key is valid for bucketing.
	KeyForRow(record *Record, field string) (key string, ok bool, err error)
}

StreamingGrouper is the optional sibling of Grouper for groupers that can derive a partition key from a single record without seeing the full record set. CATEGORY / RANGE / ROUNDED implement this interface; QUANTILE and DATE require finalize-time work over the full input.

Implementations MUST be safe to call repeatedly; the streaming path invokes KeyForRow once per filter-passing record and uses the key to index a per-group online aggregator bucket.

type StringSet added in v0.8.3 

type StringSet struct {
	// contains filtered or unexported fields
}

StringSet stores string membership (decimal128 fields, fallback path).

func (*StringSet) Add added in v0.8.3 

func (s *StringSet) Add(v string)

func (*StringSet) Contains added in v0.8.3 

func (s *StringSet) Contains(v string) bool

func (*StringSet) Kind added in v0.8.3 

func (s *StringSet) Kind() string

func (*StringSet) Len added in v0.8.3 

func (s *StringSet) Len() int

type TwoPassAttribute added in v0.2.0 

type TwoPassAttribute interface {
	RowLocalAttribute
	// PrePass folds a single record's contribution into the running
	// state used to compute population statistics.
	PrePass(record *Record, field string) error
	// Finalize closes the PrePass phase. After Finalize, Row may be
	// called for each record (typically during iter pass 2).
	Finalize() error
}

TwoPassAttribute is the streaming-friendly path for attributes that need population statistics (ZSCORE / TSCORE need mean+stddev, NORMALIZED needs min+max). The orchestrator drives PrePass over every filter-passing record, then Finalize locks the global stats, then Row emits per-record values during a second iter pass.

Mirrors feature.StreamingComputer.PrePass+Finalize+EmitRow so the streaming infrastructure (iter.Reset(), staged passes) is uniform.

Implementations MUST be safe to call PrePass repeatedly; Finalize exactly once between PrePass and Row; and Row only after Finalize. State is per-instance — callers construct a fresh instance per Process call via the AttributeFactory.

type Uint64Set added in v0.8.3 

type Uint64Set struct {
	// contains filtered or unexported fields
}

Uint64Set stores integer membership.

func (*Uint64Set) Add added in v0.8.3 

func (s *Uint64Set) Add(v uint64)

func (*Uint64Set) Contains added in v0.8.3 

func (s *Uint64Set) Contains(v uint64) bool

func (*Uint64Set) Kind added in v0.8.3 

func (s *Uint64Set) Kind() string

func (*Uint64Set) Len added in v0.8.3 

func (s *Uint64Set) Len() int

type WelfordTriple added in v0.19.1 

type WelfordTriple struct {
	Mean     float64 `json:"mean"`
	Variance float64 `json:"variance"`
	N        uint64  `json:"n"`
}

WelfordTriple is the Rich payload returned by AGG_WELFORD. Mean is the running mean, Variance is the unbiased sample variance (M2 / (n - 1); zero when N < 2), and N is the number of non-null rows that contributed.

Returned by value (not via a pointer) so downstream consumers — the crosstab MatrixCell builder, overlay handlers — can pass it through the `any`-typed Value slot without pointer-aliasing surprises across copies.

Source Files

View all Source files

Directories

Path Synopsis
Package arena provides a bump-allocator backed by a single contiguous []byte.
 Package arena provides a bump-allocator backed by a single contiguous []byte.
Package feature implements the FEAT_* operators that run pre-filter to add derived columns to a record stream.
 Package feature implements the FEAT_* operators that run pre-filter to add derived columns to a record stream.
Package regression hosts the REG_* operators that fit a regression model against the filtered record set.
 Package regression hosts the REG_* operators that fit a regression model against the filtered record set.
Package window implements the WIN_* window operators for Pulse.
 Package window implements the WIN_* window operators for Pulse.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.