Documentation
¶
Overview ¶
Package descriptor provides self-description, manifest, and predict functionality for pulse.
Index ¶
- Constants
- func BuildPayloadSchema() json.RawMessage
- func NormalizeRequest(req *types.Request, schema *encoding.Schema) *types.Request
- func ValidateFacetOverlays(env *Envelope, req *types.FacetRequest, schema *encoding.Schema)
- func ValidateLabels(env *Envelope, bindings []*types.LabelBinding, schema *encoding.Schema, ...) (augmentNames map[string]bool)
- func ValidateOverlays(env *Envelope, req *types.Request, schema *encoding.Schema, ...)
- type AggregationPredict
- type ChainOverlaySchemaDivergence
- type ChainValidationResult
- type CohortFieldType
- type Command
- type CommandAnnotations
- type ComponentKey
- type ComponentSchema
- type Components
- type ComponentsMergeability
- type ComponentsSchemasBlock
- type ComposeValidationResult
- type CrosstabCapability
- type DefaultApplied
- type DictionaryInfo
- type DistributionMeta
- type Envelope
- func Inspect(fileData io.ReadSeeker, opts *InspectOptions) *Envelope
- func InspectFromBytes(data []byte, opts *InspectOptions) *Envelope
- func NewEnvelope(data any) *Envelope
- func NewEnvelopeWithRequest(data, request any) *Envelope
- func Predict(fileData io.ReadSeeker, req *types.Request, opts *PredictOptions) *Envelope
- func PredictFromBytes(data []byte, req *types.Request, opts *PredictOptions) *Envelope
- func ValidateChain(fileData io.ReadSeeker, req *types.ChainRequest) *Envelope
- func ValidateChainFromBytes(data []byte, req *types.ChainRequest) *Envelope
- func ValidateCompose(req *types.ComposedRequest) *Envelope
- func ValidateFacet(fileData io.ReadSeeker, req *types.FacetRequest) *Envelope
- func ValidateFacetFromBytes(data []byte, req *types.FacetRequest) *Envelope
- func ValidateFacetWithExtensions(fileData io.ReadSeeker, req *types.FacetRequest, snap *ExtensionsSnapshot) *Envelope
- func ValidateJoin(leftData, rightData io.ReadSeeker, req *types.Request) *Envelope
- func ValidateJoinFromBytes(left, right []byte, req *types.Request) *Envelope
- type EnvelopeEntry
- type ExportCapability
- type ExportFormatCapability
- type ExprFunctionMeta
- type ExtensionsManifest
- type ExtensionsSnapshot
- type FacetCapability
- type FacetValidationResult
- type FiltererPredict
- type GroupPredict
- type InspectField
- type InspectOptions
- type InspectResult
- type JoinCapability
- type JoinValidationResult
- type LabelTableMeta
- type LookupTableMeta
- type MCPTool
- type Manifest
- type Operator
- type OperatorMeta
- type OperatorParamMeta
- type OverlayAppliedDescriptor
- type OverlayCapability
- type Param
- type PredictOptions
- type PredictResult
- type PredictSchemaInfo
- type ProcessChainCapability
- type RegressionMeta
- type RegressionModifier
- type ShardInfo
- type SkillMeta
- type SlotPair
- type Suggestion
- type TestMeta
Constants ¶
const ( // Mergeable forwards types.ComponentsMergeable. The components map // folds across chunks via the same associative/commutative path // as the scalar value (Welford-family, sums, counts, set masks, // weighted accumulators). Mergeable = types.ComponentsMergeable // Partial forwards types.ComponentsPartial. The components map // merges across chunks but at non-trivial allocation cost — map / // set unions where the fold is associative but not constant- // space (AGG_FREQUENCY, AGG_MODE, AGG_DISTINCT_COUNT, // AGG_SET_FREQUENCY). The orchestrator may stage the merge at // terminal flush. Partial = types.ComponentsPartial // None forwards types.ComponentsNone. The components map cannot // be computed from a per-chunk partial — the operator needs a // sorted view (or equivalent) of the full input (AGG_MEDIAN, // AGG_PERCENTILE, GROUP_QUANTILE). Streaming chunks omit // components for these operators; predict declares the slot as // buffered-components-only. None = types.ComponentsNone )
const DefaultDictionaryLimit = 100
DefaultDictionaryLimit is the default max entries shown for categorical dictionaries.
const PayloadSchemaFormatVersion = "1.1"
PayloadSchemaFormatVersion tracks the envelope format_version (see Envelope.FormatVersion / NewEnvelope). The published payload schema's $id embeds it; bump the two in lockstep. Guarded by TestPayloadSchema_VersionMatchesEnvelope.
Variables ¶
This section is empty.
Functions ¶
func BuildPayloadSchema ¶ added in v0.23.0
func BuildPayloadSchema() json.RawMessage
BuildPayloadSchema returns the bundled JSON Schema (draft 2020-12) describing every public Pulse payload — the request envelopes (Request, ComposedRequest, ChainRequest, FacetRequest, SampleRequest) and the universal output Envelope plus every operation result it can wrap.
The schema is generated three ways, not hand-maintained:
- Reflection over the Go payload structs (descriptor stays free of service/ and processing/ imports — only types + this package). A struct field change surfaces as a golden diff, forcing a regen.
- Registry-injected enums: the high-cardinality discriminant enums (operator + overlay-kind + regression families) draw their value lists from types.All*Types() / AllOverlayKinds() / AllRegressionTypes(), so registering a new operator changes the schema and trips the gate.
- Hand-tuned strict unions for the two shapes reflection cannot express faithfully: OverlayRef (at-most-one populated arm) and OverlayPayload (shape-discriminated scalar/series/matrix).
v1 boundaries (documented in docs/src/contract/payload-schema.md):
- Operator Params (json.RawMessage) stay an open object — there is no central declarative per-operator input-param source (even the MCP binder treats params as open); the per-operator param schema lives alongside each operator's processor.
- The small closed mode enums (OverlayScope, OverlayShape, CrosstabNormalize, etc.) are typed as plain strings: they have no All*() registry helper, so a hardcoded enum here would rot silently.
The result marshals deterministically: every object is a map (encoding /json sorts keys) and every enum / required list is sorted, so the golden is stable across runs and Go versions.
func NormalizeRequest ¶ added in v0.13.0
NormalizeRequest returns a clone of req with smart defaults resolved against schema. Caller-side ergonomic for envelope.Request echo — produces the exact request the engine would execute (post-defaults) without mutating the input. Returns nil when req is nil; returns the clone unchanged when schema is nil or no defaults applied.
The clone is a shallow-deep hybrid identical to the one Predict uses internally: Aggregations and Groups are deep-cloned (the slots ResolveDefaults mutates), every other slot is shared with the input. Safe to expose on read-only paths.
func ValidateFacetOverlays ¶ added in v0.19.0
func ValidateFacetOverlays(env *Envelope, req *types.FacetRequest, schema *encoding.Schema)
ValidateFacetOverlays walks every spec in req.Overlays and appends structural errors to the envelope. Sibling to ValidateOverlays — the Request-host validator — but enforces the FACET-host contract: the host is a FacetSchema FacetResult, the population reference is resolved against an alternate cohort, and the host-field the spec decorates is selected from FacetRequest.Fields.
No-op when req is nil or req.Overlays is empty. The schema is used for the per-kind host-arm gates (CHISQ rejects numeric; KS rejects categorical); when schema is nil those gates short-circuit.
func ValidateLabels ¶ added in v0.10.1
func ValidateLabels( env *Envelope, bindings []*types.LabelBinding, schema *encoding.Schema, snap *ExtensionsSnapshot, extraFields map[string]bool, ) (augmentNames map[string]bool)
ValidateLabels walks a slice of LabelBindings against the resolved schema and the extension snapshot, surfacing per-binding failures as envelope errors. ValidateLabels never reads record data — it is safe to call from any no-execute predict path.
Validation order is deterministic: bindings are inspected in input order; the first failure per binding is recorded but later bindings are still examined so callers see every problem in one pass.
`extraFields` lists output-only field names that count as occupying the schema namespace for augment-mode collision detection (e.g. aggregation output labels, attribute labels). Pass nil when only raw schema fields matter.
Returns the set of (output) field names occupied by augment-mode sibling columns so callers can fold them into downstream collision checks (group keys, sort keys against the augmented schema).
func ValidateOverlays ¶ added in v0.19.0
func ValidateOverlays(env *Envelope, req *types.Request, schema *encoding.Schema, opts *PredictOptions)
ValidateOverlays walks every spec in req.Overlays and appends structural errors to the envelope. Exported so engines and tests can run the validator standalone; Predict wires it inline so overlay issues surface alongside crosstab / aggregator / test ones.
No-op when req is nil or req.Overlays is empty. Schema is currently unused — every rule is structural — but the signature accepts it so later kinds can validate against schema-derived state (e.g. referenced field types on sibling / baseline-index families) without re-opening every call site.
Level / Within out-of-range gate: runs alongside the per- kind ref/scope checks via validateOverlayLevelWithinPredict. Mirrors the PULSE_CROSSTAB_NORMALIZE_LEVEL_OUT_OF_RANGE shape on the crosstab path — out-of-range slots surface PULSE_OVERLAY_LEVEL_OUT_OF_RANGE on the envelope with Details carrying the spec index, kind, level / within, and the axis depth. The runtime mirror lives at processing.validateOverlayLevelWithinRuntime so a programmatic Process caller that skipped predict still gets the same failure shape.
Types ¶
type AggregationPredict ¶ added in v0.20.0
type AggregationPredict struct {
// Type echoes the aggregation type from the request slot, in the
// canonical SCREAMING_SNAKE form (e.g. "AGG_MEDIAN").
Type types.AggregationType `json:"type"`
// Field echoes the field referenced by the request slot. Empty
// when the slot was authored without a field (defaults may have
// filled it; the resolved post-defaults clone is reflected here
// when applicable).
Field string `json:"field,omitempty"`
// Label echoes the response-side label the caller assigned to the
// slot (or the engine-default label rule fallback when empty).
// Predict echoes the raw label only — label fallback synthesis is
// the orchestrator's responsibility.
Label string `json:"label,omitempty"`
// ComponentSchema is the static schema for the operator the engine
// will use at runtime, projected from the capabilities table
// (capabilities_aggregators.go for built-ins; the embedder snapshot
// for extensions). Empty Keys + empty Mergeability means the slot
// references an extension operator that has not (yet) declared a
// ComponentSchema — the universal-floor populator still attaches
// {"n", "n_null"} at runtime.
ComponentSchema ComponentSchema `json:"component_schema"`
// BufferedComponents is true iff ComponentSchema.Mergeability is
// descriptor.None. The flag advertises that the slot's Components
// block will arrive only on the terminal buffered flush in a
// streaming run; it does NOT flip the overall Streamable axis (the
// data slice still streams). AGG_MEDIAN and AGG_PERCENTILE are the
// canonical Nones today.
BufferedComponents bool `json:"buffered_components,omitempty"`
}
AggregationPredict is the per-slot predict surface for one entry of req.Aggregations. Carries the operator identity echoed back to the caller plus the static ComponentSchema declared in the capabilities table — so callers can budget the runtime Components block shape before the engine runs and so streaming consumers can branch on the BufferedComponents hint without consulting the manifest.
Predict stays no-execute: every field on this struct is sourced from the descriptor capability projection (or the embedder snapshot for extension operators) — never from a constructed operator. Custom operators registered via pulse.Options.Extensions that do not (yet) supply a ComponentSchema produce an empty ComponentSchema and BufferedComponents=false; the universal-floor populator at orchestrator emission time still attaches {"n", "n_null"} at runtime.
type ChainOverlaySchemaDivergence ¶ added in v0.19.0
type ChainOverlaySchemaDivergence struct {
// Index is the position of the offending spec in
// ChainRequest.Overlays.
Index int `json:"index"`
// Kind echoes the spec's whole-chain overlay kind.
Kind types.OverlayKind `json:"kind"`
// RefIndex is the resolved stage index of the spec's Ref.
RefIndex int `json:"ref_index"`
// RefShape is the inferred OverlayShape of the Ref stage's
// Request slot.
RefShape types.OverlayShape `json:"ref_shape"`
// TargetIndex is the resolved stage index of the spec's Target
// (defaulting to the latest stage when both Target slots were
// empty per the runtime resolver contract).
TargetIndex int `json:"target_index"`
// TargetShape is the inferred OverlayShape of the Target stage's
// Request slot.
TargetShape types.OverlayShape `json:"target_shape"`
}
ChainOverlaySchemaDivergence carries one rejected (Ref, Target) chain-overlay pair from ValidateChain's overlay walk. Mirrors the per-spec envelope-error Details shape so wire consumers can read the list directly instead of stitching envelope entries together.
type ChainValidationResult ¶ added in v0.10.0
type ChainValidationResult struct {
// Valid mirrors envelope.Errors emptiness.
Valid bool `json:"valid"`
// Request echoes the input chain request unchanged.
Request *types.ChainRequest `json:"request"`
// SchemaInfo summarises the source cohort schema used for stage 0.
SchemaInfo *PredictSchemaInfo `json:"schema_info,omitempty"`
// StageSchemas exposes the inferred output schema per stage as a
// list of field names. Index i carries the columns produced by
// stage i (input to stage i+1). Useful for debugging chain breaks
// when stage N+1 cannot find a field stage N was expected to emit.
StageSchemas [][]string `json:"stage_schemas,omitempty"`
// OverlaysSchemaDivergence lists every rejected (Ref, Target)
// chain-overlay pair from the ValidateChain overlay walk per
// kind-catalog-v1 PRD §I-FR-I3. Populated alongside the matching
// PULSE_OVERLAY_CHAIN_STAGE_SHAPE_DIVERGENT envelope error so LLM
// planners can budget reshapes without re-parsing envelope details.
// Empty (and omitted on the wire) when every chain-overlay spec
// passes the shape-match gate.
OverlaysSchemaDivergence []ChainOverlaySchemaDivergence `json:"overlays_schema_divergence,omitempty"`
}
ChainValidationResult is the structured output of ValidateChain. Mirrors FacetValidationResult: the resolved request echoes back so callers can forward it after inspection, and per-stage schemas expose the inferred lineage for debugging.
type CohortFieldType ¶
type CohortFieldType struct {
Name string `json:"name"`
Categorical bool `json:"categorical"`
ShardedCapable bool `json:"sharded_capable"`
CompatibleAggregators []string `json:"compatible_aggregators,omitempty"`
CompatibleAttributes []string `json:"compatible_attributes,omitempty"`
CompatibleFilterers []string `json:"compatible_filterers,omitempty"`
CompatibleGroupers []string `json:"compatible_groupers,omitempty"`
CompatibleWindows []string `json:"compatible_windows,omitempty"`
CompatibleFeatures []string `json:"compatible_features,omitempty"`
}
CohortFieldType describes a field type available in .pulse files and the operator catalog that accepts it. The Compatible* slices are derived from the per-operator AcceptsTypes declarations and let an LLM look up "what can I do with a date field" in one place.
ShardedCapable reports whether the type participates in a shard archive without restriction. Every built-in field type is sharded- capable today; the flag exists for forward compatibility with future types that might not work across the union of shards (e.g. types whose semantics depend on per-shard locality). Embedders should treat the flag as advisory.
type Command ¶
type Command struct {
Name string `json:"name"`
Description string `json:"description"`
// Annotations carries the three per-command capability hints
// (streamable / deterministic / expensive). Embedders use these to
// decide whether to wrap a command in caching or invoke it directly.
// Always populated for built-in commands.
Annotations CommandAnnotations `json:"annotations"`
}
Command describes a CLI leaf command in the manifest.
type CommandAnnotations ¶ added in v0.11.0
type CommandAnnotations struct {
Streamable bool `json:"streamable"`
Deterministic bool `json:"deterministic"`
Expensive bool `json:"expensive"`
}
CommandAnnotations carries three capability flags per command.
- Streamable: the command has a streaming variant. Callers can invoke the streaming form for incremental output.
- Deterministic: the command produces the same output given the same inputs (including the source file's content hash). Callers can safely cache results keyed by the request hash.
- Expensive: the command is worth caching. Cheap operations may not be worth the cache machinery; expensive ones (regression, filter-to-file, profile) typically are. Hint to consumers, not a hard constraint.
type ComponentKey ¶ added in v0.20.0
type ComponentKey struct {
// Name is the snake_case key the operator emits into the runtime
// map[string]any returned by MetaAggregator.Components.
Name string `json:"name"`
// Type is the human-readable kind name of the emitted value.
Type string `json:"type"`
// Description is a single-sentence prose description suitable for
// manifest / predict / MCP surfaces.
Description string `json:"description"`
}
ComponentKey describes one named entry in an operator's components map. The triple (Name, Type, Description) is the manifest projection embedders consume to learn what shape of value to expect at runtime.
Type is a human-readable kind string (e.g. "int", "float64", "map[string]int", "[]string", "WelfordTriple") rather than a reflect.Kind so the manifest can serialize it directly into JSON without an extra mapping table. Implementations may use anything that round-trips through manifest consumers — keep it short and unambiguous.
Name is the wire-stable snake_case key the operator emits into the runtime components map (e.g. "mean", "variance", "mode_count", "range_min"). Description is a single sentence in plain prose used by the manifest, predict, and MCP surfaces. Both are required.
type ComponentSchema ¶ added in v0.20.0
type ComponentSchema struct {
// Keys enumerates the named operator-specific components, in
// emission order. May be empty when only the universal floor
// applies.
Keys []ComponentKey `json:"keys,omitempty"`
// Mergeability classifies how the components map folds across
// streaming chunks. Required for every schema (no zero-value
// default — embedders must declare intent explicitly).
Mergeability ComponentsMergeability `json:"mergeability"`
}
ComponentSchema is the per-operator declaration carried by every descriptor capability entry. The Keys slice enumerates the operator- specific components emitted at runtime (the universal floor of {"n", "n_null"} is filled unconditionally by the orchestrator and is NOT listed here); Mergeability classifies how those components fold across streaming chunks.
An empty Keys slice paired with Mergeability == Mergeable is a valid schema — it declares an operator whose only contribution is the universal floor (AGG_COUNT is the canonical example). Consumers receiving an empty Keys slice still see {"n", "n_null"} in the runtime response under the orchestrator's universal-floor pass.
Manifest serialization preserves Keys order — declare keys in a stable order matching the operator's emission for golden-test readability.
type Components ¶
type Components struct {
Aggregators []Operator `json:"aggregators"`
Attributes []Operator `json:"attributes"`
Filterers []Operator `json:"filterers"`
Groupers []Operator `json:"groupers"`
Windows []Operator `json:"windows"`
Features []Operator `json:"features"`
}
Components lists every registered processing component grouped by category. Each slice carries one Operator entry per component so LLM-side authoring has access to per-operator params, accepted field types, emit type, and streamability without further discovery round-trips.
type ComponentsMergeability ¶ added in v0.20.0
type ComponentsMergeability = types.ComponentsMergeability
ComponentsMergeability is re-exported from types/ where the canonical enum lives (see types/streamability.go). Descriptor aliases the type and forwards the constants so existing capability-file declarations (descriptor.Mergeable / descriptor.Partial / descriptor.None) and predict-side switches keep compiling unchanged while the source of truth is the leaf-package types declaration. Wire JSON values are "mergeable" / "partial" / "none".
func AggregationMergeability ¶ added in v0.20.0
func AggregationMergeability(t types.AggregationType) ComponentsMergeability
AggregationMergeability is the typed convenience that accepts a types.AggregationType directly so callers do not have to round-trip through string(...).
func AggregatorMergeability ¶ added in v0.20.0
func AggregatorMergeability(name string) ComponentsMergeability
AggregatorMergeability returns the ComponentsMergeability declared in the built-in aggregator capability table for the operator named by name (e.g. "AGG_SUM"). Unknown names return the zero-value empty string — callers should treat that as "no streaming guarantee" and either suppress per-chunk emission or fall back to terminal-only.
This is the one-shot lookup the streaming orchestrator (pulse.ProcessStreamResult) consults to decide whether to attach a per-chunk components map (Mergeable / Partial) or suppress it until the terminal chunk (None).
Lives in the descriptor package because the capability table is the single source of truth and predict already aliases the same lookup (aggregatorComponentSchemaIndex). The descriptor package stays no-execute — this helper reads only the static capabilities slice.
func GroupMergeability ¶ added in v0.20.0
func GroupMergeability(t types.GroupType) ComponentsMergeability
GroupMergeability is the typed convenience that accepts a types.GroupType directly so callers do not have to round-trip through string(...).
func GrouperMergeability ¶ added in v0.20.0
func GrouperMergeability(name string) ComponentsMergeability
GrouperMergeability returns the ComponentsMergeability declared in the built-in grouper capability table for the operator named by name (e.g. "GROUP_CATEGORY"). Unknown names return the zero-value empty string with the same "no streaming guarantee" semantics as AggregatorMergeability.
GROUP_QUANTILE is the canonical None grouper — quantile cutoffs need the sorted full input, so the streaming orchestrator suppresses the per-chunk Operator emission for that slot until the terminal chunk. Every other registered grouper today (CATEGORY, DATE, RANGE, ROUNDED, SET_VALUE, SET_PER_ELEMENT) is Mergeable.
Lives alongside AggregatorMergeability in the descriptor package so the streaming projection helper has a single import path for both axes; the underlying lookup reads only the static grouperCapabilities() slice and never touches `service/` or `processing/`.
type ComponentsSchemasBlock ¶ added in v0.20.0
type ComponentsSchemasBlock struct {
Aggregators map[string]ComponentSchema `json:"aggregators,omitempty"`
Groupers map[string]ComponentSchema `json:"groupers,omitempty"`
Filterers map[string]ComponentSchema `json:"filterers,omitempty"`
}
ComponentsSchemasBlock surfaces the per-operator components contract (ResponseComponents.Components[] payload shape) at manifest level so LLM clients can reason about meta-fields without crawling per- category Operator entries. Each map is keyed by operator name and sorted deterministically at serialization time.
An entry's ComponentSchema mirrors the same value carried on the per-Operator entry under Components.Aggregators[i].ComponentSchema. The two surfaces are deliberately redundant — Operator-level keeps the per-operator entry self-contained for category-by-category drilldown, while this top-level block lets a client materialize the whole meta-fields catalog in one O(N) scan.
type ComposeValidationResult ¶ added in v0.19.0
type ComposeValidationResult struct {
// Valid mirrors envelope.Errors emptiness.
Valid bool `json:"valid"`
// Request echoes the input composed request unchanged.
Request *types.ComposedRequest `json:"request"`
// OverlaysSchemaDivergence lists every (reference, target)
// pair the overlay walk rejected per PRD §I-FR-I3. Populated
// alongside the matching envelope error so MCP planners can
// budget reshapes without re-parsing envelope details. Empty
// (but non-nil) when every spec passes; never nil so JSON
// renderers see an empty array, matching PredictResult's
// contract.
OverlaysSchemaDivergence []SlotPair `json:"overlays_schema_divergence"`
// OverlayCost maps each COMPOSE-host overlay-spec Name to a coarse
// cost score the routing layer can consult before execution. Sibling
// to PredictResult.OverlayCost / FacetValidationResult.OverlayCost —
// streamable kinds carry overlayCostStreamable (~5% extra work) and
// buffered kinds carry overlayCostBuffered (~one extra payload
// traversal). The two multi-ref kinds (OVERLAY_PROP_Z_PANEL,
// OVERLAY_PANEL_INDEX_VS_REF) scale the base cost by
// `min(len(spec.Targets), MaxPanelTargets)` so renderers can budget
// per-panel fan-out without re-deriving the cap. Single-target kinds
// ignore the Targets slice and surface the raw kind cost. Per
// kind-catalog-v1 PRD §I-FR-I3 the value is intentionally coarse —
// callers budgeting cost across slots sum the map values. Empty (but
// non-nil) when req.Overlays is empty; never nil in JSON output.
OverlayCost map[string]float64 `json:"overlay_cost"`
}
ComposeValidationResult is the structured output of ValidateCompose. Mirrors ChainValidationResult / FacetValidationResult: the input request echoes back so callers can forward it after inspection, and the per-spec divergence slice is populated alongside the envelope errors so renderers can read the rejected (reference, target) pairs without re-parsing envelope details.
type CrosstabCapability ¶ added in v0.12.0
type CrosstabCapability struct {
// Name is the canonical entry identifier ("crosstab").
Name string `json:"name"`
// NormalizeModes lists every valid CrosstabSpec.Normalize value,
// sorted alphabetically.
NormalizeModes []string `json:"normalize_modes"`
// Shapes lists every valid CrosstabSpec.Shape value, sorted
// alphabetically.
Shapes []string `json:"shapes"`
// SummableAggregators is the alphabetized list of aggregator names
// whose margin equals the sum of cells. v1 recomputes every margin
// from raw rows for correctness; the classification is exposed so
// callers can reason about future fast-path optimizations.
SummableAggregators []string `json:"summable_aggregators"`
// MeanReducibleAggregators lists aggregators whose margin is
// derivable when each cell also carries its count.
MeanReducibleAggregators []string `json:"mean_reducible_aggregators"`
// RecomputeAggregators lists aggregators whose margin cannot be
// derived from cells and must be recomputed over the raw filter-
// passing rows.
RecomputeAggregators []string `json:"recompute_aggregators"`
// StreamingExceptions documents the only request shape that may
// stream through the standard grouped path: shape=long, no
// margins, no normalization. Intended for LLM-side reasoning.
StreamingExceptions []string `json:"streaming_exceptions"`
// RejectionRules names the structural failures the validator
// emits. Intended for LLM-side autocomplete; not a strict schema.
RejectionRules []string `json:"rejection_rules"`
// SupportsNormalizeLevel reports whether the engine honors
// CrosstabSpec.NormalizeLevel — the depth-index selector that
// picks a parent grouper as the 100% denominator on nested row /
// column axes. Always true in v1; carried so clients can detect
// the feature without inferring it from CLI version.
SupportsNormalizeLevel bool `json:"supports_normalize_level"`
// NormalizeLevelRules names the rejection rules specific to the
// NormalizeLevel slot. Intended for LLM-side autocomplete.
NormalizeLevelRules []string `json:"normalize_level_rules"`
// SupportsNormalizeWithin reports whether the engine honors
// CrosstabSpec.NormalizeWithin — the cross-axis partition
// selector that fixes a prefix of the OPPOSITE axis (columns
// when normalize=row, rows when normalize=column) inside the
// 100% denominator. Composes independently with NormalizeLevel.
SupportsNormalizeWithin bool `json:"supports_normalize_within"`
// NormalizeWithinRules names the rejection rules specific to the
// NormalizeWithin slot.
NormalizeWithinRules []string `json:"normalize_within_rules"`
// MapValuedCellAggregators is the alphabetized list of aggregator
// names whose Crosstab Cell output is map-valued (rich payload
// per cell, e.g. AGG_SET_FREQUENCY's per-label row counts). These
// aggregators emit map[string]int into MatrixCell.Value and the
// long-shape cell rows; normalize modes (row / column / total) are
// incompatible because dividing one map by another is undefined.
// Derived from types.AggregationType.MapValued() at build time.
MapValuedCellAggregators []string `json:"map_valued_cell_aggregators"`
}
CrosstabCapability describes the cross-tabulation endpoint surfaced via Request.Crosstab. The manifest carries one CrosstabCapability entry under Manifest.Crosstab so LLM clients can detect the section, learn the supported normalize / shape values, and route between the matrix and long shapes appropriately.
type DefaultApplied ¶ added in v0.5.0
type DefaultApplied struct {
Path []string `json:"path"`
Field string `json:"field"`
Type string `json:"type"`
Category string `json:"category"`
Reason string `json:"reason"`
}
DefaultApplied describes a single defaulted operator slot. Returned by ResolveDefaults so callers (predict, service, MCP transcripts) can echo exactly which slots had their Type inferred from the schema. The shape is JSON-serializable and lands on PredictResult.DefaultsApplied.
Path uses JSON-style segments e.g. ["Aggregations", "0", "Type"] so the caller can address the slot in the original request. Field names the schema column that drove inference; Type is the operator string that was filled in; Category is "aggregation" or "grouper"; Reason carries a short human-readable rule trace, e.g. "f64 → AGG_SUM (rule: numeric default)".
func ResolveDefaults ¶ added in v0.5.0
func ResolveDefaults(req *types.Request, schema *encoding.Schema) []DefaultApplied
ResolveDefaults inspects every aggregation and grouper slot in req, fills in the Type for slots that name a field but omit the Type, and returns one DefaultApplied entry per change made. Mutates req in place.
Rules:
- Defaults never override an explicit Type.
- Defaults never cross categories (a missing aggregator does not insert a grouper, and vice versa).
- Tests (req.Tests, req.PostTests) are not defaulted; hypothesis tests are intent-bearing.
- Field types absent from defaultRules (or with no rule for the slot's category) are left untouched and contribute no entry.
Returns a nil slice when nothing changed.
type DictionaryInfo ¶
type DictionaryInfo struct {
TotalEntries int `json:"total_entries"`
Truncated bool `json:"truncated"`
Values []string `json:"values"`
}
DictionaryInfo describes the categorical dictionary for a field.
type DistributionMeta ¶ added in v0.5.0
type DistributionMeta struct {
// Name is the distribution kind identifier (e.g. "lognormal").
Name string `json:"name"`
// Description is a one-sentence prose summary.
Description string `json:"description"`
// AppliesTo lists the FieldSpec types the distribution can drive.
// Values are coarse families: "numeric", "categorical", "date",
// "bool", "any".
AppliesTo []string `json:"applies_to"`
// Params lists the distribution-specific parameters.
Params []Param `json:"params"`
}
DistributionMeta describes a synth distribution entry. One entry per synth.AllDistributions() value.
type Envelope ¶
type Envelope struct {
FormatVersion string `json:"format_version"`
Data any `json:"data"`
Request any `json:"request,omitempty"`
Errors []*EnvelopeEntry `json:"errors"`
Warnings []*EnvelopeEntry `json:"warnings"`
}
Envelope is the standard JSON output wrapper for all descriptor operations. All --json output follows this shape.
Request, when non-nil, carries the *normalized* request the operation executed — smart defaults resolved, label bindings expanded, per-stage requests captured for chains. Opt-in via pulse.Options.EchoRequest / CLI --echo-request; absent (and omitted from JSON) by default so the envelope shape is unchanged for callers that do not need it. The shape varies by operation: *types.Request for process/predict, *types.ComposedRequest for compose, *types.ChainRequest for process-chain, *types.FacetRequest for facet, *types.SampleRequest for sample.
func Inspect ¶
func Inspect(fileData io.ReadSeeker, opts *InspectOptions) *Envelope
Inspect reads a .pulse file header and schema, returning structured field information. It never reads record data. This entry point handles single-file cohorts only; archive-backed cohorts route through InspectFromBytes which can magic-detect the zip container.
func InspectFromBytes ¶
func InspectFromBytes(data []byte, opts *InspectOptions) *Envelope
InspectFromBytes inspects either a single-file .pulse cohort or a Pulse shard archive. Detection is by the first four bytes: zip magic PK\x03\x04 → archive path; PULSE magic → single-file path. Other prefixes return the standard ENCODING_INVALID envelope from the single-file path.
For archive-backed cohorts, the canonical schema and dictionaries come from the reserved _schema.pulse entry; Shards enumerates every non-reserved entry in central-directory order with per-shard RecordCount populated by peeking each shard's header; the envelope-level RecordCount is the cumulative sum across shards.
func NewEnvelope ¶
NewEnvelope creates an envelope with the given data and no errors/warnings.
func NewEnvelopeWithRequest ¶ added in v0.13.0
NewEnvelopeWithRequest creates an envelope with both data and the normalized request that produced it. Use when EchoRequest is enabled at the operation boundary. Passing a nil request degrades to NewEnvelope semantics (the field is omitted from JSON via omitempty).
func Predict ¶
func Predict(fileData io.ReadSeeker, req *types.Request, opts *PredictOptions) *Envelope
Predict validates a request against a .pulse file without executing it. It reads only the header and schema, never record data. The returned Envelope contains the PredictResult in Data and any errors/warnings encountered.
func PredictFromBytes ¶
func PredictFromBytes(data []byte, req *types.Request, opts *PredictOptions) *Envelope
PredictFromBytes routes data to either the single-file predict path or the archive predict path based on the first four bytes. Archive detection is by the zip magic PK\x03\x04; single-file (or any other prefix) falls through to Predict, which surfaces the standard ENCODING_INVALID envelope on malformed input.
For archive-backed cohorts: the canonical schema is read from the reserved _schema.pulse entry and used for every validation step (field existence, type compatibility, streamability). The cumulative record total and per-shard list are added to the PredictResult; the request is validated against the canonical schema and inherits the same streamability gates that a single-file cohort with the same schema would produce.
func ValidateChain ¶ added in v0.10.0
func ValidateChain(fileData io.ReadSeeker, req *types.ChainRequest) *Envelope
ValidateChain validates a ChainRequest against a .pulse cohort header + schema. It never reads record data — the no-execute contract holds. Errors land in the envelope as SERVICE_VALIDATION or PULSE_CHAIN_NOT_MERGEABLE / PULSE_CHAIN_EMPTY; each stage's inferred output schema is propagated forward so the next stage's field references can be checked.
The validator does not import service/processing — predict's structural ban applies to the broader descriptor surface in spirit.
func ValidateChainFromBytes ¶ added in v0.10.0
func ValidateChainFromBytes(data []byte, req *types.ChainRequest) *Envelope
ValidateChainFromBytes is a convenience wrapper that creates a reader from bytes.
func ValidateCompose ¶ added in v0.19.0
func ValidateCompose(req *types.ComposedRequest) *Envelope
ValidateCompose runs the no-execute COMPOSE-host overlay walk over a *types.ComposedRequest. Returns an envelope whose Data carries a *ComposeValidationResult and whose Errors carry one entry per spec failure. The validator never reads record data; per-slot schema shape inference (MATRIX / SERIES / SCALAR) is derived from each slot's *types.Request alone via inferComposeSlotShape.
The Request slot in the envelope is left nil — Compose has no single normalized form to echo back (per-slot normalisation is handled by service.applyComposeLabelDefaults at execution time and is not the descriptor surface's responsibility).
func ValidateFacet ¶ added in v0.7.0
func ValidateFacet(fileData io.ReadSeeker, req *types.FacetRequest) *Envelope
ValidateFacet validates a FacetRequest against a .pulse cohort header + schema. It never reads record data — predict's "no execution" contract applies here too. Errors land in the envelope as SERVICE_VALIDATION; advisory issues (percentiles on a non-numeric field, a low DiscreteTopK, IncludeHistogram on a discrete-only request) become warnings.
Callers can use this to gate a UI before issuing the (potentially expensive) FacetSchema call.
func ValidateFacetFromBytes ¶ added in v0.7.0
func ValidateFacetFromBytes(data []byte, req *types.FacetRequest) *Envelope
ValidateFacetFromBytes is a convenience wrapper that creates a reader from bytes.
func ValidateFacetWithExtensions ¶ added in v0.10.1
func ValidateFacetWithExtensions(fileData io.ReadSeeker, req *types.FacetRequest, snap *ExtensionsSnapshot) *Envelope
ValidateFacetWithExtensions extends ValidateFacet with the embedder-registered extension snapshot so label-table references can be resolved. Pass nil to opt out (same behavior as ValidateFacet).
func ValidateJoin ¶ added in v0.10.0
func ValidateJoin(leftData, rightData io.ReadSeeker, req *types.Request) *Envelope
ValidateJoin validates a Request whose Joins slot carries one JoinSpec against the headers + schemas of the left and right cohorts. It never reads record data — descriptor's no-execute contract applies. Errors land as SERVICE_VALIDATION or PULSE_JOIN_*; emits JoinedFields when the join would succeed at runtime.
func ValidateJoinFromBytes ¶ added in v0.10.0
ValidateJoinFromBytes is a convenience wrapper around byte buffers.
func (*Envelope) AddWarning ¶
AddWarning appends a structured warning to the envelope.
func (*Envelope) MarshalJSON ¶
MarshalJSON produces deterministic JSON output.
func (*Envelope) WithRequest ¶ added in v0.13.0
WithRequest attaches a normalized request to an existing envelope. Returns the receiver for fluent use. Nil-safe.
type EnvelopeEntry ¶
type EnvelopeEntry struct {
Code string `json:"code"`
Message string `json:"message"`
Details map[string]any `json:"details,omitempty"`
}
EnvelopeEntry represents a single error or warning in the envelope.
func CategoricalAggregationIssues ¶ added in v0.8.4
func CategoricalAggregationIssues(req *types.Request, schema *encoding.Schema) []*EnvelopeEntry
CategoricalAggregationIssues returns one EnvelopeEntry per (Aggregation slot referencing a categorical field, aggregator from numericAggregations) pair found in req. Each entry carries the PULSE_AGG_NOT_MEANINGFUL_FOR_CATEGORICAL code, a human-readable message, and the {field, aggregation} details map. Strict-mode promotion is the caller's responsibility — the helper itself is non-judgmental and returns nil when the request, schema, or Aggregations slice is empty / nil.
Used by:
- validateRequestFields (predict path) — emits to env.Warnings / env.Errors based on PredictOptions.Strict.
- service.Process — wraps the first entry as a coded error when Service is configured strict; non-strict emission flows through the envelope at the CLI boundary.
type ExportCapability ¶ added in v0.19.0
type ExportCapability struct {
// Formats enumerates every format the export dispatcher supports,
// sorted alphabetically by Name for deterministic golden output.
// One entry per format. Empty slice never appears — Pulse always
// ships at least the canonical seven adapters.
Formats []ExportFormatCapability `json:"formats"`
}
ExportCapability is the cross-format export envelope. Carries the alphabetised per-format slice so LLM planners can pick a target format aware of its overlay-embedding shape without inspecting the io/ packages.
type ExportFormatCapability ¶ added in v0.19.0
type ExportFormatCapability struct {
// Name is the canonical format identifier matching the
// io/format package constants (csv / tsv / ndjson / jsonarray /
// parquet / arrow / excel). Stable across format-version 1.0;
// new formats land additively.
Name string `json:"name"`
// OverlaySupport names the per-format overlay-embedding strategy
// the export dispatcher follows when ExportJob.IncludeOverlays
// resolves to true. See ExportFormatCapability godoc for the
// canonical label vocabulary.
OverlaySupport string `json:"overlay_support"`
}
ExportFormatCapability describes the per-format export envelope surfaced on the manifest. One entry per format the engine's export dispatch supports, carrying the format identifier plus the per-format overlay-embedding strategy LLM planners need to decide whether a Response.Overlays slice will round-trip through an ExportJob to the chosen target.
OverlaySupport is the canonical embedding-shape label declared by research/export-embedding-shape.md:
- "sidecar" — the format carries a top-level LIST<STRUCT> column-family appended to the host stream (Arrow / Parquet). The host record stream is byte-identical to the overlay-free shape; readers that ignore the column family see the unchanged host.
- "sheets" — one workbook sheet per overlay layer named "__overlay_<layer_name>" (Excel). The host workbook sheet is unchanged from the overlay- free shape.
- "trailing_block" — a single trailing line `{"_overlays": [...]}` after the last host record (NDJSON). The host stream is byte-identical to the overlay- free shape until the trailer.
- "warn_and_skip" — the format cannot embed overlays at all (CSV / TSV). The dispatcher emits one PULSE_OVERLAY_EXPORT_CSV_UNSUPPORTED warning and writes the host body verbatim; no overlay output lands. Setting IncludeOverlays=false suppresses the warning while keeping the same body.
type ExprFunctionMeta ¶ added in v0.7.0
type ExprFunctionMeta struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Signature string `json:"signature,omitempty"`
Pure bool `json:"pure,omitempty"`
}
ExprFunctionMeta is the manifest projection of an embedder- registered expression function.
type ExtensionsManifest ¶ added in v0.7.0
type ExtensionsManifest struct {
Aggregators []OperatorMeta `json:"aggregators"`
Attributes []OperatorMeta `json:"attributes"`
Filterers []OperatorMeta `json:"filterers"`
Groupers []OperatorMeta `json:"groupers"`
Windows []OperatorMeta `json:"windows"`
Features []OperatorMeta `json:"features"`
Tests []OperatorMeta `json:"tests"`
SynthDistributions []OperatorMeta `json:"synth_distributions"`
ExprFunctions []ExprFunctionMeta `json:"expr_functions"`
LookupTables []LookupTableMeta `json:"lookup_tables"`
LabelTables []LabelTableMeta `json:"label_tables"`
}
ExtensionsManifest is the manifest-side projection of embedder- registered operators and expression-side state. Built-in operators continue to live in Manifest.Components; this block lists everything the host process adds on top, so a reviewer can see at a glance which subset is Pulse-shipped vs registered by the embedder.
All slices are sorted by Name for deterministic output. An empty Extensions block emits as nested empty slices (not null) for JSON-stability across releases.
type ExtensionsSnapshot ¶ added in v0.7.0
type ExtensionsSnapshot struct {
Aggregators []OperatorMeta
Attributes []OperatorMeta
Filterers []OperatorMeta
Groupers []OperatorMeta
Windows []OperatorMeta
Features []OperatorMeta
Tests []OperatorMeta
SynthDistributions []OperatorMeta
ExprFunctions []ExprFunctionMeta
LookupTables []LookupTableMeta
LabelTables []LabelTableMeta
// ComponentSchemas projects per-extension ComponentSchema
// declarations keyed by registered operator name. The map carries
// aggregator / grouper / filterer schemas declared via
// pulse.AggregatorRegistration.ComponentSchema (and the matching
// grouper / filterer variants) so manifest + predict surface
// extension operators on the same shape as built-ins. Missing
// entries are equivalent to floor-only registrations — the
// orchestrator emits the universal floor without operator-
// specific extras.
//
// Wired by buildExtensionsSnapshot in pulse.New; manifest assembly
// merges this map into ComponentsSchemas alongside the built-in
// capability tables, and predict's per-slot ComponentSchema
// lookups fall through to this map when the built-in capability
// table does not carry an entry for the slot's operator name.
ComponentSchemas map[string]ComponentSchema
// OverlayKinds is the forward-compat reservation slot for
// embedder-registered overlay catalog entries. The runtime
// pulse.Options.Extensions.OverlayKinds registration surface
// (referenced in types/overlay.go) is not yet implemented; this
// snapshot slot exists so the MCP per-facade schema binders
// (internal/mcp/schema_bind.go) can merge custom kind names
// into the per-facade overlay_kind enum the moment the
// registration surface lands — no churn on the binder when the
// runtime catches up. Today the slot is populated by tests
// only; production code paths leave it nil and the per-facade
// enums fall back to the built-in catalog drawn from
// types.AllOverlayKinds() via descriptor.OverlayCapabilities().
// Each entry carries Name (SCREAMING_SNAKE OVERLAY_* constant)
// alongside the standard OperatorMeta knobs so the per-facade
// classifier can route an extension kind onto the right tool's
// enum once the registration surface declares which facade(s)
// the custom kind targets. v1 lacks the per-kind facade tag —
// the binder treats every snapshot entry as Request-facade
// only (the lowest-risk fallback) until the registration
// surface lands the tag.
OverlayKinds []OperatorMeta
}
ExtensionsSnapshot is the immutable read-only view that the service hands to descriptor.BuildManifestWithExtensions and predict. It carries everything the manifest + predict surfaces need without importing the live processing registry — keeping TestPredictNoExecutionImports satisfied.
type FacetCapability ¶ added in v0.7.0
type FacetCapability struct {
// Name is the canonical entry identifier ("facet_schema").
Name string `json:"name"`
// SupportsDiscrete reports whether per-value count summaries are
// available for categorical / boolean / geo fields.
SupportsDiscrete bool `json:"supports_discrete"`
// SupportsNumeric reports whether streaming statistics
// (count, sum, min, max, mean, stddev) are produced for numeric
// fields.
SupportsNumeric bool `json:"supports_numeric"`
// SupportsPercentiles reports whether NumericPercentiles are
// computed via a buffered second-stage sort.
SupportsPercentiles bool `json:"supports_percentiles"`
// SupportsHistogram reports whether IncludeHistogram +
// HistogramRange + HistogramBins are honoured.
SupportsHistogram bool `json:"supports_histogram"`
// SupportsAdditive reports whether AdditiveFields contribution
// counts are computed.
SupportsAdditive bool `json:"supports_additive"`
// SupportsOverlays reports whether FacetRequest.Overlays is
// honoured. When true, the SupportedOverlayKinds slice enumerates
// the FACET-host overlay catalog kinds the endpoint dispatches.
// Four FACET-host kinds wire into FacetSchema's buffered exit:
// OVERLAY_INDEX_VS_POP / OVERLAY_ZSCORE_VS_POP / OVERLAY_CHISQ_VS_POP
// / OVERLAY_KS_VS_POP.
SupportsOverlays bool `json:"supports_overlays,omitempty"`
// SupportedOverlayKinds enumerates the FACET-host overlay kinds the
// endpoint dispatches. Populated only when SupportsOverlays is true;
// each entry is a canonical OverlayKind constant string. Stable
// alphabetical order so LLM clients see a deterministic enum.
SupportedOverlayKinds []string `json:"supported_overlay_kinds,omitempty"`
// StreamableConditions lists the human-readable rules that govern
// which requests run in a single pass vs. force the buffered
// secondary sort. The order is stable and intended for LLM-side
// reasoning, not strict parsing.
StreamableConditions []string `json:"streamable_conditions"`
}
FacetCapability describes the rich-facet endpoint surfaced by pulse.FacetSchema / pulse_facet_schema. The manifest carries one FacetCapability entry under Manifest.Facet so LLM clients can detect the endpoint's feature set without inspecting the source.
type FacetValidationResult ¶ added in v0.7.0
type FacetValidationResult struct {
// Valid mirrors envelope.Errors emptiness.
Valid bool `json:"valid"`
// Request echoes the input request unchanged.
Request *types.FacetRequest `json:"request"`
// SchemaInfo summarises the cohort schema used for validation.
SchemaInfo *PredictSchemaInfo `json:"schema_info,omitempty"`
// OverlaysApplied lists every FACET-host overlay spec the validator
// accepted, in req.Overlays order. Each entry echoes the catalog
// identity (Name, Kind, Scope) plus the streamability flag harvested
// from types.OverlayStreamable(kind) so the caller can reason about
// the buffered/streaming routing decision without re-parsing the
// spec. Empty when req.Overlays is empty; never nil in JSON output.
//
// Per kind-catalog-v1 PRD §I-FR-I3 the FACET-host predict surface
// mirrors the Request-host predict surface so LLM callers see the
// same per-spec descriptor shape regardless of host. The four
// FACET-host kinds (OVERLAY_INDEX_VS_POP / OVERLAY_ZSCORE_VS_POP /
// OVERLAY_CHISQ_VS_POP / OVERLAY_KS_VS_POP) route onto this
// surface; sibling to PredictResult.OverlaysApplied.
OverlaysApplied []OverlayAppliedDescriptor `json:"overlays_applied"`
// OverlayCost maps each FACET-host overlay-spec Name to a coarse
// cost score the routing layer can consult before execution. Sibling
// to PredictResult.OverlayCost — streamable kinds carry
// overlayCostStreamable (~5% extra work) and buffered kinds carry
// overlayCostBuffered (~one extra payload traversal). Per kind-
// catalog-v1 PRD §I-FR-I3 the value is intentionally coarse — callers
// budgeting cost across slots sum the map values. Empty when
// req.Overlays is empty; never nil in JSON output.
OverlayCost map[string]float64 `json:"overlay_cost"`
}
FacetValidationResult is the structured ValidateFacet output. Mirrors the predict surface: the resolved request is echoed back verbatim (callers may want to forward it to FacetSchema after inspection) and Warnings carry advisory issues that do not block execution.
type FiltererPredict ¶ added in v0.20.0
type FiltererPredict struct {
// Type echoes the filterer type from the request slot, in the
// canonical SCREAMING_SNAKE form (e.g. "FILTER_RANGE").
Type types.FiltererType `json:"type"`
// Field echoes the field referenced by the request slot. Empty
// when the slot was authored without a field (FILTER_EXPRESSION
// reads any record field through its expression body).
Field string `json:"field,omitempty"`
// ComponentSchema is the static schema for the filterer the engine
// will use at runtime, projected from the capabilities table
// (capabilities_filterers.go). Empty Keys + empty Mergeability
// means the slot references an extension filterer that has not
// (yet) declared a ComponentSchema — the universal-floor populator
// still attaches {"n_in", "n_out", "n_null_input"} at runtime.
ComponentSchema ComponentSchema `json:"component_schema"`
// BufferedComponents is true iff ComponentSchema.Mergeability is
// descriptor.None. The flag advertises that the slot's Components
// block will arrive only on the terminal buffered flush in a
// streaming run; it does NOT flip the overall Streamable axis. In
// v1 every built-in filterer is Mergeable so this flag is always
// false; the surface exists for future per-filter specifics that
// may downgrade individual entries.
BufferedComponents bool `json:"buffered_components,omitempty"`
}
FiltererPredict is the per-slot predict surface for one entry of req.Filterers. Carries the operator identity echoed back to the caller plus the static ComponentSchema declared in the capabilities table — so callers can budget the runtime Components.Filterers block shape before the engine runs and so streaming consumers can branch on the BufferedComponents hint without consulting the manifest.
Predict stays no-execute: every field on this struct is sourced from the descriptor capability projection — never from a constructed filterer. Custom filterers registered via pulse.Options.Extensions that do not (yet) supply a ComponentSchema produce an empty ComponentSchema and BufferedComponents=false; the universal-floor populator at orchestrator emission time still attaches {"n_in", "n_out", "n_null_input"} at runtime.
Components are uniform across every built-in filterer in v1: the universal floor {n_in, n_out, n_null_input} is the entire schema and Mergeability is Mergeable (counters fold trivially via integer addition). The struct mirrors AggregationPredict / GroupPredict so future per-filter specifics (e.g. n_below / n_above for FILTER_RANGE) can downgrade the slot to Partial / None without changing the predict shape.
type GroupPredict ¶ added in v0.20.0
type GroupPredict struct {
// Type echoes the grouper type from the request slot, in the
// canonical SCREAMING_SNAKE form (e.g. "GROUP_QUANTILE").
Type types.GroupType `json:"type"`
// Field echoes the field referenced by the request slot. Empty
// when the slot was authored without a field (defaults may have
// filled it; the resolved post-defaults clone is reflected here
// when applicable).
Field string `json:"field,omitempty"`
// ComponentSchema is the static schema for the grouper the engine
// will use at runtime, projected from the capabilities table
// (capabilities_groupers.go). Empty Keys + empty Mergeability
// means the slot references an extension grouper that has not
// (yet) declared a ComponentSchema — the universal-floor
// populator still attaches {"total_n", "n_null"} at runtime.
ComponentSchema ComponentSchema `json:"component_schema"`
// BufferedComponents is true iff ComponentSchema.Mergeability is
// descriptor.None. The flag advertises that the slot's Components
// block will arrive only on the terminal buffered flush in a
// streaming run; it does NOT flip the overall Streamable axis (the
// data slice still streams). GROUP_QUANTILE is the canonical
// None today.
BufferedComponents bool `json:"buffered_components,omitempty"`
}
GroupPredict is the per-slot predict surface for one entry of req.Groups. Carries the grouper identity echoed back to the caller plus the static ComponentSchema declared in the capabilities table — so callers can budget the runtime Components.Groupers block shape before the engine runs and so streaming consumers can branch on the BufferedComponents hint without consulting the manifest.
Predict stays no-execute: every field on this struct is sourced from the descriptor capability projection — never from a constructed grouper. Custom groupers registered via pulse.Options.Extensions that do not (yet) supply a ComponentSchema produce an empty ComponentSchema and BufferedComponents=false; the universal-floor populator at orchestrator emission time still attaches {"total_n", "n_null"} at runtime.
type InspectField ¶
type InspectField struct {
Name string `json:"name"`
Type string `json:"type"`
ByteOffset int `json:"byte_offset"`
BitPosition int `json:"bit_position"`
Description string `json:"description"`
DescriptionSource string `json:"description_source"`
Categorical bool `json:"categorical"`
Dictionary *DictionaryInfo `json:"dictionary,omitempty"`
// Precision is the decimal128 precision (1-38). Present only for
// decimal128 / nullable_decimal128 fields.
Precision *uint8 `json:"precision,omitempty"`
// Scale is the decimal128 scale (0-precision). Present only for
// decimal128 / nullable_decimal128 fields.
Scale *uint8 `json:"scale,omitempty"`
}
InspectField describes a single field in the inspect output.
type InspectOptions ¶
type InspectOptions struct {
// FullDict disables dictionary truncation when true.
FullDict bool
// DictionaryLimit overrides the default truncation limit.
// Zero means use DefaultDictionaryLimit.
DictionaryLimit int
}
InspectOptions controls inspect behavior.
type InspectResult ¶
type InspectResult struct {
FieldCount int `json:"field_count"`
Fields []*InspectField `json:"fields"`
Shards []ShardInfo `json:"shards"`
RecordCount int64 `json:"record_count"`
}
InspectResult holds the schema inspection output.
Shards is populated when the inspected file is a Pulse shard archive (first four bytes match the zip magic PK\x03\x04). Entries are listed in zip central-directory order, which equals shard insertion order. Single-file cohorts leave Shards as an empty slice (never nil) so the JSON envelope emits "shards": [] rather than null.
For archive-backed cohorts, FieldCount and Fields reflect the canonical schema carried in the reserved _schema.pulse entry, and RecordCount (when present in metadata) plus aggregate counting via per-shard headers populate the cumulative total — see ShardInfo.
type JoinCapability ¶ added in v0.10.0
type JoinCapability struct {
// Name is the canonical identifier ("hash_join").
Name string `json:"name"`
// MaxJoinsPerRequest is the upper bound on Request.Joins length.
// 1 today; lifts when multi-join chains land.
MaxJoinsPerRequest int `json:"max_joins_per_request"`
// Kinds lists supported JoinSpec.Kind values. "inner" today;
// "left", "outer", "anti" land once the null-bitmap correctness
// path is fully wired.
Kinds []string `json:"kinds"`
// SpillBytes reports the in-memory threshold beyond which the
// build side would spill. Zero today (no spill — the build side
// is always materialised in RAM); set when the spill path lands.
SpillBytes int64 `json:"spill_bytes"`
// SpillEnv names the environment variable that overrides the
// spill threshold. Set when the spill path lands.
SpillEnv string `json:"spill_env,omitempty"`
// Limitations lists the v1 envelope notes for LLM clients.
Limitations []string `json:"limitations"`
}
JoinCapability describes the pushdown hash-join endpoint surfaced by Request.Joins. The manifest carries one JoinCapability entry under Manifest.Join so LLM clients can detect the v1 envelope (inner-only, single-join) without inspecting the source.
type JoinValidationResult ¶ added in v0.10.0
type JoinValidationResult struct {
Valid bool `json:"valid"`
Request *types.Request `json:"request"`
LeftSchema *PredictSchemaInfo `json:"left_schema,omitempty"`
RightSchema *PredictSchemaInfo `json:"right_schema,omitempty"`
JoinedFields []string `json:"joined_fields,omitempty"`
}
JoinValidationResult is the structured output of ValidateJoin. Echoes the request unchanged and exposes the inferred output schema as a list of field names.
type LabelTableMeta ¶ added in v0.10.1
type LabelTableMeta struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
HasRowsData bool `json:"has_rows_data"`
}
LabelTableMeta is the manifest projection of a registered string- valued label table. HasRowsData distinguishes the static Rows-backed table from the function-driven Lookup table without exposing the embedder's data.
type LookupTableMeta ¶ added in v0.7.0
type LookupTableMeta struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
HasRowsData bool `json:"has_rows_data"`
}
LookupTableMeta is the manifest projection of a registered lookup table. HasRowsData distinguishes the static Rows-backed table from the function-driven Lookup table without exposing the embedder's data.
type MCPTool ¶ added in v0.5.0
type MCPTool struct {
// Name is the tool identifier (e.g. "pulse_predict").
Name string `json:"name"`
// Description mirrors the description string registered with the
// MCP server.
Description string `json:"description"`
}
MCPTool describes a single registered MCP tool. One entry per internal/mcp.RegisteredTools() value.
type Manifest ¶
type Manifest struct {
FormatVersion string `json:"format_version"`
Commands []Command `json:"commands"`
// Operations enumerates library-only entry points that do not back a
// CLI leaf (today: filter_to_file, watch, process_stream). Each entry
// carries the same CommandAnnotations as a CLI leaf so consumers can
// reason uniformly about caching / streaming. The slice is sorted by
// Name for determinism.
Operations []Command `json:"operations"`
Components Components `json:"components"`
Tests []TestMeta `json:"tests"`
PostTests []TestMeta `json:"post_tests"`
Regressions []RegressionMeta `json:"regressions"`
SynthDistributions []DistributionMeta `json:"synth_distributions"`
// ErrorCodesCount is the total number of registered error codes.
ErrorCodesCount int `json:"error_codes_count"`
// ErrorDomains is the alphabetized list of distinct domain prefixes
// (e.g. "CLI", "DATA", "ENCODING", "PROCESSING", "PULSE",
// "SERVICE"). One entry per domain, six entries in v1.
ErrorDomains []string `json:"error_domains"`
// ErrorCodes is the alphabetized list of code identifiers.
// Per-code Message + Fixup prose lives behind the
// `pulse_errors_lookup` MCP tool / `pulse errors lookup CODE` CLI
// leaf — depth-on-demand, not common-path.
ErrorCodes []string `json:"error_codes"`
MCPTools []MCPTool `json:"mcp_tools"`
CohortTypes []CohortFieldType `json:"cohort_types"`
Skills []SkillMeta `json:"skills"`
ExamplesCount int `json:"examples_count"`
ExampleCategories []string `json:"example_categories"`
ExampleTags []string `json:"example_tags"`
// Extensions enumerates embedder-registered operators + expression
// state. Built-in operators continue to live in Components; this
// block is the additive layer registered via
// pulse.Options.Extensions. Empty slices on every field for a
// host with no extensions.
Extensions ExtensionsManifest `json:"extensions"`
// Facet is the rich-facet endpoint capability descriptor. One
// entry today (facet_schema); future variants land under a slice
// when added.
Facet FacetCapability `json:"facet"`
// ProcessChain is the source-rooted linear chain endpoint
// capability descriptor (one entry today: process_chain).
// Carries the mergeable-operator allowlist and rejection rules
// so LLM clients can route between chain and per-stage fallback.
ProcessChain ProcessChainCapability `json:"process_chain"`
// Join is the pushdown hash-join capability descriptor (one
// entry today: hash_join). Carries the kind allowlist, spill
// envelope, and v1 limitations.
Join JoinCapability `json:"join"`
// Crosstab is the cross-tabulation endpoint capability
// descriptor (Request.Crosstab). Carries the normalize / shape
// allowlists plus the per-aggregator margin-reducibility
// classification so LLM clients can decide which cell aggregator
// will recompute its margin and which is summable.
Crosstab CrosstabCapability `json:"crosstab"`
// Export is the cross-format export envelope. Carries one
// ExportFormatCapability entry per format the export dispatcher
// supports, declaring the per-format overlay-embedding shape
// (sidecar / sheets / trailing_block / warn_and_skip) so LLM
// planners can route Response.Overlays through ExportJob without
// inspecting the io/ packages.
Export ExportCapability `json:"export"`
// Overlays enumerates the registered overlay catalog — one
// OverlayCapability per types.AllOverlayKinds() entry. Each entry
// declares the supported OverlayShape × OverlayScope × OverlayRef
// kinds the kind accepts plus its Buffered flag (the inverse of
// types.OverlayStreamable). LLM clients route between overlay
// catalog lookup and per-spec validation without crawling the
// type system. Sorted alphabetically by Kind via
// OverlayCapabilities() so the golden manifest stays stable.
Overlays []OverlayCapability `json:"overlays"`
// ComponentsSchemas projects the per-operator components contract
// across categories as a name-keyed map. Embedders rely on this
// block to plan ResponseComponents.Components[] consumption without
// iterating Components.* per category.
ComponentsSchemas ComponentsSchemasBlock `json:"components_schemas"`
}
Manifest is the root self-description of the Pulse system. One bootstrap call returns every fact an LLM needs to author a valid Pulse request: CLI command list, per-operator capabilities, per-test metadata (tier-1 and tier-2 as peer slices), synth distribution catalog, MCP tool list, cohort field-type catalog with operator cross-references, and embedded skill index. Error coverage is name-only — fetch per-code prose via the `pulse_errors_lookup` MCP tool or `pulse errors lookup CODE` CLI leaf on demand to keep the bootstrap payload lean.
The payload is deterministic and free of cohort data. Clients cache it for a session.
func BuildManifest ¶
func BuildManifest() *Manifest
BuildManifest constructs a deterministic Manifest from the current registries and capability tables. The result is safe to cache and share across goroutines; callers do not mutate the returned slices.
func BuildManifestWithExtensions ¶ added in v0.7.0
func BuildManifestWithExtensions(snap *ExtensionsSnapshot) *Manifest
BuildManifestWithExtensions constructs a Manifest that includes the embedder-registered extension surface. A nil snapshot is equivalent to BuildManifest — the Extensions block becomes the empty manifest (every category is `[]`, not `null`).
descriptor stays free of service / processing imports; the snapshot is the only way the live ExtensionRegistry reaches this layer.
func SlimManifest ¶ added in v0.5.0
SlimManifest returns a copy of m with every prose Description field blanked. Structural metadata (names, params, types, tiers, accept-type lists, streamability flags, cross-references, fixups) is preserved.
The slim variant is intended for size-sensitive MCP/CLI clients that would rather pay a few extra discovery round-trips than transit the full ~70 kB descriptive payload at session start. The default manifest surface remains the full one; --slim is opt-in.
SlimManifest never mutates the input; it shallow-copies the top-level struct, then per-slice shallow-copies and zeroes the Description fields.
type Operator ¶ added in v0.5.0
type Operator struct {
// Name is the operator identifier (e.g. "AGG_PERCENTILE").
Name string `json:"name"`
// Category is the family this operator belongs to. One of:
// "aggregator", "attribute", "filterer", "grouper", "window",
// "feature".
Category string `json:"category"`
// Description is a one-sentence prose summary for LLM-side selection.
Description string `json:"description"`
// Params lists every parameter the operator reads from its Params
// blob in a request. Required and optional are both listed.
Params []Param `json:"params"`
// AcceptsTypes lists the cohort field types this operator can be
// applied to. Values are field type name strings (e.g. "f64",
// "categorical_u16", "date"). Empty means "no field input".
AcceptsTypes []string `json:"accepts_types"`
// EmitsType is the field type produced for single-output operators.
// Empty when the operator's emit type is conditional on input or
// when it does not emit a typed column (e.g. an aggregator emits a
// scalar).
EmitsType string `json:"emits_type,omitempty"`
// EmitsTypeNote provides context when EmitsType is empty or
// conditional (e.g. "matches input field type", "scalar float64").
EmitsTypeNote string `json:"emits_type_note,omitempty"`
// Streamable mirrors types.X.Streamable() for the operator's type.
// Source of truth for the runtime gate.
Streamable bool `json:"streamable"`
// StreamableHint suggests the closest streaming-capable alternative
// when Streamable is false (e.g. AGG_MEDIAN -> "Use AGG_AVERAGE for
// streaming, or accept the buffered path."). Empty when Streamable
// is true or no near-equivalent exists.
StreamableHint string `json:"streamable_hint,omitempty"`
// ComponentSchema declares the per-operator components contract
// surfaced through the runtime ResponseComponents shell and the
// MetaAggregator / MetaGrouper / MetaFilterer sibling interfaces.
// The universal floor of {"n", "n_null"} is declared on every
// aggregator entry so manifest consumers see a self-contained
// schema without consulting orchestrator code; the operator-
// specific keys are appended in emission order. Mergeability
// classifies how the components map folds across streaming chunks.
ComponentSchema ComponentSchema `json:"component_schema"`
}
Operator describes a single registered processing component (aggregator, attribute, filterer, grouper, window operator, or feature operator). The manifest exposes one Operator entry per registered component. LLM clients use this metadata at session start to author valid requests without further discovery round-trips.
type OperatorMeta ¶ added in v0.7.0
type OperatorMeta struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
Description string `json:"description,omitempty"`
Streamable bool `json:"streamable"`
Accepts []string `json:"accepts,omitempty"`
Emits string `json:"emits,omitempty"`
Mode string `json:"mode,omitempty"`
Tier string `json:"tier,omitempty"`
Params []OperatorParamMeta `json:"params,omitempty"`
}
OperatorMeta is the manifest projection for an embedder-registered operator. The shape mirrors descriptor.Operator but adds Namespace (parsed from the registered name) and Mode (attribute-only) so reviewers can group operators by source without re-parsing.
type OperatorParamMeta ¶ added in v0.7.0
type OperatorParamMeta struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
JSONType string `json:"json_type"`
Required bool `json:"required,omitempty"`
Default any `json:"default,omitempty"`
}
OperatorParamMeta is the manifest-friendly mirror of pulse.ParamMeta.
type OverlayAppliedDescriptor ¶ added in v0.19.0
type OverlayAppliedDescriptor struct {
// Name echoes the renderer-facing label — either the spec's own
// Name or a synthesised default (Kind + Scope + Ref) when empty.
Name string `json:"name"`
// Kind is the on-wire SCREAMING_SNAKE OverlayKind value.
Kind types.OverlayKind `json:"kind"`
// Scope echoes the spec's scope.
Scope types.OverlayScope `json:"scope"`
// Shape echoes the resolved OverlayShape the layer will carry —
// derived from (Kind, Scope) via the capability catalog
// (descriptor.overlayCapabilityFor). For single-shape kinds the
// shape is unambiguous; for dual-shape kinds (OVERLAY_INDEX_VS_REF
// / OVERLAY_DELTA_VS_REF / OVERLAY_PANEL_INDEX_VS_REF) the resolver
// picks the shape matching the spec's scope (CELL ⇒ matrix, GROUP
// ⇒ series); for whole-chain kinds (OVERLAY_INDEX_VS_STAGE /
// OVERLAY_DELTA_VS_STAGE) the catalog declares every shape the
// kind may emit (scalar / series / matrix) and the descriptor
// leaves Shape empty — the realised shape depends on the target
// stage's host shape, which Predict cannot determine without
// inspecting the chain. Empty when the kind is unknown (the
// validator surfaces PULSE_OVERLAY_KIND_UNKNOWN for the same
// condition) or when no single shape can be resolved from the
// spec at predict time.
Shape types.OverlayShape `json:"shape,omitempty"`
// Ref echoes the resolved OverlayRef discriminated union variant
// as a renderer-friendly string. Format: "<family>:<discriminator>"
// when the family carries an on-wire discriminator, "<family>" for
// marker-only families, and "" for implicit-margin kinds (every
// CHISQ_* / FISHER_EXACT_CELL / INDEX_VS_TOTAL / FORMULA) and the
// COMPOSE-only catalog (Reference / Targets resolve via slot
// labels, not via the OverlayRef discriminated union). Concrete
// shapes:
//
// - "margin:row" / "margin:column" / "margin:grand" — Ref.Margin
// - "sibling:<field>=<value>" — Ref.Sibling
// - "baseline_index:<position>" — Ref.BaselineIndex
// - "prior:<lag>" — Ref.Prior
// - "rolling_mean" — Ref.RollingMean (marker)
// - "yoy" — Ref.YoY (marker)
// - "population:<cohort>" — Ref.Population
// - "stage:index:<index>" / "stage:name:<name>" — Ref.Stage
// - "slot:<label>" — Ref.Slot
Ref string `json:"ref,omitempty"`
// Streamable mirrors types.OverlayStreamable(kind). False for
// the buffered crosstab catalog; streamable kinds surface the
// per-kind decision uniformly.
Streamable bool `json:"streamable"`
}
OverlayAppliedDescriptor describes one accepted overlay spec the predict path observed. Mirrors DefaultApplied / Suggestion in shape: JSON-serialisable, omitempty-free for slice-friendly emit, populated from the catalog entry rather than the raw spec so the caller cannot confuse the surface with the raw OverlaySpec it submitted.
Streamable echoes types.OverlayStreamable(kind). Note that a kind can be streamable in isolation but force buffered when its host (today: crosstab) is itself buffered — predict reports the kind's own streamability, not the composed host-overlay decision. Callers that need the composed decision should consult PredictResult.Streamable + StreamableReasons.
type OverlayCapability ¶ added in v0.19.0
type OverlayCapability struct {
// Kind is the on-wire SCREAMING_SNAKE OverlayKind value.
Kind types.OverlayKind `json:"kind"`
// Shapes lists every OverlayShape this kind may emit. Sorted
// alphabetically for golden stability.
Shapes []types.OverlayShape `json:"shapes"`
// Scopes lists every OverlayScope this kind supports. Sorted
// alphabetically for golden stability.
Scopes []types.OverlayScope `json:"scopes"`
// RefKinds lists the OverlayRef union pointer-field names this
// kind consumes (e.g. "Margin"). Sorted alphabetically for golden
// stability. These are Go field-name strings — the on-wire
// discriminator (MarginAxis, sibling field, etc.) lives one level
// deeper inside the chosen pointer.
RefKinds []string `json:"ref_kinds"`
// Buffered reports whether the orchestrator must materialise
// records before evaluating this kind. Derived from
// types.OverlayStreamable(kind) — Buffered is !streamable.
Buffered bool `json:"buffered"`
// Fields lists the OverlaySpec slot names beyond Kind / Scope /
// Ref / Params that the kind's runtime honours. Sorted
// alphabetically for golden stability. "level" and "within" are
// honoured by the share / index / delta / zscore family (prefix-axis
// denominator dispatch) while the χ² / Fisher inferential family
// leaves the list empty (Level / Within must stay zero; non-zero
// values fire PULSE_OVERLAY_LEVEL_OUT_OF_RANGE). SHARE_OF_TOTAL
// declares the fields for renderer-facing parity with the rest of
// the share family even though the grand-axis denominator makes the
// slots inert at runtime — the predict gate still accepts in-range
// values without behavior change.
Fields []string `json:"fields,omitempty"`
// Description is a short human-readable summary of what the kind
// computes. Mirrors the prose in the matching skill.
Description string `json:"description"`
}
OverlayCapability is the per-kind manifest entry describing one registered overlay catalog entry. The manifest carries one OverlayCapability per types.AllOverlayKinds() entry under Manifest.Overlays.
Field shape mirrors the kind-catalog-v1 PRD §I-FR-I2 contract: kind × supported shapes × supported scopes × valid ref kinds × buffered flag × description.
func OverlayCapabilities ¶ added in v0.19.0
func OverlayCapabilities() []OverlayCapability
OverlayCapabilities returns the canonical per-kind capability list for the manifest. Deterministic ordering: alphabetised by Kind via types.AllOverlayKinds() so the golden manifest stays stable as new kinds land. Buffered flags consult types.OverlayStreamable(kind) so the static table in types/overlay_streamability.go remains the single source of truth for streaming behaviour — a kind that flips to streamable automatically flips Buffered to false here.
OVERLAY_INDEX_VS_MARGIN is the seed entry for the catalog:
- Shapes: [matrix] (CELL-scoped overlay layered onto a crosstab)
- Scopes: [cell] (CELL-only today; ROW/COLUMN/TOTAL land alongside the matching payload shapes)
- RefKinds: [Margin] (denominator is an axis-margin slot)
- Buffered: true (margins recompute from raw rows; the host crosstab path is always buffered)
Later kinds drop in by extending the switch below. TestStreamability_OverlaysKnown (types/overlay_streamability.go) and TestManifestOperatorsComplete-style follow-ups (descriptor) enforce that every catalog entry has a row in this surface.
type Param ¶ added in v0.5.0
type Param struct {
// Name is the JSON key inside the operator's Params blob.
Name string `json:"name"`
// Type is the parameter's value type. One of:
// "float", "int", "string", "bool", "field", "enum", "list", "object".
// "field" means the value names a cohort field; "enum" means the
// value is one of EnumValues; "list" means a JSON array.
Type string `json:"type"`
// Required is true when the parameter must be supplied. Optional
// parameters with a default carry Required=false and Default set.
Required bool `json:"required"`
// Default is the operator's default value when Required is false.
// Omitted from JSON when nil.
Default any `json:"default,omitempty"`
// Description is a one-sentence prose explanation.
Description string `json:"description"`
// EnumValues lists the allowed values when Type=="enum".
EnumValues []string `json:"enum_values,omitempty"`
// FieldFilter constrains acceptable field types when Type=="field".
// One of: "numeric", "categorical", "date", "any".
FieldFilter string `json:"field_filter,omitempty"`
}
Param describes a single parameter accepted by an operator, test, or synth distribution.
type PredictOptions ¶
type PredictOptions struct {
// Strict upgrades warnings to errors.
Strict bool
// Extensions is the read-only snapshot of embedder-registered
// operators + expression-side state. Nil takes the built-in-only
// path. The snapshot adds every custom operator name to the
// validator's known-types set so predict does not flag
// embedder-registered ops as unknown, and feeds streamability
// overrides into computeStreamable.
Extensions *ExtensionsSnapshot
// EchoRequest causes Predict to populate envelope.Request with the
// normalized (post-defaults) request. PredictResult.Request stays
// the raw input request — the envelope field is the new uniform
// surface across all envelope-producing endpoints. Off by default.
EchoRequest bool
}
PredictOptions controls predict behavior.
type PredictResult ¶
type PredictResult struct {
Valid bool `json:"valid"`
Request *types.Request `json:"request"`
SchemaInfo *PredictSchemaInfo `json:"schema_info,omitempty"`
// RecordCount reports the cumulative record total across all shards
// when the cohort is a shard archive, and zero for single-file
// cohorts (predict reads only headers/schemas, so single-file
// counts are not computed by this no-execute path). The count is
// derived by peeking each shard's own header.
RecordCount int64 `json:"record_count"`
// Shards mirrors InspectResult.Shards for archive-backed cohorts.
// Empty (but non-nil) for single-file cohorts. Listed in zip
// central-directory order.
Shards []ShardInfo `json:"shards"`
// Streamable reports whether ProcessStream / process --stream can
// emit rows without buffering the entire result. False whenever the
// request uses groups, attributes, windows, decimal fields, or any
// non-streamable operator. Computed via per-type Streamable() methods
// plus schema-aware checks.
//
// Streamability is a property of the request and the canonical
// schema, not of the cohort shape — archive-backed cohorts inherit
// the same streamability as a single-file cohort with the same
// schema.
Streamable bool `json:"streamable"`
// StreamableReasons lists the gates that forced Streamable=false. Empty
// when Streamable=true. Useful for users debugging why their request
// is buffering.
StreamableReasons []string `json:"streamable_reasons,omitempty"`
// Suggestions enumerates structured next-actions the caller can apply
// to repair (or improve) the request. Suggestions fire on validation
// issues — field-name typos, operator/type mismatches, date misuse,
// missing required params — and on non-streamable but otherwise valid
// requests (streamable-substitute hints). May be empty; never nil in
// JSON output.
Suggestions []Suggestion `json:"suggestions"`
// DefaultsApplied lists every operator slot whose Type was inferred
// from the named field's schema type. Predict computes this on a
// clone of the request, so the echoed Request reflects exactly what
// the engine would run; the DefaultsApplied list shows what would
// have been filled in. Empty when no defaults fire; never nil in
// JSON output.
DefaultsApplied []DefaultApplied `json:"defaults_applied"`
// Aggregations mirrors req.Aggregations in order, attaching the
// per-slot ComponentSchema the engine will use at runtime (the
// operator's `descriptor.ComponentSchema` projected from the
// capabilities table — and via the manifest snapshot for
// embedder-registered ops) plus the BufferedComponents flag.
//
// BufferedComponents is true iff the operator's
// ComponentSchema.Mergeability is None — meaning the components
// map cannot be reconstructed from per-chunk partials and the
// orchestrator emits the Components block only on the terminal
// buffered flush. The flag is a hint to streaming consumers; it
// does NOT flip the overall Streamable axis (the data slice still
// streams). Today the canonical None-mergeability operators are
// AGG_MEDIAN and AGG_PERCENTILE.
//
// Empty when req.Aggregations is empty; never nil in JSON output.
Aggregations []AggregationPredict `json:"aggregations"`
// Groups mirrors req.Groups in order, attaching the per-slot
// ComponentSchema the engine will use at runtime (the operator's
// `descriptor.ComponentSchema` projected from the capabilities
// table — same projection ComponentsSchemas.Groupers uses) plus
// the BufferedComponents flag.
//
// BufferedComponents is true iff the operator's
// ComponentSchema.Mergeability is None — meaning the components
// map cannot be reconstructed from per-chunk partials and the
// orchestrator emits the Components block only on the terminal
// buffered flush. The flag is a hint to streaming consumers; it
// does NOT flip the overall Streamable axis. Of the seven
// registered groupers today, GROUP_QUANTILE is the canonical
// None — quantile cutoffs need the sorted full input.
//
// Empty when req.Groups is empty; never nil in JSON output.
Groups []GroupPredict `json:"groups"`
// Filterers mirrors req.Filterers in order, attaching the per-slot
// ComponentSchema the engine will use at runtime (the operator's
// `descriptor.ComponentSchema` projected from the capabilities
// table — same projection ComponentsSchemas.Filterers uses) plus
// the BufferedComponents flag.
//
// Filterer components in v1 are uniform across every registered
// filterer — the orchestrator emits the universal floor
// {n_in, n_out, n_null_input} from the filter pass's per-record
// counters. Mergeability is Mergeable for every entry today
// (counters fold trivially via integer addition), so
// BufferedComponents is always false. The flag is still surfaced
// on the predict struct for symmetry with AggregationPredict and
// GroupPredict so future per-filter specifics (e.g. n_below /
// n_above for FILTER_RANGE) can downgrade the slot without
// changing the predict shape.
//
// Empty when req.Filterers is empty; never nil in JSON output.
Filterers []FiltererPredict `json:"filterers"`
// OverlaysApplied lists every overlay spec the engine accepted, in
// req.Overlays order. Each entry echoes the catalog identity (Name,
// Kind, Scope) plus the streamability flag harvested from
// types.OverlayStreamable(kind) so the caller can reason about the
// buffered/streaming routing decision without re-parsing the spec.
// Empty when req.Overlays is empty; never nil in JSON output.
//
// Per kind-catalog-v1 PRD §I-FR-I3 the predict surface is
// `OverlaysApplied + OverlaysSchemaDivergence + OverlayCost`.
// Predict emits one descriptor per spec in matching order across
// the full overlay catalog; new kinds appended to
// types.AllOverlayKinds() inherit the same per-spec emission
// without re-opening this slot. The streamability-derived
// dispatch routes streamable kinds to overlayCostStreamable and
// buffered kinds to overlayCostBuffered; every kind in
// types.AllOverlayKinds() carries a multiplier, and the
// multi-ref scaling rule lives on the COMPOSE-host equivalent at
// ComposeValidationResult.OverlayCost since Targets / MaxPanelTargets
// only exist on ComposeOverlaySpec.
OverlaysApplied []OverlayAppliedDescriptor `json:"overlays_applied"`
// OverlaysSchemaDivergence lists every (left, right) overlay-spec
// slot pair that produced incompatible result-schema shapes. Empty
// for the Request-only Predict surface — divergence detection lives
// on the Compose-host validator (ValidateCompose) — but the field is
// present so consumers can rely on the shape today. Per PRD §I-FR-I3
// the placeholder is honoured as "this slot is reserved and shipped
// empty".
OverlaysSchemaDivergence []SlotPair `json:"overlays_schema_divergence"`
// OverlayCost maps each overlay-spec Name to a coarse cost score
// the routing layer can consult before execution. The score is a
// rough record-count multiplier per overlay slot — streamable kinds
// fold inside the existing streaming pass (one extra accumulator per
// record) and carry overlayCostStreamable; buffered kinds force a
// post-host re-traversal of the materialised payload and carry
// overlayCostBuffered. Per kind-catalog-v1 PRD §I-FR-I3 the value is
// intentionally coarse — callers budgeting cost across slots sum the
// map values; renderers showing a "this overlay will buffer the
// host" warning branch on >= overlayCostBuffered. The map key is
// the overlay spec's Name when set, and the synthesised default
// (Kind + Scope + Ref) when empty — matching the Name field on
// OverlayAppliedDescriptor below. Empty when req.Overlays is empty;
// never nil in JSON output.
OverlayCost map[string]float64 `json:"overlay_cost"`
}
PredictResult holds the validated request and any diagnostics.
type PredictSchemaInfo ¶
type PredictSchemaInfo struct {
FieldCount int `json:"field_count"`
Fields []string `json:"fields"`
}
PredictSchemaInfo summarizes the schema used for prediction.
type ProcessChainCapability ¶ added in v0.10.0
type ProcessChainCapability struct {
// Name is the canonical entry identifier ("process_chain").
Name string `json:"name"`
// MaxStages is the upper bound on Stages length. Zero indicates
// no compile-time cap; runtime memory is bounded by the largest
// intermediate response.Data slice.
MaxStages int `json:"max_stages"`
// MergeableAggregators lists the aggregator names that pass the
// chain gate (mergeable + single-scalar emit). Alphabetically
// sorted, deterministic across calls.
MergeableAggregators []string `json:"mergeable_aggregators"`
// MergeableGroupers lists the grouper names that pass the chain
// gate.
MergeableGroupers []string `json:"mergeable_groupers"`
// RowLocalAttributes lists the attribute names that pass the
// chain gate (row-local only).
RowLocalAttributes []string `json:"row_local_attributes"`
// RejectionRules names the operator categories the chain gate
// rejects today. Intended for LLM-side reasoning and fallback
// routing; not a strict schema.
RejectionRules []string `json:"rejection_rules"`
// OverlayKinds lists the whole-chain overlay catalog entries
// accepted on ChainRequest.Overlays today. Alphabetically sorted.
// Minimal additive surface kept for backward compatibility; the
// richer per-kind sub-block at Overlays carries the full surface
// (shapes / scopes / ref kinds / buffered / description).
OverlayKinds []string `json:"overlay_kinds"`
// Overlays enumerates the per-kind capability rows for every
// whole-chain overlay the catalog declares. Sourced from
// descriptor.overlayCapabilityFor() so the chain capability stays
// byte-equal with the corresponding entries on Manifest.Overlays —
// single source of truth for kind metadata. Sorted alphabetically by
// Kind for golden stability. Each row carries:
//
// - Kind — the OverlayKind constant (SCREAMING_SNAKE).
// - Shapes — every OverlayShape the kind may emit; whole-chain
// kinds inherit the target stage's host shape so
// the catalog declares MATRIX + SCALAR + SERIES.
// - Scopes — the OverlayScope values the kind supports;
// whole-chain kinds decorate the entire chain
// result so Scopes == [total].
// - RefKinds — the OverlayRef union pointer-field names the kind
// consumes; the chain family consumes Stage.
// - Buffered — !types.OverlayStreamable(kind); whole-chain kinds
// stay buffered today (the streamability flag is
// the inverse of the table in
// types/overlay_streamability.go).
// - Description — short human-readable summary; mirrors the prose
// in skills/overlay-system.md.
Overlays []OverlayCapability `json:"overlays"`
}
ProcessChainCapability describes the source-rooted linear chain endpoint surfaced by pulse.ProcessChain / pulse_process_chain. The manifest carries one ProcessChainCapability entry under Manifest.ProcessChain so LLM clients can detect the chain gate and choose between the chained or per-stage fallback path.
type RegressionMeta ¶ added in v0.6.0
type RegressionMeta struct {
// Name is the operator identifier (REG_OLS, REG_GLM,
// REG_BAYES_LINEAR).
Name string `json:"name"`
// Description is a one-sentence prose summary for LLM-side
// operator selection.
Description string `json:"description"`
// AcceptsTypes lists the schema field types valid as Target /
// Predictors entries (numeric only in v1).
AcceptsTypes []string `json:"accepts_types"`
// EmitsTypeNote describes the structured output (RegressionResult)
// since no scalar field type captures the fit summary.
EmitsTypeNote string `json:"emits_type_note"`
// Streamable mirrors types.RegressionType.Streamable(): true for
// closed-form fits (REG_OLS, REG_BAYES_LINEAR), false for iterative
// fits (REG_GLM). Spec-level modifiers (Resample, Selection) can
// downgrade this further per request — see RegressionSpec.Streamable.
Streamable bool `json:"streamable"`
// StreamableHint flags the modifier downgrade rule so clients know
// non-empty Resample / Selection forces the buffered path even on
// streamable operators.
StreamableHint string `json:"streamable_hint,omitempty"`
// Params lists the per-operator parameter schema, including the
// regularization, family, prior, resample, and selection knobs that
// modify the underlying fit.
Params []Param `json:"params"`
// Modifiers names the spec-level orthogonal wrappers any regression
// type supports. Each entry is a top-level RegressionSpec field
// (Resample, Selection) plus its enum values; the field is
// duplicated under Params for completeness but exposed at the top
// level here so request authors can discover the composition story
// without parsing the full param list.
Modifiers []RegressionModifier `json:"modifiers,omitempty"`
}
RegressionMeta describes a registered REG_* operator in the manifest. The shape mirrors Operator and TestMeta enough that LLM clients can reuse the same authoring path, but regression-specific knobs (family, link, penalty, modifier slots) live alongside the shared name + streamable + params trio so the catalog stays one fetch deep.
type RegressionModifier ¶ added in v0.6.0
type RegressionModifier struct {
Name string `json:"name"`
Description string `json:"description"`
EnumValues []string `json:"enum_values"`
}
RegressionModifier describes a spec-level wrapper (Resample, Selection) that composes with any regression operator. Each modifier downgrades streamability when set; clients combine its EnumValues with their chosen Type to author the request.
type ShardInfo ¶ added in v0.8.0
ShardInfo is one shard inside a Pulse shard archive, surfaced by Inspect for archive-backed cohorts. Mirrors the public shape of service.ShardEntry without importing it — descriptor/ is header-only and must not depend on the execution layer.
Filename is the basename of the shard inside the archive (e.g. "20190101.pulse"). RecordCount is the number of records carried by the shard, computed by peeking that shard's own header (the per-shard headers are authoritative; the canonical _schema.pulse aggregate is only a sanity check).
type SlotPair ¶ added in v0.19.0
type SlotPair struct {
// ReferenceLabel is the slot label the spec's Reference field
// resolves to. May be empty when the failure is the reference
// itself being unknown / unresolvable.
ReferenceLabel string `json:"reference_label"`
// TargetLabel is the slot label the spec's Targets[j] resolves
// to. May be empty when the failure is on the Reference arm.
TargetLabel string `json:"target_label"`
// Reason is a stable, machine-readable identifier for the failure
// class. See the SlotPair doc for the enum.
Reason string `json:"reason"`
}
SlotPair carries one rejected (Reference, Target) Compose-overlay slot pair from descriptor.ValidateCompose's overlay walk. Mirrors ChainOverlaySchemaDivergence in spirit but flattens to a string-typed (ref label, target label, machine-readable reason) tuple so the PredictResult.OverlaysSchemaDivergence slot and the ComposeValidationResult.OverlaysSchemaDivergence slot can share one renderer-side shape.
PRD §I-FR-I3: the predict surface for Compose overlays is `OverlaysApplied + OverlaysSchemaDivergence + OverlayCost`. The divergence slot carries the per-spec failure reason alongside the envelope error entry so LLM planners can budget reshapes without stitching envelope details together.
Reason values are stable identifiers the renderer can branch on:
- "kind-unknown" — spec.Kind absent from the overlay catalog.
- "reference-unknown" — Reference label does not resolve to a slot.
- "target-unknown" — Targets[j] does not resolve to a slot.
- "slot-shape-divergent" — ref and target host shapes differ.
- "slot-not-crosstab" — kind requires MATRIX but a slot isn't.
- "schema-divergent" — per-axis grouper-kind tuples disagree.
- "panel-targets-over-cap" — multi-ref target count exceeds the cap.
type Suggestion ¶ added in v0.5.0
type Suggestion struct {
Path []string `json:"path"`
Reason string `json:"reason"`
Current any `json:"current,omitempty"`
Proposed []any `json:"proposed,omitempty"`
Confidence float64 `json:"confidence"`
}
Suggestion is a structured next-action attached to PredictResult. Predict computes suggestions inline so callers can repair a request without an additional inspect round-trip.
Path points at the offending request location using JSON-style segments — e.g. ["Aggregations", "0", "Field"] addresses the Field of the first aggregation.
Proposed is a ranked list of candidate values. Empty when no concrete proposal applies (e.g. ATTR_PERCENTILE has no streamable peer); the caller should treat empty Proposed as advisory.
Confidence is a static heuristic in [0, 1]: 0.9 for high-certainty single-candidate swaps and Levenshtein distance 1; 0.7 for distance 2; 0.6 for multi-candidate type-class swaps; 0.5 for missing-param fallbacks that hand the user a list to pick from; 0.8 for streamability substitutes.
type TestMeta ¶ added in v0.5.0
type TestMeta struct {
// Name is the canonical entry identifier. For tier-1 this is the
// TestType string (e.g. "TEST_ANOVA_WELCH"). For tier-2 this is the
// TestType plus the variant suffix
// (e.g. "TEST_ANOVA_WELCH/welch_one_way_post").
Name string `json:"name"`
// Family is the canonical TestType the variant belongs to
// (e.g. "TEST_PEARSON_R"). Tier-1 entries always have
// Family == Name. Tier-2 entries set Family to the underlying
// TestType so clients can pair siblings.
Family string `json:"family"`
// Tier is 1 for row tests in Request.Tests, 2 for post tests in
// Request.PostTests.
Tier int `json:"tier"`
// Variant is the algorithm flavour for tier-2 entries
// (e.g. "welch_one_way_post"). Empty for tier-1 entries.
Variant string `json:"variant,omitempty"`
// Description is a one-sentence prose summary.
Description string `json:"description"`
// Streamable mirrors types.TestType.Streamable() for tier-1 entries.
// Always false for tier-2 entries.
Streamable bool `json:"streamable"`
// Params lists the operator-specific parameters (alpha,
// success_value, etc.).
Params []Param `json:"params"`
// Requires lists the top-level Test fields that must be set for the
// test to run (e.g. "Field", "Field2", "SplitBy", "Rows", "Cols").
// Drives request authoring directly.
Requires []string `json:"requires,omitempty"`
}
TestMeta describes a statistical test entry in the manifest. Tier-1 and tier-2 tests share this shape; they live in separate top-level slices (Manifest.Tests for tier-1, Manifest.PostTests for tier-2). The Family field ties variants of the same underlying test together across tiers so clients can filter by family.
Source Files
¶
- capabilities_aggregators.go
- capabilities_attributes.go
- capabilities_chain.go
- capabilities_crosstab.go
- capabilities_distributions.go
- capabilities_errors.go
- capabilities_export.go
- capabilities_facet.go
- capabilities_features.go
- capabilities_filterers.go
- capabilities_groupers.go
- capabilities_join.go
- capabilities_mcp.go
- capabilities_overlay.go
- capabilities_regressions.go
- capabilities_tests.go
- capabilities_window_ops.go
- chain.go
- chain_overlay.go
- components.go
- compose.go
- crosstab.go
- defaults.go
- doc.go
- envelope.go
- extensions.go
- facet.go
- inspect.go
- join.go
- labels.go
- manifest.go
- manifest_slim.go
- mergeability_lookup.go
- operator.go
- overlay.go
- overlay_facet.go
- overlay_formula.go
- predict.go
- predict_extensions.go
- predict_feature.go
- predict_regression.go
- predict_suggestions.go
- predict_test_ops.go
- predict_window.go
- schema.go