README
¶
Pulse Examples
Runnable JSON request files organized by topic. Every example resolves
its cohort from the shared fixture set under fixtures/ so adding a new
example is just authoring JSON — no new CSV, schema, or build step.
Layout
internal/examples/
├── README.md # this file
├── fixtures/ # shared CSVs + schemas + generator + build script
│ ├── gen.go # deterministic CSV generator (run once)
│ ├── build.sh # CSV -> .pulse importer (writes ./.data)
│ ├── *.csv # checked-in source data
│ └── schemas/*.json # hand-tuned schemas with descriptions
├── attributes/ # ATTR_* per-record derivations (6)
├── features/ # FEAT_* feature engineering (10)
├── filterers/ # FILTER_* row-level predicates (6)
├── groupers/ # GROUP_* partitioning (7)
├── windows/ # WIN_* per-row analytics (10)
├── aggregations/ # AGG_* requests over the all_types cohort (5)
├── tests/ # TEST_* tier-1 and tier-2 statistical tests (10)
├── matrices/ # MAT_* matrix operators over virtual vectors (1)
└── <new-category>/ # each: README.md + run-all.sh + *.json
One-time setup
Build the binary, generate fixtures, import to .pulse:
make build
./internal/examples/fixtures/build.sh
That writes transactions.pulse, customers.pulse, orders.pulse,
training_data.pulse, all_types.pulse, and experiment.pulse into
.data/ at the repo root. Every example request has
cohort.data_dir = ".data" so the requests run unmodified.
Available cohorts
| File | Fields |
|---|---|
transactions.pulse |
id (u32), amount (f64) |
customers.pulse |
id (u32), age (u8), income (f64), region (categorical), city (categorical) |
orders.pulse |
id (u32), order_date (date), revenue (f64), ordered_at (datetime) |
training_data.pulse |
id (u32), region/occupation/category (categorical), label (u8 binary), price/income (f64), signup_date (date) |
all_types.pulse |
One column for every one of the 17 supported types (u8/u16/u32/u64, f32/f64, nullable_bool/u4/u8/u16, date, packed_bool, categorical_u8/u16/u32, decimal128, nullable_decimal128). Hand-curated 5-row CSV — exists so a single fixture exercises every type-aware code path. |
experiment.pulse |
A/B test cohort: id (u32), treatment/region/segment/converted (categorical), revenue/session_minutes (f64), period (date). Planted effects across treatment, region, segment, and time so every TEST_* example produces a meaningful, non-trivial result. |
The generator (fixtures/gen.go) is deterministic — re-running with the
same seed produces byte-identical CSVs. Re-generate from scratch:
go run internal/examples/fixtures/gen.go
./internal/examples/fixtures/build.sh
Running
A category-specific runner exercises everything in its folder:
./internal/examples/attributes/run-all.sh
./internal/examples/features/run-all.sh
./internal/examples/filterers/run-all.sh
./internal/examples/groupers/run-all.sh
./internal/examples/windows/run-all.sh
./internal/examples/tests/run-all.sh
Or invoke individual requests:
bin/pulse api predict --request internal/examples/features/01_log_transform.json --json
bin/pulse api process --request internal/examples/features/01_log_transform.json --json
The _meta block
Every example JSON carries a top-level _meta object; it is stripped
before the body is served or run (examples.Get), and request decoding
ignores it.
| Key | Required | Meaning |
|---|---|---|
name |
yes | unique example ID (pulse_examples_get name=...) |
category |
yes | must equal the parent directory |
description |
yes | one-sentence summary served by search |
tags |
yes | drawn from CanonicalTags |
operators |
yes (may be []) |
exactly the "type": "AGG_/ATTR_/FILTER_/GROUP_/WIN_/FEAT_/TEST_/REG_..." values in the body (TestExamples_OperatorsMatchBody); overlay kinds ride overlays[].kind instead |
intents |
expected | intent-taxonomy IDs (TestExamples_IntentsFromTaxonomy) |
capabilities |
optional | non-operator feature names in feature-profile spelling (capability:stream), for capabilities the body carries no structural signal for |
capabilities exists so an example is joined to every capability it
needs by an explicit edge in the discovery graph rather than by a scan of
its prose. The graph builder (internal/descriptor/ontology_examples.go)
already DETECTS capability:crosstab (a crosstab key), capability:compose
(a root requests), capability:process_chain (a root stages),
capability:joins (a joins key) and capability:facet (category or tag
facet) from the body, plus every overlays[].kind; declare the rest —
capability:stream, watch, shard, index, sample, labels,
range_tables, filter_to_file — in capabilities. An example with an
empty operators list must declare at least one capability. Gates
(internal/descriptor): TestExamples_CapabilitiesFromFeatures (every
value is a known non-operator feature), TestExamples_EdgeCoverage (every
operator a body or description names is an edge; a declared capability
never contradicts its detector), and
TestExamples_EmptyOperatorExamplesDeclareCapabilities.
Adding a new example category
- Pick a category name (e.g.
windows,aggregation,attributes). - Create
internal/examples/<category>/with aREADME.mdand request*.jsonfiles. - Each JSON sets
cohort.data_dir = ".data"and references one of the shared cohort filenames above. - If your category needs a field that's not in any shared cohort, add
it to
fixtures/gen.goand the matching schema infixtures/schemas/. Re-run the generator and check in the new CSV. - Drop in a
run-all.shfor the category (copy fromfeatures/and adjust the glob). - Optionally add an end-to-end test in
examples_test.gomirroringTestFeatureExamples_RunEndToEnd.
CI coverage
TestExamples_RunEndToEnd (in examples_test.go) builds the shared
fixtures into a temp directory and runs every example across every
category through pulse.Process, asserting no errors. Adding a new
category is one line in the test's categories slice.
go test -run TestExamples -v .
One sub-test runs per example file, grouped by category.
Documentation
¶
Overview ¶
Package examples embeds the catalogue of runnable Pulse request JSON files and exposes a search/get API over them.
Every embedded JSON carries a top-level _meta block describing the example's name, category, taxonomy tags, the operators it uses, and a one-sentence description. Pulse's types.Request unmarshaler ignores unknown fields by default, so _meta is invisible at execution time — the examples remain runnable verbatim.
Get returns the example with the _meta block stripped, so callers can hand the JSON straight to pulse_process / pulse_predict.
Index ¶
- Variables
- func AllCategories() []string
- func AllTags() []string
- func Capabilities() map[string][]string
- func Count() int
- func DeriveOperators(body []byte) []string
- func Intents() map[string][]string
- func IsCanonicalTag(tag string) bool
- func Parse(data []byte, strict bool) (*Example, Meta, error)
- func Tokenize(s string) []string
- type Example
- type ExampleSummary
- type Meta
- type Options
- type Query
Constants ¶
This section is empty.
Variables ¶
var CanonicalTags = []string{
"time-series", "cohort-analysis", "experiment-analysis", "correlation-analysis",
"comparison", "before-after", "top-n", "distribution-shape", "cross-tabulation",
"proportion-analysis", "trend-detection", "outlier-detection", "cardinality-analysis",
"data-quality", "financial", "feature-engineering",
"hypothesis-test", "t-test", "parametric", "nonparametric", "paired", "one-sample",
"two-sample", "k-sample", "repeated-measures", "post-hoc",
"normality-test", "homogeneity-test", "exact-test",
"regression", "ecological",
"ols", "glm", "logistic", "bayesian",
"regularization", "ridge", "lasso", "elasticnet",
"polynomial", "resampling", "jackknife", "selection", "stepwise",
"tier-1-test", "tier-2-test", "composed", "pre-filter", "feature-pipeline",
"window-operator", "streaming-friendly", "buffered-pipeline",
"matrix", "covariance", "multivariate",
"leakage-risk", "small-sample",
"facet",
"sharded", "anchor",
"overlay",
"byte-equal-test",
"compose",
"crosstab",
"welch",
"welford-triple",
"z",
}
CanonicalTags is the curated taxonomy every annotated example must draw from. Kept here (rather than embedded in a separate file) so the validation gate can compare directly without round-tripping through a disk asset.
var ErrNoMeta = errors.New("missing _meta block")
ErrNoMeta is Parse's error for a JSON object without a _meta block.
Functions ¶
func AllCategories ¶
func AllCategories() []string
AllCategories returns every directory the library indexes, sorted alphabetically.
func AllTags ¶
func AllTags() []string
AllTags returns every taxonomy tag the library currently uses, sorted alphabetically. Derived from the indexed examples — not the canonical allowlist — so callers see exactly what is tagged today.
func Capabilities ¶
Capabilities returns every example's optional _meta.capabilities, keyed by example name; an example without the key is absent. Values are feature-profile spellings of non-operator features (`capability:stream`), validated by internal/descriptor (TestExamples_CapabilitiesFromFeatures), which this package cannot import. The key exists for capabilities a request body carries no structural signal for; internal/descriptor's ontology builder turns each value into an `example requires_capability <feature>` edge.
func DeriveOperators ¶
DeriveOperators returns the sorted, distinct operator names a body carries as a `"type"` value — what `_meta.operators` must equal (TestExamples_OperatorsMatchBody; embedder examples at pulse.New).
func Intents ¶
Intents returns every example's optional _meta.intents, keyed by example name; an example without the key is absent. Values are validated against the intent taxonomy by internal/descriptor (TestExamples_IntentsFromTaxonomy), which this package cannot import.
func IsCanonicalTag ¶
IsCanonicalTag reports whether tag belongs to the curated taxonomy.
func Parse ¶
Parse reads one example file: a JSON object carrying a _meta block plus the request body. It returns the Example — Body is the object minus _meta, re-marshaled deterministically — and the parsed Meta. strict refuses an unknown _meta key (embedder examples); the embedded library parses leniently. Parse does not validate Meta's values.
Types ¶
type Example ¶
type Example struct {
Name string `json:"name"`
Category string `json:"category"`
Tags []string `json:"tags"`
Operators []string `json:"operators"`
Description string `json:"description"`
// Intents is the example's _meta.intents (intent-taxonomy IDs);
// absent on the wire when the example declares none.
Intents []string `json:"intents,omitempty"`
Body json.RawMessage `json:"body"`
}
Example is the full record returned by Get. Body carries the request JSON with the _meta block stripped, so it can be handed verbatim to pulse_process / pulse_predict.
type ExampleSummary ¶
type ExampleSummary struct {
Name string `json:"name"`
Category string `json:"category"`
Tags []string `json:"tags"`
Operators []string `json:"operators"`
Description string `json:"description"`
// Intents is the example's _meta.intents (intent-taxonomy IDs);
// absent on the wire when the example declares none.
Intents []string `json:"intents,omitempty"`
}
ExampleSummary is the lightweight projection used by Search results.
func Search ¶
func Search(query string, tags []string, category string) []ExampleSummary
Search returns summaries matching all three filters. An empty filter is treated as "no constraint" for that dimension. It is SearchQuery over the embedded library with no synonym tables; SearchQuery documents the matching rules. Never nil.
func SearchIn ¶
func SearchIn(lib []*Example, query string, tags []string, category string) []ExampleSummary
SearchIn is Search over lib, which must be sorted by name (the no-query order and the score tie-break). An instance serving embedder examples searches the merged library through it.
func SearchLibrary ¶
func SearchLibrary(q Query, opts Options) []ExampleSummary
SearchLibrary is SearchQuery over the embedded library.
func SearchQuery ¶
func SearchQuery(lib []*Example, q Query, syn Options) []ExampleSummary
SearchQuery returns the summaries of lib matching q, ranked. lib must be sorted by name (the no-query order and the score tie-break).
Matching runs as ordered tiers; the first tier with any hit wins:
- exact — a query that is a single plain word (letters and digits only) keeps the legacy rule: a case-insensitive substring of the name, the description or any operator, so a one-word query answers exactly as before. Any other query is tokenized (lower-cased, split on non-alphanumerics, no stemming, common stop words dropped) and an example matches when every query token is an exact token of at least one of its fields (name, description, operators, intents, tags); a field counts when it holds any query token.
- synonym — a query phrase that is a Purpose.KnownAs alias expands the query to that operator, and a query phrase found inside an intent's Sound expands it to that intent. An example matches when its operators or intents carry an expansion; the fields matched by the expansions and by any query token count.
Score = distinct matched fields; ties break alphabetically by name. No query lists every example passing the filters, alphabetically. Returns a non-nil empty slice when nothing matches.
type Meta ¶
type Meta struct {
Name string `json:"name"`
Category string `json:"category"`
Tags []string `json:"tags"`
Operators []string `json:"operators"`
Description string `json:"description"`
// Intents is optional; every value must be an intent-taxonomy ID
// (validated by internal/descriptor's TestExamples_IntentsFromTaxonomy,
// since this package cannot import the taxonomy).
Intents []string `json:"intents,omitempty"`
// Capabilities is optional; every value must be a non-operator
// feature name (validated by internal/descriptor's
// TestExamples_CapabilitiesFromFeatures).
Capabilities []string `json:"capabilities,omitempty"`
}
Meta mirrors the _meta block every example JSON carries — built-in and embedder (pulse.Extensions.Examples) alike.
type Options ¶
type Options struct {
// Aliases maps a Purpose.KnownAs alias to the operator declaring it.
Aliases map[string]string
// Sounds maps an intent Sound to its intent ID.
Sounds map[string]string
// KeepIntent, when set, drops every _meta.intents value it refuses
// from matching and from the returned summaries (an intent the
// instance hides). Nil keeps all.
KeepIntent func(id string) bool
// KeepExample, when set, drops every example it refuses before any
// tier runs, so a hidden example never decides which tier answers.
KeepExample func(name string) bool
}
Options carries an instance's view into a search: the synonym tables of the synonym match tier and the visibility filters. Every table key is in internal/descriptor's FoldAlias form (lower-cased, whitespace collapsed, punctuation kept). The zero value searches the whole library with the synonym tier disabled.
type Query ¶
type Query struct {
Query string `json:"query,omitempty"`
Tags []string `json:"tags,omitempty"`
Category string `json:"category,omitempty"`
Intent string `json:"intent,omitempty"`
}
Query is one example search: every filter optional, all ANDed.
- Query is matched against an example's name, description, operators, _meta.intents and tags (see SearchQuery for the match tiers).
- Tags is ANDed: an example must carry every requested tag.
- Category is an exact directory match.
- Intent is an intent-taxonomy ID the example's _meta.intents must carry. This package does not validate it (it cannot import the taxonomy); internal/descriptor refuses an unknown or hidden one before searching.
Source Files