index

package
v0.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 25, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package index defines the consumer-side chunker contract: the bridge between the SDK's per-space change feed and the search indexer (internal/indexer). Chunkers turn one object's records into a stream of IndexEntry values ordered by ApplySeq — the SDK's per-space, peer-local, monotonic apply counter (advances on every apply that mutates a record, including non-DAG ones). Record deletions are first-class: a deleted record yields a removal entry (Data == ""); structural removals (object deletion, type detach) are derived by the indexer from the shared objects row and applied as primary-key prefix deletes. See docs/13-index.md for the full contract.

Index

Constants

View Source 
const (
	ScopeBasic = "basic" // editor blocks, object names/descriptions
	ScopeChat  = "chat"  // chat messages
	// ScopeProps is the default scope for user property values (see
	// PropChunker). FTS-only: the indexer never embeds props-scope docs —
	// short "Name: value" entries embed badly and would pollute vector
	// recall. A meta.index override into another scope opts back in.
	ScopeProps = "props"
)

Conventional scope slugs. Scopes are an OPEN set — any slug passing ValidScope works (property meta flags carry arbitrary scopes); these constants are just the established vocabulary.

View Source 
const (
	// LinkKindMention — an identity referenced in text (`m` URI).
	LinkKindMention = "mention"
	// LinkKindLink — an ordinary reference in text: an object, one of
	// its records or property values, a file.
	LinkKindLink = "link"
	// LinkKindRelation — a value of a link-bearing property or dataset
	// field (the `relation` slug, or an explicit `links` marker on a
	// value field).
	LinkKindRelation = "relation"
	// LinkKindCard — an editor block that is exactly one link to an
	// object or file, rendered as a card (the whole-line rule the web
	// client promotes on).
	LinkKindCard = "card"
	// LinkKindEmbed — an editor block that embeds another object's
	// content: the synced-block envelope referencing a source block.
	LinkKindEmbed = "embed"
)

Link kinds. An open slug set the extractors mint; these are the established vocabulary.

View Source 
const (
	XFormatLinksKey = "links"

	LinkModeOne      = "link"
	LinkModeMany     = "links"
	LinkModeMarkdown = "markdown"
	LinkModeNone     = "none"
)

Link markers on a descriptor: the explicit `links` key, or the slug that implies one. What the marker means for the value: 

link      the string value is one reference
links     the array value lists references
markdown  the text is scanned for references
none      the value is never scanned, whatever the slug implies
View Source 
const (
	NamePropRecordId        = "name"
	DescriptionPropRecordId = "description"
)

Reserved RecordIds for the always-indexed built-in `any` properties. Both are valid base58, so a collision with a real hash-derived propId is theoretically possible — and harmless: it would merge two text sources under one doc id.

View Source 
const ApplySeqField = "_applySeq"

ApplySeqField is the reserved record field carrying the per-space applySeq watermark. Mirrors the SDK's crdt.ApplySeqField — duplicated here so the index package doesn't depend on an SDK internal. It is the record-level twin of space.ObjectChange.ApplySeq, so the chunker window and the change-index cursor share one ordering axis.

View Source 
const DatasetProp = "prop"

DatasetProp is the VIRTUAL dataset name the prop chunker writes under (the middle segment of its doc ids). No SDK handler registers a dataset called "prop", so it can't collide with real dataset chunks.

View Source 
const DatasetSchemaVirtual = "schema"

DatasetSchemaVirtual is the SchemaChunker's VIRTUAL Dataset() name. Never used in doc ids — every entry carries its real runtime dataset name — but it must stay unique among chunkers, and (like DatasetProp) reserved against user dataset names so runtime doc ids can't collide with a chunker's namespace.

View Source 
const MetaCollectionLabel = "__collection__"

MetaCollectionLabel is the `any.type` value of a collection object.

View Source 
const MetaIndexKey = "index"

MetaIndexKey is the property-definition meta key controlling value indexing. Three states: absent/empty → indexed under the default ScopeProps; a scope slug (e.g. "agent") → indexed under that scope; MetaIndexNone → excluded (opt-out for blobs and noisy enums).

View Source 
const MetaIndexNone = "none"

MetaIndexNone is the MetaIndexKey value that excludes a property from indexing. The literal word — an empty string means "default".

View Source 
const MetaTypeLabel = "__type__"

MetaTypeLabel is the SDK's reserved meta-type marker: type-definition objects carry `any.type = "__type__"` and their `any.name` is the type name. The SDK keeps the literal internal, but it is wire-visible on every objects-query row, so it is restated here. Type definitions are schema, not knowledge (discovery is `GET /types`) — the prop chunker always excludes them: their one-word names otherwise win BM25 on field-length normalization and surface as top search hits.

Variables

This section is empty.

Functions

func IsDeleted 

func IsDeleted(rec *anyenc.Value) bool

IsDeleted reports whether rec is a tombstone — the SDK wipes content and sets _deletedAt on delete, preserving _ver / _traces / _applySeq.

func LinkMode 

func LinkMode(xf map[string]any) string

LinkMode resolves a descriptor's link marker: the explicit `links` key wins whatever it holds (`none`, an unknown or a malformed value disable — the gate refuses the latter two on write, so they only arrive from a vendor or a bypass), else the `relation` slug implies `links` and the `markdown` slug implies `markdown`. Empty when the field carries no links.

func Members 

func Members(row *anyenc.Value) map[string]bool

Members reads an objects row's membership into a set: its one type (`any.type`), its collections (`any.collections`) and — a definition object implements itself — its own id when the row carries a marker. Nil for a nil row.

func RecordsSince 

func RecordsSince(ctx context.Context, q space.Query, since uint64, yield func(rec *anyenc.Value, applySeq uint64) error) error

RecordsSince is the shared windowed streamer every chunker uses. It chains the deletion opt-in (Projection{IncludeDeleted: true}), the cursor window (Filter {"_applySeq": {"$gt": since}}), and ascending ApplySeq order onto the supplied query, then iterates and yields each record paired with its parsed ApplySeq.

Projection{IncludeDeleted: true} is what makes tombstones visible — without it the SDK's find path skips deleted rows, so deletions would never reach the indexer. Tombstones carry _applySeq (the SDK stamps it on delete too), so they land in the same window as content. The $gt window mirrors the SDK's own ChangedSince contract: rows never stamped with _applySeq (pre-applySeq writes) sort below any since >= 0 and are excluded — "index from the next change."

A yield error stops iteration and is returned after the iterator is closed.

func ValidLinkMode 

func ValidLinkMode(s string) bool

ValidLinkMode reports whether s is one of the marker values.

func ValidScope 

func ValidScope(s string) bool

ValidScope reports whether s is an acceptable scope slug: 1..64 chars of [a-z0-9_-]. Scopes are an open set — property meta flags carry arbitrary slugs — so the search handler and the prop-chunker catalog share this sanity rule instead of a fixed list.

Types

type CatalogInvalidator 

type CatalogInvalidator interface {
	Invalidate(spaceId string)
}

CatalogInvalidator is an optional Chunker capability for chunkers holding a per-space catalog snapshot with a TTL (the prop chunker): the worker calls Invalidate when a type object changed in the feed, so a property added a moment ago is in the catalog when the values written right after it are extracted.

type Chunker 

type Chunker interface {
	// Dataset is the dataset this chunker writes — the middle segment
	// of the index doc id (objectId:dataset:recordId). May be virtual
	// (see DatasetProp): it only has to be unique among chunkers,
	// colon-free, and stable.
	Dataset() string
	// TypeId is the type gating this chunker: the indexer runs
	// ChunksSince only while the object has the type (or is the type
	// itself), and prefix-evicts objectId:<dataset>: when it does not
	// — covering a retype. Empty = ungated, runs for every object.
	TypeId() string
	// ChunksSince streams every entry of objectId with ApplySeq > since,
	// ascending by ApplySeq. Cleared/deleted records yield removal
	// entries (Data == ""). yield is called once per entry; a yield
	// error stops the stream and is returned.
	ChunksSince(ctx context.Context, sp space.Space, objectId string, since uint64, yield func(IndexEntry) error) error
}

Chunker streams the IndexEntry values for one dataset on one object.

type DynamicChunker 

type DynamicChunker interface {
	Chunker
	// EvictDatasets returns the runtime dataset names to structurally
	// evict for an object with the given membership set (its type and
	// its collections): catalog datasets whose owning type is not the
	// object's (the retype path), plus names retired since process
	// start (definition removed).
	EvictDatasets(ctx context.Context, sp space.Space, attached map[string]bool) ([]string, error)
}

DynamicChunker is an optional Chunker capability for chunkers whose dataset set is defined at runtime (schema-driven datasets). The worker replaces the static Dataset()/TypeId() gate for such a chunker: it prefix-evicts objectId:<ds>: for every name returned by EvictDatasets, then runs ChunksSince as usual — the chunker self-gates its active set, so an evicted dataset is never also streamed in the same page.

type IndexEntry 

type IndexEntry struct {
	Scope    string
	ObjectId string
	Dataset  string
	RecordId string
	Data     string // text to index; empty = remove this record from the index
	Title    string // optional BM25F-boosted field (heading / signature / summary);
	// its terms should also appear in Data — Title only adds ranking weight,
	// and the content hash (embed-skip) is over Data alone.
	ApplySeq uint64 // peer-local, per-space monotonic apply counter
	// Links are the edges found in the record(s) this entry covers —
	// the link sink's input (docs/13-index.md § Links). Each carries
	// its own source place: a coalesced editor window reports the
	// links of every member block under that block's id. A streaming
	// chunker's entry replaces its record's edges (nil = the record
	// links nothing); a reconciling chunker's set replaces the
	// collection's edges. Not part of the content hash.
	Links []LinkEntry
}

IndexEntry is one unit handed to the indexer: the text of a single record, tagged with enough identity to address it (Scope + ObjectId + Dataset + RecordId) and ordered by ApplySeq.

An empty Data is a record-level removal signal: "this record is gone (or has nothing to index), remove it." Removing a never-indexed record is a no-op on the indexer side, so removals are safe to emit unconditionally. Structural removals — a whole object (deletion) or one object's dataset (type detach) — are NOT expressed as entries: the indexer derives them from the shared objects row inside the same ChangedSince window and applies them as id-prefix deletes in the same page transaction.

type LinkEntry 

type LinkEntry struct {
	ObjectId string
	Dataset  string
	RecordId string
	// TypeId names the type declaring a property value's source
	// (DatasetProp edges — a value lives at record[typeId][propId]);
	// empty for record sources.
	TypeId string
	// Field names the record field the reference was read from when
	// the record has several link-bearing fields (runtime datasets);
	// empty when the source has one text (blocks, messages) or is a
	// property value.
	Field  string
	Kind   string
	Target anyuri.URI // canonical (URI.Canonical), never a space
}

LinkEntry is one edge: the record it was found in (the source place — same identity vocabulary as IndexEntry, `DatasetProp` + propId for a property value) and the canonical target. 

func TextLinks(spaceId, objectId, dataset, recordId, text string) []LinkEntry

TextLinks scans markdown text for references and returns the edges for source (objectId, dataset, recordId) in spaceId: mentions as LinkKindMention, everything else as LinkKindLink. Space references, unknown kinds and a link to the source object itself (no record, no property — the parent is not a backlink) are dropped; duplicates merge on the canonical target. 

func ValueLinks(spaceId, objectId, dataset, recordId, mode string, v *anyenc.Value) []LinkEntry

ValueLinks reads references off a value field by its marker mode: one string for LinkModeOne, an array of strings for LinkModeMany, scanned text for LinkModeMarkdown. Value links are LinkKindRelation; text links keep their text kinds. Malformed elements are skipped.

type ModuleChunker 

type ModuleChunker struct {
	// contains filtered or unexported fields
}

ModuleChunker indexes every collection a dataset module serves in a space: the module's shared canonical collection and each namespaced `<typeId>_<key>` instance a type declares. One registered chunker per module, resolved per space from Space.Datasets (the `Module` of each discovered dataset) — the same in-memory snapshot the SDK refreshes synchronously when a declaration applies, so a read is never stale relative to the applySeq window being processed.

The gate is collection ownership: an object holds a collection when it carries one of the collection's Owners — the one declaring type of a namespaced instance, any of the types sharing a canonical one. The worker structurally evicts objectId:<collection>: for every collection the object cannot hold (DynamicChunker), plus collections that vanished from the catalog since process start (a part removed — the SchemaChunker's retired set, same restart caveat: docs/13-index.md § Removal semantics). Entries carry the real collection as their dataset, so doc ids stay per collection; Dataset() is the module's virtual name.

Exactly one of stream / reconcile is set. A reconciling module implements MultiReconciler: the worker diffs each collection's stored docs against the returned set, so an edit re-embeds only the windows that changed.

func NewModuleChunker 

func NewModuleChunker(module string, shape any) *ModuleChunker

NewModuleChunker constructs a chunker for module. Pass either a ModuleStream or a ModuleReconcile as the shape.

func (*ModuleChunker) ChunksSince 

func (c *ModuleChunker) ChunksSince(ctx context.Context, sp space.Space, objectId string, since uint64, yield func(IndexEntry) error) error

ChunksSince streams every active collection's entries (stream shape). A reconciling module answers through ReconcileAll instead; this path then renders its full sets, which the worker treats as a plain stream only when it does not implement MultiReconciler.

func (*ModuleChunker) Dataset 

func (c *ModuleChunker) Dataset() string

Dataset is the module's virtual name — unique among chunkers, never a doc-id segment (entries carry their collection).

func (*ModuleChunker) EvictDatasets 

func (c *ModuleChunker) EvictDatasets(ctx context.Context, sp space.Space, attached map[string]bool) ([]string, error)

EvictDatasets returns the collections to structurally evict for an object with the given membership: every collection of the module none of whose owners is attached, plus collections retired from the catalog.

func (*ModuleChunker) Module 

func (c *ModuleChunker) Module() string

Module returns the module slug this chunker indexes.

func (*ModuleChunker) ReconcileAll 

func (c *ModuleChunker) ReconcileAll(ctx context.Context, sp space.Space, objectId string, since uint64) (map[string][]IndexEntry, error)

ReconcileAll returns, per active collection, the full current entry set (reconcile shape). Only meaningful for reconciling modules; the worker checks MultiReconciler membership through Reconciles.

func (*ModuleChunker) Reconciles 

func (c *ModuleChunker) Reconciles() bool

Reconciles reports whether this chunker diffs full sets per collection (reconcile shape) rather than streaming records.

func (*ModuleChunker) TypeId 

func (c *ModuleChunker) TypeId() string

TypeId is empty: gating is per collection through EvictDatasets.

type ModuleReconcile 

type ModuleReconcile func(ctx context.Context, sp space.Space, objectId, collection string) ([]IndexEntry, error)

ModuleReconcile renders one collection's full current entry set — the coalesced shape (editor windows), diffed against the store by the indexer.

type ModuleStream 

type ModuleStream func(ctx context.Context, sp space.Space, objectId, collection string, since uint64, yield func(IndexEntry) error) error

ModuleStream streams one collection's entries past the cursor — the per-record shape (chat messages).

type MultiReconciler 

type MultiReconciler interface {
	DynamicChunker
	Reconciles() bool
	ReconcileAll(ctx context.Context, sp space.Space, objectId string, since uint64) (map[string][]IndexEntry, error)
}

MultiReconciler is a DynamicChunker whose index unit spans multiple records on SEVERAL collections at once (a module's canonical collection plus namespaced instances). ReconcileAll returns the full current entry set per active collection; the indexer diffs each collection's stored docs (prefix objectId:<collection>:) against its set exactly as Reconcile does for one dataset. Reconciles reports whether the chunker takes this path at all — a streaming module chunker returns false and is served through ChunksSince.

type PropChunker 

type PropChunker struct {
	// contains filtered or unexported fields
}

PropChunker indexes property VALUES from the shared `objects` collection: one entry per (object, indexed property), doc id objectId:prop:propId. User properties index BY DEFAULT under the dedicated scope ScopeProps with self-describing entry text ("<prop name>: <value>"); a property definition's meta["index"] = "<scope>" overrides the scope and meta["index"] = "none" opts out (see the SDK's PropertyDraft.Meta). The built-in any.name and any.description are always indexed under scope "basic", raw (no name prefix). Ungated: it runs for every object except type-definition rows (MetaTypeLabel) and the wired-in excludeTypes.

func NewPropChunker 

func NewPropChunker(excludeTypeIds ...string) *PropChunker

NewPropChunker constructs the chunker with an empty catalog cache. excludeTypeIds names types whose objects are skipped entirely (e.g. a diagnostic type — see PropChunker.excludeTypes); MetaTypeLabel is always excluded on top of them.

func (*PropChunker) ChunksSince 

func (c *PropChunker) ChunksSince(ctx context.Context, sp space.Space, objectId string, since uint64, yield func(IndexEntry) error) error

ChunksSince streams the object's property entries past the cursor. Per live row it emits, unconditionally:

  • any.name / any.description under scope "basic";
  • one entry per catalog property — value rendered when its type is attached, Data "" otherwise (idempotent record-level eviction of cleared values and detached-type props).

Tombstoned rows yield nothing — structural eviction is the indexer's job (it never calls chunkers for deleted objects).

func (*PropChunker) Dataset 

func (c *PropChunker) Dataset() string

func (*PropChunker) Invalidate 

func (c *PropChunker) Invalidate(spaceId string)

Invalidate drops the space's catalog snapshot (test/ops hook — the next ChunksSince re-resolves immediately instead of waiting out the TTL).

func (*PropChunker) LinksReplaceCollection 

func (c *PropChunker) LinksReplaceCollection()

LinksReplaceCollection marks the chunker's streams as complete per row (WholeCollectionLinks): a removed definition's edges are evicted on the row's next stream.

func (*PropChunker) TypeId 

func (c *PropChunker) TypeId() string

type Reconciler 

type Reconciler interface {
	Chunker
	// Reconcile returns every current index entry for objectId on this
	// chunker's dataset. since is the indexer's cursor (the current
	// implementation rebuilds the full set and ignores it; a future
	// incremental implementation could use it to scope work).
	Reconcile(ctx context.Context, sp space.Space, objectId string, since uint64) ([]IndexEntry, error)
}

Reconciler is an optional chunker capability for datasets whose index unit spans MULTIPLE records — e.g. coalesced editor windows, where one index doc concatenates several blocks. An incremental per-record stream (ChunksSince) can't express such a unit: a window's text needs sibling records below the cursor, and a deleted record's position is wiped from its tombstone, so the affected window can't be located incrementally.

Reconcile returns the object's FULL current doc set for the dataset. The indexer diffs it against what is already stored (by content hash): docs that vanished are deleted, new/changed docs are upserted, and unchanged docs are left untouched — so their vectors are preserved and not needlessly re-embedded. The indexer prefers Reconcile over ChunksSince when a chunker implements this interface.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry holds the set of chunkers known to the process. Phase 1 has no consumer — it's constructed at boot and stored on server deps so the phase-2 indexer can look up "which chunkers feed this dataset" without re-wiring. Read-only after construction; safe for concurrent reads.

func NewRegistry 

func NewRegistry(chunkers ...Chunker) *Registry

NewRegistry builds a registry over the given chunkers, in order.

func (*Registry) All 

func (r *Registry) All() []Chunker

All returns every registered chunker, in registration order.

func (*Registry) ForDataset 

func (r *Registry) ForDataset(dataset string) []Chunker

ForDataset returns the chunkers reading the given dataset. Usually one, but the contract allows several (e.g. multiple scopes over one dataset). Empty when no chunker covers the dataset — datasets excluded from indexing return nothing.

type SchemaChunker 

type SchemaChunker struct {
	// contains filtered or unexported fields
}

SchemaChunker indexes records of RUNTIME-DEFINED datasets by their schema's x-search {title, text, scope} mapping: one registered chunker covers every dataset any type in the space declares with a search annotation. Entries carry the real dataset name (doc ids objectId:<dataset>:<recordId>) under the declared scope — absent defaults to "basic" (runtime dataset records are user content on par with editor blocks), an invalid slug makes the dataset unsearchable (a broken override must not silently land in the default scope — the resolveIndexedProp stance).

Unlike PropChunker there is NO catalog TTL cache: Space.Datasets() is an atomic in-memory snapshot the SDK refreshes synchronously when a definitions change applies, so a fresh read is never stale relative to the applySeq window being processed — and a cached one could skip records in the primary define-then-write flow while the cursor advances past them.

It implements DynamicChunker: per object, datasets whose owning type is not attached are structurally evicted by the worker (the DetachType path), and dataset names that vanish from the catalog (definition removed) are remembered for the process lifetime and evicted the same way as objects get dirty. Restart drops the retired set — stale docs of a removed definition survive until a boot-time sweep exists (docs/13-index.md § Removal semantics).

func NewSchemaChunker 

func NewSchemaChunker(staticDatasets ...string) *SchemaChunker

NewSchemaChunker constructs the chunker. staticDatasets are the compiled-in dataset names (indexed or not) to never treat as runtime; the virtual chunker names are always skipped on top of them.

func (*SchemaChunker) ChunksSince 

func (c *SchemaChunker) ChunksSince(ctx context.Context, sp space.Space, objectId string, since uint64, yield func(IndexEntry) error) error

ChunksSince streams the object's runtime-dataset entries past the cursor: per active dataset (owning type attached), every record with ApplySeq > since maps x-search.title → Title and title + the joined text fields → Data. Tombstones and records whose mapped fields render empty yield removal entries (Data == ""). Datasets the worker just evicted (type not attached) are never streamed in the same page.

func (*SchemaChunker) Dataset 

func (c *SchemaChunker) Dataset() string

func (*SchemaChunker) EvictDatasets 

func (c *SchemaChunker) EvictDatasets(ctx context.Context, sp space.Space, attached map[string]bool) ([]string, error)

EvictDatasets implements DynamicChunker: searchable catalog datasets whose owning type is not attached to this object, every unsearchable runtime name (x-search absent or cleared), and every retired name. The resolved catalog is stashed for the paired ChunksSince.

func (*SchemaChunker) EvictText 

func (c *SchemaChunker) EvictText(_ context.Context, sp space.Space, attached map[string]bool) []string

EvictText implements TextEvictor: link-only datasets (link fields, no usable search mapping — a cleared `x-search` included) keep their edges but must lose their text docs every page, the stance `unsearchable` takes for datasets with nothing to keep.

func (*SchemaChunker) TypeId 

func (c *SchemaChunker) TypeId() string

type TextEvictor 

type TextEvictor interface {
	EvictText(ctx context.Context, sp space.Space, attached map[string]bool) []string
}

TextEvictor is an optional DynamicChunker capability for datasets whose TEXT docs must go while their edges stay: the worker prefix-evicts `objectId:<ds>:` on the text collection only for every name returned, and the chunker keeps streaming the dataset for its links (docs/13-index.md § Links). A dataset that lost its search mapping but keeps link fields is the case. 

type WholeCollectionLinks interface {
	LinksReplaceCollection()
}

WholeCollectionLinks is an optional Chunker capability for streaming chunkers whose every stream carries the record's complete edge set for the whole collection — the prop chunker emits one entry per catalog property per row, so a property that left the catalog (definition removed) simply stops being emitted. The worker then replaces the collection's edges on every stream instead of only the streamed records', and the removed property's edges fall out.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.