graphembedding

package
v0.1.0-alpha.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 27 Imported by: 0

README

graph-embedding

The embedding component of the graph subsystem.

Overview

graph-embedding follows the ENTITY_STATES bucket, which graph-ingest writes, and keeps one embedding record per entity that has text in the EMBEDDING_INDEX bucket. An embedding is a vector of numbers computed from the entity's text so that similar texts get similar vectors. At tier 0 (no external provider) the only embedder is BM25, which runs in this process; nothing is sent to a model service. EMBEDDING_DEDUP holds content-addressed embeddings, keyed by the text and the embedder, so identical text is embedded once.

A revision is the number NATS gives each write to a bucket. The component has two hops, and a revision counts as applied only after both:

  1. Hop 1 reads the entity and writes a pending record with its text, or removes the record when the entity is deleted or has no text.
  2. Hop 2, the Worker (graph/embedding), turns each pending record into a generated record (the vector) or a failed one (a bounded reason). Its result is the record's terminal outcome.

The component also answers two query verbs, graph.embedding.query.search (rank entities against a text query) and graph.embedding.query.similar (rank entities against one entity's vector). Both read a vector cache kept in memory from EMBEDDING_INDEX; "Query verbs" below says what they accept and return.

Architecture

ENTITY_STATES ──► graph/follower ──► hop 1 ──► EMBEDDING_INDEX (pending) ──► Worker ──► EMBEDDING_INDEX (generated
  (KV watch)       (the one watch,    (this      ▲                          (hop 2)      or failed), EMBEDDING_DEDUP
                    the watermark,    package)   │
                    GRAPH_STATUS)                └── repair lane retries hop 1's failures every 30 s

graph/follower owns the watch, the replay, the watermark and the readiness record it writes to GRAPH_STATUS under the key graph-embedding. Hop 1 runs one entry at a time on the follower's goroutine; with coalesce_ms above 0 it instead merges updates to one entity and reads the entity's current value when the window ends. The component reports a revision complete to the follower when the Worker reports its terminal outcome, or at once when hop 1 ends it (no text, deleted, or a generated record already standing). So readiness waits for hop 2.

Text

Inline text is the string values of the entity's non-relationship triples whose predicate ends in one of text_suffixes, joined with spaces and cut to max_text_len characters at a word boundary. An entity whose body is offloaded to an object store (a StorageRef) is embedded from its identity text followed by the body, which the Worker reads through the store registry. If no store is registered for the body's storage instance, only the inline text is embedded and content_unresolved_total counts the exclusion.

Configuration

{
  "type": "processor",
  "name": "graph-embedding",
  "enabled": true,
  "config": {
    "ports": {
      "inputs": [
        {"name": "entity_watch", "config": {"kind": "kv-watch", "bucket": "ENTITY_STATES"}},
        {"name": "content_store", "config": {"kind": "store-read", "bucket": "MESSAGES"}}
      ]
    },
    "embedder_type": "bm25",
    "workers": 5,
    "coalesce_ms": 0
  }
}
Option Type Default Description
ports object the ports above Inputs only. A ports block that is present is used as written; one that is absent takes the defaults
embedder_type string bm25 The only value accepted at tier 0; any other, http included, is refused
workers int 5 Concurrent Worker goroutines; negative is refused
max_text_len int 4000 Characters of text embedded per entity, 0 to 1,000,000. A change applies to records generated after it; a record already generated keeps its text until the entity has a new revision, such as a re-ingest
text_suffixes array .title .content .description .summary .text .name .body .abstract .subject Predicate suffixes whose string values are embedded
startup_attempts int 30 Tries to find ENTITY_STATES at startup
startup_interval_ms int 500 Pause between those tries
coalesce_ms int 0 Window in milliseconds for merging updates to one entity; 0 applies each at once

The configuration is decoded strictly: a key not listed here is refused at construction, naming it. This includes batch_size and cache_ttl. The inputs must be a kv-watch of every ENTITY_STATES key with no history (at least one is required) and, optionally, store-read inputs; the store registry is used only when a store-read input is declared. The component declares no outputs and refuses a configuration that does. Registering the factory asks boot for no model registry. MIGRATION.md lists what a SemStreams configuration changes.

Ports

Direction Name Kind Bucket
input entity_watch kv-watch ENTITY_STATES
input content_store store-read MESSAGES

The component writes EMBEDDING_INDEX and EMBEDDING_DEDUP itself and declares no output port for them.

Health and readiness

Health is Healthy once started, and unhealthy while any of these holds: a stored ENTITY_STATES value could not be decoded (Health reads graph state reset required: <reason>, and both query verbs are refused with the error code graph_state_reset_required), failed entities are unresolved, the ENTITY_STATES feed is lost, the Worker's watch on EMBEDDING_INDEX ended while the component ran, or the vector cache's watch on EMBEDDING_INDEX ended (Health reads vector cache watch lost; restart the process). The error count (ErrorCount) rises by one when an entity becomes failed (it was not failed before; see "Failed entities and the repair lane" below) and by one for each stored value refused this way; a repeated failure of an entity already failed, such as a repair pass failing again, and a failure older than the entity's latest outcome add nothing. graph-index counts the same way. The count never lowers Healthy by itself (ADR-109). A lost Worker watch, a lost cache watch and a lost feed hold until the process restarts; nothing rebinds them. DataFlow counts one input per delivered ENTITY_STATES entry, update or delete, before merging.

The follower publishes the readiness record to GRAPH_STATUS at once and then every heartbeat, with indexed_revision, target_revision, failed_count and failed_reasons (the count of failed entities per bounded reason). It reads degraded when entities are failed, the feed is lost, or the component has been behind for 30 seconds with no revision completed. Any completion counts, so one entity held while others finish reads not ready, not degraded (owner ruling (a1), PR #550 comment 6721867279). A graph with no entities is caught up once the snapshot is delivered.

Failed entities and the repair lane

The component keeps, per entity, the source revision and result of its latest terminal outcome; a failure for an older revision is not counted, and a success at a revision clears a failure at or below it. The map is filled from the failed records EMBEDDING_INDEX holds at start. Three failures are hop 1's own: a failed record delete, a failed pending write and a failed entity read. Every 30 seconds the repair lane retries them against the entity's current value; any later convergence of the entity in hop 1 (hop 1 writes its pending record, removes its record, or finds a generated record already standing) clears the failure. The Worker's failures are not retried: they wait for a new revision of the entity or a restart.

Query verbs

Both verbs answer a graph.QueryResponse with producer graph-embedding, from Start until Stop returns. Each reply's data holds results (a JSON array, [] when empty; each hit is an entity_id and a similarity), unreadable_records (records in EMBEDDING_INDEX that could not be decoded) and failed_records (entities whose latest embedding failed, so they cannot be hits). search also sends embedder_type, always bm25. A 0 in both counts does not prove every entity is searchable: an entity whose embedding is still pending is left out and counted in neither.

Subject Request keys Ranks
graph.embedding.query.search query (required, not blank), limit, scope, min_revision entities against the query text
graph.embedding.query.similar entity_id (required), limit, min_revision entities against that entity's vector; the entity itself is never a hit
  • Limit. 0 means 10, 1 to 100 is used as sent, and anything else is refused invalid_request, not cut down.
  • Scope. scope is a list of dot-delimited entity ID prefixes; an entity is a candidate if it matches any of them, and an empty prefix matches every entity. The filter runs before ranking, so a limit of 1 gives the best entity inside the scope. A prefix that fails types.ValidateEntityIDPrefix is refused invalid_request.
  • Hits. Ranked by similarity, ties by entity ID. A hit scoring 0 is dropped.
  • Strict requests. A key the request does not define, or data after the JSON object, is refused invalid_request, naming what was refused.
  • No revision claim. The verbs do not say how current the cache is. indexed_revision is always 0, and a min_revision above 0 is refused invalid_request rather than answered.
  • similar needs a vector. An entity with no record, a record that is not generated, or a generated record with no vector is refused class invalid, code embedding_unavailable; a record that cannot be decoded is refused class invalid with no code.
When a call is withheld

Before answering, each verb asks the follower whether the graph state is ready. It is refused transient index_not_ready before the first build completes, while the ENTITY_STATES feed is lost, while a hop-1 failure (delete_failed, pending_write_failed, entity_read_failed) stands, and while the cache has not applied its first snapshot or has lost its watch. A latched reset is refused fatal graph_state_reset_required. A failure in the Worker (hop 2, such as a content_error for a document whose text cannot be extracted) does not withhold anything: the verbs answer from the other vectors and failed_records counts it. The status the component publishes still reads degraded while such a failure stands. A malformed request is refused invalid_request whatever the readiness.

Metrics

Collectors register on the injected metric.MetricsRegistry, never on Prometheus' global registry. This component registers one, semengine_graph_embedding_content_unresolved_total (counter): entities whose offloaded body was left out because no store is registered for its storage instance. The follower adds readiness, lag, bootstrap_complete, readiness_state, status_publish_failures_total, indexed_revision and target_revision under the same prefix.

Known limits

  • Offloaded-body fetch has no deadline. A store that hangs holds one Worker lane until the process stops (#551).
  • Double count after a restart without the store. If the process restarts with no store registered for an offloaded body, the Worker's replay of the old pending record and hop 1's re-read of the entity are two exclusions, and content_unresolved_total counts both.
  • No retry for the Worker's failures. See the repair lane above.
  • Hop 1 is serial. A slow pending write delays delivery of every entry behind it; the heartbeat keeps publishing, and staleness_ms rises.

Dependencies

  • Upstream: graph-ingest writes ENTITY_STATES, which this component follows.
  • Reads: graph/follower (watch, watermark, readiness), graph/embedding (BM25 embedder, Worker, storage), graph/kvcatalog, storage/storeregistry (offloaded bodies).
  • Writes: EMBEDDING_INDEX and EMBEDDING_DEDUP, which this component owns.

Documentation

Overview

Package graphembedding is the graph-embedding component: it follows ENTITY_STATES and keeps one BM25 embedding record per entity with text in EMBEDDING_INDEX, with its dedup keys in EMBEDDING_DEDUP, and reports its progress on the graph-embedding readiness key. At tier 0 the only embedder is BM25, which runs in-process; nothing reaches an external provider.

A composition registers the factory with Register and creates the component through the component manager, with a configuration that Config describes.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateGraphEmbedding

func CreateGraphEmbedding(rawConfig json.RawMessage, deps component.Dependencies) (component.Discoverable, error)

CreateGraphEmbedding is the factory function for creating graph-embedding components. It opens no connection.

func DeclarePorts

func DeclarePorts(rawConfig json.RawMessage, _ string) (component.PortConfig, error)

DeclarePorts is the component.PortDeclarer for graph-embedding: the ports CreateGraphEmbedding will report for rawConfig, computed without dependencies.

func Register

func Register(registry *component.Registry) error

Register registers the graph-embedding factory with the component registry. It asks boot for no model registry: at tier 0 the embedder is BM25, which needs no provider.

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component implements the graph-embedding processor.

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow reports input messages: one per delivered ENTITY_STATES entry, update or delete, with the delivered value's length, counted before coalescing.

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health reports the shell's verdict (component.Lifecycle.Health) over the follower's stuck states (a latched reset, unresolved writes, a lost feed), a lost Worker watch and a lost vector cache watch.

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize marks the component initialized (no I/O); the factory validated the configuration.

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns the input ports, fixed at construction.

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns the output ports, fixed at construction.

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start begins following ENTITY_STATES and writing pending records. The lifecycle state is the shell's (component.Lifecycle): Start before Initialize, a nil or ended context and a second Start are refused, the second with errs.ErrAlreadyStarted, and a Start that fails is rolled back through cleanup. A component with no NATS client is refused first.

ctx bounds Start only: the wait for ENTITY_STATES, opening the buckets and seeding the failed map run under it. The follower's watch, the repair lane, the Worker (hop 2) and the coalescer run under a run context start creates and cleanup ends, so ending ctx after Start returned stops nothing (ADR-113).

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop stops the component through the shell, which runs cleanup: Stop before Start does nothing, and a Stop whose context ends first returns that context's error and keeps what is left for the next Stop.

type Config

type Config struct {
	Ports *component.PortConfig `json:"ports" schema:"type:ports,description:Port configuration,category:basic"`
	// EmbedderType is the embedder; at tier 0 only bm25 is accepted, and empty means bm25.
	EmbedderType string `json:"embedder_type" schema:"type:string,description:Embedder type (bm25 only at tier 0),category:basic"`

	// Workers is the number of concurrent embedding worker goroutines; 0 means 5.
	Workers int `` /* 137-byte string literal not displayed */

	// MaxTextLen is the source-text cap, in characters, applied before embedding; text past it
	// is truncated at a word boundary. It is part of the dedup key, so changing it re-embeds
	// the entities it affects. 0 means BM25's 4,000.
	MaxTextLen int `` /* 217-byte string literal not displayed */

	// TextSuffixes are the predicate suffixes whose string values are embedded; empty means
	// defaultTextSuffixes.
	TextSuffixes []string `` /* 191-byte string literal not displayed */

	// Dependency startup configuration
	StartupAttempts int `` /* 130-byte string literal not displayed */
	StartupInterval int `` /* 134-byte string literal not displayed */
	CoalesceMs      int `` /* 141-byte string literal not displayed */
}

Config holds configuration for graph-embedding component. Decoding is strict: a key it does not define, batch_size among them, is refused (design D4).

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a valid default configuration

func (*Config) ApplyDefaults

func (c *Config) ApplyDefaults()

ApplyDefaults sets default values for configuration, resolving each zero value to the value its reader uses. Validate only checks.

func (*Config) Validate

func (c *Config) Validate() error

Validate implements component.Validatable interface

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL