graphindex

package
v0.1.0-alpha.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 30 Imported by: 0

README

graph-index

The relationship index component of the graph subsystem.

Overview

graph-index follows the ENTITY_STATES bucket, which graph-ingest writes, and keeps five index buckets in step with it so that a caller can ask "what does this entity point to", "what points at it", "which entities carry this predicate" or "which entity has this name" without reading every entity. It answers those questions as request/reply verbs on graph.index.query.*.

A revision is the number NATS gives each write to a bucket. A watermark is the highest revision graph-index has applied with nothing below it still waiting. The index is rebuilt from ENTITY_STATES on every start, so after a start it is not ready until it has caught up (see Query refusals).

Architecture

                                       ┌─► OUTGOING_INDEX  (KV)
ENTITY_STATES ──► graph/follower ──►   │─► INCOMING_INDEX  (KV)
   (KV watch)      (the one watch,     │─► ALIAS_INDEX     (KV)
                    the watermark,     │─► PREDICATE_INDEX (KV)
                    GRAPH_STATUS)      └─► NAME_INDEX      (KV)
                                              ▲
graph.index.query.* (request/reply) ──────────┘ read by the eight verbs

graph/follower owns the watch on ENTITY_STATES, the replay, the watermark and the readiness record it writes to GRAPH_STATUS under the key graph-index. graph-index takes each entry from it, merges entries that arrive close together (coalesce_ms), and applies the current stored state of each entity on a pool of lanes that runs entries of one entity in order (internal/dispatch).

Indexes

Index Key Value Answers
OUTGOING_INDEX the entity ID the entity's outgoing relationships what the entity points to
INCOMING_INDEX target.source.hex(predicate) none what points at the target
ALIAS_INDEX hash(alias).entity.hex(predicate) none which entity carries an alias
PREDICATE_INDEX predicate.entity (a 3-part predicate, a 6-part entity ID) none which entities carry a predicate
NAME_INDEX hash(name).entity.hex(predicate) the name as written and its label priority which entities carry a name

NAME_INDEX is graph-index's own and is not a configurable output; the other four are the configured outputs. A membership key encodes who owns it, so an update replaces exactly the rows its entity owns: the layouts are ADR-077 and ADR-078 (the alias layout is ADR-114). An alias is trimmed and case-folded before it is hashed, as a name is, so SN-1 and sn-1 are one alias. Any non-empty string is indexed, including one that is not a valid NATS KV key (for example two words): the key holds the hash, not the alias. The alias predicates are discovered once when graph-index starts; a predicate registered after that is not indexed until the next start.

Query verbs

Every verb is declared in graph.QueryVerbs() (graph/query_verbs.go) with its subject; spell no subject as a literal. All eight are on graph.index.query. followed by the name.

Verb Request Reply data
outgoing {"entity_id": ...} graph.OutgoingRelationshipsData
incoming {"entity_id": ...} graph.IncomingRelationshipsData, sorted by source then predicate
alias {"alias": ...} graph.AliasData: the one live entity carrying the alias, or no ID
predicate {"predicate": ..., "value": ..., "limit": ...} (value and limit optional) graph.PredicateData
predicateList {"namespace": ...} (optional) graph.PredicateListData
predicateStats {"predicate": ..., "sample_limit": ...} graph.PredicateStatsData
predicateCompound {"predicates": [...], "operator": "AND" or "OR", "limit": ...} graph.CompoundPredicateData
byName {"name": ..., "limit": ...} graph.NameData

A successful reply is a graph.QueryResponse:

{"data": {}, "indexed_revision": 1234, "producer": "graph-index", "timestamp": "2026-10-27T12:00:00Z"}

indexed_revision is the watermark read before the handler's first bucket read, so every ENTITY_STATES write at or below it is reflected in data. A caller holding the kv_revision of its own write compares it with this number to know whether the answer includes that write, for an alias reply too. A refusal is a classified error and carries no envelope.

Reading at a revision

Every verb accepts an optional min_revision, the kv_revision of a write the answer must include (graph.MinRevisionField, a number; 0 or absent means any). If the indexed revision is below it, the verb refuses with transient revision_not_reached (graph.ErrorCodeRevisionNotReached), the revision reached in the detail. The verb does not wait; the caller retries. In graph.QueryVerbs() all eight verbs are marked Enveloped, the mark that says a reply is a graph.QueryResponse; graph-query's byName, predicate, entityByAlias and relationships pass the field through.

A malformed request is refused as invalid_request before any readiness check, so a bad request fails the same way during a rebuild.

Query refusals

Every one of the eight verbs refuses:

  • with the transient code index_not_ready during the startup rebuild, until the watermark first reaches the ENTITY_STATES revision that was newest when the rebuild's snapshot ended. Once reached, the index stays ready; a later lag does not refuse queries;
  • with the transient code index_not_ready for as long as the follower has lost its feed (below), and while index writes that failed are unresolved;
  • with the fatal code graph_state_reset_required after a stored ENTITY_STATES value could not be decoded (ADR-074). The state holds until the process restarts; the stored state has to be wiped and rebuilt first.

A caller treats index_not_ready as retry later and graph_state_reset_required as do not retry.

Alias lookups

The alias verb answers the one live entity that carries the alias, under any number of alias predicates. If no entity carries it, the reply holds no ID. If two or more entities do, the verb refuses with class invalid and code alias_ambiguous (graph.ErrorCodeAliasAmbiguous), the message naming the alias; it answers again once all but one holder has dropped the alias or been deleted and graph-index has indexed that change. An alias that is empty after trimming is refused as invalid_request. A stored ALIAS_INDEX row that is not in the layout above is left out of the answer, logged, and counted in query_rows_skipped_total under alias.

Configuration

{
  "type": "processor",
  "name": "graph-index",
  "enabled": true,
  "config": {
    "ports": {
      "inputs": [
        {"name": "entity_watch", "config": {"kind": "kv-watch", "bucket": "ENTITY_STATES"}}
      ],
      "outputs": [
        {"name": "outgoing_index", "config": {"kind": "kv-write", "bucket": "OUTGOING_INDEX"}},
        {"name": "incoming_index", "config": {"kind": "kv-write", "bucket": "INCOMING_INDEX"}},
        {"name": "alias_index", "config": {"kind": "kv-write", "bucket": "ALIAS_INDEX"}},
        {"name": "predicate_index", "config": {"kind": "kv-write", "bucket": "PREDICATE_INDEX"}}
      ]
    },
    "workers": 1,
    "coalesce_ms": 0
  }
}
Option Type Default Description
ports object the ports above Inputs and outputs. A ports block that is present is used as written; one that is absent takes the defaults
workers int 1 Index lanes, 1 to 16. 0 means the default, one lane
coalesce_ms int 0 Window in milliseconds for merging updates to one entity; 0 applies each at once
startup_attempts int 30 Tries to find ENTITY_STATES at startup
startup_interval_ms int 500 Pause between those tries

The configuration is decoded strictly: a key not listed here is refused at construction, naming it. This includes batch_size, which the SemStreams version accepted and nothing read; a configuration that still sets it fails at boot until the line is deleted. The one input must be a kv-watch of every ENTITY_STATES key with no history, and the four outputs must be exactly the four index buckets above; an input or output that is anything else is refused.

Ports

Direction Name Kind Bucket
input entity_watch kv-watch ENTITY_STATES
output outgoing_index kv-write OUTGOING_INDEX
output incoming_index kv-write INCOMING_INDEX
output alias_index kv-write ALIAS_INDEX
output predicate_index kv-write PREDICATE_INDEX

Health and readiness

Health is Healthy once started, and unhealthy while any of these holds: a stored value could not be decoded (graph_state_reset_required), index writes are unresolved, or the ENTITY_STATES feed is lost. The error count is reported and never lowers Healthy by itself (ADR-109). DataFlow counts one input per delivered ENTITY_STATES entry, update or delete, before merging.

The follower publishes the readiness record to GRAPH_STATUS at once and then every heartbeat, with indexed_revision, target_revision and failed_count. The state reads degraded when index writes are unresolved, the feed is lost, or the follower has been behind for 30 seconds with no completed write. Any completed write counts, so one entity held while others finish reads not ready, not degraded (owner ruling (a1), PR #550 comment 6721867279).

Lost feed

The follower declares the feed lost when the watch cannot start, when it closes while the component is running, or when ENTITY_STATES' backing stream is a different stream from the one it bound (the bucket was deleted and created again). Nothing rebinds the watch: the state holds until the process restarts. How long a watch stalls after a broker restart (about 20 s in one probe of nats.go v1.54.0, PR #414 comment 6425769680) and when a deleted bucket closes it (about 18 s, comment 6426443564) are that library's timings and are not promised here. A bucket deleted and created again within one stall of the watch is caught at the next heartbeat; a stream restored with the same creation time is not.

Metrics

Collectors register on the injected metric.MetricsRegistry, never on Prometheus' global registry. Names are semengine_graph_index_<name>:

Name Type Meaning
events_processed_total counter ENTITY_STATES entries processed
updates_total counter vec index updates by index_type; alias counts once per entity indexed or deleted
kv_operations_total counter vec KV operations by operation and kv_bucket
watch_events_total counter vec watch events by event_type
write_failures_total counter entities whose required index writes failed after retry
reindex_events_total counter vec re-indexes by result (changed or unchanged)
reconcile_operations_total counter vec owner reconciliation operations by index_type, operation, result
query_rows_skipped_total counter vec stored index rows a query left out, by index_type

The follower adds readiness, lag, bootstrap_complete, readiness_state, status_publish_failures_total, indexed_revision and target_revision under the same prefix, and the lane pool its dispatch_* collectors. A second graph-index on one registry writes the collectors the first registered.

Known limits

  • Overwritten revisions. ENTITY_STATES keeps one revision per key, so a revision overwritten before the watch reads it is never delivered. The watermark tolerates this.
  • Rebuild on every start. The index is rebuilt from ENTITY_STATES at every start, and queries are refused for the length of the rebuild. Issue #108 holds the decision on resuming instead.
  • Memory. One string per entity is kept to count reindex_events_total.

Dependencies

  • Upstream: graph-ingest writes ENTITY_STATES, which this component follows.
  • Reads: graph/follower (watch, watermark, readiness), graph/kvcatalog (the index buckets), internal/dispatch (the lane pool).
  • Consumers read through graph.QueryVerbs() and the graph.QueryResponse envelope; none reads the index buckets directly.

Documentation

Overview

Package graphindex provides graph-index, the component that keeps the graph's relationship indexes: it follows the ENTITY_STATES bucket through graph/follower, writes five index buckets it owns (OUTGOING_INDEX, INCOMING_INDEX, ALIAS_INDEX, PREDICATE_INDEX and NAME_INDEX), and answers the eight graph.index.query.* verbs declared for it in graph.QueryVerbs. Each answer is a graph.QueryResponse carrying the revision it reflects (indexed_revision) and the producer that gave it.

Until the index has rebuilt from ENTITY_STATES after a start, and for as long as it has lost its feed, every verb refuses with the transient error code index_not_ready; after a stored entity that cannot be decoded, it refuses with the fatal graph_state_reset_required.

An alias lookup answers the one live entity carrying the alias (trimmed and case-folded, as names are), answers no ID when none does, and refuses with class invalid and code alias_ambiguous when two or more do. A dropped or changed alias, or a deleted entity's alias, stops resolving once the revision is indexed. The alias predicates are read once, when graph-index starts; a predicate registered later is not indexed until the next start.

The package README.md describes its ports, configuration, query verbs, metrics and health.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateGraphIndex

func CreateGraphIndex(rawConfig json.RawMessage, deps component.Dependencies) (component.Discoverable, error)

CreateGraphIndex is the factory function for creating graph-index components. It registers the component's collectors on deps.MetricsRegistry and opens no connection.

func DeclarePorts

func DeclarePorts(rawConfig json.RawMessage, _ string) (component.PortConfig, error)

DeclarePorts is the component.PortDeclarer for graph-index: the ports CreateGraphIndex will report for rawConfig, computed without dependencies.

func Register

func Register(registry *component.Registry) error

Register registers the graph-index factory with the component registry.

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component implements the graph-index processor.

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow reports input messages: one per delivered ENTITY_STATES entry, update or delete, with the delivered value's length, counted before coalescing (design D4). ErrorRate is the errors counted (Health's ErrorCount) per input.

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health reports the shell's verdict (component.Lifecycle.Health) over the follower's stuck states: a latched reset, unresolved index writes, a lost feed. ErrorCount is reported and never lowers Healthy (ADR-109 Decision 2), so a write failure lowers it only until its repair.

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize marks the component initialized (no I/O); the factory validated the configuration.

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns the input ports, fixed at construction.

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns the output ports, fixed at construction.

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start begins following ENTITY_STATES and indexing it. The lifecycle state is the shell's (component.Lifecycle): Start before Initialize, a nil or ended context and a second Start are refused, the second with errs.ErrAlreadyStarted, and a Start that fails is rolled back through cleanup. A component with no NATS client is refused first.

ctx bounds Start only: the wait for ENTITY_STATES and opening the buckets run under it, and a ctx that ends first fails Start, which then holds nothing. The follower's watch, the keyed pool, the coalescer and the repair loop run under a run context start creates and cleanup ends, so ending ctx after Start returned stops nothing (ADR-113).

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop stops the component through the shell, which runs cleanup: Stop before Start does nothing, and a Stop whose context ends first returns that context's error and keeps what is left for the next Stop.

type Config

type Config struct {
	Ports *component.PortConfig `json:"ports" schema:"type:ports,description:Port configuration,category:basic"`
	// Workers is the index lane count, 1 to 16; 0 means the default, 1 (semsource sends 0
	// to mean "use the default").
	Workers int `json:"workers" schema:"type:int,description:Index lane count (1-16; 0 = default 1),category:advanced"`

	// Dependency startup configuration
	StartupAttempts int `` /* 130-byte string literal not displayed */
	StartupInterval int `` /* 134-byte string literal not displayed */
	CoalesceMs      int `` /* 141-byte string literal not displayed */
}

Config holds configuration for graph-index component. Decoding is strict: a key it does not define, batch_size among them, is refused (design D9).

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a valid default configuration

func (*Config) ApplyDefaults

func (c *Config) ApplyDefaults()

ApplyDefaults sets default values for configuration. Validate only checks.

func (*Config) Validate

func (c *Config) Validate() error

Validate implements component.Validatable interface

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL