graphclustering

package
v0.1.0-alpha.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 24 Imported by: 0

README

graph-clustering

The community component of the graph subsystem.

Overview

graph-clustering follows the ENTITY_STATES bucket, which graph-ingest writes, and keeps the COMMUNITY_INDEX bucket current. A community here is a group: every entity whose ID shares the same first 4 parts (level 0), first 3 parts (level 1) or first 2 parts (level 2). Groups are derived from IDs, not detected from relationships (ADR-099), so no external provider and no other index is involved. For each group the component keeps one bounded record (counts, keywords, a few sample IDs, never a member list); graph/clustering defines the record, and its README lists the fields.

A revision is the number NATS gives each write to a bucket. The component has two parts:

  1. The feed. graph/follower watches ENTITY_STATES, replays what is stored, then delivers each change with the value it carries. The component hands each value and its revision to the graph/clustering Maintainer, in delivery order, and reports the revision complete at once. A value that does not decode is handed over as a refused entity, which contributes nothing to its groups.
  2. The ticker. Every detection_interval the component asks the Maintainer to write the groups that changed since the last tick. The Maintainer removes the record of a group whose last member was deleted, but only after every write of the tick succeeded.

graph-query's localSearch and the text tier of globalSearch and searchGraph read these records. Until the first tick has published, they refuse transient index_not_ready.

Architecture

ENTITY_STATES ──► graph/follower ──► deliver ──► Maintainer ──► ticker ──► COMMUNITY_INDEX
  (KV watch)       (watch, replay,    (this       (graph/         (every      (changed groups
                    watermark,         package)    clustering)     interval)   only)
                    GRAPH_STATUS)

graph/follower owns the watch, the replay, the watermark and the readiness record it writes to GRAPH_STATUS under the key graph-clustering.

What the readiness key means

The key's indexed_revision counts revisions the Maintainer holds, not revisions published. It reaches a write's revision as soon as the Maintainer has applied it, before the next tick. To know whether a group's record includes a write, read the record's entity_revision, which is exact per group. "Health and readiness" below lists when the key reads degraded.

Configuration

{
  "type": "processor",
  "name": "graph-clustering",
  "enabled": true,
  "config": {
    "ports": {
      "inputs": [
        {"name": "entity_watch", "config": {"kind": "kv-watch", "bucket": "ENTITY_STATES"}}
      ]
    },
    "detection_interval": "30s"
  }
}
Option Type Default Description
ports object the input above Inputs only. A ports block that is present is used as written; one that is absent takes the default
detection_interval string 30s Time between publications, a Go duration such as 30s or 5m. A value that does not parse or is not above zero is refused
startup_attempts int 30 Tries to find ENTITY_STATES at startup; negative is refused
startup_interval_ms int 500 Pause between those tries; negative is refused

The configuration is decoded strictly: a key not listed here is refused at construction, naming it. These keys are refused with a line saying what replaces them: batch_size, enable_llm, enhancement_workers, min_community_size, max_iterations, entity_id_edges, semantic_edges, allow_ungated_reads, enable_anomaly_detection and anomaly_config. The input must be exactly one kv-watch of every ENTITY_STATES key with no history. The component declares no outputs and refuses a configuration that does. Registering the factory asks boot for no model registry. MIGRATION.md lists what a SemStreams configuration changes.

Ports

Direction Name Kind Bucket
input entity_watch kv-watch ENTITY_STATES

The component writes COMMUNITY_INDEX itself and declares no output port for it.

When records are published

The ticker starts a tick only while the follower says the graph state is ready. So nothing is written before the replay of ENTITY_STATES has been applied, and once the follower reports a lost feed (a watch closed, a bucket recreated, a revision out of order) or a latched reset (a stored value that cannot be decoded during the replay), no later tick starts. A tick already running when the loss is detected finishes, so a record published just before the loss was noticed may already be out of date. The first tick runs one detection_interval after Start. A tick that takes longer than the interval delays the next; ticks never overlap.

Health and readiness

Health is Healthy once started, and unhealthy while any of these holds:

  • a stored ENTITY_STATES value could not be decoded during the replay (graph state reset required: <reason>, until the process restarts);
  • the ENTITY_STATES feed is lost (until the process restarts);
  • the last tick left entities out of the records because the Maintainer refused them (reason refused_entity, one count per entity); the other groups are still published, and the state clears on a tick that refuses none;
  • the last tick's publication failed (reason publication_failed, count 1); the changed groups stay pending and the next tick retries them. The state clears on a tick that succeeds.

The readiness record on GRAPH_STATUS carries failed_count and failed_reasons (the two tick reasons above), and reads degraded while either is non-zero or the feed is lost. A failed write returns no refusal count, so while publication fails the last known refused_entity count is kept until a tick reports one again or succeeds. Nothing withholds ticks: the component reports 0 to the follower's withhold. DataFlow counts one input per delivered ENTITY_STATES entry, update or delete.

Metrics

Collectors register on the injected metric.MetricsRegistry, never on Prometheus' global registry. This component registers one, semengine_graph_clustering_publication_failures_total (counter): ticks whose publication to COMMUNITY_INDEX failed. The follower adds its readiness gauges and counters under the same prefix.

Known limits

  • No LLM summaries. A record's summary is a one-line template over its counts. Language-model summaries are tier 2 (#485) and not present.
  • Replay-time reset is permanent. A stored ENTITY_STATES value that cannot be decoded during the replay stops publication until the process restarts.
  • Records update on successful ticks. detection_interval is how often a tick starts, not a bound on how stale a record can be: a slow or failing publication, or a lost feed, delays updates further. A group's record is as current as the last successful tick; its entity_revision says which write it includes.

Dependencies

  • Upstream: graph-ingest writes ENTITY_STATES, which this component follows.
  • Reads: graph/follower (watch, watermark, readiness), graph/clustering (Maintainer, storage, record type), graph/kvcatalog.
  • Writes: COMMUNITY_INDEX, which this component owns, and its readiness record on GRAPH_STATUS through the follower.

Documentation

Overview

Package graphclustering is the graph-clustering component: it follows ENTITY_STATES and keeps COMMUNITY_INDEX current, one bounded record per entity-ID prefix group, through the graph/clustering Maintainer, and reports on the graph-clustering readiness key. A group is every entity that shares the first 4, 3 or 2 parts of its ID. At tier 0 nothing reaches an external provider: communities are derived from IDs, not detected (ADR-099).

Every delivered change is applied to the Maintainer in delivery order. A ticker publishes the changed groups every detection_interval, and only after the follower has applied the whole replay and while its feed is intact.

A composition registers the factory with Register and creates the component through the component manager, with a configuration that Config describes.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateGraphClustering

func CreateGraphClustering(rawConfig json.RawMessage, deps component.Dependencies) (component.Discoverable, error)

CreateGraphClustering is the factory function for graph-clustering components. It opens no connection.

func DeclarePorts

func DeclarePorts(rawConfig json.RawMessage, _ string) (component.PortConfig, error)

DeclarePorts is the component.PortDeclarer for graph-clustering: the ports CreateGraphClustering will report for rawConfig, computed without dependencies.

func Register

func Register(registry *component.Registry) error

Register registers the graph-clustering factory with the component registry. It asks boot for no model registry: at tier 0 the records need no provider.

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component implements the graph-clustering processor.

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow reports input messages: one per delivered ENTITY_STATES entry, update or delete, with the delivered value's length.

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health reports the shell's verdict (component.Lifecycle.Health) over the follower's stuck states: a latched reset, the failures the last tick reported (refused entities, a failed publication), a lost feed.

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize marks the component initialized (no I/O); the factory validated the configuration.

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns the input ports, fixed at construction.

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns no ports: graph-clustering writes COMMUNITY_INDEX itself.

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start begins following ENTITY_STATES into the maintainer. The lifecycle state is the shell's (component.Lifecycle): Start before Initialize, a nil or ended context and a second Start are refused, the second with errs.ErrAlreadyStarted, and a Start that fails is rolled back through cleanup. A component with no NATS client is refused first.

ctx bounds Start only: the wait for ENTITY_STATES and opening COMMUNITY_INDEX run under it. The follower's watch and the ticker, which publishes every detection_interval, run under a run context start creates and cleanup ends, so ending ctx after Start returned stops nothing (ADR-113).

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop stops the component through the shell, which runs cleanup: Stop before Start does nothing, and a Stop whose context ends first returns that context's error and keeps what is left for the next Stop.

type Config

type Config struct {
	Ports *component.PortConfig `json:"ports" schema:"type:ports,description:Port configuration,category:basic"`
	// DetectionInterval is the time between publications of the changed groups, a Go duration
	// such as 30s or 5m; empty means 30s.
	DetectionInterval string `` /* 148-byte string literal not displayed */

	// Dependency startup configuration
	StartupAttempts int `` /* 130-byte string literal not displayed */
	StartupInterval int `` /* 134-byte string literal not displayed */
}

Config holds the graph-clustering configuration. Decoding is strict: a key it does not define is refused, and a removed key is refused with what replaces it (design D3).

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a valid default configuration.

func (*Config) ApplyDefaults

func (c *Config) ApplyDefaults()

ApplyDefaults resolves each zero value to the value its reader uses. Validate only checks.

func (*Config) Validate

func (c *Config) Validate() error

Validate implements component.Validatable.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL