graphingest

package
v0.1.0-alpha.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 36 Imported by: 0

README

graph-ingest

Entity and triple ingestion component for the graph subsystem.

Overview

The graph-ingest component is the entry point for entity data flowing into the graph. It consumes entity messages from JetStream, serves the typed graph-mutation operations over NATS request/reply, and stores every entity in the ENTITY_STATES KV bucket, which it owns. It also answers the authoritative entity reads declared for it in the verb table in graph (graph.QueryVerbs).

Architecture

entity.>  (JetStream) ──┐
                        │  ┌──────────────┐
graph.mutation.> ───────┼─►│ graph-ingest ├──► ENTITY_STATES (KV)
  (request/reply)       │  └──────┬───────┘
                        │         └── answers graph.ingest.query.{entity,batch,prefix}

Features

  • Entity writes: create, merge, reconcile and delete entities.
  • Triple mutations: append triples to an entity, and replace one source's triples of named predicates (reconcile).
  • Hierarchy inference: optionally creates container entities from the 6-part entity ID structure.
  • At-least-once delivery: JetStream consumption with explicit acknowledgment; a redelivery the applied-sequence guard has already seen is dropped and counted. The guard keeps, for each entity and stream, the stream's generation and the last sequence applied from it in GRAPH_INGEST_APPLIED_SEQ, as sixteen bytes. The generation is the stream's creation time on the server, which Start reads before it binds the stream's consumer; every input that consumer delivers carries it. An input from a generation other than the stored one is not dropped because of the stored sequence. In some cases an input from a stream that was deleted and created again, or restored, still carries the stored generation, and is then dropped and counted as a redelivery when its sequence is not newer; the graph-ingest-recovery spec lists those cases under "Replay protection uses the generation read before the bind". A stored value of any other length, including the eight-byte sequence SemStreams wrote, cannot be decoded, so the input is neither applied nor acknowledged: it is delivered again, each refusal is counted, and the key is logged once. Deleting the key repairs it, and the next delivery is applied as first seen.

Configuration

{
  "type": "processor",
  "name": "graph-ingest",
  "enabled": true,
  "config": {
    "ports": {
      "inputs": [
        {
          "name": "graph_mutations",
          "required": true,
          "config": {
            "kind": "nats-request",
            "subject": "graph.mutation.>",
            "interface": {"type": "semengine.graph.mutation", "version": "v1"}
          }
        },
        {
          "name": "entity_stream",
          "config": {"kind": "jetstream", "stream_name": "ENTITY", "subjects": ["entity.>"]}
        }
      ],
      "outputs": [
        {
          "name": "entity_states",
          "config": {"kind": "kv-write", "bucket": "ENTITY_STATES"}
        }
      ]
    },
    "enable_hierarchy": true
  }
}
Configuration options
Option Type Default Description
ports object the default ports Input and output ports. Without the key, the component takes the default ports: entity_stream (JetStream ENTITY, entity.>), mutations (the request port below) and entity_states. A ports block that is present is the whole port set, used as written and never merged with the defaults. Exactly one input must be the required nats-request port on graph.mutation.> with the semengine.graph.mutation v1 interface; any other input with that interface type, or with a NATS subject starting with graph.mutation. (a nats or jetstream port included), is refused. At least one output is required. Every output must be a kv-write port on ENTITY_STATES, the only place graph-ingest writes; any other output is refused when the component is built.
enable_hierarchy bool false Create hierarchy container entities and edges (below).
ingest_lanes int 8 Number of ingest lanes. Messages for one entity ID always use the same lane, so they apply in arrival order; different entities apply in parallel. 1 is fully serial. 0 counts as unset and takes the default of 8; a negative value is clamped to 1.

A configuration key not in this table is refused when the component is built, with an error naming the key.

The component is also refused when its dependencies carry no NATS client, no payload registry, or no deployment authority (platform.org and platform.id). The registry is the type authority every create is checked against, so graph-ingest is never built without one, and no write path checks for a missing registry (O-15).

Ports

Inputs
Name Kind Subject Description
mutations nats-request graph.mutation.> The typed mutation operations (create, reconcile, append, delete)
entity_stream jetstream entity.> Entity messages to merge into ENTITY_STATES
Outputs
Name Kind Bucket Description
entity_states kv-write ENTITY_STATES Entity state storage

Request/reply verbs

graph-ingest subscribes to exactly the verbs graph.QueryVerbs declares with responder graph-ingest, and to no other query subject. Replies are bare JSON, not wrapped in an envelope.

Verb Subject Reply
entity graph.ingest.query.entity graph.ExactEntity
batch graph.ingest.query.batch graph.EntityBatchResponse
prefix graph.ingest.query.prefix graph.PrefixQueryResponse

graph.ingest.query.prefix accepts graph.PrefixQueryRequest, which filters keys either by a 1–6 part literal prefix (Prefix) or by an exact 6-position * wildcard pattern (Pattern, validated by pkg/types.ValidateEntityIDPattern and evaluated via KVStore.KeysByFilter). prefix and pattern are mutually exclusive; omitting both matches all entities. Its reply's total is the length of the key listing, before the cursor and any limit, on every path; it is true as listed, not a snapshot.

Hierarchy inference

When enable_hierarchy is on, the component creates container entities from the 6-part entity ID structure:

org.platform.system.domain.type.instance
 │      │       │      │     │      │
 └──────┴───────┴──────┴─────┴──────┴─► Real entity
        │       │      │     │
        └───────┴──────┴─────┴─► Type container (5-part prefix + .group)
                │      │
                └──────┴─► Taxonomy container (4-part prefix, source × domain, + .group.container)
                       │
                       └─► Source container (3-part prefix + .group.container.level)

The padding tokens group, container and level are reserved instance values (pkg/types.IsReservedInstanceToken). Containers are minted only for entities under the deployment's own authority; an imported entity gets none. The payload registry must hold the hierarchy container type, or the component is refused at construction.

Dependencies

  • Upstream: none; graph-ingest is the entry point.
  • Downstream: components that watch ENTITY_STATES. None is ported to SemEngine yet.

Metrics

Every collector is registered on the MetricsRegistry in the component's dependencies; with none, the collectors are registered nowhere, and nothing is registered on Prometheus' global registry.

Give each graph-ingest its own registry. A registry holds one instance's series: gauges such as the readiness gauges are set from that instance's own state. A second graph-ingest on the same registry shares each collector, so the two overwrite each other's gauges, and which instance's value a scrape shows is not promised. No configuration we know of runs two on one registry.

Metric Type Labels
semengine_datamanager_entities_updated_total counter
semengine_graph_ingest_indexing_profile_default_total counter message_type
semengine_graph_ingest_mutation_rejections_total counter subject, reason
semengine_graph_ingest_graph_mutation_outcomes_total counter operation, outcome
semengine_graph_ingest_batch_query_missing_total counter reason
semengine_graph_ingest_predicate_contract_rejections_total counter lane, reason
semengine_graph_ingest_entity_state_contract_rejections_total counter lane, field, reason
semengine_graph_ingest_processing_duration_seconds histogram
semengine_graph_ingest_ingest_lag_seconds histogram
semengine_graph_ingest_redeliveries_dropped_total counter
semengine_graph_ingest_guard_record_refusals_total counter
semengine_graph_ingest_unguarded_inputs_total counter
semengine_graph_ingest_work_interrupted_total counter cause
semengine_graph_ingest_cas_retries_total counter
semengine_graph_ingest_stale_sets_total counter
semengine_graph_ingest_duplicate_triples_suppressed_total counter lane
semengine_graph_ingest_poisoned_entities gauge

The readiness gauges of graph/readiness (readiness, lag, bootstrap_complete, readiness_state and status_publish_failures_total, under the graph_ingest subsystem) and the ingest pool's dispatch_* collectors are registered on the same registry.

Health

Health() reports stuck states, not past errors (ADR-109 Decision 2). The component is healthy when it is running, no stored entity fails to decode (the poison inventory is empty), and the latest readiness status tick read the backlog of every consumer it bound. While either of the last two fails, Healthy is false, Status is degraded, and LastError names each: a sample of the poisoned entities, or the consumers whose backlog could not be read. Both clear without a restart: a repair or a valid write clears the poison record, and the next tick that reads every backlog clears the other. A lost durable consumer reads unhealthy this way; graph-ingest does not bind a new one (#216).

A third stuck state does not clear. If a status tick, run while the component is running and before Stop begins, sees that a consume loop (the goroutine that pulls messages for one consumer) has ended, graph-ingest records the binding as lost. Healthy is then false with LastError reading input binding lost for consumer <stream>/<consumer>: its consume loop ended; restart the process to consume again, the readiness envelope is degraded, and graph-ingest logs an Error once per consumer. This is what a stream deleted and created again while the broker stays up leaves behind. Creating the consumer again under the same name does not clear it: graph-ingest never rebinds, stops or deletes that consumer, and the process restart is the recovery. Restart on /readyz failing, not /healthz, which stays 200 while the listener serves; after the restart, publish again from the source. After a broker restart with a memory stream, whether the consume loop ends is nats.go behaviour that was measured once and is not promised: in the probe (openspec/changes/archive/2026-10-27-fix-216-consumer-rebind/inventory.md section 1.2, case a) the loop stayed open and pulled a consumer that no longer existed, so the unhealthy reading there comes from the failed backlog lookup, not from an ended loop. A file stream resumes without action in the same probe (case c).

ErrorCount counts processing errors since start and never lowers Healthy.

DataFlow() counts input messages since start: each delivery on a stream input, including a redelivery and a message graph-ingest refuses, and each mutation request (graph.mutation.*) that reaches its handler. BytesPerSecond counts those messages' payload sizes. Queries, CreateEntity calls made inside the process, and the containers hierarchy inference creates are not input messages and are not counted. ErrorRate is ErrorCount divided by the input messages counted; an append counts an error for each subject it cannot write, an absent one included, so the rate can exceed 1.

semengine_datamanager_entities_updated_total counts every write that changed an entry: a create, a reconcile that applied, a stream merge, a container born by hierarchy inference, and each subject an append applied. A delete is not counted; deletes show on graph_mutation_outcomes_total{operation="entity.delete"}.

Running implies the boot sweep of ENTITY_STATES completed: Start fails when the sweep cannot reach the end of the snapshot. So the readiness envelope's bootstrap_complete is true from the first one graph-ingest publishes, whatever backlog stands; ready and lag report the backlog.

Documentation

Overview

Package graphingest provides the graph-ingest component for entity and triple ingestion.

Package graphingest provides graph-ingest, the component that writes entities and their triples into the ENTITY_STATES KV bucket, which it owns: it consumes entity messages from JetStream, serves the graph-mutation operations over NATS request/reply, and answers the entity reads declared for it in graph.QueryVerbs. The package README.md describes its ports, configuration, metrics and health.

Package graphingest query handlers

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateGraphIngest

func CreateGraphIngest(rawConfig json.RawMessage, deps component.Dependencies) (component.Discoverable, error)

CreateGraphIngest is the factory function for creating graph-ingest components

func DeclarePorts

func DeclarePorts(rawConfig json.RawMessage, _ string) (component.PortConfig, error)

DeclarePorts is the component.PortDeclarer for graph-ingest: the ports CreateGraphIngest will report for rawConfig, computed without dependencies.

func Register

func Register(registry *component.Registry) error

Register registers the graph-ingest factory with the component registry

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component implements the graph-ingest processor

func (*Component) CreateEntity

func (c *Component) CreateEntity(ctx context.Context, entity *graph.EntityState) error

CreateEntity atomically creates a new entity. Existing keys are never overwritten; callers observe natsclient.ErrKVKeyExists and decide whether that conflict is acceptable for their operation. A nil context is refused as invalid before anything is read or written.

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow reports input messages: each delivery on a stream input, redeliveries included, and each mutation request. Queries, in-process creates and the containers hierarchy inference creates are not input messages. ErrorRate is the processing errors counted (Health's ErrorCount) per input message.

func (*Component) DebugStatus

func (c *Component) DebugStatus() any

DebugStatus implements component.DebugStatusProvider: the FULL poison inventory, sorted by entity ID, so a mass-poison event is enumerable in-band (design D6 — the Health message and gauge are bounded; this is not).

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health reports the shell's verdict (component.Lifecycle.Health) over graph-ingest's stuck states, not a count of past errors (ADR-109 Decision 2): a poisoned entity, a backlog the latest status tick could not read, or a lost input binding, which only a process restart clears (#216, observeLostBinding). ErrorCount is reported and never lowers Healthy. A non-empty poison inventory degrades Health per-entity: status is "degraded", NOT "reset_required" (that stays a per-read error code, design D6), and the message carries count + a bounded ID/reason sample. Full enumeration rides DebugStatus.

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize marks the component initialized (no I/O). The factory validated the configuration.

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns input port definitions. Reads directly from config so ports are available before Initialize().

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns output port definitions. Reads directly from config so ports are available before Initialize().

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start begins processing (must be initialized first). The lifecycle state is the shell's (component.Lifecycle, over internal/lifecycleguard): a nil or ended context and a second Start are refused before anything is acquired; a Start that fails is rolled back through release, and a rollback that fails leaves release pending for the next Stop. A component with no NATS client, or an uninitialized one, is refused before the guard is consulted, so a later Start after Initialize is still admitted.

ctx bounds Start only. Everything Start does before it returns runs under ctx: connecting to NATS (or waiting while the client reconnects), bucket setup, the ENTITY_STATES sweep and consumer setup. A ctx that ends first fails Start. Subscriptions, the ingest pool, the entity cache's expiry sweep and the status loop run under a run context start creates and release ends, so ending ctx after Start returned stops nothing (ADR-113). Request handlers run under a context natsclient owns; release drains their subscriptions, which waits for running handlers.

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop gracefully shuts down the component through the shell, which runs release, the cleanup Start gave it: Stop before Start releases nothing; Stop during Start waits for it, or returns ctx's error first; a Stop whose release fails keeps it pending, and the next Stop runs it again; after a release that succeeded, Stop returns nil and runs nothing. Stop logs nothing itself: release logs "component stopped gracefully" when it succeeds.

type Config

type Config struct {
	Ports           *component.PortConfig `json:"ports" schema:"type:ports,description:Port configuration,category:basic"`
	EnableHierarchy bool                  `json:"enable_hierarchy" schema:"type:bool,description:Enable hierarchy inference,default:false,category:advanced"`
	// IngestLanes is the number of keyed-concurrent ingest lanes (SemStreams ADR-072,
	// SemStreams gh#480). Messages are partitioned by entity ID (same entity → one lane →
	// serial in arrival order, preserving the arrival-order merge; different
	// entities → parallel), so ingest is no longer bound to a single serial
	// Get+CAS round-trip chain. Default 8 (concurrent); `1` selects the prior
	// fully-serial behavior.
	IngestLanes int `` /* 166-byte string literal not displayed */
}

Config holds configuration for graph-ingest component

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a valid default configuration

func (*Config) ApplyDefaults

func (c *Config) ApplyDefaults()

ApplyDefaults sets default values for configuration. Validate only checks (ruling K, #91 comment 6062681355).

func (*Config) Validate

func (c *Config) Validate() error

Validate implements component.Validatable interface

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL