graphingest

package
v0.1.0-alpha.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 15, 2026 License: MIT Imports: 35 Imported by: 0

README

graph-ingest

Entity and triple ingestion component for the graph subsystem.

Overview

The graph-ingest component is the entry point for entity data flowing into the graph. It consumes entity messages from JetStream, serves the typed graph-mutation operations over NATS request/reply, and stores every entity in the ENTITY_STATES KV bucket, which it owns. It also answers the authoritative entity reads declared for it in the verb table in graph (graph.QueryVerbs).

Architecture

entity.>  (JetStream) ──┐
                        │  ┌──────────────┐
graph.mutation.> ───────┼─►│ graph-ingest ├──► ENTITY_STATES (KV)
  (request/reply)       │  └──────┬───────┘
                        │         └── answers graph.ingest.query.{entity,batch,prefix}

Features

  • Entity writes: create, merge, reconcile and delete entities.
  • Triple mutations: append triples to an entity, and replace one source's triples of named predicates (reconcile).
  • Hierarchy inference: optionally creates container entities from the 6-part entity ID structure.
  • At-least-once delivery: JetStream consumption with explicit acknowledgment; a redelivery the applied-sequence guard has already seen is dropped and counted. The guard keeps, for each entity and stream, the stream's generation and the last sequence applied from it in GRAPH_INGEST_APPLIED_SEQ, as sixteen bytes. The generation is the stream's creation time on the server, which Start reads before it binds the stream's consumer; every input that consumer delivers carries it. An input from a generation other than the stored one is not dropped because of the stored sequence. In some cases an input from a stream that was deleted and created again, or restored, still carries the stored generation, and is then dropped and counted as a redelivery when its sequence is not newer; the graph-ingest-recovery spec lists those cases under "Replay protection uses the generation read before the bind". A stored value of any other length, including the eight-byte sequence SemStreams wrote, cannot be decoded, so the input is neither applied nor acknowledged: it is delivered again, each refusal is counted, and the key is logged once. Deleting the key repairs it, and the next delivery is applied as first seen.

Configuration

{
  "type": "processor",
  "name": "graph-ingest",
  "enabled": true,
  "config": {
    "ports": {
      "inputs": [
        {
          "name": "graph_mutations",
          "required": true,
          "config": {
            "kind": "nats-request",
            "subject": "graph.mutation.>",
            "interface": {"type": "semengine.graph.mutation", "version": "v1"}
          }
        },
        {
          "name": "entity_stream",
          "config": {"kind": "jetstream", "stream_name": "ENTITY", "subjects": ["entity.>"]}
        }
      ],
      "outputs": [
        {
          "name": "entity_states",
          "config": {"kind": "kv-write", "bucket": "ENTITY_STATES"}
        }
      ]
    },
    "enable_hierarchy": true
  }
}
Configuration options
Option Type Default Description
ports object required Input and output ports. Exactly one input must be the required nats-request port on graph.mutation.> with the semengine.graph.mutation v1 interface, and at least one output is required. Every output must be a kv-write port on ENTITY_STATES, the only place graph-ingest writes; any other output is refused when the component is built.
enable_hierarchy bool false Create hierarchy container entities and edges (below).
ingest_lanes int 8 Number of ingest lanes. Messages for one entity ID always use the same lane, so they apply in arrival order; different entities apply in parallel. 1 is fully serial. 0 counts as unset and takes the default of 8; a negative value is clamped to 1.

A configuration key not in this table is refused when the component is built, with an error naming the key.

Ports

Inputs
Name Kind Subject Description
graph_mutations nats-request graph.mutation.> The typed mutation operations (create, reconcile, append, delete)
entity_stream jetstream entity.> Entity messages to merge into ENTITY_STATES
Outputs
Name Kind Bucket Description
entity_states kv-write ENTITY_STATES Entity state storage

Request/reply verbs

graph-ingest subscribes to exactly the verbs graph.QueryVerbs declares with responder graph-ingest, and to no other query subject. Replies are bare JSON, not wrapped in an envelope.

Verb Subject Reply
entity graph.ingest.query.entity graph.ExactEntity
batch graph.ingest.query.batch graph.EntityBatchResponse
prefix graph.ingest.query.prefix graph.PrefixQueryResponse

graph.ingest.query.prefix accepts graph.PrefixQueryRequest, which filters keys either by a 1–6 part literal prefix (Prefix) or by an exact 6-position * wildcard pattern (Pattern, validated by pkg/types.ValidateEntityIDPattern and evaluated via KVStore.KeysByFilter). prefix and pattern are mutually exclusive; omitting both matches all entities.

Hierarchy inference

When enable_hierarchy is on, the component creates container entities from the 6-part entity ID structure:

org.platform.system.domain.type.instance
 │      │       │      │     │      │
 └──────┴───────┴──────┴─────┴──────┴─► Real entity
        │       │      │     │
        └───────┴──────┴─────┴─► Type container (5-part prefix + .group)
                │      │
                └──────┴─► Taxonomy container (4-part prefix, source × domain, + .group.container)
                       │
                       └─► Source container (3-part prefix + .group.container.level)

The padding tokens group, container and level are reserved instance values (pkg/types.IsReservedInstanceToken). Containers are minted only for entities under the deployment's own authority; an imported entity gets none. The payload registry must hold the hierarchy container type, or the component is refused at construction.

Dependencies

  • Upstream: none; graph-ingest is the entry point.
  • Downstream: components that watch ENTITY_STATES. None is ported to SemEngine yet.

Metrics

Every collector is registered on the MetricsRegistry in the component's dependencies; with none, the collectors are registered nowhere, and nothing is registered on Prometheus' global registry. Two components on one registry share each collector.

Metric Type Labels
semengine_datamanager_entities_updated_total counter
semengine_graph_ingest_indexing_profile_default_total counter message_type
semengine_graph_ingest_mutation_rejections_total counter subject, reason
semengine_graph_ingest_graph_mutation_outcomes_total counter operation, outcome
semengine_graph_ingest_batch_query_missing_total counter reason
semengine_graph_ingest_predicate_contract_rejections_total counter lane, reason
semengine_graph_ingest_entity_state_contract_rejections_total counter lane, field, reason
semengine_graph_ingest_processing_duration_seconds histogram
semengine_graph_ingest_ingest_lag_seconds histogram
semengine_graph_ingest_redeliveries_dropped_total counter
semengine_graph_ingest_guard_record_refusals_total counter
semengine_graph_ingest_unguarded_inputs_total counter
semengine_graph_ingest_work_interrupted_total counter cause
semengine_graph_ingest_cas_retries_total counter
semengine_graph_ingest_stale_sets_total counter
semengine_graph_ingest_duplicate_triples_suppressed_total counter lane
semengine_graph_ingest_poisoned_entities gauge

The readiness gauges of graph/readiness (readiness, lag, bootstrap_complete, readiness_state and status_publish_failures_total, under the graph_ingest subsystem) and the ingest pool's dispatch_* collectors are registered on the same registry.

Health

The component reports healthy when it is running, no processing error has been counted, and no stored entity fails to decode (the poison inventory is empty). Running implies the boot sweep of ENTITY_STATES completed: Start fails when the sweep cannot reach the end of the snapshot.

Documentation

Overview

Package graphingest provides the graph-ingest component for entity and triple ingestion.

Package graphingest provides the graph-ingest component for entity and triple ingestion.

Overview

The graph-ingest component is responsible for ingesting entities and triples into the graph system. It subscribes to JetStream subjects for incoming entity data and stores them in the ENTITY_STATES KV bucket.

Tier

Tier: ALL TIERS (Tier 0, 1, 2) - Required for all deployments.

Architecture

graph-ingest is a core component required for all deployment tiers (Structural, Statistical, Semantic). It serves as the entry point for all entity data flowing into the graph subsystem.

                    ┌─────────────────┐
objectstore.stored ─┤                 │
                    │  graph-ingest   ├──► ENTITY_STATES (KV)
sensor.processed   ─┤                 │
                    └─────────────────┘

Features

  • Entity CRUD operations (create, read, update, delete)
  • Triple mutations (add, remove)
  • Hierarchy inference (optional) - creates container entities based on 6-part entity ID structure
  • JetStream subscription with at-least-once delivery semantics
  • KV storage with atomic updates

Configuration

The component is configured via JSON with the following structure:

{
  "ports": {
    "inputs": [
      {"name":"objectstore_in","config":{"kind":"jetstream","subjects":["objectstore.stored.entity"]}},
      {"name":"sensor_in","config":{"kind":"jetstream","subjects":["sensor.processed.entity"]}}
    ],
    "outputs": [
      {"name":"entity_states","config":{"kind":"kv-write","bucket":"ENTITY_STATES"}}
    ]
  },
  "enable_hierarchy": true
}

Port Definitions

Inputs:

  • JetStream subscriptions for entity events (objectstore.stored.entity, sensor.processed.entity)

Outputs:

  • KV bucket: ENTITY_STATES - stores entity state with triples

Hierarchy Inference

When enable_hierarchy is true, the component automatically creates container entities based on the 6-part entity ID structure (org.platform.system.domain.type.instance). This creates edges for:

  • Type containers (*.group)
  • System containers (*.container)
  • Domain containers (*.level)

Usage

Register the component with the component registry:

import graphingest "github.com/c360studio/semengine/processor/graph-ingest"

func init() {
    graphingest.Register(registry)
}

Dependencies

This component has no upstream graph component dependencies. It is the entry point for entity data and other graph components (graph-index, graph-embedding, etc.) watch its output KV bucket.

Package graphingest query handlers

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateGraphIngest

func CreateGraphIngest(rawConfig json.RawMessage, deps component.Dependencies) (component.Discoverable, error)

CreateGraphIngest is the factory function for creating graph-ingest components

func DeclarePorts

func DeclarePorts(rawConfig json.RawMessage, _ string) (component.PortConfig, error)

DeclarePorts is the component.PortDeclarer for graph-ingest: the ports CreateGraphIngest will report for rawConfig, computed without dependencies.

func Register

func Register(registry *component.Registry) error

Register registers the graph-ingest factory with the component registry

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component implements the graph-ingest processor

func (*Component) ConfigSchema

func (c *Component) ConfigSchema() component.ConfigSchema

ConfigSchema returns the configuration schema

func (*Component) CreateEntity

func (c *Component) CreateEntity(ctx context.Context, entity *graph.EntityState) error

CreateEntity atomically creates a new entity. Existing keys are never overwritten; callers observe natsclient.ErrKVKeyExists and decide whether that conflict is acceptable for their operation. A nil context is refused as invalid before anything is read or written.

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow returns current data flow metrics

func (*Component) DebugStatus

func (c *Component) DebugStatus() any

DebugStatus implements component.DebugStatusProvider: the FULL poison inventory, sorted by entity ID, so a mass-poison event is enumerable in-band (design D6 — the Health message and gauge are bounded; this is not).

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health returns current health status

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize validates configuration and sets up ports (no I/O)

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns input port definitions. Reads directly from config so ports are available before Initialize().

func (*Component) Meta

func (c *Component) Meta() component.Metadata

Meta returns component metadata

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns output port definitions. Reads directly from config so ports are available before Initialize().

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start begins processing (must be initialized first). The lifecycle state is the guard's (internal/lifecycleguard, design D13): a nil or ended context and a second Start are refused before anything is acquired; a Start that fails is rolled back through release, and a rollback that fails leaves release pending for the next Stop. An uninitialized component, or one with no NATS client, is refused before the guard is consulted, so a later Start after Initialize is still admitted.

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop gracefully shuts down the component through the guard: Stop before Start releases nothing; Stop during Start waits for it, or returns ctx's error first; a Stop whose release fails keeps it pending, and the next Stop runs it again; after a release that succeeded, Stop returns nil and runs nothing.

type Config

type Config struct {
	Ports           *component.PortConfig `json:"ports" schema:"type:ports,description:Port configuration,category:basic"`
	EnableHierarchy bool                  `json:"enable_hierarchy" schema:"type:bool,description:Enable hierarchy inference,default:false,category:advanced"`
	// IngestLanes is the number of keyed-concurrent ingest lanes (ADR-072,
	// gh#480). Messages are partitioned by entity ID (same entity → one lane →
	// serial in arrival order, preserving the arrival-order merge; different
	// entities → parallel), so ingest is no longer bound to a single serial
	// Get+CAS round-trip chain. Default 8 (concurrent); `1` selects the prior
	// fully-serial behavior.
	IngestLanes int `` /* 166-byte string literal not displayed */
}

Config holds configuration for graph-ingest component

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a valid default configuration

func (*Config) ApplyDefaults

func (c *Config) ApplyDefaults()

ApplyDefaults sets default values for configuration

func (*Config) Validate

func (c *Config) Validate() error

Validate implements component.Validatable interface

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL