graphlexical

package
v0.1.0-alpha.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 26 Imported by: 0

README

graph-lexical

The word-search component of the graph subsystem.

Overview

graph-lexical follows the ENTITY_STATES bucket, which graph-ingest writes, and keeps an in-memory word index (graph/lexical) of each entity's text. A word index maps each word to the entities that contain it, so a query such as pump -telemetry finds entities by the words in their name and body and ranks them with BM25, a standard scoring formula that rewards a word that is frequent in an entity and rare across all entities. Nothing is sent to a model or other service, so it runs at tier 0, and the index is never written to disk: it is rebuilt from ENTITY_STATES on every start. Which of an entity's values count as its text is decided by graph/entitytext, the same rule graph-embedding uses.

Each process holds its own index. A revision is the number NATS gives each write to a bucket; a revision counts as applied only once its text is in the index, and the component reports that to graph/follower, which publishes the readiness record to GRAPH_STATUS under the key graph-lexical.

Who word search helps

Only entities that have text: documents, notes, descriptions, names. An entity whose facts are numbers, such as sensor or drone telemetry, has no words to match and gains nothing from word search. Such entities are reached through the structure of the graph instead (by ID prefix, predicate, relationship, name and alias, or group); the graph/lexical README lists the read requests. Two limits apply today: the graph has no numeric range filter (the predicate request matches a stored value only as an exact string), and no component serves a time-range request, so "temperature above 80" or "readings from the last hour" cannot be asked of the graph yet.

Query verb

The component answers graph.lexical.query.search. The request is JSON:

{"query": "pump -telemetry", "limit": 10, "scope": ["acme.ops.docs"], "min_revision": 0}
Field Meaning
query Plain words, quoted phrases and - exclusions; the rules are in the graph/lexical README
limit 0 means 10; 1 to 100 is used as given; anything else is refused
scope Entity ID prefixes; an entity must start with one of them. Empty or "" means every entity. The scope applies before the limit
min_revision Answer only once the index holds at least this revision; otherwise a transient revision_not_reached error

An unknown request key is refused, naming it. The reply is a graph.QueryResponse whose data holds:

  • results: the best matches, each an entity_id and a score, ordered by score and then entity ID. A score has no fixed scale and is comparable only within one reply.
  • matched: how many entities matched within the scope, before the limit. matched above the number of results means the limit cut the answer.
  • excluded: the query's excluded terms, each term's words joined by a space (go test -race gives ["race"]), omitted when there are none. Every entity holding one was left out, so a caller can say why an expected entity is absent.
  • bodies_not_indexed: how many entities in the whole index, not only the hits, have an offloaded body the index does not hold (next section). Such an entity is found only by its inline text.

While the first build is unfinished, and while the ENTITY_STATES feed is lost, the verb refuses with the transient code index_not_ready. After that, no body state withholds it.

Entity text and offloaded bodies

An entity's name is its text values whose predicate ends in .name or .title, plus the values of the vocabulary's label predicates; its body is its other text values, then the offloaded body. Text values are the string values of predicates ending in one of text_suffixes. Words in the name count twice when ranking.

An entity's body can be offloaded: stored in an object store, with the entity holding only a reference to it. Such an entity is indexed by its inline text at once and by its body once a single goroutine has read the body from the store.

A body is read once at start, and again only when the entity's reference or its inline text changes. A revision that changes neither, such as one that only updates telemetry values, leaves the entity's words as they are and reads nothing, so an entity updated many times a second keeps its body's words. Two references are the same only when every field is: storage instance, key, content type and size. A reference therefore names fixed content: a producer that stores new content must give it a new reference, as the object store's StoreContent does with a new key on every call. Content replaced behind an unchanged reference is not read again until the reference or the inline text changes, or graph-lexical restarts. When the inline text changes under the same reference, the new inline words are found at once and the body's words again once its read returns; a read already queued or in flight for that reference serves the newer revision, joined to its inline text, so updates arriving faster than reads still end with the body indexed.

Until it is read, a body not in the index is in one of four states, counted on the gauge semengine_graph_lexical_bodies_not_indexed{state}:

State Meaning What happens next
queued Not read yet It is read in turn
unread A read failed, or the store was skipped after another read on it failed Retried every 30 seconds. It is the only failure: the readiness record reads degraded with the reason body_unread, and Health names it
missing The store has no such object Not asked for again until the entity's reference or inline text changes, or graph-lexical restarts
no_store No store is registered for the body's storage instance Looked up again every 30 seconds

A read that fails for any reason other than a missing object skips that store until the next 30-second pass, so a store that does not answer, but honours its read's context, costs one 10-second read deadline per pass, not one per body. A store that ignores its context is not bounded by the deadline. The first missing body and the first unregistered store since start are logged once each.

The text cap

max_text_len is the most characters of an entity's body that are indexed. Text past it is cut, preferring a word boundary, otherwise at the cap, and the name is never cut. The default is 16,000 characters, long enough that a code body is almost never cut; graph-embedding's default is 4,000, so the two can index different amounts of the same entity. The gauge semengine_graph_lexical_bodies_cut counts the entities whose body is cut, and the first cut since start is logged once, naming the entity and the cap. Words past the cap cannot be found. A larger value reads more of each offloaded body and holds more words; the accepted range is 0 to 1,000,000.

File paths are not indexed unless a product adds the suffix of its path predicate to text_suffixes.

Configuration

{
  "type": "processor",
  "name": "graph-lexical",
  "enabled": true,
  "config": {
    "ports": {
      "inputs": [
        {"name": "entity_watch", "config": {"kind": "kv-watch", "bucket": "ENTITY_STATES"}}
      ]
    },
    "max_text_len": 16000
  }
}
Option Type Default Description
ports object the input above Exactly one input, a kv-watch of every ENTITY_STATES key with no history; no outputs. A ports block that is present is used as written
text_suffixes array .title .content .description .summary .text .name .body .abstract .subject Predicate suffixes whose string values are indexed as text
max_text_len int 16000 Characters of body indexed per entity, 0 to 1,000,000; 0 means the default
startup_attempts int 30 Tries to find ENTITY_STATES at startup
startup_interval_ms int 500 Pause between those tries

The configuration is decoded strictly: a key not listed here is refused at construction, naming it. Offloaded bodies are read from the stores in the composition's store registry, which the component looks up at each read; no store port is configured.

A product adds the component to its composition by calling graphlexical.Register on its component registry and listing graph-lexical among its components.

Memory and restart cost

The index is in memory and is rebuilt on every start, so a restart costs a rebuild and the index holds whatever fits in the process.

  • Memory, measured on one host: 50,000 entities with about 1.2 KB of text each took 110 MB of heap (about 2.2 KB per entity); 32,000 entities with about 640 bytes of text each took 44 MB (about 1.4 KB per entity). One million entities is a projection, not a measurement: about 1.4 GB at the smaller text size and about 2.2 GB at the larger. Real text has more distinct words than the repeated test text, so treat these as floors.
  • Search time grows with the number of entities that match. A query on a common word against a very large index holds the index's read lock while it scores, and writes wait; this was estimated at 200 to 400 ms for a million entities, not measured here.
  • First build. Every offloaded body is read on every start. For 32,000 entities with offloaded bodies of about 600 characters, the first search was answered after about 16 seconds on one Apple M3 Pro machine against a local NATS server in Docker, about 0.4 to 0.5 ms per body; 100,000 took 50 to 54 seconds. On another host or against a remote store the time is not measured. Until the build is applied, search is refused with index_not_ready.

Metrics

All on the registry the composition injects.

Gauge Meaning
semengine_graph_lexical_index_entities Entities in the index
semengine_graph_lexical_index_words Words in the index, every occurrence in every entity counted
semengine_graph_lexical_bodies_not_indexed{state} Offloaded bodies not in the index, by the four states above
semengine_graph_lexical_bodies_cut Entities whose body is cut at max_text_len

Documentation

Overview

Package graphlexical is the graph-lexical component: it follows ENTITY_STATES and keeps an in-memory word index (graph/lexical) of each entity's text, rebuilt from the bucket on every start, and reports on the graph-lexical readiness key. An entity's text is chosen by graph/entitytext, the rule graph-embedding uses, split into the entity's name and its body. No model or external provider is involved at any tier.

Every delivered change is applied in delivery order, and a delivery at or below the revision the index holds for its key is ignored. A revision is reported applied only once the index holds its text.

An entity whose body is offloaded to a store is indexed by its inline text at once, and by its body once the one goroutine that reads every body has read it. A body the index does not hold is queued, unread, missing or has no store. Only unread is a failure: the readiness key reads degraded and Health names it until a repair pass, every 30 seconds, reads the body. A read that fails on a store skips that store until the next pass.

The index answers graph.lexical.query.search (graph.LexicalSearchRequest, replying graph.LexicalSearchData in a graph.QueryResponse) from Start until Stop. It refuses transient index_not_ready until the initial build is applied and while the feed is lost, and answers min_revision against the follower's floor as graph-index does. No body state withholds it.

A composition registers the factory with Register and creates the component through the component manager, with a configuration that Config describes.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateGraphLexical

func CreateGraphLexical(rawConfig json.RawMessage, deps component.Dependencies) (component.Discoverable, error)

CreateGraphLexical is the factory function for graph-lexical components. It opens no connection. It reads the vocabulary's label predicates, so a product registers its own first. Offloaded bodies are read from the stores in deps.StoreRegistry, looked up at each read.

func DeclarePorts

func DeclarePorts(rawConfig json.RawMessage, _ string) (component.PortConfig, error)

DeclarePorts is the component.PortDeclarer for graph-lexical: the ports CreateGraphLexical will report for rawConfig, computed without dependencies.

func Register

func Register(registry *component.Registry) error

Register registers the graph-lexical factory with the component registry. It asks boot for no model registry: the word index needs no provider at any tier.

Types

type Component

type Component struct {
	// contains filtered or unexported fields
}

Component implements the graph-lexical processor.

func (*Component) DataFlow

func (c *Component) DataFlow() component.FlowMetrics

DataFlow reports input messages: one per delivered ENTITY_STATES entry, update or delete, with the delivered value's length.

func (*Component) Health

func (c *Component) Health() component.HealthStatus

Health reports the shell's verdict (component.Lifecycle.Health) over the follower's stuck states: a latched reset or a lost feed.

func (*Component) Initialize

func (c *Component) Initialize() error

Initialize marks the component initialized (no I/O); the factory validated the configuration.

func (*Component) InputPorts

func (c *Component) InputPorts() []component.Port

InputPorts returns the input ports, fixed at construction.

func (*Component) OutputPorts

func (c *Component) OutputPorts() []component.Port

OutputPorts returns no ports: the index lives in the component's memory.

func (*Component) Start

func (c *Component) Start(ctx context.Context) error

Start begins following ENTITY_STATES into a new, empty index, so every start rebuilds it. The lifecycle state is the shell's (component.Lifecycle): Start before Initialize, a nil or ended context and a second Start are refused, the second with errs.ErrAlreadyStarted, and a Start that fails is rolled back through cleanup. A component with no NATS client is refused first.

ctx bounds Start only: the wait for ENTITY_STATES and the query subscriptions run under it. The follower's watch runs under a run context start creates and cleanup ends, so ending ctx after Start returned stops nothing (ADR-113). The search verb is served from Start until Stop.

func (*Component) Stop

func (c *Component) Stop(ctx context.Context) error

Stop stops the component through the shell, which runs cleanup: Stop before Start does nothing, and any non-nil Stop on a started component starts every teardown step, waits for them only as long as its context allows, and keeps what has not finished for the next Stop.

type Config

type Config struct {
	Ports *component.PortConfig `json:"ports" schema:"type:ports,description:Port configuration,category:basic"`

	// TextSuffixes are the predicate suffixes whose string values are an entity's text, as for
	// graph-embedding; empty means entitytext.DefaultSuffixes.
	TextSuffixes []string `` /* 199-byte string literal not displayed */

	// MaxTextLen is the most characters of an entity's body indexed, as for graph-embedding; the
	// body's tail past it is cut at a word boundary, and the name is never cut. 0 means 16,000.
	MaxTextLen int `` /* 205-byte string literal not displayed */

	// Dependency startup configuration
	StartupAttempts int `` /* 130-byte string literal not displayed */
	StartupInterval int `` /* 134-byte string literal not displayed */
}

Config holds the graph-lexical configuration. Decoding is strict: a key it does not define is refused, naming it.

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns a valid default configuration.

func (*Config) ApplyDefaults

func (c *Config) ApplyDefaults()

ApplyDefaults resolves each zero value to the value its reader uses. Validate only checks.

func (*Config) Validate

func (c *Config) Validate() error

Validate implements component.Validatable.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL