readiness

package
v0.1.0-alpha.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 11, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package readiness writes and reads the readiness envelope a producer keeps in the GRAPH_STATUS KV bucket, and holds the gate that judges one.

The envelope (graph.IndexStatusResponse) is a producer's own account of its progress: whether it has finished its initial build, how far behind it is, and whether it is broken. A producer such as graph-ingest writes it to its own key on every heartbeat tick through a Publisher, and Gauges exposes the same fields to Prometheus. ComputeBacklogStatus fills the envelope for a producer whose progress is a message backlog, as graph-ingest's is.

The bucket is information, not a gate (ADR-109 Decision 3). It is for operators and for tools that show a producer's state to people or agents. Nothing in SemEngine waits on it.

EvaluateReadinessGate answers one Reading with a DeferReason: proceed, or why not. Its one reader in SemEngine will be graph-index, which asks it in process about the status it has just computed.

The Watcher reads one producer's key from another process and holds the last envelope, so a read needs no NATS round trip. It judges freshness by when each value arrived, never by comparing the producer's clock with its own (ADR-083 Decision 2). Nothing in SemEngine reads a Watcher yet: it is kept for its reader at the SemStreams pin, graph-clustering. Fusion reads no status (ADR-109 D1): pkg/fusion/fusionnats sees readiness as index_not_ready on each reply.

The key names live here, for producers and readers both, so the two cannot drift onto different names. The envelope type stays in graph, so a reader of GRAPH_STATUS needs no NATS import.

Index

Constants

View Source
const (
	// KeyGraphIndex is graph-index's status key (one key per producer).
	KeyGraphIndex = "graph-index"
	// KeyGraphEmbedding is graph-embedding's status key.
	KeyGraphEmbedding = "graph-embedding"
	// KeyGraphClustering is graph-clustering's status key. Its indexed revision counts what
	// the community maintainer holds, not what is published; a community record's own
	// entity_revision says what is published (graph.IndexStatusResponse.IndexedRevision).
	KeyGraphClustering = "graph-clustering"
	// KeyGraphIngest is graph-ingest's status key. Its envelope is a BACKLOG
	// envelope (ComputeBacklogStatus): Lag is in MESSAGES, and the revision
	// fields are absent because graph-ingest consumes multiple streams whose
	// sequence spaces are independent.
	KeyGraphIngest = "graph-ingest"
	// KeyRule is the rule processor's status key, reporting bootstrap-replay
	// completion for its entity watchers.
	KeyRule = "rule"
)

The GRAPH_STATUS keys, shared by producers and consumers. The bucket's name is graph.BucketGraphStatus.

View Source
const (
	// DefaultHeartbeat is the producers' status tick: they publish the envelope
	// every tick unconditionally, so the write doubles as a liveness heartbeat.
	DefaultHeartbeat = 5 * time.Second
)

Variables

This section is empty.

Functions

func ComputeBacklogStatus

func ComputeBacklogStatus(in BacklogStatusInputs) graph.IndexStatusResponse

ComputeBacklogStatus builds the readiness envelope for a BACKLOG producer — one whose "caught up" is the absence of un-applied messages rather than a revision watermark. It is a projection of its own beside graph-index's revision-lag projection, not a mode of it.

  • Ready = Outstanding == 0 && BootstrapComplete.
  • State = observationFailed ? "degraded" : (ready ? "ready" : "building").
  • Lag = Outstanding, IN MESSAGES — a different unit from the revision-lag producers' Lag.
  • StalenessMs = 0 when Ready or when OldestOutstandingAt is unknown, else now-OldestOutstandingAt with a 1ms floor (shared presence encoding).

WHY NOT the revision-lag projection: it computes Ready = target > 0 && indexed >= target, which is FALSE at 0/0 — exactly the steady state of an idle backlog producer with nothing to do. Bending it to accommodate that would also risk changing graph-index's published output.

IndexedRevision and TargetRevision are DELIBERATELY NOT SET, and both are omitempty so they stay off the wire. Those fields are contractually in the ENTITY_STATES KV revision space — ADR-084 D3 pins them as comparable to a caller's kv_revision by a test. A backlog producer consumes multiple streams whose sequence spaces are independent, so there is no single scalar revision to report; writing a stream sequence into a KV-revision field would silently corrupt every read-your-writes check in the system. A test asserts their absence on the wire.

The gate's PRODUCER INVARIANT (Ready == true implies BootstrapComplete == true) holds here by construction: BootstrapComplete is a conjunct of Ready.

func ComputeIndexStatus

func ComputeIndexStatus(in IndexStatusInputs) graph.IndexStatusResponse

ComputeIndexStatus builds the readiness envelope of a revision-lag producer (ADR-066, with ADR-083's age-of-view staleness) from its watermark floor, the target revision, its stuck verdict, the oldest unapplied write's commit time and its failed count.

  • Ready = Target > 0 && Indexed >= Target.
  • Lag = Target - Indexed, never below zero.
  • State = degraded when FailedCount > 0, else ready when Ready, else degraded when Stuck, else building.
  • StalenessMs = 0 when Ready or OldestUnappliedAt is unknown, else now-OldestUnappliedAt with a 1ms floor.

The staleness subtraction is the one place a NATS server commit time meets the local clock; under skew it is off by the skew, which ADR-083 D3 accepts.

Types

type BacklogStatusInputs

type BacklogStatusInputs struct {
	// Outstanding is total un-applied work across every bound consumer, IN MESSAGES —
	// the sum of NumPending (server-side, undelivered) and NumAckPending
	// (delivered, not yet acked) over all of them.
	//
	// THE SUM IS INVARIANT TO WHICH COUNTER HOLDS A MESSAGE, and that is why it is a
	// sum rather than either half. A message moves between the two continuously —
	// delivered (pending -> ack-pending), nak'd back (ack-pending -> pending),
	// redelivered again — so any single counter oscillates while work is steady.
	// Only the total is monotone with respect to real outstanding work. Do not
	// "simplify" this to NumPending: that under-reports by the whole in-process lane
	// queue (up to defaultIngestLanes(8) x ingestLaneQueueDepth(256) = 2048 messages
	// held delivered-but-unacked), which is precisely the backlog SemStreams gh#712
	// tripped over.
	//
	// WHY NOT THE ACK FLOOR: measured 2026-07-30 against both deployed NATS versions
	// (2.10, 2.12), AckFloor.Stream does not advance past a MaxDeliver-exhausted
	// message, then jumps PAST it on the next unrelated ack — reading
	// permanently-not-caught-up while idle and falsely-covered under traffic. It never
	// means "everything at or below this is durable". See the SemStreams change's
	// design.md D0.
	//
	// HONESTY BOUNDARY: Outstanding == 0 means NO OUTSTANDING WORK, not EVERYTHING WAS
	// APPLIED. A MaxDeliver-parked message leaves BOTH counters (measured), so it is
	// invisible here. Caught-up is a backlog claim and cannot license an absence
	// claim; operator visibility for parked messages is SemStreams gh#742.
	Outstanding uint64
	// BootstrapComplete is the producer's own initial-build latch. It is an INPUT, not
	// a projection: what constitutes the initial build differs per producer (a drained
	// boot sweep, a watcher replay sentinel), and only the producer can say.
	BootstrapComplete bool
	// BootstrapScope is the size of that initial build in the producer's unit; see the
	// field of the same name on graph.IndexStatusResponse. Passed through untouched — the
	// projection never compares it to anything.
	BootstrapScope uint64
	// ObservationFailed reports that the producer could not read its own backlog (a
	// consumer.Info() failure). It projects to degraded AND forces Ready false,
	// mirroring graph-index's precedent for a failed target read
	// (pin: processor/graph-index/watermark.go:69-80): a backend fault cannot honestly
	// confirm caught-up, and "building" would read as ordinary progress rather than a
	// fault. See the divergence note in ComputeBacklogStatus for why this is stronger
	// than the revision-lag projection's FailedCount handling.
	//
	// Outstanding may still be a PARTIAL sum when this is set (some consumers read,
	// one failed). That partial is kept on Lag as an honest lower bound rather than
	// zeroed — a lower bound under a degraded verdict is strictly more useful to an
	// operator than a fabricated 0, and no gate proceeds on degraded anyway.
	ObservationFailed bool
	// OldestOutstandingAt is the JetStream timestamp of the oldest outstanding
	// message. ZERO means "not computable" — the projection then leaves StalenessMs at
	// 0 rather than fabricating a fresh-looking view, matching the presence encoding
	// on that field.
	OldestOutstandingAt time.Time
	// Now is the compute instant; the zero value means time.Now().
	Now time.Time
}

BacklogStatusInputs are the observations ComputeBacklogStatus projects into the readiness envelope. It is SEPARATE from IndexStatusInputs on purpose: the two producer shapes have disjoint inputs (a revision watermark vs. a message backlog), and merging them would make mutually-exclusive fields co-resident, an invalid state made representable.

type BucketSource

type BucketSource interface {
	GetKeyValueBucket(ctx context.Context, name string) (jetstream.KeyValue, error)
}

BucketSource opens a KV bucket by name. *natsclient.Client satisfies it through GetKeyValueBucket, whose signature this must match exactly. The Watcher takes this one method rather than the client, so its tests need no live NATS.

type DeferReason

type DeferReason string

DeferReason is the typed cause of a defer. Its string form is the label value of the defer_total{reason} counter and of the structured defer log field, so the set is CLOSED: four reasons an operator acts on differently — a broken index, an index that has not finished building, a dead status feed, and an envelope this consumer cannot interpret.

const (
	// DeferNone is the reason attached to a proceed.
	DeferNone DeferReason = ""
	// DeferHardStop is a degraded or reset_required index: broken, not behind.
	DeferHardStop DeferReason = "hard_stop"
	// DeferStatusUnknown is a status feed the consumer cannot vouch for (never
	// received, or older than the freshness window). It fails closed and is never
	// mistaken for index state.
	DeferStatusUnknown DeferReason = "status_unknown"
	// DeferUnrecognizedState is an envelope whose State is blank or outside
	// graph.IndexStates(). It is NOT the same operator problem as a dead feed: the value
	// arrived and decoded, so the producer is talking — it is saying something this
	// consumer does not understand, which means version skew (a newer producer, a
	// partially-written envelope, a hand-edited key). Failing closed on it is the
	// whole posture of ADR-084: State became load-bearing when health replaced
	// coverage, so an uninterpretable State cannot be waved through as "not degraded".
	DeferUnrecognizedState DeferReason = "unrecognized_state"
	// DeferBootstrapIncomplete is a producer that has not finished its initial build
	// in this process lifetime, such as an index halfway through a format cutover
	// (SemStreams gh#474), where the keyset is half-built and a plausible small Lag
	// is actively misleading. TargetRevision==0 cannot stand in for it: that is false
	// during a cutover, and true for an empty graph whose build has finished.
	DeferBootstrapIncomplete DeferReason = "bootstrap_incomplete"
)

The closed defer-reason set.

func DeferReasons

func DeferReasons() []DeferReason

DeferReasons returns the closed set of defer reasons a gate can return, as a new slice on every call, so no caller can change what the next one reads. It lives next to the const block so it is the single source of truth, as graph.IndexStates() is for wire states: adding a reason means adding it HERE, once, and every consumer that pre-initializes counter series or validates a label set picks it up automatically.

The alternative — each consumer hand-maintaining its own copy — fails SILENTLY in the direction that matters. A counter keyed off a private list drops any reason missing from it (dropping is correct: an unbounded label is a cardinality leak), so a newly added reason would defer in production while incrementing nothing, and the dashboard would read as "no defers" rather than "unknown defers".

DeferNone is deliberately absent: it is the reason attached to a PROCEED, not a defer, and pre-initializing a "" series would publish a permanently-zero label.

func EvaluateReadinessGate

func EvaluateReadinessGate(r Reading) DeferReason

EvaluateReadinessGate is the single home for readiness gate semantics. It answers the health question for one Reading and returns the typed defer reason; the caller proceeds when that is DeferNone. A Watcher reader passes w.Read(), so status and freshness come from one read; an IN-PROCESS producer that computes its own envelope passes Reading{Status: s, Fresh: true}, having no transport to lose. How a caller REACTS to a defer (error, skip a tick, serve an honest empty envelope) stays with the caller.

Evaluation order is load-bearing:

  1. Unknown status short-circuits BEFORE anything else. A dead status feed is a transport fact; nothing licenses proceeding on state the consumer cannot vouch for.
  2. An UNRECOGNIZED State defers. The check is an allow-list over graph.IndexStates(), not a deny-list for degraded/reset_required: a blank or future state must not read as "not degraded, therefore healthy".
  3. Hard stops (degraded, reset_required) defer.
  4. An incomplete initial build defers likewise. This guards against reading an index halfway through a format cutover (SemStreams gh#474), and is why bootstrap_complete is on the wire.
  5. Everything else proceeds, however far behind the view is. Lag is not a fault; it is a property of the answer, carried on graph.IndexStatusResponse.StalenessMs for the consumer to stamp on its output.

Ready is deliberately absent. It is COVERAGE, and coverage answers no question this gate asks: it cannot license a proceed a health verdict already denied, and it cannot withhold one a health verdict already allowed. A caller with a specific revision in hand compares it against IndexedRevision itself.

PRODUCER INVARIANT: an envelope with Ready == true must also carry BootstrapComplete == true. Health is answered before coverage is consulted anywhere, so a producer that published a caught-up envelope while reporting an unfinished build would see every consumer defer on it forever, pointing an operator at a cutover that already finished. A caller comparing its own revision with IndexedRevision also assumes the two agree. Every producer keeps the invariant; ComputeBacklogStatus does by construction.

type Gauges

type Gauges struct {
	// contains filtered or unexported fields
}

Gauges is the readiness envelope's Prometheus projection, owned ONCE for every producer.

Set is the single projection site, so a field added to IndexStatusResponse cannot be wired in one producer and forgotten in another. At the SemStreams pin, two producers each built their own set, and both left out bootstrap_complete, the field EvaluateReadinessGate reads.

func NewGauges

func NewGauges(subsystem string) *Gauges

NewGauges builds the readiness gauge set for one producer, named by its Prometheus subsystem ("graph_ingest", "graph_index"). Register keys each collector under the same subsystem.

The emitted metric NAMES are an external contract (ADR-066): dashboards and operator alerts read them. They are the names the SemStreams pin emits, and a test compares them against the live registry.

func NewRevisionGauges

func NewRevisionGauges(subsystem string) *Gauges

NewRevisionGauges builds the gauge set for a revision-lag producer (graph-index's follower): NewGauges' set plus indexed_revision and target_revision, the envelope's floor and target in ENTITY_STATES revisions (ADR-066). A backlog producer uses NewGauges, which exports neither.

func (*Gauges) RecordPublishFailure

func (g *Gauges) RecordPublishFailure()

RecordPublishFailure counts one failed GRAPH_STATUS heartbeat write (ADR-083).

func (*Gauges) Register

func (g *Gauges) Register(registry *metric.MetricsRegistry) error

Register registers every collector on registry through metric.RegisterOrGet and keeps the collectors it returns: when another Gauges already registered a metric under the same subsystem and name, this one writes that collector, so its writes are the ones gathered. A nil registry registers nothing: the gauges still take writes, and nothing gathers them. Nothing is registered on Prometheus' global registry.

func (*Gauges) Set

func (g *Gauges) Set(resp graph.IndexStatusResponse)

Set projects one readiness envelope onto the gauges.

THE SINGLE PROJECTION SITE. Every producer funnels through here, so adding a field to IndexStatusResponse means adding it once.

The state one-hot iterates graph.IndexStates() rather than the states this producer happens to emit, so a new readiness state renders as an explicit 0 instead of a silently absent series.

type IndexStatusInputs

type IndexStatusInputs struct {
	// Indexed is the watermark's floor: every delivered ENTITY_STATES revision at
	// or below it has been applied.
	Indexed uint64
	// Target is the ENTITY_STATES stream LastSeq the producer must reach, read at
	// compute time.
	Target uint64
	// Stuck is the producer's own stuck-watermark verdict; it projects to degraded
	// when not ready.
	Stuck bool
	// OldestUnappliedAt is the KV commit time of the oldest delivered revision not
	// yet applied, or of the newest delivered one when none is in flight
	// (revlag.Watermark.IndexedAt). ZERO means "not computable": the projection then
	// leaves StalenessMs at 0 rather than reporting a fresh-looking view.
	OldestUnappliedAt time.Time
	// Now is the compute instant; the zero value means time.Now().
	Now time.Time
	// FailedCount is the number of entries the producer currently holds in a failed
	// state. Above zero it projects to degraded before the ready branch: a producer
	// caught up over failures is covered (Ready stays accurate) but not healthy
	// (ADR-085).
	FailedCount uint64
}

IndexStatusInputs are the observations ComputeIndexStatus projects into the readiness envelope of a revision-lag producer (graph-index's follower). It is a struct rather than positional arguments because two inputs are time.Time (OldestUnappliedAt, Now), and adjacent same-typed parameters swap silently.

type Publisher

type Publisher struct {
	// contains filtered or unexported fields
}

Publisher writes one producer's readiness envelope to its GRAPH_STATUS key. One instance per producer; Publish is called from the status tick and is safe to call from any single goroutine.

func NewPublisher

func NewPublisher(bucket StatusWriter, key string) (*Publisher, error)

NewPublisher builds a publisher for one producer key (KeyGraphIndex, KeyGraphEmbedding, KeyGraphIngest). It refuses a nil bucket and an empty key, so a producer wired without either fails its Start instead of publishing nothing while consumers read the key as absent. A typed-nil bucket is not detected.

func (*Publisher) Key

func (p *Publisher) Key() string

Key reports the producer key this publisher writes, for the caller's failure log.

func (*Publisher) Publish

func (p *Publisher) Publish(ctx context.Context, status graph.IndexStatusResponse) error

Publish writes the envelope as the KV value for this producer's key.

The value is PLAIN graph.IndexStatusResponse JSON, with no BaseMessage wrapper. The payload registry governs polymorphic message publishes on subjects, where a receiver must discriminate a type it did not choose; this is a KV value on a fixed key whose type is fixed by the contract, and wrapping it would break every reader's plain decode, the Watcher's included, for nothing.

Callers publish on EVERY tick, unconditionally, without comparing to the last value: the write is the liveness heartbeat that lets a consumer tell "not ready" from "the producer is gone", and skipping unchanged values would make a healthy steady state indistinguishable from a dead one.

The error is returned, never swallowed, so the tick loop can count and log it; the loop must keep ticking regardless, because the next heartbeat is the recovery. A nil context is refused with an error before any write.

type Reading

type Reading struct {
	// Status is the last envelope decoded from the key.
	Status graph.IndexStatusResponse
	// Fresh reports that the held envelope arrived within three
	// heartbeats, was not already older than that when it arrived, and is still the
	// key's current value (no undecodable value has replaced it). False means
	// UNKNOWN — fail closed.
	Fresh bool
	// Known reports that an envelope has been received at least once. It separates
	// "never saw the producer" (absent bucket/key, standalone deployment) from "the
	// feed went quiet", which are different operator problems.
	Known bool
	// Age is how long ago the held envelope arrived, consumer-local. Zero when
	// !Known.
	Age time.Duration
	// Err is the last bucket/watch/decode failure, if any, for the structured defer
	// log. It is cleared by the next good update.
	Err error
}

Reading is the consumer-side answer: the last-known envelope plus everything needed to attribute a defer without correlating log lines.

Status is authoritative ONLY when Fresh. When Fresh is false the envelope is last-known-and-aged, or no longer the key's current value (or the zero value if none ever arrived), and is carried for DIAGNOSTICS only — a caller must fail closed, never read the stale bits as index state, so a dead feed is never mistaken for an index that is not ready.

type StatusWriter

type StatusWriter interface {
	Put(ctx context.Context, key string, value []byte) (uint64, error)
}

StatusWriter writes one status value. jetstream.KeyValue satisfies it; taking the one method the publisher needs keeps the seam narrow and the unit tests honest.

type Watcher

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher holds the last-known readiness envelope for ONE producer key. One instance serves every gate decision in a consumer: Read is safe from any goroutine, and Run runs on a goroutine the owning component starts and stops by ending Run's context.

func NewWatcher

func NewWatcher(src BucketSource, key string, heartbeat time.Duration, logger *slog.Logger) *Watcher

NewWatcher builds a watcher for one producer key (KeyGraphIndex, KeyGraphEmbedding) in the GRAPH_STATUS bucket. heartbeat is the producer's publish interval (DefaultHeartbeat for the framework's producers), which sets the freshness window to three heartbeats; logger receives bind and decode failures. It performs no I/O and checks nothing; Run refuses incomplete wiring.

func (*Watcher) Read

func (w *Watcher) Read() Reading

Read returns the current reading. It never blocks on I/O — the answer comes from held state, which is the point of watching rather than polling.

func (*Watcher) Run

func (w *Watcher) Run(ctx context.Context) error

Run watches the key until ctx ends, then returns ctx.Err(). The caller runs it on a goroutine it owns; Run starts no goroutine of its own. It stops the KV watch and waits for the watch's delivery to close before it returns, so its return is the join, for at most 5 s after the watcher's own Stop returns, whether or not ctx has ended; past that bound it returns with the delivery still open. Why a stopped nats.go watch needs that wait is in internal/kvwatch's doc. A failure to stop the watch is logged at Debug.

Run does NOT fail when the bucket or key is absent: a deployment running a consumer without its producer is a legitimate configuration whose readiness is simply UNKNOWN, and Read reports it as such (fail closed) while Run keeps trying to bind, rebinding after any failure. It never fabricates an envelope. Misconfiguration is refused at the call instead: a nil context, a nil bucket source, an empty key, a heartbeat of zero or less, or a nil logger returns an error before any I/O.

func (*Watcher) WaitForFirst

func (w *Watcher) WaitForFirst(ctx context.Context) error

WaitForFirst blocks until an envelope has been received on the key, or ctx ends. It returns nil once something has arrived (fresh or not — Read is what judges that) and ctx.Err() otherwise.

It is for a consumer that binds the watch lazily, on its first gate decision, as fusion's NATS client does at the SemStreams pin: without a bounded wait, that first decision would fail closed only because the watch had not delivered yet. Waiting once at bind, with a bounded context, delays only the first gated call; every later call is a local read. Callers must wait AT MOST once: with no producer deployed nothing will ever arrive, and a per-call wait would turn the standalone deployment into a per-call stall. A nil context returns an error, even after an envelope has arrived.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL