Documentation
¶
Overview ¶
Package readiness owns BOTH sides of the ADR-083 readiness distribution contract. Producers (publisher.go) create the GRAPH_STATUS KV bucket and write their envelope to their own key on every heartbeat tick; consumers (watcher.go) watch that key, hold the last-known ADR-066 envelope, and answer "(status, fresh | unknown)" with no per-decision NATS round-trip.
It exists because per-decision request/reply died under exactly the load that makes readiness interesting: in a single-binary deployment the status request travels the same connection as the ENTITY_STATES firehose, so an in-process requester pays the saturated stream twice per round-trip, times out, and fails closed — indistinguishable in the logs from a genuine not-ready (gh#590). The KV twofer inverts it: the producer pays one write per heartbeat and N consumers hold state.
Freshness is judged by CONSUMER-LOCAL arrival time. No producer timestamp is ever compared against the consumer's clock (ADR-083 D2) — arrival-time freshness needs no clock agreement between processes, and cross-process skew would buy nothing. The envelope's published_at, which Publisher.Publish stamps, is not read here; what a reader does with it is #110's.
This package holds the bucket/key identifiers for BOTH sides of the contract, so producers and consumers cannot drift onto different names, with the computation that fills the envelope (ComputeBacklogStatus) and the gate that interprets it (EvaluateReadinessGate). The envelope type itself stays in graph. The Watcher takes a BucketSource, the narrow method set *natsclient.Client already satisfies, rather than the client, which keeps its unit tests free of a live NATS.
Index ¶
- Constants
- Variables
- func ComputeBacklogStatus(in BacklogStatusInputs) graph.IndexStatusResponse
- func EnsureBucket(ctx context.Context, client *natsclient.Client) (jetstream.KeyValue, error)
- func FreshnessWindow(heartbeat time.Duration) time.Duration
- type BacklogStatusInputs
- type BucketSource
- type DeferReason
- type Gauges
- type Option
- type ProducerNames
- type Publisher
- type Reading
- type StatusReading
- type StatusWriter
- type Watcher
Constants ¶
const ( // BucketGraphStatus is the dedicated readiness bucket. It is operational // component status, NOT domain entity state: it never routes through // graph-ingest, carries no ADR-055 semantic envelope, and is kept out of // ENTITY_STATES so the sole-writer invariant and the graph itself stay clean. // // Re-exported from graph.BucketGraphStatus (the single source of truth) so // this name and the framework-owned-bucket set that write-protects it against // a generic rule update_kv (framework-owned-bucket-guards F3) can never drift. // Producers and consumers keep referring to readiness.BucketGraphStatus. BucketGraphStatus = graph.BucketGraphStatus // KeyGraphIndex is graph-index's status key (one key per producer). KeyGraphIndex = "graph-index" // KeyGraphEmbedding is graph-embedding's status key. KeyGraphEmbedding = "graph-embedding" // KeyGraphIngest is graph-ingest's status key. Its envelope is a BACKLOG // envelope (graph.ComputeBacklogStatus): Lag is in MESSAGES, and the revision // fields are absent because graph-ingest consumes multiple streams whose // sequence spaces are independent. KeyGraphIngest = "graph-ingest" // KeyRule is the rule processor's status key, reporting bootstrap-replay // completion for its entity watchers. KeyRule = "rule" )
The GRAPH_STATUS identifiers, shared by producers and consumers.
const ( // DefaultHeartbeat is the producers' status tick: they publish the envelope // every tick unconditionally, so the write doubles as a liveness heartbeat. DefaultHeartbeat = 5 * time.Second // FreshnessMultiplier is how many heartbeats a held status stays fresh. It is a // CONSTANT, not config (owner decision): 3 tolerates a lost write and a slow // delivery without letting a dead feed masquerade as live, and one fewer knob // is one fewer way to configure a fail-open. // // "Freshness" here is about the ENVELOPE, not the view it describes — can this // consumer still vouch for the status reading it is holding, or has the producer // gone quiet? That is a transport-liveness question, and it is what // StatusReading.Fresh answers and what the health gate fails closed on. It is NOT // the retired view-age concept (how far behind ENTITY_STATES the index itself is); // view age is reported on IndexStatusResponse.StalenessMs and gates nothing. FreshnessMultiplier = 3 )
Variables ¶
var AllDeferReasons = []DeferReason{ DeferHardStop, DeferStatusUnknown, DeferUnrecognizedState, DeferBootstrapIncomplete, }
AllDeferReasons is the closed iteration domain for the typed defer reasons a gate can return. It lives next to the const block so it is the single source of truth, exactly as graph.AllIndexStates is for wire states: adding a reason means adding it HERE, once, and every consumer that pre-initializes counter series or validates a label set picks it up automatically.
The alternative — each consumer hand-maintaining its own copy — fails SILENTLY in the direction that matters. A counter keyed off a private list drops any reason missing from it (dropping is correct: an unbounded label is a cardinality leak), so a newly added reason would defer in production while incrementing nothing, and the dashboard would read as "no defers" rather than "unknown defers".
DeferNone is deliberately absent: it is the reason attached to a PROCEED, not a defer, and pre-initializing a "" series would publish a permanently-zero label.
Functions ¶
func ComputeBacklogStatus ¶
func ComputeBacklogStatus(in BacklogStatusInputs) graph.IndexStatusResponse
ComputeBacklogStatus builds the readiness envelope for a BACKLOG producer — one whose "caught up" is the absence of un-applied messages rather than a revision watermark. It is a projection of its own beside graph-index's revision-lag projection, not a mode of it.
- Ready = Outstanding == 0 && BootstrapComplete.
- State = observationFailed ? "degraded" : (ready ? "ready" : "building").
- Lag = Outstanding, IN MESSAGES — a different unit from the revision-lag producers' Lag, which the spec states explicitly.
- StalenessMs = 0 when Ready or when OldestOutstandingAt is unknown, else now-OldestOutstandingAt with a 1ms floor (shared presence encoding).
WHY NOT the revision-lag projection: it computes Ready = target > 0 && indexed >= target, which is FALSE at 0/0 — exactly the steady state of an idle backlog producer with nothing to do. Bending it to accommodate that would also risk byte-drift in graph-index's published output, which the current spec protects.
IndexedRevision and TargetRevision are DELIBERATELY NOT SET, and both are omitempty so they stay off the wire. Those fields are contractually in the ENTITY_STATES KV revision space — ADR-084 D3 pins them as comparable to a caller's kv_revision by a test. A backlog producer consumes multiple streams whose sequence spaces are independent, so there is no single scalar revision to report; writing a stream sequence into a KV-revision field would silently corrupt every read-your-writes check in the system. A test asserts their absence on the wire.
The gate's PRODUCER INVARIANT (Ready == true implies BootstrapComplete == true) holds here by construction: BootstrapComplete is a conjunct of Ready.
func EnsureBucket ¶
EnsureBucket acquires the GRAPH_STATUS bucket through the catalog owner seam and returns a handle. Every producer calls it at Start, EAGERLY — before any consumer could bind — because a consumer that binds a watch to a not-yet-existent bucket reads permanently unknown (fail-closed) until it happens to rebind, and the whole point of this contract is that unknown means something is wrong.
It is IDEMPOTENT across producers and restarts: producers in one process (graph-index and graph-embedding at the pin) both call it, in either order and possibly concurrently; the seam's create-or-open resolves the race, and an adopted bucket is RECONCILED to the catalog declaration (History, no-lifecycle retention) rather than adopted config-unseen.
func FreshnessWindow ¶
FreshnessWindow is how long a status value stays fresh for a producer publishing on the given heartbeat. Exported so consumers and their tests express the window in one vocabulary instead of re-deriving 3x from the two constants.
Types ¶
type BacklogStatusInputs ¶
type BacklogStatusInputs struct {
// Outstanding is total un-applied work across every bound consumer, IN MESSAGES —
// the sum of NumPending (server-side, undelivered) and NumAckPending
// (delivered, not yet acked) over all of them.
//
// THE SUM IS INVARIANT TO WHICH COUNTER HOLDS A MESSAGE, and that is why it is a
// sum rather than either half. A message moves between the two continuously —
// delivered (pending -> ack-pending), nak'd back (ack-pending -> pending),
// redelivered again — so any single counter oscillates while work is steady.
// Only the total is monotone with respect to real outstanding work. Do not
// "simplify" this to NumPending: that under-reports by the whole in-process lane
// queue (up to defaultIngestLanes(8) x ingestLaneQueueDepth(256) = 2048 messages
// held delivered-but-unacked), which is precisely the backlog gh#712 tripped over.
//
// WHY NOT THE ACK FLOOR: measured 2026-07-30 against both deployed NATS versions
// (2.10, 2.12), AckFloor.Stream does not advance past a MaxDeliver-exhausted
// message, then jumps PAST it on the next unrelated ack — reading
// permanently-not-caught-up while idle and falsely-covered under traffic. It never
// means "everything at or below this is durable". See the change's design.md D0.
//
// HONESTY BOUNDARY: Outstanding == 0 means NO OUTSTANDING WORK, not EVERYTHING WAS
// APPLIED. A MaxDeliver-parked message leaves BOTH counters (measured), so it is
// invisible here. Caught-up is a backlog claim and cannot license an absence
// claim; operator visibility for parked messages is gh#742.
Outstanding uint64
// BootstrapComplete is the producer's own initial-build latch. It is an INPUT, not
// a projection: what constitutes the initial build differs per producer (a drained
// boot sweep, a watcher replay sentinel), and only the producer can say.
BootstrapComplete bool
// BootstrapScope is the size of that initial build in the producer's unit; see the
// field of the same name on graph.IndexStatusResponse. Passed through untouched — the
// projection never compares it to anything.
BootstrapScope uint64
// ObservationFailed reports that the producer could not read its own backlog (a
// consumer.Info() failure). It projects to degraded AND forces Ready false,
// mirroring graph-index's precedent for a failed target read
// (processor/graph-index/watermark.go:69-80): a backend fault cannot honestly
// confirm caught-up, and "building" would read as ordinary progress rather than a
// fault. See the divergence note in ComputeBacklogStatus for why this is stronger
// than the revision-lag projection's FailedCount handling.
//
// Outstanding may still be a PARTIAL sum when this is set (some consumers read,
// one failed). That partial is kept on Lag as an honest lower bound rather than
// zeroed — a lower bound under a degraded verdict is strictly more useful to an
// operator than a fabricated 0, and no gate proceeds on degraded anyway.
ObservationFailed bool
// OldestOutstandingAt is the JetStream timestamp of the oldest outstanding
// message. ZERO means "not computable" — the projection then leaves StalenessMs at
// 0 rather than fabricating a fresh-looking view, matching the presence encoding
// on that field.
OldestOutstandingAt time.Time
// Now is the compute instant; the zero value means time.Now().
Now time.Time
}
BacklogStatusInputs are the observations ComputeBacklogStatus projects into the readiness envelope. It is SEPARATE from the inputs of graph-index's revision-lag projection on purpose: the two producer shapes have disjoint inputs (a revision watermark vs. a message backlog), and merging them would make mutually-exclusive fields co-resident — an invalid state made representable in the one projection whose output gates authoritative-absence claims.
type BucketSource ¶
type BucketSource interface {
GetKeyValueBucket(ctx context.Context, name string) (jetstream.KeyValue, error)
}
BucketSource opens a KV bucket by name. *natsclient.Client satisfies it via GetKeyValueBucket — declared here as the narrow method set (with the EXACT signature, or the structural match silently breaks) rather than importing natsclient, so this package stays a leaf and tests need no live NATS.
type DeferReason ¶
type DeferReason string
DeferReason is the typed cause of a defer. Its string form is the label value of the defer_total{reason} counter and of the structured defer log field, so the set is CLOSED: four reasons an operator acts on differently — a broken index, an index that has not finished building, a dead status feed, and an envelope this consumer cannot interpret. The gh#590 investigation cost three comment cycles because the defer line was a bare constant.
const ( // DeferNone is the reason attached to a proceed. DeferNone DeferReason = "" // DeferHardStop is a degraded or reset_required index: broken, not behind. DeferHardStop DeferReason = "hard_stop" // DeferStatusUnknown is a status feed the consumer cannot vouch for (never // received, or older than the freshness window). It fails closed and is never // mistaken for index state. DeferStatusUnknown DeferReason = "status_unknown" // DeferUnrecognizedState is an envelope whose State is blank or outside // graph.AllIndexStates. It is NOT the same operator problem as a dead feed: the value // arrived and decoded, so the producer is talking — it is saying something this // consumer does not understand, which means version skew (a newer producer, a // partially-written envelope, a hand-edited key). Failing closed on it is the // whole posture of ADR-084: State became load-bearing when health replaced // coverage, so an uninterpretable State cannot be waved through as "not degraded". DeferUnrecognizedState DeferReason = "unrecognized_state" // DeferBootstrapIncomplete is a producer that has not finished its initial build // in this process lifetime — the gh#474 cutover window, where the keyset is // half-materialised and a plausible small Lag is actively misleading. It replaces // the former `empty` reason, whose TargetRevision==0 proxy was wrong in both // directions: false during a cutover, and true for the authoritatively empty // graph it then wrongly deferred. DeferBootstrapIncomplete DeferReason = "bootstrap_incomplete" )
The closed defer-reason set.
func EvaluateReadinessGate ¶
func EvaluateReadinessGate(reading StatusReading) (proceed bool, reason DeferReason)
EvaluateReadinessGate is the single home for readiness gate semantics. It answers the health question and reports whether to proceed plus the typed defer reason. How a caller REACTS to a defer (error, skip a tick, serve an honest empty envelope) stays with the caller.
Evaluation order is load-bearing:
- Unknown status short-circuits BEFORE anything else. A dead status feed is a transport fact; nothing licenses proceeding on state the consumer cannot vouch for.
- An UNRECOGNIZED State defers. The check is an allow-list over graph.AllIndexStates, not a deny-list for degraded/reset_required: a blank or future state must not read as "not degraded, therefore healthy".
- Hard stops (degraded, reset_required) defer.
- An incomplete initial build defers likewise — this is the wire-observable gh#474 guard, and the reason bootstrap_complete had to leave the producer's process.
- Everything else proceeds, however far behind the view is. Lag is not a fault; it is a property of the answer, carried on graph.IndexStatusResponse.StalenessMs for the consumer to stamp on its output.
Ready is deliberately absent. It is COVERAGE, and coverage answers no question this gate asks: it cannot license a proceed a health verdict already denied, and it cannot withhold one a health verdict already allowed. A caller with a specific revision in hand compares it against IndexedRevision itself.
PRODUCER INVARIANT this ordering still relies on: an envelope with Ready == true must also carry BootstrapComplete == true. Health is answered before coverage is consulted anywhere, so a producer that published a caught-up envelope while reporting an unfinished build would see every consumer defer on it forever — pointing an operator at a cutover that already finished. It survived the ADR-085 collapse: deleting the Ready fast path removed the gate's own READ of the field, not the obligation on producers, because graph-index's sticky responder gate and every consumer's IndexedRevision comparison still assume the two agree. graph-index maintains it in latchBootstrap and on its caged early-boot branch; a new producer must do the same.
type Gauges ¶
type Gauges struct {
// contains filtered or unexported fields
}
Gauges is the readiness envelope's Prometheus projection, owned ONCE for every producer.
It exists because the duplication it replaces did not merely permit a bug, it produced one twice: graph-index and graph-embedding each hand-rolled this set, and both independently omitted bootstrap_complete — the field the KV envelope treats as load-bearing and EvaluateReadinessGate consumes. A second producer shape (backlog) would have made four copies with the same drift mode.
Set is the single projection site, so a field added to IndexStatusResponse cannot be wired in three producers and forgotten in a fourth.
func NewGauges ¶
func NewGauges(names ProducerNames) *Gauges
NewGauges builds the readiness gauge set for one producer.
The emitted metric NAMES are an external contract — dashboards and operator alerts consume them, and the graph-index-readiness spec explicitly protects graph-index's published output. They are therefore reproduced here exactly as the two hand-rolled sets emitted them; MetricNames pins that, and a test compares it against the live registry.
func (*Gauges) MetricNames ¶
MetricNames returns the registry metric names this set emits, in registration order. Exported so a producer's test can pin its external contract without reaching into unexported fields.
func (*Gauges) RecordPublishFailure ¶
func (g *Gauges) RecordPublishFailure()
RecordPublishFailure counts one failed GRAPH_STATUS heartbeat write (ADR-083).
func (*Gauges) Register ¶
func (g *Gauges) Register(registry *metric.MetricsRegistry) error
Register registers every collector on registry through metric.RegisterOrGet and keeps the collectors it returns: when another Gauges already registered a metric under the same service and name, this one writes that collector, so its writes are the ones gathered. A nil registry registers nothing: the gauges still take writes, and nothing gathers them. Nothing is registered on Prometheus' global registry.
func (*Gauges) Set ¶
func (g *Gauges) Set(resp graph.IndexStatusResponse)
Set projects one readiness envelope onto the gauges.
THE SINGLE PROJECTION SITE. Every producer funnels through here, so adding a field to IndexStatusResponse means adding it once rather than remembering four places — which is exactly the omission that left bootstrap_complete unexposed on two independently-written producers.
The state one-hot iterates graph.AllIndexStates rather than the states this producer happens to emit, so a new readiness state renders as an explicit 0 instead of a silently absent series.
type Option ¶
type Option func(*Watcher)
Option configures a Watcher.
func WithHeartbeat ¶
WithHeartbeat declares the producer's publish interval, which sets the freshness window (FreshnessMultiplier x heartbeat). Defaults to DefaultHeartbeat.
func WithLogger ¶
WithLogger sets the logger used for bind/decode failures.
type ProducerNames ¶
type ProducerNames struct {
// Service is the metrics-registry key, hyphenated: "graph-index".
Service string
// Subsystem is the Prometheus subsystem, underscored: "graph_index".
Subsystem string
}
ProducerNames carries the two naming vocabularies this repo already uses for the same component.
A struct rather than two string parameters: they differ only in punctuation ("graph-index" vs "graph_index"), so positional arguments are a silent-swap footgun whose failure mode is renaming EVERY metric the producer emits — with no error raised anywhere and dashboards simply going dark. It is also the reason this package refuses to DERIVE one spelling from the other: guessing wrong fails silently, so the caller states both.
type Publisher ¶
type Publisher struct {
// contains filtered or unexported fields
}
Publisher writes one producer's readiness envelope to its GRAPH_STATUS key. One instance per producer; Publish is called from the status tick and is safe to call from any single goroutine.
func NewPublisher ¶
func NewPublisher(bucket StatusWriter, key string) *Publisher
NewPublisher builds a publisher for one producer key (KeyGraphIndex, KeyGraphEmbedding). It returns nil when the wiring is incomplete, and a nil *Publisher's Publish is a safe no-op: readiness reporting must never be the thing that panics the component whose health it reports.
func (*Publisher) Key ¶
Key reports the producer key this publisher writes, for the caller's failure log.
func (*Publisher) Publish ¶
Publish writes the envelope as the KV value for this producer's key.
The value is PLAIN graph.IndexStatusResponse JSON — the same struct fusion decodes, with no BaseMessage wrapper. The payload registry governs polymorphic message publishes on subjects, where a receiver must discriminate a type it did not choose; this is a KV value on a fixed key whose type is fixed by the contract, and wrapping it would break every consumer's plain decode (graph/readiness, pkg/fusion) for nothing.
Callers publish on EVERY tick, unconditionally, without comparing to the last value: the write is the liveness heartbeat that lets a consumer tell "not ready" from "the producer is gone", and skipping unchanged values would make a healthy steady state indistinguishable from a dead one.
Publish sets the envelope's PublishedAt to the wall-clock time of this write, in UTC, RFC 3339 with nanoseconds, replacing whatever the caller passed, so no producer can leave it out or carry an old one forward. It states when the producer wrote, not when anything in the graph was asserted.
The error is returned, never swallowed, so the tick loop can count and log it; the loop must keep ticking regardless, because the next heartbeat is the recovery.
type Reading ¶
type Reading struct {
// Status is the last envelope decoded from the key.
Status graph.IndexStatusResponse
// Raw is the untouched wire value the Status field was decoded from, for a caller
// that must decode into its OWN field-identical struct (pkg/fusion.IndexStatus).
// Handing such a caller only the decoded graph.IndexStatusResponse would force a
// hand-copied field remap, which is exactly the bug that silently dropped
// IndexedRevision/Lag once before and read downstream as a false caught-up. Nil
// when !Known.
Raw []byte
// Fresh reports that the held envelope arrived within FreshnessMultiplier
// heartbeats, was not already older than that when it arrived, and is still the
// key's current value (no undecodable value has replaced it). False means
// UNKNOWN — fail closed.
Fresh bool
// Known reports that an envelope has been received at least once. It separates
// "never saw the producer" (absent bucket/key, standalone deployment) from "the
// feed went quiet", which are different operator problems.
Known bool
// Age is how long ago the held envelope arrived, consumer-local. Zero when
// !Known.
Age time.Duration
// Err is the last bucket/watch/decode failure, if any, for the structured defer
// log. It is cleared by the next good update.
Err error
}
Reading is the consumer-side answer: the last-known envelope plus everything needed to attribute a defer without correlating log lines.
Status is authoritative ONLY when Fresh. When Fresh is false the envelope is last-known-and-aged, or no longer the key's current value (or the zero value if none ever arrived), and is carried for DIAGNOSTICS only — a caller must fail closed, never read the stale bits as index state. That distinction is the whole point: gh#590 cost three investigation cycles because a transport failure wore not-ready's log line.
type StatusReading ¶
type StatusReading struct {
// Status is the last envelope received from the producer.
Status graph.IndexStatusResponse
// Fresh reports that the reading is recent enough to vouch for. False means
// UNKNOWN, which fails closed — not "not ready".
Fresh bool
}
StatusReading is a consumer's local view of a producer's envelope: the envelope itself plus the one fact only the consumer knows — whether it can still vouch for the feed. It is a struct rather than positional arguments so a new consumer-local fact can be added without breaking every adopter.
Reading maps onto it directly; an IN-PROCESS producer that computes its own envelope passes Fresh: true, having no transport to lose.
type StatusWriter ¶
StatusWriter writes one status value. jetstream.KeyValue satisfies it; taking the one method the publisher needs keeps the seam narrow and the unit tests honest.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher holds the last-known readiness envelope for ONE producer key. One instance serves every gate decision in a consumer: Read is safe from any goroutine, and Run runs on a goroutine the owning component starts and stops by ending Run's context.
func NewWatcher ¶
func NewWatcher(src BucketSource, key string, opts ...Option) *Watcher
NewWatcher builds a watcher for one producer key (KeyGraphIndex, KeyGraphEmbedding) in the GRAPH_STATUS bucket. It performs no I/O; call Run.
func (*Watcher) Read ¶
Read returns the current reading. It never blocks on I/O — the answer comes from held state, which is the point of watching rather than polling.
func (*Watcher) Run ¶
Run watches the key until ctx ends, then returns ctx.Err(). The caller runs it on a goroutine it owns; Run starts no goroutine of its own and releases the KV watch before it returns, so its return is the join.
Run does NOT fail when the bucket or key is absent: a deployment running a consumer without its producer is a legitimate configuration whose readiness is simply UNKNOWN, and Read reports it as such (fail closed) while Run keeps trying to bind, rebinding after any failure. The caller's own escape hatch (e.g. allow_ungated_reads) is what covers that case — never a fabricated envelope. Misconfiguration is refused at the call instead: a nil context, a nil bucket source or an empty key returns an error before any I/O.
func (*Watcher) WaitForFirst ¶
WaitForFirst blocks until an envelope has been received on the key, or ctx ends. It returns nil once something has arrived (fresh or not — Read is what judges that) and ctx.Err() otherwise.
It exists for lazily bound consumers such as fusion: they bind the watch on their first gate decision, and without a bounded wait that first decision would fail closed purely because the watch had not delivered yet — a regression against the request/reply this replaces, which always got an answer or an error. Waiting once at bind, bounded by the same timeout the request used, keeps the worst-case latency of the first gated call unchanged while every later call becomes a local read. Callers must wait AT MOST once: with no producer deployed nothing will ever arrive, and a per-call wait would turn the standalone deployment into a per-call stall.