Documentation
¶
Overview ¶
Package graphview provides a shared read-side fan-out primitive over a NATS KV bucket (ADR-081): ONE WatchAll feeding one validated in-memory current-state projection, coalesced to a view-rate tick and fanned out to N local subscribers with snapshot+delta consistency, per-subscriber at-most-once backpressure, and an honest degraded-path contract.
It exists to avoid the O(N x writeRate) cost of one watcher per client: each write is serialized, decoded, and contract-validated exactly once regardless of subscriber count. The view is domain-agnostic — the bucket handle and a validating decode func are injected; the decode func receives the key, raw value, and authoritative entry metadata (the server write time) and decides what a servable record is ((T, keep, err): keep=false maps the key to absent, err poisons the key per SemStreams ADR-079 semantics).
Coherence: why fan-out is one critical section (G1, the tick seam) ¶
A materialized view IS a cache, so a read-through cache's read-after-write coherence class applies here at two seams:
Attach seam: {snapshot capture + subscriber registration} is atomic with delta application at one view sequence S under the projection mutex. The snapshot reflects every applied change at <= S; the subscriber receives exactly the changes > S. Snapshot capture holds the lock (bounded map copy); snapshot delivery happens after unlock.
Tick seam: a coalescer that fires its callbacks OUTSIDE its lock lets a detached batch holding K@R5 be enqueued to a subscriber that attached with a snapshot already at K@R6 — stale delivery. This package closes that seam STRUCTURALLY: batch detach, subscriber-set iteration, and per-subscriber enqueue are ONE critical section under the projection mutex (fanOut). A detached batch never exists outside the lock, so no apply and no attach can interleave between capture and enqueue — the stale-delivery interleaving is unrepresentable rather than guarded at each enqueue site (a guard every future enqueue path would have to remember).
The work under the lock is bounded map writes (changed keys x subscribers) — never a channel send, decode, or I/O. Actual delivery to subscriber channels happens in per-subscriber goroutines outside the lock, so a slow subscriber degrades to staleness (its pending deltas coalesce last-writer-wins per key, bounded by live changed-key cardinality) and never blocks the watcher, the projection apply, or its peers.
Degraded paths (G5/G6) ¶
Readiness is caught-up, not started: attach and point reads before the initial WatchAll replay completes fail with ErrNotReady. A view runs once: its owner calls Run on a goroutine of its own, and Run holds the watch until its context ends or the watch is lost. Ending the context is a stop: every subscriber's delta channel closes with ErrViewStopped surfaced via Err(), and Run returns the context's error. Loss of the shared watcher (its update channel closing while Run's context is live) fails closed: every subscriber's delta channel closes with ErrWatcherLost surfaced via Err(), the frozen projection is never served as live, and Run returns the loss. The owner replaces a view by constructing a new one, which starts from an empty projection and serves only what its own replay delivers. Decode/contract failures surface as typed per-key *PoisonError signals in the delta lane and on point reads; they never launder through as upserts and never halt delivery for unrelated keys.
The view coexists with raw WatchAll: consumers needing independent filters, historical replay, per-revision delivery, or independent ack semantics keep opening real JetStream consumers.
Index ¶
- Variables
- type DecodeFunc
- type Delta
- type DeltaOp
- type Entry
- type EntryMeta
- type Hooks
- type KeyedEntry
- type Option
- type PoisonError
- type Snapshot
- type Subscription
- type View
- func (v *View[T]) Applied() (revision uint64, appliedAt time.Time)
- func (v *View[T]) Get(key string) (T, uint64, error)
- func (v *View[T]) List(prefix string, limit int) ([]KeyedEntry[T], error)
- func (v *View[T]) ListCounted(prefix string) ([]KeyedEntry[T], int, error)
- func (v *View[T]) Run(ctx context.Context) error
- func (v *View[T]) SnapshotAndSubscribe(ctx context.Context) (Snapshot[T], *Subscription[T], error)
- func (v *View[T]) Subscribe(ctx context.Context) (*Subscription[T], error)
- func (v *View[T]) WaitCaughtUp(ctx context.Context) error
- type WatcherSource
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotReady gates attach and point reads before the initial WatchAll // replay has completed (readiness = caught-up, not started) and after a // watcher loss (fail-closed is a not-ready state). ErrNotReady = errors.New("graphview: view not ready") // ErrWatcherLost marks the fail-closed state after the shared watcher // died (its update channel closed while Run's context was live) or // could not open. The frozen projection is never served as live. ErrWatcherLost = errors.New("graphview: watcher lost") // ErrViewStopped is returned by every operation once Run's context has // ended, and by Run on a view whose Run has returned; it is the terminal // reason reported by Subscription.Err after a stop. ErrViewStopped = errors.New("graphview: view stopped") // ErrKeyNotFound is returned by Get for a key absent from the projection. ErrKeyNotFound = errors.New("graphview: key not found") // ErrAlreadyStarted is returned by Run on a view whose Run is running. ErrAlreadyStarted = errors.New("graphview: view already started") )
Sentinel errors returned by view operations. Failed-state errors wrap BOTH ErrNotReady and ErrWatcherLost so callers can gate on not-ready semantics or distinguish "wait for bootstrap" from "the watcher died; restart".
Functions ¶
This section is empty.
Types ¶
type DecodeFunc ¶
type DecodeFunc[T any] func(key string, value []byte, meta EntryMeta) (decoded T, keep bool, err error)
DecodeFunc decodes and validates one stored KV value; meta carries the entry's server write time. keep=false means the key is not part of the view (non-record keys, present-but-not-ready records) and maps it to absence; a non-nil err poisons the key (G6) — the value is never delivered as an upsert while other keys keep flowing. Decode runs exactly once per delivered write, on Run's goroutine, regardless of subscriber count.
type Delta ¶
type Delta[T any] struct { // Op is the operation kind. Op DeltaOp // Key is the KV key. Key string // Value is the decoded value; set only for DeltaUpsert. Value T // Revision is the KV revision of the operation. Revision uint64 // Created is the KV server timestamp of the operation (entry.Created()), // carried for every op including tombstones — deletes have no decoded T, // so the delta lane is their only channel for the write time. Created time.Time // Err is the *PoisonError; set only for DeltaPoison. Err error }
Delta is one coalesced per-key operation delivered to subscribers: at most one per changed key per tick window, carrying the greatest-revision operation observed in that window. Values are shared across subscribers (for pointer-shaped T, do not mutate delivered values).
type DeltaOp ¶
type DeltaOp uint8
DeltaOp is the kind of a Delta: upsert, delete (tombstone), or poison.
const ( // DeltaUpsert carries the newest decoded value for a key. DeltaUpsert DeltaOp = iota // DeltaDelete is a tombstone: the key converged to absence. DeltaDelete // DeltaPoison is a typed per-key poison signal; Delta.Err holds the // *PoisonError. DeltaPoison )
Delta operation kinds. Tombstones and poison signals ride the same ordered, coalesced lane as upserts — last-writer-wins by revision (G4).
type Entry ¶
type Entry[T any] struct { // Value is the decoded value (shared, not copied per subscriber). Value T // Revision is the KV revision that produced the value. Revision uint64 }
Entry is a projection value in a Snapshot.
type EntryMeta ¶
type EntryMeta struct {
// Created is the KV server timestamp of the write (entry.Created()).
Created time.Time
}
EntryMeta carries the authoritative KV entry metadata into the validating decode: the server write timestamp (entry.Created()), so decoded records that embed their own write time need not reach back into the delta lane for it.
type Hooks ¶
type Hooks struct {
// OnApply fires after every delivered entry has been applied (including
// skips and tombstones), with the entry's key and KV revision.
OnApply func(key string, revision uint64)
// OnCaughtUp fires when the view transitions to caught-up (initial
// bootstrap of this instance).
OnCaughtUp func()
// OnWatcherLost fires once per watcher loss with the cause.
OnWatcherLost func(err error)
// OnPoison fires per poisoned write with the key and *PoisonError.
OnPoison func(key string, err error)
// OnSubscribers fires with the new subscriber count after every attach
// and every effective detach (Unsubscribe, subscription-context cancel,
// and the batch detaches at a stop and a watcher loss, which report zero).
// Counts are delivered outside the projection mutex, so under concurrent
// attach/detach, or across a stop, two callbacks may arrive out of order:
// the last value is not promised to be the current count. A consumer that
// needs zero once the view has ended sets it when Run returns.
OnSubscribers func(n int)
// OnFanOut fires after every tick window that enqueued at least one
// changed key to at least one subscriber, with the number of pending
// deltas that were overwritten before delivery across all subscribers in
// the window (the at-most-once coalescing drop on undrained buffers —
// the slow-subscriber staleness signal) and the largest per-subscriber
// pending-buffer size after enqueue.
OnFanOut func(overwritten, maxPending int)
}
Hooks receives view-internal signals. All callbacks are optional, are invoked outside the projection mutex, and MUST be fast and non-blocking — they run on Run's goroutine, or on the goroutine of an attach or detach, and a blocking callback holds Run. This is the observability seam for metrics.
type KeyedEntry ¶
type KeyedEntry[T any] struct { // Key is the KV key. Key string // Value is the decoded value. Value T // Revision is the KV revision that produced the value. Revision uint64 }
KeyedEntry is a projection value returned by List.
type Option ¶
type Option func(*options)
Option configures a View at construction.
func WithTickInterval ¶
WithTickInterval sets the fan-out coalescing window (default 250 ms). Non-positive values fall back to the default.
type PoisonError ¶
type PoisonError struct {
// Key is the poisoned KV key.
Key string
// Revision is the KV revision whose value failed decode.
Revision uint64
// Err is the decode/contract error from the injected DecodeFunc.
Err error
}
PoisonError is the typed per-key poison signal (G6, SemStreams ADR-079 semantics): a stored value for Key failed the injected validating decode at Revision. It is surfaced in the delta lane (Delta.Op == DeltaPoison, Delta.Err), on point reads of the poisoned key, and in Snapshot.Poisoned — never as a normal upsert. It wraps the decode/contract error so consumer poison latches can errors.As through to their contract-error types.
func (*PoisonError) Unwrap ¶
func (e *PoisonError) Unwrap() error
Unwrap exposes the underlying decode/contract error for errors.Is/As.
type Snapshot ¶
type Snapshot[T any] struct { // Entries maps key to its current decoded value. Entries map[string]Entry[T] // Poisoned maps key to its sticky poison signal (G6) so consumers that // attach after a poison event can still drive their latches. Poisoned map[string]*PoisonError // Revision is the applied-revision watermark at capture. Revision uint64 // AppliedAt is the KV server write time of the Revision watermark — how // current this snapshot was when it was captured, taken in the same // critical section as Entries and Revision so it names the same // instant. It is REPORTING ONLY (nothing in this package gates on it) and // is the zero time when currency is not computable; see View.Applied for // the floor, zero-value, and clock-skew contract. AppliedAt time.Time }
Snapshot is the consistent projection copy returned by SnapshotAndSubscribe: every change the view applied before the attach is present (a deleted key is absent), and the paired subscription delivers exactly the changes after it — no gap, no duplicate, no inversion (G1).
type Subscription ¶
type Subscription[T any] struct { // contains filtered or unexported fields }
Subscription is one subscriber's attachment to a View. Deltas() delivers coalesced batches; channel close is the explicit terminal signal and Err() reports why (nil after Unsubscribe, the subscription context's error after its cancellation, ErrViewStopped after a stop, a wrapped ErrWatcherLost after a watcher loss). A subscription that detached first keeps its own reason. Slowness never disconnects a subscription: an undrained subscriber's pending deltas coalesce last-writer-wins per key until it resumes.
func (*Subscription[T]) Deltas ¶
func (s *Subscription[T]) Deltas() <-chan []Delta[T]
Deltas returns the delivery channel. Each batch holds at most one delta per key (the greatest-revision operation), sorted by revision ascending. The channel closes on Unsubscribe, subscription-context cancellation, a view stop, or watcher loss — check Err() after close. After Unsubscribe or the subscription's context ends, a batch already taken may still be delivered before the close.
func (*Subscription[T]) Err ¶
func (s *Subscription[T]) Err() error
Err reports the terminal reason once Deltas has closed: nil for a clean Unsubscribe, the context error for a context detach, ErrViewStopped for view shutdown, or a wrapped ErrWatcherLost staleness signal.
func (*Subscription[T]) Unsubscribe ¶
func (s *Subscription[T]) Unsubscribe()
Unsubscribe detaches the subscription and releases its buffered state. Idempotent; safe to call concurrently with delivery. It does not wait for the delivery goroutine; Run does.
type View ¶
type View[T any] struct { // contains filtered or unexported fields }
View maintains one authoritative in-memory projection of a KV bucket from a single WatchAll and fans coalesced deltas out to local subscribers. It is explicitly constructed and owned — inject it into consumers; there is no process-global registry. Its owner calls Run once, on a goroutine of its own. All methods are safe for concurrent use.
func New ¶
func New[T any](source WatcherSource, decode DecodeFunc[T], opts ...Option) (*View[T], error)
New constructs a View over source with the injected validating decode. The view does not touch the bucket until Run.
func (*View[T]) Applied ¶
Applied returns the applied-revision watermark together with the KV SERVER WRITE TIME (entry.Created()) of the newest revision the view has applied — the "this view reflects the bucket as of T" pair. It is not the time graph-index projects into the ADR-083 staleness_ms field: that is readiness.IndexStatusInputs.OldestUnappliedAt (graph/readiness), the commit time of the oldest delivered write graph-index has not yet applied.
Both values are read in ONE critical section, so the revision and the timestamp always name the same write; reading them through two accessors could observe a torn pair.
REPORTING ONLY — no graphview API gates on the timestamp, and none may grow one. This package's gates stay exactly the two it has: ErrNotReady until the initial replay completes (still building) and fail-closed on watcher loss (broken). Age is neither, so a healthy view that is merely behind serves and the CONSUMER stamps currency on its own output, the way community detection reports staleness_at_detection_ms alongside its answer.
It is a FLOOR on currency, not an oracle. Only a DELIVERED write can age it, so a feed that stalls entirely upstream leaves the pair frozen and looks arbitrarily current by this measure alone. Detecting a total stall is the watcher-loss path's job, not this value's; graph.IndexStatusResponse.StalenessMs carries the same caveat for the same reason.
appliedAt is the ZERO time until something has been applied, and stays zero rather than being fabricated as time.Now(). A caller computing an age MUST check IsZero and treat it as "currency not computable" — the contract readiness.IndexStatusInputs.OldestUnappliedAt uses, where the projection deliberately declines to make an uncomputable view look fresh. It is zero for the same reason when the newest applied write carried no server timestamp (real KV entries always do; fakes and synthetic replays need not).
Subtracting it from a local time.Now() crosses a NATS server clock against a local one and is off by the skew between them. Accepted, the same trade as ADR-083 decision 3: the alternative — measuring currency in revisions — is wrong by multiples under a coalescing change alone.
func (*View[T]) Get ¶
Get is the coherent point read: it serves the projection under the same mutex applies run under, so it can never return a value older than one the view already applied. It honors readiness (ErrNotReady before caught-up, wrapped ErrWatcherLost after a loss, ErrViewStopped after a stop) and poison (a poisoned key returns its *PoisonError, never a stale value).
func (*View[T]) List ¶
func (v *View[T]) List(prefix string, limit int) ([]KeyedEntry[T], error)
List returns projection entries whose key has the given prefix (empty prefix matches all), sorted by key, truncated to limit when limit > 0. The complete match set is sorted before the limit is applied (deterministic bounded reads). Poisoned keys are excluded — they have no servable value. It honors the same readiness gate as Get.
func (*View[T]) ListCounted ¶
func (v *View[T]) ListCounted(prefix string) ([]KeyedEntry[T], int, error)
ListCounted returns what List(prefix, 0) returns and the number of poisoned keys with the prefix, both read under one hold of the projection, so the count describes exactly the listing beside it: every key the view holds under the prefix is either listed or counted, never both; a key the decode skips is held as neither. It honors the same readiness gate as Get. The entries' values are the stored ones, which an apply replaces whole and never edits; callers read them only.
func (*View[T]) Run ¶
Run holds the view's one WatchAll on the caller's goroutine until ctx ends or the watch is lost (the background-work spec's Run; the component that runs it owns ctx, ADR-113). A view runs once: Run refuses a nil context before touching the source, returns ErrAlreadyStarted while another Run runs, and ErrViewStopped once a Run has returned, for any reason. A replacement is a new view.
Run watches under a context it derives from ctx, and reads the watch and the tick window itself. Ending ctx is a stop: reads and attaches return ErrViewStopped, every attached subscription closes with ErrViewStopped, OnWatcherLost does not fire, and Run returns ctx.Err(). Ending the watch's context closes its update channel too (measured at nats.go v1.54.0; not promised), so a closed channel is a loss only while ctx is live. On a loss, reads and attaches return an error matching ErrNotReady and ErrWatcherLost, every attached subscription closes with an error matching ErrWatcherLost, OnWatcherLost fires once after that, and Run returns an error matching ErrWatcherLost. A WatchAll that fails returns ctx.Err() when ctx has ended, else its error, with the view failed and OnWatcherLost not called.
Run returns only after every subscription goroutine and every hook call an attach or detach made has returned, and after the watch has been stopped through kvwatch.Stop, which reads the update channel until nats.go closes it, for at most kvwatch.DrainBudget; when that budget runs out the error is joined with kvwatch.ErrDeliveryOpen. A blocking hook or decode holds Run; the composing component bounds its own wait.
func (*View[T]) SnapshotAndSubscribe ¶
SnapshotAndSubscribe atomically captures a snapshot and registers the subscription at one view sequence S under the projection mutex (G1): the snapshot holds every change applied at <= S, the subscription delivers exactly the changes > S. Capture holds the lock (bounded copy); delivery to the caller happens after unlock. ctx bounds the subscription: its cancellation detaches like Unsubscribe. Fails with ErrNotReady until the view is caught up.
func (*View[T]) Subscribe ¶
func (v *View[T]) Subscribe(ctx context.Context) (*Subscription[T], error)
Subscribe is the delta-only attach for trigger-shaped consumers: no snapshot, no second projection copy — the subscription delivers the changes after the attach sequence. Same readiness gate as SnapshotAndSubscribe.
type WatcherSource ¶
type WatcherSource interface {
WatchAll(ctx context.Context, opts ...jetstream.WatchOpt) (jetstream.KeyWatcher, error)
}
WatcherSource is the narrow method set the view needs from a KV bucket. jetstream.KeyValue satisfies it; unit tests inject a deterministic fake.