Documentation
¶
Overview ¶
Package embedding turns the text of graph entities into vectors, stores them, and finds the nearest ones. It needs no external provider: its embedder, BM25Embedder, runs in the process.
Embedding is two hops. Hop 1 is a component outside this package that writes one pending record per entity revision to the EMBEDDING_INDEX bucket (Storage.SavePendingGuarded). Hop 2 is the Worker, which turns each pending record into a generated one holding the vector, or a failed one holding a reason. The Worker is what this package runs; the hop-1 component is processor/graph-embedding, which is not part of this package.
Parts ¶
Embedder is the interface an embedding source implements, for documents (Generate) and for search queries (GenerateQuery). BM25Embedder is the tier-0 implementation: it hashes terms into a fixed number of dimensions and weights them with BM25, so it matches shared words, not meaning. A product that wants neural vectors supplies its own Embedder.
Storage reads and writes two NATS KV buckets: EMBEDDING_INDEX, one record per entity, and EMBEDDING_DEDUP, one vector per content key so identical text is embedded once. Every read-modify-write is a compare-and-set through natsclient.KVStore.UpdateWithRetryRead: the decision is made from the bytes and revision it read, and a lost race is retried. Saves are ordered by source revision (the ENTITY_STATES revision a record was made from): a save for an older revision never replaces a generated record for a newer one. SaveGenerated and SaveFailed refuse such a save with ErrSupersededRevision (or ErrRecordGone for an absent record) and write nothing; SavePendingGuarded reports it as not saved, with a nil error. None of these is a failure.
When a record's text resolves to nothing, the Worker removes the record only if its key is still at the KV revision the Worker read; if newer work has moved the key, nothing is removed and the outcome is OutcomeDeleted.
VectorCache keeps the generated vectors in memory on a graphview.View of EMBEDDING_INDEX, and FindSimilar ranks them by cosine similarity and counts the records it could not decode, which it does not score. Before the view has read the bucket's initial contents, after the watch was lost, and after Run returned, FindSimilar returns an error matching graphview.ErrNotReady.
Running ¶
Worker.Run(ctx) and VectorCache.Run(ctx) each block until ctx ends or their watch is lost, and return ctx.Err() or an error saying the watch was lost. The caller owns ctx and ends it to stop them. Each refuses a nil context and a second Run while one is running. A lost watch is an error to act on, not a normal stop: the caller decides whether to report it or run again.
Ordering ¶
The Worker hands each watch entry to a keyed pool (internal/dispatch.KeyedPool), keyed by entity ID. One entity's entries are processed one at a time, in the order the watch delivered them, from reading the record through the store write to GeneratedCallback and TerminalCallback. Different entities run in parallel on the WithWorkers lanes (default 5). Callbacks for different entities are unordered. The package promises nothing about the order the watch delivers entries in; the revision checks in Storage are what keep an older save from replacing a newer generated record.
Where this differs from SemStreams ¶
The package is a port of SemStreams' graph/embedding without its HTTP embedder. The changes are listed in the admission ledger row for graph/embedding.
Index ¶
- Constants
- Variables
- func CosineSimilarity(a, b []float32) float64
- func DedupKey(id EmbedderIdentity, text string) string
- func InProcessDedupKey(id EmbedderIdentity, text string) string
- func ScoredEntityLess(a, b ScoredEntity) bool
- type BM25Config
- type BM25Embedder
- type DedupRecord
- type Embedder
- type EmbedderIdentity
- type FailedEntry
- type GeneratedCallback
- type Record
- type ScoredEntity
- type Status
- type Storage
- func (s *Storage) DeleteEmbedding(ctx context.Context, entityID string) error
- func (s *Storage) GetByContentHash(ctx context.Context, contentHash string) (*DedupRecord, error)
- func (s *Storage) GetEmbedding(ctx context.Context, entityID string) (*Record, error)
- func (s *Storage) SaveDedup(ctx context.Context, contentHash string, vector []float32, model string, ...) error
- func (s *Storage) SaveFailed(ctx context.Context, entityID, errorMsg, reason string, sourceRevision uint64) error
- func (s *Storage) SaveGenerated(ctx context.Context, entityID string, vector []float32, model string, ...) error
- func (s *Storage) SavePendingGuarded(ctx context.Context, record *Record) (saved bool, err error)
- func (s *Storage) ScanFailed(ctx context.Context) ([]FailedEntry, error)
- type StorageRef
- type StoreResolver
- type TerminalCallback
- type TerminalOutcome
- type VectorCache
- type Worker
- func (w *Worker) Run(ctx context.Context) error
- func (w *Worker) WithEmbedderType(t string) *Worker
- func (w *Worker) WithMaxSourceTextLen(n int) *Worker
- func (w *Worker) WithMetrics(m WorkerMetrics) *Worker
- func (w *Worker) WithOnGenerated(cb GeneratedCallback) *Worker
- func (w *Worker) WithOnTerminal(cb TerminalCallback) *Worker
- func (w *Worker) WithStoreResolver(r StoreResolver) *Worker
- func (w *Worker) WithWorkers(n int) *Worker
- type WorkerMetrics
Constants ¶
const ( // ReasonDeleteFailed marks an entity whose derived-record delete failed // (tombstone or reconcile-observed absence); the record is still present // and queryable until repair converges it. ReasonDeleteFailed = "delete_failed" // ReasonPendingWriteFailed marks an entity whose hop-1 pending write // (SavePendingGuarded) failed; the revision stays uncompleted // (#613 F2) and repair re-queues from authoritative state. ReasonPendingWriteFailed = "pending_write_failed" // ReasonEntityReadFailed marks an entity whose authoritative ENTITY_STATES // read failed transiently during a reconcile; repair re-reads until the // source answers. ReasonEntityReadFailed = "entity_read_failed" )
Derived-write/read failure reasons (#625). They classify entries in graph-embedding's PROCESS-LOCAL current-failed map when a hop-1 derived write or authoritative read fails — a failed EMBEDDING_INDEX delete, a failed pending write, a failed ENTITY_STATES read — so those failures surface as `degraded` immediately and are re-driven by the background repair loop.
IN-MEMORY ONLY, by design. These reasons are NEVER persisted into a stored Record: they must not be passed to SaveFailed (a derived-write failure means the durable state could not be brought to truth — writing a durable "failed" record about it is the same class of write and would race the same fault), and they are deliberately NOT members of normalizeFailureReason's closed enum — a persisted-but-unreachable enum member is a phantom signal. The bounded-reasons contract of the readiness envelope still holds: this is a closed const set.
const MaxSourceTextLenCeiling = 1_000_000
MaxSourceTextLenCeiling is the upper bound on the text cap, in runes. A larger configured cap is clamped to it on both lanes, so the offloaded lane's byte budget (utf8.UTFMax*cap+1) can never overflow into a negative io.LimitReader bound, which would read an empty body and remove the pending record as having no text. 1,000,000 characters is far past any real embedding input while bounding the worst-case offloaded read at utf8.UTFMax MB.
Variables ¶
var ErrRecordGone = errors.New("embedding index record no longer exists")
ErrRecordGone reports that the EMBEDDING_INDEX record a save was meant to UPDATE no longer exists, so the save was dropped without writing.
It is a normal outcome, not a fault: the hop-1 entity tombstone deletes an entity's index key while a hop-2 worker may still be inside an embedder round trip for that same entity. Callers should treat it as "this entity is no longer supposed to have an embedding" and stop — in particular they must not report it as a generation failure or fire the generated callback, which would push a vector for a dead entity into caches.
var ErrSupersededRevision = errors.New("embedding index write superseded by a newer source revision")
ErrSupersededRevision reports that the save was dropped because a newer source revision's outcome already landed for this entity (the ordering guard). Like ErrRecordGone it is a normal, non-failure outcome: the caller must NOT fire the generated callback, because the vector it holds is the OLDER revision's and firing would push a stale vector into any WithOnGenerated consumer's cache. Callers must not report it as a failure either; the newer outcome is authoritative.
Functions ¶
func CosineSimilarity ¶
CosineSimilarity computes the cosine similarity between two vectors.
Returns a value between -1 and 1, where:
- 1 means vectors are identical
- 0 means vectors are orthogonal (unrelated)
- -1 means vectors are opposite
Formula: cos(θ) = (A · B) / (||A|| × ||B||)
func DedupKey ¶
func DedupKey(id EmbedderIdentity, text string) string
DedupKey derives the EMBEDDING_DEDUP key for text under a given embedder identity, or "" when the identity is not yet resolved.
Every field is length-prefixed before hashing so that no two distinct field tuples can serialize to the same byte stream (without prefixes, {"ab","c"} and {"a","bc"} would collide).
An identity with a non-positive Dimensions has not learned its vector width yet (a remote embedder resolves it from its first response) and gets NO key. "" is the package's established "dedup disabled for this record" signal, which both hops already honour: it is stored as the pending record's ContentHash and getOrGenerateEmbedding generates unconditionally for it. The alternative of keying on a placeholder width is what split EMBEDDING_DEDUP into two disjoint keyspaces per embedder; omitting Dimensions from the key instead would be worse still, since keys would then collide across models of different width — exactly what folding identity into the key was meant to prevent. The cost is redundant embeds during the brief unresolved window, once per process.
func InProcessDedupKey ¶
func InProcessDedupKey(id EmbedderIdentity, text string) string
InProcessDedupKey derives a STABLE process-local key for collapsing concurrent byte-identical embedder calls (#630), EXCLUDING the learned Dimensions.
Unlike DedupKey it ALWAYS returns a key — even before the embedder has resolved its vector width (a remote embedder reports Dimensions()==0 until its first response). That is deliberate: at cold start DedupKey withholds a key, so a burst of K workers holding identical content would otherwise each issue its own paid Generate call. Keying the in-process singleflight on this width-independent value collapses them to one call, while the DURABLE dedup cache stays correctly withheld (a vector must not be stored under a wrong-dimensions key).
Excluding Dimensions is safe here precisely because this key never leaves the process: the embedder — and therefore the width it will eventually resolve — is the SAME instance across every worker goroutine, so two workers with identical content are guaranteed to be producing vectors in the same space regardless of whether the width is known yet.
func ScoredEntityLess ¶
func ScoredEntityLess(a, b ScoredEntity) bool
ScoredEntityLess is the CANONICAL ordering for similarity results: by similarity descending, then entity ID ascending as a total-order tie-break. Every similarity path sorts through this ONE comparator so that an equal-similarity group straddling the top-k boundary resolves to the SAME subset regardless of map- or KV-enumeration order. Sorting by similarity alone leaves the k-boundary tie to Go's randomized map iteration or the KV's key-delivery order, which yields nondeterministic neighbor sets downstream. Entity IDs are unique, so this is a strict total order — deterministic even under the non-stable sort.Slice.
Types ¶
type BM25Config ¶
type BM25Config struct {
// Dimensions is the output embedding dimension (default: 384 for compatibility)
Dimensions int
// K1 controls term frequency saturation (default: 1.5)
// Higher values give more weight to term frequency
K1 float64
// B controls length normalization (default: 0.75)
// B=1.0 means full normalization. B=0 selects the default 0.75, so no
// normalization cannot be configured.
B float64
}
BM25Config configures the BM25 embedder.
type BM25Embedder ¶
type BM25Embedder struct {
// contains filtered or unexported fields
}
BM25Embedder implements pure Go lexical embeddings using BM25 algorithm.
It is the tier-0 embedder: it needs no external provider. It uses BM25 (Best Matching 25) scoring - a term-frequency based ranking function widely used in information retrieval.
The embedder generates fixed-dimension vectors by:
- Tokenizing text (lowercase, split on non-alphanumeric)
- Computing term frequencies
- Hashing terms to fixed dimensions (feature hashing)
- Applying BM25 weighting (TF with IDF and length normalization)
- L2 normalizing for cosine similarity compatibility
Parameters:
- k1: Controls term frequency saturation (default 1.5)
- b: Controls document length normalization (default 0.75)
This is a lexical approach - it won't understand semantic similarity like neural models, but provides reasonable results for exact term matches and common phrases.
func NewBM25Embedder ¶
func NewBM25Embedder(cfg BM25Config) *BM25Embedder
NewBM25Embedder creates a new BM25-based embedder.
func (*BM25Embedder) Close ¶
func (b *BM25Embedder) Close() error
Close releases resources (no-op for BM25).
func (*BM25Embedder) Dimensions ¶
func (b *BM25Embedder) Dimensions() int
Dimensions returns the dimensionality of embeddings.
func (*BM25Embedder) Generate ¶
Generate creates BM25-based embeddings for the given texts.
This updates internal document statistics incrementally, so the embedder "learns" vocabulary and IDF scores from all texts it processes.
func (*BM25Embedder) GenerateQuery ¶
GenerateQuery embeds query-side text. BM25 is a symmetric bag-of-words model — there is no query/document asymmetry, so the returned vector is what Generate would produce for the same text against the same statistics (gh#438).
It is READ-ONLY over the corpus statistics (gh#619). Searching is not observing a document: folding the query's terms into docCount/termDocCount and its (typically 2-4 token) length into avgDocLength shifted the IDF weights and the length-normalization denominator for every document embedded afterwards, so a graph's rankings drifted as a function of how many times it had been searched. Generate remains the only mutator.
This does NOT make BM25 a correct lexical index — the statistics are still process-local, unpersisted, and order-dependent across restarts. That is the open decision in gh#619 (real index over an immutable snapshot vs. a stateless hashed TF vector); this only stops searches from corrupting the corpus.
func (*BM25Embedder) Model ¶
func (b *BM25Embedder) Model() string
Model returns the model identifier.
type DedupRecord ¶
type DedupRecord struct {
Vector []float32 `json:"vector"`
FirstGenerated time.Time `json:"first_generated"`
Model string `json:"model,omitempty"`
Dimensions int `json:"dimensions,omitempty"`
}
DedupRecord stores content-addressed embeddings for deduplication.
Model and Dimensions record WHICH vector space the stored vector belongs to. The dedup KEY already folds in embedder identity (see DedupKey), so a mismatch here should be unreachable; carrying the fields anyway makes a stale record detectable after the fact instead of silently servable. FirstGenerated has no reader in code; it is what an operator sees when listing the bucket.
type Embedder ¶
type Embedder interface {
// Generate creates embeddings for the given texts (the DOCUMENT side — what is
// embedded at ingest).
//
// This is the primary method - batch operations are natural for all providers.
// For single text, pass a slice with one element.
// Returns a slice of float32 slices, where each inner slice is an embedding vector.
Generate(ctx context.Context, texts []string) ([][]float32, error)
// GenerateQuery creates embeddings for QUERY-side text (semantic search),
// as opposed to Generate which embeds documents at ingest. Asymmetric retrieval
// models (Snowflake arctic-embed, BGE, E5) are trained to embed a query with an
// instruction prefix while documents are embedded raw; omitting the query prefix
// is a measured retrieval-quality cliff, not a rounding error (gh#438). Symmetric
// embedders (BM25) implement this identically to Generate.
GenerateQuery(ctx context.Context, texts []string) ([][]float32, error)
// Dimensions returns the dimensionality of embeddings produced by this embedder,
// or 0 if it is not yet known.
//
// For example, all-MiniLM-L6-v2 produces 384-dimensional vectors.
//
// Implementations that discover the width from a remote endpoint report 0 until
// the first successful response. Implementations MUST return 0
// rather than a plausible default: callers fold this value into durable
// content-addressed keys (DedupKey), where a guessed width silently partitions
// the dedup bucket into pre-guess and post-guess halves. Callers must treat 0
// as "no stable vector space yet" and skip keyed dedup entirely.
//
// This value must be safe to read concurrently with Generate — the graph-embedding
// component reads it from the hop-1 watcher goroutine and from every hop-2 worker.
Dimensions() int
// Model returns the model identifier used by this embedder.
//
// This is useful for debugging and logging which model is being used.
Model() string
// Close releases any resources held by the embedder.
//
// Must be called when the embedder is no longer needed. For a remote provider
// this is typically a no-op; a local model releases its GPU/CPU resources here.
Close() error
}
Embedder generates vector embeddings for text.
Implementations can use different providers (BM25 here, or a product's own) while maintaining a consistent interface. Every implementation takes a batch of texts.
type EmbedderIdentity ¶
type EmbedderIdentity struct {
// Type is the configured embedder kind ("bm25", or a product embedder's name).
Type string
// Model is the embedder's model identifier.
Model string
// Dimensions is the vector width the embedder produces.
Dimensions int
// MaxTextLen is the source-text truncation cap applied before generation.
// It belongs in the key because the cap is derived from the embedder type
// (4000 for bm25, 8000 otherwise): flipping type changes the TEXT that gets
// embedded, not only the model that embeds it.
MaxTextLen int
}
EmbedderIdentity captures everything a stored vector depends on BESIDES the input text. Two embedders that disagree on any of these fields produce vectors in different, incomparable vector spaces.
It exists because EMBEDDING_DEDUP is durable, untimed, and never cleared: an operator who switches embedder_type from bm25 to another embedder against the same NATS state would otherwise get every already-embedded entity's OLD vector back, stamped with the NEW model's name. When both spaces happen to share a dimension count (384 is the default for both BM25 here and TEI's all-MiniLM-L6-v2), cosine similarity returns a plausible-looking score across unrelated spaces and no health signal fires. Folding identity into the key makes that impossible: stale keys simply never match again and age out.
type FailedEntry ¶
FailedEntry is one entity currently in a failed embedding terminal state, as read by the current-failed bootstrap scan: the entity ID, its bounded reason, and the source revision the failure was recorded at.
Reason is ALWAYS a member of the bounded failure enum (or reasonUnknown) — ScanFailed normalizes it, so a stored record can never inject an arbitrary string into the readiness envelope's reason histogram.
SourceRevision is the durable record's revision, seeded into the component's in-memory failed map so its revision check (a superseded older completion must NOT clear a newer failure) starts from the correct baseline after a restart.
type GeneratedCallback ¶
GeneratedCallback is called when an embedding has been stored. It receives the context Run was given, the entity ID and the stored vector, and must return promptly once that context has ended.
It runs on the entity's lane of the Worker's keyed pool: callbacks for one entity, this one and TerminalCallback, arrive in the order the watch delivered its entries to the Worker, each entry's callbacks before the next entry of that entity is processed. A save that was refused (a newer source revision is stored, or the record is gone) fires no generated callback. Callbacks for different entities may run concurrently and are unordered.
type Record ¶
type Record struct {
EntityID string `json:"entity_id"`
Vector []float32 `json:"vector,omitempty"`
ContentHash string `json:"content_hash"`
// SourceText is the INLINE lane's WHOLE embedding text (the identity triples
// extracted at hop 1). It is meaningful ONLY on the inline lane (StorageRef == nil);
// on the offloaded lane it is left EMPTY and the identity prefix travels in
// IdentityText instead.
SourceText string `json:"source_text,omitempty"`
// IdentityText is the inline identity PREFIX (title/.signature/.comment, per the
// configured text suffixes) for the OFFLOADED (StorageRef) lane ONLY: hop 2 embeds it
// AHEAD of the fetched body, identity-first, in one vector so the text suffixes take
// effect on offloaded entities too. Empty on the inline lane, and empty on an
// offloaded record with no inline identity text (body-only).
IdentityText string `json:"identity_text,omitempty"`
Model string `json:"model,omitempty"`
Dimensions int `json:"dimensions,omitempty"`
GeneratedAt time.Time `json:"generated_at,omitempty"`
Status Status `json:"status"`
ErrorMsg string `json:"error_msg,omitempty"` // If status=failed
// Reason is a BOUNDED classification of a failure (status=failed), stored next to
// the raw ErrorMsg. It is the value the failures metric is labelled by — the raw
// ErrorMsg is unbounded and must NEVER be a metric label (cardinality blowup). It is
// also what the current-failed bootstrap scan reads to seed the reason breakdown
// after a restart.
Reason string `json:"reason,omitempty"` // Bounded reason enum when status=failed
// SourceRevision is the ENTITY_STATES stream revision that produced this record.
// It is threaded from the hop-1 watcher so hop 2 can complete the embedding
// readiness watermark at the terminal transition (ADR-066 §3), and it is
// persisted onto generated and failed records so SaveGenerated and SaveFailed can
// order writes by it: a record already carrying a higher SourceRevision has a
// newer outcome, so a late older-revision write is dropped. 0 means unknown and
// is treated as oldest.
SourceRevision uint64 `json:"source_revision,omitempty"`
// StorageRef, when set, points at the entity's body in a content store; the
// Worker fetches the text from there.
StorageRef *StorageRef `json:"storage_ref,omitempty"`
}
Record represents a stored embedding with metadata
type ScoredEntity ¶
ScoredEntity pairs an entity ID with its cosine similarity score.
type Storage ¶
type Storage struct {
// contains filtered or unexported fields
}
Storage handles persistence of embeddings to two NATS KV buckets, EMBEDDING_INDEX and EMBEDDING_DEDUP.
Every read-modify-write goes through natsclient.KVStore.UpdateWithRetryRead, which decides from the bytes and revision it read and retries a revision conflict (ADR-110). The retry budget and the per-call timeout are the KVStore's, set by whoever builds it; when the retries run out a save returns an error matching natsclient.ErrKVMaxRetriesExceeded and has written nothing.
Every method that takes an entity ID refuses one that is not a literal six-part ID (types.ValidateEntityID), and every method that takes a dedup key refuses one that is not a literal KV key (natsclient.ValidateKVLiteralKey), with an invalid-input error before any bucket call.
func NewStorage ¶
func NewStorage(index, dedup *natsclient.KVStore) *Storage
NewStorage creates a new embedding storage instance over the EMBEDDING_INDEX and EMBEDDING_DEDUP buckets.
func (*Storage) DeleteEmbedding ¶
DeleteEmbedding removes an embedding record. Removing an absent record is not an error.
func (*Storage) GetByContentHash ¶
GetByContentHash retrieves a dedup record by its key, or nil when there is none.
func (*Storage) GetEmbedding ¶
GetEmbedding retrieves an embedding by entity ID, or nil when there is none.
func (*Storage) SaveDedup ¶
func (s *Storage) SaveDedup(ctx context.Context, contentHash string, vector []float32, model string, dimensions int) error
SaveDedup saves a content-addressed embedding for deduplication.
model and dimensions identify the vector space the vector belongs to; callers pass the generating embedder's own values so a stale record is auditable. contentHash MUST come from DedupKey: the durable dedup bucket outlives any one embedder configuration.
A stored record from the same vector space (same model and dimensions) stands and nothing is written. Any other record, or none, is replaced with this vector: it was generated by the current embedder and is authoritative for the key, so a record from another or unknown space must not merely be stamped with the current identity, which would serve its old vector under the new model's name.
func (*Storage) SaveFailed ¶
func (s *Storage) SaveFailed(ctx context.Context, entityID, errorMsg, reason string, sourceRevision uint64) error
SaveFailed marks an embedding as failed under the same guard as SaveGenerated, so a stale failure cannot clobber a newer success, and the failed record keeps its own source revision for later ordering.
reason is the BOUNDED classification stored alongside the raw errorMsg; it is what the failures metric is labelled by and what the current-failed bootstrap scan reads. Pass "" only for a failure that has no classified reason.
It returns ErrRecordGone when the key is absent and ErrSupersededRevision when the stored record carries a newer source revision, or is generated at the same one: a generated outcome wins over a failure at an equal revision. Neither writes.
func (*Storage) SaveGenerated ¶
func (s *Storage) SaveGenerated( ctx context.Context, entityID string, vector []float32, model string, dimensions int, contentHash string, sourceRevision uint64, ) error
SaveGenerated persists a generated embedding, ordered by source revision.
contentHash is the hop-2 dedup key of the exact bytes embedded; it is stored as the record's content hash rather than copied from the pending record. sourceRevision is the ENTITY_STATES revision this generation is completing. ContentHash and Vector are both taken from the arguments, never from the stored record, so they cannot desync across revisions.
It returns ErrRecordGone, writing nothing, when the key is absent: the entity was tombstoned or its pending record removed, and writing would re-create a vector for an entity that must not have one. It returns ErrSupersededRevision, writing nothing, when the stored record carries a newer source revision.
func (*Storage) SavePendingGuarded ¶
SavePendingGuarded persists a hop-1 pending record with a generated-record guard. It is the write behind graph-embedding's sole hop-1 record writer, for both pending lanes (inline and storage-ref — the caller shapes the record; Status is forced to StatusPending here).
The guard: when the stored record is a GENERATED vector at a same-or-newer source revision, nothing is written and saved is false. An unconditional write here would let a stale repair re-drive downgrade a fresh generated record to pending, and the guard makes a restart's re-delivery of an already-generated revision a cheap skip. Any other stored record, pending or failed at any revision, or generated at an OLDER revision, is overwritten, so an older pending record replaces a newer pending one. An absent key, a delete or purge marker included, is created.
The decision and the write are one compare-and-set (UpdateWithRetryRead), so a concurrent hop-2 commit cannot bypass the guard.
func (*Storage) ScanFailed ¶
func (s *Storage) ScanFailed(ctx context.Context) ([]FailedEntry, error)
ScanFailed reads EMBEDDING_INDEX through a watch's initial snapshot and returns every record currently in the StatusFailed terminal state, with its bounded reason. It seeds the component's current-failed map at start so its failed count is accurate immediately after bootstrap, independent of re-delivery timing. It returns at the snapshot's end. A record that will not decode is skipped (best-effort seed); aborting would leave the failed count at 0, which is worse than a partial seed corrected by re-delivery. When ctx ends during the scan, the entries read so far are returned with a nil error, for the same reason; when it ends before the watch opens, none are. The watch is stopped through kvwatch.Stop; if its delivery does not close, ScanFailed returns that error and no entries.
type StorageRef ¶
StorageRef is a simplified reference for embedding storage. Mirrors message.StorageReference structure.
type StoreResolver ¶
type StoreResolver interface {
Streamable(instance string) (storage.StreamableStore, bool)
}
StoreResolver resolves a StorageReference's StorageInstance to its live streaming store (ADR-063); *storeregistry.Registry satisfies it. The Worker resolves per fetch and never caches or closes the returned store, which the storage component owns.
type TerminalCallback ¶
type TerminalCallback func(ctx context.Context, entityID string, sourceRevision uint64, outcome TerminalOutcome, reason string)
TerminalCallback is called once for each pending embedding the Worker carries to a terminal outcome, with the context Run was given, the entity ID, the ENTITY_STATES source revision that produced the record, the outcome and, for OutcomeFailed only, the bounded failure reason. It is what completes the hop-1 readiness watermark at the end of the two-hop pipeline (ADR-066, ADR-085): an entry that does not reach a terminal outcome (its processing panicked, a save's retries ran out, the record could not be marked failed, or its no-text removal failed) fires nothing and stays pending, so readiness stays withheld for it. sourceRevision 0 means unknown; ^uint64(0) drains an unreadable record whose revision cannot be recovered.
Ordering is GeneratedCallback's: per entity, in the order the entries were delivered, each after that entry's store write; unordered across entities.
type TerminalOutcome ¶
type TerminalOutcome int
TerminalOutcome classifies how a pending embedding reached its terminal state, so the caller can route its current-failed map: OutcomeFailed adds the entity with its bounded reason, every other outcome removes it. Every outcome is terminal for readiness: a permanently failing or text-less entity still completes.
const ( // OutcomeGenerated means a usable vector was stored. OutcomeGenerated TerminalOutcome = iota // OutcomeFailed means a durable StatusFailed record was written for this entity, the only // outcome that adds to the current-failed map. OutcomeFailed // OutcomeSkipped means nothing to embed (no text, and the pending record was removed), a // corrupt record drained, or a failure that did not persist a durable failed record // (superseded, record gone, or the Worker's context cancelled). None is a current failure. OutcomeSkipped // OutcomeDeleted means the record was removed, or newer work for the entity was stored, // while this entry was processed, so its result was dropped. Not a current failure. OutcomeDeleted )
type VectorCache ¶
type VectorCache struct {
// contains filtered or unexported fields
}
VectorCache is an in-memory copy of the generated vectors in EMBEDDING_INDEX, kept by Run on a graphview.View of the bucket. FindSimilar answers from the view while it is live, and refuses with an error matching graphview.ErrNotReady otherwise.
func NewVectorCache ¶
func NewVectorCache(storage *Storage) *VectorCache
NewVectorCache returns a cache over the storage's EMBEDDING_INDEX bucket. It answers nothing until Run has applied the watch's initial snapshot.
func (*VectorCache) FindSimilar ¶
func (c *VectorCache) FindSimilar(excludeID string, query []float32, keep func(string) bool, limit int) ([]ScoredEntity, int, error)
FindSimilar returns up to limit cached entities by cosine similarity to query, best first, ties by entity ID (ScoredEntityLess), sorted before the limit applies, and the number of records in EMBEDDING_INDEX that could not be decoded, which are not scored. The vectors and the count are read at one moment (graphview.View.ListCounted); scoring runs after the read, so a write applied meanwhile shows in the next call. The count is every unreadable record, whether or not keep accepts its key. excludeID is never scored ("" excludes nothing). keep, when non-nil, is applied before scoring and before the limit: only the entity IDs it accepts are scored, so a scoped search (ADR-071) returns the best in-scope entities. A limit of 0 or less returns no entities.
It returns an error matching graphview.ErrNotReady, and no entities, before Run has applied the watch's initial snapshot, after the watch was lost, and after Run returned.
func (*VectorCache) Run ¶
func (c *VectorCache) Run(ctx context.Context) error
Run keeps the cache in step with EMBEDDING_INDEX on the caller's goroutine until ctx ends or the watch is lost (the background-work spec's Run; the component that runs it owns ctx, ADR-113). Each Run builds a new graphview.View of the bucket and runs it, so it starts from an empty cache, which answers once the watch's initial snapshot has been applied.
The view stops as graphview.View.Run says: it watches under a context derived from ctx, ends it, and stops the watch through kvwatch.Stop, which reads the update channel until nats.go closes it, for at most kvwatch.DrainBudget (TestVectorCacheRunLeavesNoWatchDelivery drives a backlogged delivery). nats.go's own goroutine that unsubscribes the watch when its context ends is released, not joined: TestVectorCacheRunReportsLostWatch waits for it to go after Run returns. Run does not wait for a FindSimilar in progress, which answers from what it read.
Run returns ctx.Err() when ctx ended and an error matching graphview.ErrWatcherLost when the watch's update channel closed while ctx was live, joined with kvwatch.Stop's error when the watch did not stop cleanly. A watch that cannot open returns its error, or ctx.Err() when ctx has ended. Run refuses a nil context, and a second Run while one is running, before building a view. After Run returns, Run may be called again.
type Worker ¶
type Worker struct {
// contains filtered or unexported fields
}
Worker turns the pending records of EMBEDDING_INDEX into generated or failed ones (hop 2). Run watches the bucket and processes each pending record on a keyed pool, one entity's entries at a time.
func NewWorker ¶
NewWorker creates a Worker over storage's EMBEDDING_INDEX bucket, with 5 lanes. A nil logger means slog.Default.
func (*Worker) Run ¶
Run processes the pending records of EMBEDDING_INDEX until ctx ends or the watch is lost (the background-work spec's Run; the component that runs it owns ctx, ADR-113).
Run opens one watch of the bucket under a context it derives from ctx and reads it on the caller's goroutine. It hands each put to a keyed pool (internal/dispatch.KeyedPool) under the entity ID, so one entity's entries are processed one at a time in the order the watch delivered them, through its store write and both callbacks, while different entities run in parallel on the WithWorkers lanes. A full lane makes the read wait. A pending record is always processed; a failed record only when it arrives in the watch's initial snapshot, so each Run retries failures once, and the Worker's own failed writes never loop.
When ctx ends, or the watch's update channel closes while ctx is live, Run stops in this order: it stops reading and cancels the derived context, which ends a read waiting on a full lane and releases the goroutine nats.go keeps for the watch's context; it begins the pool's drain and joins its lanes (entries already accepted are processed under ctx, ended or not; an entry whose read was cut short stays pending in the bucket and returns in the next Run's snapshot); and it stops the watch through kvwatch.Stop, which reads the update channel until nats.go closes it, for at most kvwatch.DrainBudget (TestWorkerRunLeavesNoWatchDelivery drives a backlogged delivery). The join has no bound of its own: a callback that blocks holds Run.
Run returns ctx.Err() when ctx ended and an error saying the watch was lost otherwise, joined with kvwatch.Stop's error when the watch did not stop cleanly. A watch that cannot open returns its error, or ctx.Err() when ctx has ended. Run refuses a nil context, and a second Run while one is running, before opening a watch. After Run returns, Run may be called again.
func (*Worker) WithEmbedderType ¶
WithEmbedderType sets the Type axis of the dedup key identity ("bm25", or a remote embedder's type). The Worker folds it, with the embedder's live Model and Dimensions and the text cap, into the dedup key, so changing the embedder type cannot serve a vector from the previous vector space.
func (*Worker) WithMaxSourceTextLen ¶
WithMaxSourceTextLen sets the text cap in runes, applied on both lanes; text beyond it is truncated at a word boundary. 0 or less means 8,000; a cap above MaxSourceTextLenCeiling is clamped to it.
func (*Worker) WithMetrics ¶
func (w *Worker) WithMetrics(m WorkerMetrics) *Worker
WithMetrics sets the metrics receiver.
func (*Worker) WithOnGenerated ¶
func (w *Worker) WithOnGenerated(cb GeneratedCallback) *Worker
WithOnGenerated sets the callback invoked after an embedding is stored (see GeneratedCallback for its ordering).
func (*Worker) WithOnTerminal ¶
func (w *Worker) WithOnTerminal(cb TerminalCallback) *Worker
WithOnTerminal sets the callback invoked when a pending embedding reaches a terminal outcome (see TerminalCallback).
func (*Worker) WithStoreResolver ¶
func (w *Worker) WithStoreResolver(r StoreResolver) *Worker
WithStoreResolver sets the shared store resolver (ADR-063), the only path by which an offloaded body is fetched: a StorageRef's StorageInstance is resolved per fetch to the exact registered instance it names.
func (*Worker) WithWorkers ¶
WithWorkers sets the number of lanes entities are processed on in parallel, floored at 1: a zero or negative count would be a Worker that consumes nothing while every health signal stays green.
type WorkerMetrics ¶
type WorkerMetrics interface {
// IncDedupHits counts a stored vector that came from the dedup bucket. It fires only when
// the save succeeded, so dedup hits stay a subset of stored vectors.
IncDedupHits()
// IncDedupSkipped counts an embedding generated without consulting the dedup bucket, with
// the reason (today only an embedder whose vector width is unresolved, so no dedup key can
// be derived).
IncDedupSkipped(reason string)
// IncTruncated counts one source text truncated at the cap.
IncTruncated()
// IncOffloadedIdentityIncluded counts one offloaded entity whose stored vector included its
// inline identity text ahead of the body. It fires only when the save succeeded.
IncOffloadedIdentityIncluded()
// IncOffloadedIdentityAbsent counts one offloaded entity whose stored vector came from its
// body alone. It fires only when the save succeeded.
IncOffloadedIdentityAbsent()
// IncFailed counts one embedding failure.
IncFailed()
// IncFailedReason counts one embedding failure by its bounded reason, on the same events
// as IncFailed, never with the raw error message as the label.
IncFailedReason(reason string)
// IncContentResolveError counts a body fetch that failed after its store was resolved (a
// read error, a deleted bucket), as distinct from an unresolved store.
IncContentResolveError()
// IncContentResolved counts a body fetched from a resolved store.
IncContentResolved()
// IncContentUnresolved counts a body excluded because no live store is registered for its
// StorageInstance; the entity's inline identity text is still embedded.
IncContentUnresolved()
}
WorkerMetrics receives the Worker's counts, so the package depends on no metrics library.