Documentation
¶
Overview ¶
Package collectionindex is the curated-collection consumer of the shared indexjobs framework (#550). It registers a Source/Sink pair under source_kind = "portal-collections" so collections are embedded off the request path. A newly created collection, one whose name/description changed, and one whose sections changed each enqueue their own job at write time (#1256); the reconciler is the backstop for those a write could not produce and for the corpus a provider model swap invalidates.
Like the prompt, memory, and asset consumers, collections store their vectors inline on the portal_collections table (one embedding per row). So this package's Store reads and writes the embedding / embedding_model / embedding_text_hash columns directly: a collection IS its own indexing unit. SourceID is the collection id; each unit yields exactly one Item whose text is portal.CollectionIndexText (name + description + the denormalized sections_text, i.e. section titles + descriptions).
Only non-deleted collections are indexed. Gap detection and coverage both filter on deleted_at IS NULL.
Index ¶
- Constants
- type Sink
- func (s *Sink) Coverage(ctx context.Context) (indexjobs.Coverage, error)
- func (s *Sink) FindGaps(ctx context.Context) ([]string, error)
- func (*Sink) Kind() string
- func (s *Sink) ListExisting(ctx context.Context, key indexjobs.Key) (map[string]indexjobs.Vector, error)
- func (*Sink) StampExpected(context.Context, indexjobs.Key, int) error
- func (s *Sink) Upsert(ctx context.Context, key indexjobs.Key, rows []indexjobs.Vector) error
- func (s *Sink) UpsertBatch(ctx context.Context, key indexjobs.Key, rows []indexjobs.Vector) error
- type Source
- type Store
- func (s *Store) Coverage(ctx context.Context) (indexed, expected int, err error)
- func (s *Store) FindGaps(ctx context.Context, currentModel string) ([]string, error)
- func (s *Store) GetIndexText(ctx context.Context, id string) (string, error)
- func (s *Store) ListVectors(ctx context.Context, id string) (map[string]indexjobs.Vector, error)
- func (s *Store) UpsertVectors(ctx context.Context, id string, rows []indexjobs.Vector) error
Constants ¶
const SourceKind = "portal-collections"
SourceKind is the indexjobs source_kind this package serves.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Sink ¶
type Sink struct {
// contains filtered or unexported fields
}
Sink implements indexjobs.Sink for the portal-collections kind over the embedding columns of the portal_collections table. currentModel is the provider model the gap query diffs stored rows against, so a model swap re-embeds rows stamped with the previous model.
func NewSink ¶
NewSink returns a Sink backed by the given store. currentModel is the embedding provider's model identifier (embedding.ModelName); pass "" on a deployment whose provider does not name its model, in which case every row matches "" and only NULL-embedding rows are treated as gaps.
func (*Sink) Coverage ¶
Coverage reports the portal-collections kind's indexed-vs-expected totals. ExpectedKnown is true: every non-deleted collection is expected to converge to one vector.
func (*Sink) FindGaps ¶
FindGaps returns non-deleted collection ids whose embedding is missing or was produced by a model other than the current one.
func (*Sink) ListExisting ¶
func (s *Sink) ListExisting(ctx context.Context, key indexjobs.Key) (map[string]indexjobs.Vector, error)
ListExisting returns the collection's persisted vector keyed by item id for the worker's dedup pass.
func (*Sink) StampExpected ¶
StampExpected is a no-op for collections. Gap detection is condition-based, not count-based, so there is no expected count to record per unit.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source implements indexjobs.Source for the portal-collections kind. A unit is one collection (SourceID = collection id) and yields exactly one item: the collection's composed embed text. The worker embeds it and the Sink writes the vector back onto the same row.
func (*Source) LoadItems ¶
LoadItems returns the collection's single embeddable item. A collection soft-deleted between enqueue and claim yields an empty slice (a clean completion that writes no vector), per the Source contract.
func (*Source) OnSucceeded ¶
OnSucceeded is a no-op: the ranked search reads embeddings from the portal_collections table directly on every query, so there is no in-memory cache to refresh after a backfill writes a vector.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store reads and writes collection embedding state on the portal_collections table for the indexjobs collections consumer. It touches only the embedding columns (embedding, embedding_model, embedding_text_hash) and the denormalized sections_text it reads, so it does not widen the request-path store contract. The request-path Update/SetSections clears the embedding when a collection's indexed text changes; this Store writes it back.
func (*Store) Coverage ¶
Coverage returns the number of non-deleted collections with an embedding (indexed) and the total number of non-deleted collections (expected).
func (*Store) FindGaps ¶
FindGaps returns the ids of non-deleted collections whose embedding is missing or was produced by a model other than the current provider's.
func (*Store) GetIndexText ¶
GetIndexText returns the composed embed text for a non-deleted collection. A collection soft-deleted between enqueue and claim yields errNotIndexable so the Source returns an empty item set. The composition is portal.CollectionIndexText, the same one the request-path search ranks against (name + description + sections_text).
func (*Store) ListVectors ¶
ListVectors returns the collection's persisted embedding keyed by item id (the collection id), for the worker's text-hash + model dedup pass. A collection with no embedding yields an empty map, so the worker embeds it.
Source Files