Documentation
¶
Overview ¶
Package knowledgepageindex is the knowledge-page consumer of the shared indexjobs framework (#633). It registers a Source/Sink pair under source_kind = "portal-knowledge-pages" so canonical knowledge pages are embedded off the request path. A newly created page and a content edit each enqueue their own job at write time (#1256), so a page a session just wrote is findable through ranked search in that same session; the reconciler is the backstop for those a write could not produce and for the corpus a provider model swap invalidates.
SourceID is the page id, and a unit yields one Item per CHUNK of the page's composed text (title + body + tags, split by knowledgepage.IndexChunks to the provider's input budget), because a page's body routinely exceeds what an embedding provider accepts in one call — before #1242 everything past that budget was trimmed off and never reached the model. The vectors live in portal_knowledge_page_embedding_chunks, one row per chunk, and search keeps the best-scoring chunk per page so results stay page-granular. The body is indexed (unlike assets, whose body lives unindexed in S3), so page CONTENT is semantically searchable.
Only live pages with indexable text are indexed. Gap detection and coverage share one predicate, so a soft-deleted or text-less page is never embedded and never counted as missing coverage.
Index ¶
- Constants
- func RegisterConsumer(reg interface{ ... }, db *sql.DB, currentModel string, maxInputBytes int) error
- type Content
- type Sink
- func (s *Sink) Coverage(ctx context.Context) (indexjobs.Coverage, error)
- func (s *Sink) FindGaps(ctx context.Context) ([]string, error)
- func (*Sink) Kind() string
- func (s *Sink) ListExisting(ctx context.Context, key indexjobs.Key) (map[string]indexjobs.Vector, error)
- func (s *Sink) StampExpected(ctx context.Context, key indexjobs.Key, _ int) error
- func (s *Sink) Upsert(ctx context.Context, key indexjobs.Key, rows []indexjobs.Vector) error
- func (s *Sink) UpsertBatch(ctx context.Context, key indexjobs.Key, rows []indexjobs.Vector) error
- type Source
- type Store
- func (s *Store) Coverage(ctx context.Context, currentModel string) (indexed, expected int, err error)
- func (s *Store) FindGaps(ctx context.Context, currentModel string) ([]string, error)
- func (s *Store) GetContent(ctx context.Context, id string) (Content, error)
- func (s *Store) ListVectors(ctx context.Context, pageID string) (map[string]indexjobs.Vector, error)
- func (s *Store) ReplaceVectors(ctx context.Context, pageID string, rows []indexjobs.Vector) error
- func (s *Store) StampModel(ctx context.Context, pageID, model string) error
- func (s *Store) UpsertVectors(ctx context.Context, pageID string, rows []indexjobs.Vector) error
Constants ¶
const SourceKind = "portal-knowledge-pages"
SourceKind is the indexjobs source_kind this package serves.
Variables ¶
This section is empty.
Functions ¶
func RegisterConsumer ¶
func RegisterConsumer(reg interface {
Register(indexjobs.Source, indexjobs.Sink) error
}, db *sql.DB, currentModel string, maxInputBytes int,
) error
RegisterConsumer registers the knowledge-pages Source/Sink pair on the shared indexjobs registry. Keeping the wiring here (rather than inline in the platform) keeps the platform package thin. currentModel is the embedding provider's model identifier; maxInputBytes is that provider's per-text input budget (embedding.MaxInputBytes), the size pages are chunked to.
Types ¶
type Content ¶ added in v1.120.0
Content is the indexed text of one page: exactly the fields knowledgepage.IndexChunks composes an embed text from.
type Sink ¶
type Sink struct {
// contains filtered or unexported fields
}
Sink implements indexjobs.Sink for the portal-knowledge-pages kind over the page's embedding-chunk table. currentModel is the provider model the gap query diffs stored pages against, so a model swap re-embeds pages stamped with the previous model.
func NewSink ¶
NewSink returns a Sink backed by the given store. currentModel is the embedding provider's model identifier (embedding.ModelName); pass "" on a deployment whose provider does not name its model, in which case a page converges once it has been through the worker at all.
func (*Sink) Coverage ¶
Coverage reports the portal-knowledge-pages kind's indexed-vs-expected totals (pages converged on the current model vs all indexable pages). ExpectedKnown is true: every indexable page is expected to converge to a chunk set.
func (*Sink) FindGaps ¶
FindGaps returns the indexable page ids whose chunk set is missing or was produced by a model other than the current one.
func (*Sink) ListExisting ¶
func (s *Sink) ListExisting(ctx context.Context, key indexjobs.Key) (map[string]indexjobs.Vector, error)
ListExisting returns the page's persisted chunk vectors keyed by item id for the worker's dedup pass, so an edit re-embeds only the chunks whose text moved.
func (*Sink) StampExpected ¶
StampExpected marks the page's chunk set as produced by the current model. The count is not stored: gap detection is condition-based (the page's marker versus the current model), not a count comparison, because the number of chunks a page produces is a function of its text and the provider's budget, not a target the reconciler could independently derive.
The worker calls this only after a successful embed pass, which is exactly the convergence signal this marker records; a failure here is non-fatal (the next sweep re-enqueues the page and its unchanged chunks are reused by the dedup pass, so the retry costs no provider calls).
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source implements indexjobs.Source for the portal-knowledge-pages kind. A unit is one page (SourceID = page id) and yields one item per embeddable chunk of that page: the page's composed text split so no chunk exceeds the embedding provider's input budget (#1242). The worker embeds each chunk and the Sink writes the vectors into the page's chunk table.
func NewSource ¶
NewSource returns a Source backed by the given store, chunking page text to maxInputBytes per item.
func (*Source) LoadItems ¶
LoadItems returns one item per embeddable chunk of the page, in chunk order. A page soft-deleted between enqueue and claim, or one with no indexable text at all, yields an empty slice (a clean completion that writes no vectors), per the Source contract.
func (*Source) OnSucceeded ¶
OnSucceeded is a no-op: the ranked search reads embeddings from the chunk table directly on every query, so there is no in-memory cache to refresh after a backfill writes a vector.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store reads and writes knowledge-page embedding state for the indexjobs knowledge-pages consumer: the page's chunk vectors in portal_knowledge_page_embedding_chunks, plus the set-level index marker (embedding_model) on portal_knowledge_pages. It is intentionally separate from portal.KnowledgePageStore: it touches only indexing state and is scoped to the backfill path, so it does not widen the request-path store contract. The request-path Create/Update clears that state when a page's indexed text changes; this Store writes it back.
func (*Store) Coverage ¶
func (s *Store) Coverage(ctx context.Context, currentModel string) (indexed, expected int, err error)
Coverage returns the number of indexable pages whose chunk set is current (indexed) and the total number of indexable pages (expected), against the supplied provider model. Both halves apply the same indexable predicate as FindGaps, so a fully converged corpus reports 100% rather than a permanent shortfall from pages that can never carry a vector.
func (*Store) FindGaps ¶
FindGaps returns the ids of indexable pages whose chunk set is missing or was produced by a model other than the current provider's.
func (*Store) GetContent ¶ added in v1.120.0
GetContent returns the indexed fields of a non-deleted page. A page soft-deleted between enqueue and claim yields errNotIndexable so the Source returns an empty item set. The Source composes the embed text from these fields with knowledgepage.IndexChunks, the same composition the request-path search ranks against.
func (*Store) ListVectors ¶
func (s *Store) ListVectors(ctx context.Context, pageID string) (map[string]indexjobs.Vector, error)
ListVectors returns the page's persisted chunk vectors keyed by item id, for the worker's text-hash + model dedup pass. A page with no chunks yields an empty map, so the worker embeds every chunk. Per-chunk hashes are what make an edit to one section re-embed only the chunks whose text actually moved.
func (*Store) ReplaceVectors ¶ added in v1.120.0
ReplaceVectors writes the page's chunk set atomically: every supplied row is upserted and any chunk outside the set is deleted, so a page that shrinks (an edit that removes a section) does not leave an orphan vector ranking against text the page no longer has. An empty row set deletes every chunk, which is how the worker clears a unit whose source is gone.
The page's own updated_at is deliberately untouched here and in StampModel: a background embed is not a user-visible edit, so the page's "last modified" timestamp must not move.
func (*Store) StampModel ¶ added in v1.120.0
StampModel records that the page's chunk set was produced by the given model. It is the set-level convergence marker the gap query reads: one row per page, so detecting "this page still owes work" does not have to count chunk rows or know how many chunks the page's current text would produce. A page with no indexable text converges here with zero chunks, which is what keeps it out of the reconciler's sweep instead of being re-enqueued forever.
func (*Store) UpsertVectors ¶
UpsertVectors writes one batch of chunk vectors in place, leaving every chunk outside the batch alone, so a job that fails mid-pass leaves its completed chunks visible to the next attempt's dedup read.
Source Files