Documentation
¶
Overview ¶
Package knowledgepage is the store and ranked-search backend for canonical business/domain knowledge pages (#633): org-shared markdown documents stored inline in Postgres so their content is vector- and full-text searchable. It is a sibling of the portal package (not part of it) so the portal package stays within its size budget; the portal REST handler and the unified-search provider consume this package's Store and Searcher.
Index ¶
- Constants
- Variables
- func AssetRef(id string) string
- func ConnectionRef(kind, name string) string
- func IndexText(title, body string, tags []string) string
- func InsightRef(id string) string
- func MemoryRef(id string) string
- func NewID() string
- func NewRefID() string
- func NewVersionID() string
- func PageReference(id string) string
- func PromptRef(id string) string
- func SplitSuggestion(body string, byteThreshold, sectionThreshold int) (string, bool)
- type DedupCandidate
- type DuplicateProber
- type EntityRef
- type Filter
- type Page
- type PageGuards
- type PageGuardsConfig
- type PageRef
- type ReverseLookup
- type ScoredPage
- type SearchQuery
- type Searcher
- type Store
- type StoreSearcher
- type Update
- type Version
Constants ¶
const ( RefTargetAsset = "asset" RefTargetPrompt = "prompt" RefTargetCollection = "collection" RefTargetKnowledgePage = "knowledge_page" RefTargetConnection = "connection" RefTargetDataHub = "datahub" // RefTargetInsight is a captured insight (a knowledge-dimension memory record). // It is fetchable by its owner, but NOT citable on a shared knowledge page (#699): // an insight is ScopePerUser, so a citation would resolve only for its capturer. // Promote the insight to the catalog via apply_knowledge and cite the resulting // urn:li:... entity instead. The page-citation path rejects it (ParseCitableRef). RefTargetInsight = "insight" // RefTargetMemory is a personal memory record. It is fetchable by its owner but // must NOT be cited on a shared knowledge page: a per-user reference would // resolve only for its owner and be a broken citation for everyone else (#699). // It has no DB column and is rejected by the page-citation path (ParseCitableRef). RefTargetMemory = "memory" )
Entity-reference target types. Exactly one target is set on a reference row: an internal entity by foreign key, or an external DataHub URN.
const ( RefSourcePromoted = "promoted" // carried from a source insight by apply_knowledge RefSourceManual = "manual" // added explicitly through the authoring picker RefSourceInline = "inline" // derived from a mention in the page body )
Entity-reference sources, recording how a reference came to be so the inline body-scan (a later phase) can reconcile only its own rows without clobbering picked or promoted references.
const ( // DefaultOversizeBytes is the body byte size at or above which the split // suggestion fires (~16 KiB of markdown, well under the 1 MiB hard cap but // large enough that a single page is becoming an index's worth of content). DefaultOversizeBytes = 16384 // DefaultOversizeSections is the markdown-heading count at or above which the // split suggestion fires: many top-level sections is the structural tell that a // page is covering several topics that each want their own page. DefaultOversizeSections = 12 )
Default oversized-page thresholds (#705). A page crossing either bound gets a non-blocking suggestion to split into focused, cross-linked sub-pages; the write always succeeds. The platform signals, the agent performs the semantic split.
const DefaultDedupThreshold = 0.85
DefaultDedupThreshold is the cosine similarity at or above which a create is treated as a near-duplicate of an existing page (#705). It is a raw cosine in [0,1] from SemanticSearch (NOT the fused hybrid search score), so the value reads directly as "how similar": 0.85 catches "same topic, different slug" duplicates (e.g. "Return Policy" vs "ACME Returns Policy") while leaving genuinely distinct pages free to be created. Deployments can override it; 0 disables the gate.
const (
DefaultSearchLimit = 20
)
Search result limits, mirroring pkg/portal so every ranked surface clamps the same way. DefaultSearchLimit is the top-K when the caller does not specify one; maxSearchLimit bounds an explicit request.
Variables ¶
var ErrNotFound = errors.New("knowledge page not found")
ErrNotFound is returned when a page id/slug does not resolve to a live page.
var ErrRefTargetNotFound = errors.New("entity reference target does not exist")
ErrRefTargetNotFound is returned when a reference points at an internal entity (asset, prompt, collection, page, connection) that does not exist, so the foreign key rejects it. Callers map it to a client error rather than a 500.
Functions ¶
func AssetRef ¶ added in v1.90.0
AssetRef returns the canonical reference for an asset, or "" if id is empty.
func ConnectionRef ¶ added in v1.90.0
ConnectionRef returns the canonical reference for a connection, or "" if kind or name is empty.
func IndexText ¶
IndexText composes the text a page is embedded and lexically indexed on: its title, body, and tags. The indexjobs knowledge-pages consumer and the request-path search MUST agree on this composition so a stored embedding lives in the same space as the query; portal_knowledge_page_fts (migration 000070) composes the same corpus from the same columns. Empty fields are skipped so a sparse page does not pad the text.
func InsightRef ¶ added in v1.93.0
InsightRef returns the canonical reference for a captured insight, or "" if id is empty. An insight is fetchable by its owner but is NOT citable on a knowledge page (#699): it is per-user, so a shared-page citation would resolve for no one else. The page-citation path rejects it (ParseCitableRef); promote the insight to the catalog and cite the resulting urn:li:... entity instead.
func MemoryRef ¶ added in v1.93.0
MemoryRef returns the canonical reference for a personal memory record, or "" if id is empty. Memory is fetchable by its owner but is NOT citable on a knowledge page (#699); the page-citation path rejects it.
func NewRefID ¶
func NewRefID() string
NewRefID returns a unique id for an entity-reference row ("kpr_<uuid>").
func NewVersionID ¶
func NewVersionID() string
NewVersionID returns a unique id for a page version ("kpv_<uuid>").
func PageReference ¶ added in v1.90.0
PageReference returns the canonical reference for a knowledge page, or "" if id is empty. It is not named PageRef to avoid colliding with the PageRef type.
func PromptRef ¶ added in v1.90.0
PromptRef returns the canonical reference for a prompt, or "" when id is not a UUID (file-defined prompts have no prompts row and are not referenceable).
func SplitSuggestion ¶ added in v1.94.0
SplitSuggestion reports whether a page body is large enough to suggest splitting it into focused, cross-linked sub-pages, and the human-readable suggestion when so (#705, Part B). It is a pure signal: it never blocks a write. A non-positive threshold disables that arm. ok is true when either the byte size or the heading count crosses its (positive) threshold.
Types ¶
type DedupCandidate ¶ added in v1.94.0
type DedupCandidate struct {
ID string `json:"id"`
Slug string `json:"slug,omitempty"`
Title string `json:"title"`
Score float64 `json:"score"`
}
DedupCandidate is an existing page the dedup gate flags as a near-duplicate of a page being created (#705): enough to either re-apply against its slug (an update) or, with force_new, knowingly create a separate page. Score is the cosine similarity in [0,1].
func NearDuplicatePages ¶ added in v1.94.0
func NearDuplicatePages(ctx context.Context, p DuplicateProber, embedding []float32, threshold float64) ([]DedupCandidate, error)
NearDuplicatePages returns the existing pages whose cosine similarity to the candidate is at or above threshold, the create-time dedup gate shared by the MCP apply path and the portal REST create path (#705). It is the recall-first analog of memory_capture, adapted for shared pages: surface-and-require rather than auto-supersede, so a human or agent owns the merge decision.
The gate is meaningful only with a real embedding: a nil embedding makes it a no-op (returns no candidates) and the create proceeds, the same graceful degradation the platform applies wherever no embedding provider is configured. A non-positive threshold also disables the gate.
The probe ranks by SemanticSearch (pure cosine), not Search (fused semantic+lexical), so threshold is a true similarity. The caller embeds the candidate with IndexText so the query vector lives in the same text space as the stored page embeddings.
type DuplicateProber ¶ added in v1.94.0
type DuplicateProber interface {
SemanticSearch(ctx context.Context, embedding []float32, limit int) ([]ScoredPage, error)
}
DuplicateProber ranks pages by pure embedding cosine similarity for the dedup gate (#705). It is the SemanticSearch slice of the store, declared separately from Searcher because the gate needs the raw cosine, not Search's fused semantic+lexical score (which is uncalibrated as a similarity threshold).
type EntityRef ¶
type EntityRef struct {
ID string `json:"id,omitempty"`
PageID string `json:"page_id,omitempty"`
TargetType string `json:"target_type"`
AssetID string `json:"asset_id,omitempty"`
PromptID string `json:"prompt_id,omitempty"`
CollectionID string `json:"collection_id,omitempty"`
RefPageID string `json:"ref_page_id,omitempty"`
ConnectionKind string `json:"connection_kind,omitempty"`
ConnectionName string `json:"connection_name,omitempty"`
EntityURN string `json:"entity_urn,omitempty"`
InsightID string `json:"insight_id,omitempty"`
// MemoryID is set only by the parser for an mcp:memory: reference (fetch-only);
// it is never persisted, since memory is not citable on a page (#699).
MemoryID string `json:"memory_id,omitempty"`
Source string `json:"source,omitempty"`
CreatedBy string `json:"created_by,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
EntityRef is a typed reference from a knowledge page to an entity it provides knowledge about. Exactly one target is populated, matching TargetType.
func DataHubRef ¶
DataHubRef builds a reference to an external DataHub entity (a urn:li: URN). This is the only reference type Phase 0 writes: the references apply_knowledge carries from a promoted insight.
func ParseCitableRef ¶ added in v1.93.0
ParseCitableRef parses a reference for attachment to a knowledge page. It is ParseEntityRef plus the page-citation policy: a reference type that is fetchable but not citable on a shared page is rejected here, even though it parses and dereferences. Today those are the per-user sources, personal memory (mcp:memory:<id>) and captured insights (mcp:insight:<id>): both are ScopePerUser, so a citation embedded in a shared page would resolve only for its owner and be a broken citation for everyone else (#699). An insight that is promoted to the catalog via apply_knowledge becomes a shared DataHub entity, which IS citable as its urn:li:... form. Use this on the page-authoring paths (apply_knowledge references, the REST picker, the inline body scan); fetch keeps using ParseEntityRef so both forms remain fetchable by their owner.
func ParseEntityRef ¶
ParseEntityRef parses a serialized reference (the inverse of URN) into a typed EntityRef. A "urn:" reference is an external (DataHub) URN, stored verbatim; an "mcp:" reference resolves to the matching internal target. Source and the owning page id are not part of the serialized form and are left to the caller.
func ScanBodyRefs ¶
ScanBodyRefs extracts the entity references mentioned in a page's markdown body. It is content-agnostic: a reference is found whether it appears as a markdown link href, an autolink, or inline text. It uses ParseCitableRef, so a mention of a fetch-only-but-not-citable form (mcp:memory:, mcp:insight:) is skipped exactly like an unparseable token rather than producing a reference no page-citation path can persist (#699). Unparseable matches are skipped, the result is de-duplicated by target, and every ref is marked source=inline so a reconcile can replace only the inline set without touching promoted or manual references.
type Filter ¶
Filter narrows a knowledge page listing. Tag filters to pages carrying the tag; Query is a substring match on title for the browse UI. Only non-deleted pages are ever returned.
type Page ¶
type Page struct {
ID string `json:"id" example:"kp_01HK7R8Z8M0Y6A5G1R6FQ2VQNK"`
Slug string `json:"slug,omitempty" example:"fiscal-calendar"`
Title string `json:"title" example:"Fiscal Calendar"`
Summary string `json:"summary,omitempty" example:"How the company defines fiscal quarters."`
Body string `json:"body" example:"# Fiscal Calendar\n\nQ1 begins..."`
Tags []string `json:"tags"`
CreatedBy string `json:"created_by,omitempty" example:"alice@example.com"`
CreatedEmail string `json:"created_email,omitempty" example:"alice@example.com"`
UpdatedBy string `json:"updated_by,omitempty" example:"bob@example.com"`
CurrentVersion int `json:"current_version" example:"3"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
DeletedAt *time.Time `json:"deleted_at,omitempty"`
}
Page is a canonical unit of business/domain knowledge: a markdown page in the platform's internal knowledge store (the sibling of DataHub). It is org-shared, not owner-scoped: every caller can read it, and personas with apply_knowledge access edit it. The markdown body is stored inline (not in S3) so page CONTENT is directly embeddable and full-text searchable.
type PageGuards ¶ added in v1.94.0
PageGuards are the resolved write-guard thresholds the apply path and portal consume. A zero DedupThreshold disables the gate; a zero Oversize* threshold disables that arm of the split suggestion.
type PageGuardsConfig ¶ added in v1.94.0
type PageGuardsConfig struct {
// DedupThreshold is the cosine similarity [0,1] at or above which creating a page
// is blocked as a near-duplicate. Defaults to DefaultDedupThreshold. The gate only
// acts when a real embedding provider is configured (cosine is undefined without).
DedupThreshold float64 `yaml:"dedup_threshold"`
// DedupDisabled turns the duplicate gate off entirely.
DedupDisabled bool `yaml:"dedup_disabled"`
// OversizeBytes is the body byte size at or above which the (non-blocking) split
// suggestion fires. Defaults to DefaultOversizeBytes; negative disables this arm.
OversizeBytes int `yaml:"oversize_bytes"`
// OversizeSections is the markdown-heading count at or above which the split
// suggestion fires. Defaults to DefaultOversizeSections; negative disables it.
OversizeSections int `yaml:"oversize_sections"`
}
PageGuardsConfig is the YAML configuration for the knowledge-page write guards (#705): the create-time duplicate gate and the oversized-page split suggestion, shared by the MCP apply path and the portal REST create path. Zero values select documented defaults; the gate is turned off explicitly with DedupDisabled so a deployment that wants no gate is unambiguous (not confused with "left at default").
func (PageGuardsConfig) Resolve ¶ added in v1.94.0
func (c PageGuardsConfig) Resolve() PageGuards
Resolve applies the documented defaults: DedupDisabled (or a non-positive threshold) yields 0 (gate off); an unset threshold yields DefaultDedupThreshold. A negative oversize threshold disables that arm (0); an unset one yields the default.
type PageRef ¶
PageRef identifies a knowledge page that references an entity. It is the result of the reverse lookup (the pages that reference a target), the counterpart of ListEntityRefs (a page's references).
func PagesForURNs ¶ added in v1.89.0
func PagesForURNs(ctx context.Context, store ReverseLookup, urns []string, limit int) ([]PageRef, error)
PagesForURNs returns the distinct pages that reference any of the given entity URNs, in first-seen order, capped at limit (limit <= 0 means no cap). A URN that does not parse as a reference is skipped; a lookup error is returned. It backs the cross-enrichment that surfaces the knowledge about the entities a tool returns.
type ReverseLookup ¶ added in v1.89.0
type ReverseLookup interface {
ListPagesReferencing(ctx context.Context, ref EntityRef) ([]PageRef, error)
}
ReverseLookup is the reverse-lookup capability PagesForURNs needs (the pages that reference a target), satisfied by Store and by lighter adapters in callers.
type ScoredPage ¶
ScoredPage pairs a page with its relevance score in [0,1].
type SearchQuery ¶
type SearchQuery struct {
Embedding []float32 // query vector; nil selects lexical-only ranking
QueryText string // raw query text for the lexical arm
Limit int // max results; clamped into [1, maxSearchLimit]
}
SearchQuery describes a relevance ranking request over canonical knowledge pages. Unlike asset search, there is NO owner scope: knowledge pages are org-shared, so every non-deleted page is rankable for every caller. A nil Embedding selects lexical-only ranking (graceful degradation when no embedding provider is configured); a non-nil Embedding selects hybrid ranking.
func (SearchQuery) EffectiveLimit ¶
func (q SearchQuery) EffectiveLimit() int
EffectiveLimit clamps the requested limit into the search bounds.
type Searcher ¶
type Searcher interface {
Search(ctx context.Context, q SearchQuery) ([]ScoredPage, error)
}
Searcher ranks knowledge pages by relevance to a query. It is a capability separate from Store so the feature degrades to absent (rather than forcing every store to carry a ranking query) on a deployment without pgvector.
type Store ¶
type Store interface {
Insert(ctx context.Context, page Page) error
Get(ctx context.Context, id string) (*Page, error)
GetBySlug(ctx context.Context, slug string) (*Page, error)
List(ctx context.Context, filter Filter) ([]Page, int, error)
Update(ctx context.Context, id string, updates Update) error
SoftDelete(ctx context.Context, id string) error
ListVersions(ctx context.Context, pageID string, limit, offset int) ([]Version, int, error)
GetVersion(ctx context.Context, pageID string, version int) (*Version, error)
// Entity references (#664): the entities a page provides knowledge about.
ListEntityRefs(ctx context.Context, pageID string) ([]EntityRef, error)
// ValidateRefTargets checks each FK-backed reference target exists, so a
// citation to a missing entity is rejected before the page is written (#690).
ValidateRefTargets(ctx context.Context, refs []EntityRef) error
// FilterExistingRefTargets returns the subset of refs whose target exists,
// dropping stale references carried from a source insight (#690).
FilterExistingRefTargets(ctx context.Context, refs []EntityRef) ([]EntityRef, error)
AddEntityRefs(ctx context.Context, pageID string, refs []EntityRef) error
ReplaceEntityRefs(ctx context.Context, pageID string, refs []EntityRef) error
ReplaceEntityRefsBySource(ctx context.Context, pageID, source string, refs []EntityRef) error
// ListPagesReferencing is the reverse lookup: the pages that reference a target.
ListPagesReferencing(ctx context.Context, ref EntityRef) ([]PageRef, error)
}
Store persists and queries canonical knowledge pages.
func NewPostgresStore ¶
NewPostgresStore creates a PostgreSQL knowledge page store.
type StoreSearcher ¶ added in v1.94.0
type StoreSearcher interface {
Store
Searcher
DuplicateProber
}
StoreSearcher is a Store that also ranks pages by relevance (Searcher) and by pure cosine for the dedup gate (DuplicateProber). The postgres store implements all three; the knowledge apply path needs them to write a page and run the create-time duplicate probe (#705), so it takes this combined capability rather than asserting the optional interfaces off a bare Store at runtime.
func NewPostgresStoreSearcher ¶ added in v1.94.0
func NewPostgresStoreSearcher(db *sql.DB) StoreSearcher
NewPostgresStoreSearcher is NewPostgresStore typed as a StoreSearcher, for the apply path which needs Search (the dedup gate) alongside the write methods.
type Update ¶
type Update struct {
Slug *string
Title *string
Summary *string
Body *string
Tags *[]string
UpdatedBy string
ChangeSummary string
}
Update carries the editable fields of a page. A nil pointer means "leave unchanged"; a non-nil pointer (including empty string) sets the field. Whenever Title, Body, or Tags change, the store clears the embedding columns so the indexjobs reconciler re-embeds the new content off the request path.
type Version ¶
type Version struct {
ID string `json:"id" example:"kpv_01HK7R9A"`
PageID string `json:"page_id" example:"kp_01HK7R8Z8M0Y6A5G1R6FQ2VQNK"`
Version int `json:"version" example:"2"`
Title string `json:"title" example:"Fiscal Calendar"`
Summary string `json:"summary,omitempty"`
Body string `json:"body"`
Tags []string `json:"tags"`
CreatedBy string `json:"created_by,omitempty" example:"bob@example.com"`
ChangeSummary string `json:"change_summary,omitempty" example:"Clarified Q1 start"`
CreatedAt time.Time `json:"created_at"`
}
Version records a single saved version of a page's content. The body is stored inline (pages are text-first and bounded), unlike asset versions which reference S3 keys.