knowledgepage

package
v1.97.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 4, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package knowledgepage is the store and ranked-search backend for canonical business/domain knowledge pages (#633): org-shared markdown documents stored inline in Postgres so their content is vector- and full-text searchable. It is a sibling of the portal package (not part of it) so the portal package stays within its size budget; the portal REST handler and the unified-search provider consume this package's Store and Searcher.

Index

Constants

View Source
const (
	RefTargetAsset         = "asset"
	RefTargetPrompt        = "prompt"
	RefTargetCollection    = "collection"
	RefTargetKnowledgePage = "knowledge_page"
	RefTargetConnection    = "connection"
	RefTargetDataHub       = "datahub"
	// RefTargetInsight is a captured insight (a knowledge-dimension memory record).
	// It is fetchable by its owner, but NOT citable on a shared knowledge page (#699):
	// an insight is ScopePerUser, so a citation would resolve only for its capturer.
	// Promote the insight to the catalog via apply_knowledge and cite the resulting
	// urn:li:... entity instead. The page-citation path rejects it (ParseCitableRef).
	RefTargetInsight = "insight"
	// RefTargetMemory is a personal memory record. It is fetchable by its owner but
	// must NOT be cited on a shared knowledge page: a per-user reference would
	// resolve only for its owner and be a broken citation for everyone else (#699).
	// It has no DB column and is rejected by the page-citation path (ParseCitableRef).
	RefTargetMemory = "memory"
)

Entity-reference target types. Exactly one target is set on a reference row: an internal entity by foreign key, or an external DataHub URN.

View Source
const (
	RefSourcePromoted = "promoted" // carried from a source insight by apply_knowledge
	RefSourceManual   = "manual"   // added explicitly through the authoring picker
	RefSourceInline   = "inline"   // derived from a mention in the page body
)

Entity-reference sources, recording how a reference came to be so the inline body-scan (a later phase) can reconcile only its own rows without clobbering picked or promoted references.

View Source
const (
	// DefaultOversizeBytes is the body byte size at or above which the split
	// suggestion fires (~16 KiB of markdown, well under the 1 MiB hard cap but
	// large enough that a single page is becoming an index's worth of content).
	DefaultOversizeBytes = 16384
	// DefaultOversizeSections is the markdown-heading count at or above which the
	// split suggestion fires: many top-level sections is the structural tell that a
	// page is covering several topics that each want their own page.
	DefaultOversizeSections = 12
)

Default oversized-page thresholds (#705). A page crossing either bound gets a non-blocking suggestion to split into focused, cross-linked sub-pages; the write always succeeds. The platform signals, the agent performs the semantic split.

View Source
const DefaultDedupThreshold = 0.85

DefaultDedupThreshold is the cosine similarity at or above which a create is treated as a near-duplicate of an existing page (#705). It is a raw cosine in [0,1] from SemanticSearch (NOT the fused hybrid search score), so the value reads directly as "how similar": 0.85 catches "same topic, different slug" duplicates (e.g. "Return Policy" vs "ACME Returns Policy") while leaving genuinely distinct pages free to be created. Deployments can override it; 0 disables the gate.

View Source
const (
	DefaultSearchLimit = 20
)

Search result limits, mirroring pkg/portal so every ranked surface clamps the same way. DefaultSearchLimit is the top-K when the caller does not specify one; maxSearchLimit bounds an explicit request.

Variables

View Source
var ErrNotFound = errors.New("knowledge page not found")

ErrNotFound is returned when a page id/slug does not resolve to a live page.

View Source
var ErrRefTargetNotFound = errors.New("entity reference target does not exist")

ErrRefTargetNotFound is returned when a reference points at an internal entity (asset, prompt, collection, page, connection) that does not exist, so the foreign key rejects it. Callers map it to a client error rather than a 500.

Functions

func AssetRef added in v1.90.0

func AssetRef(id string) string

AssetRef returns the canonical reference for an asset, or "" if id is empty.

func ConnectionRef added in v1.90.0

func ConnectionRef(kind, name string) string

ConnectionRef returns the canonical reference for a connection, or "" if kind or name is empty.

func IndexText

func IndexText(title, body string, tags []string) string

IndexText composes the text a page is embedded and lexically indexed on: its title, body, and tags. The indexjobs knowledge-pages consumer and the request-path search MUST agree on this composition so a stored embedding lives in the same space as the query; portal_knowledge_page_fts (migration 000070) composes the same corpus from the same columns. Empty fields are skipped so a sparse page does not pad the text.

func InsightRef added in v1.93.0

func InsightRef(id string) string

InsightRef returns the canonical reference for a captured insight, or "" if id is empty. An insight is fetchable by its owner but is NOT citable on a knowledge page (#699): it is per-user, so a shared-page citation would resolve for no one else. The page-citation path rejects it (ParseCitableRef); promote the insight to the catalog and cite the resulting urn:li:... entity instead.

func MemoryRef added in v1.93.0

func MemoryRef(id string) string

MemoryRef returns the canonical reference for a personal memory record, or "" if id is empty. Memory is fetchable by its owner but is NOT citable on a knowledge page (#699); the page-citation path rejects it.

func NewID

func NewID() string

NewID returns a unique id for a knowledge page ("kp_<uuid>").

func NewRefID

func NewRefID() string

NewRefID returns a unique id for an entity-reference row ("kpr_<uuid>").

func NewVersionID

func NewVersionID() string

NewVersionID returns a unique id for a page version ("kpv_<uuid>").

func PageReference added in v1.90.0

func PageReference(id string) string

PageReference returns the canonical reference for a knowledge page, or "" if id is empty. It is not named PageRef to avoid colliding with the PageRef type.

func PromptRef added in v1.90.0

func PromptRef(id string) string

PromptRef returns the canonical reference for a prompt, or "" when id is not a UUID (file-defined prompts have no prompts row and are not referenceable).

func SplitSuggestion added in v1.94.0

func SplitSuggestion(body string, byteThreshold, sectionThreshold int) (string, bool)

SplitSuggestion reports whether a page body is large enough to suggest splitting it into focused, cross-linked sub-pages, and the human-readable suggestion when so (#705, Part B). It is a pure signal: it never blocks a write. A non-positive threshold disables that arm. ok is true when either the byte size or the heading count crosses its (positive) threshold.

Types

type DedupCandidate added in v1.94.0

type DedupCandidate struct {
	ID    string  `json:"id"`
	Slug  string  `json:"slug,omitempty"`
	Title string  `json:"title"`
	Score float64 `json:"score"`
}

DedupCandidate is an existing page the dedup gate flags as a near-duplicate of a page being created (#705): enough to either re-apply against its slug (an update) or, with force_new, knowingly create a separate page. Score is the cosine similarity in [0,1].

func NearDuplicatePages added in v1.94.0

func NearDuplicatePages(ctx context.Context, p DuplicateProber, embedding []float32, threshold float64) ([]DedupCandidate, error)

NearDuplicatePages returns the existing pages whose cosine similarity to the candidate is at or above threshold, the create-time dedup gate shared by the MCP apply path and the portal REST create path (#705). It is the recall-first analog of memory_capture, adapted for shared pages: surface-and-require rather than auto-supersede, so a human or agent owns the merge decision.

The gate is meaningful only with a real embedding: a nil embedding makes it a no-op (returns no candidates) and the create proceeds, the same graceful degradation the platform applies wherever no embedding provider is configured. A non-positive threshold also disables the gate.

The probe ranks by SemanticSearch (pure cosine), not Search (fused semantic+lexical), so threshold is a true similarity. The caller embeds the candidate with IndexText so the query vector lives in the same text space as the stored page embeddings.

type DuplicateProber added in v1.94.0

type DuplicateProber interface {
	SemanticSearch(ctx context.Context, embedding []float32, limit int) ([]ScoredPage, error)
}

DuplicateProber ranks pages by pure embedding cosine similarity for the dedup gate (#705). It is the SemanticSearch slice of the store, declared separately from Searcher because the gate needs the raw cosine, not Search's fused semantic+lexical score (which is uncalibrated as a similarity threshold).

type EntityRef

type EntityRef struct {
	ID             string `json:"id,omitempty"`
	PageID         string `json:"page_id,omitempty"`
	TargetType     string `json:"target_type"`
	AssetID        string `json:"asset_id,omitempty"`
	PromptID       string `json:"prompt_id,omitempty"`
	CollectionID   string `json:"collection_id,omitempty"`
	RefPageID      string `json:"ref_page_id,omitempty"`
	ConnectionKind string `json:"connection_kind,omitempty"`
	ConnectionName string `json:"connection_name,omitempty"`
	EntityURN      string `json:"entity_urn,omitempty"`
	InsightID      string `json:"insight_id,omitempty"`
	// MemoryID is set only by the parser for an mcp:memory: reference (fetch-only);
	// it is never persisted, since memory is not citable on a page (#699).
	MemoryID  string    `json:"memory_id,omitempty"`
	Source    string    `json:"source,omitempty"`
	CreatedBy string    `json:"created_by,omitempty"`
	CreatedAt time.Time `json:"created_at"`
}

EntityRef is a typed reference from a knowledge page to an entity it provides knowledge about. Exactly one target is populated, matching TargetType.

func DataHubRef

func DataHubRef(urn, source string) EntityRef

DataHubRef builds a reference to an external DataHub entity (a urn:li: URN). This is the only reference type Phase 0 writes: the references apply_knowledge carries from a promoted insight.

func ParseCitableRef added in v1.93.0

func ParseCitableRef(s string) (EntityRef, error)

ParseCitableRef parses a reference for attachment to a knowledge page. It is ParseEntityRef plus the page-citation policy: a reference type that is fetchable but not citable on a shared page is rejected here, even though it parses and dereferences. Today those are the per-user sources, personal memory (mcp:memory:<id>) and captured insights (mcp:insight:<id>): both are ScopePerUser, so a citation embedded in a shared page would resolve only for its owner and be a broken citation for everyone else (#699). An insight that is promoted to the catalog via apply_knowledge becomes a shared DataHub entity, which IS citable as its urn:li:... form. Use this on the page-authoring paths (apply_knowledge references, the REST picker, the inline body scan); fetch keeps using ParseEntityRef so both forms remain fetchable by their owner.

func ParseEntityRef

func ParseEntityRef(s string) (EntityRef, error)

ParseEntityRef parses a serialized reference (the inverse of URN) into a typed EntityRef. A "urn:" reference is an external (DataHub) URN, stored verbatim; an "mcp:" reference resolves to the matching internal target. Source and the owning page id are not part of the serialized form and are left to the caller.

func ScanBodyRefs

func ScanBodyRefs(body string) []EntityRef

ScanBodyRefs extracts the entity references mentioned in a page's markdown body. It is content-agnostic: a reference is found whether it appears as a markdown link href, an autolink, or inline text. It uses ParseCitableRef, so a mention of a fetch-only-but-not-citable form (mcp:memory:, mcp:insight:) is skipped exactly like an unparseable token rather than producing a reference no page-citation path can persist (#699). Unparseable matches are skipped, the result is de-duplicated by target, and every ref is marked source=inline so a reconcile can replace only the inline set without touching promoted or manual references.

func (EntityRef) URN

func (r EntityRef) URN() string

URN returns the serialized projection of a reference: the mcp: form for an internal entity (resolved by id, so a rename does not change it), or the external URN as-is for a DataHub reference. The empty string is returned for an unrecognized target type.

type Filter

type Filter struct {
	Tag    string
	Query  string
	Limit  int
	Offset int
}

Filter narrows a knowledge page listing. Tag filters to pages carrying the tag; Query is a substring match on title for the browse UI. Only non-deleted pages are ever returned.

type Page

type Page struct {
	ID             string     `json:"id" example:"kp_01HK7R8Z8M0Y6A5G1R6FQ2VQNK"`
	Slug           string     `json:"slug,omitempty" example:"fiscal-calendar"`
	Title          string     `json:"title" example:"Fiscal Calendar"`
	Summary        string     `json:"summary,omitempty" example:"How the company defines fiscal quarters."`
	Body           string     `json:"body" example:"# Fiscal Calendar\n\nQ1 begins..."`
	Tags           []string   `json:"tags"`
	CreatedBy      string     `json:"created_by,omitempty" example:"alice@example.com"`
	CreatedEmail   string     `json:"created_email,omitempty" example:"alice@example.com"`
	UpdatedBy      string     `json:"updated_by,omitempty" example:"bob@example.com"`
	CurrentVersion int        `json:"current_version" example:"3"`
	CreatedAt      time.Time  `json:"created_at"`
	UpdatedAt      time.Time  `json:"updated_at"`
	DeletedAt      *time.Time `json:"deleted_at,omitempty"`
}

Page is a canonical unit of business/domain knowledge: a markdown page in the platform's internal knowledge store (the sibling of DataHub). It is org-shared, not owner-scoped: every caller can read it, and personas with apply_knowledge access edit it. The markdown body is stored inline (not in S3) so page CONTENT is directly embeddable and full-text searchable.

type PageGuards added in v1.94.0

type PageGuards struct {
	DedupThreshold   float64
	OversizeBytes    int
	OversizeSections int
}

PageGuards are the resolved write-guard thresholds the apply path and portal consume. A zero DedupThreshold disables the gate; a zero Oversize* threshold disables that arm of the split suggestion.

type PageGuardsConfig added in v1.94.0

type PageGuardsConfig struct {
	// DedupThreshold is the cosine similarity [0,1] at or above which creating a page
	// is blocked as a near-duplicate. Defaults to DefaultDedupThreshold. The gate only
	// acts when a real embedding provider is configured (cosine is undefined without).
	DedupThreshold float64 `yaml:"dedup_threshold"`
	// DedupDisabled turns the duplicate gate off entirely.
	DedupDisabled bool `yaml:"dedup_disabled"`
	// OversizeBytes is the body byte size at or above which the (non-blocking) split
	// suggestion fires. Defaults to DefaultOversizeBytes; negative disables this arm.
	OversizeBytes int `yaml:"oversize_bytes"`
	// OversizeSections is the markdown-heading count at or above which the split
	// suggestion fires. Defaults to DefaultOversizeSections; negative disables it.
	OversizeSections int `yaml:"oversize_sections"`
}

PageGuardsConfig is the YAML configuration for the knowledge-page write guards (#705): the create-time duplicate gate and the oversized-page split suggestion, shared by the MCP apply path and the portal REST create path. Zero values select documented defaults; the gate is turned off explicitly with DedupDisabled so a deployment that wants no gate is unambiguous (not confused with "left at default").

func (PageGuardsConfig) Resolve added in v1.94.0

func (c PageGuardsConfig) Resolve() PageGuards

Resolve applies the documented defaults: DedupDisabled (or a non-positive threshold) yields 0 (gate off); an unset threshold yields DefaultDedupThreshold. A negative oversize threshold disables that arm (0); an unset one yields the default.

type PageRef

type PageRef struct {
	ID    string `json:"id"`
	Slug  string `json:"slug"`
	Title string `json:"title"`
}

PageRef identifies a knowledge page that references an entity. It is the result of the reverse lookup (the pages that reference a target), the counterpart of ListEntityRefs (a page's references).

func PagesForURNs added in v1.89.0

func PagesForURNs(ctx context.Context, store ReverseLookup, urns []string, limit int) ([]PageRef, error)

PagesForURNs returns the distinct pages that reference any of the given entity URNs, in first-seen order, capped at limit (limit <= 0 means no cap). A URN that does not parse as a reference is skipped; a lookup error is returned. It backs the cross-enrichment that surfaces the knowledge about the entities a tool returns.

type ReverseLookup added in v1.89.0

type ReverseLookup interface {
	ListPagesReferencing(ctx context.Context, ref EntityRef) ([]PageRef, error)
}

ReverseLookup is the reverse-lookup capability PagesForURNs needs (the pages that reference a target), satisfied by Store and by lighter adapters in callers.

type ScoredPage

type ScoredPage struct {
	Page  Page    `json:"page"`
	Score float64 `json:"score"`
}

ScoredPage pairs a page with its relevance score in [0,1].

type SearchQuery

type SearchQuery struct {
	Embedding []float32 // query vector; nil selects lexical-only ranking
	QueryText string    // raw query text for the lexical arm
	Limit     int       // max results; clamped into [1, maxSearchLimit]
}

SearchQuery describes a relevance ranking request over canonical knowledge pages. Unlike asset search, there is NO owner scope: knowledge pages are org-shared, so every non-deleted page is rankable for every caller. A nil Embedding selects lexical-only ranking (graceful degradation when no embedding provider is configured); a non-nil Embedding selects hybrid ranking.

func (SearchQuery) EffectiveLimit

func (q SearchQuery) EffectiveLimit() int

EffectiveLimit clamps the requested limit into the search bounds.

type Searcher

type Searcher interface {
	Search(ctx context.Context, q SearchQuery) ([]ScoredPage, error)
}

Searcher ranks knowledge pages by relevance to a query. It is a capability separate from Store so the feature degrades to absent (rather than forcing every store to carry a ranking query) on a deployment without pgvector.

type Store

type Store interface {
	Insert(ctx context.Context, page Page) error
	Get(ctx context.Context, id string) (*Page, error)
	GetBySlug(ctx context.Context, slug string) (*Page, error)
	List(ctx context.Context, filter Filter) ([]Page, int, error)
	Update(ctx context.Context, id string, updates Update) error
	SoftDelete(ctx context.Context, id string) error
	ListVersions(ctx context.Context, pageID string, limit, offset int) ([]Version, int, error)
	GetVersion(ctx context.Context, pageID string, version int) (*Version, error)
	// Entity references (#664): the entities a page provides knowledge about.
	ListEntityRefs(ctx context.Context, pageID string) ([]EntityRef, error)
	// ValidateRefTargets checks each FK-backed reference target exists, so a
	// citation to a missing entity is rejected before the page is written (#690).
	ValidateRefTargets(ctx context.Context, refs []EntityRef) error
	// FilterExistingRefTargets returns the subset of refs whose target exists,
	// dropping stale references carried from a source insight (#690).
	FilterExistingRefTargets(ctx context.Context, refs []EntityRef) ([]EntityRef, error)
	AddEntityRefs(ctx context.Context, pageID string, refs []EntityRef) error
	ReplaceEntityRefs(ctx context.Context, pageID string, refs []EntityRef) error
	ReplaceEntityRefsBySource(ctx context.Context, pageID, source string, refs []EntityRef) error
	// ListPagesReferencing is the reverse lookup: the pages that reference a target.
	ListPagesReferencing(ctx context.Context, ref EntityRef) ([]PageRef, error)
}

Store persists and queries canonical knowledge pages.

func NewPostgresStore

func NewPostgresStore(db *sql.DB) Store

NewPostgresStore creates a PostgreSQL knowledge page store.

type StoreSearcher added in v1.94.0

type StoreSearcher interface {
	Store
	Searcher
	DuplicateProber
}

StoreSearcher is a Store that also ranks pages by relevance (Searcher) and by pure cosine for the dedup gate (DuplicateProber). The postgres store implements all three; the knowledge apply path needs them to write a page and run the create-time duplicate probe (#705), so it takes this combined capability rather than asserting the optional interfaces off a bare Store at runtime.

func NewPostgresStoreSearcher added in v1.94.0

func NewPostgresStoreSearcher(db *sql.DB) StoreSearcher

NewPostgresStoreSearcher is NewPostgresStore typed as a StoreSearcher, for the apply path which needs Search (the dedup gate) alongside the write methods.

type Update

type Update struct {
	Slug          *string
	Title         *string
	Summary       *string
	Body          *string
	Tags          *[]string
	UpdatedBy     string
	ChangeSummary string
}

Update carries the editable fields of a page. A nil pointer means "leave unchanged"; a non-nil pointer (including empty string) sets the field. Whenever Title, Body, or Tags change, the store clears the embedding columns so the indexjobs reconciler re-embeds the new content off the request path.

type Version

type Version struct {
	ID            string    `json:"id" example:"kpv_01HK7R9A"`
	PageID        string    `json:"page_id" example:"kp_01HK7R8Z8M0Y6A5G1R6FQ2VQNK"`
	Version       int       `json:"version" example:"2"`
	Title         string    `json:"title" example:"Fiscal Calendar"`
	Summary       string    `json:"summary,omitempty"`
	Body          string    `json:"body"`
	Tags          []string  `json:"tags"`
	CreatedBy     string    `json:"created_by,omitempty" example:"bob@example.com"`
	ChangeSummary string    `json:"change_summary,omitempty" example:"Clarified Q1 start"`
	CreatedAt     time.Time `json:"created_at"`
}

Version records a single saved version of a page's content. The body is stored inline (pages are text-first and bounded), unlike asset versions which reference S3 keys.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL