knowledge

package
v1.94.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 28, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package knowledge is the unified read path for platform knowledge (#632).

The platform holds knowledge in several stores that an agent must today search separately (captured memory, reviewed insights, managed assets, and later the technical catalog and prompts). Each store has its own tool, its own scope rules, and its own relevance scoring, so the agent pays a discovery tax to find anything and usually declines to pay it.

This package collapses those stores behind one Provider interface and a Router that fans a single query across every registered provider, normalizes each provider's local relevance score onto a common scale, fuses the results into one ranked list, and enforces per-user scope so a shared search can never surface one user's private records to another.

The same Router is exposed two ways from one code path: as the search agent tool (pull), and later as a retriever wired into the enrichment middleware (push). PR1 (#632) builds the pull path with the memory, insights, and assets providers; the technical catalog (datahub) and prompt providers, and push injection, land in follow-up PRs.

Index

Constants

View Source
const SourceAssets = "assets"

SourceAssets is the provenance label for asset-provider hits.

View Source
const SourceCatalog = "catalog"

SourceCatalog is the provenance label for technical-catalog hits.

View Source
const SourceConnections = "connections"

SourceConnections is the provenance label for connection hits.

View Source
const SourceContextDocuments = "context_documents"

SourceContextDocuments is the provenance label for DataHub context-document hits.

View Source
const SourceEndpoints = "endpoints"

SourceEndpoints is the provenance label for API-endpoint hits.

View Source
const SourceFeedback = "feedback"

SourceFeedback is the provenance label for feedback-thread hits.

View Source
const SourceInsights = "insights"

SourceInsights is the provenance label for insight-provider hits.

View Source
const SourceKnowledgePages = "knowledge_pages"

SourceKnowledgePages is the provenance label for knowledge-page hits.

View Source
const SourceMemory = "memory"

SourceMemory is the provenance label for memory-provider hits.

View Source
const SourcePrompts = "prompts"

SourcePrompts is the provenance label for prompt hits.

Variables

View Source
var (
	// ErrUnknownSource is returned when the requested source name matches no known
	// source (a typo or an unsupported name).
	ErrUnknownSource = errors.New("unknown source")
	// ErrSourceNotBrowsable is returned when the source is known but cannot be
	// enumerated: it is not configured on this deployment, it does not implement
	// browse, or the caller's identity does not grant it.
	ErrSourceNotBrowsable = errors.New("source cannot be enumerated")
)

Errors the browse surface distinguishes so the caller can explain a failed enumeration precisely instead of returning an opaque empty page.

View Source
var ErrNotFound = errors.New("reference not found")

ErrNotFound is the fetch contract's sentinel for a reference that resolves to no content: no provider owns it, it never existed, it was deleted, or it names content the caller may not read. The Router returns it (wrapped) from Fetch, and the fetch surface maps it to a structured not-found rather than an error, so a stale citation is a normal answer ("that reference is gone"), not a failure (#694). A Fetcher signals the same condition for a reference it owns by returning ErrNotFound; the Router passes it through unchanged.

Functions

func KnownSources added in v1.91.0

func KnownSources() []string

KnownSources returns the valid search-source names (the Source* provenance constants) sorted, so other packages and their drift guards validate against one authority instead of a hand-maintained copy.

Types

type AssetSearcher added in v1.92.0

type AssetSearcher interface {
	SearchAssets(ctx context.Context, q portal.AssetSearchQuery) ([]portal.ScoredAsset, error)
	Get(ctx context.Context, id string) (*portal.Asset, error)
}

AssetSearcher is what the provider needs from the portal asset store: relevance search over a caller's assets (the text path) and a by-id read (fetch). The concrete postgres asset store satisfies it; declared here so the provider depends on the capability and the platform asserts one authority for "a searchable, fetchable asset store".

type AssetsProvider

type AssetsProvider struct {
	// contains filtered or unexported fields
}

AssetsProvider exposes a caller's managed assets (saved artifacts) to the router. It is per-user: results are restricted to assets the caller owns (assets.owner_id == caller UUID), which is why it keys on Caller.UserID rather than the email the memory and insight providers use.

func NewAssetsProvider

func NewAssetsProvider(searcher AssetSearcher) *AssetsProvider

NewAssetsProvider builds the assets provider over an asset searcher.

func (*AssetsProvider) Fetch added in v1.92.0

func (p *AssetsProvider) Fetch(ctx context.Context, ref string, caller Caller) (*Document, bool, error)

Fetch dereferences an mcp:asset:<id> reference to the asset's full metadata (#694), folding what manage_artifact's get returns into the one fetch verb. It owns only the asset reference form; any other reference is declined (owned=false). Assets are per-user, so the read is scoped to the caller exactly as Search is: an asset the caller does not own, a missing id, or a soft-deleted asset all return ErrNotFound, so fetch never reveals another owner's asset (or even its existence). The blob bytes live in S3 and are reached with s3_get_object / s3_presign_url; this returns the metadata record (name, description, tags, S3 location, size, provenance).

func (*AssetsProvider) Name

func (*AssetsProvider) Name() string

Name returns the provenance label.

func (*AssetsProvider) Scope

func (*AssetsProvider) Scope() Scope

Scope marks this provider per-user; the router supplies the caller identity and must skip it when that identity is absent.

func (*AssetsProvider) Search

func (p *AssetsProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns the caller's assets ranked by relevance. It fails closed on a missing caller UUID rather than searching across all owners.

type BrowsePage added in v1.93.0

type BrowsePage struct {
	Hits   []Hit `json:"hits"`
	Total  int   `json:"total"`
	Offset int   `json:"offset"`
	Limit  int   `json:"limit"`
}

BrowsePage is one page of an enumeration: the members on this page (as Hits, so each carries the same Reference search emits and fetch consumes), the Total members in the source (so a caller knows how many pages remain), and the effective Offset/Limit the Router applied after clamping.

type BrowseQuery added in v1.93.0

type BrowseQuery struct {
	Caller Caller
	Offset int
	Limit  int
}

BrowseQuery is one enumeration request: page the source from Offset for Limit members, scoped to Caller. Unlike a search Query it carries no intent and no entity URNs; it enumerates a single source in full rather than ranking across many.

type Browser added in v1.93.0

type Browser interface {
	// Browse returns one page of this provider's members. It sets Hits and Total;
	// the Router stamps the effective Offset and Limit onto the returned page.
	Browse(ctx context.Context, q BrowseQuery) (BrowsePage, error)
}

Browser is the optional capability of a Provider to enumerate its members in full, paginated and with a total count and no relevance threshold (#695). It is the complement of Search: Search ranks a relevant few, Browse lists the whole set so an agent can audit, dedup, govern, or migrate the corpus. A provider that does not implement Browser is search-only, and the Router reports its source as not browsable.

A per-user provider's Browse must scope to the caller exactly as its Search does; the Router additionally refuses to browse a per-user provider for an anonymous caller, so enumeration can never widen what a caller could otherwise see.

type Caller

type Caller struct {
	// UserID is the caller's UUID identity, the owner key for assets.
	UserID string
	// Email is the caller's canonical identity, the owner key for memory
	// and insights.
	Email string
	// Persona is the caller's resolved persona. It scopes entity-keyed memory
	// lookups and selects which persona-scoped prompts are visible. It is not a
	// security boundary on its own (per-user records are scoped by Email/UserID);
	// it narrows persona-targeted content.
	Persona string
}

Caller is the resolved identity of the requester. Per-user providers scope on it, and they do not all key on the same field: captured memory and insights are owned by email (memory_records.created_by), while managed assets are owned by the user's UUID (assets.owner_id). Both fields travel on every request so each provider selects the one it scopes on; a provider whose key is empty must return no results rather than query unscoped.

func (Caller) Anonymous

func (c Caller) Anonymous() bool

Anonymous reports whether the caller carries no identity at all. The Router skips every per-user provider for an anonymous caller.

type CatalogProvider added in v1.91.0

type CatalogProvider struct {
	// contains filtered or unexported fields
}

CatalogProvider exposes the technical catalog (DataHub) to the router. It serves two query shapes: a relevance search on Intent (folding datahub_search into search) and an exact entity-keyed lookup on EntityURNs that returns the catalog entity itself, so handing search a dataset URN surfaces its catalog entry alongside the URN-linked memory and insights. Structured catalog navigation (platform/domain/tag/entity-type filters) stays in datahub_browse.

It is shared: the catalog is global, so it is queried for every request and needs no caller identity.

DataHub ranks search results but does not return a numeric score, so the provider derives a descending positional score from the result order; entity-keyed hits take the max score. The router's per-provider normalization then places these on the common scale.

func NewCatalogProvider added in v1.91.0

func NewCatalogProvider(searcher tableSearcher) *CatalogProvider

NewCatalogProvider builds the catalog provider over a catalog searcher.

func (*CatalogProvider) Fetch added in v1.92.0

func (p *CatalogProvider) Fetch(ctx context.Context, ref string, _ Caller) (*Document, bool, error)

Fetch dereferences a urn:li:dataset:<id> reference to the dataset's full catalog context (#694), folding what datahub_get_entity returns into the one fetch verb. It owns only the dataset URN form; any other reference is declined (owned=false). A URN that does not parse as a dataset, that the catalog has no entry for, or that the catalog errors on is ErrNotFound: DataHub reports a missing/deleted entity as an error rather than an empty result (mcp-datahub GetEntity), and the search entity path treats that same lookup error as a skip (searchByEntity), so a stale dataset citation must be a clean not-found here too, not a hard tool failure. The catalog is global, so no per-caller scope applies.

func (*CatalogProvider) Name added in v1.91.0

func (*CatalogProvider) Name() string

Name returns the provenance label.

func (*CatalogProvider) Scope added in v1.91.0

func (*CatalogProvider) Scope() Scope

Scope marks the catalog shared (global, always queried).

func (*CatalogProvider) Search added in v1.91.0

func (p *CatalogProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns catalog entities for the query: the entities named by EntityURNs (entity path) plus those relevant to Intent (text path), merged and de-duplicated by URN.

type ConnectionInfo

type ConnectionInfo struct {
	Name        string
	Kind        string
	Description string
}

ConnectionInfo is one configured data connection, reduced to the fields a relevance search needs. The knowledge package defines it (rather than importing the platform's connection types) so the federation engine stays decoupled; the platform adapts its connection registry to ConnectionLister.

type ConnectionLister

type ConnectionLister interface {
	Connections() []ConnectionInfo
}

ConnectionLister enumerates the deployment's configured connections. The platform implements it over the same toolkit registry list_connections uses, so the search corpus and the connections tool stay in agreement.

type ConnectionsProvider

type ConnectionsProvider struct {
	// contains filtered or unexported fields
}

ConnectionsProvider exposes configured connections to the router as a relevance search. Connections are in the default corpus by design (#645): an agent should discover that, say, a "stripe" or "warehouse" connection exists from one search, rather than having to know to call list_connections first. list_connections stays the full enumeration; this surfaces the connections relevant to a query.

It is shared: connection metadata (name, kind, description) is already globally visible through list_connections, so there is no per-user record to scope here. The security-sensitive boundary is which connections a persona may use, and that is enforced fail-closed at tool-call time on the scoped tools (trino_query, api_invoke_endpoint, ...), unchanged by this provider.

func NewConnectionsProvider

func NewConnectionsProvider(lister ConnectionLister) *ConnectionsProvider

NewConnectionsProvider builds the connections provider over a lister.

func (*ConnectionsProvider) Fetch added in v1.92.0

Fetch dereferences an mcp:connection:(kind,name) reference to the connection's descriptor (#694), folding what list_connections enumerates into the one fetch verb. It owns only the connection reference form; any other reference is declined (owned=false). Connection metadata is global (the same list_connections exposes), so no per-caller scope applies; a reference that matches no current connection is ErrNotFound. As with search, this surfaces only the descriptor: which personas may USE a connection is enforced fail-closed at tool-call time on the scoped tools, unchanged by this read.

func (*ConnectionsProvider) Name

func (*ConnectionsProvider) Name() string

Name returns the provenance label.

func (*ConnectionsProvider) Scope

func (*ConnectionsProvider) Scope() Scope

Scope marks connections shared: their metadata is already global via list_connections.

func (*ConnectionsProvider) Search

func (p *ConnectionsProvider) Search(_ context.Context, q Query) ([]Hit, error)

Search returns connections whose name, kind, or description match the intent, ranked by a lexical token-overlap score. Connections carry no embeddings, so ranking is lexical; the score still feeds the allocator's per-source normalization. It responds to the text path only.

type ContextDocumentsProvider added in v1.91.0

type ContextDocumentsProvider struct {
	// contains filtered or unexported fields
}

ContextDocumentsProvider surfaces DataHub context documents in unified search (#692). Context documents are the non-dataset knowledge home that predates knowledge pages; before this source they were undiscoverable from search (the catalog provider only searched datasets), so the knowledge in them was stranded. The catalog is global, so this is a shared source queried for every request.

func NewContextDocumentsProvider added in v1.91.0

func NewContextDocumentsProvider(searcher semantic.DocumentSearcher) *ContextDocumentsProvider

NewContextDocumentsProvider builds the documents provider over a document searcher.

func (*ContextDocumentsProvider) Browse added in v1.93.0

Browse enumerates context documents in full (#695): the offset/limit page of the complete corpus plus the total document count, with no relevance threshold and no visibility/status filter, so a governance sweep sees every document (drafts and hidden ones included) and the page matches the total. The catalog is global, so no per-caller scope applies. Each member carries the same Reference search emits, so a browse page feeds directly into fetch.

func (*ContextDocumentsProvider) Fetch added in v1.92.0

Fetch dereferences a urn:li:document:<id> reference to the document's full body (#694). Context documents are snippet-only in search; this is the only MCP path to their complete content. It owns only the document URN form; any other reference is declined (owned=false). A URN that resolves to no document (semantic.ErrDocumentNotFound) is ErrNotFound.

No publication or visibility filter is applied: browse (#695) enumerates the whole corpus, drafts and hidden documents included, so a draft is a legitimately discoverable reference and fetch must read it (else browse would list documents fetch refuses, and a migration task could not open the drafts it is sent to clean up). The catalog is global, so no per-caller scope applies; relevance search still hides drafts and hidden documents from its ranked results (the curated view), while browse + fetch are the exhaustive read path.

func (*ContextDocumentsProvider) Name added in v1.91.0

Name returns the provenance label.

func (*ContextDocumentsProvider) Scope added in v1.91.0

Scope marks the catalog shared (global, always queried).

func (*ContextDocumentsProvider) Search added in v1.91.0

func (p *ContextDocumentsProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns context documents for the query: those linked to the requested EntityURNs (entity path) plus those relevant to Intent (text path), merged and de-duplicated by document URN so a document found both ways appears once, the entity match ranked first at the exact-match score (mirroring CatalogProvider and PagesProvider). Both arms exclude drafts; the text arm additionally excludes documents hidden from global search, which the entity (linked-asset) arm surfaces.

type Document added in v1.92.0

type Document struct {
	// Reference is the canonical citation that was dereferenced (echoed back so a
	// caller can re-cite or cache by it).
	Reference string `json:"reference"`
	// Source is the provider provenance label (the same Source a search Hit carries).
	Source string `json:"source"`
	// Title is a short human label for the content (page title, document title,
	// dataset name, asset name, prompt display name, connection name).
	Title string `json:"title,omitempty"`
	// Body is the full textual content for text-bodied sources (knowledge page
	// markdown, context-document body, prompt content). Empty for purely structured
	// sources (a dataset, an asset, a connection), which carry their payload in
	// Content instead.
	Body string `json:"body,omitempty"`
	// Content is the source-native full payload for sources whose record is
	// structured (a dataset's TableContext, an asset's metadata, a connection
	// descriptor, the full prompt record). A source may populate both Body and
	// Content when it has a primary text body and structured metadata around it (a
	// prompt fills Body with its text and Content with the whole record); a purely
	// text-bodied source (knowledge page, context document) leaves Content nil.
	Content any `json:"content,omitempty"`
	// EntityURNs are the catalog entities this content is about, when the source
	// links any (a context document's related assets, a dataset's own URN).
	EntityURNs []string `json:"entity_urns,omitempty"`
	// References are the outbound links this content declares: the other pages and
	// entities it points to, as citable references a caller can fetch in turn (#705).
	// A knowledge page populates these from its tracked entity references, so an agent
	// can deliberately deep-crawl the knowledge graph (fetch the index, follow into
	// the relevant branch) instead of re-parsing links out of the markdown body.
	References []DocumentRef `json:"references,omitempty"`
}

Document is the full, dereferenced content of a reference that search emitted on a Hit. It is the consumer of the Reference field that every searchable source already produces: search returns navigational pointers with truncated snippets, and fetch turns one pointer back into the whole record.

A source fills the shape that fits it: text-bodied sources (knowledge pages, context documents, prompts) put their full text in Body, while structured sources (a catalog dataset's context, a managed asset's metadata, a connection's descriptor) put their native payload in Content. Reference echoes the resolved reference and Source is the provider provenance, so a caller can cite the result the same way search let it cite the pointer.

type DocumentRef added in v1.94.0

type DocumentRef struct {
	Reference string `json:"reference"`
	Type      string `json:"type"`
}

DocumentRef is one outbound link a Document declares (#705): a reference string a caller can fetch, plus its target type so the caller can decide whether to crawl it (another knowledge page, a dataset, an asset, a connection, ...).

type EndpointCandidate

type EndpointCandidate struct {
	Connection  string
	OperationID string
	Method      string
	Path        string
	Summary     string
	Spec        string
	Score       float64
}

EndpointCandidate is one API operation matched by an endpoint searcher, already ranked and scoped within its API gateway connection. The knowledge package defines it (rather than importing the apigateway concrete) so the federation engine stays decoupled from any one toolkit; the platform adapts each apigateway toolkit to EndpointSearcher.

type EndpointSearcher

type EndpointSearcher interface {
	SearchEndpoints(ctx context.Context, intent string, limit int) ([]EndpointCandidate, error)
}

EndpointSearcher ranks API operations across the connections of one API gateway toolkit, applying that toolkit's per-connection route policy so a caller never sees an operation their persona could not invoke. The platform wires one EndpointSearcher per apigateway toolkit.

type EndpointsProvider

type EndpointsProvider struct {
	// contains filtered or unexported fields
}

EndpointsProvider exposes API endpoints to the router as a relevance search, aggregated across every API gateway toolkit. API endpoints are in the default corpus by design (#645): an agent searching "customer retention" should see a relevant operation next to the dataset and the insight without first having to know an API gateway exists, list connections, and search each one. api_list_endpoints stays the scoped drill-down, the way datahub_browse is the scoped counterpart to catalog search.

It is shared: endpoints are global to the deployment, and each searcher enforces its own per-connection route policy fail-closed, so the provider needs no caller identity of its own.

func NewEndpointsProvider

func NewEndpointsProvider(searchers ...EndpointSearcher) *EndpointsProvider

NewEndpointsProvider builds the endpoints provider over one or more endpoint searchers (one per API gateway toolkit).

func (*EndpointsProvider) Name

func (*EndpointsProvider) Name() string

Name returns the provenance label.

func (*EndpointsProvider) Scope

func (*EndpointsProvider) Scope() Scope

Scope marks endpoints shared (always queried); each searcher self-filters operations to those the caller's persona may invoke.

func (*EndpointsProvider) Search

func (p *EndpointsProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns API operations relevant to the intent, aggregated across every configured API gateway. It responds to the text path only; a query with no intent yields nothing. A single searcher erroring is logged and skipped so one unhealthy gateway does not blank the endpoints group.

type Fetcher added in v1.92.0

type Fetcher interface {
	// Fetch dereferences ref to its full content for caller.
	//
	// owned reports whether ref is a form this provider recognizes. When false the
	// Router moves on to the next provider and doc/err are ignored, so a provider
	// must cheaply decline references that are not its own rather than erroring.
	//
	// When owned is true: a nil err returns doc; ErrNotFound means the reference is
	// well-formed and this provider's, but resolves to nothing the caller may read
	// (deleted, never existed, or out of the caller's scope) and is rendered as a
	// structured not-found; any other err is a real fetch failure (store/transport).
	//
	// A per-user provider must scope the read to caller exactly as its Search does:
	// a reference the caller could not have searched must return ErrNotFound, not
	// content, so fetch never widens what a persona can see.
	Fetch(ctx context.Context, ref string, caller Caller) (doc *Document, owned bool, err error)
}

Fetcher is the optional capability of a Provider to dereference a reference (the canonical citation string search emits on a Hit) to its full content. A provider that can search a store can usually also read one record from it by id; implementing Fetcher exposes that as the other half of search. A provider that does not implement Fetcher is search-only, and the Router skips it when fetching.

The Router asks each scope-permitted Fetcher in turn whether it owns a reference and returns the first owner's content, so ownership across providers is partitioned by reference form (mcp:knowledge_page:, urn:li:dataset:, ...) and never ambiguous.

type Hit

type Hit struct {
	Text       string   `json:"text"`
	Source     string   `json:"source"`
	Ref        string   `json:"ref"`
	Score      float64  `json:"score"`
	Status     string   `json:"status,omitempty"`
	EntityURNs []string `json:"entity_urns,omitempty"`
	Dimension  string   `json:"dimension,omitempty"`
	// CapturedBy is the author (email) of an authored result such as an insight,
	// so a reviewer can see who recorded it. Omitted for sources with no single
	// author (catalog datasets, API endpoints, connections).
	CapturedBy string `json:"captured_by,omitempty"`
	// Reference is the canonical citation string for the hit's entity
	// (mcp:<type>:<key> for an internal entity, urn:... for DataHub), so an agent
	// can reference it from a knowledge page without hand-assembling or guessing
	// the form. Omitted when the entity is not referenceable.
	Reference string `json:"reference,omitempty"`
}

Hit is one knowledge record matched by a provider. Score is the provider's own relevance score; the Router normalizes it across providers before fusing, so callers see a fused rank, not the raw provider score. Source is the provider name, surfaced as provenance. Ref is the record's stable identifier within its source (memory id, insight id, asset id) so a caller can fetch the full record.

The optional fields carry what the specialized search tools returned, so folding them into one search loses nothing: Status is a review or lifecycle state (insight pending/approved/...), EntityURNs are the linked catalog entities (provenance), and Dimension is the memory dimension or category. They are omitted when a source does not populate them.

Temporal validity (valid_from/valid_until) and a live-vs-captured freshness flag remain deferred until a provider populates them (the wiki carries season windows); adding them now would be unexercised fields.

type InsightsProvider

type InsightsProvider struct {
	// contains filtered or unexported fields
}

InsightsProvider exposes captured domain knowledge (insights) to the router.

Insights are knowledge-dimension memory rows owned by the caller (insight.captured_by == caller email). The underlying store scopes to that owner and to the knowledge dimension, so this provider covers exactly the records the MemoryProvider skips.

Scope note (#632): the epic envisions reviewed insights becoming shared across callers. The current store has no review-state-aware sharing, and searching it without an owner would expose every user's personal insights, so PR1 keeps this provider per-user. Promoting reviewed insights to ScopeShared is deferred to the write-path/review work (#633).

func NewInsightsProvider

func NewInsightsProvider(store insightSource) *InsightsProvider

NewInsightsProvider builds the insights provider over a searchable insight store.

func (*InsightsProvider) Fetch added in v1.93.0

func (p *InsightsProvider) Fetch(ctx context.Context, ref string, caller Caller) (*Document, bool, error)

Fetch dereferences an mcp:insight:<id> reference to the full insight (#699), following the AssetsProvider precedent. Insights are per-user, so the read is scoped to the caller: it returns an insight only when the caller captured it (captured_by == caller email); a non-owner, a missing id, or an anonymous caller is ErrNotFound, so fetch never reveals an insight the caller could not have searched. It does NOT additionally gate on review status: Search retracts non-live insights only from the default (no-status) discovery path, while an explicit status query surfaces them, so a caller can search any of their own insights by status and fetch must dereference any reference search hands out. The knowledge-dimension scope is enforced by the store adapter's Get, so a reference that names a non-knowledge memory record resolves to not-found here.

func (*InsightsProvider) Name

func (*InsightsProvider) Name() string

Name returns the provenance label.

func (*InsightsProvider) Scope

func (*InsightsProvider) Scope() Scope

Scope marks this provider per-user; see the type doc for why reviewed-insight sharing is deferred.

func (*InsightsProvider) Search

func (p *InsightsProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns the caller's captured insights. It serves both query shapes: an exact entity-keyed lookup on EntityURNs (insights linked to the requested datasets, lineage-expanded by the Router) and a relevance search on Intent. Results from both paths are merged and de-duplicated by insight id. Each hit carries the insight's review status and linked entity URNs as provenance. It fails closed on a missing caller email rather than searching across all users.

type LineageExpander

type LineageExpander interface {
	Expand(ctx context.Context, urns []string) []string
}

LineageExpander optionally widens a set of entity URNs along lineage so an entity-keyed lookup also recalls knowledge about upstream and downstream datasets (the old memory_recall "graph" strategy). Implemented by an adapter over the semantic provider; a nil expander disables expansion, leaving a plain entity lookup.

It lives on the Router, not on any single provider, so the expansion runs once per search and every entity-keyed provider (memory, insights, the technical catalog) sees the same widened URN set, the same way the query embedding is computed once and shared.

func NewLineageExpander added in v1.88.0

func NewLineageExpander(sem semantic.Provider) LineageExpander

NewLineageExpander returns a LineageExpander backed by a DataHub semantic provider. Pass it to NewRouter so entity-keyed searches recall knowledge about upstream and downstream datasets, not just the exact URNs.

type MemoryProvider

type MemoryProvider struct {
	// contains filtered or unexported fields
}

MemoryProvider exposes a caller's personal memory to the knowledge router.

It is per-user: results are restricted to records the caller owns (memory_records.created_by == caller email), the same identity the portal's "my knowledge" search scopes on. It serves two query shapes: relevance search on Intent, and an exact entity-keyed lookup on EntityURNs. Lineage expansion of the entity URNs is the Router's responsibility (so it runs once for every entity-keyed provider), so this provider takes the URN set as given.

It deliberately omits the knowledge dimension on both paths. Captured insights and remembered knowledge are knowledge-dimension memory rows owned by the InsightsProvider; surfacing them here too would double-list the same record. This provider covers the caller's non-knowledge memory (preferences, events, entities, relationships).

func NewMemoryProvider

func NewMemoryProvider(store memorySearcher) *MemoryProvider

NewMemoryProvider builds the memory provider over a memory store.

func (*MemoryProvider) Fetch added in v1.93.0

func (p *MemoryProvider) Fetch(ctx context.Context, ref string, caller Caller) (*Document, bool, error)

Fetch dereferences an mcp:memory:<id> reference to the full memory record (#699), following the AssetsProvider precedent. Memory is per-user, so the read is scoped to the caller exactly as Search is: it returns a record only when the caller owns it (created_by == caller email), it is active, and it is not a knowledge-dimension record (those are insights, addressed by mcp:insight:); anything else, a missing id, or an anonymous caller is ErrNotFound, so fetch never reveals a record the caller could not have searched (nor even its existence).

func (*MemoryProvider) Name

func (*MemoryProvider) Name() string

Name returns the provenance label.

func (*MemoryProvider) Scope

func (*MemoryProvider) Scope() Scope

Scope marks this provider per-user; the router supplies the caller identity and must skip it when that identity is absent.

func (*MemoryProvider) Search

func (p *MemoryProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns the caller's active, non-knowledge memory. When EntityURNs are given it does an exact entity lookup (lineage-expanded when configured); when Intent is given it ranks by relevance (hybrid with an embedding, lexical otherwise). Results from both paths are merged and de-duplicated by record id. It fails closed: an empty caller email yields no results rather than an unscoped search across all users.

type PageSearcher added in v1.89.0

type PageSearcher interface {
	Search(ctx context.Context, q knowledgepage.SearchQuery) ([]knowledgepage.ScoredPage, error)
	ListPagesReferencing(ctx context.Context, ref knowledgepage.EntityRef) ([]knowledgepage.PageRef, error)
	// Get reads one page by id (the other half of search: a hit's reference
	// dereferenced to the full body). Returns knowledgepage.ErrNotFound for a
	// missing id.
	Get(ctx context.Context, id string) (*knowledgepage.Page, error)
	// List returns the offset/limit page of live pages plus the total live-page
	// count, ordered deterministically, for exhaustive enumeration (#695).
	List(ctx context.Context, filter knowledgepage.Filter) ([]knowledgepage.Page, int, error)
	// ListEntityRefs returns the references a page declares (the pages and entities
	// it links to), so a fetch can surface the page's outbound graph edges (#705).
	ListEntityRefs(ctx context.Context, pageID string) ([]knowledgepage.EntityRef, error)
}

PageSearcher is what the provider needs from the knowledge-page store: relevance search over page content (the text path) and the reverse lookup of the pages that reference an entity (the entity path, #634). Declared locally so the provider depends on the capability and tests can supply a fake.

type PagesProvider added in v1.88.0

type PagesProvider struct {
	// contains filtered or unexported fields
}

PagesProvider exposes the platform's canonical knowledge pages (the internal-knowledge home for business/domain ontology) to the router. Pages are org-shared, so this provider is shared: it is queried for every request and needs no caller identity, and it never holds per-user records.

func NewKnowledgePagesProvider added in v1.88.0

func NewKnowledgePagesProvider(searcher PageSearcher) *PagesProvider

NewKnowledgePagesProvider builds the knowledge-pages provider over a searcher.

func (*PagesProvider) Browse added in v1.93.0

Browse enumerates knowledge pages in full (#695): the offset/limit page of live pages plus the total live-page count, with no relevance threshold, so an agent can page the whole corpus to audit or migrate it. Pages are org-shared, so no per-caller scope applies; the store's List excludes soft-deleted pages and orders by (updated_at DESC, id) - a deterministic total order whose unique id tiebreaker keeps pagination stable across pages even when timestamps collide, so a sweep neither skips nor double-returns a page of a fixed corpus. Each member carries the same Reference search emits, so a browse page feeds directly into fetch.

func (*PagesProvider) Fetch added in v1.92.0

func (p *PagesProvider) Fetch(ctx context.Context, ref string, _ Caller) (*Document, bool, error)

Fetch dereferences an mcp:knowledge_page:<id> reference to the page's full body (#694), the consumer the search snippet was built to anticipate ("a hit conveys what the page covers without a fetch"). It owns only the knowledge-page reference form; any other reference is declined (owned=false) so the Router tries the next provider. A missing id, or a soft-deleted page, is ErrNotFound: a page-handler Get returns soft-deleted rows (it is the editor's undelete path), so the live read must filter them exactly as the portal HTTP handler does.

func (*PagesProvider) Name added in v1.88.0

func (*PagesProvider) Name() string

Name returns the provenance label.

func (*PagesProvider) Scope added in v1.88.0

func (*PagesProvider) Scope() Scope

Scope marks knowledge pages shared (global canonical knowledge, always queried).

func (*PagesProvider) Search added in v1.88.0

func (p *PagesProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns knowledge pages matching the query: the pages that REFERENCE each requested entity (the entity path, via the reverse lookup) plus the pages relevant to the intent (the text path, ranked over title/body/tags), merged and de-duplicated by page id. So an entity-keyed search surfaces "the knowledge about this entity", not just text matches (#634).

type PromptSearcher added in v1.92.0

type PromptSearcher interface {
	Search(ctx context.Context, q prompt.SearchQuery) ([]prompt.ScoredPrompt, error)
	GetByID(ctx context.Context, id string) (*prompt.Prompt, error)
}

PromptSearcher is what the prompts provider needs from the prompt store: relevance search over visible prompts (the text path) and a by-id read (fetch). The concrete postgres prompt store satisfies it; declared here so the provider depends on the capability and the platform asserts one authority for "a searchable, fetchable prompt store".

type PromptsProvider

type PromptsProvider struct {
	// contains filtered or unexported fields
}

PromptsProvider exposes operational prompts to the router. Prompt visibility is mixed: global prompts are visible to everyone, while persona- and personal-scoped prompts are visible only to the matching caller. The underlying searcher enforces that visibility from the caller identity, so the provider is shared (always queried, returning at least the global prompts even for an anonymous caller) yet never leaks another caller's personal prompts.

func NewPromptsProvider

func NewPromptsProvider(searcher PromptSearcher) *PromptsProvider

NewPromptsProvider builds the prompts provider over a prompt searcher.

func (*PromptsProvider) Fetch added in v1.92.0

func (p *PromptsProvider) Fetch(ctx context.Context, ref string, caller Caller) (*Document, bool, error)

Fetch dereferences an mcp:prompt:<id> reference to the prompt's full content (#694), folding what manage_prompt's get returns into the one fetch verb. It owns only the prompt reference form; any other reference is declined (owned=false).

GetByID reads a prompt regardless of visibility, so this re-applies the same scope filter the search path enforces: a global prompt is visible to everyone, a persona prompt only to a caller carrying that persona, and a personal prompt only to its owner. A prompt the caller could not have searched (wrong persona, another owner's personal prompt) returns ErrNotFound, so fetch never widens persona scope; a missing id is likewise ErrNotFound.

func (*PromptsProvider) Name

func (*PromptsProvider) Name() string

Name returns the provenance label.

func (*PromptsProvider) Scope

func (*PromptsProvider) Scope() Scope

Scope marks prompts shared (always queried); the searcher self-filters persona/personal prompts to the caller.

func (*PromptsProvider) Search

func (p *PromptsProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns prompts visible to the caller, ranked by relevance to the intent. It responds to the text path only; a query with no intent yields nothing.

type Provider

type Provider interface {
	Name() string
	Scope() Scope
	Search(ctx context.Context, q Query) ([]Hit, error)
}

Provider is one searchable knowledge store behind the Router. Name is the provenance label stamped on every Hit. Scope drives the Router's access rules. Search returns the provider's own ranked hits for the query; the Router owns cross-provider normalization and fusion, so a provider only needs to rank within itself.

type Query

type Query struct {
	Intent     string
	Embedding  []float32
	EntityURNs []string
	Status     string
	Caller     Caller
	Limit      int
	Sources    []string
}

Query is one knowledge search. It carries two complementary ways to match, and a provider uses whichever it supports:

  • Intent is natural-language text matched by relevance. Embedding is the query vector the Router computes once from Intent and shares across providers; nil selects lexical-only ranking.
  • EntityURNs is an exact, entity-keyed lookup: return knowledge linked to these DataHub URNs (memory uses this, optionally expanded along lineage).

At least one of Intent or EntityURNs is set. Status optionally filters by lifecycle/review state where a provider tracks one (insight review status). Caller carries the identity per-user providers scope on. Limit caps the candidate list each provider returns before the allocator builds the balanced display set.

Sources optionally narrows the federation to a subset of provider names (e.g. ["datahub"]). It only narrows: an empty Sources queries every provider the caller can access, and a name in Sources never opts a caller into a provider their scope would otherwise exclude.

type Result

type Result struct {
	Groups   []SourceGroup
	Coverage []SourceCoverage
	Ranking  string
	// UnknownSources lists requested Sources names that match no known source (a
	// typo or an unsupported source), so a caller is told why a narrowed search
	// returned little or nothing instead of being silently given an empty result.
	UnknownSources []string
}

Result is one knowledge search response: the balanced, grouped-by-source display set, the coverage summary (per-source matched vs shown counts so the agent sees breadth beyond what is displayed), and the ranking mode used to produce it.

type Router

type Router struct {
	// contains filtered or unexported fields
}

Router fans one query across every registered provider, normalizes each provider's local relevance scores onto a common scale, fuses them into one ranked list, and enforces per-user scope. It is the single read path behind both the search tool and (later) push injection, so the scope and fusion rules live here once rather than in each surface.

func NewRouter

func NewRouter(embedder embedding.Provider, lineage LineageExpander, providers ...Provider) *Router

NewRouter builds a router over an embedder, an optional lineage expander, and a set of providers. The embedder may be nil or the noop placeholder; the router then ranks lexically. lineage may be nil, leaving entity-keyed lookups unexpanded. Provider order does not affect ranking (scores are fused), only the deterministic tie-break.

func (*Router) BrowsableSources added in v1.93.0

func (r *Router) BrowsableSources() []string

BrowsableSources returns the names of the sources that implement Browser, sorted, so a caller (and the tool description) can name exactly what can be enumerated on this deployment rather than guessing.

func (*Router) Browse added in v1.93.0

func (r *Router) Browse(ctx context.Context, source string, q BrowseQuery) (BrowsePage, error)

Browse enumerates a single source in full: the page of members at the requested offset plus the source's total member count. It is the exhaustive counterpart to Search, for auditing or migrating a corpus that relevance ranking cannot list.

The source must be named exactly (browse enumerates one source; a single offset and total are meaningless across a federation). A name that matches no known source returns ErrUnknownSource; a known source that is not configured here, does not implement Browser, or is a per-user source for an anonymous caller returns ErrSourceNotBrowsable. Offset is floored at zero and Limit is clamped to the browse bounds; both effective values are echoed on the returned page.

func (*Router) Fetch added in v1.92.0

func (r *Router) Fetch(ctx context.Context, ref string, caller Caller) (*Document, error)

Fetch dereferences a reference to its full content. It asks each scope-permitted Fetcher provider, in registration order, whether it owns the reference and returns the first owner's content.

Scope mirrors Search exactly: a per-user provider is consulted only when the caller carries an identity (an anonymous caller is never offered per-user content), and each provider re-applies its own ownership filter, so fetch can never read content the same caller could not have searched. A reference no provider owns, an empty reference, or one an owner cannot resolve all return ErrNotFound, which the caller renders as a structured not-found.

Unlike Search this does not fan out concurrently: ownership is partitioned by reference form, so exactly one provider does real work and the rest decline in a cheap prefix check; a serial walk is both correct and the lower-overhead choice.

func (*Router) Providers

func (r *Router) Providers() []Provider

Providers returns the registered providers, for introspection and wiring checks.

func (*Router) Search

func (r *Router) Search(ctx context.Context, q Query) (Result, error)

Search runs one knowledge search from a caller-built Query. It embeds the intent once (when present) and shares the vector across providers, queries every shared provider plus every per-user provider for which the caller carries an identity, fuses the results, and trims to limit. The query may be text-based (Intent), entity-keyed (EntityURNs), or both; each provider uses the parts it supports.

Provider failures are tolerated: a single provider erroring is logged and its results omitted, so one unhealthy store does not blank the whole search. An error is returned only when every queried provider failed, so an all-stores- down condition is not reported as an empty-but-successful result.

type Scope

type Scope int

Scope declares whether a provider's records are visible to every caller or only to the caller who owns them. The Router uses it to decide which providers a request may touch and with what identity.

const (
	// ScopeShared marks a provider that is queried for every request, with or
	// without a caller identity, because it can always return at least some
	// content visible to everyone (the technical catalog, global prompts). A
	// shared provider may still use the caller identity to widen what it
	// returns (a prompt provider adds the caller's persona/personal prompts to
	// the global ones); "shared" means "always queried", not "ignores the
	// caller". It must never return another caller's private records.
	ScopeShared Scope = iota

	// ScopePerUser marks a provider whose records belong to individual
	// callers (personal memory, personal assets). The Router queries a
	// per-user provider only when the request carries the identity that
	// provider scopes on, and the provider must restrict results to that
	// identity. This is the security boundary that keeps one user's private
	// records out of another user's search.
	ScopePerUser
)

func (Scope) String

func (s Scope) String() string

String renders a Scope for logs and test failures.

type SourceCoverage

type SourceCoverage struct {
	Source  string `json:"source"`
	Matched int    `json:"matched"`
	Shown   int    `json:"shown"`
}

SourceCoverage reports, per source, how many candidates matched the query and how many of those are shown in the grouped result. Matched can exceed Shown when the balanced allocator spent its budget elsewhere; that gap is the anti-tunnel signal that tells the agent where unshown answers live ("14 datasets matched, 3 shown"). Matched is the count of candidates the provider returned for this query, capped at the per-source candidate fetch limit, not a full-corpus count.

type SourceGroup

type SourceGroup struct {
	Source string `json:"source"`
	Hits   []Hit  `json:"hits"`
}

SourceGroup is the displayed hits for one source, in that source's own relevance order. Grouping by source (rather than one flat relevance list) is the anti-tunnel shape: the agent sees that answers exist across memory, the catalog, endpoints, and prompts at once, instead of a top list one strong source dominates.

type ThreadSearcher added in v1.90.0

type ThreadSearcher interface {
	SearchThreads(ctx context.Context, ownerEmail, intent string, limit int) ([]threads.Thread, error)
}

ThreadSearcher is the lexical feedback-thread search the provider needs. The postgres thread store implements it; declared here so the provider depends on the capability and tests can supply a fake.

type ThreadsProvider added in v1.90.0

type ThreadsProvider struct {
	// contains filtered or unexported fields
}

ThreadsProvider federates a caller's feedback threads into unified search, so feedback they have given is discoverable knowledge alongside the catalog, insights, and pages. It closes the gap where feedback was the one knowledge source not in the search corpus (#686): the loop now reads feedback, not only writes it. Threads carry no embedding, so this is a lexical source; it is per-user (author email) and fails closed on an empty caller email, never surfacing another user's feedback.

func NewThreadsProvider added in v1.90.0

func NewThreadsProvider(store ThreadSearcher) *ThreadsProvider

NewThreadsProvider builds the feedback provider over a thread searcher.

func (*ThreadsProvider) Name added in v1.90.0

func (*ThreadsProvider) Name() string

Name returns the provenance label.

func (*ThreadsProvider) Scope added in v1.90.0

func (*ThreadsProvider) Scope() Scope

Scope marks this provider per-user: a caller searches only their own feedback, because the thread store has no organization-wide visibility model and a shared scope would leak every user's feedback to every searcher.

func (*ThreadsProvider) Search added in v1.90.0

func (p *ThreadsProvider) Search(ctx context.Context, q Query) ([]Hit, error)

Search returns the caller's feedback threads relevant to the intent. It fails closed on a missing caller email rather than searching across all users, and a query with no intent yields nothing (feedback has no entity-keyed path).

Directories

Path Synopsis
Package federation adapts the platform's live toolkit registry to the knowledge package's source interfaces, so the universal search router can federate API endpoints and connections without the knowledge engine depending on any concrete toolkit.
Package federation adapts the platform's live toolkit registry to the knowledge package's source interfaces, so the universal search router can federate API endpoints and connections without the knowledge engine depending on any concrete toolkit.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL