semantic

package
v1.148.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package semantic provides semantic layer abstractions.

Package semantic provides abstractions for semantic metadata providers.

Index

Constants

View Source
const MaxStringLength = 2000

MaxStringLength is the maximum length for sanitized strings.

View Source
const TotalUnknown = -1

TotalUnknown is the match count reported when no provider in the chain can count. It is deliberately negative rather than zero so it can never be read as "no matches" or compared against a page length as if it were a total: a caller must branch on it explicitly.

Variables

View Source
var ErrNotFound = errors.New("entity not found")

ErrNotFound reports that the catalog holds no entity for the reference a by-URN read was given. The read returns it (wrapped) so a caller can tell a reference the catalog has never ingested from a transport or authorization failure, and so CachedProvider can remember the miss for its TTL the way it remembers a hit. It is defined here, on the abstraction, rather than per-implementation so every provider agrees on the sentinel (#1610).

Functions

func DetectAndLogInjection added in v1.96.0

func DetectAndLogInjection(sanitizer *Sanitizer, source, field, input string) bool

DetectAndLogInjection checks input for prompt-injection patterns and emits a structured slog warning for each detection, so operators can search the log stream by entity source, field, and matched patterns. Returns true when patterns were detected.

Types

type AssertionResult added in v1.44.0

type AssertionResult struct {
	AssertionURN string `json:"assertion_urn,omitempty"` // URN identifying the assertion
	Type         string `json:"type"`                    // FRESHNESS, SCHEMA, DATA_QUALITY
}

AssertionResult represents a single assertion reference within a data contract.

type CacheConfig

type CacheConfig struct {
	TTL time.Duration
}

CacheConfig configures the cache.

type CachedProvider

type CachedProvider struct {
	// contains filtered or unexported fields
}

CachedProvider wraps a Provider with caching.

func NewCachedProvider

func NewCachedProvider(provider Provider, cfg CacheConfig) *CachedProvider

NewCachedProvider creates a caching wrapper around a provider.

func (*CachedProvider) Close

func (c *CachedProvider) Close() error

Close closes the underlying provider.

func (*CachedProvider) GetColumnContext

func (c *CachedProvider) GetColumnContext(ctx context.Context, column ColumnIdentifier) (*ColumnContext, error)

GetColumnContext retrieves column context with caching.

func (*CachedProvider) GetColumnsContext

func (c *CachedProvider) GetColumnsContext(ctx context.Context, table TableIdentifier) (map[string]*ColumnContext, error)

GetColumnsContext retrieves columns context with caching.

func (*CachedProvider) GetCuratedQueryCount added in v0.25.0

func (c *CachedProvider) GetCuratedQueryCount(ctx context.Context, urn string) (int, error)

GetCuratedQueryCount retrieves curated query count with caching.

func (*CachedProvider) GetGlossaryTerm

func (c *CachedProvider) GetGlossaryTerm(ctx context.Context, urn string) (*GlossaryTerm, error)

GetGlossaryTerm retrieves a glossary term with caching.

func (*CachedProvider) GetLineage

func (c *CachedProvider) GetLineage(ctx context.Context, table TableIdentifier, direction LineageDirection, maxDepth int) (*LineageInfo, error)

GetLineage retrieves lineage with caching.

func (*CachedProvider) GetTableContext

func (c *CachedProvider) GetTableContext(ctx context.Context, table TableIdentifier) (*TableContext, error)

GetTableContext retrieves table context with caching.

func (*CachedProvider) Invalidate

func (c *CachedProvider) Invalidate()

Invalidate clears the cache. It empties each map rather than replacing it, so the map values themselves are fixed at construction: cachedRead and cacheWrite take one as an argument, which the caller reads outside the lock, and a reassignment here would race with that read.

func (*CachedProvider) Name

func (c *CachedProvider) Name() string

Name returns the underlying provider name.

func (*CachedProvider) SearchTables

func (c *CachedProvider) SearchTables(ctx context.Context, filter SearchFilter) ([]TableSearchResult, error)

SearchTables searches without caching (queries vary too much).

func (*CachedProvider) Unwrap added in v1.91.0

func (c *CachedProvider) Unwrap() Provider

Unwrap returns the wrapped provider, so a capability probe can inspect the real provider behind the decorator instead of the decorator's unconditional pass-throughs (SearchDocuments below always exists on CachedProvider, which would otherwise make an optional-capability type-assertion falsely succeed).

type CatalogPicker added in v1.102.0

type CatalogPicker interface {
	// ListDomains returns every domain; the caller filters client-side.
	ListDomains(ctx context.Context) ([]EntityRef, error)
	// SearchGlossaryTerms name-searches glossary terms, bounded by limit. An
	// empty query lists: an implementation must substitute its backend's
	// match-everything query rather than forward the empty string, which a
	// relevance backend reads as "match nothing".
	SearchGlossaryTerms(ctx context.Context, query string, limit int) ([]EntityRef, error)
	// SearchDataProducts name-searches data products, bounded by limit, under
	// the same empty-query listing rule.
	SearchDataProducts(ctx context.Context, query string, limit int) ([]EntityRef, error)
}

CatalogPicker enumerates business-context entities (domains, glossary terms, data products) for topic-shaped argument autocompletion. These are not part of the core Provider interface (the platform catalog implements them, the noop provider does not), so completion probes for the capability with CatalogPickerFrom. Glossary/domain names are catalog metadata; callers apply their own persona gating.

func CatalogPickerFrom added in v1.102.0

func CatalogPickerFrom(p Provider) (CatalogPicker, bool)

CatalogPickerFrom reports the catalog-picker capability of p, returning the innermost provider that implements it: the unwrapped provider rather than p, because the caching decorator does not forward these picker methods; the picker lists (domains, glossary terms) are small and not per-table, so bypassing the cache is correct. ok is false when no provider in the chain can pick.

type ColumnContext

type ColumnContext struct {
	// Basic info
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`

	// Classification
	Tags          []string       `json:"tags,omitempty"`
	GlossaryTerms []GlossaryTerm `json:"glossary_terms,omitempty"`

	// Sensitivity
	IsPII       bool `json:"is_pii,omitempty"`
	IsSensitive bool `json:"is_sensitive,omitempty"`

	// Business metadata
	BusinessName string `json:"business_name,omitempty"`

	// InheritedFrom is set when metadata was inherited from upstream lineage.
	InheritedFrom *InheritedMetadata `json:"inherited_from,omitempty"`
}

ColumnContext provides semantic context for a column.

func (*ColumnContext) HasContent added in v0.24.0

func (c *ColumnContext) HasContent() bool

HasContent reports whether the column has any meaningful metadata worth including in enrichment responses. Columns with no description, tags, glossary terms, sensitivity flags, business name, or inherited metadata are considered empty and can be omitted to save tokens.

type ColumnIdentifier

type ColumnIdentifier struct {
	TableIdentifier
	Column string `json:"column"`
}

ColumnIdentifier uniquely identifies a column.

func (ColumnIdentifier) String

func (c ColumnIdentifier) String() string

String returns a dot-separated representation including the column.

type DataContractStatus added in v1.44.0

type DataContractStatus struct {
	Status           string            `json:"status"` // PASSING or FAILING
	AssertionResults []AssertionResult `json:"assertion_results,omitempty"`
}

DataContractStatus represents the pass/fail status of a data contract.

type Deprecation

type Deprecation struct {
	Deprecated bool       `json:"deprecated"`
	Note       string     `json:"note,omitempty"`
	Actor      string     `json:"actor,omitempty"`
	DecommDate *time.Time `json:"decommission_date,omitempty"`
}

Deprecation indicates if an entity is deprecated.

type Domain

type Domain struct {
	URN         string `json:"urn"`
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
}

Domain represents a data domain.

type EntityRef added in v1.100.2

type EntityRef struct {
	URN         string `json:"urn"`
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
}

EntityRef is a lightweight reference + display name. The catalog metadata pickers use it to resolve a human-readable name (e.g. "Revenue") to the catalog reference (e.g. "mcp:glossary_term:<id>") a write takes, so a user never types a raw reference.

func SearchGlossaryTermsCounted added in v1.120.0

func SearchGlossaryTermsCounted(ctx context.Context, picker CatalogPicker, query string, limit int) (refs []EntityRef, total int, err error)

SearchGlossaryTermsCounted searches picker's glossary and reports the total match count, falling back to the uncounted picker search (total TotalUnknown) when the picker cannot count. It takes the picker rather than the Provider because CatalogPickerFrom already resolves to the innermost implementation, which is the same provider that counts.

type GlossaryMatchCounter added in v1.120.0

type GlossaryMatchCounter interface {
	// SearchGlossaryTermsCounted runs the same search as
	// CatalogPicker.SearchGlossaryTerms, under the same empty-query listing rule,
	// and also reports the backend's total match count.
	SearchGlossaryTermsCounted(ctx context.Context, query string, limit int) (refs []EntityRef, total int, err error)
}

GlossaryMatchCounter is the glossary counterpart of TableMatchCounter, kept a separate capability rather than a second method on it so a backend that can count one search is never mistaken for one that can count both. It carries the same rationale, doubly so: the glossary page is bounded by the picker's own limit before the client's clamp applies.

type GlossaryTerm

type GlossaryTerm struct {
	URN         string `json:"urn"`
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	// ParentNode is the URN of the glossary node the term is filed under, empty
	// for a term at the glossary root.
	ParentNode string `json:"parent_node,omitempty"`
	// Owners are the term's stewards.
	Owners []Owner `json:"owners,omitempty"`
	// CustomProperties are the term's free-form key/value properties.
	CustomProperties map[string]string `json:"custom_properties,omitempty"`
}

GlossaryTerm represents a business glossary term. As an attribute of a dataset (TableContext.GlossaryTerms) only the first three fields are filled; a by-URN read (GetGlossaryTerm) also carries where the term sits in the glossary, who owns it, and its custom properties (#1590).

type Incident added in v1.44.0

type Incident struct {
	URN         string `json:"urn"`
	Type        string `json:"type"`
	Title       string `json:"title"`
	Description string `json:"description,omitempty"`
	State       string `json:"state"`
	Created     int64  `json:"created,omitempty"`
}

Incident represents an active data incident.

type InheritedMetadata added in v0.8.0

type InheritedMetadata struct {
	// SourceURN is the catalog reference of the upstream dataset.
	SourceURN string `json:"source_urn"`

	// SourceColumn is the column name in the upstream dataset.
	SourceColumn string `json:"source_column"`

	// Hops is the distance from the target dataset (1 = direct upstream).
	Hops int `json:"hops"`

	// MatchMethod indicates how the column was matched.
	// Values: "column_lineage", "name_exact", "name_transformed", "alias"
	MatchMethod string `json:"match_method"`
}

InheritedMetadata tracks the provenance of inherited column metadata.

type LineageDirection

type LineageDirection string

LineageDirection indicates the direction of lineage traversal.

const (
	LineageUpstream   LineageDirection = "upstream"
	LineageDownstream LineageDirection = "downstream"
)

Lineage direction constants.

type LineageEdge

type LineageEdge struct {
	URN            string `json:"urn"`
	Type           string `json:"type,omitempty"`
	TransformLogic string `json:"transform_logic,omitempty"`
}

LineageEdge represents an edge in the lineage graph.

type LineageEntity

type LineageEntity struct {
	URN      string        `json:"urn"`
	Type     string        `json:"type"`
	Name     string        `json:"name"`
	Platform string        `json:"platform,omitempty"`
	Depth    int           `json:"depth"`
	Parents  []LineageEdge `json:"parents,omitempty"`
	Children []LineageEdge `json:"children,omitempty"`
	Context  *TableContext `json:"context,omitempty"`
}

LineageEntity represents an entity in a lineage graph.

type LineageInfo

type LineageInfo struct {
	Direction LineageDirection `json:"direction"`
	Entities  []LineageEntity  `json:"entities"`
	MaxDepth  int              `json:"max_depth"`
}

LineageInfo contains lineage information for an entity.

type NoopProvider

type NoopProvider struct{}

NoopProvider is a no-op implementation for testing.

func NewNoopProvider

func NewNoopProvider() *NoopProvider

NewNoopProvider creates a new no-op provider.

func (*NoopProvider) Close

func (*NoopProvider) Close() error

Close does nothing.

func (*NoopProvider) GetColumnContext

func (*NoopProvider) GetColumnContext(_ context.Context, _ ColumnIdentifier) (*ColumnContext, error)

GetColumnContext returns empty context.

func (*NoopProvider) GetColumnsContext

func (*NoopProvider) GetColumnsContext(_ context.Context, _ TableIdentifier) (map[string]*ColumnContext, error)

GetColumnsContext returns empty map.

func (*NoopProvider) GetCuratedQueryCount added in v0.25.0

func (*NoopProvider) GetCuratedQueryCount(_ context.Context, _ string) (int, error)

GetCuratedQueryCount returns zero for the noop provider.

func (*NoopProvider) GetGlossaryTerm

func (*NoopProvider) GetGlossaryTerm(_ context.Context, _ string) (*GlossaryTerm, error)

GetGlossaryTerm returns an empty term.

func (*NoopProvider) GetLineage

func (*NoopProvider) GetLineage(_ context.Context, _ TableIdentifier, dir LineageDirection, maxDepth int) (*LineageInfo, error)

GetLineage returns empty lineage.

func (*NoopProvider) GetTableContext

func (*NoopProvider) GetTableContext(_ context.Context, _ TableIdentifier) (*TableContext, error)

GetTableContext returns empty context.

func (*NoopProvider) Name

func (*NoopProvider) Name() string

Name returns the provider name.

func (*NoopProvider) SearchTables

SearchTables returns empty results.

type Owner

type Owner struct {
	URN   string    `json:"urn"`
	Type  OwnerType `json:"type"`
	Name  string    `json:"name,omitempty"`
	Email string    `json:"email,omitempty"`
}

Owner represents a data owner.

type OwnerType

type OwnerType string

OwnerType indicates the type of owner.

const (
	OwnerTypeUser  OwnerType = "user"
	OwnerTypeGroup OwnerType = "group"
)

Owner type constants.

type Provider

type Provider interface {
	// Name returns the provider name.
	Name() string

	// GetTableContext retrieves semantic context for a table.
	GetTableContext(ctx context.Context, table TableIdentifier) (*TableContext, error)

	// GetColumnContext retrieves semantic context for a single column.
	GetColumnContext(ctx context.Context, column ColumnIdentifier) (*ColumnContext, error)

	// GetColumnsContext retrieves semantic context for all columns of a table.
	GetColumnsContext(ctx context.Context, table TableIdentifier) (map[string]*ColumnContext, error)

	// GetLineage retrieves lineage information for a table.
	GetLineage(ctx context.Context, table TableIdentifier, direction LineageDirection, maxDepth int) (*LineageInfo, error)

	// GetGlossaryTerm retrieves a glossary term by URN.
	GetGlossaryTerm(ctx context.Context, urn string) (*GlossaryTerm, error)

	// SearchTables searches for tables matching the filter.
	SearchTables(ctx context.Context, filter SearchFilter) ([]TableSearchResult, error)

	// GetCuratedQueryCount returns the number of curated/saved queries for a dataset.
	GetCuratedQueryCount(ctx context.Context, urn string) (int, error)

	// Close releases resources.
	Close() error
}

Provider retrieves semantic metadata from the catalog. The platform catalog implements it (internal/platform/catalogsemantic).

type SanitizeConfig added in v0.2.0

type SanitizeConfig struct {
	// MaxLength is the maximum length for strings (default: 2000).
	MaxLength int

	// StripInjectionPatterns removes detected injection patterns instead of flagging.
	StripInjectionPatterns bool

	// LogInjectionAttempts enables logging of detected injection attempts.
	LogInjectionAttempts bool
}

SanitizeConfig configures sanitization behavior.

func DefaultSanitizeConfig added in v0.2.0

func DefaultSanitizeConfig() SanitizeConfig

DefaultSanitizeConfig returns a safe default configuration.

type Sanitizer added in v0.2.0

type Sanitizer struct {
	// contains filtered or unexported fields
}

Sanitizer sanitizes metadata strings to prevent prompt injection and other attacks.

func NewSanitizer added in v0.2.0

func NewSanitizer(cfg SanitizeConfig) *Sanitizer

NewSanitizer creates a new sanitizer with the given configuration.

func (*Sanitizer) DetectInjection added in v0.2.0

func (*Sanitizer) DetectInjection(input string) (detected bool, patterns []string)

DetectInjection checks if the input contains potential prompt injection patterns. Returns true if injection is detected along with matched patterns.

func (*Sanitizer) SanitizeColumnContext added in v0.2.0

func (s *Sanitizer) SanitizeColumnContext(cc *ColumnContext) *ColumnContext

SanitizeColumnContext sanitizes all string fields in a ColumnContext.

func (*Sanitizer) SanitizeDescription added in v0.2.0

func (s *Sanitizer) SanitizeDescription(desc string) string

SanitizeDescription sanitizes a description field.

func (*Sanitizer) SanitizeGlossaryTerm added in v1.129.0

func (s *Sanitizer) SanitizeGlossaryTerm(term *GlossaryTerm) *GlossaryTerm

SanitizeGlossaryTerm sanitizes every string field of a glossary term. The URNs (the term's own and its parent node's) are system identifiers and pass through unchanged.

func (*Sanitizer) SanitizeString added in v0.2.0

func (s *Sanitizer) SanitizeString(input string) string

SanitizeString sanitizes a string by removing control characters, truncating to max length, and optionally stripping injection patterns.

func (*Sanitizer) SanitizeTableContext added in v0.2.0

func (s *Sanitizer) SanitizeTableContext(tc *TableContext) *TableContext

SanitizeTableContext sanitizes all string fields in a TableContext.

func (*Sanitizer) SanitizeTag added in v0.2.0

func (*Sanitizer) SanitizeTag(tag string) string

SanitizeTag validates and sanitizes a tag name. Returns empty string if the tag is invalid.

func (*Sanitizer) SanitizeTags added in v0.2.0

func (s *Sanitizer) SanitizeTags(tags []string) []string

SanitizeTags sanitizes a slice of tags, removing invalid ones.

type SearchFilter

type SearchFilter struct {
	Query    string   `json:"query"`
	Platform string   `json:"platform,omitempty"`
	Tags     []string `json:"tags,omitempty"`
	Domain   string   `json:"domain,omitempty"`
	Owner    string   `json:"owner,omitempty"`
	Limit    int      `json:"limit,omitempty"`
	Offset   int      `json:"offset,omitempty"`

	// Mode selects the search strategy: "keyword" (default) or "semantic".
	Mode string `json:"mode,omitempty"`
}

SearchFilter defines criteria for searching tables.

type StructuredProperty added in v1.44.0

type StructuredProperty struct {
	QualifiedName string `json:"qualified_name"`
	DisplayName   string `json:"display_name,omitempty"`
	Values        []any  `json:"values"`
}

StructuredProperty represents a typed custom property.

type TableContext

type TableContext struct {
	// Basic info
	URN         string `json:"urn,omitempty"`
	Description string `json:"description,omitempty"`

	// Ownership
	Owners []Owner `json:"owners,omitempty"`

	// Classification
	Tags          []string       `json:"tags,omitempty"`
	GlossaryTerms []GlossaryTerm `json:"glossary_terms,omitempty"`
	Domain        *Domain        `json:"domain,omitempty"`

	// TagRefs carries the same tags as Tags but as URN + display-name pairs, so a
	// governance editor can remove or dedupe a tag by its URN (Tags holds only the
	// display name, which is not the identifier a write needs). Populated on the
	// entity-detail read; enrichment reads Tags, not this field (#785).
	TagRefs []EntityRef `json:"tag_refs,omitempty"`

	// Status
	Deprecation *Deprecation `json:"deprecation,omitempty"`

	// Quality
	QualityScore *float64 `json:"quality_score,omitempty"`

	// Metadata
	CustomProperties map[string]string `json:"custom_properties,omitempty"`
	LastModified     *time.Time        `json:"last_modified,omitempty"`

	// Structured properties
	StructuredProperties []StructuredProperty `json:"structured_properties,omitempty"`

	// Incidents
	ActiveIncidents int        `json:"active_incidents,omitempty"`
	Incidents       []Incident `json:"incidents,omitempty"`

	// Data contracts
	DataContract *DataContractStatus `json:"data_contract,omitempty"`
}

TableContext provides semantic context for a table.

type TableIdentifier

type TableIdentifier struct {
	Catalog string `json:"catalog,omitempty"`
	Schema  string `json:"schema"`
	Table   string `json:"table"`
}

TableIdentifier uniquely identifies a table.

func (TableIdentifier) String

func (t TableIdentifier) String() string

String returns a dot-separated representation.

type TableMatchCounter added in v1.120.0

type TableMatchCounter interface {
	// SearchTablesCounted runs the same search as Provider.SearchTables and also
	// reports the backend's total match count. The total may exceed len(results)
	// both because the caller's limit bounded the page and because the backend
	// bounded it further.
	SearchTablesCounted(ctx context.Context, filter SearchFilter) (results []TableSearchResult, total int, err error)
}

TableMatchCounter is the optional total-matches capability for the dataset search: it reports how many matches the backend found, not merely how many rows it put in the page it returned. Only a backend that knows whether its page is the whole answer implements it; the platform catalog does, because it returns exactly the page it is asked for.

It exists because a page-bounded backend can silently reduce the requested limit, so a caller cannot discover "more matches exist" by asking for one row more than it needs: the extra row never arrives and the short page reads as a complete set (#1238). A total that rides back on the same response costs no extra round trip.

type TableSearchResult

type TableSearchResult struct {
	URN          string   `json:"urn"`
	Name         string   `json:"name"`
	Platform     string   `json:"platform,omitempty"`
	Description  string   `json:"description,omitempty"`
	Tags         []string `json:"tags,omitempty"`
	Domain       string   `json:"domain,omitempty"`
	MatchedField string   `json:"matched_field,omitempty"`
}

TableSearchResult represents a search result.

func SearchTablesCounted added in v1.120.0

func SearchTablesCounted(ctx context.Context, p Provider, filter SearchFilter) (results []TableSearchResult, total int, err error)

SearchTablesCounted searches p and reports the backend's total match count alongside the page. When no provider in p's chain can count, total is TotalUnknown and the caller has learned nothing about the matches beyond the rows it holds — in particular it must not read a full page as proof that more exist, nor a short one as proof that none do.

The count is read from the innermost counting provider, bypassing the cache decorator, which is what CachedProvider.SearchTables does with the search itself: searches vary too much to cache, so the decorator only forwards.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL