Documentation
¶
Overview ¶
Package semantic provides semantic layer abstractions.
Package semantic provides abstractions for semantic metadata providers.
Index ¶
- Constants
- Variables
- func DetectAndLogInjection(sanitizer *Sanitizer, source, field, input string) bool
- type AssertionResult
- type CacheConfig
- type CachedProvider
- func (c *CachedProvider) BrowseDocuments(ctx context.Context, offset, limit int) ([]DocumentResult, int, error)
- func (c *CachedProvider) Close() error
- func (c *CachedProvider) GetColumnContext(ctx context.Context, column ColumnIdentifier) (*ColumnContext, error)
- func (c *CachedProvider) GetColumnsContext(ctx context.Context, table TableIdentifier) (map[string]*ColumnContext, error)
- func (c *CachedProvider) GetCuratedQueryCount(ctx context.Context, urn string) (int, error)
- func (c *CachedProvider) GetDocument(ctx context.Context, urn string) (*DocumentResult, error)
- func (c *CachedProvider) GetGlossaryTerm(ctx context.Context, urn string) (*GlossaryTerm, error)
- func (c *CachedProvider) GetLineage(ctx context.Context, table TableIdentifier, direction LineageDirection, ...) (*LineageInfo, error)
- func (c *CachedProvider) GetRelatedDocuments(ctx context.Context, urn string) ([]DocumentResult, error)
- func (c *CachedProvider) GetTableContext(ctx context.Context, table TableIdentifier) (*TableContext, error)
- func (c *CachedProvider) Invalidate()
- func (c *CachedProvider) Name() string
- func (c *CachedProvider) SearchDocuments(ctx context.Context, query string, limit int) ([]DocumentResult, error)
- func (c *CachedProvider) SearchTables(ctx context.Context, filter SearchFilter) ([]TableSearchResult, error)
- func (c *CachedProvider) Unwrap() Provider
- type CatalogPicker
- type ColumnContext
- type ColumnIdentifier
- type DataContractStatus
- type DataProduct
- type DataProductReader
- type Dataset
- type DatasetReader
- type DatasetSchema
- type Deprecation
- type DocumentResult
- type DocumentSearcher
- type Domain
- type EntityRef
- type FieldFilter
- type ForeignKey
- type GlossaryChildren
- type GlossaryMatchCounter
- type GlossaryNode
- type GlossaryTerm
- type GovernanceReader
- type Incident
- type InheritedMetadata
- type LineageDirection
- type LineageEdge
- type LineageEntity
- type LineageInfo
- type NoopProvider
- func (*NoopProvider) Close() error
- func (*NoopProvider) GetColumnContext(_ context.Context, _ ColumnIdentifier) (*ColumnContext, error)
- func (*NoopProvider) GetColumnsContext(_ context.Context, _ TableIdentifier) (map[string]*ColumnContext, error)
- func (*NoopProvider) GetCuratedQueryCount(_ context.Context, _ string) (int, error)
- func (*NoopProvider) GetGlossaryTerm(_ context.Context, _ string) (*GlossaryTerm, error)
- func (*NoopProvider) GetLineage(_ context.Context, _ TableIdentifier, dir LineageDirection, maxDepth int) (*LineageInfo, error)
- func (*NoopProvider) GetTableContext(_ context.Context, _ TableIdentifier) (*TableContext, error)
- func (*NoopProvider) Name() string
- func (*NoopProvider) SearchTables(_ context.Context, _ SearchFilter) ([]TableSearchResult, error)
- type Owner
- type OwnerType
- type Provider
- type SanitizeConfig
- type Sanitizer
- func (*Sanitizer) DetectInjection(input string) (detected bool, patterns []string)
- func (s *Sanitizer) SanitizeColumnContext(cc *ColumnContext) *ColumnContext
- func (s *Sanitizer) SanitizeDataProduct(p *DataProduct) *DataProduct
- func (s *Sanitizer) SanitizeDataset(d *Dataset) *Dataset
- func (s *Sanitizer) SanitizeDescription(desc string) string
- func (s *Sanitizer) SanitizeGlossaryTerm(term *GlossaryTerm) *GlossaryTerm
- func (s *Sanitizer) SanitizeString(input string) string
- func (s *Sanitizer) SanitizeTableContext(tc *TableContext) *TableContext
- func (*Sanitizer) SanitizeTag(tag string) string
- func (s *Sanitizer) SanitizeTags(tags []string) []string
- type SavedQuery
- type SchemaField
- type SearchFilter
- type StructuredProperty
- type TableContext
- type TableIdentifier
- type TableMatchCounter
- type TableSearchResult
- type URNResolver
Constants ¶
const FilterFieldGlossaryTerms = "glossaryTerms"
FilterFieldGlossaryTerms is the catalog search-filter field that matches the datasets carrying a glossary term. DataHub folds a column-level assignment into the dataset's glossaryTerms index, so this field matches a dataset whose TABLE or whose COLUMN carries the term; fieldGlossaryTerms narrows to column-level assignments only. Exported so the governance search source and the portal's catalog REST surface name the field from one authority.
const MaxStringLength = 2000
MaxStringLength is the maximum length for sanitized strings.
const TotalUnknown = -1
TotalUnknown is the match count reported when no provider in the chain can count. It is deliberately negative rather than zero so it can never be read as "no matches" or compared against a page length as if it were a total: a caller must branch on it explicitly.
Variables ¶
var ErrDocumentNotFound = errors.New("document not found")
ErrDocumentNotFound reports that a document URN did not resolve to a document. GetDocument returns it (wrapped) so a caller can distinguish a stale reference from a transport failure. It is defined here, on the capability interface, rather than per-implementation so every DocumentSearcher agrees on the sentinel.
Functions ¶
func DetectAndLogInjection ¶ added in v1.96.0
DetectAndLogInjection checks input for prompt-injection patterns and emits a structured slog warning for each detection, so operators can search the log stream by entity source, field, and matched patterns. Returns true when patterns were detected.
Types ¶
type AssertionResult ¶ added in v1.44.0
type AssertionResult struct {
AssertionURN string `json:"assertion_urn,omitempty"` // URN identifying the assertion
Type string `json:"type"` // FRESHNESS, SCHEMA, DATA_QUALITY
}
AssertionResult represents a single assertion reference within a data contract.
type CachedProvider ¶
type CachedProvider struct {
// contains filtered or unexported fields
}
CachedProvider wraps a Provider with caching.
func NewCachedProvider ¶
func NewCachedProvider(provider Provider, cfg CacheConfig) *CachedProvider
NewCachedProvider creates a caching wrapper around a provider.
func (*CachedProvider) BrowseDocuments ¶ added in v1.93.0
func (c *CachedProvider) BrowseDocuments(ctx context.Context, offset, limit int) ([]DocumentResult, int, error)
BrowseDocuments forwards the document enumeration (#695) to the wrapped provider, preserving the DocumentSearcher capability through the cache decorator. It is not cached: a browse pages a mutating corpus and reports a live total, so a stale cached page or count would be worse than a fresh round trip. A wrapped provider without the capability yields an empty page and a zero total.
func (*CachedProvider) Close ¶
func (c *CachedProvider) Close() error
Close closes the underlying provider.
func (*CachedProvider) GetColumnContext ¶
func (c *CachedProvider) GetColumnContext(ctx context.Context, column ColumnIdentifier) (*ColumnContext, error)
GetColumnContext retrieves column context with caching.
func (*CachedProvider) GetColumnsContext ¶
func (c *CachedProvider) GetColumnsContext(ctx context.Context, table TableIdentifier) (map[string]*ColumnContext, error)
GetColumnsContext retrieves columns context with caching.
func (*CachedProvider) GetCuratedQueryCount ¶ added in v0.25.0
GetCuratedQueryCount retrieves curated query count with caching.
func (*CachedProvider) GetDocument ¶ added in v1.92.0
func (c *CachedProvider) GetDocument(ctx context.Context, urn string) (*DocumentResult, error)
GetDocument forwards the single-document read (#694) to the wrapped provider, preserving the DocumentSearcher capability through the cache decorator. It is not cached: a fetch is an explicit, low-frequency dereference whose whole point is the current full body, so serving a stale cached copy (e.g. just after an edit) would defeat it; the hot search path is what the snippet caches serve.
func (*CachedProvider) GetGlossaryTerm ¶
func (c *CachedProvider) GetGlossaryTerm(ctx context.Context, urn string) (*GlossaryTerm, error)
GetGlossaryTerm retrieves a glossary term with caching.
func (*CachedProvider) GetLineage ¶
func (c *CachedProvider) GetLineage(ctx context.Context, table TableIdentifier, direction LineageDirection, maxDepth int) (*LineageInfo, error)
GetLineage retrieves lineage with caching.
func (*CachedProvider) GetRelatedDocuments ¶ added in v1.91.0
func (c *CachedProvider) GetRelatedDocuments(ctx context.Context, urn string) ([]DocumentResult, error)
GetRelatedDocuments forwards the entity-keyed document lookup (#692) to the wrapped provider, preserving the DocumentSearcher capability through the cache decorator. It is keyed on a single entity URN (like GetGlossaryTerm/GetTableContext), so it is cached by URN: lineage expansion produces overlapping URN sets across successive searches, and serving repeats from cache spares the DataHub round trip on the hot search path.
func (*CachedProvider) GetTableContext ¶
func (c *CachedProvider) GetTableContext(ctx context.Context, table TableIdentifier) (*TableContext, error)
GetTableContext retrieves table context with caching.
func (*CachedProvider) Invalidate ¶
func (c *CachedProvider) Invalidate()
Invalidate clears the cache.
func (*CachedProvider) Name ¶
func (c *CachedProvider) Name() string
Name returns the underlying provider name.
func (*CachedProvider) SearchDocuments ¶ added in v1.91.0
func (c *CachedProvider) SearchDocuments(ctx context.Context, query string, limit int) ([]DocumentResult, error)
SearchDocuments forwards the optional document-search capability (#692) to the wrapped provider, preserving it through the cache decorator. A wrapped provider that does not implement it (e.g. a noop catalog) yields no documents, so the capability is absent rather than always-empty. Not cached: queries vary too much.
func (*CachedProvider) SearchTables ¶
func (c *CachedProvider) SearchTables(ctx context.Context, filter SearchFilter) ([]TableSearchResult, error)
SearchTables searches without caching (queries vary too much).
func (*CachedProvider) Unwrap ¶ added in v1.91.0
func (c *CachedProvider) Unwrap() Provider
Unwrap returns the wrapped provider, so a capability probe can inspect the real provider behind the decorator instead of the decorator's unconditional pass-throughs (SearchDocuments below always exists on CachedProvider, which would otherwise make an optional-capability type-assertion falsely succeed).
type CatalogPicker ¶ added in v1.102.0
type CatalogPicker interface {
// ListDomains returns every DataHub domain; the caller filters client-side.
ListDomains(ctx context.Context) ([]EntityRef, error)
// SearchGlossaryTerms name-searches glossary terms, bounded by limit. An
// empty query lists: an implementation must substitute its backend's
// match-everything query rather than forward the empty string, which a
// relevance backend reads as "match nothing".
SearchGlossaryTerms(ctx context.Context, query string, limit int) ([]EntityRef, error)
}
CatalogPicker enumerates business-context entities (domains, glossary terms) for topic-shaped argument autocompletion. These are not part of the core Provider interface — only a real semantic backend (the DataHub adapter) implements them — so completion probes for the capability with CatalogPickerFrom. Glossary/domain names are catalog metadata; callers apply their own persona gating.
func CatalogPickerFrom ¶ added in v1.102.0
func CatalogPickerFrom(p Provider) (CatalogPicker, bool)
CatalogPickerFrom reports the catalog-picker capability of p, returning the innermost provider that implements it. Unlike DocumentSearcherFrom it returns the unwrapped provider rather than p, because the caching decorator does not forward these picker methods; the picker lists (domains, glossary terms) are small and not per-table, so bypassing the cache is correct. ok is false when no provider in the chain can pick.
type ColumnContext ¶
type ColumnContext struct {
// Basic info
Name string `json:"name"`
Description string `json:"description,omitempty"`
// Classification
Tags []string `json:"tags,omitempty"`
GlossaryTerms []GlossaryTerm `json:"glossary_terms,omitempty"`
// Sensitivity
IsPII bool `json:"is_pii,omitempty"`
IsSensitive bool `json:"is_sensitive,omitempty"`
// Business metadata
BusinessName string `json:"business_name,omitempty"`
// InheritedFrom is set when metadata was inherited from upstream lineage.
InheritedFrom *InheritedMetadata `json:"inherited_from,omitempty"`
}
ColumnContext provides semantic context for a column.
func (*ColumnContext) HasContent ¶ added in v0.24.0
func (c *ColumnContext) HasContent() bool
HasContent reports whether the column has any meaningful metadata worth including in enrichment responses. Columns with no description, tags, glossary terms, sensitivity flags, business name, or inherited metadata are considered empty and can be omitted to save tokens.
type ColumnIdentifier ¶
type ColumnIdentifier struct {
TableIdentifier
Column string `json:"column"`
}
ColumnIdentifier uniquely identifies a column.
func (ColumnIdentifier) String ¶
func (c ColumnIdentifier) String() string
String returns a dot-separated representation including the column.
type DataContractStatus ¶ added in v1.44.0
type DataContractStatus struct {
Status string `json:"status"` // PASSING or FAILING
AssertionResults []AssertionResult `json:"assertion_results,omitempty"`
}
DataContractStatus represents the pass/fail status of a data contract from DataHub 1.4.x.
type DataProduct ¶ added in v1.129.0
type DataProduct struct {
URN string `json:"urn"`
Name string `json:"name"`
Description string `json:"description,omitempty"`
// Domain is the domain the product is filed under, nil when it has none.
Domain *Domain `json:"domain,omitempty"`
// Owners are the product's owners.
Owners []Owner `json:"owners,omitempty"`
// Assets are the datasets that make up the product.
Assets []EntityRef `json:"assets,omitempty"`
// CustomProperties are the product's free-form key/value properties.
CustomProperties map[string]string `json:"custom_properties,omitempty"`
}
DataProduct is a catalog data product: a named, owned grouping of datasets under a domain (#1590). It is what a fetch of a urn:li:dataProduct: reference returns.
type DataProductReader ¶ added in v1.129.0
type DataProductReader interface {
// GetDataProduct reads one data product by URN. A URN the catalog has no
// product for is an error, as the other by-URN reads report one.
GetDataProduct(ctx context.Context, urn string) (*DataProduct, error)
}
DataProductReader is the optional by-URN read of a catalog data product (#1590), the read behind fetch's urn:li:dataProduct: arm. Only a real catalog backend implements it.
func DataProductReaderFrom ¶ added in v1.129.0
func DataProductReaderFrom(p Provider) (DataProductReader, bool)
DataProductReaderFrom reports the data-product read capability of p, returning the innermost provider that implements it, under the same rule as DatasetReaderFrom.
type Dataset ¶ added in v1.129.0
type Dataset struct {
TableContext
// Name is the catalog's display name for the dataset.
Name string `json:"name,omitempty"`
// Type is the catalog entity type ("DATASET").
Type string `json:"type,omitempty"`
// Platform is the data platform the dataset belongs to (the URN's
// dataPlatform segment, e.g. "trino").
Platform string `json:"platform,omitempty"`
// SubTypes are the catalog's sub-type classifications ("table", "view").
SubTypes []string `json:"sub_types,omitempty"`
// Created is when the catalog first recorded the dataset.
Created *time.Time `json:"created,omitempty"`
// Schema is the declared schema, nil when the catalog holds none.
Schema *DatasetSchema `json:"schema,omitempty"`
// Queries are the saved queries the catalog holds against the dataset, and
// TotalQueries is the catalog's own count of them (the list may be a page).
Queries []SavedQuery `json:"queries,omitempty"`
TotalQueries int `json:"total_queries,omitempty"`
// RelatedDocuments are the context documents linked to the dataset.
RelatedDocuments []DocumentResult `json:"related_documents,omitempty"`
// this read ("schema", "queries", "related_documents"), so an absent part is
// distinguishable from an empty one. The entity itself always resolved.
Unavailable []string `json:"unavailable,omitempty"`
}
Dataset is the full record of one catalog dataset: the business context a TableContext carries, the identity fields the catalog holds beside it, the declared schema, and the saved queries and context documents attached to it (#1590). It is what a fetch of a urn:li:dataset: reference returns, folding the former datahub_get_entity, datahub_get_schema, and datahub_get_queries reads into one answer to "tell me about this dataset". TableContext is embedded so its fields serialize at the top level, where enrichment readers already look for them.
type DatasetReader ¶ added in v1.129.0
type DatasetReader interface {
// GetDataset reads the dataset the table identifier names. A table the
// catalog has no entry for is an error, as GetTableContext reports one.
GetDataset(ctx context.Context, table TableIdentifier) (*Dataset, error)
}
DatasetReader is the optional full-record read of one catalog dataset (#1590): business context, identity, declared schema, saved queries, and linked context documents in one call. It is the read behind fetch's urn:li:dataset: arm, where the former datahub_get_entity, datahub_get_schema, and datahub_get_queries tools were folded. Only a real catalog backend (the DataHub adapter) implements it; the noop provider does not, so a noop catalog leaves fetch on the enrichment-shaped TableContext read alone.
func DatasetReaderFrom ¶ added in v1.129.0
func DatasetReaderFrom(p Provider) (DatasetReader, bool)
DatasetReaderFrom reports the full-dataset read capability of p, returning the innermost provider that implements it. It resolves like CatalogPickerFrom because the caching decorator does not forward it: a fetch is one read of one record, not the per-query enrichment read the cache exists for.
type DatasetSchema ¶ added in v1.129.0
type DatasetSchema struct {
// Version is the catalog's schema version.
Version int64 `json:"version,omitempty"`
// Fields are the declared fields in declaration order.
Fields []SchemaField `json:"fields"`
// PrimaryKeys lists the primary key field paths.
PrimaryKeys []string `json:"primary_keys,omitempty"`
// ForeignKeys lists the declared foreign key relationships.
ForeignKeys []ForeignKey `json:"foreign_keys,omitempty"`
}
DatasetSchema is a dataset's declared schema as the catalog holds it.
type Deprecation ¶
type Deprecation struct {
Deprecated bool `json:"deprecated"`
Note string `json:"note,omitempty"`
Actor string `json:"actor,omitempty"`
DecommDate *time.Time `json:"decommission_date,omitempty"`
}
Deprecation indicates if an entity is deprecated.
type DocumentResult ¶ added in v1.91.0
type DocumentResult struct {
URN string `json:"urn"`
Title string `json:"title"`
SubType string `json:"sub_type,omitempty"`
Snippet string `json:"snippet,omitempty"`
// Body is the full, untruncated document content. It is populated only by a
// single-document read (GetDocument), where the whole point is to return the
// complete content a search snippet elides; the relevance-search paths
// (SearchDocuments, GetRelatedDocuments) leave it empty and populate the
// bounded Snippet instead, so a multi-result search does not carry N full
// bodies.
Body string `json:"body,omitempty"`
// Status is the publication state (PUBLISHED/UNPUBLISHED). The upstream search
// applies no status filter, so a consumer carries this to exclude drafts.
Status string `json:"status,omitempty"`
// ShowInGlobalContext reports whether the document is meant to appear in global
// search. The upstream search returns documents regardless of this flag, so a
// search consumer must filter on it to honor a steward's choice to hide a document.
ShowInGlobalContext bool `json:"show_in_global_context"`
RelatedAssetURNs []string `json:"related_asset_urns,omitempty"`
}
DocumentResult is one DataHub context document returned by a relevance search (#692). Context documents are the non-dataset knowledge home that predates knowledge pages; surfacing them in search makes them discoverable (and migratable). The URN (urn:li:document:<id>) drills in, Snippet shows relevance, and ShowInGlobalContext distinguishes globally-visible documents from hidden ones.
type DocumentSearcher ¶ added in v1.91.0
type DocumentSearcher interface {
// SearchDocuments ranks context documents by relevance to query; a query of "*"
// lists all (an empty query does not list). Results carry ShowInGlobalContext and
// Status so the caller can filter to globally-visible, published documents. limit
// caps results (0 means the provider default).
SearchDocuments(ctx context.Context, query string, limit int) ([]DocumentResult, error)
// GetRelatedDocuments returns the context documents linked to an entity URN (the
// reverse of a document's related assets), for entity-keyed discovery. Results
// carry the same fields as SearchDocuments so the caller applies the same filter.
GetRelatedDocuments(ctx context.Context, urn string) ([]DocumentResult, error)
// GetDocument reads one context document by its URN, returning the full
// untruncated body (in DocumentResult.Body) so an agent can dereference a
// urn:li:document:<id> reference search emitted to the complete content. A URN
// that resolves to no document returns ErrDocumentNotFound, which the fetch
// surface maps to a structured not-found rather than an error.
GetDocument(ctx context.Context, urn string) (*DocumentResult, error)
// BrowseDocuments enumerates context documents for the browse surface (#695):
// the offset/limit page of the complete document set plus the total document
// count, so an agent can page the whole corpus to audit, dedup, or migrate it.
// Unlike SearchDocuments this applies NO relevance threshold and NO
// visibility/status filter: every document is enumerable (drafts and hidden
// documents included), so the returned page and total describe the same complete
// set. Results carry the same fields as SearchDocuments (a bounded Snippet, not
// the full Body) since a listing shows what each document is, not its contents.
BrowseDocuments(ctx context.Context, offset, limit int) (docs []DocumentResult, total int, err error)
}
DocumentSearcher is the optional document-search capability (#692): relevance search over DataHub context documents, the non-dataset knowledge home that predates knowledge pages. Only a real catalog provider implements it (the DataHub adapter); the noop provider does not, so a noop catalog adds no documents search source. The cache decorator forwards it. A consumer type-asserts a Provider to this to decide whether to register a documents search source.
func DocumentSearcherFrom ¶ added in v1.91.0
func DocumentSearcherFrom(p Provider) (DocumentSearcher, bool)
DocumentSearcherFrom reports the document-search capability of p, unwrapping any decorator chain (e.g. CachedProvider) so the answer reflects the real underlying provider rather than a decorator's unconditional pass-through. ok is false when no provider in the chain can search documents (so no documents source is registered); when ok, the returned searcher is p itself, so searches still flow through the decorator (and its cache/forwarding) rather than bypassing it.
type Domain ¶
type Domain struct {
URN string `json:"urn"`
Name string `json:"name"`
Description string `json:"description,omitempty"`
}
Domain represents a data domain.
type EntityRef ¶ added in v1.100.2
type EntityRef struct {
URN string `json:"urn"`
Name string `json:"name"`
Description string `json:"description,omitempty"`
}
EntityRef is a lightweight URN + display name reference. The catalog metadata pickers use it to resolve a human-readable name (e.g. "Revenue") to the DataHub URN (e.g. "urn:li:glossaryTerm:Revenue") the write endpoints require, so a user never types a raw URN.
func SearchGlossaryTermsCounted ¶ added in v1.120.0
func SearchGlossaryTermsCounted(ctx context.Context, picker CatalogPicker, query string, limit int) (refs []EntityRef, total int, err error)
SearchGlossaryTermsCounted searches picker's glossary and reports the total match count, falling back to the uncounted picker search (total TotalUnknown) when the picker cannot count. It takes the picker rather than the Provider because CatalogPickerFrom already resolves to the innermost implementation, which is the same provider that counts.
type FieldFilter ¶ added in v1.50.0
type FieldFilter struct {
// Field is the filter field (e.g., "fieldPaths", "fieldTags", "platform", "owners").
Field string `json:"field"`
// Values to match against.
Values []string `json:"values"`
// Condition is the match operator: CONTAIN, EQUAL (default), IN, EXISTS.
Condition string `json:"condition,omitempty"`
// Negated inverts the filter (exclude matching entities).
Negated bool `json:"negated,omitempty"`
}
FieldFilter is a single search filter criterion.
type ForeignKey ¶ added in v1.129.0
type ForeignKey struct {
// Name is the constraint name, when the catalog holds one.
Name string `json:"name,omitempty"`
// SourceFields are the field paths on this dataset.
SourceFields []string `json:"source_fields"`
// ForeignDataset is the URN of the referenced dataset, and ForeignFields the
// field paths on it.
ForeignDataset string `json:"foreign_dataset"`
ForeignFields []string `json:"foreign_fields"`
}
ForeignKey is a declared foreign key relationship between two datasets.
type GlossaryChildren ¶ added in v1.119.0
type GlossaryChildren struct {
Nodes []GlossaryNode `json:"nodes"`
Terms []GlossaryTerm `json:"terms"`
Start int `json:"start"`
Count int `json:"count"`
Total int `json:"total"`
}
GlossaryChildren is a page of what sits directly under a glossary node. The backend pages nodes and terms as one mixed collection, so Start, Count, and Total describe the combined page rather than either slice on its own: a caller pages by Start/Count against Total and reads whichever slices the page happened to contain.
type GlossaryMatchCounter ¶ added in v1.120.0
type GlossaryMatchCounter interface {
// SearchGlossaryTermsCounted runs the same search as
// CatalogPicker.SearchGlossaryTerms, under the same empty-query listing rule,
// and also reports the backend's total match count.
SearchGlossaryTermsCounted(ctx context.Context, query string, limit int) (refs []EntityRef, total int, err error)
}
GlossaryMatchCounter is the glossary counterpart of TableMatchCounter, kept a separate capability rather than a second method on it so a backend that can count one search is never mistaken for one that can count both. It carries the same rationale, doubly so: the glossary page is bounded by the picker's own limit before the client's clamp applies.
type GlossaryNode ¶ added in v1.119.0
type GlossaryNode struct {
URN string `json:"urn"`
Name string `json:"name"`
Description string `json:"description,omitempty"`
// ParentNode is the URN of the containing node, empty at the root.
ParentNode string `json:"parent_node,omitempty"`
TermsCount int `json:"terms_count"`
NodesCount int `json:"nodes_count"`
}
GlossaryNode is a directory in the business glossary: it holds glossary terms and other glossary nodes. TermsCount and NodesCount are the backend's own tally of the direct children, so a browser can render an expandable branch without first fetching it (#1155).
type GlossaryTerm ¶
type GlossaryTerm struct {
URN string `json:"urn"`
Name string `json:"name"`
Description string `json:"description,omitempty"`
// ParentNode is the URN of the glossary node the term is filed under, empty
// for a term at the glossary root.
ParentNode string `json:"parent_node,omitempty"`
// Owners are the term's stewards.
Owners []Owner `json:"owners,omitempty"`
// CustomProperties are the term's free-form key/value properties.
CustomProperties map[string]string `json:"custom_properties,omitempty"`
}
GlossaryTerm represents a business glossary term. As an attribute of a dataset (TableContext.GlossaryTerms) only the first three fields are filled; a by-URN read (GetGlossaryTerm) also carries where the term sits in the glossary, who owns it, and its custom properties (#1590).
type GovernanceReader ¶ added in v1.119.0
type GovernanceReader interface {
CatalogPicker
// SearchTags name-searches tags, bounded by limit. An empty query lists them,
// under the same substitution rule SearchGlossaryTerms carries.
SearchTags(ctx context.Context, query string, limit int) ([]EntityRef, error)
// GetGlossaryTerm reads one term by URN. It is the only by-URN read any
// governance vocabulary has, which is why a tag or a domain is resolved by
// listing its vocabulary and matching instead.
GetGlossaryTerm(ctx context.Context, urn string) (*GlossaryTerm, error)
// SearchTables ranks datasets; filtered by tag, domain, or glossary term it
// lists the datasets carrying that governance entity.
SearchTables(ctx context.Context, filter SearchFilter) ([]TableSearchResult, error)
}
GovernanceReader is the optional governance-vocabulary capability (#1160): the reads that make DataHub's glossary terms, tags, and domains discoverable and readable as entities in their own right rather than as attributes of a dataset. Only a real catalog backend (the DataHub adapter) implements it, so a noop catalog registers no governance source.
The three kinds are deliberately not read through one uniform method, because upstream does not offer one: a glossary term has a by-URN read and a name search, a tag has a name search only, and a domain has neither and is enumerated whole. SearchTables completes the set — with a tag, domain, or glossary-term filter it lists the datasets that carry a governance entity, which is what makes the entity useful to read.
func GovernanceReaderFrom ¶ added in v1.119.0
func GovernanceReaderFrom(p Provider) (GovernanceReader, bool)
GovernanceReaderFrom reports the governance-read capability of p, returning the innermost provider that implements it. It resolves like CatalogPickerFrom (and unlike DocumentSearcherFrom) because it builds on the same picker reads, which the caching decorator does not forward. ok is false when no provider in the chain can read the governance vocabulary.
type Incident ¶ added in v1.44.0
type Incident struct {
URN string `json:"urn"`
Type string `json:"type"`
Title string `json:"title"`
Description string `json:"description,omitempty"`
State string `json:"state"`
Created int64 `json:"created,omitempty"`
}
Incident represents an active data incident from DataHub 1.4.x.
type InheritedMetadata ¶ added in v0.8.0
type InheritedMetadata struct {
// SourceURN is the DataHub URN of the upstream dataset.
SourceURN string `json:"source_urn"`
// SourceColumn is the column name in the upstream dataset.
SourceColumn string `json:"source_column"`
// Hops is the distance from the target dataset (1 = direct upstream).
Hops int `json:"hops"`
// MatchMethod indicates how the column was matched.
// Values: "column_lineage", "name_exact", "name_transformed", "alias"
MatchMethod string `json:"match_method"`
}
InheritedMetadata tracks the provenance of inherited column metadata.
type LineageDirection ¶
type LineageDirection string
LineageDirection indicates the direction of lineage traversal.
const ( LineageUpstream LineageDirection = "upstream" LineageDownstream LineageDirection = "downstream" )
Lineage direction constants.
type LineageEdge ¶
type LineageEdge struct {
URN string `json:"urn"`
Type string `json:"type,omitempty"`
TransformLogic string `json:"transform_logic,omitempty"`
}
LineageEdge represents an edge in the lineage graph.
type LineageEntity ¶
type LineageEntity struct {
URN string `json:"urn"`
Type string `json:"type"`
Name string `json:"name"`
Platform string `json:"platform,omitempty"`
Depth int `json:"depth"`
Parents []LineageEdge `json:"parents,omitempty"`
Children []LineageEdge `json:"children,omitempty"`
Context *TableContext `json:"context,omitempty"`
}
LineageEntity represents an entity in a lineage graph.
type LineageInfo ¶
type LineageInfo struct {
Direction LineageDirection `json:"direction"`
Entities []LineageEntity `json:"entities"`
MaxDepth int `json:"max_depth"`
}
LineageInfo contains lineage information for an entity.
type NoopProvider ¶
type NoopProvider struct{}
NoopProvider is a no-op implementation for testing.
func NewNoopProvider ¶
func NewNoopProvider() *NoopProvider
NewNoopProvider creates a new no-op provider.
func (*NoopProvider) GetColumnContext ¶
func (*NoopProvider) GetColumnContext(_ context.Context, _ ColumnIdentifier) (*ColumnContext, error)
GetColumnContext returns empty context.
func (*NoopProvider) GetColumnsContext ¶
func (*NoopProvider) GetColumnsContext(_ context.Context, _ TableIdentifier) (map[string]*ColumnContext, error)
GetColumnsContext returns empty map.
func (*NoopProvider) GetCuratedQueryCount ¶ added in v0.25.0
GetCuratedQueryCount returns zero for the noop provider.
func (*NoopProvider) GetGlossaryTerm ¶
func (*NoopProvider) GetGlossaryTerm(_ context.Context, _ string) (*GlossaryTerm, error)
GetGlossaryTerm returns an empty term.
func (*NoopProvider) GetLineage ¶
func (*NoopProvider) GetLineage(_ context.Context, _ TableIdentifier, dir LineageDirection, maxDepth int) (*LineageInfo, error)
GetLineage returns empty lineage.
func (*NoopProvider) GetTableContext ¶
func (*NoopProvider) GetTableContext(_ context.Context, _ TableIdentifier) (*TableContext, error)
GetTableContext returns empty context.
func (*NoopProvider) SearchTables ¶
func (*NoopProvider) SearchTables(_ context.Context, _ SearchFilter) ([]TableSearchResult, error)
SearchTables returns empty results.
type Owner ¶
type Owner struct {
URN string `json:"urn"`
Type OwnerType `json:"type"`
Name string `json:"name,omitempty"`
Email string `json:"email,omitempty"`
}
Owner represents a data owner.
type Provider ¶
type Provider interface {
// Name returns the provider name.
Name() string
// GetTableContext retrieves semantic context for a table.
GetTableContext(ctx context.Context, table TableIdentifier) (*TableContext, error)
// GetColumnContext retrieves semantic context for a single column.
GetColumnContext(ctx context.Context, column ColumnIdentifier) (*ColumnContext, error)
// GetColumnsContext retrieves semantic context for all columns of a table.
GetColumnsContext(ctx context.Context, table TableIdentifier) (map[string]*ColumnContext, error)
// GetLineage retrieves lineage information for a table.
GetLineage(ctx context.Context, table TableIdentifier, direction LineageDirection, maxDepth int) (*LineageInfo, error)
// GetGlossaryTerm retrieves a glossary term by URN.
GetGlossaryTerm(ctx context.Context, urn string) (*GlossaryTerm, error)
// SearchTables searches for tables matching the filter.
SearchTables(ctx context.Context, filter SearchFilter) ([]TableSearchResult, error)
// GetCuratedQueryCount returns the number of curated/saved queries for a dataset.
GetCuratedQueryCount(ctx context.Context, urn string) (int, error)
// Close releases resources.
Close() error
}
Provider retrieves semantic metadata from catalog systems. DataHub implements this. Future alternatives (Atlas, Unity Catalog) can too.
type SanitizeConfig ¶ added in v0.2.0
type SanitizeConfig struct {
// MaxLength is the maximum length for strings (default: 2000).
MaxLength int
// StripInjectionPatterns removes detected injection patterns instead of flagging.
StripInjectionPatterns bool
// LogInjectionAttempts enables logging of detected injection attempts.
LogInjectionAttempts bool
}
SanitizeConfig configures sanitization behavior.
func DefaultSanitizeConfig ¶ added in v0.2.0
func DefaultSanitizeConfig() SanitizeConfig
DefaultSanitizeConfig returns a safe default configuration.
type Sanitizer ¶ added in v0.2.0
type Sanitizer struct {
// contains filtered or unexported fields
}
Sanitizer sanitizes metadata strings to prevent prompt injection and other attacks.
func NewSanitizer ¶ added in v0.2.0
func NewSanitizer(cfg SanitizeConfig) *Sanitizer
NewSanitizer creates a new sanitizer with the given configuration.
func (*Sanitizer) DetectInjection ¶ added in v0.2.0
DetectInjection checks if the input contains potential prompt injection patterns. Returns true if injection is detected along with matched patterns.
func (*Sanitizer) SanitizeColumnContext ¶ added in v0.2.0
func (s *Sanitizer) SanitizeColumnContext(cc *ColumnContext) *ColumnContext
SanitizeColumnContext sanitizes all string fields in a ColumnContext.
func (*Sanitizer) SanitizeDataProduct ¶ added in v1.129.0
func (s *Sanitizer) SanitizeDataProduct(p *DataProduct) *DataProduct
SanitizeDataProduct sanitizes every string field of a data product.
func (*Sanitizer) SanitizeDataset ¶ added in v1.129.0
SanitizeDataset sanitizes every string field of a full dataset record: the embedded table context through SanitizeTableContext, then the identity, schema, saved-query, and document fields the record adds.
func (*Sanitizer) SanitizeDescription ¶ added in v0.2.0
SanitizeDescription sanitizes a description field.
func (*Sanitizer) SanitizeGlossaryTerm ¶ added in v1.129.0
func (s *Sanitizer) SanitizeGlossaryTerm(term *GlossaryTerm) *GlossaryTerm
SanitizeGlossaryTerm sanitizes every string field of a glossary term. The URNs (the term's own and its parent node's) are system identifiers and pass through unchanged.
func (*Sanitizer) SanitizeString ¶ added in v0.2.0
SanitizeString sanitizes a string by removing control characters, truncating to max length, and optionally stripping injection patterns.
func (*Sanitizer) SanitizeTableContext ¶ added in v0.2.0
func (s *Sanitizer) SanitizeTableContext(tc *TableContext) *TableContext
SanitizeTableContext sanitizes all string fields in a TableContext.
func (*Sanitizer) SanitizeTag ¶ added in v0.2.0
SanitizeTag validates and sanitizes a tag name. Returns empty string if the tag is invalid.
func (*Sanitizer) SanitizeTags ¶ added in v0.2.0
SanitizeTags sanitizes a slice of tags, removing invalid ones.
type SavedQuery ¶ added in v1.129.0
type SavedQuery struct {
URN string `json:"urn,omitempty"`
Name string `json:"name,omitempty"`
Statement string `json:"statement"`
Description string `json:"description,omitempty"`
// Source says how the query was recorded ("MANUAL", "SYSTEM").
Source string `json:"source,omitempty"`
// CreatedBy is the actor that recorded it; Created is when.
CreatedBy string `json:"created_by,omitempty"`
Created *time.Time `json:"created,omitempty"`
}
SavedQuery is one query the catalog holds against a dataset.
type SchemaField ¶ added in v1.129.0
type SchemaField struct {
// FieldPath is the full path of the field ("user.address.city" for a nested one).
FieldPath string `json:"field_path"`
// Type is the catalog's normalized type; NativeType is the platform's own.
Type string `json:"type"`
NativeType string `json:"native_type,omitempty"`
// Description is the field's documented meaning.
Description string `json:"description,omitempty"`
// Nullable reports whether the field admits NULL.
Nullable bool `json:"nullable"`
// IsPartitionKey marks a partition column.
IsPartitionKey bool `json:"is_partition_key,omitempty"`
// Tags and GlossaryTerms are the field-level classifications.
Tags []string `json:"tags,omitempty"`
GlossaryTerms []GlossaryTerm `json:"glossary_terms,omitempty"`
}
SchemaField is one declared field of a dataset schema.
type SearchFilter ¶
type SearchFilter struct {
Query string `json:"query"`
Platform string `json:"platform,omitempty"`
Tags []string `json:"tags,omitempty"`
Domain string `json:"domain,omitempty"`
Owner string `json:"owner,omitempty"`
Limit int `json:"limit,omitempty"`
Offset int `json:"offset,omitempty"`
// EntityTypes restricts search to specific entity types (e.g., "DATASET", "DASHBOARD").
// If empty, defaults to the DataHub client's DefaultEntityType.
EntityTypes []string `json:"entity_types,omitempty"`
// Mode selects the search strategy: "keyword" (default) or "semantic".
Mode string `json:"mode,omitempty"`
// Filters provides advanced field-level filtering (e.g., by column name, column tag).
// All filters are AND'd together. These map directly to DataHub's searchAcrossEntities
// orFilters and support fields like fieldPaths, fieldTags, fieldGlossaryTerms, etc.
Filters []FieldFilter `json:"filters,omitempty"`
}
SearchFilter defines criteria for searching tables.
type StructuredProperty ¶ added in v1.44.0
type StructuredProperty struct {
QualifiedName string `json:"qualified_name"`
DisplayName string `json:"display_name,omitempty"`
Values []any `json:"values"`
}
StructuredProperty represents a typed custom property from DataHub 1.4.x.
type TableContext ¶
type TableContext struct {
// Basic info
URN string `json:"urn,omitempty"`
Description string `json:"description,omitempty"`
// Ownership
Owners []Owner `json:"owners,omitempty"`
// Classification
Tags []string `json:"tags,omitempty"`
GlossaryTerms []GlossaryTerm `json:"glossary_terms,omitempty"`
Domain *Domain `json:"domain,omitempty"`
// TagRefs carries the same tags as Tags but as URN + display-name pairs, so a
// governance editor can remove or dedupe a tag by its URN (Tags holds only the
// display name, which is not the identifier a write needs). Populated on the
// entity-detail read; enrichment reads Tags, not this field (#785).
TagRefs []EntityRef `json:"tag_refs,omitempty"`
// Status
Deprecation *Deprecation `json:"deprecation,omitempty"`
// Quality
QualityScore *float64 `json:"quality_score,omitempty"`
// Metadata
CustomProperties map[string]string `json:"custom_properties,omitempty"`
LastModified *time.Time `json:"last_modified,omitempty"`
// Structured properties (DataHub 1.4.x)
StructuredProperties []StructuredProperty `json:"structured_properties,omitempty"`
// Incidents (DataHub 1.4.x)
ActiveIncidents int `json:"active_incidents,omitempty"`
Incidents []Incident `json:"incidents,omitempty"`
// Data contracts (DataHub 1.4.x)
DataContract *DataContractStatus `json:"data_contract,omitempty"`
}
TableContext provides semantic context for a table.
type TableIdentifier ¶
type TableIdentifier struct {
Catalog string `json:"catalog,omitempty"`
Schema string `json:"schema"`
Table string `json:"table"`
}
TableIdentifier uniquely identifies a table.
func (TableIdentifier) String ¶
func (t TableIdentifier) String() string
String returns a dot-separated representation.
type TableMatchCounter ¶ added in v1.120.0
type TableMatchCounter interface {
// SearchTablesCounted runs the same search as Provider.SearchTables and also
// reports the backend's total match count. The total may exceed len(results)
// both because the caller's limit bounded the page and because the backend
// bounded it further.
SearchTablesCounted(ctx context.Context, filter SearchFilter) (results []TableSearchResult, total int, err error)
}
TableMatchCounter is the optional total-matches capability for the dataset search: it reports how many matches the backend found, not merely how many rows it put in the page it returned. Only a real catalog backend implements it (the DataHub adapter), because only the backend knows the total.
It exists because a page-bounded backend silently reduces the requested limit — the DataHub client caps every search at its MaxLimit (100) — so a caller cannot discover "more matches exist" by asking for one row more than it needs. The extra row never arrives and the short page reads as a complete set (#1238). The total survives the clamp and rides back on the same response, so reading it costs no extra round trip.
type TableSearchResult ¶
type TableSearchResult struct {
URN string `json:"urn"`
Name string `json:"name"`
Platform string `json:"platform,omitempty"`
Description string `json:"description,omitempty"`
Tags []string `json:"tags,omitempty"`
Domain string `json:"domain,omitempty"`
MatchedField string `json:"matched_field,omitempty"`
}
TableSearchResult represents a search result.
func SearchTablesCounted ¶ added in v1.120.0
func SearchTablesCounted(ctx context.Context, p Provider, filter SearchFilter) (results []TableSearchResult, total int, err error)
SearchTablesCounted searches p and reports the backend's total match count alongside the page. When no provider in p's chain can count, total is TotalUnknown and the caller has learned nothing about the matches beyond the rows it holds — in particular it must not read a full page as proof that more exist, nor a short one as proof that none do.
The count is read from the innermost counting provider, bypassing the cache decorator, which is what CachedProvider.SearchTables does with the search itself: searches vary too much to cache, so the decorator only forwards.
type URNResolver ¶
type URNResolver interface {
// ResolveURN converts a URN to a table identifier.
ResolveURN(ctx context.Context, urn string) (*TableIdentifier, error)
// BuildURN creates a URN from a table identifier.
BuildURN(ctx context.Context, table TableIdentifier) (string, error)
}
URNResolver can resolve URNs to table identifiers.