Documentation
¶
Overview ¶
Package datahub provides a DataHub implementation of the semantic provider.
Index ¶
- type Adapter
- func (a *Adapter) BrowseDocuments(ctx context.Context, offset, limit int) ([]semantic.DocumentResult, int, error)
- func (a *Adapter) BuildURN(_ context.Context, table semantic.TableIdentifier) (string, error)
- func (a *Adapter) Close() error
- func (a *Adapter) GetColumnContext(ctx context.Context, column semantic.ColumnIdentifier) (*semantic.ColumnContext, error)
- func (a *Adapter) GetColumnsContext(ctx context.Context, table semantic.TableIdentifier) (map[string]*semantic.ColumnContext, error)
- func (a *Adapter) GetCuratedQueryCount(ctx context.Context, urn string) (int, error)
- func (a *Adapter) GetDocument(ctx context.Context, urn string) (*semantic.DocumentResult, error)
- func (a *Adapter) GetGlossaryTerm(ctx context.Context, urn string) (*semantic.GlossaryTerm, error)
- func (a *Adapter) GetLineage(ctx context.Context, table semantic.TableIdentifier, ...) (*semantic.LineageInfo, error)
- func (a *Adapter) GetRelatedDocuments(ctx context.Context, urn string) ([]semantic.DocumentResult, error)
- func (a *Adapter) GetTableContext(ctx context.Context, table semantic.TableIdentifier) (*semantic.TableContext, error)
- func (a *Adapter) LineageConfig() LineageConfig
- func (a *Adapter) ListDomains(ctx context.Context) ([]semantic.EntityRef, error)
- func (*Adapter) Name() string
- func (*Adapter) ResolveURN(_ context.Context, urn string) (*semantic.TableIdentifier, error)
- func (a *Adapter) SearchDocuments(ctx context.Context, query string, limit int) ([]semantic.DocumentResult, error)
- func (a *Adapter) SearchGlossaryTerms(ctx context.Context, query string, limit int) ([]semantic.EntityRef, error)
- func (a *Adapter) SearchTables(ctx context.Context, filter semantic.SearchFilter) ([]semantic.TableSearchResult, error)
- func (a *Adapter) SearchTags(ctx context.Context, query string, limit int) ([]semantic.EntityRef, error)
- func (a *Adapter) SetMetrics(m *observability.Metrics)
- type AliasConfig
- type Client
- type ColumnTransformConfig
- type Config
- type LineageConfig
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Adapter ¶
type Adapter struct {
// contains filtered or unexported fields
}
Adapter implements semantic.Provider using DataHub.
func NewWithClient ¶
NewWithClient creates a new DataHub adapter with a provided client (for testing).
func (*Adapter) BrowseDocuments ¶ added in v1.93.0
func (a *Adapter) BrowseDocuments(ctx context.Context, offset, limit int) ([]semantic.DocumentResult, int, error)
BrowseDocuments enumerates context documents for the browse surface (#695): the offset/limit page of the complete corpus plus the total count. The list query "*" returns every document with no relevance threshold; WithOffset/WithLimit page it. The total comes from a separate count-only searchAcrossEntities scoped to the DOCUMENT entity type (the document-scoped SearchDocuments query fetches the total in GraphQL but the upstream client does not surface it, so the total is read from the typed SearchResult instead, with count=1 to avoid transferring a second full page). No visibility/status filter is applied, so the page and total describe the same complete set (drafts and hidden documents included).
func (*Adapter) GetColumnContext ¶
func (a *Adapter) GetColumnContext(ctx context.Context, column semantic.ColumnIdentifier) (*semantic.ColumnContext, error)
GetColumnContext retrieves column context from DataHub.
func (*Adapter) GetColumnsContext ¶
func (a *Adapter) GetColumnsContext(ctx context.Context, table semantic.TableIdentifier) (map[string]*semantic.ColumnContext, error)
GetColumnsContext retrieves all columns context from DataHub. When lineage is enabled, it inherits metadata from upstream datasets for undocumented columns.
func (*Adapter) GetCuratedQueryCount ¶ added in v0.25.0
GetCuratedQueryCount returns the number of curated/saved queries for a dataset.
func (*Adapter) GetDocument ¶ added in v1.92.0
GetDocument reads one context document by its URN and maps it to semantic.DocumentResult with the FULL body (not the truncated search snippet), so the fetch surface can dereference a urn:li:document:<id> reference to the complete content (#694). The upstream ErrNotFound (a URN that resolves to no document) is translated to semantic.ErrDocumentNotFound so the caller can render a structured not-found instead of a transport error.
func (*Adapter) GetGlossaryTerm ¶
GetGlossaryTerm retrieves a glossary term from DataHub.
func (*Adapter) GetLineage ¶
func (a *Adapter) GetLineage(ctx context.Context, table semantic.TableIdentifier, direction semantic.LineageDirection, maxDepth int) (*semantic.LineageInfo, error)
GetLineage retrieves lineage from DataHub.
func (*Adapter) GetRelatedDocuments ¶ added in v1.91.0
func (a *Adapter) GetRelatedDocuments(ctx context.Context, urn string) ([]semantic.DocumentResult, error)
GetRelatedDocuments returns the context documents linked to an entity URN (the reverse of a document's related assets) and maps them to semantic.DocumentResult, for entity-keyed document discovery (#692). It wraps the upstream client's GetRelatedDocuments.
func (*Adapter) GetTableContext ¶
func (a *Adapter) GetTableContext(ctx context.Context, table semantic.TableIdentifier) (*semantic.TableContext, error)
GetTableContext retrieves table context from DataHub.
func (*Adapter) LineageConfig ¶ added in v0.8.1
func (a *Adapter) LineageConfig() LineageConfig
LineageConfig returns the lineage configuration. This allows verifying that configuration was wired correctly.
func (*Adapter) ListDomains ¶ added in v1.100.2
ListDomains lists DataHub domains for the catalog domain picker (#785). Domains are not returned by name from searchAcrossEntities (no inline fragment upstream), so this wraps the client's dedicated ListDomains, which returns every domain with its display name; the picker filters client-side.
func (*Adapter) ResolveURN ¶
ResolveURN converts a DataHub URN to a table identifier.
func (*Adapter) SearchDocuments ¶ added in v1.91.0
func (a *Adapter) SearchDocuments(ctx context.Context, query string, limit int) ([]semantic.DocumentResult, error)
SearchDocuments searches DataHub context documents by relevance and maps them to the neutral semantic.DocumentResult for the unified search corpus (#692). It wraps the upstream client's SearchDocuments (mcp-datahub v1.10.0+), which searches the DOCUMENT entity type via searchAcrossEntities and returns ALL matching documents regardless of visibility or publication state: filtering to globally-visible, published documents is the caller's job (each result carries ShowInGlobalContext and Status for that). A query of "*" lists; the query is passed through verbatim.
func (*Adapter) SearchGlossaryTerms ¶ added in v1.100.2
func (a *Adapter) SearchGlossaryTerms(ctx context.Context, query string, limit int) ([]semantic.EntityRef, error)
SearchGlossaryTerms searches DataHub glossary terms by display name for the catalog glossary picker (#785). Glossary terms have no dedicated list method, so this uses searchAcrossEntities scoped to the GLOSSARY_TERM entity type, whose Name is resolved from the term's properties by the upstream client.
func (*Adapter) SearchTables ¶
func (a *Adapter) SearchTables(ctx context.Context, filter semantic.SearchFilter) ([]semantic.TableSearchResult, error)
SearchTables searches for tables in DataHub using searchAcrossEntities. Supports advanced field-level filtering (column names, tags, etc.) via SearchFilter.Filters.
func (*Adapter) SearchTags ¶ added in v1.100.2
func (a *Adapter) SearchTags(ctx context.Context, query string, limit int) ([]semantic.EntityRef, error)
SearchTags searches DataHub tags by display name for the catalog tag picker (#785), returning URN + name refs the picker resolves under the hood. It wraps the upstream client's ListTags, which name-searches the TAG entity type; an empty query lists tags. The returned name/URN come straight from DataHub so a selected value is always a real tag URN. ListTags has no server-side limit, so the page is truncated before sanitizing to avoid sanitizing discarded entries.
func (*Adapter) SetMetrics ¶ added in v1.70.0
func (a *Adapter) SetMetrics(m *observability.Metrics)
SetMetrics wraps the adapter's client in an instrumenting decorator that records DataHub metrics AND opens a per-operation span. The caller installs it only when metrics OR tracing is enabled; the metric record is nil-safe and the span is a no-op outside an active trace, so the public Provider interface is unchanged and the off subsystem costs effectively nothing.
type AliasConfig ¶ added in v0.8.0
type AliasConfig struct {
// Source is the fully-qualified source table name.
Source string `yaml:"source"`
// Targets are glob patterns matching target table names.
Targets []string `yaml:"targets"`
// ColumnMapping provides explicit column name mappings.
// Key: target column, Value: source column
ColumnMapping map[string]string `yaml:"column_mapping,omitempty"`
}
AliasConfig defines an explicit source-target relationship.
type Client ¶
type Client interface {
SearchAcrossEntities(ctx context.Context, query string, opts ...dhclient.SearchOption) (*types.SearchResult, error)
SemanticSearch(ctx context.Context, query string, opts ...dhclient.SearchOption) (*types.SearchResult, error)
SearchDocuments(ctx context.Context, query string, opts ...dhclient.SearchOption) ([]types.Document, error)
GetRelatedDocuments(ctx context.Context, urn string) ([]types.Document, error)
GetDocument(ctx context.Context, urn string) (*types.Document, error)
GetEntity(ctx context.Context, urn string) (*types.Entity, error)
GetSchema(ctx context.Context, urn string) (*types.SchemaMetadata, error)
GetSchemas(ctx context.Context, urns []string) (map[string]*types.SchemaMetadata, error)
GetLineage(ctx context.Context, urn string, opts ...dhclient.LineageOption) (*types.LineageResult, error)
GetColumnLineage(ctx context.Context, urn string) (*types.ColumnLineage, error)
GetGlossaryTerm(ctx context.Context, urn string) (*types.GlossaryTerm, error)
GetQueries(ctx context.Context, urn string) (*types.QueryList, error)
ListTags(ctx context.Context, filter string) ([]types.Tag, error)
ListDomains(ctx context.Context) ([]types.Domain, error)
Ping(ctx context.Context) error
Close() error
}
Client defines the interface for DataHub operations. This allows for mocking in tests.
type ColumnTransformConfig ¶ added in v0.8.0
type ColumnTransformConfig struct {
// TargetPattern is a glob pattern matching target dataset names.
TargetPattern string `yaml:"target_pattern"`
// StripPrefix removes this prefix from target column names.
StripPrefix string `yaml:"strip_prefix,omitempty"`
// StripSuffix removes this suffix from target column names.
StripSuffix string `yaml:"strip_suffix,omitempty"`
}
ColumnTransformConfig defines a path normalization rule.
type Config ¶
type Config struct {
URL string
Token string
Platform string // Default platform for URN building (e.g., "trino", "postgres")
Timeout time.Duration
Debug bool // Enable debug logging
// CatalogMapping maps query engine catalog names to metadata catalog names.
// For example: {"rdbms": "warehouse"} means the Trino "rdbms" catalog
// corresponds to the "warehouse" catalog in DataHub URNs.
CatalogMapping map[string]string
// Lineage configuration for inheritance-aware column resolution.
Lineage LineageConfig
}
Config holds DataHub adapter configuration.
type LineageConfig ¶ added in v0.8.0
type LineageConfig struct {
// Enabled activates lineage traversal for missing documentation.
Enabled bool `yaml:"enabled"`
// MaxHops limits upstream traversal depth. Range: 1-5. Default: 2.
MaxHops int `yaml:"max_hops"`
// Inherit specifies which metadata types to inherit.
// Valid: "glossary_terms", "descriptions", "tags"
Inherit []string `yaml:"inherit"`
// ConflictResolution determines behavior when multiple upstreams
// define metadata for the same column.
// Values: "nearest" (closest upstream wins), "all" (merge), "skip" (no inheritance on conflict)
ConflictResolution string `yaml:"conflict_resolution"`
// PreferColumnLineage uses DataHub's column-level lineage edges when available.
PreferColumnLineage bool `yaml:"prefer_column_lineage"`
// ColumnTransforms defines path normalization rules.
ColumnTransforms []ColumnTransformConfig `yaml:"column_transforms"`
// Aliases defines explicit source-target mappings that bypass lineage lookup.
Aliases []AliasConfig `yaml:"aliases"`
// CacheTTL for lineage graphs.
CacheTTL time.Duration `yaml:"cache_ttl"`
// Timeout for the entire inheritance operation.
Timeout time.Duration `yaml:"timeout"`
}
LineageConfig controls lineage-aware semantic enrichment.
func DefaultLineageConfig ¶ added in v0.8.0
func DefaultLineageConfig() LineageConfig
DefaultLineageConfig returns sensible defaults.