dbt

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 19, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

README

dbt

Extract metadata from dbt project artifacts including models, sources, lineage, and ownership.

Usage

source:
  name: dbt
  scope: my-dbt-project
  config:
    manifest: target/manifest.json
    catalog: target/catalog.json

Configuration

Key Type Required Description
manifest string Yes Path to dbt manifest.json file.
catalog string No Path to dbt catalog.json file. When provided, column types are enriched with actual warehouse types.

Entities

Entity: model

dbt models (tables/views produced by dbt transformations).

Field Sample Value
urn urn:dbt:my-project:model:model.jaffle_shop.customers
name customers
description This table has basic information about a customer...
properties.database analytics
properties.schema jaffle_shop
properties.materialization table
properties.sql_path models/customers.sql
properties.language sql
properties.tags ["finance", "daily"]
properties.meta {"owner": "analytics-team"}
properties.columns [{"name": "id", "data_type": "INTEGER", "description": "Primary key"}]
Entity: source

External tables that dbt reads from (declared in sources.yml).

Field Sample Value
urn urn:dbt:my-project:source:source.jaffle_shop.raw.orders
name orders
description Raw orders table from the payment system
properties.database raw
properties.schema jaffle_shop
properties.source_name raw
properties.loader stitch
properties.columns [{"name": "id", "data_type": "INTEGER"}]
Edges
Type Source Target Description
derived_from model model Model depends on another model
derived_from model source Model depends on a source
owned_by model user Model ownership from meta.owner
owned_by source user Source ownership from meta.owner

How It Works

The extractor parses dbt's manifest.json artifact, which is generated by any dbt command (dbt run, dbt build, dbt compile, etc.). This file contains the full project graph — models, sources, tests, and their relationships.

When catalog.json is also provided (generated by dbt docs generate), column definitions are enriched with actual warehouse data types.

This approach works with both dbt Core (local artifacts in target/) and dbt Cloud (download artifacts via the API or CI).

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Catalog 

type Catalog struct {
	Nodes map[string]CatalogNode `json:"nodes"`
}

Catalog represents a dbt catalog.json file.

type CatalogColumn 

type CatalogColumn struct {
	Name    string `json:"name"`
	Type    string `json:"type"`
	Index   int    `json:"index"`
	Comment string `json:"comment"`
}

CatalogColumn holds actual column metadata from the warehouse.

type CatalogNode 

type CatalogNode struct {
	Columns map[string]CatalogColumn `json:"columns"`
	Stats   map[string]CatalogStat   `json:"stats"`
}

CatalogNode holds physical table metadata from the warehouse.

type CatalogStat 

type CatalogStat struct {
	ID          string `json:"id"`
	Label       string `json:"label"`
	Value       any    `json:"value"`
	Description string `json:"description"`
}

CatalogStat holds a table-level statistic.

type Config 

type Config struct {
	Manifest string `json:"manifest" yaml:"manifest" mapstructure:"manifest" validate:"required,file"`
	Catalog  string `json:"catalog" yaml:"catalog" mapstructure:"catalog" validate:"omitempty,file"`
}

type DependsOn 

type DependsOn struct {
	Nodes []string `json:"nodes"`
}

DependsOn lists upstream dependencies of a node.

type Extractor 

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

func New 

func New(logger log.Logger) *Extractor

func (*Extractor) Extract 

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

func (*Extractor) Init 

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

type Manifest 

type Manifest struct {
	Nodes   map[string]ManifestNode   `json:"nodes"`
	Sources map[string]ManifestSource `json:"sources"`
}

Manifest represents a dbt manifest.json file.

type ManifestColumn 

type ManifestColumn struct {
	Name        string `json:"name"`
	Description string `json:"description"`
	DataType    string `json:"data_type"`
}

ManifestColumn represents a column definition.

type ManifestNode 

type ManifestNode struct {
	UniqueID     string                    `json:"unique_id"`
	ResourceType string                    `json:"resource_type"`
	Name         string                    `json:"name"`
	Schema       string                    `json:"schema"`
	Database     string                    `json:"database"`
	Description  string                    `json:"description"`
	Columns      map[string]ManifestColumn `json:"columns"`
	DependsOn    DependsOn                 `json:"depends_on"`
	Tags         []string                  `json:"tags"`
	Meta         map[string]any            `json:"meta"`
	Config       NodeConfig                `json:"config"`
	Path         string                    `json:"path"`
	Language     string                    `json:"language"`
}

ManifestNode represents a model, seed, snapshot, or test in the manifest.

type ManifestSource 

type ManifestSource struct {
	UniqueID    string                    `json:"unique_id"`
	Name        string                    `json:"name"`
	SourceName  string                    `json:"source_name"`
	Schema      string                    `json:"schema"`
	Database    string                    `json:"database"`
	Description string                    `json:"description"`
	Columns     map[string]ManifestColumn `json:"columns"`
	Loader      string                    `json:"loader"`
	Tags        []string                  `json:"tags"`
	Meta        map[string]any            `json:"meta"`
}

ManifestSource represents a source definition in the manifest.

type NodeConfig 

type NodeConfig struct {
	Materialized string `json:"materialized"`
}

NodeConfig holds dbt node configuration.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.