Help us shape the future of Go! Take the 2026 Go Developer Survey

dbt

package
v0.5.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 22, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Documentation

Overview

Package dbt renders a dbt project's manifest.json into Dexaflow tasks (ADR 0042).

dbt already compiles a project into target/manifest.json — the canonical DAG of nodes (seeds, models, snapshots, tests) with their dependencies. This package reads that file in Go and emits flat Dexaflow tasks, so a dbt project becomes a Dexaflow DAG with no Cosmos at runtime and no Airflow in the parser.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Compile

func Compile(manifestJSON []byte, meta Meta) (domain.DAGSpec, error)

Compile renders a dbt manifest.json into tasks and assembles a complete dag.json DAGSpec around them. DagID is required; an empty Schedule yields an unscheduled DAG (the field is omitted, not set to "").

func EmbedGroup

func EmbedGroup(groupName string, tasks []domain.TaskSpec, upstream []string) ([]domain.TaskSpec, []string, error)

EmbedGroup namespaces a rendered group's tasks under groupName (taskID -> "<group>__<taskID>", since task_id forbids dots) and wires its external edges: the upstream task_ids become dependencies of the group's roots (tasks with no internal dependency). It returns the namespaced tasks and the group's leaf ids (tasks that nothing inside the group depends on) so the caller can attach them to the group's downstream tasks (ADR 0043).

func ExpandGroups

func ExpandGroups(tasks []domain.TaskSpec, render func(group string) ([]domain.TaskSpec, error)) ([]domain.TaskSpec, error)

ExpandGroups replaces each dbt_group placeholder task with the rendered + embedded dbt group's tasks (namespaced), rewiring any downstream dependents from the placeholder onto the group's leaves. render maps a group name to its rendered dbt tasks (the caller supplies manifest loading + Render).

func Render

func Render(manifestJSON []byte, opts Options) ([]domain.TaskSpec, error)

Render parses a dbt manifest.json and returns Dexaflow bash tasks. At node granularity it emits one task per executable node (`dbt <verb> --select`); at a grouped granularity it contracts nodes into per-group tasks (`dbt build --select`) and rejects a grouping that makes the task graph cyclic. Tasks are sorted by TaskID for deterministic output.

Types

type Granularity

type Granularity string

Granularity controls how dbt nodes are partitioned into Dexaflow tasks.

const (
	GranularityNode   Granularity = "node"
	GranularityLevel  Granularity = "level"
	GranularityFolder Granularity = "folder"
)

Granularity strategies. node is one task per dbt node; the rest contract a group of nodes into one task (a quotient of the dbt DAG), which must stay acyclic — the semantic strategy (folder) is validated, level is acyclic by construction. Tag/selector grouping is tracked separately (issue #398) because its overlap semantics (a node with several tags) needs a deliberate rule.

type Meta

type Meta struct {
	// Warn, when set, receives advisories that are not build failures — passed
	// through to Render. nil means say nothing (#1114).
	Warn        func(string)
	DagID       string
	DagVersion  string
	Image       string
	Owner       string
	Description string
	Tags        []string
	Schedule    string
	Granularity Granularity
	// Connections and Variables are the secret names the dexaflow.yaml declares
	// (ADR 0045 / ADR 0055). They must reach the spec: what a task pod is allowed
	// to see is derived from what the DAG declares. Dropping them here does not
	// merely omit a field — under `auth.secret_scoping: enforce` the pod is
	// delivered nothing and the DAG fails inside the task rather than at compile
	// (#997). Under the default permissive scoping the pod still receives the
	// whole tenant vault, so the declarations are simply not honored; the
	// exception there is a secret that lives only in an external backend, which
	// is requested by declared name and so is never fetched at all. The dag.py
	// path emits both; this path built its spec from an explicit field list that
	// omitted them.
	Connections []string
	Variables   []string
	// Connection and Profile, when set, wrap each task's dbt command with the
	// runtime step that writes profiles.yml from the managed connection (ADR 0043).
	Connection string
	Profile    string
	// Schema overrides the dbt target schema in the generated profile.
	Schema string
	// ProjectDir scopes the dbt commands with --project-dir for a subdir project.
	ProjectDir string
	// ProfilesDir, when set, is passed as --profiles-dir. It is how a project
	// that ships its own profiles.yml gets honored: the base image points
	// DBT_PROFILES_DIR at an ephemeral /tmp dir so dbt's WRITES stay off the
	// read-only project (#852), and the side effect is that dbt never looks
	// inside the project at all (#994). Only reads happen through this — target
	// and log paths keep their own env and still land in /tmp. Ignored when a
	// managed connection is set: that path generates a profile into
	// DBT_PROFILES_DIR and must win over a checked-in file.
	ProfilesDir string
	// Local marks a Lite/host build: with no Connection, each task is prefixed with
	// the step that writes a default duckdb profiles.yml (the zero-config local
	// warehouse). Ignored on the Pro/image path. See Options.Local.
	Local bool
}

Meta carries the DAG metadata a dbt manifest does not provide: identity, version, image, ownership, schedule, and the granularity strategy. These come from the Dexaflow project config, not from dbt.

type Options

type Options struct {
	// Granularity selects the partition strategy; the empty value means node.
	Granularity Granularity
	// Warn, when set, receives advisories that are not build failures — today
	// only a folder group-key collision (#1114). This package is otherwise pure,
	// so the caller decides where a warning goes; nil means say nothing.
	Warn func(string)
	// Connection, when set, is a managed Dexaflow connection id; each task's dbt
	// command is prefixed with the runtime step that writes profiles.yml from it
	// (ADR 0043 #2), so no credential is baked into the image.
	Connection string
	// Profile is the dbt project's profile name (dbt_project.yml `profile:`), used
	// as the generated profiles.yml key. Meaningful only when Connection is set.
	Profile string
	// Schema, when set, overrides the dbt target schema in the generated profile
	// (otherwise the connection's or the "public" default is used).
	Schema string
	// ProjectDir scopes each command with --project-dir so dbt finds dbt_project.yml
	// when the project is a subdirectory of the DAG (#401). "." or empty adds nothing.
	ProjectDir string
	// ProfilesDir, when set, is passed as --profiles-dir. It is how a project that
	// ships its own profiles.yml gets honored: the base image points
	// DBT_PROFILES_DIR at an ephemeral /tmp dir so dbt's WRITES stay off the
	// read-only project (#852), and the side effect is that dbt never looks inside
	// the project at all (#994). Only reads go through this — DBT_TARGET_PATH and
	// DBT_LOG_PATH still carry the writes to /tmp. Ignored when Connection is set:
	// that path generates a profile into DBT_PROFILES_DIR and must win over a
	// file checked into the project.
	ProfilesDir string
	// Local marks a Lite/host build: with no Connection, each task is prefixed with a
	// step that writes a default duckdb profiles.yml — a zero-config local warehouse,
	// no server and no connection needed (L4). Ignored on the Pro/image path.
	Local bool
	// DagConnections are the connection ids declared at the top level of
	// dexaflow.yaml. They are stamped onto each task ALONGSIDE Connection, not
	// instead of it: declaredConnections (internal/storage) returns the task's
	// list whenever it is non-empty, so stamping only the managed connection
	// shadows the DAG's own declarations and they never reach the pod — a
	// pre-hook's connection silently missing under enforce scoping or an
	// external secrets backend.
	DagConnections []string
}

Options configures a Render call.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL