Documentation
¶
Index ¶
- Constants
- Variables
- func ExtractTableReferences(sql string) []string
- func GetMacrosForModel(project *BuildProject, model *Model) string
- func LoadSeed(seed *Seed, connName string, fullRefresh bool) error
- func MakeSeedConfig(seed *Seed, connName string) *sling.Config
- func RewriteTableReferences(sql string, project *BuildProject, selfName string) (string, []string)
- func RunForHook(path string, opts sling.HookBuildRunOptions) (map[string]any, error)
- func SplitModelSQL(sql, dialect string) (preStatements []string, modelQuery string, postStatements []string, err error)
- func ValidateReadOnlyQuery(sql string, dbType dbio.Type) error
- func WarmParser()
- type Build
- func (b *Build) Compile() error
- func (b *Build) CompileJSONPayload() map[string]any
- func (b *Build) Execute() error
- func (b *Build) GetModelMode(model *Model) string
- func (b *Build) GetTarget() string
- func (b *Build) PrintCompileJSON()
- func (b *Build) PrintCompileOutput()
- func (b *Build) PrintListJSON()
- func (b *Build) PrintListOutput()
- type BuildConfig
- type BuildDefaults
- type BuildModelState
- type BuildOptions
- type BuildProject
- type BuildState
- func (bs *BuildState) GetStore() map[string]any
- func (bs *BuildState) Marshall() string
- func (bs *BuildState) SetStateData(id string, data map[string]any)
- func (bs *BuildState) SetStateKeyValue(id, key string, value any)
- func (bs *BuildState) SetStoreData(key string, value any, del bool)
- func (bs *BuildState) StepExecution() *sling.PipelineStepExecution
- func (bs *BuildState) TaskExecution() *sling.TaskExecution
- type BuildTargetState
- type DAG
- func (dag *DAG) DetectCycles() [][]string
- func (dag *DAG) GetDownstream(name string) []string
- func (dag *DAG) GetDownstreamN(name string, n int) []string
- func (dag *DAG) GetExecutionLevels() [][]string
- func (dag *DAG) GetUpstream(name string) []string
- func (dag *DAG) GetUpstreamN(name string, n int) []string
- func (dag *DAG) TopologicalSort() ([]string, error)
- type DAGNode
- type DbtProjectConfig
- type DevConfig
- type ExecutionResult
- type Executor
- type IncrementalContext
- type MacroFile
- type Model
- type ModelConfig
- type ModelSQL
- type Range
- type RangeChunk
- type RangeConfig
- type Seed
- type Selector
- type Style
- type TemplateEngine
Constants ¶
const ConfigFileName = "sling_build.yml"
ConfigFileName is the standard config file name.
const DefaultThreads = 4
DefaultThreads is the default parallelism for model execution.
Variables ¶
var ValidModes = map[string]bool{ "full-refresh": true, "view": true, "truncate": true, "incremental": true, "append": true, "snapshot": true, }
ValidModes are the recognized materialization modes.
Functions ¶
func ExtractTableReferences ¶
ExtractTableReferences extracts table references from FROM/JOIN clauses in SQL. It returns unique table references (schema.table or just table) found in the SQL, excluding CTEs and Jinja template expressions.
func GetMacrosForModel ¶
func GetMacrosForModel(project *BuildProject, model *Model) string
GetMacrosForModel returns the concatenated macro SQL applicable to a model. Macros are ordered root-first (outermost to innermost scope).
func LoadSeed ¶
LoadSeed loads a seed file into the target database using the existing sling task infrastructure. This gets CSV/JSON/Parquet parsing, type inference, bulk loading, and all 30+ connectors for free. Seeds always use full-refresh.
func MakeSeedConfig ¶
MakeSeedConfig creates a sling.Config for loading a seed file without executing it. Useful for testing and compile mode.
func RewriteTableReferences ¶
func RewriteTableReferences(sql string, project *BuildProject, selfName string) (string, []string)
RewriteTableReferences scans compiled SQL for table references matching prod-mode names of known models/seeds and rewrites them to current-mode FullTableNames. Returns the rewritten SQL and matched model/seed names (for DependsOn).
func RunForHook ¶
RunForHook compiles and executes a build project for a pipeline/replication hook. Returns a state map with per-node results for state.<step_id>.results.
func SplitModelSQL ¶
func SplitModelSQL(sql, dialect string) (preStatements []string, modelQuery string, postStatements []string, err error)
SplitModelSQL splits multi-statement SQL into pre-statements, the model query, and post-statements. The model query is the single SELECT/WITH/UNION statement.
Returns an error if zero or more than one query statement is found.
func ValidateReadOnlyQuery ¶
ValidateReadOnlyQuery returns an error unless sql is a single read-only statement for the given database type. It parses with golyglot; if the parser is unavailable or cannot parse the dialect, a conservative keyword guard decides. Neither layer can catch side-effecting functions (select my_procedure()) — use a read-only database user for connections a project token can reach.
func WarmParser ¶
func WarmParser()
WarmParser starts loading the golyglot dylib in the background so the first ValidateReadOnlyQuery does not pay the download/load cost. Init failure is non-fatal — the keyword fallback covers it.
Types ¶
type Build ¶
type Build struct {
Project *BuildProject
DAG *DAG
Engine *TemplateEngine
Options BuildOptions
Selected []string // selected node names after selector filtering
SubBuilds []*Build // compiled sub-projects (for multi-target compile mode)
// contains filtered or unexported fields
}
Build is the main orchestrator for sling build. It loads a project, compiles templates, builds the DAG, applies selectors, and provides compile output.
func NewBuild ¶
func NewBuild(dir string, opts BuildOptions) (*Build, error)
NewBuild creates a new Build from the given project directory and options.
func (*Build) Compile ¶
Compile loads the project, compiles all model templates, builds the DAG, and applies selectors. After Compile(), the Build is ready for execution or compile output.
func (*Build) CompileJSONPayload ¶
CompileJSONPayload returns the --compile --json object. Safe when Compile did not finish (e.g. a cycle): order/nodes stay empty.
func (*Build) Execute ¶
Execute runs the compiled build against the target database. Must be called after Compile().
func (*Build) GetModelMode ¶
GetModelMode returns the effective mode for a model, considering: 1. CLI --full-refresh flag overrides everything 2. Model config() block mode 3. Project defaults.mode 4. Default: full-refresh
func (*Build) PrintCompileJSON ¶
func (b *Build) PrintCompileJSON()
PrintCompileJSON prints compile output as JSON (models, deps, SQL).
func (*Build) PrintCompileOutput ¶
func (b *Build) PrintCompileOutput()
PrintCompileOutput prints the compile output in YAML format for each selected node.
func (*Build) PrintListJSON ¶
func (b *Build) PrintListJSON()
PrintListJSON prints selected nodes as JSON.
func (*Build) PrintListOutput ¶
func (b *Build) PrintListOutput()
PrintListOutput prints the selected models/seeds and exits.
type BuildConfig ¶
type BuildConfig struct {
Target string `yaml:"target"`
Dev *DevConfig `yaml:"dev,omitempty"`
DbtProject any `yaml:"dbt_project,omitempty"`
Vars map[string]any `yaml:"vars,omitempty"`
Defaults BuildDefaults `yaml:"defaults,omitempty"`
}
BuildConfig represents the contents of sling_build.yml.
type BuildDefaults ¶
type BuildDefaults struct {
Mode string `yaml:"mode,omitempty"`
Schema string `yaml:"schema,omitempty"`
Tags []string `yaml:"tags,omitempty"` // additive across nesting
UniqueKey any `yaml:"unique_key,omitempty"`
UpdateKey string `yaml:"update_key,omitempty"`
MergeStrategy string `yaml:"merge_strategy,omitempty"`
Enabled *bool `yaml:"enabled,omitempty"`
Hooks sling.HookMap `yaml:"hooks,omitempty"` // additive across nesting
DropCascade *bool `yaml:"drop_cascade,omitempty"`
}
BuildDefaults holds default settings for models.
type BuildModelState ¶
type BuildModelState struct {
Name string `json:"name,omitempty"`
Schema string `json:"schema,omitempty"`
FullName string `json:"full_name,omitempty"`
Mode string `json:"mode,omitempty"`
}
BuildModelState holds model metadata available in hooks.
type BuildOptions ¶
type BuildOptions struct {
Target string
Schema string
Prod bool
Vars map[string]any
FullRefresh bool
Select []string
Exclude []string
Compile bool
Threads int
FailFast bool
List bool
NoSeeds bool
Range *string // CLI --range: "start,end[,step]"
Recursive bool // CLI --recursive/-R: discover sling_build.yml in immediate subdirectories
Test bool // CLI --test: run data tests only (no materialization)
JSON bool // CLI --json: machine-readable compile/list output
}
BuildOptions holds CLI-provided overrides.
type BuildProject ¶
type BuildProject struct {
Dir string
Config *BuildConfig // from sling_build.yml (nil if missing)
Models map[string]*Model // keyed by unique model name
Seeds map[string]*Seed // keyed by unique seed name
Macros []*MacroFile // collected .macros.sql files
Mode string // "dev" or "prod"
SchemaOverride string // dev mode schema
DefaultSchema string // default schema for root-level files (default: "public")
ChildConfigs map[string]*BuildConfig // child sling_build.yml configs keyed by relative dir
SubProjects []*BuildProject // independent build projects (when no root yml)
Recursive bool // CLI --recursive: keep immediate child projects
}
BuildProject represents a sling build project discovered from a directory.
func LoadProject ¶
func LoadProject(dir string, opts ...BuildOptions) (*BuildProject, error)
LoadProject loads a build project from the given directory.
func (*BuildProject) AllNames ¶
func (p *BuildProject) AllNames() []string
AllNames returns a sorted list of all model and seed names.
func (*BuildProject) BuildProdNameIndex ¶
func (p *BuildProject) BuildProdNameIndex() map[string]prodNameEntry
BuildProdNameIndex builds a lookup map from lowercased prod FullTableName (and unqualified name) to the model/seed entry. Qualified names take priority over unqualified for the same key.
func (*BuildProject) GetEffectiveConfig ¶
func (p *BuildProject) GetEffectiveConfig(dir string) *BuildConfig
GetEffectiveConfig returns the merged config for the project, applying child overrides.
func (*BuildProject) LookupFullTableName ¶
func (p *BuildProject) LookupFullTableName(name string) (string, bool)
LookupFullTableName returns the full table name for a given model or seed name.
type BuildState ¶
type BuildState struct {
State map[string]map[string]any `json:"state,omitempty"`
Store map[string]any `json:"store,omitempty"`
Env map[string]any `json:"env,omitempty"`
Timestamp sling.DateTimeState `json:"timestamp,omitempty"`
Model BuildModelState `json:"model,omitempty"`
Target BuildTargetState `json:"target,omitempty"`
}
BuildState implements sling.RuntimeState for build model hooks.
func (*BuildState) GetStore ¶
func (bs *BuildState) GetStore() map[string]any
func (*BuildState) Marshall ¶
func (bs *BuildState) Marshall() string
func (*BuildState) SetStateData ¶
func (bs *BuildState) SetStateData(id string, data map[string]any)
func (*BuildState) SetStateKeyValue ¶
func (bs *BuildState) SetStateKeyValue(id, key string, value any)
func (*BuildState) SetStoreData ¶
func (bs *BuildState) SetStoreData(key string, value any, del bool)
func (*BuildState) StepExecution ¶
func (bs *BuildState) StepExecution() *sling.PipelineStepExecution
func (*BuildState) TaskExecution ¶
func (bs *BuildState) TaskExecution() *sling.TaskExecution
type BuildTargetState ¶
type BuildTargetState struct {
Name string `json:"name,omitempty"`
}
BuildTargetState holds target connection metadata available in hooks.
type DAG ¶
DAG represents a directed acyclic graph of models and seeds.
func BuildDAG ¶
func BuildDAG(project *BuildProject) (*DAG, error)
BuildDAG constructs a dependency graph from a project's models and seeds. Models must have been compiled first (DependsOn populated).
func (*DAG) DetectCycles ¶
DetectCycles finds cycles in the graph using DFS.
func (*DAG) GetDownstream ¶
GetDownstream returns all transitive downstream dependents of the given node.
func (*DAG) GetDownstreamN ¶
GetDownstreamN returns downstream dependents up to N degrees from the given node.
func (*DAG) GetExecutionLevels ¶
GetExecutionLevels groups nodes by depth for parallel execution. Each level contains nodes that can be executed concurrently.
func (*DAG) GetUpstream ¶
GetUpstream returns all transitive upstream dependencies of the given node.
func (*DAG) GetUpstreamN ¶
GetUpstreamN returns upstream dependencies up to N degrees from the given node.
func (*DAG) TopologicalSort ¶
TopologicalSort performs Kahn's algorithm to produce a topological ordering. Returns an error if a cycle is detected.
type DAGNode ¶
type DAGNode struct {
Name string
Model *Model // nil for seeds
Seed *Seed // nil for models
Dependencies []string
Dependents []string
Depth int
}
DAGNode represents a node in the dependency graph.
type DbtProjectConfig ¶
type DbtProjectConfig struct {
ModelsPath string `yaml:"models_path,omitempty"` // default: "models"
SeedsPath string `yaml:"seeds_path,omitempty"` // default: "seeds"
}
DbtProjectConfig holds dbt project compatibility settings.
type DevConfig ¶
type DevConfig struct {
Target string `yaml:"target,omitempty"` // optional, falls back to top-level target
Schema string `yaml:"schema"` // mandatory for dev mode
}
DevConfig holds dev-mode settings in sling_build.yml. When present, dev mode is the default (override with --prod).
type ExecutionResult ¶
type ExecutionResult struct {
Name string
NodeType string // "seed" or "model"
Mode string // "full-refresh", "view", "truncate", "incremental", "append"
Duration time.Duration
Err error
Skipped bool
}
ExecutionResult holds the outcome of executing one node.
type Executor ¶
type Executor struct {
Build *Build
ConnName string // resolved target connection name
DbConn database.Connection // database connection (nil until Connect)
Results []ExecutionResult // per-node results
RunID string // unique per Execute() for temp table isolation
// contains filtered or unexported fields
}
Executor runs a compiled Build against a target database.
func NewExecutor ¶
NewExecutor creates an Executor from a compiled Build.
func (*Executor) CreateSchemas ¶
CreateSchemas creates all unique schemas needed by the selected nodes.
type IncrementalContext ¶
type IncrementalContext struct {
WhereCond string // e.g., `"created_at" > '2024-01-01'` or "1=1"
Value string // e.g., `'2024-01-01'` or "null"
IsIncremental bool // drives is_incremental() for dbt-style models
}
IncrementalContext carries values used by CompileModel to resolve incremental placeholders and flags. One type serves both styles:
Style A (dbt): CompileModel reads IsIncremental only and passes it to the is_incremental() Jinja function. WhereCond/Value are ignored because the SQL does not contain the {incremental_where_cond}/{incremental_value} placeholders.
Style B (sling): CompileModel reads WhereCond/Value and substitutes them into the rendered SQL via g.R(). is_incremental() is still registered but returns IsIncremental (which callers typically leave false for sling-style models).
A nil *IncrementalContext is equivalent to DefaultIncrementalContext(): first-run semantics (WhereCond=1=1, Value=null, IsIncremental=false).
func DefaultIncrementalContext ¶
func DefaultIncrementalContext() *IncrementalContext
DefaultIncrementalContext returns a first-run context: placeholders resolve to "1=1"/"null" and is_incremental() returns false.
type MacroFile ¶
type MacroFile struct {
FilePath string // absolute path
Dir string // directory relative to project root ("" for root, "staging", "marts/core")
RawSQL string // raw file content with {% macro %} definitions
}
MacroFile represents a .macros.sql file containing Jinja macro definitions.
type Model ¶
type Model struct {
Name string // e.g., "dim_customers"
FilePath string // absolute path
RelPath string // relative path from project root
Schema string // derived from folder or override
Prefix string // underscore-joined nested folder names
FullTableName string // schema.prefix_name (current mode)
ProdFullTableName string // schema.prefix_name (always prod-mode, for SQL matching)
RawSQL string // raw file content (frontmatter stripped)
CompiledSQL string // after Jinja rendering
PreStatements []string // SQL statements before the model query (from multi-statement splitting)
PostStatements []string // SQL statements after the model query (from multi-statement splitting)
Config ModelConfig // from YAML frontmatter or config() block
HasFrontmatter bool // true if config was set via YAML frontmatter (config() becomes no-op)
Style Style // detected incremental pattern: StyleDbt or StyleSling (populated at load)
Refs []string // ref() dependencies
Sources []string // src() references
DependsOn []string // all DAG dependencies (refs + bare refs + auto-detected)
// contains filtered or unexported fields
}
Model represents a SQL model file in the project.
type ModelConfig ¶
type ModelConfig struct {
Mode string `yaml:"mode,omitempty"`
Materialized string `yaml:"materialized,omitempty"` // dbt alias for mode
UniqueKey any `yaml:"unique_key,omitempty"` // string or []string
MergeStrategy string `yaml:"merge_strategy,omitempty"`
UpdateKey string `yaml:"update_key,omitempty"`
Tags []string `yaml:"tags,omitempty"`
Hooks sling.HookMap `yaml:"hooks,omitempty"`
PreHook string `yaml:"pre_hook,omitempty"` // deprecated: kept for validation only
PostHook string `yaml:"post_hook,omitempty"` // deprecated: kept for validation only
Schema string `yaml:"schema,omitempty"`
Enabled *bool `yaml:"enabled,omitempty"`
Engine string `yaml:"engine,omitempty"`
Range *RangeConfig `yaml:"range,omitempty"`
DropCascade *bool `yaml:"drop_cascade,omitempty"` // default false; CASCADE on DROP when true
Rewrite *bool `yaml:"rewrite,omitempty"` // default true; set false to skip bare-name rewrite
Tests []any `yaml:"tests,omitempty"` // declarative data tests
}
ModelConfig holds configuration extracted from the config() block in a SQL model.
type ModelSQL ¶
ModelSQL holds the split result of a multi-statement SQL model file.
func MakeModelSQL ¶
MakeModelSQL splits a SQL model file into pre-statements, the model query, and post-statements. The model query is the single SELECT/WITH/UNION statement. All other statements are classified as pre (before) or post (after) the query.
type Range ¶
type Range struct {
Chunks []RangeChunk // ordered; 0 chunks = no-op
UpdateState bool // advance SLING_STATE after last chunk succeeds
FromCLI bool // came from --range (print resume hint on failure)
CLIRaw string // original raw --range value (for resume hint)
Step string // parsed step for resume hint, may be ""
}
Range is the resolved set of chunks for a single build model execution.
type RangeChunk ¶
type RangeChunk struct {
Lower string // already-quoted SQL literal, or "" for unbounded
Upper string // already-quoted SQL literal, or "" for unbounded
LowerInclusive bool // true → use >= for lower
ColType iop.ColumnType // for state writes
LowerRaw string // raw display form for logs/resume hint
UpperRaw string // raw display form for logs/resume hint
}
RangeChunk is a single [lower, upper) window for one merge pass.
func (RangeChunk) Describe ¶
func (c RangeChunk) Describe(updateKey string) string
Describe is used for log lines.
type RangeConfig ¶
type RangeConfig struct {
Start string `yaml:"start,omitempty"` // literal value; parsed lazily at execution time
Advance string `yaml:"advance,omitempty"` // duration (5m, 5h, 5d, 5w, 1mo, 1y) — per-run forward advance
Lookback string `yaml:"lookback,omitempty"` // duration
}
RangeConfig holds the range block from model front-matter. It drives the unified incremental / lookback / paged-backfill behavior (sling style only).
func (*RangeConfig) HasAdvance ¶
func (r *RangeConfig) HasAdvance() bool
HasAdvance returns true if paged-advance mode is enabled.
func (*RangeConfig) HasLookback ¶
func (r *RangeConfig) HasLookback() bool
HasLookback returns true if a lookback window is configured.
type Seed ¶
type Seed struct {
Name string // e.g., "country_codes"
FilePath string // absolute path
RelPath string // relative path from project root
Schema string
Prefix string
FullTableName string // schema.prefix_name (current mode)
ProdFullTableName string // schema.prefix_name (always prod-mode, for SQL matching)
Format string // csv, json, parquet
}
Seed represents a seed file (CSV, JSON, Parquet) in the project.
type Selector ¶
Selector filters DAG nodes based on include/exclude patterns.
func NewSelector ¶
NewSelector creates a new selector from include and exclude patterns.
type Style ¶
type Style int
Style identifies which incremental pattern a model uses.
const ( // StyleDbt is the dbt-compatible pattern: models use is_incremental() and {{ this }} // to write their own WHERE clauses. This is the zero value and the harmless default // for non-incremental models. StyleDbt Style = iota // StyleSling is the sling-native pattern: models use {incremental_where_cond} and/or // {incremental_value} placeholders, and sling owns the WHERE clause / watermark. StyleSling )
type TemplateEngine ¶
type TemplateEngine struct {
// contains filtered or unexported fields
}
TemplateEngine compiles SQL model templates using Jinja-like syntax.
func NewTemplateEngine ¶
func NewTemplateEngine(project *BuildProject, vars map[string]any) *TemplateEngine
NewTemplateEngine creates a new template engine for the given project.
func (*TemplateEngine) CompileAll ¶
func (te *TemplateEngine) CompileAll(incCtx *IncrementalContext) error
CompileAll compiles all models in the project using the provided incremental context. A nil incCtx is equivalent to DefaultIncrementalContext().
func (*TemplateEngine) CompileModel ¶
func (te *TemplateEngine) CompileModel(model *Model, incCtx *IncrementalContext) (string, error)
CompileModel compiles a model's SQL template, extracting config and resolving references. The incCtx parameter carries incremental-pattern values:
- Style A (dbt) models read incCtx.IsIncremental via the is_incremental() Jinja function.
- Style B (sling) models read incCtx.WhereCond / incCtx.Value via {incremental_where_cond} and {incremental_value} placeholders, substituted after Jinja rendering.
A nil incCtx is equivalent to DefaultIncrementalContext() — first-run semantics.