hcl

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 31, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

View Source
const (
	StatusAdded   = "added"
	StatusDropped = "dropped"
	StatusAltered = "altered"
)

Comparison statuses. Status is right-relative: "added" means the right side of the Diff has the object and the left does not.

View Source
const (
	KindTable            = "table"
	KindMaterializedView = "materialized_view"
	KindView             = "view"
	KindDictionary       = "dictionary"
	KindRaw              = "raw"
)

Object kinds accepted by RenderObjectSQL and RenderObjectHCL. They match the segment used in the web UI's object routes.

View Source
const (
	OpCreate = "CREATE"
	OpAlter  = "ALTER"
	OpDrop   = "DROP"
	OpRename = "RENAME"
)

Operation kinds.

View Source
const (
	DepMVSource          = "mv_source"          // a materialized view reads from this table
	DepMVDest            = "mv_dest"            // a materialized view writes into this table
	DepDistributedRemote = "distributed_remote" // a Distributed table forwards to this table
	DepTimeSeriesTarget  = "ts_target"          // a TimeSeries table references an external samples/tags/metrics target
	DepBufferDestination = "buffer_dest"        // a Buffer table forwards writes into this table
	DepViewSource        = "view_source"        // a plain view reads from this table

	// KindMVColumn flags a materialized view that references a column its
	// single source table does not provide (declared columns plus the
	// engine's virtual columns). This is a heuristic check, restricted to
	// virtual-prefixed names (those starting with `_`) and to MVs with a
	// single resolvable source and no JOIN/CTE/UNION/subquery/SELECT *.
	KindMVColumn = "mv_column"

	// KindDistributedColumn flags a Distributed table whose columns disagree
	// with its resolved remote storage table: a column the proxy declares is
	// absent from the remote, or its type/nullability differs. In strict mode
	// it also flags remote columns the proxy omits.
	KindDistributedColumn = "distributed_column"
)

Dependency kinds.

View Source
const KindNamedCollection = "named_collection"

KindNamedCollection is the object_type for named collections; the other object_type values reuse the Kind* constants in render.go (table, materialized_view, view, dictionary, raw).

View Source
const RedactedValue = "[HIDDEN]"

RedactedValue is the literal string ClickHouse substitutes for a secret when the connecting user lacks displaySecretsInShowAndSelect / the server lacks display_secrets_in_show_and_select / a query omits format_display_secrets_in_show_and_select. It shows up in system.named_collections values and in a dictionary's create_table_query SOURCE(...) credentials.

It is kept in-band: introspection stores the marker as the value, so it round-trips through an HCL dump and every comparison — live, dump file, or drift — can tell "this secret is unknown to hclexp" apart from "this object has no secret". The diff layer treats it as unknown ("don't generate a change") rather than as a literal target, and sqlgen refuses to emit any statement containing it: writing the string "[HIDDEN]" over a real cluster credential would be actively destructive.

Authored production HCL should use an external named collection instead of carrying either a real secret or this marker.

Variables

This section is empty.

Functions

func ApplySQL

func ApplySQL(schema *Schema, sql, defaultDatabase string, allowRaw bool) (int, error)

ApplySQL parses ClickHouse DDL and applies each statement, in order, to schema. It is the engine behind `hclexp sql2hcl`: developers express a change as the SQL they already know and it is folded into their declarative HCL.

Supported statements (declarative schema changes only):

  • CREATE TABLE | MATERIALIZED VIEW | VIEW | DICTIONARY — adds the object, or replaces an existing object of the same name.
  • ALTER TABLE … ADD/DROP/MODIFY/RENAME COLUMN, ADD/DROP INDEX, MODIFY/REMOVE TTL, MODIFY/RESET SETTING — edits the matching table block.
  • ALTER TABLE <mv> MODIFY QUERY … — replaces a materialized view's query.
  • DROP TABLE | VIEW | DICTIONARY | DATABASE — removes the object.
  • RENAME TABLE a TO b — renames (and moves across databases).

Data and partition operations (TRUNCATE, ALTER … DELETE, attach/detach/drop/ freeze/replace partition, materialize index/projection) carry no declarative schema meaning and are rejected with a clear error.

defaultDatabase supplies the database for unqualified object names. When it is empty and the schema has exactly one database, that database is used.

allowRaw mirrors `introspect`: a CREATE the schema model cannot express is captured verbatim as a raw{} block instead of failing. (A statement that does not even parse aborts the whole call — per-statement isolation is impossible once the parser has rejected the input.)

It returns the number of statements applied.

func BeautifySQL

func BeautifySQL(sql string) (string, bool)

BeautifySQL parses a single SQL statement and returns it re-rendered in the parser's beautified (indented, multi-line) form — the same visitor that produces readable view/MV queries elsewhere. It returns ok=false with the input unchanged when the statement can't be parsed, so callers can fall back to the verbatim SQL (e.g. for DDL the parser doesn't yet handle).

func FilterSchema

func FilterSchema(s *Schema, m *ExcludeMatcher)

FilterSchema removes every object the matcher excludes, in place. Both sides of a comparison are filtered before Diff so excluded objects appear in no output and no count. Databases are kept even when emptied.

func IntrospectMacros

func IntrospectMacros(ctx context.Context, conn driver.Conn) (map[string]string, error)

IntrospectMacros returns the ClickHouse macros configured on the connected node, read from system.macros. Macros carry a node's identity within the cluster — shard, replica, hostClusterRole, hostClusterType — substituted into ON CLUSTER DDL and ReplicatedMergeTree zoo paths. The returned map is keyed by macro name.

func IsVirtualColumn

func IsVirtualColumn(e Engine, name string) bool

IsVirtualColumn reports whether name is one of the implicit columns the engine contributes. Used as a name guard in the diff path; mode- and settings-independent on purpose — Kafka's stream-mode extras are included in the membership test so a column never silently shifts status when a setting changes.

The bare Nested parent (e.g. "_headers" for Kafka) is also recognised in addition to the dot-access form ("_headers.name").

func LayerFiles

func LayerFiles(path string) ([]string, error)

LayerFiles returns the .hcl files a layer path contributes, in load order: for a directory, every *.hcl inside it (lexical filename order); for a regular file, the file itself, which must have the .hcl extension so a stack entry pointing at a dump or a .sql script fails loudly rather than being parsed as HCL.

func MatchesPattern

func MatchesPattern(pattern, database, name string) bool

MatchesPattern reports whether a locate pattern (a filepath.Match glob, or a plain name) matches an object, trying the bare name and the qualified "<database>.<name>" form — the same convention exclude patterns use.

func RedactedParamKeys

func RedactedParamKeys(nc NamedCollectionSpec) []string

RedactedParamKeys returns the param keys whose introspected value is the literal "[HIDDEN]" placeholder ClickHouse returns when the user lacks displaySecretsInShowAndSelect / the server-side display_secrets_in_show_and_select is off. Returns an empty slice when the spec is fully un-redacted. Callers use this to warn operators that the diff cannot be trusted for those specific params.

func RenderDiffJSON

func RenderDiffJSON(cs ChangeSet, gen GeneratedSQL, left, right *Schema) ([]byte, error)

RenderDiffJSON builds the structured, dependency-ordered operation list from a GenerateSQL result. The operation order is the index in the already dependency-sorted statement list. Engine/replicated are looked up from the resolved schemas — the target (right) for CREATE/ALTER, falling back to the current (left) for DROP — because an ALTER that does not change the engine carries no engine information of its own. Unsafe flags come from the existing gen.Unsafe list (no separate diff path), matched by database + object.

func RenderObjectComparisons

func RenderObjectComparisons(w io.Writer, objs []ObjectComparison)

RenderObjectComparisons prints comparisons as the indented, +/-/~ marked summary used by `diff` (text mode) and `drift -details`. Objects render in input order under a header per database; named collections (empty database) render under a "named_collections" header. It consumes the same []ObjectComparison the JSON emits, so text and JSON cannot disagree.

func RenderObjectHCL

func RenderObjectHCL(database, kind, name string, db *DatabaseSpec) (string, error)

RenderObjectHCL returns the canonical HCL for a single object named name of the given kind within db, using the same writers as the schema dumper. It returns an error if the kind is unknown or no such object exists.

func RenderObjectSQL

func RenderObjectSQL(database, kind, name string, db *DatabaseSpec) (string, error)

RenderObjectSQL returns the ClickHouse CREATE DDL for a single object named name of the given kind within db. It reuses the same generators that GenerateSQL drives, so the output matches what a fresh `diff -sql` would emit. It returns an error if the kind is unknown or no such object exists.

func Resolve

func Resolve(s *Schema) error

Resolve walks each database, applies object patches, resolves extend chains, drops abstract tables and materialized views, and validates the remaining objects. All mutation happens in place on the supplied slice.

func SelectSchema

func SelectSchema(s *Schema, m *ExcludeMatcher)

SelectSchema keeps only the objects the matcher matches, in place — the inverse of FilterSchema, for "emit only these objects" selections (load -only). An empty matcher selects nothing to keep... which would erase the schema, so it is treated as "no selection" and leaves the schema unchanged; callers pass a matcher only when a selector was actually given. Databases are kept even when emptied (a split layer still needs the database{} wrapper and its cluster default); node blocks are untouched.

func Write

func Write(w io.Writer, schema *Schema) error

Write emits dbs as canonical HCL. Tables are sorted alphabetically; columns, indexes, and engine fields keep their structural order. Settings entries are sorted by key.

The dumper assumes the input has already been resolved: extend/abstract/ override are consumed, patches applied, engines decoded. Fields tagged diff:"-" in the type definitions are intentionally never emitted.

func WriteLayer

func WriteLayer(w io.Writer, schema *Schema) error

WriteLayer emits an unresolved authored layer. Unlike Write, it preserves patch blocks so generated decompositions can be loaded back through LoadLayers and Resolve. Declarations use the same canonical writers as dumps; patch collections are sorted by target name for deterministic output.

Types

type ChangeSet

type ChangeSet struct {
	Databases        []DatabaseChange
	NamedCollections []NamedCollectionChange
}

ChangeSet describes the changes required to evolve a `from` schema into a `to` schema. Empty databases (no tables to add/drop/alter) are omitted.

func Diff

func Diff(from, to *Schema) ChangeSet

Diff compares two resolved schemas and returns a deterministic ChangeSet. Both inputs must already have been resolved (engines decoded, abstracts dropped, extend/patches consumed).

func DiffWithOptions

func DiffWithOptions(from, to *Schema, options DiffOptions) ChangeSet

DiffWithOptions compares two resolved schemas using the supplied semantic policy. Both inputs must already have been resolved (engines decoded, abstracts dropped, extend/patches consumed).

func (ChangeSet) IsEmpty

func (cs ChangeSet) IsEmpty() bool

IsEmpty reports whether the change set has no work to do.

type ClusterSet

type ClusterSet struct {
	// contains filtered or unexported fields
}

ClusterSet maps external cluster names to the schema objects each cluster declares. It lets Validate resolve a Distributed table's remote against the composition of *another* cluster: the remote database is almost always "posthog" on every cluster, so the cluster name is the only discriminator.

A cluster is either mapped (a composition was loaded for it), absent (the env has no local composition for it, e.g. a node type with no instance here), or an alias of another cluster. References into an absent cluster are structurally unresolvable and count as satisfied; a ClickHouse remote_servers alias (e.g. "posthog_writable") shares the composition of its base cluster, so it resolves through that base. The zero value is a valid empty set.

func NewClusterSet

func NewClusterSet() ClusterSet

NewClusterSet returns an empty, initialized ClusterSet.

func (*ClusterSet) Add

func (cs *ClusterSet) Add(name string, dbs []DatabaseSpec)

Add records the composition of an external cluster: every table, view, materialized view, and raw object it declares becomes resolvable under name.

func (*ClusterSet) AddAbsent

func (cs *ClusterSet) AddAbsent(name string)

AddAbsent declares a cluster that has no composition available in this env. References into it are treated as satisfied (see the resolution algorithm in Validate).

func (*ClusterSet) AddAlias

func (cs *ClusterSet) AddAlias(alias, base string)

AddAlias records that alias resolves to base: a Distributed remote on the alias cluster is resolved against base's composition (or base's @absent status). base itself may be another alias; resolveCluster follows the chain.

type ColumnChange

type ColumnChange struct {
	Name string
	Old  ColumnSpec
	New  ColumnSpec
}

ColumnChange is an in-name modification of a column: its type and/or any modifier (default/materialized/ephemeral/alias/codec/ttl/comment/nullable) differs between Old and New. Name is the column's (current) name.

func (ColumnChange) IsUnsafe

func (c ColumnChange) IsUnsafe() bool

IsUnsafe reports whether the change switches the column's storage class — into or out of ALIAS / MATERIALIZED / EPHEMERAL. ClickHouse accepts such a MODIFY COLUMN but it is data-affecting (e.g. plain → ALIAS silently replaces stored values with the computed expression), so it is never auto-emitted. Changes within the same kind, or plain ↔ DEFAULT, plus codec/comment/ttl/ nullable/type changes, are in-place safe.

type ColumnSpec

type ColumnSpec struct {
	Name string `hcl:"name,label"`
	Type string `hcl:"type"`

	// Nullable wraps Type in Nullable(...) at SQL gen time. Combining
	// Nullable = true with a Type that already starts with "Nullable("
	// is rejected, matching ClickHouse's own behavior.
	Nullable bool `hcl:"nullable,optional"`

	// Default / Materialized / Ephemeral / Alias are mutually exclusive
	// default-value expressions, matching ClickHouse's
	// DEFAULT|MATERIALIZED|EPHEMERAL|ALIAS clauses. Ephemeral can be the
	// empty string, meaning a bare EPHEMERAL with no expression.
	Default      *string `hcl:"default,optional"`
	Materialized *string `hcl:"materialized,optional"`
	Ephemeral    *string `hcl:"ephemeral,optional"`
	Alias        *string `hcl:"alias,optional"`

	Comment *string `hcl:"comment,optional"`
	Codec   *string `hcl:"codec,optional"`
	TTL     *string `hcl:"ttl,optional"`

	// RenamedFrom declares that this column previously existed under another
	// name. The diff engine uses it to emit RENAME COLUMN instead of DROP +
	// ADD. Tagged diff:"-" because it's transient metadata, not part of the
	// schema being compared.
	RenamedFrom *string `hcl:"renamed_from,optional" diff:"-"`

	// After / First position a patch_table column add: the new column is
	// inserted immediately after the named column (or at the front) instead
	// of appended, so an env's mid-table extras can interleave without
	// redeclaring the table (#158). Patch-only — a declared table or MV
	// column carrying either is a resolve error, and modify_column rejects
	// them (repositioning base columns is genuine drift). Consumed and
	// cleared at patch application; the diff engine may also populate these
	// fields on its private AddColumns copies so SQL generation preserves the
	// target order. Tagged diff:"-" as transient placement metadata, like
	// RenamedFrom.
	After *string `hcl:"after,optional" diff:"-"`
	First bool    `hcl:"first,optional" diff:"-"`
}

type CompareSummary

type CompareSummary struct {
	TablesAdded             int `json:"tables_added"`
	TablesDropped           int `json:"tables_dropped"`
	TablesAltered           int `json:"tables_altered"`
	MVsAdded                int `json:"mvs_added"`
	MVsDropped              int `json:"mvs_dropped"`
	MVsAltered              int `json:"mvs_altered"`
	ViewsAdded              int `json:"views_added"`
	ViewsDropped            int `json:"views_dropped"`
	ViewsAltered            int `json:"views_altered"`
	DictsAdded              int `json:"dicts_added"`
	DictsDropped            int `json:"dicts_dropped"`
	DictsAltered            int `json:"dicts_altered"`
	RawsAdded               int `json:"raws_added"`
	RawsDropped             int `json:"raws_dropped"`
	RawsAltered             int `json:"raws_altered"`
	NamedCollectionsChanged int `json:"named_collections_changed"`
}

CompareSummary counts comparisons by object type and status.

func SummarizeComparisons

func SummarizeComparisons(objs []ObjectComparison) CompareSummary

SummarizeComparisons counts comparisons by object type and status.

func (CompareSummary) OneLiner

func (s CompareSummary) OneLiner() string

OneLiner renders the summary as the compact drift line, e.g. "+1 table, ~2 mv, ~1 raw". An all-zero summary falls back to "changed".

type ConstraintChange

type ConstraintChange struct {
	Name string
	Old  ConstraintSpec
	New  ConstraintSpec
}

ConstraintChange is an in-name modification of a table constraint: its CHECK/ASSUME predicate differs between Old and New. ClickHouse has no MODIFY CONSTRAINT, so it is emitted as DROP CONSTRAINT + ADD CONSTRAINT.

type ConstraintSpec

type ConstraintSpec struct {
	Name   string  `hcl:"name,label"`
	Check  *string `hcl:"check,optional"`
	Assume *string `hcl:"assume,optional"`
}

ConstraintSpec is a table-level constraint. Either Check or Assume must be set, but not both. Check enforces a row-level boolean predicate on INSERT; Assume is an optimizer hint (the constraint is presumed true).

type DatabaseChange

type DatabaseChange struct {
	Database    string
	AddTables   []TableSpec // emitted via CREATE TABLE
	DropTables  []TableSpec // emitted via DROP TABLE
	AlterTables []TableDiff

	AddMaterializedViews   []MaterializedViewSpec // emitted via CREATE MATERIALIZED VIEW
	DropMaterializedViews  []string               // emitted via DROP VIEW
	AlterMaterializedViews []MaterializedViewDiff

	AddDictionaries   []DictionarySpec // emitted via CREATE OR REPLACE DICTIONARY
	DropDictionaries  []string         // emitted via DROP DICTIONARY
	AlterDictionaries []DictionaryDiff

	AddViews   []ViewSpec // emitted via CREATE VIEW
	DropViews  []string   // emitted via DROP VIEW
	AlterViews []ViewDiff

	AddRaws   []RawSpec   // emitted verbatim via the stored CREATE DDL
	DropRaws  []RawSpec   // emitted via a kind-mapped DROP
	AlterRaws []RawChange // emitted as DROP + CREATE (recreate)
}

DatabaseChange holds the per-database differences.

func (DatabaseChange) IsEmpty

func (dc DatabaseChange) IsEmpty() bool

type DatabaseSpec

type DatabaseSpec struct {
	Name string `hcl:"name,label"`

	// Cluster is the default ON CLUSTER target for every table declared in
	// this database. Tables can override via TableSpec.Cluster. Inheritance
	// happens during resolution; after Resolve, every TableSpec.Cluster
	// reflects the effective value.
	Cluster *string `hcl:"cluster,optional"`

	Tables  []TableSpec      `hcl:"table,block"`
	Patches []PatchTableSpec `hcl:"patch_table,block" diff:"-"`

	// ViewPatches and DictionaryPatches are the patch_view /
	// patch_dictionary counterparts of Patches. Like table patches they
	// accumulate across layers and are consumed during resolution.
	MaterializedViewPatches []PatchMaterializedViewSpec `hcl:"patch_materialized_view,block" diff:"-"`
	ViewPatches             []PatchViewSpec             `hcl:"patch_view,block"              diff:"-"`
	DictionaryPatches       []PatchDictionarySpec       `hcl:"patch_dictionary,block"        diff:"-"`

	// MaterializedViews are a sibling collection to Tables, not a flavored
	// TableSpec. They may `extend` an `abstract = true` table to inherit
	// its column list (and optionally cluster/comment).
	MaterializedViews []MaterializedViewSpec `hcl:"materialized_view,block"`

	// Dictionaries are a third sibling collection. Like MVs, they do not
	// participate in the table inheritance system.
	Dictionaries []DictionarySpec `hcl:"dictionary,block"`

	// Views are plain (non-materialized) views. Like MVs and dictionaries
	// they live as a sibling collection and do not participate in the
	// table inheritance system.
	Views []ViewSpec `hcl:"view,block"`

	// Raws are escape-hatch objects whose CREATE DDL could not be parsed
	// or expressed in this schema language. They are stored verbatim,
	// round-tripped unchanged, and recreated (DROP+CREATE) on change.
	Raws []RawSpec `hcl:"raw,block"`
}

func Introspect

func Introspect(ctx context.Context, conn driver.Conn, database string, allowRaw bool) (*DatabaseSpec, error)

Introspect builds a DatabaseSpec from a live ClickHouse database. The returned spec is "flat" — no extend/abstract/patch_table — and every TableSpec has its Engine.Decoded populated. Tables and column order match the declaration order ClickHouse reports.

The function reads `create_table_query` for each table and parses it with a ClickHouse SQL parser (orian/clickhouse-sql-parser). All per-table data is derived from that AST: columns (including TTL, CODEC, default/materialized/ephemeral/alias, comment), indexes, constraints, engine, ORDER BY, PARTITION BY, SAMPLE BY, table TTL, SETTINGS, and table comment.

func IntrospectWithExclude

func IntrospectWithExclude(ctx context.Context, conn driver.Conn, database string, allowRaw bool, exclude *ExcludeMatcher) (*DatabaseSpec, error)

IntrospectWithExclude is Introspect with an optional exclude matcher: objects whose name matches a pattern are skipped before their DDL is parsed, so transient tables (e.g. ClickHouse's _tmp_replace_*, migration tmp_* tables) neither appear in the dump nor abort introspection when their DDL can't be parsed. A nil matcher excludes nothing.

type Declaration

type Declaration struct {
	ObjectType string // KindTable, KindMaterializedView, KindView, KindDictionary, KindRaw, KindNamedCollection
	Database   string // empty for named collections (cluster-scoped)
	Name       string
	File       string
	Line       int

	Abstract bool
	Override bool
	Patch    bool   // declared via patch_table (strictly additive)
	Extends  string // the extend = "<parent>" attribute, when present
	RawKind  string // raw blocks only: the kind label
}

Declaration is one declaration site of a schema object in an HCL source file. It comes from a lightweight syntax scan — no gohcl decode, no Resolve — which is what preserves the source position and the raw inheritance flags (abstract/override/extend/patch_table) that resolution consumes and the resolved specs no longer carry.

func ScanDeclarations

func ScanDeclarations(files []string) ([]Declaration, error)

ScanDeclarations records every object declaration site in the given files, in file order. Files must be native HCL syntax, same as the loader; a parse error aborts the scan.

func ScanFileDeclarations

func ScanFileDeclarations(path string) ([]Declaration, string, error)

ScanFileDeclarations scans one file and additionally returns the label of its first node{} block ("" when the file has none). Per-node dumps carry exactly one node block, so the name attributes the file's declarations to their host.

type DeclaredColumn

type DeclaredColumn struct {
	Name string
	Type string
}

DeclaredColumn is a single column from the object declared in a CREATE statement: name + ClickHouse type expression as plain text. Used by tests that need to materialize stub source tables.

func ColumnsProvidedBy

func ColumnsProvidedBy(t TableSpec, r TableResolver) []DeclaredColumn

ColumnsProvidedBy returns every column readable from t: declared columns first (in declaration order), then engine-contributed virtual columns the user has not shadowed by declaring a same-named column.

Declared wins: a user-written `column "_key" { ... }` on a Kafka table (e.g. via kafka_map_virtual_columns_on_write) appears in the output and the virtual is suppressed.

func ExtractDeclaredColumns

func ExtractDeclaredColumns(createSQL string) ([]DeclaredColumn, error)

ExtractDeclaredColumns parses a CREATE statement and returns the columns declared on the object being created:

  • CREATE TABLE / VIEW with explicit columns: the column list
  • CREATE MATERIALIZED VIEW: the destination column list (TO dest (…))
  • CREATE DICTIONARY: the attribute list (name+type only)

Returns an empty slice (no error) when the statement carries no column list (e.g. CREATE VIEW v AS SELECT … without an explicit schema).

func VirtualColumnsFor

func VirtualColumnsFor(e Engine, r TableResolver) []DeclaredColumn

VirtualColumnsFor returns the engine-defined implicit columns for e, or nil if the engine has none / e is nil. Dispatch is by interface assertion, not by Kind() switch — adding a new engine with virtuals only requires implementing one of the optional interfaces.

type Dependency

type Dependency struct {
	From ObjectRef // the dependent object (a materialized view or Distributed table)
	To   ObjectRef // the object it requires
	Kind string    // one of the Dep* constants

	// Cluster is the ClickHouse cluster the remote is expected on. It is set
	// only for DepDistributedRemote (from EngineDistributed.ClusterName) and
	// is the discriminator for cross-cluster resolution: the remote database
	// is almost always "posthog" on every cluster. Empty for all other kinds.
	Cluster string
}

Dependency records that one object (From) requires another (To) to exist before it can be created.

func CollectDependencies

func CollectDependencies(dbs []DatabaseSpec) ([]Dependency, error)

CollectDependencies walks every database and returns the creation-order dependencies carried by materialized views (source + destination tables) and Distributed-engine tables (their remote table).

type DictionaryAttribute

type DictionaryAttribute struct {
	Name         string  `hcl:"name,label"`
	Type         string  `hcl:"type"`
	Default      *string `hcl:"default,optional"`
	Expression   *string `hcl:"expression,optional"`
	Hierarchical bool    `hcl:"hierarchical,optional"`
	Injective    bool    `hcl:"injective,optional"`
	IsObjectID   bool    `hcl:"is_object_id,optional"`
}

type DictionaryDiff

type DictionaryDiff struct {
	Name    string
	Changed []string

	// Old is the source spec. Dictionary reconciliation rewrites the whole
	// object, so sqlgen needs the old source as well as the target to detect a
	// credential that exists on the live object but was redacted to [HIDDEN].
	// Without it, a target that omits the credential looks safe even though
	// CREATE OR REPLACE would silently remove the unknown live value (#178).
	Old DictionarySpec

	// New is the target spec. The CREATE OR REPLACE DICTIONARY that
	// reconciles the change is rendered from it.
	New DictionarySpec

	// SkippedRedactedSecrets names source fields (the DDL argument name, e.g.
	// "password") whose comparison was suppressed because one side holds the
	// RedactedValue marker and the other a real value. hclexp cannot tell
	// whether they differ, so it says so rather than guessing. Mirrors
	// NamedCollectionChange.SkippedRedactedParams.
	SkippedRedactedSecrets []string
}

DictionaryDiff describes a change to a dictionary. ClickHouse has no useful in-place ALTER DICTIONARY, so a change is normally materialized as CREATE OR REPLACE DICTIONARY. Replacing the object is non-destructive, but sqlgen still refuses a rewrite when a redacted source credential cannot be preserved. Changed lists field paths that differ, for rendering.

func (DictionaryDiff) IsEmpty

func (d DictionaryDiff) IsEmpty() bool

IsEmpty counts a skipped secret as a difference: a dictionary whose only divergence is an unverifiable credential is reported (with no DDL) rather than certified equal.

func (DictionaryDiff) IsUnsafe

func (d DictionaryDiff) IsUnsafe() bool

type DictionaryLayout

type DictionaryLayout interface{ Kind() string }

func DecodeDictionaryLayout

func DecodeDictionaryLayout(spec *DictionaryLayoutSpec) (DictionaryLayout, error)

DecodeDictionaryLayout dispatches on spec.Kind and decodes the body into the matching typed layout struct. Returns (nil, nil) when spec is nil.

type DictionaryLayoutSpec

type DictionaryLayoutSpec struct {
	Kind string   `hcl:"kind,label" diff:"-"`
	Body hcl.Body `hcl:",remain"    diff:"-"`

	Decoded DictionaryLayout
}

type DictionaryLifetime

type DictionaryLifetime struct {
	Min *int64 `hcl:"min,optional"`
	Max *int64 `hcl:"max,optional"`
}

DictionaryLifetime models LIFETIME(...). The simple form LIFETIME(n) sets only Min (Max remains nil). The range form LIFETIME(MIN x MAX y) sets both.

type DictionaryRange

type DictionaryRange struct {
	Min string `hcl:"min"`
	Max string `hcl:"max"`
}

type DictionarySource

type DictionarySource interface{ Kind() string }

func DecodeDictionarySource

func DecodeDictionarySource(spec *DictionarySourceSpec) (DictionarySource, error)

DecodeDictionarySource dispatches on spec.Kind and decodes the body into the matching typed source struct. Returns (nil, nil) when spec is nil.

type DictionarySourceSpec

type DictionarySourceSpec struct {
	Kind string   `hcl:"kind,label" diff:"-"`
	Body hcl.Body `hcl:",remain"    diff:"-"`

	Decoded DictionarySource
}

DictionarySourceSpec mirrors EngineSpec: Kind label + opaque hcl.Body + typed Decoded value. Decoded is populated by DecodeDictionarySource after the initial gohcl decode; Kind and Body are diff-skipped artifacts.

type DictionarySpec

type DictionarySpec struct {
	Name       string                `hcl:"name,label"`
	PrimaryKey []string              `hcl:"primary_key"`
	Attributes []DictionaryAttribute `hcl:"attribute,block"`
	Source     *DictionarySourceSpec `hcl:"source,block"`
	Layout     *DictionaryLayoutSpec `hcl:"layout,block"`
	Lifetime   *DictionaryLifetime   `hcl:"lifetime,block"`
	Range      *DictionaryRange      `hcl:"range,block"`
	Settings   map[string]string     `hcl:"settings,optional"`
	Cluster    *string               `hcl:"cluster,optional"`
	Comment    *string               `hcl:"comment,optional"`
}

DictionarySpec models a ClickHouse dictionary. Dictionaries are a sibling collection on DatabaseSpec, not a flavored table; they do not participate in the table inheritance system (extend/abstract/patch_table). Source and layout are modeled typed-per-kind via DictionarySource/DictionaryLayout — the same pattern engines use.

type DiffJSON

type DiffJSON struct {
	Objects    []ObjectComparison `json:"objects"`
	Operations []JSONOperation    `json:"operations"`
	Unsafe     []JSONUnsafe       `json:"unsafe,omitempty"`
	Summary    CompareSummary     `json:"summary"`
}

DiffJSON is the top-level document emitted by `hclexp diff -format json`. Objects is the per-object comparison view; Operations is the flat, dependency-ordered execution view of the same diff. Summary counts derive from Objects.

type DiffOptions

type DiffOptions struct {
	IgnoreColumnOrder bool
}

DiffOptions controls comparison semantics shared by diff, drift, and web. Column order is significant by default because it changes SELECT * output and must be preserved by a convergent migration. Callers may explicitly ignore it when node roles or operational policy do not require uniform physical ordering.

type DriftGroup

type DriftGroup struct {
	Key       string      `json:"key"`
	Reference string      `json:"reference"`
	Nodes     int         `json:"nodes"`
	Drifters  []DriftNode `json:"drifters"`
}

DriftGroup is one set of nodes expected to share a schema.

type DriftJSON

type DriftJSON struct {
	Groups  []DriftGroup    `json:"groups"`
	Summary DriftRunSummary `json:"summary"`
}

DriftJSON is the document emitted by `hclexp drift -format json`.

type DriftNode

type DriftNode struct {
	Node    string             `json:"node"`
	File    string             `json:"file"`
	Macros  map[string]string  `json:"macros,omitempty"`
	Objects []ObjectComparison `json:"objects"`
	Summary CompareSummary     `json:"summary"`
}

DriftNode is one drifting node's comparison against its group reference.

type DriftRunSummary

type DriftRunSummary struct {
	Nodes           int `json:"nodes"`
	Groups          int `json:"groups"`
	GroupsWithDrift int `json:"groups_with_drift"`
	DriftingNodes   int `json:"drifting_nodes"`
}

DriftRunSummary aggregates a whole drift run.

type DuplicateGroup

type DuplicateGroup struct {
	Database     string
	Name         string
	Declarations []Declaration // every site for the name, legitimate ones included
}

DuplicateGroup is one object name defined at more than one site without the composition system explaining the extras: patch sites are additive, override = true is a deliberate replacement, and extend-carrying sites refine an inherited definition. Abstract declarations still define schema and therefore count. A group qualifies when at least two definition sites remain.

func FindDuplicates

func FindDuplicates(decls []Declaration) []DuplicateGroup

FindDuplicates groups declarations by (database, name) — the ClickHouse namespace, which object types share — and returns groups holding two or more definition sites: declarations that are not patches or overrides and do not carry extend. Every declaration is retained in a reported group's output. Results are sorted by database then name.

type Engine

type Engine interface {
	Kind() string
}

Engine is the typed, decoded form of an engine block. Each ClickHouse engine kind is a distinct concrete type. Implementations are value types.

func DecodeEngine

func DecodeEngine(spec *EngineSpec) (Engine, error)

DecodeEngine dispatches on spec.Kind and decodes the body into a kind-specific struct. Returns (nil, nil) when spec is nil.

func ParseEngineString

func ParseEngineString(engineFull string) (Engine, error)

ParseEngineString parses a ClickHouse engine_full value into a typed HCL Engine. Strips the trailing ORDER BY / PARTITION BY / SAMPLE BY / TTL / SETTINGS / COMMENT clauses, then dispatches on the engine name.

type EngineAggregatingMergeTree

type EngineAggregatingMergeTree struct{}

func (EngineAggregatingMergeTree) Kind

func (EngineAggregatingMergeTree) Virtuals

type EngineBuffer

type EngineBuffer struct {
	Database  string `hcl:"database"`
	Table     string `hcl:"table"`
	NumLayers int64  `hcl:"num_layers"`
	MinTime   int64  `hcl:"min_time"`
	MaxTime   int64  `hcl:"max_time"`
	MinRows   int64  `hcl:"min_rows"`
	MaxRows   int64  `hcl:"max_rows"`
	MinBytes  int64  `hcl:"min_bytes"`
	MaxBytes  int64  `hcl:"max_bytes"`

	FlushTime  *int64 `hcl:"flush_time,optional"`
	FlushRows  *int64 `hcl:"flush_rows,optional"`
	FlushBytes *int64 `hcl:"flush_bytes,optional"`
}

EngineBuffer wraps another table with an in-memory write buffer that flushes periodically based on time/row/byte thresholds. CH syntax:

Buffer(database, table, num_layers, min_time, max_time, min_rows, max_rows, min_bytes, max_bytes [, flush_time [, flush_rows [, flush_bytes]]])

The optional flush_* triplet is captured as pointers; nil = not set (CH defaults apply).

func (EngineBuffer) Kind

func (EngineBuffer) Kind() string

type EngineChange

type EngineChange struct {
	Old Engine
	New Engine
}

type EngineCollapsingMergeTree

type EngineCollapsingMergeTree struct {
	SignColumn string `hcl:"sign_column"`
}

func (EngineCollapsingMergeTree) Kind

func (EngineCollapsingMergeTree) Virtuals

type EngineDistributed

type EngineDistributed struct {
	ClusterName    string  `hcl:"cluster_name"`
	RemoteDatabase string  `hcl:"remote_database"`
	RemoteTable    string  `hcl:"remote_table"`
	ShardingKey    *string `hcl:"sharding_key,optional"`
	PolicyName     *string `hcl:"policy_name,optional"`
}

func (EngineDistributed) DynamicVirtuals

func (e EngineDistributed) DynamicVirtuals(r TableResolver) []DeclaredColumn

DynamicVirtuals returns _shard_num plus the virtuals of the remote table, recursing through chained Distributed tables. Cycles are broken by a visited set. With r == nil or the remote not modelled in the schema, only _shard_num is returned.

func (EngineDistributed) Kind

func (EngineDistributed) Kind() string

func (EngineDistributed) Virtuals

func (EngineDistributed) Virtuals() []DeclaredColumn

Virtuals returns only the Distributed-local virtuals — used as the static fallback when no TableResolver is available. With a resolver, DynamicVirtuals returns the transitive set.

type EngineJoin

type EngineJoin struct {
	Strictness string   `hcl:"strictness"` // ANY | ALL | SEMI | ANTI
	JoinType   string   `hcl:"type"`       // LEFT | INNER | RIGHT | FULL
	Keys       []string `hcl:"keys"`
}

EngineJoin is a memory-resident table designed for the right-hand side of JOINs (and joinGet calls). CH syntax:

Join(strictness, type, k1[, k2, ...])

Strictness/type are bare identifiers in the DDL (not quoted strings) and must match the JOIN operation the table is used with — see the CH docs for the Join-table engine.

func (EngineJoin) Kind

func (EngineJoin) Kind() string

type EngineKafka

type EngineKafka struct {
	// Collection is the named-collection reference. The remaining fields are
	// optional per-table overrides when Collection is set.
	Collection *string `hcl:"collection,optional"`

	// BrokerList, TopicList, GroupName, and Format are required only when
	// Collection is nil.
	BrokerList *string `hcl:"broker_list,optional"`
	TopicList  *string `hcl:"topic_list,optional"`
	GroupName  *string `hcl:"group_name,optional"`
	Format     *string `hcl:"format,optional"`

	// Optional auth.
	SecurityProtocol *string `hcl:"security_protocol,optional"`
	SaslMechanism    *string `hcl:"sasl_mechanism,optional"`
	SaslUsername     *string `hcl:"sasl_username,optional"`
	SaslPassword     *string `hcl:"sasl_password,optional"`

	// Optional numeric tuning.
	NumConsumers         *int64 `hcl:"num_consumers,optional"`
	MaxBlockSize         *int64 `hcl:"max_block_size,optional"`
	SkipBrokenMessages   *int64 `hcl:"skip_broken_messages,optional"`
	PollTimeoutMs        *int64 `hcl:"poll_timeout_ms,optional"`
	PollMaxBatchSize     *int64 `hcl:"poll_max_batch_size,optional"`
	FlushIntervalMs      *int64 `hcl:"flush_interval_ms,optional"`
	ConsumerRescheduleMs *int64 `hcl:"consumer_reschedule_ms,optional"`
	MaxRowsPerMessage    *int64 `hcl:"max_rows_per_message,optional"`
	CompressionLevel     *int64 `hcl:"compression_level,optional"`

	// Optional booleans (introspected as 0/1, presented as bool in HCL).
	CommitEveryBatch     *bool `hcl:"commit_every_batch,optional"`
	ThreadPerConsumer    *bool `hcl:"thread_per_consumer,optional"`
	CommitOnSelect       *bool `hcl:"commit_on_select,optional"`
	AutodetectClientRack *bool `hcl:"autodetect_client_rack,optional"`

	// Optional strings.
	ClientID         *string `hcl:"client_id,optional"`
	Schema           *string `hcl:"schema,optional"`
	HandleErrorMode  *string `hcl:"handle_error_mode,optional"`
	CompressionCodec *string `hcl:"compression_codec,optional"`

	// Extra is the escape valve for kafka_* settings ClickHouse adds in
	// versions we don't yet model. Keys are passed through verbatim and
	// MUST include the `kafka_` prefix (the typed fields above strip it).
	Extra map[string]string `hcl:"extra,optional"`
}

func (EngineKafka) Kind

func (EngineKafka) Kind() string

func (EngineKafka) Virtuals

func (e EngineKafka) Virtuals() []DeclaredColumn

type EngineLog

type EngineLog struct{}

func (EngineLog) Kind

func (EngineLog) Kind() string

type EngineMemory

type EngineMemory struct{}

EngineMemory holds rows in RAM; non-durable, lost on restart. Common in test pipelines and short-lived staging tables.

func (EngineMemory) Kind

func (EngineMemory) Kind() string

type EngineMerge

type EngineMerge struct {
	DBRegex    string `hcl:"db_regex"`
	TableRegex string `hcl:"table_regex"`
}

EngineMerge is a read-only union over multiple physical tables matched by regex. CH syntax: Merge(db_regex, table_regex).

func (EngineMerge) Kind

func (EngineMerge) Kind() string

type EngineMergeTree

type EngineMergeTree struct{}

func (EngineMergeTree) Kind

func (EngineMergeTree) Kind() string

func (EngineMergeTree) Virtuals

func (EngineMergeTree) Virtuals() []DeclaredColumn

type EngineNull

type EngineNull struct{}

EngineNull discards every write. Often used as a MV source that fans out into multiple downstream MVs without storing rows itself.

func (EngineNull) Kind

func (EngineNull) Kind() string

type EngineReplacingMergeTree

type EngineReplacingMergeTree struct {
	VersionColumn *string `hcl:"version_column,optional"`
	// IsDeletedColumn is ClickHouse's optional is_deleted parameter (1 = the
	// row is a delete marker). ClickHouse only accepts it together with ver,
	// so version_column must be set too (enforced at resolve time).
	IsDeletedColumn *string `hcl:"is_deleted_column,optional"`
}

func (EngineReplacingMergeTree) Kind

func (EngineReplacingMergeTree) Virtuals

type EngineReplicatedAggregatingMergeTree

type EngineReplicatedAggregatingMergeTree struct {
	ZooPath     string `hcl:"zoo_path"`
	ReplicaName string `hcl:"replica_name"`
}

func (EngineReplicatedAggregatingMergeTree) Kind

func (EngineReplicatedAggregatingMergeTree) Virtuals

type EngineReplicatedCollapsingMergeTree

type EngineReplicatedCollapsingMergeTree struct {
	ZooPath     string `hcl:"zoo_path"`
	ReplicaName string `hcl:"replica_name"`
	SignColumn  string `hcl:"sign_column"`
}

func (EngineReplicatedCollapsingMergeTree) Kind

func (EngineReplicatedCollapsingMergeTree) Virtuals

type EngineReplicatedMergeTree

type EngineReplicatedMergeTree struct {
	ZooPath     string `hcl:"zoo_path"`
	ReplicaName string `hcl:"replica_name"`
}

func (EngineReplicatedMergeTree) Kind

func (EngineReplicatedMergeTree) Virtuals

type EngineReplicatedReplacingMergeTree

type EngineReplicatedReplacingMergeTree struct {
	ZooPath         string  `hcl:"zoo_path"`
	ReplicaName     string  `hcl:"replica_name"`
	VersionColumn   *string `hcl:"version_column,optional"`
	IsDeletedColumn *string `hcl:"is_deleted_column,optional"`
}

func (EngineReplicatedReplacingMergeTree) Kind

func (EngineReplicatedReplacingMergeTree) Virtuals

type EngineReplicatedSummingMergeTree

type EngineReplicatedSummingMergeTree struct {
	ZooPath     string   `hcl:"zoo_path"`
	ReplicaName string   `hcl:"replica_name"`
	SumColumns  []string `hcl:"sum_columns,optional"`
}

func (EngineReplicatedSummingMergeTree) Kind

func (EngineReplicatedSummingMergeTree) Virtuals

type EngineSharedAggregatingMergeTree

type EngineSharedAggregatingMergeTree struct {
	ZooPath     string `hcl:"zoo_path"`
	ReplicaName string `hcl:"replica_name"`
}

func (EngineSharedAggregatingMergeTree) Kind

func (EngineSharedAggregatingMergeTree) Virtuals

type EngineSharedCollapsingMergeTree

type EngineSharedCollapsingMergeTree struct {
	ZooPath     string `hcl:"zoo_path"`
	ReplicaName string `hcl:"replica_name"`
	SignColumn  string `hcl:"sign_column"`
}

func (EngineSharedCollapsingMergeTree) Kind

func (EngineSharedCollapsingMergeTree) Virtuals

type EngineSharedMergeTree

type EngineSharedMergeTree struct {
	ZooPath     string `hcl:"zoo_path"`
	ReplicaName string `hcl:"replica_name"`
}

func (EngineSharedMergeTree) Kind

func (EngineSharedMergeTree) Virtuals

type EngineSharedReplacingMergeTree

type EngineSharedReplacingMergeTree struct {
	ZooPath         string  `hcl:"zoo_path"`
	ReplicaName     string  `hcl:"replica_name"`
	VersionColumn   *string `hcl:"version_column,optional"`
	IsDeletedColumn *string `hcl:"is_deleted_column,optional"`
}

func (EngineSharedReplacingMergeTree) Kind

func (EngineSharedReplacingMergeTree) Virtuals

type EngineSharedSummingMergeTree

type EngineSharedSummingMergeTree struct {
	ZooPath     string   `hcl:"zoo_path"`
	ReplicaName string   `hcl:"replica_name"`
	SumColumns  []string `hcl:"sum_columns,optional"`
}

func (EngineSharedSummingMergeTree) Kind

func (EngineSharedSummingMergeTree) Virtuals

type EngineSpec

type EngineSpec struct {
	Kind string   `hcl:"kind,label"   diff:"-"`
	Body hcl.Body `hcl:",remain"      diff:"-"`

	// Decoded is populated post-parse via DecodeEngine; gohcl ignores it
	// because it has no hcl tag.
	Decoded Engine
}

EngineSpec holds both the raw HCL representation (Kind label + Body) and the typed Decoded value. Decoded is populated by ParseFile after the initial HCL decode. Diff comparisons use Decoded only — Kind and Body are raw artifacts skipped via the diff tag.

type EngineSummingMergeTree

type EngineSummingMergeTree struct {
	SumColumns []string `hcl:"sum_columns,optional"`
}

func (EngineSummingMergeTree) Kind

func (EngineSummingMergeTree) Virtuals

type EngineTimeSeries

type EngineTimeSeries struct {
	// Settings declared via `engine "time_series" { settings = {...} }`.
	// Free-form for forward compatibility. Only two are ALTER-able
	// (id_generator, filter_by_min_time_and_max_time); the rest are
	// baked into the CREATE.
	Settings map[string]string `hcl:"settings,optional"`

	// TagsToColumns shapes the inner tags-target table by promoting
	// specific label keys into their own columns. CH treats this as a
	// setting at the SQL level; promoted here for HCL clarity. sqlgen
	// folds it back into the SETTINGS clause at emit time.
	TagsToColumns map[string]string `hcl:"tags_to_columns,optional"`

	// Target sub-blocks. Nil = CH default applies (auto-generate the
	// inner target for that kind).
	Samples *TimeSeriesTarget `hcl:"samples,block"`
	Tags    *TimeSeriesTarget `hcl:"tags,block"`
	Metrics *TimeSeriesTarget `hcl:"metrics,block"`

	// KeywordHint preserves whether the samples target's source SQL used
	// SAMPLES or DATA. Populated by the introspector; sqlgen reads it to
	// round-trip the same word. HCL authors never set this — no hcl tag
	// so gohcl ignores it during decode.
	KeywordHint string `diff:"-"`
}

EngineTimeSeries models the (experimental) ClickHouse TimeSeries engine. It stores Prometheus-style time series across three sibling target tables (samples/tags/metrics), each either an external CH table reference or an inline declaration with INNER COLUMNS + optional INNER ENGINE.

TimeSeries exposes no engine-virtual columns — its outer columns are real declared columns. Virtuals() returns nil.

func (EngineTimeSeries) Kind

func (EngineTimeSeries) Kind() string

func (EngineTimeSeries) Virtuals

func (EngineTimeSeries) Virtuals() []DeclaredColumn

type EngineWithDynamicVirtuals

type EngineWithDynamicVirtuals interface {
	Engine
	DynamicVirtuals(r TableResolver) []DeclaredColumn
}

EngineWithDynamicVirtuals is implemented by engines whose implicit column set depends on other tables in the schema — for example, Distributed, which exposes _shard_num plus the virtuals of its remote table. When an engine implements both interfaces, the dynamic form is authoritative.

type EngineWithVirtuals

type EngineWithVirtuals interface {
	Engine
	Virtuals() []DeclaredColumn
}

EngineWithVirtuals is the optional interface an Engine implements when ClickHouse exposes implicit columns on tables of that kind. The returned set is static — derived from the engine's settings only, not from other tables in the schema. Engines with no virtuals (e.g. Log) simply don't implement it.

type ExcludeMatcher

type ExcludeMatcher struct {
	// contains filtered or unexported fields
}

ExcludeMatcher decides whether an object should be skipped during introspection or dropped from a schema before it is compared. It holds a list of glob patterns (filepath.Match syntax) matched against both the bare object name and its database-qualified "<database>.<name>" form, so patterns like "tmp_*" or "posthog.*_backup" both work, plus an optional set of object types that are excluded wholesale regardless of name. A nil matcher excludes nothing.

func LoadExcludeConfig

func LoadExcludeConfig(path string) (*ExcludeMatcher, error)

LoadExcludeConfig parses an HCL exclude config:

exclude {
  patterns     = ["tmp_*", "_tmp_replace_*", "*_backup", ...]
  object_types = ["named_collection"]   # optional: drop a whole class
}

Patterns are validated as globs and object types against the known object kinds at load time. A file with no exclude block (or no patterns and no object types) yields a matcher that excludes nothing.

func NewExcludeMatcher

func NewExcludeMatcher(patterns ...string) *ExcludeMatcher

NewExcludeMatcher builds a matcher from explicit patterns (used by tests).

func NewExcludeMatcherWithTypes

func NewExcludeMatcherWithTypes(objectTypes []string, patterns ...string) *ExcludeMatcher

NewExcludeMatcherWithTypes builds a matcher from explicit object types and patterns (used by tests and callers that don't load a config file).

func (*ExcludeMatcher) Empty

func (m *ExcludeMatcher) Empty() bool

Empty reports whether the matcher has no patterns and no object types (so it excludes nothing).

func (*ExcludeMatcher) Match

func (m *ExcludeMatcher) Match(database, name string) (pattern string, ok bool)

Match reports whether an object is excluded and, if so, the pattern that matched (for logging). It tries each pattern against the bare name and the "<database>.<name>" qualified form.

func (*ExcludeMatcher) Matches

func (m *ExcludeMatcher) Matches(database, name string) bool

Matches reports whether an object is excluded.

func (*ExcludeMatcher) MatchesObject

func (m *ExcludeMatcher) MatchesObject(objectType, database, name string) bool

MatchesObject reports whether an object is excluded, by its type or by a name pattern.

type FieldChange

type FieldChange struct {
	Field  string `json:"field"`
	Change string `json:"change"` // add | drop | modify | rename
	Old    string `json:"old,omitempty"`
	New    string `json:"new,omitempty"`
}

FieldChange is one attribute-level difference on an altered object. The Field vocabulary is a public contract, documented in docs/README.hcl.md.

type GeneratedSQL

type GeneratedSQL struct {
	Statements []string
	Ops        []Operation
	Unsafe     []UnsafeChange
}

GeneratedSQL holds the result of GenerateSQL: the DDL statements for the migration, plus a list of unsafe changes (engine, order_by, partition_by, sample_by) that ClickHouse cannot apply in place and require manual table recreation.

Ops is a structured, parallel view of Statements: Ops[i] describes Statements[i] (same order, same length), carrying the operation kind and the target object's identity so consumers don't have to re-parse the SQL text or re-derive the dependency order.

A statement whose Ops[i].Manual is true (e.g. MATERIALIZE INDEX) is generated for the operator, never for automatic execution: anything that executes Statements must skip it, and text rendering comments it out.

func GenerateSQL

func GenerateSQL(cs ChangeSet) GeneratedSQL

GenerateSQL turns a ChangeSet into ClickHouse DDL. Statements are ordered: CREATE TABLE, CREATE MATERIALIZED VIEW, ALTER TABLE, ALTER ... MODIFY QUERY, DROP VIEW, DROP TABLE — so materialized views are created after their destination table and dropped before it. Within the CREATE TABLE phase, tables are ordered by dependency so a Distributed table comes after the local table it forwards to; the DROP TABLE phase uses the reverse order. Unsafe or explicitly operator-reviewed changes (engine swap, existing-column reorder, ORDER BY change, materialized view recreation, etc.) are collected into Unsafe; the generator does not synthesize a recreate-and-swap procedure.

type IndexSpec

type IndexSpec struct {
	Name        string `hcl:"name,label"`
	Expr        string `hcl:"expr"`
	Type        string `hcl:"type"`
	Granularity int    `hcl:"granularity,optional"`

	// After / First position a patch_table index add, exactly like the
	// column placement fields (#158/#160): patch-only, consumed and
	// cleared at application, so the composed table renders identically
	// to a flat declaration in that order. A declared table index
	// carrying either is a resolve error.
	After *string `hcl:"after,optional" diff:"-"`
	First bool    `hcl:"first,optional" diff:"-"`
}

type JSONOperation

type JSONOperation struct {
	Order        int    `json:"order"`
	Kind         string `json:"kind"`        // CREATE | ALTER | DROP | RENAME
	ObjectType   string `json:"object_type"` // table | materialized_view | view | dictionary | named_collection | raw
	Database     string `json:"database"`    // empty for named collections
	Object       string `json:"object"`
	Engine       string `json:"engine"`     // ClickHouse engine family, tables only (e.g. ReplicatedMergeTree)
	Replicated   bool   `json:"replicated"` // true when the engine family is a Replicated* variant
	SQL          string `json:"sql"`        // statement without trailing ';'
	Manual       bool   `json:"manual"`     // operator-run only (e.g. MATERIALIZE INDEX); executors must skip it
	Unsafe       bool   `json:"unsafe"`     // this object has a change that can't be applied in place
	UnsafeReason string `json:"unsafe_reason"`
}

JSONOperation is one entry in the structured, dependency-ordered diff output emitted by `hclexp diff -format json`. It mirrors one generated statement (Operations is parallel to GeneratedSQL.Statements), enriched with the object's engine family and an unsafe flag.

type JSONUnsafe

type JSONUnsafe struct {
	Database string `json:"database"`
	Object   string `json:"object"`
	Reason   string `json:"reason"`
}

JSONUnsafe is one destructive change that is never auto-emitted. The top-level list surfaces every UnsafeChange — including those that produce no statement (e.g. an engine or ORDER BY change that requires a full recreate) — so consumers of the JSON output see the same warnings the text mode prints as `-- UNSAFE:` comments.

type LayoutCache

type LayoutCache struct {
	SizeInCells int64 `hcl:"size_in_cells"`
}

func (LayoutCache) Kind

func (LayoutCache) Kind() string

type LayoutComplexKeyCache

type LayoutComplexKeyCache struct {
	SizeInCells int64 `hcl:"size_in_cells"`
}

LayoutComplexKeyCache is the complex-key (multi-column key) counterpart to LayoutCache. Same single required parameter — size_in_cells controls the number of cells in the LRU cache.

func (LayoutComplexKeyCache) Kind

type LayoutComplexKeyDirect

type LayoutComplexKeyDirect struct{}

LayoutComplexKeyDirect is the complex-key counterpart to LayoutDirect: every lookup hits the source, no in-process caching. No parameters.

func (LayoutComplexKeyDirect) Kind

type LayoutComplexKeyHashed

type LayoutComplexKeyHashed struct {
	// Preallocate is accepted by ClickHouse for backwards compatibility
	// but no longer documented; kept so old DDL introspects faithfully.
	Preallocate           *int64   `hcl:"preallocate,optional"`
	Shards                *int64   `hcl:"shards,optional"`
	ShardLoadQueueBacklog *int64   `hcl:"shard_load_queue_backlog,optional"`
	MaxLoadFactor         *float64 `hcl:"max_load_factor,optional"`
}

func (LayoutComplexKeyHashed) Kind

type LayoutComplexKeyHashedArray

type LayoutComplexKeyHashedArray struct {
	Shards *int64 `hcl:"shards,optional"`
}

LayoutComplexKeyHashedArray is the complex-key counterpart to LayoutHashedArray. Same optional `shards` parameter.

func (LayoutComplexKeyHashedArray) Kind

type LayoutComplexKeyRangeHashed

type LayoutComplexKeyRangeHashed struct {
	RangeLookupStrategy *string `hcl:"range_lookup_strategy,optional"`
}

func (LayoutComplexKeyRangeHashed) Kind

type LayoutComplexKeySparseHashed

type LayoutComplexKeySparseHashed struct {
	Shards                *int64   `hcl:"shards,optional"`
	ShardLoadQueueBacklog *int64   `hcl:"shard_load_queue_backlog,optional"`
	MaxLoadFactor         *float64 `hcl:"max_load_factor,optional"`
}

func (LayoutComplexKeySparseHashed) Kind

type LayoutDirect

type LayoutDirect struct{}

func (LayoutDirect) Kind

func (LayoutDirect) Kind() string

type LayoutFlat

type LayoutFlat struct {
	InitialArraySize *int64 `hcl:"initial_array_size,optional"`
	MaxArraySize     *int64 `hcl:"max_array_size,optional"`
}

func (LayoutFlat) Kind

func (LayoutFlat) Kind() string

type LayoutHashed

type LayoutHashed struct {
	Shards                *int64   `hcl:"shards,optional"`
	ShardLoadQueueBacklog *int64   `hcl:"shard_load_queue_backlog,optional"`
	MaxLoadFactor         *float64 `hcl:"max_load_factor,optional"`
}

func (LayoutHashed) Kind

func (LayoutHashed) Kind() string

type LayoutHashedArray

type LayoutHashedArray struct {
	Shards *int64 `hcl:"shards,optional"`
}

LayoutHashedArray is a memory-efficient hash layout that stores values in a packed array. `shards` partitions the dictionary across N parallel hash tables for concurrent reads (default 1).

func (LayoutHashedArray) Kind

func (LayoutHashedArray) Kind() string

type LayoutIPTrie

type LayoutIPTrie struct {
	AccessToKeyFromAttributes *bool `hcl:"access_to_key_from_attributes,optional"`
}

func (LayoutIPTrie) Kind

func (LayoutIPTrie) Kind() string

type LayoutRangeHashed

type LayoutRangeHashed struct {
	RangeLookupStrategy *string `hcl:"range_lookup_strategy,optional"`
}

func (LayoutRangeHashed) Kind

func (LayoutRangeHashed) Kind() string

type LayoutRegexpTree

type LayoutRegexpTree struct{}

LayoutRegexpTree backs a dictionary that matches keys against an ordered tree of regular expressions. The layout itself takes no inner parameters.

func (LayoutRegexpTree) Kind

func (LayoutRegexpTree) Kind() string

type LayoutSparseHashed

type LayoutSparseHashed struct {
	Shards                *int64   `hcl:"shards,optional"`
	ShardLoadQueueBacklog *int64   `hcl:"shard_load_queue_backlog,optional"`
	MaxLoadFactor         *float64 `hcl:"max_load_factor,optional"`
}

func (LayoutSparseHashed) Kind

func (LayoutSparseHashed) Kind() string

type MaterializedViewDiff

type MaterializedViewDiff struct {
	Name        string
	QueryChange *StringChange // the AS SELECT body changed
	Recreate    bool          // the change cannot be applied with MODIFY QUERY

	// Detail fields used by renderers. ColumnsChanged can accompany a safe
	// QueryChange when the desired output is a compatible additive projection.
	ToTableChange  *StringChange
	ColumnsChanged bool
}

MaterializedViewDiff is the set of mutations to a single existing materialized view. Query changes are applied in place via ALTER TABLE ... MODIFY QUERY. A destination change or a non-additive/incompatible column change requires recreating the view and is flagged unsafe.

func (MaterializedViewDiff) IsEmpty

func (mvd MaterializedViewDiff) IsEmpty() bool

func (MaterializedViewDiff) IsUnsafe

func (mvd MaterializedViewDiff) IsUnsafe() bool

IsUnsafe reports whether the diff requires recreating the view.

type MaterializedViewSpec

type MaterializedViewSpec struct {
	Name string `hcl:"name,label"`

	// Inheritance fields. Consumed during resolution; not part of the
	// schema for diff purposes. Extend names an abstract table whose
	// columns (and optionally cluster/comment) are merged into this MV.
	// Abstract MVs are dropped before diff, like abstract tables.
	Extend   *string `hcl:"extend,optional"   diff:"-"`
	Abstract bool    `hcl:"abstract,optional" diff:"-"`
	Override bool    `hcl:"override,optional" diff:"-"`

	ToTable string       `hcl:"to_table,optional"` // TO <db.>table target (required when not abstract)
	Columns []ColumnSpec `hcl:"column,block"`      // explicit column list (may be empty; merged with parent on extend)
	Query   string       `hcl:"query,optional"`    // the AS SELECT ... body (required when not abstract)
	Cluster *string      `hcl:"cluster,optional"`  // ON CLUSTER
	Comment *string      `hcl:"comment,optional"`
}

MaterializedViewSpec models a ClickHouse materialized view in its `TO <table>` form: the MV reads from its source (referenced in Query) and writes rows into an existing destination table. Inner-engine MVs (ENGINE = ...), refreshable MVs (REFRESH ...), and window views are not supported and are rejected with a clear error during introspection.

type NamedCollectionChange

type NamedCollectionChange struct {
	Name string

	// Add is set for fresh adds AND for the create-half of a recreate
	// (so the create has the full target spec).
	Add *NamedCollectionSpec

	// Drop is true for pure drops AND for the drop-half of a recreate.
	Drop bool

	// Recreate is true when ON CLUSTER changed; sqlgen emits
	// DROP then CREATE adjacently.
	Recreate bool

	// Surgical (non-recreate) changes:
	SetParams     []NamedCollectionParam
	DeleteParams  []string
	CommentChange *StringChange

	// SkippedRedactedParams lists param keys whose diff was suppressed
	// because the value is the redacted "[HIDDEN]" placeholder on one side
	// while the other side holds a real value, differs in flags, or lacks
	// the param entirely. Identically-redacted params (hidden on both
	// sides, flags equal) are not a difference and are not listed. The CLI
	// surfaces these so operators know hclexp couldn't verify equality.
	SkippedRedactedParams []string

	// Error is non-empty when the diff describes an unsupported transition.
	// sqlgen emits no DDL; the CLI surfaces it.
	Error string
}

NamedCollectionChange describes a planned change to a named collection.

func (NamedCollectionChange) IsEmpty

func (c NamedCollectionChange) IsEmpty() bool

func (NamedCollectionChange) IsUnsafe

func (c NamedCollectionChange) IsUnsafe() bool

type NamedCollectionParam

type NamedCollectionParam struct {
	Key         string `hcl:"name,label"`
	Value       string `hcl:"value"`
	Overridable *bool  `hcl:"overridable,optional"`
}

type NamedCollectionSpec

type NamedCollectionSpec struct {
	Name     string                 `hcl:"name,label"`
	External bool                   `hcl:"external,optional"`
	Override bool                   `hcl:"override,optional" diff:"-"`
	Cluster  *string                `hcl:"cluster,optional"`
	Comment  *string                `hcl:"comment,optional"`
	Params   []NamedCollectionParam `hcl:"param,block"`
}

NamedCollectionSpec models a ClickHouse named collection — a cluster-scoped key/value bag of configuration values that other objects (most notably Kafka tables) can reference by name.

External = true marks a collection as managed outside hclexp — for example, defined in the ClickHouse server XML config under /etc/clickhouse-server/config.d/. hclexp emits no DDL for external collections (no CREATE / ALTER / DROP); they exist in HCL only as declarations so engine `collection = "x"` references can resolve and be validated. Params is optional when External = true (the values live in the XML config, not in HCL).

func IntrospectNamedCollections

func IntrospectNamedCollections(ctx context.Context, conn driver.Conn) ([]NamedCollectionSpec, error)

IntrospectNamedCollections returns every named collection the live ClickHouse cluster exposes via system.named_collections.

The cluster's system.named_collections schema varies by ClickHouse version. Newer versions expose `source` (DDL vs. config file) and `create_query` (the full CREATE statement, parseable for ON CLUSTER, comments, and OVERRIDABLE flags). Older versions expose only `name` and `collection` (a Map of key→value pairs).

This implementation reads only the columns guaranteed across versions — `name` and `collection`. As a consequence:

  • Config-file (XML) collections are NOT filtered out at introspection; the operator-side `external = true` declaration is the way to keep hclexp from trying to manage them.
  • ON CLUSTER, comment, and OVERRIDABLE/NOT OVERRIDABLE flags are not captured (the `collection` map doesn't carry them). Round-tripping these requires a newer ClickHouse with `create_query` exposed — extending the introspection then is a follow-up.

The query also enables `format_display_secrets_in_show_and_select` so values come back as their real strings rather than the redacted `[HIDDEN]` placeholder. The connecting user must have the `displaySecretsInShowAndSelect` access (granted via SQL `GRANT displaySecretsInShowAndSelect ON *.* TO <user>`, or in users.xml via `<show_named_collections_secrets>1</show_named_collections_secrets>`) for the setting to actually take effect — otherwise ClickHouse keeps returning `[HIDDEN]`.

type NodeSpec

type NodeSpec struct {
	Name   string            `hcl:"name,label"`
	Macros map[string]string `hcl:"macros,optional"`
}

NodeSpec records the identity of a single physical ClickHouse node, dumped alongside its schema. Name is the node hostname; Macros is the raw key/value bag from `SELECT * FROM system.macros`.

func IntrospectNode

func IntrospectNode(ctx context.Context, conn driver.Conn, nameOverride string) (NodeSpec, error)

IntrospectNode builds a NodeSpec for the connected node: its macros plus a name. When nameOverride is non-empty it is used as the node name; otherwise the name is read from ClickHouse via hostName().

type ObjectComparison

type ObjectComparison struct {
	Database   string `json:"database"` // empty for named collections
	Object     string `json:"object"`
	ObjectType string `json:"object_type"` // table | materialized_view | view | dictionary | named_collection | raw

	// RawKind is a raw{} block's inner object kind (table | view |
	// dictionary | materialized_view). Only a raw table holds rows on disk,
	// so its DROP+CREATE recreate is the destructive one — the kind is what
	// tells a consumer that apart.
	RawKind string `json:"raw_kind,omitempty"`

	Status       string          `json:"status"`            // added | dropped | altered
	Changes      []FieldChange   `json:"changes,omitempty"` // altered only
	Operations   []JSONOperation `json:"operations"`        // the DDL that reconciles this object; may be empty (unsafe-only changes)
	Unsafe       bool            `json:"unsafe"`
	UnsafeReason string          `json:"unsafe_reason,omitempty"`
	Error        string          `json:"error,omitempty"` // unsupported transition
}

ObjectComparison describes how one object differs between two schemas. It is a projection of the ChangeSet — the same structs the text summary renders — so JSON, text, and counts can never contradict each other.

func BuildObjectComparisons

func BuildObjectComparisons(cs ChangeSet, gen GeneratedSQL, left, right *Schema) []ObjectComparison

BuildObjectComparisons flattens a ChangeSet into one entry per differing object, attaching each generated operation to its object. Status is right-relative; nested operations keep their global dependency order, so the object view and the flat operation list can never disagree about sequencing.

type ObjectRef

type ObjectRef struct {
	Database string
	Name     string
}

ObjectRef identifies a schema object (table or materialized view) by its database and name.

func ExtractReferencedTables

func ExtractReferencedTables(createSQL string) ([]ObjectRef, error)

ExtractReferencedTables parses a CREATE statement (TABLE, MATERIALIZED VIEW, VIEW, or DICTIONARY) and returns every table it references that is NOT the object being created itself:

  • MV destination (TO dest) and SELECT FROM/JOIN sources
  • View SELECT FROM/JOIN sources
  • Dictionary SOURCE(CLICKHOUSE(TABLE 'x')) targets and SOURCE(CLICKHOUSE(QUERY 'SELECT ... FROM y')) inner-query sources
  • Distributed engine remote_table

CTE names introduced by WITH ... AS are filtered out (they're query-local, not real tables). The object being created itself is excluded. Database-unqualified refs are returned with Database = "".

Use this from test setup to derive the list of tables that must exist before the CREATE statement is executable.

func (ObjectRef) String

func (o ObjectRef) String() string

type Operation

type Operation struct {
	Kind       string // OpCreate | OpAlter | OpDrop | OpRename
	ObjectType string // table | materialized_view | view | dictionary | named_collection | raw
	Database   string // empty for named collections (cluster-scoped)
	Object     string
	SQL        string // the statement, without a trailing ';'
	Manual     bool   // operator-run only (heavy mutation, e.g. MATERIALIZE INDEX); never execute automatically
}

Operation is the typed description of one generated DDL statement.

type OrderByChange

type OrderByChange struct {
	Old []string
	New []string
}

type PatchColumnSpec

type PatchColumnSpec struct {
	Name string `hcl:"name,label"`

	Type     *string `hcl:"type,optional"`
	Nullable *bool   `hcl:"nullable,optional"`

	Default      *string `hcl:"default,optional"`
	Materialized *string `hcl:"materialized,optional"`
	Ephemeral    *string `hcl:"ephemeral,optional"`
	Alias        *string `hcl:"alias,optional"`

	Comment *string `hcl:"comment,optional"`
	Codec   *string `hcl:"codec,optional"`
	TTL     *string `hcl:"ttl,optional"`
}

PatchColumnSpec is a child-local, partial overlay for a column inherited through table Extend. Pointer fields distinguish an omitted attribute (preserve the parent value) from an explicit zero value such as nullable = false. If one default-kind attribute is set it replaces the inherited DEFAULT / MATERIALIZED / EPHEMERAL / ALIAS family wholesale. Optional modifiers cannot currently be removed; put the modifier-free shape on the abstract parent and add it on the child that needs it.

type PatchDictionarySpec

type PatchDictionarySpec struct {
	Name     string                `hcl:"name,label"`
	Source   *DictionarySourceSpec `hcl:"source,block"`
	Layout   *DictionaryLayoutSpec `hcl:"layout,block"`
	Lifetime *DictionaryLifetime   `hcl:"lifetime,block"`
	Settings map[string]string     `hcl:"settings,optional"`
}

PatchDictionarySpec is the dictionary counterpart of patch_table. Source, Layout, and Lifetime replace the target's wholesale when set; Settings merge patch-wins, like patch_table's.

type PatchMaterializedViewSpec

type PatchMaterializedViewSpec struct {
	Name          string       `hcl:"name,label"`
	Columns       []ColumnSpec `hcl:"column,block"`
	ModifyColumns []ColumnSpec `hcl:"modify_column,block"`
	DropColumns   []string     `hcl:"drop_columns,optional"`
	Query         *string      `hcl:"query,optional"`
}

PatchMaterializedViewSpec is the materialized-view counterpart of patch_table. Query replaces when set; columns follow patch_table's modify -> drop -> add semantics. It is consumed after MV extend resolution, so inherited columns may be modified or dropped too.

type PatchTableSpec

type PatchTableSpec struct {
	Name          string            `hcl:"name,label"`
	Columns       []ColumnSpec      `hcl:"column,block"`
	ModifyColumns []ColumnSpec      `hcl:"modify_column,block"`
	DropColumns   []string          `hcl:"drop_columns,optional"`
	Indexes       []IndexSpec       `hcl:"index,block"`
	DropIndexes   []string          `hcl:"drop_indexes,optional"`
	Projections   []ProjectionSpec  `hcl:"projection,block"`
	OrderBy       []string          `hcl:"order_by,optional"`
	PartitionBy   *string           `hcl:"partition_by,optional"`
	SampleBy      *string           `hcl:"sample_by,optional"`
	TTL           *string           `hcl:"ttl,optional"`
	Settings      map[string]string `hcl:"settings,optional"`
	Engine        *EngineSpec       `hcl:"engine,block"`
}

PatchTableSpec is a cross-layer modification of a table: the table stays declared once, and an env layer patches just its delta (#152, #154). Consumed during resolution; patches accumulate and apply in layer order, after their target inherits its parent and before any child inherits the resolved target. Inherited columns and indexes are therefore visible, and every descendant sees the patched result. Per-field semantics:

  • Columns add (a name already on the target errors); ModifyColumns replace an existing column in place (an unknown name errors); DropColumns remove (unknown errors). Applied modify → drop → add, so add sees the post-drop state.
  • Indexes add, DropIndexes remove; drops apply first, so a drop+add pair in one patch redefines an index.
  • Projections add (an existing name errors).
  • OrderBy / PartitionBy / SampleBy / TTL replace the target's value when set.
  • Engine replaces the target's engine block wholesale — merging engine sub-arguments is not meaningful.
  • Settings merge into the target's map, patch wins on key collision.

type PatchViewSpec

type PatchViewSpec struct {
	Name    string  `hcl:"name,label"`
	Query   *string `hcl:"query,optional"`
	Comment *string `hcl:"comment,optional"`
}

PatchViewSpec is the view counterpart of patch_table: the view stays declared once and an env layer replaces just the fields it sets. Query is normalized to the canonical beautified form at parse, like a declared view's.

type PlanOperation

type PlanOperation struct {
	Order        int      `json:"order"`
	Kind         string   `json:"kind"`
	ObjectType   string   `json:"object_type"`
	Database     string   `json:"database"`
	Object       string   `json:"object"`
	Engine       string   `json:"engine"`
	Replicated   bool     `json:"replicated"`
	SQL          string   `json:"sql"`
	Manual       bool     `json:"manual"` // operator-run only (e.g. MATERIALIZE INDEX); executors must skip it
	Roles        []string `json:"roles"`
	Unsafe       bool     `json:"unsafe"`
	UnsafeReason string   `json:"unsafe_reason"`
}

PlanOperation is one globally-ordered operation across all roles, with role provenance. It extends the diff -format json operation shape (#64) with the roles that contribute it; identical statements across roles collapse to one operation carrying the union of roles.

type PlanResult

type PlanResult struct {
	Operations []PlanOperation  `json:"operations"`
	Unsafe     []JSONUnsafe     `json:"unsafe,omitempty"`
	Roles      []RoleComparison `json:"roles"`
}

PlanResult is the merged, globally-ordered plan across every role.

func BuildPlan

func BuildPlan(roles []RoleDiff) PlanResult

BuildPlan diffs every role, unions the per-role operations into one global operation list, and orders it by cross-role dependency graphs built over the unioned desired and current schemas. Because every role's objects share one (database, object) keyspace, a Distributed/Buffer object that forwards to a table physically hosted on another role (e.g. a per-role writable_query_log_archive -> the OPS sharded_query_log_archive) gets a real dependency edge — the cross-role ordering the issue is about.

Ordering: CREATE and widening ALTER follow dependency direction (a referenced object before the object that references it — storage before proxies before the MV); DROP runs in reverse. Identical statements across roles dedupe to a single operation carrying the union of contributing roles.

type ProjectionSpec

type ProjectionSpec struct {
	Name     string            `hcl:"name,label"`
	Query    string            `hcl:"query"`
	Settings map[string]string `hcl:"settings,optional"`
}

ProjectionSpec is a table PROJECTION: a named SELECT (implicit FROM the parent table) materialized per part. Settings maps to the newer `WITH SETTINGS (…)` clause (projection-level MergeTree settings); ClickHouse before 26.5 neither emits nor accepts it.

type RawChange

type RawChange struct {
	Kind   string
	Name   string
	OldSQL string
	NewSQL string
}

RawChange describes a change to a raw escape-hatch object. Raw SQL is opaque, so the only reconciliation is DROP + CREATE. Kind is the object's kind (drives the DROP form); OldSQL/NewSQL are the stored DDL on each side.

func (RawChange) IsUnsafe

func (rc RawChange) IsUnsafe() bool

IsUnsafe reports whether recreating the object risks data loss. Only a table holds rows on disk; recreating a view/dictionary/materialized_view is not destructive.

type RawSpec

type RawSpec struct {
	Kind string `hcl:"kind,label"` // table | materialized_view | view | dictionary
	Name string `hcl:"name,label"`
	SQL  string `hcl:"sql"`
}

RawSpec is an opaque object captured as its original CREATE DDL. It is the escape hatch for DDL the parser cannot handle or the HCL model cannot express. The two labels mirror Terraform's `resource "<type>" "<name>"`: Kind drives the DROP form on a recreate, Name is the object name, and SQL is emitted verbatim on apply.

type RenameColumn

type RenameColumn struct {
	Old string
	New string
}

RenameColumn captures an explicit renamed_from directive that the diff engine has matched against the source schema. Old is the previous name, New is the current name.

type RoleComparison

type RoleComparison struct {
	Role    string             `json:"role"`
	Objects []ObjectComparison `json:"objects"`
	Summary CompareSummary     `json:"summary"`
}

RoleComparison is one role's per-object view of its diff. Unlike the merged Operations list it is deliberately NOT deduped across roles — triage is per (env, role), so a shared object drifting on two roles appears under both.

type RoleDiff

type RoleDiff struct {
	Role    string
	Desired *Schema
	Current *Schema
}

RoleDiff pairs a node role with its desired and current schema. Desired is the composed single-role authored schema (the target); Current is the matching node from a topology dump (one representative per role+shard, replicas collapsed). The plan emits the migration that brings Current to Desired.

type Schema

type Schema struct {
	Databases        []DatabaseSpec
	NamedCollections []NamedCollectionSpec

	// Nodes carries per-node identity captured at introspection time:
	// the node hostname (label) and its ClickHouse macros (shard,
	// replica, hostClusterRole, hostClusterType, …). It is metadata only
	// — Diff() ignores it — and exists so multi-node drift analysis can
	// group nodes by their authoritative macros rather than by filename.
	Nodes []NodeSpec
}

Schema is what ParseFile returns. It carries both top-level kinds hclexp tracks: databases (with their tables/MVs/dictionaries) and named collections (cluster-scoped, separate from any database).

func LoadLayers

func LoadLayers(layerPaths []string) (*Schema, error)

LoadLayers parses the .hcl files each layer path contributes, in the given order, and merges them into a combined raw spec set. A layer path is either a directory (every *.hcl inside it, in lexical filename order) or a single .hcl file. Across layers (and across files), a duplicate table or materialized-view name is an error unless the later declaration sets override = true. Patch blocks always accumulate.

LoadLayers does NOT call Resolve; callers run that explicitly so they can inspect the merged-but-unresolved input first.

func ParseFile

func ParseFile(path string) (*Schema, error)

ParseFile parses a single HCL file and returns the declared schema. Diagnostics are formatted into the returned error.

func ScopeSchemaToObjects

func ScopeSchemaToObjects(source, scope *Schema) *Schema

ScopeSchemaToObjects returns a non-mutating copy of source containing only objects whose logical identity occurs in scope. Database wrappers and node metadata are retained. It is the shared primitive behind directional diff scope and desired-scoped topology planning.

type SettingChange

type SettingChange struct {
	Key      string
	OldValue string
	NewValue string
}

type SkipSet

type SkipSet struct {
	// contains filtered or unexported fields
}

SkipSet selects dependent objects whose dependency validation should be skipped. It is built from a comma-separated list of object names; the special value "*" skips every object.

func ParseSkipSet

func ParseSkipSet(csv string) SkipSet

ParseSkipSet parses a comma-separated list of dependent object names. Each entry is matched against an object's bare name or its "database.name" form. The single entry "*" skips all objects.

func (SkipSet) Skips

func (s SkipSet) Skips(o ObjectRef) bool

Skips reports whether validation for the given dependent object is skipped.

type SourceClickHouse

type SourceClickHouse struct {
	Collection      *string `hcl:"collection,optional"`
	Host            *string `hcl:"host,optional"`
	Port            *int64  `hcl:"port,optional"`
	User            *string `hcl:"user,optional"`
	Password        *string `hcl:"password,optional"`
	DB              *string `hcl:"db,optional"`
	Table           *string `hcl:"table,optional"`
	Query           *string `hcl:"query,optional"`
	InvalidateQuery *string `hcl:"invalidate_query,optional"`
	UpdateField     *string `hcl:"update_field,optional"`
	UpdateLag       *int64  `hcl:"update_lag,optional"`
}

SourceClickHouse — SOURCE(CLICKHOUSE(...)).

func (SourceClickHouse) Kind

func (SourceClickHouse) Kind() string

type SourceExecutable

type SourceExecutable struct {
	Command     string `hcl:"command"`
	Format      string `hcl:"format"`
	ImplicitKey *bool  `hcl:"implicit_key,optional"`
}

SourceExecutable — SOURCE(EXECUTABLE(...)).

func (SourceExecutable) Kind

func (SourceExecutable) Kind() string

type SourceFile

type SourceFile struct {
	Path   string `hcl:"path"`
	Format string `hcl:"format"`
}

SourceFile — SOURCE(FILE(...)).

func (SourceFile) Kind

func (SourceFile) Kind() string

type SourceHTTP

type SourceHTTP struct {
	Collection          *string `hcl:"collection,optional"`
	URL                 string  `hcl:"url,optional"`
	Format              string  `hcl:"format,optional"`
	CredentialsUser     *string `hcl:"credentials_user,optional"`
	CredentialsPassword *string `hcl:"credentials_password,optional"`
}

SourceHTTP — SOURCE(HTTP(...)).

func (SourceHTTP) Kind

func (SourceHTTP) Kind() string

type SourceMySQL

type SourceMySQL struct {
	Collection      *string `hcl:"collection,optional"`
	Host            *string `hcl:"host,optional"`
	Port            *int64  `hcl:"port,optional"`
	User            *string `hcl:"user,optional"`
	Password        *string `hcl:"password,optional"`
	DB              *string `hcl:"db,optional"`
	Table           *string `hcl:"table,optional"`
	Query           *string `hcl:"query,optional"`
	InvalidateQuery *string `hcl:"invalidate_query,optional"`
	UpdateField     *string `hcl:"update_field,optional"`
	UpdateLag       *int64  `hcl:"update_lag,optional"`
}

SourceMySQL — SOURCE(MYSQL(...)).

func (SourceMySQL) Kind

func (SourceMySQL) Kind() string

type SourceNull

type SourceNull struct{}

SourceNull — SOURCE(NULL()).

func (SourceNull) Kind

func (SourceNull) Kind() string

type SourcePostgreSQL

type SourcePostgreSQL struct {
	Collection      *string `hcl:"collection,optional"`
	Host            *string `hcl:"host,optional"`
	Port            *int64  `hcl:"port,optional"`
	User            *string `hcl:"user,optional"`
	Password        *string `hcl:"password,optional"`
	DB              *string `hcl:"db,optional"`
	Table           *string `hcl:"table,optional"`
	Query           *string `hcl:"query,optional"`
	InvalidateQuery *string `hcl:"invalidate_query,optional"`
	UpdateField     *string `hcl:"update_field,optional"`
	UpdateLag       *int64  `hcl:"update_lag,optional"`
}

SourcePostgreSQL — SOURCE(POSTGRESQL(...)).

func (SourcePostgreSQL) Kind

func (SourcePostgreSQL) Kind() string

type StringChange

type StringChange struct {
	Old *string
	New *string
}

StringChange describes the transition of an optional string-valued attribute. A nil pointer on a side means "unset on that side."

type TableDiff

type TableDiff struct {
	Table string

	RenameColumns []RenameColumn
	AddColumns    []ColumnSpec
	DropColumns   []string
	ModifyColumns []ColumnChange

	AddIndexes  []IndexSpec
	DropIndexes []string

	// Projections keyed by name; a modify is emitted as DROP + ADD (there
	// is no ALTER MODIFY PROJECTION). Adding one to an existing table also
	// generates a manual MATERIALIZE PROJECTION.
	AddProjections  []ProjectionSpec
	DropProjections []string

	EngineChange *EngineChange
	// ColumnOrderChange records a reorder among columns present on both
	// sides. New columns carry FIRST/AFTER on AddColumns instead.
	ColumnOrderChange *OrderByChange
	OrderByChange     *OrderByChange
	PartitionByChange *StringChange
	SampleByChange    *StringChange
	TTLChange         *StringChange

	SettingsAdded   map[string]string
	SettingsRemoved []string
	SettingsChanged []SettingChange

	// Constraints keyed by name; a modify is emitted as DROP + ADD. All
	// in-place ALTER-able.
	AddConstraints    []ConstraintSpec
	DropConstraints   []string
	ModifyConstraints []ConstraintChange

	// PrimaryKeyChange is in-place-impossible (surfaced via unsafeReasons,
	// like ORDER BY). CommentChange is ALTER-able (MODIFY COMMENT).
	PrimaryKeyChange *OrderByChange
	CommentChange    *StringChange
}

TableDiff is the set of mutations to a single existing table. Any unset field means "no change in that aspect."

func (TableDiff) IsEmpty

func (td TableDiff) IsEmpty() bool

func (TableDiff) IsUnsafe

func (td TableDiff) IsUnsafe() bool

IsUnsafe reports whether the diff includes a change hclexp will not apply automatically. Most require table recreation; reordering existing columns is in-place-capable but deliberately left for explicit operator review.

type TableResolver

type TableResolver interface {
	LookupTable(db, name string) (TableSpec, bool)
}

TableResolver is the minimum lookup surface dynamic-virtuals engines need. Pass nil when no schema context is available — dynamic engines must degrade gracefully (Distributed returns just _shard_num).

func NewSchemaResolver

func NewSchemaResolver(dbs []DatabaseSpec) TableResolver

NewSchemaResolver builds a resolver over the loaded databases. Safe to call with an empty slice; the resolver returns ok=false for every lookup. Tables collide on (db, name); the last entry wins (Resolve already errors on duplicates within a single db, so collisions only happen across multiple DatabaseSpec entries for the same db name — rare, and the legacy multi-layer loader already merged them).

type TableSpec

type TableSpec struct {
	Name string `hcl:"name,label"`

	// Inheritance / control fields. Consumed during resolution; not part
	// of the schema for diff purposes.
	Extend   *string `hcl:"extend,optional"   diff:"-"`
	Abstract bool    `hcl:"abstract,optional" diff:"-"`
	Override bool    `hcl:"override,optional" diff:"-"`

	PrimaryKey  []string          `hcl:"primary_key,optional"`
	OrderBy     []string          `hcl:"order_by,optional"`
	PartitionBy *string           `hcl:"partition_by,optional"`
	SampleBy    *string           `hcl:"sample_by,optional"`
	TTL         *string           `hcl:"ttl,optional"`
	Settings    map[string]string `hcl:"settings,optional"`
	Comment     *string           `hcl:"comment,optional"`

	// Cluster is the ON CLUSTER target. May be set on the table itself, or
	// inherited from DatabaseSpec.Cluster during resolution.
	Cluster *string `hcl:"cluster,optional"`

	Columns []ColumnSpec `hcl:"column,block"`

	// ColumnPatches partially specialize columns inherited through Extend.
	// They are applied after the parent has resolved, preserve the inherited
	// column's position and every field they do not set, then are consumed.
	// A patch may only target an inherited column; ordinary column blocks
	// remain additions and still collide with inherited names.
	ColumnPatches []PatchColumnSpec `hcl:"patch_column,block" diff:"-"`

	Indexes     []IndexSpec      `hcl:"index,block"`
	Projections []ProjectionSpec `hcl:"projection,block"`
	Constraints []ConstraintSpec `hcl:"constraint,block"`
	Engine      *EngineSpec      `hcl:"engine,block"`
}

type TimeSeriesInnerTable

type TimeSeriesInnerTable struct {
	Columns     []ColumnSpec      `hcl:"column,block"`
	Engine      *EngineSpec       `hcl:"engine,block"`
	PrimaryKey  []string          `hcl:"primary_key,optional"`
	OrderBy     []string          `hcl:"order_by,optional"`
	PartitionBy *string           `hcl:"partition_by,optional"`
	Settings    map[string]string `hcl:"settings,optional"`
}

TimeSeriesInnerTable mirrors a small slice of TableSpec — only the fields CH actually emits inside INNER COLUMNS / INNER ENGINE.

type TimeSeriesTarget

type TimeSeriesTarget struct {
	Target *string               `hcl:"target,optional"`
	Inner  *TimeSeriesInnerTable `hcl:"inner,block"`
}

TimeSeriesTarget is one of {samples, tags, metrics}. Exactly one of Target or Inner must be set; both/neither is a resolve-time error.

type UnsafeChange

type UnsafeChange struct {
	Database string
	Table    string
	Reason   string
}

UnsafeChange describes a diff entry that can't be expressed as an ALTER. Database and Table identify the target; Reason is a human-readable explanation of what would need to change.

type ValidateOptions

type ValidateOptions struct {
	// StrictProxyColumns requires a Distributed table and its remote to have
	// exactly the same set of columns. When false (default), the proxy's
	// columns need only be a subset of the remote's (every proxy column must
	// exist on the remote with a matching type) — a proxy may legitimately
	// omit columns.
	StrictProxyColumns bool

	// StrictClusters requires every Distributed remote to resolve against a
	// real composition: a remote on an @absent cluster is an error instead of
	// being satisfied. Use it to enforce full validation when the whole fleet
	// is composed, so a stale @absent (a cluster that has since been mapped)
	// cannot silently pass.
	StrictClusters bool
}

ValidateOptions tunes optional checks layered on top of Validate's core dependency resolution.

type ValidationError

type ValidationError struct {
	Object  ObjectRef // the dependent object
	Missing ObjectRef // the unresolved reference
	Kind    string
	Cluster string // Distributed cluster name; empty for non-Distributed findings
	Reason  string
}

ValidationError describes a dependency that could not be satisfied.

func Validate

func Validate(dbs []DatabaseSpec, skip SkipSet, clusters ClusterSet) []ValidationError

Validate checks that every dependency carried by the loaded databases is satisfied: the referenced table or materialized view must be declared in one of the loaded databases. A reference into a database that is not loaded is itself an error, since it cannot be checked. Dependent objects matched by skip are not validated. All errors are collected and returned together.

Distributed remotes are resolved cluster-aware against clusters: a proxy routinely forwards to a storage table that lives on another cluster's composition (same database name, different cluster). See resolveDistributedRemote for the routing rules. Materialized-view and plain- view sources are likewise resolved against the union of mapped clusters when missing locally (a co-located read carries no cluster name). Pass ClusterSet{} when no external cluster mappings are available.

Once a Distributed remote resolves to an inspectable table, the proxy's columns are checked against it (subset by default; see ValidateOptions).

func ValidateOpts

func ValidateOpts(dbs []DatabaseSpec, skip SkipSet, clusters ClusterSet, opts ValidateOptions) []ValidationError

ValidateOpts is Validate with explicit options.

func (ValidationError) Error

func (e ValidationError) Error() string

type ViewDiff

type ViewDiff struct {
	Name        string
	QueryChange *StringChange
	Comment     *StringChange
	Recreate    bool

	// RecreateChanged names the attributes that forced Recreate, in a fixed
	// order, so consumers can report which one moved.
	RecreateChanged []string
}

ViewDiff is the set of mutations to a single existing plain view. A query-only change is applied in place via ALTER TABLE ... MODIFY QUERY. A comment-only change is applied via ALTER TABLE ... MODIFY COMMENT. Any change to ColumnAliases / SQLSecurity / Definer / Cluster requires recreating the view (DROP + CREATE) and is flagged unsafe.

func (ViewDiff) IsEmpty

func (vd ViewDiff) IsEmpty() bool

func (ViewDiff) IsUnsafe

func (vd ViewDiff) IsUnsafe() bool

type ViewSpec

type ViewSpec struct {
	Name          string   `hcl:"name,label"`
	Query         string   `hcl:"query"`
	ColumnAliases []string `hcl:"column_aliases,optional"`
	SQLSecurity   *string  `hcl:"sql_security,optional"`
	Definer       *string  `hcl:"definer,optional"`
	Cluster       *string  `hcl:"cluster,optional"`
	Comment       *string  `hcl:"comment,optional"`
}

ViewSpec models a ClickHouse plain (non-materialized) view: a saved SELECT executed on every read of the view. The Query is stored verbatim as text. Live views, refreshable materialized views, and window views are unsupported and rejected with a clear error during introspection (they parse into different AST types).

SQLSecurity is the canonical lowercase form of the SQL SECURITY clause: one of "definer", "invoker", or "none". Definer is the user the view runs as; required iff SQLSecurity == "definer" with a named user. The literal "current_user" is accepted unquoted.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL