Documentation
¶
Index ¶
- Constants
- func ApplySQL(schema *Schema, sql, defaultDatabase string, allowRaw bool) (int, error)
- func BeautifySQL(sql string) (string, bool)
- func FilterSchema(s *Schema, m *ExcludeMatcher)
- func InferExternalNamedCollections(s *Schema) []string
- func IntrospectMacros(ctx context.Context, conn driver.Conn) (map[string]string, error)
- func IsVirtualColumn(e Engine, name string) bool
- func LayerFiles(path string) ([]string, error)
- func MatchesColumnPattern(pattern, database, table, name string) bool
- func MatchesPattern(pattern, database, name string) bool
- func RedactedParamKeys(nc NamedCollectionSpec) []string
- func RenderDiffJSON(cs ChangeSet, gen GeneratedSQL, left, right *Schema) ([]byte, error)
- func RenderObjectComparisons(w io.Writer, objs []ObjectComparison)
- func RenderObjectHCL(database, kind, name string, db *DatabaseSpec) (string, error)
- func RenderObjectSQL(database, kind, name string, db *DatabaseSpec) (string, error)
- func Resolve(s *Schema) error
- func SelectSchema(s *Schema, m *ExcludeMatcher)
- func Write(w io.Writer, schema *Schema) error
- func WriteLayer(w io.Writer, schema *Schema) error
- type ChangeSet
- type ClusterSet
- type ColumnChange
- type ColumnSpec
- type CompareSummary
- type ConstraintChange
- type ConstraintSpec
- type DatabaseChange
- type DatabaseSpec
- type Declaration
- type DeclaredColumn
- type Dependency
- type DictionaryAttribute
- type DictionaryDiff
- type DictionaryLayout
- type DictionaryLayoutSpec
- type DictionaryLifetime
- type DictionaryRange
- type DictionarySource
- type DictionarySourceSpec
- type DictionarySpec
- type DiffJSON
- type DiffOptions
- type DriftGroup
- type DriftJSON
- type DriftNode
- type DriftRunSummary
- type DuplicateGroup
- type Engine
- type EngineAggregatingMergeTree
- type EngineBuffer
- type EngineChange
- type EngineCollapsingMergeTree
- type EngineDistributed
- type EngineJoin
- type EngineKafka
- type EngineLog
- type EngineMemory
- type EngineMerge
- type EngineMergeTree
- type EngineNull
- type EngineReplacingMergeTree
- type EngineReplicatedAggregatingMergeTree
- type EngineReplicatedCollapsingMergeTree
- type EngineReplicatedMergeTree
- type EngineReplicatedReplacingMergeTree
- type EngineReplicatedSummingMergeTree
- type EngineSharedAggregatingMergeTree
- type EngineSharedCollapsingMergeTree
- type EngineSharedMergeTree
- type EngineSharedReplacingMergeTree
- type EngineSharedSummingMergeTree
- type EngineSpec
- type EngineSummingMergeTree
- type EngineTimeSeries
- type EngineWithDynamicVirtuals
- type EngineWithVirtuals
- type ExcludeMatcher
- type FieldChange
- type GeneratedSQL
- type IndexSpec
- type JSONOperation
- type JSONUnsafe
- type LayoutCache
- type LayoutComplexKeyCache
- type LayoutComplexKeyDirect
- type LayoutComplexKeyHashed
- type LayoutComplexKeyHashedArray
- type LayoutComplexKeyRangeHashed
- type LayoutComplexKeySparseHashed
- type LayoutDirect
- type LayoutFlat
- type LayoutHashed
- type LayoutHashedArray
- type LayoutIPTrie
- type LayoutRangeHashed
- type LayoutRegexpTree
- type LayoutSparseHashed
- type MaterializedViewDiff
- type MaterializedViewSpec
- type NamedCollectionChange
- type NamedCollectionParam
- type NamedCollectionSpec
- type NodeSpec
- type ObjectComparison
- type ObjectRef
- type Operation
- type OrderByChange
- type PatchColumnSpec
- type PatchDictionarySpec
- type PatchMaterializedViewSpec
- type PatchTableSpec
- type PatchViewSpec
- type PlanOperation
- type PlanResult
- type ProjectionSpec
- type RawChange
- type RawSpec
- type RenameColumn
- type RoleComparison
- type RoleDiff
- type Schema
- type SettingChange
- type SkipSet
- type SourceClickHouse
- type SourceExecutable
- type SourceFile
- type SourceHTTP
- type SourceMySQL
- type SourceNull
- type SourcePostgreSQL
- type StringChange
- type TableDiff
- type TableResolver
- type TableSpec
- type TimeSeriesInnerTable
- type TimeSeriesTarget
- type UnsafeChange
- type ValidateOptions
- type ValidationError
- type ViewDiff
- type ViewSpec
Constants ¶
const ( StatusAdded = "added" StatusDropped = "dropped" StatusAltered = "altered" )
Comparison statuses. Status is right-relative: "added" means the right side of the Diff has the object and the left does not.
const ( KindTable = "table" KindMaterializedView = "materialized_view" KindView = "view" KindDictionary = "dictionary" KindRaw = "raw" )
Object kinds accepted by RenderObjectSQL and RenderObjectHCL. They match the segment used in the web UI's object routes.
const ( OpCreate = "CREATE" OpAlter = "ALTER" OpDrop = "DROP" OpRename = "RENAME" )
Operation kinds.
const ( DepMVSource = "mv_source" // a materialized view reads from this table DepMVDest = "mv_dest" // a materialized view writes into this table DepDistributedRemote = "distributed_remote" // a Distributed table forwards to this table DepTimeSeriesTarget = "ts_target" // a TimeSeries table references an external samples/tags/metrics target DepBufferDestination = "buffer_dest" // a Buffer table forwards writes into this table DepViewSource = "view_source" // a plain view reads from this table // KindMVColumn flags a materialized view that references a column its // single source table does not provide (declared columns plus the // engine's virtual columns). This is a heuristic check, restricted to // virtual-prefixed names (those starting with `_`) and to MVs with a // single resolvable source and no JOIN/CTE/UNION/subquery/SELECT *. KindMVColumn = "mv_column" // KindDistributedColumn flags a Distributed table whose columns disagree // with its resolved remote storage table: a column the proxy declares is // absent from the remote, or its type/nullability differs. In strict mode // it also flags remote columns the proxy omits. KindDistributedColumn = "distributed_column" )
Dependency kinds.
const KindNamedCollection = "named_collection"
KindNamedCollection is the object_type for named collections; the other object_type values reuse the Kind* constants in render.go (table, materialized_view, view, dictionary, raw).
const RedactedValue = "[HIDDEN]"
RedactedValue is the literal string ClickHouse substitutes for a secret when the connecting user lacks displaySecretsInShowAndSelect / the server lacks display_secrets_in_show_and_select / a query omits format_display_secrets_in_show_and_select. It shows up in system.named_collections values and in a dictionary's create_table_query SOURCE(...) credentials.
It is kept in-band: introspection stores the marker as the value, so it round-trips through an HCL dump and every comparison — live, dump file, or drift — can tell "this secret is unknown to hclexp" apart from "this object has no secret". The diff layer treats it as unknown ("don't generate a change") rather than as a literal target, and sqlgen refuses to emit any statement containing it: writing the string "[HIDDEN]" over a real cluster credential would be actively destructive.
Authored production HCL should use an external named collection instead of carrying either a real secret or this marker.
Variables ¶
This section is empty.
Functions ¶
func ApplySQL ¶
ApplySQL parses ClickHouse DDL and applies each statement, in order, to schema. It is the engine behind `hclexp sql2hcl`: developers express a change as the SQL they already know and it is folded into their declarative HCL.
Supported statements (declarative schema changes only):
- CREATE TABLE | MATERIALIZED VIEW | VIEW | DICTIONARY — adds the object, or replaces an existing object of the same name.
- ALTER TABLE … ADD/DROP/MODIFY/RENAME COLUMN, ADD/DROP INDEX, MODIFY/REMOVE TTL, MODIFY/RESET SETTING — edits the matching table block.
- ALTER TABLE <mv> MODIFY QUERY … — replaces a materialized view's query.
- DROP TABLE | VIEW | DICTIONARY | DATABASE — removes the object.
- RENAME TABLE a TO b — renames (and moves across databases).
Data and partition operations (TRUNCATE, ALTER … DELETE, attach/detach/drop/ freeze/replace partition, materialize index/projection) carry no declarative schema meaning and are rejected with a clear error.
defaultDatabase supplies the database for unqualified object names. When it is empty and the schema has exactly one database, that database is used.
allowRaw mirrors `introspect`: a CREATE the schema model cannot express is captured verbatim as a raw{} block instead of failing. (A statement that does not even parse aborts the whole call — per-statement isolation is impossible once the parser has rejected the input.)
It returns the number of statements applied.
func BeautifySQL ¶
BeautifySQL parses a single SQL statement and returns it re-rendered in the parser's beautified (indented, multi-line) form — the same visitor that produces readable view/MV queries elsewhere. It returns ok=false with the input unchanged when the statement can't be parsed, so callers can fall back to the verbatim SQL (e.g. for DDL the parser doesn't yet handle).
func FilterSchema ¶
func FilterSchema(s *Schema, m *ExcludeMatcher)
FilterSchema removes every object the matcher excludes, in place. Both sides of a comparison are filtered before Diff so excluded objects appear in no output and no count. Databases are kept even when emptied.
func InferExternalNamedCollections ¶ added in v0.3.0
InferExternalNamedCollections declares collection references that are missing from a captured schema as external. Live ClickHouse users can be allowed to use config-backed named collections without being allowed to enumerate them through system.named_collections. In that case introspection still sees the collection name in table/dictionary DDL, but cannot capture its values or an explicit declaration.
This is intentionally not part of Resolve: authored schemas must continue to declare every collection explicitly. Callers use it only at the live/dump boundary, where a reference is evidence that the collection is provisioned outside the captured schema. The returned names are the declarations added, sorted for deterministic diagnostics and dumps.
func IntrospectMacros ¶
IntrospectMacros returns the ClickHouse macros configured on the connected node, read from system.macros. Macros carry a node's identity within the cluster — shard, replica, hostClusterRole, hostClusterType — substituted into ON CLUSTER DDL and ReplicatedMergeTree zoo paths. The returned map is keyed by macro name.
func IsVirtualColumn ¶
IsVirtualColumn reports whether name is one of the implicit columns the engine contributes. Used as a name guard in the diff path; mode- and settings-independent on purpose — Kafka's stream-mode extras are included in the membership test so a column never silently shifts status when a setting changes.
The bare Nested parent (e.g. "_headers" for Kafka) is also recognised in addition to the dot-access form ("_headers.name").
func LayerFiles ¶
LayerFiles returns the .hcl files a layer path contributes, in load order: for a directory, every *.hcl inside it (lexical filename order); for a regular file, the file itself, which must have the .hcl extension so a stack entry pointing at a dump or a .sql script fails loudly rather than being parsed as HCL.
func MatchesColumnPattern ¶ added in v0.3.0
MatchesColumnPattern matches a column by bare name, table.column, or database.table.column. It uses the same filepath.Match glob syntax as MatchesPattern.
func MatchesPattern ¶
MatchesPattern reports whether a locate pattern (a filepath.Match glob, or a plain name) matches an object, trying the bare name and the qualified "<database>.<name>" form — the same convention exclude patterns use.
func RedactedParamKeys ¶
func RedactedParamKeys(nc NamedCollectionSpec) []string
RedactedParamKeys returns the param keys whose introspected value is the literal "[HIDDEN]" placeholder ClickHouse returns when the user lacks displaySecretsInShowAndSelect / the server-side display_secrets_in_show_and_select is off. Returns an empty slice when the spec is fully un-redacted. Callers use this to warn operators that the diff cannot be trusted for those specific params.
func RenderDiffJSON ¶
func RenderDiffJSON(cs ChangeSet, gen GeneratedSQL, left, right *Schema) ([]byte, error)
RenderDiffJSON builds the structured, dependency-ordered operation list from a GenerateSQL result. The operation order is the index in the already dependency-sorted statement list. Engine/replicated are looked up from the resolved schemas — the target (right) for CREATE/ALTER, falling back to the current (left) for DROP — because an ALTER that does not change the engine carries no engine information of its own. Unsafe flags come from the existing gen.Unsafe list (no separate diff path), matched by database + object.
func RenderObjectComparisons ¶
func RenderObjectComparisons(w io.Writer, objs []ObjectComparison)
RenderObjectComparisons prints comparisons as the indented, +/-/~ marked summary used by `diff` (text mode) and `drift -details`. Objects render in input order under a header per database; named collections (empty database) render under a "named_collections" header. It consumes the same []ObjectComparison the JSON emits, so text and JSON cannot disagree.
func RenderObjectHCL ¶
func RenderObjectHCL(database, kind, name string, db *DatabaseSpec) (string, error)
RenderObjectHCL returns the canonical HCL for a single object named name of the given kind within db, using the same writers as the schema dumper. It returns an error if the kind is unknown or no such object exists.
func RenderObjectSQL ¶
func RenderObjectSQL(database, kind, name string, db *DatabaseSpec) (string, error)
RenderObjectSQL returns the ClickHouse CREATE DDL for a single object named name of the given kind within db. It reuses the same generators that GenerateSQL drives, so the output matches what a fresh `diff -sql` would emit. It returns an error if the kind is unknown or no such object exists.
func Resolve ¶
Resolve walks each database, applies object patches, resolves extend chains, drops abstract tables and materialized views, and validates the remaining objects. All mutation happens in place on the supplied slice.
func SelectSchema ¶
func SelectSchema(s *Schema, m *ExcludeMatcher)
SelectSchema keeps only the objects the matcher matches, in place — the inverse of FilterSchema, for "emit only these objects" selections (load -only). An empty matcher selects nothing to keep... which would erase the schema, so it is treated as "no selection" and leaves the schema unchanged; callers pass a matcher only when a selector was actually given. Databases are kept even when emptied (a split layer still needs the database{} wrapper and its cluster default); node blocks are untouched.
func Write ¶
Write emits dbs as canonical HCL. Tables are sorted alphabetically; columns, indexes, and engine fields keep their structural order. Settings entries are sorted by key.
The dumper assumes the input has already been resolved: extend/abstract/ override are consumed, patches applied, engines decoded. Fields tagged diff:"-" in the type definitions are intentionally never emitted.
func WriteLayer ¶
WriteLayer emits an unresolved authored layer. Unlike Write, it preserves patch blocks so generated decompositions can be loaded back through LoadLayers and Resolve. Declarations use the same canonical writers as dumps; patch collections are sorted by target name for deterministic output.
Types ¶
type ChangeSet ¶
type ChangeSet struct {
Databases []DatabaseChange
NamedCollections []NamedCollectionChange
}
ChangeSet describes the changes required to evolve a `from` schema into a `to` schema. Empty databases (no tables to add/drop/alter) are omitted.
func Diff ¶
Diff compares two resolved schemas and returns a deterministic ChangeSet. Both inputs must already have been resolved (engines decoded, abstracts dropped, extend/patches consumed).
func DiffWithOptions ¶
func DiffWithOptions(from, to *Schema, options DiffOptions) ChangeSet
DiffWithOptions compares two resolved schemas using the supplied semantic policy. Both inputs must already have been resolved (engines decoded, abstracts dropped, extend/patches consumed).
type ClusterSet ¶
type ClusterSet struct {
// contains filtered or unexported fields
}
ClusterSet maps external cluster names to the schema objects each cluster declares. It lets Validate resolve a Distributed table's remote against the composition of *another* cluster: the remote database is almost always "posthog" on every cluster, so the cluster name is the only discriminator.
A cluster is either mapped (a composition was loaded for it), absent (the env has no local composition for it, e.g. a node type with no instance here), or an alias of another cluster. References into an absent cluster are structurally unresolvable and count as satisfied; a ClickHouse remote_servers alias (e.g. "posthog_writable") shares the composition of its base cluster, so it resolves through that base. The zero value is a valid empty set.
func NewClusterSet ¶
func NewClusterSet() ClusterSet
NewClusterSet returns an empty, initialized ClusterSet.
func (*ClusterSet) Add ¶
func (cs *ClusterSet) Add(name string, dbs []DatabaseSpec)
Add records the composition of an external cluster: every table, view, materialized view, and raw object it declares becomes resolvable under name.
func (*ClusterSet) AddAbsent ¶
func (cs *ClusterSet) AddAbsent(name string)
AddAbsent declares a cluster that has no composition available in this env. References into it are treated as satisfied (see the resolution algorithm in Validate).
func (*ClusterSet) AddAlias ¶
func (cs *ClusterSet) AddAlias(alias, base string)
AddAlias records that alias resolves to base: a Distributed remote on the alias cluster is resolved against base's composition (or base's @absent status). base itself may be another alias; resolveCluster follows the chain.
type ColumnChange ¶
type ColumnChange struct {
Name string
Old ColumnSpec
New ColumnSpec
}
ColumnChange is an in-name modification of a column: its type and/or any modifier (default/materialized/ephemeral/alias/codec/ttl/comment/nullable) differs between Old and New. Name is the column's (current) name.
func (ColumnChange) IsUnsafe ¶
func (c ColumnChange) IsUnsafe() bool
IsUnsafe reports whether the change switches the column's storage class — into or out of ALIAS / MATERIALIZED / EPHEMERAL. ClickHouse accepts such a MODIFY COLUMN but it is data-affecting (e.g. plain → ALIAS silently replaces stored values with the computed expression), so it is never auto-emitted. Changes within the same kind, or plain ↔ DEFAULT, plus codec/comment/ttl/ nullable/type changes, are in-place safe.
type ColumnSpec ¶
type ColumnSpec struct {
Name string `hcl:"name,label"`
Type string `hcl:"type"`
// Nullable wraps Type in Nullable(...) at SQL gen time. Combining
// Nullable = true with a Type that already starts with "Nullable("
// is rejected, matching ClickHouse's own behavior.
Nullable bool `hcl:"nullable,optional"`
// Default / Materialized / Ephemeral / Alias are mutually exclusive
// default-value expressions, matching ClickHouse's
// DEFAULT|MATERIALIZED|EPHEMERAL|ALIAS clauses. Ephemeral can be the
// empty string, meaning a bare EPHEMERAL with no expression.
Default *string `hcl:"default,optional"`
Materialized *string `hcl:"materialized,optional"`
Ephemeral *string `hcl:"ephemeral,optional"`
Alias *string `hcl:"alias,optional"`
Comment *string `hcl:"comment,optional"`
Codec *string `hcl:"codec,optional"`
TTL *string `hcl:"ttl,optional"`
// RenamedFrom declares that this column previously existed under another
// name. The diff engine uses it to emit RENAME COLUMN instead of DROP +
// ADD. Tagged diff:"-" because it's transient metadata, not part of the
// schema being compared.
RenamedFrom *string `hcl:"renamed_from,optional" diff:"-"`
// After / First position a patch_table column add: the new column is
// inserted immediately after the named column (or at the front) instead
// of appended, so an env's mid-table extras can interleave without
// redeclaring the table (#158). Patch-only — a declared table or MV
// column carrying either is a resolve error, and modify_column rejects
// them (repositioning base columns is genuine drift). Consumed and
// cleared at patch application; the diff engine may also populate these
// fields on its private AddColumns copies so SQL generation preserves the
// target order. Tagged diff:"-" as transient placement metadata, like
// RenamedFrom.
After *string `hcl:"after,optional" diff:"-"`
First bool `hcl:"first,optional" diff:"-"`
}
type CompareSummary ¶
type CompareSummary struct {
TablesAdded int `json:"tables_added"`
TablesDropped int `json:"tables_dropped"`
TablesAltered int `json:"tables_altered"`
MVsAdded int `json:"mvs_added"`
MVsDropped int `json:"mvs_dropped"`
MVsAltered int `json:"mvs_altered"`
ViewsAdded int `json:"views_added"`
ViewsDropped int `json:"views_dropped"`
ViewsAltered int `json:"views_altered"`
DictsAdded int `json:"dicts_added"`
DictsDropped int `json:"dicts_dropped"`
DictsAltered int `json:"dicts_altered"`
RawsAdded int `json:"raws_added"`
RawsDropped int `json:"raws_dropped"`
RawsAltered int `json:"raws_altered"`
NamedCollectionsChanged int `json:"named_collections_changed"`
}
CompareSummary counts comparisons by object type and status.
func SummarizeComparisons ¶
func SummarizeComparisons(objs []ObjectComparison) CompareSummary
SummarizeComparisons counts comparisons by object type and status.
func (CompareSummary) OneLiner ¶
func (s CompareSummary) OneLiner() string
OneLiner renders the summary as the compact drift line, e.g. "+1 table, ~2 mv, ~1 raw". An all-zero summary falls back to "changed".
type ConstraintChange ¶
type ConstraintChange struct {
Name string
Old ConstraintSpec
New ConstraintSpec
}
ConstraintChange is an in-name modification of a table constraint: its CHECK/ASSUME predicate differs between Old and New. ClickHouse has no MODIFY CONSTRAINT, so it is emitted as DROP CONSTRAINT + ADD CONSTRAINT.
type ConstraintSpec ¶
type ConstraintSpec struct {
Name string `hcl:"name,label"`
Check *string `hcl:"check,optional"`
Assume *string `hcl:"assume,optional"`
}
ConstraintSpec is a table-level constraint. Either Check or Assume must be set, but not both. Check enforces a row-level boolean predicate on INSERT; Assume is an optimizer hint (the constraint is presumed true).
type DatabaseChange ¶
type DatabaseChange struct {
Database string
AddTables []TableSpec // emitted via CREATE TABLE
DropTables []TableSpec // emitted via DROP TABLE
AlterTables []TableDiff
AddMaterializedViews []MaterializedViewSpec // emitted via CREATE MATERIALIZED VIEW
DropMaterializedViews []string // emitted via DROP VIEW
AlterMaterializedViews []MaterializedViewDiff
AddDictionaries []DictionarySpec // emitted via CREATE OR REPLACE DICTIONARY
DropDictionaries []string // emitted via DROP DICTIONARY
AlterDictionaries []DictionaryDiff
AddViews []ViewSpec // emitted via CREATE VIEW
DropViews []string // emitted via DROP VIEW
AlterViews []ViewDiff
AddRaws []RawSpec // emitted verbatim via the stored CREATE DDL
DropRaws []RawSpec // emitted via a kind-mapped DROP
AlterRaws []RawChange // emitted as DROP + CREATE (recreate)
}
DatabaseChange holds the per-database differences.
func (DatabaseChange) IsEmpty ¶
func (dc DatabaseChange) IsEmpty() bool
type DatabaseSpec ¶
type DatabaseSpec struct {
Name string `hcl:"name,label"`
// Cluster is the default ON CLUSTER target for every table declared in
// this database. Tables can override via TableSpec.Cluster. Inheritance
// happens during resolution; after Resolve, every TableSpec.Cluster
// reflects the effective value.
Cluster *string `hcl:"cluster,optional"`
Tables []TableSpec `hcl:"table,block"`
Patches []PatchTableSpec `hcl:"patch_table,block" diff:"-"`
// ViewPatches and DictionaryPatches are the patch_view /
// patch_dictionary counterparts of Patches. Like table patches they
// accumulate across layers and are consumed during resolution.
MaterializedViewPatches []PatchMaterializedViewSpec `hcl:"patch_materialized_view,block" diff:"-"`
ViewPatches []PatchViewSpec `hcl:"patch_view,block" diff:"-"`
DictionaryPatches []PatchDictionarySpec `hcl:"patch_dictionary,block" diff:"-"`
// MaterializedViews are a sibling collection to Tables, not a flavored
// TableSpec. They may `extend` an `abstract = true` table to inherit
// its column list (and optionally cluster/comment).
MaterializedViews []MaterializedViewSpec `hcl:"materialized_view,block"`
// Dictionaries are a third sibling collection. Like MVs, they do not
// participate in the table inheritance system.
Dictionaries []DictionarySpec `hcl:"dictionary,block"`
// Views are plain (non-materialized) views. Like MVs and dictionaries
// they live as a sibling collection and do not participate in the
// table inheritance system.
Views []ViewSpec `hcl:"view,block"`
// Raws are escape-hatch objects whose CREATE DDL could not be parsed
// or expressed in this schema language. They are stored verbatim,
// round-tripped unchanged, and recreated (DROP+CREATE) on change.
Raws []RawSpec `hcl:"raw,block"`
}
func Introspect ¶
func Introspect(ctx context.Context, conn driver.Conn, database string, allowRaw bool) (*DatabaseSpec, error)
Introspect builds a DatabaseSpec from a live ClickHouse database. The returned spec is "flat" — no extend/abstract/patch_table — and every TableSpec has its Engine.Decoded populated. Tables and column order match the declaration order ClickHouse reports.
The function reads `create_table_query` for each table and parses it with a ClickHouse SQL parser (orian/clickhouse-sql-parser). All per-table data is derived from that AST: columns (including TTL, CODEC, default/materialized/ephemeral/alias, comment), indexes, constraints, engine, ORDER BY, PARTITION BY, SAMPLE BY, table TTL, SETTINGS, and table comment.
func IntrospectWithExclude ¶
func IntrospectWithExclude(ctx context.Context, conn driver.Conn, database string, allowRaw bool, exclude *ExcludeMatcher) (*DatabaseSpec, error)
IntrospectWithExclude is Introspect with an optional exclude matcher: objects whose name matches a pattern are skipped before their DDL is parsed, so transient tables (e.g. ClickHouse's _tmp_replace_*, migration tmp_* tables) neither appear in the dump nor abort introspection when their DDL can't be parsed. A nil matcher excludes nothing.
type Declaration ¶
type Declaration struct {
ObjectType string // KindTable, KindMaterializedView, KindView, KindDictionary, KindRaw, KindNamedCollection
Database string // empty for named collections (cluster-scoped)
Name string
File string
Line int
Abstract bool
Override bool
Patch bool // declared via patch_table (strictly additive)
Extends string // the extend = "<parent>" attribute, when present
RawKind string // raw blocks only: the kind label
}
Declaration is one declaration site of a schema object in an HCL source file. It comes from a lightweight syntax scan — no gohcl decode, no Resolve — which is what preserves the source position and the raw inheritance flags (abstract/override/extend/patch_table) that resolution consumes and the resolved specs no longer carry.
func ScanDeclarations ¶
func ScanDeclarations(files []string) ([]Declaration, error)
ScanDeclarations records every object declaration site in the given files, in file order. Files must be native HCL syntax, same as the loader; a parse error aborts the scan.
func ScanFileDeclarations ¶
func ScanFileDeclarations(path string) ([]Declaration, string, error)
ScanFileDeclarations scans one file and additionally returns the label of its first node{} block ("" when the file has none). Per-node dumps carry exactly one node block, so the name attributes the file's declarations to their host.
type DeclaredColumn ¶
DeclaredColumn is a single column from the object declared in a CREATE statement: name + ClickHouse type expression as plain text. Used by tests that need to materialize stub source tables.
func ColumnsProvidedBy ¶
func ColumnsProvidedBy(t TableSpec, r TableResolver) []DeclaredColumn
ColumnsProvidedBy returns every column readable from t: declared columns first (in declaration order), then engine-contributed virtual columns the user has not shadowed by declaring a same-named column.
Declared wins: a user-written `column "_key" { ... }` on a Kafka table (e.g. via kafka_map_virtual_columns_on_write) appears in the output and the virtual is suppressed.
func ExtractDeclaredColumns ¶
func ExtractDeclaredColumns(createSQL string) ([]DeclaredColumn, error)
ExtractDeclaredColumns parses a CREATE statement and returns the columns declared on the object being created:
- CREATE TABLE / VIEW with explicit columns: the column list
- CREATE MATERIALIZED VIEW: the destination column list (TO dest (…))
- CREATE DICTIONARY: the attribute list (name+type only)
Returns an empty slice (no error) when the statement carries no column list (e.g. CREATE VIEW v AS SELECT … without an explicit schema).
func VirtualColumnsFor ¶
func VirtualColumnsFor(e Engine, r TableResolver) []DeclaredColumn
VirtualColumnsFor returns the engine-defined implicit columns for e, or nil if the engine has none / e is nil. Dispatch is by interface assertion, not by Kind() switch — adding a new engine with virtuals only requires implementing one of the optional interfaces.
type Dependency ¶
type Dependency struct {
From ObjectRef // the dependent object (a materialized view or Distributed table)
To ObjectRef // the object it requires
Kind string // one of the Dep* constants
// Cluster is the ClickHouse cluster the remote is expected on. It is set
// only for DepDistributedRemote (from EngineDistributed.ClusterName) and
// is the discriminator for cross-cluster resolution: the remote database
// is almost always "posthog" on every cluster. Empty for all other kinds.
Cluster string
}
Dependency records that one object (From) requires another (To) to exist before it can be created.
func CollectDependencies ¶
func CollectDependencies(dbs []DatabaseSpec) ([]Dependency, error)
CollectDependencies walks every database and returns the creation-order dependencies carried by materialized views (source + destination tables) and Distributed-engine tables (their remote table).
type DictionaryAttribute ¶
type DictionaryAttribute struct {
Name string `hcl:"name,label"`
Type string `hcl:"type"`
Default *string `hcl:"default,optional"`
Expression *string `hcl:"expression,optional"`
Hierarchical bool `hcl:"hierarchical,optional"`
Injective bool `hcl:"injective,optional"`
IsObjectID bool `hcl:"is_object_id,optional"`
}
type DictionaryDiff ¶
type DictionaryDiff struct {
Name string
Changed []string
// Old is the source spec. Dictionary reconciliation rewrites the whole
// object, so sqlgen needs the old source as well as the target to detect a
// credential that exists on the live object but was redacted to [HIDDEN].
// Without it, a target that omits the credential looks safe even though
// CREATE OR REPLACE would silently remove the unknown live value (#178).
Old DictionarySpec
// New is the target spec. The CREATE OR REPLACE DICTIONARY that
// reconciles the change is rendered from it.
New DictionarySpec
// SkippedRedactedSecrets names source fields (the DDL argument name, e.g.
// "password") whose comparison was suppressed because one side holds the
// RedactedValue marker and the other a real value. hclexp cannot tell
// whether they differ, so it says so rather than guessing. Mirrors
// NamedCollectionChange.SkippedRedactedParams.
SkippedRedactedSecrets []string
}
DictionaryDiff describes a change to a dictionary. ClickHouse has no useful in-place ALTER DICTIONARY, so a change is normally materialized as CREATE OR REPLACE DICTIONARY. Replacing the object is non-destructive, but sqlgen still refuses a rewrite when a redacted source credential cannot be preserved. Changed lists field paths that differ, for rendering.
func (DictionaryDiff) IsEmpty ¶
func (d DictionaryDiff) IsEmpty() bool
IsEmpty counts a skipped secret as a difference: a dictionary whose only divergence is an unverifiable credential is reported (with no DDL) rather than certified equal.
func (DictionaryDiff) IsUnsafe ¶
func (d DictionaryDiff) IsUnsafe() bool
type DictionaryLayout ¶
type DictionaryLayout interface{ Kind() string }
func DecodeDictionaryLayout ¶
func DecodeDictionaryLayout(spec *DictionaryLayoutSpec) (DictionaryLayout, error)
DecodeDictionaryLayout dispatches on spec.Kind and decodes the body into the matching typed layout struct. Returns (nil, nil) when spec is nil.
type DictionaryLayoutSpec ¶
type DictionaryLayoutSpec struct {
Kind string `hcl:"kind,label" diff:"-"`
Body hcl.Body `hcl:",remain" diff:"-"`
Decoded DictionaryLayout
}
type DictionaryLifetime ¶
DictionaryLifetime models LIFETIME(...). The simple form LIFETIME(n) sets only Min (Max remains nil). The range form LIFETIME(MIN x MAX y) sets both.
type DictionaryRange ¶
type DictionarySource ¶
type DictionarySource interface{ Kind() string }
func DecodeDictionarySource ¶
func DecodeDictionarySource(spec *DictionarySourceSpec) (DictionarySource, error)
DecodeDictionarySource dispatches on spec.Kind and decodes the body into the matching typed source struct. Returns (nil, nil) when spec is nil.
type DictionarySourceSpec ¶
type DictionarySourceSpec struct {
Kind string `hcl:"kind,label" diff:"-"`
Body hcl.Body `hcl:",remain" diff:"-"`
Decoded DictionarySource
}
DictionarySourceSpec mirrors EngineSpec: Kind label + opaque hcl.Body + typed Decoded value. Decoded is populated by DecodeDictionarySource after the initial gohcl decode; Kind and Body are diff-skipped artifacts.
type DictionarySpec ¶
type DictionarySpec struct {
Name string `hcl:"name,label"`
Override bool `hcl:"override,optional" diff:"-"`
PrimaryKey []string `hcl:"primary_key"`
Attributes []DictionaryAttribute `hcl:"attribute,block"`
Source *DictionarySourceSpec `hcl:"source,block"`
Layout *DictionaryLayoutSpec `hcl:"layout,block"`
Lifetime *DictionaryLifetime `hcl:"lifetime,block"`
Range *DictionaryRange `hcl:"range,block"`
Settings map[string]string `hcl:"settings,optional"`
Cluster *string `hcl:"cluster,optional"`
Comment *string `hcl:"comment,optional"`
}
DictionarySpec models a ClickHouse dictionary. Dictionaries are a sibling collection on DatabaseSpec, not a flavored table; they do not participate in the table inheritance system (extend/abstract/patch_table). Source and layout are modeled typed-per-kind via DictionarySource/DictionaryLayout — the same pattern engines use.
type DiffJSON ¶
type DiffJSON struct {
Objects []ObjectComparison `json:"objects"`
Operations []JSONOperation `json:"operations"`
Unsafe []JSONUnsafe `json:"unsafe,omitempty"`
Summary CompareSummary `json:"summary"`
}
DiffJSON is the top-level document emitted by `hclexp diff -format json`. Objects is the per-object comparison view; Operations is the flat, dependency-ordered execution view of the same diff. Summary counts derive from Objects.
type DiffOptions ¶
type DiffOptions struct {
IgnoreColumnOrder bool
}
DiffOptions controls comparison semantics shared by diff, drift, and web. Column order is significant by default because it changes SELECT * output and must be preserved by a convergent migration. Callers may explicitly ignore it when node roles or operational policy do not require uniform physical ordering.
type DriftGroup ¶
type DriftGroup struct {
Key string `json:"key"`
Reference string `json:"reference"`
Nodes int `json:"nodes"`
Drifters []DriftNode `json:"drifters"`
}
DriftGroup is one set of nodes expected to share a schema.
type DriftJSON ¶
type DriftJSON struct {
Groups []DriftGroup `json:"groups"`
Summary DriftRunSummary `json:"summary"`
}
DriftJSON is the document emitted by `hclexp drift -format json`.
type DriftNode ¶
type DriftNode struct {
Node string `json:"node"`
File string `json:"file"`
Macros map[string]string `json:"macros,omitempty"`
Objects []ObjectComparison `json:"objects"`
Summary CompareSummary `json:"summary"`
}
DriftNode is one drifting node's comparison against its group reference.
type DriftRunSummary ¶
type DriftRunSummary struct {
Nodes int `json:"nodes"`
Groups int `json:"groups"`
GroupsWithDrift int `json:"groups_with_drift"`
DriftingNodes int `json:"drifting_nodes"`
}
DriftRunSummary aggregates a whole drift run.
type DuplicateGroup ¶
type DuplicateGroup struct {
Database string
Name string
Declarations []Declaration // every site for the name, legitimate ones included
}
DuplicateGroup is one object name defined at more than one site without the composition system explaining the extras: patch sites are additive, override = true is a deliberate replacement, and extend-carrying sites refine an inherited definition. Abstract declarations still define schema and therefore count. A group qualifies when at least two definition sites remain.
func FindDuplicates ¶
func FindDuplicates(decls []Declaration) []DuplicateGroup
FindDuplicates groups declarations by (database, name) — the ClickHouse namespace, which object types share — and returns groups holding two or more definition sites: declarations that are not patches or overrides and do not carry extend. Every declaration is retained in a reported group's output. Results are sorted by database then name.
type Engine ¶
type Engine interface {
Kind() string
}
Engine is the typed, decoded form of an engine block. Each ClickHouse engine kind is a distinct concrete type. Implementations are value types.
func DecodeEngine ¶
func DecodeEngine(spec *EngineSpec) (Engine, error)
DecodeEngine dispatches on spec.Kind and decodes the body into a kind-specific struct. Returns (nil, nil) when spec is nil.
func ParseEngineString ¶
ParseEngineString parses a ClickHouse engine_full value into a typed HCL Engine. Strips the trailing ORDER BY / PARTITION BY / SAMPLE BY / TTL / SETTINGS / COMMENT clauses, then dispatches on the engine name.
type EngineAggregatingMergeTree ¶
type EngineAggregatingMergeTree struct{}
func (EngineAggregatingMergeTree) Kind ¶
func (EngineAggregatingMergeTree) Kind() string
func (EngineAggregatingMergeTree) Virtuals ¶
func (EngineAggregatingMergeTree) Virtuals() []DeclaredColumn
type EngineBuffer ¶
type EngineBuffer struct {
Database string `hcl:"database"`
Table string `hcl:"table"`
NumLayers int64 `hcl:"num_layers"`
MinTime int64 `hcl:"min_time"`
MaxTime int64 `hcl:"max_time"`
MinRows int64 `hcl:"min_rows"`
MaxRows int64 `hcl:"max_rows"`
MinBytes int64 `hcl:"min_bytes"`
MaxBytes int64 `hcl:"max_bytes"`
FlushTime *int64 `hcl:"flush_time,optional"`
FlushRows *int64 `hcl:"flush_rows,optional"`
FlushBytes *int64 `hcl:"flush_bytes,optional"`
}
EngineBuffer wraps another table with an in-memory write buffer that flushes periodically based on time/row/byte thresholds. CH syntax:
Buffer(database, table, num_layers, min_time, max_time, min_rows, max_rows, min_bytes, max_bytes [, flush_time [, flush_rows [, flush_bytes]]])
The optional flush_* triplet is captured as pointers; nil = not set (CH defaults apply). Because these are positional parameters, configured values must form a contiguous prefix: flush_rows requires flush_time, and flush_bytes requires both preceding fields.
func (EngineBuffer) Kind ¶
func (EngineBuffer) Kind() string
type EngineChange ¶
type EngineCollapsingMergeTree ¶
type EngineCollapsingMergeTree struct {
SignColumn string `hcl:"sign_column"`
}
func (EngineCollapsingMergeTree) Kind ¶
func (EngineCollapsingMergeTree) Kind() string
func (EngineCollapsingMergeTree) Virtuals ¶
func (EngineCollapsingMergeTree) Virtuals() []DeclaredColumn
type EngineDistributed ¶
type EngineDistributed struct {
ClusterName string `hcl:"cluster_name"`
RemoteDatabase string `hcl:"remote_database"`
RemoteTable string `hcl:"remote_table"`
ShardingKey *string `hcl:"sharding_key,optional"`
PolicyName *string `hcl:"policy_name,optional"`
}
func (EngineDistributed) DynamicVirtuals ¶
func (e EngineDistributed) DynamicVirtuals(r TableResolver) []DeclaredColumn
DynamicVirtuals returns _shard_num plus the virtuals of the remote table, recursing through chained Distributed tables. Cycles are broken by a visited set. With r == nil or the remote not modelled in the schema, only _shard_num is returned.
func (EngineDistributed) Kind ¶
func (EngineDistributed) Kind() string
func (EngineDistributed) Virtuals ¶
func (EngineDistributed) Virtuals() []DeclaredColumn
Virtuals returns only the Distributed-local virtuals — used as the static fallback when no TableResolver is available. With a resolver, DynamicVirtuals returns the transitive set.
type EngineJoin ¶
type EngineJoin struct {
Strictness string `hcl:"strictness"` // ANY | ALL | SEMI | ANTI
JoinType string `hcl:"type"` // LEFT | INNER | RIGHT | FULL
Keys []string `hcl:"keys"`
}
EngineJoin is a memory-resident table designed for the right-hand side of JOINs (and joinGet calls). CH syntax:
Join(strictness, type, k1[, k2, ...])
Strictness/type are bare identifiers in the DDL (not quoted strings) and must match the JOIN operation the table is used with — see the CH docs for the Join-table engine.
func (EngineJoin) Kind ¶
func (EngineJoin) Kind() string
type EngineKafka ¶
type EngineKafka struct {
// Collection is the named-collection reference. The remaining fields are
// optional per-table overrides when Collection is set.
Collection *string `hcl:"collection,optional"`
// BrokerList, TopicList, GroupName, and Format are required only when
// Collection is nil.
BrokerList *string `hcl:"broker_list,optional"`
TopicList *string `hcl:"topic_list,optional"`
GroupName *string `hcl:"group_name,optional"`
Format *string `hcl:"format,optional"`
// Optional auth.
SecurityProtocol *string `hcl:"security_protocol,optional"`
SaslMechanism *string `hcl:"sasl_mechanism,optional"`
SaslUsername *string `hcl:"sasl_username,optional"`
SaslPassword *string `hcl:"sasl_password,optional"`
// Optional numeric tuning.
NumConsumers *int64 `hcl:"num_consumers,optional"`
MaxBlockSize *int64 `hcl:"max_block_size,optional"`
SkipBrokenMessages *int64 `hcl:"skip_broken_messages,optional"`
PollTimeoutMs *int64 `hcl:"poll_timeout_ms,optional"`
PollMaxBatchSize *int64 `hcl:"poll_max_batch_size,optional"`
FlushIntervalMs *int64 `hcl:"flush_interval_ms,optional"`
ConsumerRescheduleMs *int64 `hcl:"consumer_reschedule_ms,optional"`
MaxRowsPerMessage *int64 `hcl:"max_rows_per_message,optional"`
CompressionLevel *int64 `hcl:"compression_level,optional"`
// Optional booleans (introspected as 0/1, presented as bool in HCL).
CommitEveryBatch *bool `hcl:"commit_every_batch,optional"`
ThreadPerConsumer *bool `hcl:"thread_per_consumer,optional"`
CommitOnSelect *bool `hcl:"commit_on_select,optional"`
// Optional strings.
ClientID *string `hcl:"client_id,optional"`
Schema *string `hcl:"schema,optional"`
HandleErrorMode *string `hcl:"handle_error_mode,optional"`
CompressionCodec *string `hcl:"compression_codec,optional"`
AutodetectClientRack *string `hcl:"autodetect_client_rack,optional"`
// Extra is the escape valve for kafka_* settings ClickHouse adds in
// versions we don't yet model. Keys are passed through verbatim and
// MUST include the `kafka_` prefix (the typed fields above strip it).
Extra map[string]string `hcl:"extra,optional"`
}
func (EngineKafka) Kind ¶
func (EngineKafka) Kind() string
func (EngineKafka) Virtuals ¶
func (e EngineKafka) Virtuals() []DeclaredColumn
type EngineMemory ¶
type EngineMemory struct{}
EngineMemory holds rows in RAM; non-durable, lost on restart. Common in test pipelines and short-lived staging tables.
func (EngineMemory) Kind ¶
func (EngineMemory) Kind() string
type EngineMerge ¶
EngineMerge is a read-only union over multiple physical tables matched by regex. CH syntax: Merge(db_regex, table_regex).
func (EngineMerge) Kind ¶
func (EngineMerge) Kind() string
type EngineMergeTree ¶
type EngineMergeTree struct{}
func (EngineMergeTree) Kind ¶
func (EngineMergeTree) Kind() string
func (EngineMergeTree) Virtuals ¶
func (EngineMergeTree) Virtuals() []DeclaredColumn
type EngineNull ¶
type EngineNull struct{}
EngineNull discards every write. Often used as a MV source that fans out into multiple downstream MVs without storing rows itself.
func (EngineNull) Kind ¶
func (EngineNull) Kind() string
type EngineReplacingMergeTree ¶
type EngineReplacingMergeTree struct {
VersionColumn *string `hcl:"version_column,optional"`
// IsDeletedColumn is ClickHouse's optional is_deleted parameter (1 = the
// row is a delete marker). ClickHouse only accepts it together with ver,
// so version_column must be set too (enforced at resolve time).
IsDeletedColumn *string `hcl:"is_deleted_column,optional"`
}
func (EngineReplacingMergeTree) Kind ¶
func (EngineReplacingMergeTree) Kind() string
func (EngineReplacingMergeTree) Virtuals ¶
func (EngineReplacingMergeTree) Virtuals() []DeclaredColumn
type EngineReplicatedAggregatingMergeTree ¶
type EngineReplicatedAggregatingMergeTree struct {
ZooPath string `hcl:"zoo_path"`
ReplicaName string `hcl:"replica_name"`
}
func (EngineReplicatedAggregatingMergeTree) Kind ¶
func (EngineReplicatedAggregatingMergeTree) Kind() string
func (EngineReplicatedAggregatingMergeTree) Virtuals ¶
func (EngineReplicatedAggregatingMergeTree) Virtuals() []DeclaredColumn
type EngineReplicatedCollapsingMergeTree ¶
type EngineReplicatedCollapsingMergeTree struct {
ZooPath string `hcl:"zoo_path"`
ReplicaName string `hcl:"replica_name"`
SignColumn string `hcl:"sign_column"`
}
func (EngineReplicatedCollapsingMergeTree) Kind ¶
func (EngineReplicatedCollapsingMergeTree) Kind() string
func (EngineReplicatedCollapsingMergeTree) Virtuals ¶
func (EngineReplicatedCollapsingMergeTree) Virtuals() []DeclaredColumn
type EngineReplicatedMergeTree ¶
type EngineReplicatedMergeTree struct {
ZooPath string `hcl:"zoo_path"`
ReplicaName string `hcl:"replica_name"`
}
func (EngineReplicatedMergeTree) Kind ¶
func (EngineReplicatedMergeTree) Kind() string
func (EngineReplicatedMergeTree) Virtuals ¶
func (EngineReplicatedMergeTree) Virtuals() []DeclaredColumn
type EngineReplicatedReplacingMergeTree ¶
type EngineReplicatedReplacingMergeTree struct {
ZooPath string `hcl:"zoo_path"`
ReplicaName string `hcl:"replica_name"`
VersionColumn *string `hcl:"version_column,optional"`
IsDeletedColumn *string `hcl:"is_deleted_column,optional"`
}
func (EngineReplicatedReplacingMergeTree) Kind ¶
func (EngineReplicatedReplacingMergeTree) Kind() string
func (EngineReplicatedReplacingMergeTree) Virtuals ¶
func (EngineReplicatedReplacingMergeTree) Virtuals() []DeclaredColumn
type EngineReplicatedSummingMergeTree ¶
type EngineReplicatedSummingMergeTree struct {
ZooPath string `hcl:"zoo_path"`
ReplicaName string `hcl:"replica_name"`
SumColumns []string `hcl:"sum_columns,optional"`
}
func (EngineReplicatedSummingMergeTree) Kind ¶
func (EngineReplicatedSummingMergeTree) Kind() string
func (EngineReplicatedSummingMergeTree) Virtuals ¶
func (EngineReplicatedSummingMergeTree) Virtuals() []DeclaredColumn
type EngineSharedAggregatingMergeTree ¶
type EngineSharedAggregatingMergeTree struct {
}
func (EngineSharedAggregatingMergeTree) Kind ¶
func (EngineSharedAggregatingMergeTree) Kind() string
func (EngineSharedAggregatingMergeTree) Virtuals ¶
func (EngineSharedAggregatingMergeTree) Virtuals() []DeclaredColumn
type EngineSharedCollapsingMergeTree ¶
type EngineSharedCollapsingMergeTree struct {
}
func (EngineSharedCollapsingMergeTree) Kind ¶
func (EngineSharedCollapsingMergeTree) Kind() string
func (EngineSharedCollapsingMergeTree) Virtuals ¶
func (EngineSharedCollapsingMergeTree) Virtuals() []DeclaredColumn
type EngineSharedMergeTree ¶
type EngineSharedMergeTree struct {
}
func (EngineSharedMergeTree) Kind ¶
func (EngineSharedMergeTree) Kind() string
func (EngineSharedMergeTree) Virtuals ¶
func (EngineSharedMergeTree) Virtuals() []DeclaredColumn
type EngineSharedReplacingMergeTree ¶
type EngineSharedReplacingMergeTree struct {
}
func (EngineSharedReplacingMergeTree) Kind ¶
func (EngineSharedReplacingMergeTree) Kind() string
func (EngineSharedReplacingMergeTree) Virtuals ¶
func (EngineSharedReplacingMergeTree) Virtuals() []DeclaredColumn
type EngineSharedSummingMergeTree ¶
type EngineSharedSummingMergeTree struct {
}
func (EngineSharedSummingMergeTree) Kind ¶
func (EngineSharedSummingMergeTree) Kind() string
func (EngineSharedSummingMergeTree) Virtuals ¶
func (EngineSharedSummingMergeTree) Virtuals() []DeclaredColumn
type EngineSpec ¶
type EngineSpec struct {
Kind string `hcl:"kind,label" diff:"-"`
Body hcl.Body `hcl:",remain" diff:"-"`
// Decoded is populated post-parse via DecodeEngine; gohcl ignores it
// because it has no hcl tag.
Decoded Engine
}
EngineSpec holds both the raw HCL representation (Kind label + Body) and the typed Decoded value. Decoded is populated by ParseFile after the initial HCL decode. Diff comparisons use Decoded only — Kind and Body are raw artifacts skipped via the diff tag.
type EngineSummingMergeTree ¶
type EngineSummingMergeTree struct {
SumColumns []string `hcl:"sum_columns,optional"`
}
func (EngineSummingMergeTree) Kind ¶
func (EngineSummingMergeTree) Kind() string
func (EngineSummingMergeTree) Virtuals ¶
func (EngineSummingMergeTree) Virtuals() []DeclaredColumn
type EngineTimeSeries ¶
type EngineTimeSeries struct {
// Settings declared via `engine "time_series" { settings = {...} }`.
// Free-form for forward compatibility. Only two are ALTER-able
// (id_generator, filter_by_min_time_and_max_time); the rest are
// baked into the CREATE.
Settings map[string]string `hcl:"settings,optional"`
// TagsToColumns shapes the inner tags-target table by promoting
// specific label keys into their own columns. CH treats this as a
// setting at the SQL level; promoted here for HCL clarity. sqlgen
// folds it back into the SETTINGS clause at emit time.
TagsToColumns map[string]string `hcl:"tags_to_columns,optional"`
// Target sub-blocks. Nil = CH default applies (auto-generate the
// inner target for that kind).
Samples *TimeSeriesTarget `hcl:"samples,block"`
Tags *TimeSeriesTarget `hcl:"tags,block"`
Metrics *TimeSeriesTarget `hcl:"metrics,block"`
// RecentSamples is the RECENT SAMPLES target (ClickHouse 26.8+): a
// short-lived copy of the newest samples, on by default
// (recent_samples_ttl_seconds).
RecentSamples *TimeSeriesTarget `hcl:"recent_samples,block"`
// KeywordHint preserves whether the samples target's source SQL used
// SAMPLES or DATA. Populated by the introspector; sqlgen reads it to
// round-trip the same word. HCL authors never set this — no hcl tag
// so gohcl ignores it during decode.
KeywordHint string `diff:"-"`
}
EngineTimeSeries models the (experimental) ClickHouse TimeSeries engine. It stores Prometheus-style time series across three sibling target tables (samples/tags/metrics), each either an external CH table reference or an inline declaration with INNER COLUMNS + optional INNER ENGINE.
TimeSeries exposes no engine-virtual columns — its outer columns are real declared columns. Virtuals() returns nil.
func (EngineTimeSeries) Kind ¶
func (EngineTimeSeries) Kind() string
func (EngineTimeSeries) Virtuals ¶
func (EngineTimeSeries) Virtuals() []DeclaredColumn
type EngineWithDynamicVirtuals ¶
type EngineWithDynamicVirtuals interface {
Engine
DynamicVirtuals(r TableResolver) []DeclaredColumn
}
EngineWithDynamicVirtuals is implemented by engines whose implicit column set depends on other tables in the schema — for example, Distributed, which exposes _shard_num plus the virtuals of its remote table. When an engine implements both interfaces, the dynamic form is authoritative.
type EngineWithVirtuals ¶
type EngineWithVirtuals interface {
Engine
Virtuals() []DeclaredColumn
}
EngineWithVirtuals is the optional interface an Engine implements when ClickHouse exposes implicit columns on tables of that kind. The returned set is static — derived from the engine's settings only, not from other tables in the schema. Engines with no virtuals (e.g. Log) simply don't implement it.
type ExcludeMatcher ¶
type ExcludeMatcher struct {
// contains filtered or unexported fields
}
ExcludeMatcher decides whether an object should be skipped during introspection or dropped from a schema before it is compared. It holds a list of glob patterns (filepath.Match syntax) matched against both the bare object name and its database-qualified "<database>.<name>" form, so patterns like "tmp_*" or "posthog.*_backup" both work, plus an optional set of object types that are excluded wholesale regardless of name. A nil matcher excludes nothing.
func LoadExcludeConfig ¶
func LoadExcludeConfig(path string) (*ExcludeMatcher, error)
LoadExcludeConfig parses an HCL exclude config:
exclude {
patterns = ["tmp_*", "_tmp_replace_*", "*_backup", ...]
object_types = ["named_collection"] # optional: drop a whole class
}
Patterns are validated as globs and object types against the known object kinds at load time. A file with no exclude block (or no patterns and no object types) yields a matcher that excludes nothing.
func NewExcludeMatcher ¶
func NewExcludeMatcher(patterns ...string) *ExcludeMatcher
NewExcludeMatcher builds a matcher from explicit patterns (used by tests).
func NewExcludeMatcherWithTypes ¶
func NewExcludeMatcherWithTypes(objectTypes []string, patterns ...string) *ExcludeMatcher
NewExcludeMatcherWithTypes builds a matcher from explicit object types and patterns (used by tests and callers that don't load a config file).
func (*ExcludeMatcher) Empty ¶
func (m *ExcludeMatcher) Empty() bool
Empty reports whether the matcher has no patterns and no object types (so it excludes nothing).
func (*ExcludeMatcher) Match ¶
func (m *ExcludeMatcher) Match(database, name string) (pattern string, ok bool)
Match reports whether an object is excluded and, if so, the pattern that matched (for logging). It tries each pattern against the bare name and the "<database>.<name>" qualified form.
func (*ExcludeMatcher) Matches ¶
func (m *ExcludeMatcher) Matches(database, name string) bool
Matches reports whether an object is excluded.
func (*ExcludeMatcher) MatchesObject ¶
func (m *ExcludeMatcher) MatchesObject(objectType, database, name string) bool
MatchesObject reports whether an object is excluded, by its type or by a name pattern.
type FieldChange ¶
type FieldChange struct {
Field string `json:"field"`
Change string `json:"change"` // add | drop | modify | rename
Old string `json:"old,omitempty"`
New string `json:"new,omitempty"`
}
FieldChange is one attribute-level difference on an altered object. The Field vocabulary is a public contract, documented in docs/README.hcl.md.
type GeneratedSQL ¶
type GeneratedSQL struct {
Statements []string
Ops []Operation
Unsafe []UnsafeChange
}
GeneratedSQL holds the result of GenerateSQL: the DDL statements for the migration, plus a list of unsafe changes (engine, order_by, partition_by, sample_by) that ClickHouse cannot apply in place and require manual table recreation.
Ops is a structured, parallel view of Statements: Ops[i] describes Statements[i] (same order, same length), carrying the operation kind and the target object's identity so consumers don't have to re-parse the SQL text or re-derive the dependency order.
A statement whose Ops[i].Manual is true (e.g. MATERIALIZE INDEX) is generated for the operator, never for automatic execution: anything that executes Statements must skip it, and text rendering comments it out.
func GenerateSQL ¶
func GenerateSQL(cs ChangeSet) GeneratedSQL
GenerateSQL turns a ChangeSet into ClickHouse DDL. Statements are ordered: CREATE TABLE, CREATE MATERIALIZED VIEW, ALTER TABLE, ALTER ... MODIFY QUERY, DROP VIEW, DROP TABLE — so materialized views are created after their destination table and dropped before it. Within the CREATE TABLE phase, tables are ordered by dependency so a Distributed table comes after the local table it forwards to; the DROP TABLE phase uses the reverse order. Unsafe or explicitly operator-reviewed changes (engine swap, existing-column reorder, ORDER BY change, materialized view recreation, etc.) are collected into Unsafe; the generator does not synthesize a recreate-and-swap procedure.
type IndexSpec ¶
type IndexSpec struct {
Name string `hcl:"name,label"`
Expr string `hcl:"expr"`
Type string `hcl:"type"`
Granularity int `hcl:"granularity,optional"`
// After / First position a patch_table index add, exactly like the
// column placement fields (#158/#160): patch-only, consumed and
// cleared at application, so the composed table renders identically
// to a flat declaration in that order. A declared table index
// carrying either is a resolve error.
After *string `hcl:"after,optional" diff:"-"`
First bool `hcl:"first,optional" diff:"-"`
}
type JSONOperation ¶
type JSONOperation struct {
Order int `json:"order"`
Kind string `json:"kind"` // CREATE | ALTER | DROP | RENAME
ObjectType string `json:"object_type"` // table | materialized_view | view | dictionary | named_collection | raw
Database string `json:"database"` // empty for named collections
Object string `json:"object"`
Engine string `json:"engine"` // ClickHouse engine family, tables only (e.g. ReplicatedMergeTree)
Replicated bool `json:"replicated"` // true when the engine family is a Replicated* variant
SQL string `json:"sql"` // statement without trailing ';'
Manual bool `json:"manual"` // operator-run only (e.g. MATERIALIZE INDEX); executors must skip it
Unsafe bool `json:"unsafe"` // this object has a change that can't be applied in place
UnsafeReason string `json:"unsafe_reason"`
}
JSONOperation is one entry in the structured, dependency-ordered diff output emitted by `hclexp diff -format json`. It mirrors one generated statement (Operations is parallel to GeneratedSQL.Statements), enriched with the object's engine family and an unsafe flag.
type JSONUnsafe ¶
type JSONUnsafe struct {
Database string `json:"database"`
Object string `json:"object"`
Reason string `json:"reason"`
}
JSONUnsafe is one destructive change that is never auto-emitted. The top-level list surfaces every UnsafeChange — including those that produce no statement (e.g. an engine or ORDER BY change that requires a full recreate) — so consumers of the JSON output see the same warnings the text mode prints as `-- UNSAFE:` comments.
type LayoutCache ¶
type LayoutCache struct {
SizeInCells int64 `hcl:"size_in_cells"`
}
func (LayoutCache) Kind ¶
func (LayoutCache) Kind() string
type LayoutComplexKeyCache ¶
type LayoutComplexKeyCache struct {
SizeInCells int64 `hcl:"size_in_cells"`
}
LayoutComplexKeyCache is the complex-key (multi-column key) counterpart to LayoutCache. Same single required parameter — size_in_cells controls the number of cells in the LRU cache.
func (LayoutComplexKeyCache) Kind ¶
func (LayoutComplexKeyCache) Kind() string
type LayoutComplexKeyDirect ¶
type LayoutComplexKeyDirect struct{}
LayoutComplexKeyDirect is the complex-key counterpart to LayoutDirect: every lookup hits the source, no in-process caching. No parameters.
func (LayoutComplexKeyDirect) Kind ¶
func (LayoutComplexKeyDirect) Kind() string
type LayoutComplexKeyHashed ¶
type LayoutComplexKeyHashed struct {
// Preallocate is accepted by ClickHouse for backwards compatibility
// but no longer documented; kept so old DDL introspects faithfully.
Preallocate *int64 `hcl:"preallocate,optional"`
Shards *int64 `hcl:"shards,optional"`
ShardLoadQueueBacklog *int64 `hcl:"shard_load_queue_backlog,optional"`
MaxLoadFactor *float64 `hcl:"max_load_factor,optional"`
}
func (LayoutComplexKeyHashed) Kind ¶
func (LayoutComplexKeyHashed) Kind() string
type LayoutComplexKeyHashedArray ¶
type LayoutComplexKeyHashedArray struct {
Shards *int64 `hcl:"shards,optional"`
}
LayoutComplexKeyHashedArray is the complex-key counterpart to LayoutHashedArray. Same optional `shards` parameter.
func (LayoutComplexKeyHashedArray) Kind ¶
func (LayoutComplexKeyHashedArray) Kind() string
type LayoutComplexKeyRangeHashed ¶
type LayoutComplexKeyRangeHashed struct {
RangeLookupStrategy *string `hcl:"range_lookup_strategy,optional"`
}
func (LayoutComplexKeyRangeHashed) Kind ¶
func (LayoutComplexKeyRangeHashed) Kind() string
type LayoutComplexKeySparseHashed ¶
type LayoutComplexKeySparseHashed struct {
Shards *int64 `hcl:"shards,optional"`
ShardLoadQueueBacklog *int64 `hcl:"shard_load_queue_backlog,optional"`
MaxLoadFactor *float64 `hcl:"max_load_factor,optional"`
}
func (LayoutComplexKeySparseHashed) Kind ¶
func (LayoutComplexKeySparseHashed) Kind() string
type LayoutDirect ¶
type LayoutDirect struct{}
func (LayoutDirect) Kind ¶
func (LayoutDirect) Kind() string
type LayoutFlat ¶
type LayoutFlat struct {
InitialArraySize *int64 `hcl:"initial_array_size,optional"`
MaxArraySize *int64 `hcl:"max_array_size,optional"`
}
func (LayoutFlat) Kind ¶
func (LayoutFlat) Kind() string
type LayoutHashed ¶
type LayoutHashed struct {
Shards *int64 `hcl:"shards,optional"`
ShardLoadQueueBacklog *int64 `hcl:"shard_load_queue_backlog,optional"`
MaxLoadFactor *float64 `hcl:"max_load_factor,optional"`
}
func (LayoutHashed) Kind ¶
func (LayoutHashed) Kind() string
type LayoutHashedArray ¶
type LayoutHashedArray struct {
Shards *int64 `hcl:"shards,optional"`
}
LayoutHashedArray is a memory-efficient hash layout that stores values in a packed array. `shards` partitions the dictionary across N parallel hash tables for concurrent reads (default 1).
func (LayoutHashedArray) Kind ¶
func (LayoutHashedArray) Kind() string
type LayoutIPTrie ¶
type LayoutIPTrie struct {
AccessToKeyFromAttributes *bool `hcl:"access_to_key_from_attributes,optional"`
}
func (LayoutIPTrie) Kind ¶
func (LayoutIPTrie) Kind() string
type LayoutRangeHashed ¶
type LayoutRangeHashed struct {
RangeLookupStrategy *string `hcl:"range_lookup_strategy,optional"`
}
func (LayoutRangeHashed) Kind ¶
func (LayoutRangeHashed) Kind() string
type LayoutRegexpTree ¶
type LayoutRegexpTree struct{}
LayoutRegexpTree backs a dictionary that matches keys against an ordered tree of regular expressions. The layout itself takes no inner parameters.
func (LayoutRegexpTree) Kind ¶
func (LayoutRegexpTree) Kind() string
type LayoutSparseHashed ¶
type LayoutSparseHashed struct {
Shards *int64 `hcl:"shards,optional"`
ShardLoadQueueBacklog *int64 `hcl:"shard_load_queue_backlog,optional"`
MaxLoadFactor *float64 `hcl:"max_load_factor,optional"`
}
func (LayoutSparseHashed) Kind ¶
func (LayoutSparseHashed) Kind() string
type MaterializedViewDiff ¶
type MaterializedViewDiff struct {
Name string
QueryChange *StringChange // the AS SELECT body changed
Recreate bool // the change cannot be applied with MODIFY QUERY
// Detail fields used by renderers. ColumnsChanged can accompany a safe
// QueryChange when the desired output is a compatible additive projection.
ToTableChange *StringChange
// ColumnOrderChange is populated when the column names and definitions are
// identical and declaration order is the only column difference. The view
// still requires recreation; this field makes that reason diagnosable.
ColumnOrderChange *OrderByChange
ColumnsChanged bool
}
MaterializedViewDiff is the set of mutations to a single existing materialized view. Query changes are applied in place via ALTER TABLE ... MODIFY QUERY. A destination change or a non-additive/incompatible column change requires recreating the view and is flagged unsafe.
func (MaterializedViewDiff) IsEmpty ¶
func (mvd MaterializedViewDiff) IsEmpty() bool
func (MaterializedViewDiff) IsUnsafe ¶
func (mvd MaterializedViewDiff) IsUnsafe() bool
IsUnsafe reports whether the diff requires recreating the view.
type MaterializedViewSpec ¶
type MaterializedViewSpec struct {
Name string `hcl:"name,label"`
// Inheritance fields. Consumed during resolution; not part of the
// schema for diff purposes. Extend names an abstract table whose
// columns (and optionally cluster/comment) are merged into this MV.
// Abstract MVs are dropped before diff, like abstract tables.
Extend *string `hcl:"extend,optional" diff:"-"`
Abstract bool `hcl:"abstract,optional" diff:"-"`
Override bool `hcl:"override,optional" diff:"-"`
ToTable string `hcl:"to_table,optional"` // TO <db.>table target (required when not abstract)
Columns []ColumnSpec `hcl:"column,block"` // explicit column list (may be empty; merged with parent on extend)
Query string `hcl:"query,optional"` // the AS SELECT ... body (required when not abstract)
Cluster *string `hcl:"cluster,optional"` // ON CLUSTER
Comment *string `hcl:"comment,optional"`
}
MaterializedViewSpec models a ClickHouse materialized view in its `TO <table>` form: the MV reads from its source (referenced in Query) and writes rows into an existing destination table. Inner-engine MVs (ENGINE = ...), refreshable MVs (REFRESH ...), and window views are not supported and are rejected with a clear error during introspection.
type NamedCollectionChange ¶
type NamedCollectionChange struct {
Name string
// Add is set for fresh adds AND for the create-half of a recreate
// (so the create has the full target spec).
Add *NamedCollectionSpec
// Drop is true for pure drops AND for the drop-half of a recreate.
Drop bool
// Recreate is true when ON CLUSTER changed; sqlgen emits
// DROP then CREATE adjacently.
Recreate bool
// Surgical (non-recreate) changes:
SetParams []NamedCollectionParam
DeleteParams []string
CommentChange *StringChange
// SkippedRedactedParams lists param keys whose diff was suppressed
// because the value is the redacted "[HIDDEN]" placeholder on one side
// while the other side holds a real value, differs in flags, or lacks
// the param entirely. Identically-redacted params (hidden on both
// sides, flags equal) are not a difference and are not listed. The CLI
// surfaces these so operators know hclexp couldn't verify equality.
SkippedRedactedParams []string
// Error is non-empty when the diff describes an unsupported transition.
// sqlgen emits no DDL; the CLI surfaces it.
Error string
}
NamedCollectionChange describes a planned change to a named collection.
func (NamedCollectionChange) IsEmpty ¶
func (c NamedCollectionChange) IsEmpty() bool
func (NamedCollectionChange) IsUnsafe ¶
func (c NamedCollectionChange) IsUnsafe() bool
type NamedCollectionParam ¶
type NamedCollectionSpec ¶
type NamedCollectionSpec struct {
Name string `hcl:"name,label"`
External bool `hcl:"external,optional"`
Override bool `hcl:"override,optional" diff:"-"`
Cluster *string `hcl:"cluster,optional"`
Comment *string `hcl:"comment,optional"`
Params []NamedCollectionParam `hcl:"param,block"`
}
NamedCollectionSpec models a ClickHouse named collection — a cluster-scoped key/value bag of configuration values that other objects (most notably Kafka tables) can reference by name.
External = true marks a collection as managed outside hclexp — for example, defined in the ClickHouse server XML config under /etc/clickhouse-server/config.d/. hclexp emits no DDL for external collections (no CREATE / ALTER / DROP); they exist in HCL only as declarations so engine `collection = "x"` references can resolve and be validated. Params is optional when External = true (the values live in the XML config, not in HCL).
func IntrospectNamedCollections ¶
func IntrospectNamedCollections(ctx context.Context, conn driver.Conn) ([]NamedCollectionSpec, error)
IntrospectNamedCollections returns every named collection the live ClickHouse cluster exposes via system.named_collections.
The cluster's system.named_collections schema varies by ClickHouse version. Newer versions expose `source` (DDL vs. config file) and `create_query` (the full CREATE statement, parseable for ON CLUSTER, comments, and OVERRIDABLE flags). Older versions expose only `name` and `collection` (a Map of key→value pairs).
This implementation reads only the columns guaranteed across versions — `name` and `collection`. As a consequence:
- Config-file (XML) collections are NOT filtered out at introspection; the operator-side `external = true` declaration is the way to keep hclexp from trying to manage them.
- ON CLUSTER, comment, and OVERRIDABLE/NOT OVERRIDABLE flags are not captured (the `collection` map doesn't carry them). Round-tripping these requires a newer ClickHouse with `create_query` exposed — extending the introspection then is a follow-up.
The query also enables `format_display_secrets_in_show_and_select` so values come back as their real strings rather than the redacted `[HIDDEN]` placeholder. The connecting user must have the `displaySecretsInShowAndSelect` access (granted via SQL `GRANT displaySecretsInShowAndSelect ON *.* TO <user>`, or in users.xml via `<show_named_collections_secrets>1</show_named_collections_secrets>`) for the setting to actually take effect — otherwise ClickHouse keeps returning `[HIDDEN]`.
type NodeSpec ¶
type NodeSpec struct {
Name string `hcl:"name,label"`
Macros map[string]string `hcl:"macros,optional"`
}
NodeSpec records the identity of a single physical ClickHouse node, dumped alongside its schema. Name is the node hostname; Macros is the raw key/value bag from `SELECT * FROM system.macros`.
func IntrospectNode ¶
IntrospectNode builds a NodeSpec for the connected node: its macros plus a name. When nameOverride is non-empty it is used as the node name; otherwise the name is read from ClickHouse via hostName().
type ObjectComparison ¶
type ObjectComparison struct {
Database string `json:"database"` // empty for named collections
Object string `json:"object"`
ObjectType string `json:"object_type"` // table | materialized_view | view | dictionary | named_collection | raw
// RawKind is a raw{} block's inner object kind (table | view |
// dictionary | materialized_view). Only a raw table holds rows on disk,
// so its DROP+CREATE recreate is the destructive one — the kind is what
// tells a consumer that apart.
RawKind string `json:"raw_kind,omitempty"`
Status string `json:"status"` // added | dropped | altered
Changes []FieldChange `json:"changes,omitempty"` // altered only
Operations []JSONOperation `json:"operations"` // the DDL that reconciles this object; may be empty (unsafe-only changes)
Unsafe bool `json:"unsafe"`
UnsafeReason string `json:"unsafe_reason,omitempty"`
Error string `json:"error,omitempty"` // unsupported transition
}
ObjectComparison describes how one object differs between two schemas. It is a projection of the ChangeSet — the same structs the text summary renders — so JSON, text, and counts can never contradict each other.
func BuildObjectComparisons ¶
func BuildObjectComparisons(cs ChangeSet, gen GeneratedSQL, left, right *Schema) []ObjectComparison
BuildObjectComparisons flattens a ChangeSet into one entry per differing object, attaching each generated operation to its object. Status is right-relative; nested operations keep their global dependency order, so the object view and the flat operation list can never disagree about sequencing.
type ObjectRef ¶
ObjectRef identifies a schema object (table or materialized view) by its database and name.
func ExtractReferencedTables ¶
ExtractReferencedTables parses a CREATE statement (TABLE, MATERIALIZED VIEW, VIEW, or DICTIONARY) and returns every table it references that is NOT the object being created itself:
- MV destination (TO dest) and SELECT FROM/JOIN sources
- View SELECT FROM/JOIN sources
- Dictionary SOURCE(CLICKHOUSE(TABLE 'x')) targets and SOURCE(CLICKHOUSE(QUERY 'SELECT ... FROM y')) inner-query sources
- Distributed engine remote_table
CTE names introduced by WITH ... AS are filtered out (they're query-local, not real tables). The object being created itself is excluded. Database-unqualified refs are returned with Database = "".
Use this from test setup to derive the list of tables that must exist before the CREATE statement is executable.
type Operation ¶
type Operation struct {
Kind string // OpCreate | OpAlter | OpDrop | OpRename
ObjectType string // table | materialized_view | view | dictionary | named_collection | raw
Database string // empty for named collections (cluster-scoped)
Object string
SQL string // the statement, without a trailing ';'
Manual bool // operator-run only (heavy mutation, e.g. MATERIALIZE INDEX); never execute automatically
}
Operation is the typed description of one generated DDL statement.
type OrderByChange ¶
type PatchColumnSpec ¶
type PatchColumnSpec struct {
Name string `hcl:"name,label"`
Type *string `hcl:"type,optional"`
Nullable *bool `hcl:"nullable,optional"`
Default *string `hcl:"default,optional"`
Materialized *string `hcl:"materialized,optional"`
Ephemeral *string `hcl:"ephemeral,optional"`
Alias *string `hcl:"alias,optional"`
Comment *string `hcl:"comment,optional"`
Codec *string `hcl:"codec,optional"`
TTL *string `hcl:"ttl,optional"`
}
PatchColumnSpec is a child-local, partial overlay for a column inherited through table Extend. Pointer fields distinguish an omitted attribute (preserve the parent value) from an explicit zero value such as nullable = false. If one default-kind attribute is set it replaces the inherited DEFAULT / MATERIALIZED / EPHEMERAL / ALIAS family wholesale. Optional modifiers cannot currently be removed; put the modifier-free shape on the abstract parent and add it on the child that needs it.
type PatchDictionarySpec ¶
type PatchDictionarySpec struct {
Name string `hcl:"name,label"`
Source *DictionarySourceSpec `hcl:"source,block"`
Layout *DictionaryLayoutSpec `hcl:"layout,block"`
Lifetime *DictionaryLifetime `hcl:"lifetime,block"`
Settings map[string]string `hcl:"settings,optional"`
}
PatchDictionarySpec is the dictionary counterpart of patch_table. Source, Layout, and Lifetime replace the target's wholesale when set; Settings merge patch-wins, like patch_table's.
type PatchMaterializedViewSpec ¶
type PatchMaterializedViewSpec struct {
Name string `hcl:"name,label"`
ToTable *string `hcl:"to_table,optional"`
Columns []ColumnSpec `hcl:"column,block"`
ModifyColumns []ColumnSpec `hcl:"modify_column,block"`
DropColumns []string `hcl:"drop_columns,optional"`
Query *string `hcl:"query,optional"`
}
PatchMaterializedViewSpec is the materialized-view counterpart of patch_table. ToTable and Query replace when set; columns follow patch_table's modify -> drop -> add semantics. It is consumed after MV extend resolution, so inherited columns may be modified or dropped too.
type PatchTableSpec ¶
type PatchTableSpec struct {
Name string `hcl:"name,label"`
Columns []ColumnSpec `hcl:"column,block"`
ModifyColumns []ColumnSpec `hcl:"modify_column,block"`
DropColumns []string `hcl:"drop_columns,optional"`
Indexes []IndexSpec `hcl:"index,block"`
DropIndexes []string `hcl:"drop_indexes,optional"`
Projections []ProjectionSpec `hcl:"projection,block"`
OrderBy []string `hcl:"order_by,optional"`
PartitionBy *string `hcl:"partition_by,optional"`
SampleBy *string `hcl:"sample_by,optional"`
TTL *string `hcl:"ttl,optional"`
Settings map[string]string `hcl:"settings,optional"`
Engine *EngineSpec `hcl:"engine,block"`
}
PatchTableSpec is a cross-layer modification of a table: the table stays declared once, and an env layer patches just its delta (#152, #154). Consumed during resolution; patches accumulate and apply in layer order, after their target inherits its parent and before any child inherits the resolved target. Inherited columns and indexes are therefore visible, and every descendant sees the patched result. Per-field semantics:
- Columns add (a name already on the target errors); ModifyColumns replace an existing column in place (an unknown name errors); DropColumns remove (unknown errors). Applied modify → drop → add, so add sees the post-drop state.
- Indexes add, DropIndexes remove; drops apply first, so a drop+add pair in one patch redefines an index.
- Projections add (an existing name errors).
- OrderBy / PartitionBy / SampleBy / TTL replace the target's value when set.
- Engine replaces the target's engine block wholesale — merging engine sub-arguments is not meaningful.
- Settings merge into the target's map, patch wins on key collision.
type PatchViewSpec ¶
type PatchViewSpec struct {
Name string `hcl:"name,label"`
Query *string `hcl:"query,optional"`
Comment *string `hcl:"comment,optional"`
}
PatchViewSpec is the view counterpart of patch_table: the view stays declared once and an env layer replaces just the fields it sets. Query is normalized to the canonical beautified form at parse, like a declared view's.
type PlanOperation ¶
type PlanOperation struct {
Order int `json:"order"`
Kind string `json:"kind"`
ObjectType string `json:"object_type"`
Database string `json:"database"`
Object string `json:"object"`
Engine string `json:"engine"`
Replicated bool `json:"replicated"`
SQL string `json:"sql"`
Manual bool `json:"manual"` // operator-run only (e.g. MATERIALIZE INDEX); executors must skip it
Roles []string `json:"roles"`
Unsafe bool `json:"unsafe"`
UnsafeReason string `json:"unsafe_reason"`
}
PlanOperation is one globally-ordered operation across all roles, with role provenance. It extends the diff -format json operation shape (#64) with the roles that contribute it; identical statements across roles collapse to one operation carrying the union of roles.
type PlanResult ¶
type PlanResult struct {
Operations []PlanOperation `json:"operations"`
Unsafe []JSONUnsafe `json:"unsafe,omitempty"`
Roles []RoleComparison `json:"roles"`
}
PlanResult is the merged, globally-ordered plan across every role.
func BuildPlan ¶
func BuildPlan(roles []RoleDiff) PlanResult
BuildPlan diffs every role, unions the per-role operations into one global operation list, and orders it by cross-role dependency graphs built over the unioned desired and current schemas. Because every role's objects share one (database, object) keyspace, a Distributed/Buffer object that forwards to a table physically hosted on another role (e.g. a per-role writable_query_log_archive -> the OPS sharded_query_log_archive) gets a real dependency edge — the cross-role ordering the issue is about.
Ordering: CREATE and widening ALTER follow dependency direction (a referenced object before the object that references it — storage before proxies before the MV); DROP runs in reverse. Identical statements across roles dedupe to a single operation carrying the union of contributing roles.
func BuildPlanWithOptions ¶ added in v0.3.0
func BuildPlanWithOptions(roles []RoleDiff, options DiffOptions) PlanResult
BuildPlanWithOptions is BuildPlan with the same comparison policy controls used by the other schema-comparison entry points.
type ProjectionSpec ¶
type ProjectionSpec struct {
Name string `hcl:"name,label"`
Query string `hcl:"query"`
Settings map[string]string `hcl:"settings,optional"`
}
ProjectionSpec is a table PROJECTION: a named SELECT (implicit FROM the parent table) materialized per part. Settings maps to the newer `WITH SETTINGS (…)` clause (projection-level MergeTree settings); ClickHouse before 26.5 neither emits nor accepts it.
type RawChange ¶
RawChange describes a change to a raw escape-hatch object. Raw SQL is opaque, so the only reconciliation is DROP + CREATE. Kind is the object's kind (drives the DROP form); OldSQL/NewSQL are the stored DDL on each side.
type RawSpec ¶
type RawSpec struct {
Kind string `hcl:"kind,label"` // table | materialized_view | view | dictionary
Name string `hcl:"name,label"`
Override bool `hcl:"override,optional" diff:"-"`
SQL string `hcl:"sql"`
}
RawSpec is an opaque object captured as its original CREATE DDL. It is the escape hatch for DDL the parser cannot handle or the HCL model cannot express. The two labels mirror Terraform's `resource "<type>" "<name>"`: Kind drives the DROP form on a recreate, Name is the object name, and SQL is emitted verbatim on apply.
type RenameColumn ¶
RenameColumn captures an explicit renamed_from directive that the diff engine has matched against the source schema. Old is the previous name, New is the current name.
type RoleComparison ¶
type RoleComparison struct {
Role string `json:"role"`
Objects []ObjectComparison `json:"objects"`
Summary CompareSummary `json:"summary"`
}
RoleComparison is one role's per-object view of its diff. Unlike the merged Operations list it is deliberately NOT deduped across roles — triage is per (env, role), so a shared object drifting on two roles appears under both.
type RoleDiff ¶
RoleDiff pairs a node role with its desired and current schema. Desired is the composed single-role authored schema (the target); Current is the matching node from a topology dump (one representative per role+shard, replicas collapsed). The plan emits the migration that brings Current to Desired.
type Schema ¶
type Schema struct {
Databases []DatabaseSpec
NamedCollections []NamedCollectionSpec
// Nodes carries per-node identity captured at introspection time:
// the node hostname (label) and its ClickHouse macros (shard,
// replica, hostClusterRole, hostClusterType, …). It is metadata only
// — Diff() ignores it — and exists so multi-node drift analysis can
// group nodes by their authoritative macros rather than by filename.
Nodes []NodeSpec
}
Schema is what ParseFile returns. It carries both top-level kinds hclexp tracks: databases (with their tables/MVs/dictionaries) and named collections (cluster-scoped, separate from any database).
func LoadLayers ¶
LoadLayers parses the .hcl files each layer path contributes, in the given order, and merges them into a combined raw spec set. A layer path is either a directory (every *.hcl inside it, in lexical filename order) or a single .hcl file. Across layers (and across files), duplicate managed-object names are errors unless the later declaration sets override = true. Raw objects are keyed by (kind, name). Patch blocks always accumulate.
LoadLayers does NOT call Resolve; callers run that explicitly so they can inspect the merged-but-unresolved input first.
func ParseFile ¶
ParseFile parses a single HCL file and returns the declared schema. Diagnostics are formatted into the returned error.
func ScopeSchemaToObjects ¶
ScopeSchemaToObjects returns a non-mutating copy of source containing only objects whose logical identity occurs in scope. Database wrappers and node metadata are retained. It is the shared primitive behind directional diff scope and desired-scoped topology planning.
type SettingChange ¶
type SkipSet ¶
type SkipSet struct {
// contains filtered or unexported fields
}
SkipSet selects dependent objects whose dependency validation should be skipped. It is built from a comma-separated list of object names; the special value "*" skips every object.
func ParseSkipSet ¶
ParseSkipSet parses a comma-separated list of dependent object names. Each entry is matched against an object's bare name or its "database.name" form. The single entry "*" skips all objects.
type SourceClickHouse ¶
type SourceClickHouse struct {
Collection *string `hcl:"collection,optional"`
Host *string `hcl:"host,optional"`
Port *int64 `hcl:"port,optional"`
User *string `hcl:"user,optional"`
Password *string `hcl:"password,optional"`
DB *string `hcl:"db,optional"`
Table *string `hcl:"table,optional"`
Query *string `hcl:"query,optional"`
InvalidateQuery *string `hcl:"invalidate_query,optional"`
UpdateField *string `hcl:"update_field,optional"`
UpdateLag *int64 `hcl:"update_lag,optional"`
}
SourceClickHouse — SOURCE(CLICKHOUSE(...)).
func (SourceClickHouse) Kind ¶
func (SourceClickHouse) Kind() string
type SourceExecutable ¶
type SourceExecutable struct {
Command string `hcl:"command"`
Format string `hcl:"format"`
ImplicitKey *bool `hcl:"implicit_key,optional"`
}
SourceExecutable — SOURCE(EXECUTABLE(...)).
func (SourceExecutable) Kind ¶
func (SourceExecutable) Kind() string
type SourceFile ¶
SourceFile — SOURCE(FILE(...)).
func (SourceFile) Kind ¶
func (SourceFile) Kind() string
type SourceHTTP ¶
type SourceHTTP struct {
Collection *string `hcl:"collection,optional"`
URL string `hcl:"url,optional"`
Format string `hcl:"format,optional"`
CredentialsUser *string `hcl:"credentials_user,optional"`
CredentialsPassword *string `hcl:"credentials_password,optional"`
}
SourceHTTP — SOURCE(HTTP(...)).
func (SourceHTTP) Kind ¶
func (SourceHTTP) Kind() string
type SourceMySQL ¶
type SourceMySQL struct {
Collection *string `hcl:"collection,optional"`
Host *string `hcl:"host,optional"`
Port *int64 `hcl:"port,optional"`
User *string `hcl:"user,optional"`
Password *string `hcl:"password,optional"`
DB *string `hcl:"db,optional"`
Table *string `hcl:"table,optional"`
Query *string `hcl:"query,optional"`
InvalidateQuery *string `hcl:"invalidate_query,optional"`
UpdateField *string `hcl:"update_field,optional"`
UpdateLag *int64 `hcl:"update_lag,optional"`
}
SourceMySQL — SOURCE(MYSQL(...)).
func (SourceMySQL) Kind ¶
func (SourceMySQL) Kind() string
type SourceNull ¶
type SourceNull struct{}
SourceNull — SOURCE(NULL()).
func (SourceNull) Kind ¶
func (SourceNull) Kind() string
type SourcePostgreSQL ¶
type SourcePostgreSQL struct {
Collection *string `hcl:"collection,optional"`
Host *string `hcl:"host,optional"`
Port *int64 `hcl:"port,optional"`
User *string `hcl:"user,optional"`
Password *string `hcl:"password,optional"`
DB *string `hcl:"db,optional"`
Table *string `hcl:"table,optional"`
Query *string `hcl:"query,optional"`
InvalidateQuery *string `hcl:"invalidate_query,optional"`
UpdateField *string `hcl:"update_field,optional"`
UpdateLag *int64 `hcl:"update_lag,optional"`
}
SourcePostgreSQL — SOURCE(POSTGRESQL(...)).
func (SourcePostgreSQL) Kind ¶
func (SourcePostgreSQL) Kind() string
type StringChange ¶
StringChange describes the transition of an optional string-valued attribute. A nil pointer on a side means "unset on that side."
type TableDiff ¶
type TableDiff struct {
Table string
RenameColumns []RenameColumn
AddColumns []ColumnSpec
DropColumns []string
ModifyColumns []ColumnChange
AddIndexes []IndexSpec
DropIndexes []string
// Projections keyed by name; a modify is emitted as DROP + ADD (there
// is no ALTER MODIFY PROJECTION). Adding one to an existing table also
// generates a manual MATERIALIZE PROJECTION.
AddProjections []ProjectionSpec
DropProjections []string
EngineChange *EngineChange
// ColumnOrderChange records a reorder among columns present on both
// sides. New columns carry FIRST/AFTER on AddColumns instead.
ColumnOrderChange *OrderByChange
OrderByChange *OrderByChange
PartitionByChange *StringChange
SampleByChange *StringChange
TTLChange *StringChange
SettingsAdded map[string]string
SettingsRemoved []string
SettingsChanged []SettingChange
// Constraints keyed by name; a modify is emitted as DROP + ADD. All
// in-place ALTER-able.
AddConstraints []ConstraintSpec
DropConstraints []string
ModifyConstraints []ConstraintChange
// PrimaryKeyChange is in-place-impossible (surfaced via unsafeReasons,
// like ORDER BY). CommentChange is ALTER-able (MODIFY COMMENT).
PrimaryKeyChange *OrderByChange
CommentChange *StringChange
}
TableDiff is the set of mutations to a single existing table. Any unset field means "no change in that aspect."
type TableResolver ¶
TableResolver is the minimum lookup surface dynamic-virtuals engines need. Pass nil when no schema context is available — dynamic engines must degrade gracefully (Distributed returns just _shard_num).
func NewSchemaResolver ¶
func NewSchemaResolver(dbs []DatabaseSpec) TableResolver
NewSchemaResolver builds a resolver over the loaded databases. Safe to call with an empty slice; the resolver returns ok=false for every lookup. Tables collide on (db, name); the last entry wins (Resolve already errors on duplicates within a single db, so collisions only happen across multiple DatabaseSpec entries for the same db name — rare, and the legacy multi-layer loader already merged them).
type TableSpec ¶
type TableSpec struct {
Name string `hcl:"name,label"`
// Inheritance / control fields. Consumed during resolution; not part
// of the schema for diff purposes.
Extend *string `hcl:"extend,optional" diff:"-"`
Abstract bool `hcl:"abstract,optional" diff:"-"`
Override bool `hcl:"override,optional" diff:"-"`
PrimaryKey []string `hcl:"primary_key,optional"`
OrderBy []string `hcl:"order_by,optional"`
PartitionBy *string `hcl:"partition_by,optional"`
SampleBy *string `hcl:"sample_by,optional"`
TTL *string `hcl:"ttl,optional"`
Settings map[string]string `hcl:"settings,optional"`
Comment *string `hcl:"comment,optional"`
// Cluster is the ON CLUSTER target. May be set on the table itself, or
// inherited from DatabaseSpec.Cluster during resolution.
Cluster *string `hcl:"cluster,optional"`
Columns []ColumnSpec `hcl:"column,block"`
// ColumnPatches partially specialize columns inherited through Extend.
// They are applied after the parent has resolved, preserve the inherited
// column's position and every field they do not set, then are consumed.
// A patch may only target an inherited column; ordinary column blocks
// remain additions and still collide with inherited names.
ColumnPatches []PatchColumnSpec `hcl:"patch_column,block" diff:"-"`
Indexes []IndexSpec `hcl:"index,block"`
Projections []ProjectionSpec `hcl:"projection,block"`
Constraints []ConstraintSpec `hcl:"constraint,block"`
Engine *EngineSpec `hcl:"engine,block"`
}
type TimeSeriesInnerTable ¶
type TimeSeriesInnerTable struct {
Columns []ColumnSpec `hcl:"column,block"`
Engine *EngineSpec `hcl:"engine,block"`
PrimaryKey []string `hcl:"primary_key,optional"`
OrderBy []string `hcl:"order_by,optional"`
PartitionBy *string `hcl:"partition_by,optional"`
TTL *string `hcl:"ttl,optional"`
Settings map[string]string `hcl:"settings,optional"`
}
TimeSeriesInnerTable mirrors a small slice of TableSpec — only the fields CH actually emits inside INNER COLUMNS / INNER ENGINE.
type TimeSeriesTarget ¶
type TimeSeriesTarget struct {
Target *string `hcl:"target,optional"`
Inner *TimeSeriesInnerTable `hcl:"inner,block"`
}
TimeSeriesTarget is one of {samples, tags, metrics, recent_samples}. Exactly one of Target or Inner must be set; both/neither is a resolve-time error.
type UnsafeChange ¶
UnsafeChange describes a diff entry that can't be expressed as an ALTER. Database and Table identify the target; Reason is a human-readable explanation of what would need to change.
type ValidateOptions ¶
type ValidateOptions struct {
// StrictProxyColumns requires a Distributed table and its remote to have
// exactly the same set of columns. When false (default), the proxy's
// columns need only be a subset of the remote's (every proxy column must
// exist on the remote with a matching type) — a proxy may legitimately
// omit columns.
StrictProxyColumns bool
// StrictClusters requires every Distributed remote to resolve against a
// real composition: a remote on an @absent cluster is an error instead of
// being satisfied. Use it to enforce full validation when the whole fleet
// is composed, so a stale @absent (a cluster that has since been mapped)
// cannot silently pass.
StrictClusters bool
}
ValidateOptions tunes optional checks layered on top of Validate's core dependency resolution.
type ValidationError ¶
type ValidationError struct {
Object ObjectRef // the dependent object
Missing ObjectRef // the unresolved reference
Kind string
Cluster string // Distributed cluster name; empty for non-Distributed findings
Reason string
}
ValidationError describes a dependency that could not be satisfied.
func Validate ¶
func Validate(dbs []DatabaseSpec, skip SkipSet, clusters ClusterSet) []ValidationError
Validate checks that every dependency carried by the loaded databases is satisfied: the referenced table or materialized view must be declared in one of the loaded databases. A reference into a database that is not loaded is itself an error, since it cannot be checked. Dependent objects matched by skip are not validated. All errors are collected and returned together.
Distributed remotes are resolved cluster-aware against clusters: a proxy routinely forwards to a storage table that lives on another cluster's composition (same database name, different cluster). See resolveDistributedRemote for the routing rules. Materialized-view and plain- view sources are likewise resolved against the union of mapped clusters when missing locally (a co-located read carries no cluster name). Pass ClusterSet{} when no external cluster mappings are available.
Once a Distributed remote resolves to an inspectable table, the proxy's columns are checked against it (subset by default; see ValidateOptions).
func ValidateOpts ¶
func ValidateOpts(dbs []DatabaseSpec, skip SkipSet, clusters ClusterSet, opts ValidateOptions) []ValidationError
ValidateOpts is Validate with explicit options.
func (ValidationError) Error ¶
func (e ValidationError) Error() string
type ViewDiff ¶
type ViewDiff struct {
Name string
QueryChange *StringChange
Comment *StringChange
Recreate bool
// RecreateChanged names the attributes that forced Recreate, in a fixed
// order, so consumers can report which one moved.
RecreateChanged []string
}
ViewDiff is the set of mutations to a single existing plain view. A query-only change is applied in place via ALTER TABLE ... MODIFY QUERY. A comment-only change is applied via ALTER TABLE ... MODIFY COMMENT. Any change to ColumnAliases / SQLSecurity / Definer / Cluster requires recreating the view (DROP + CREATE) and is flagged unsafe.
type ViewSpec ¶
type ViewSpec struct {
Name string `hcl:"name,label"`
Override bool `hcl:"override,optional" diff:"-"`
Query string `hcl:"query"`
ColumnAliases []string `hcl:"column_aliases,optional"`
SQLSecurity *string `hcl:"sql_security,optional"`
Definer *string `hcl:"definer,optional"`
Cluster *string `hcl:"cluster,optional"`
Comment *string `hcl:"comment,optional"`
}
ViewSpec models a ClickHouse plain (non-materialized) view: a saved SELECT executed on every read of the view. The Query is stored verbatim as text. Live views, refreshable materialized views, and window views are unsupported and rejected with a clear error during introspection (they parse into different AST types).
SQLSecurity is the canonical lowercase form of the SQL SECURITY clause: one of "definer", "invoker", or "none". Definer is the user the view runs as; required iff SQLSecurity == "definer" with a named user. The literal "current_user" is accepted unquoted.
Source Files
¶
- clusterset.go
- compare.go
- dictionary_dump.go
- dictionary_introspect.go
- dictionary_layouts.go
- dictionary_sources.go
- dictionary_sqlgen.go
- diff.go
- drift_doc.go
- dump.go
- engines.go
- exclude.go
- hclfuncs.go
- introspect.go
- layers.go
- locate.go
- macros_introspect.go
- named_collection_diff.go
- named_collection_dump.go
- named_collection_introspect.go
- named_collection_references.go
- named_collection_sqlgen.go
- parse.go
- parser.go
- plan.go
- query_normalize.go
- render.go
- render_json.go
- render_text.go
- resolver.go
- scope.go
- sql_edit.go
- sqlgen.go
- types.go
- validate.go
- virtual_columns.go