Documentation
¶
Overview ¶
Package schema provides schema migration generation for ClickHouse.
This package compares current and target schemas to generate executable migration files with SQL statements. It supports all major ClickHouse schema objects including databases, tables, dictionaries, and views, ensuring safe and predictable migrations for ClickHouse deployments.
Key features:
- Intelligent diff detection between current and target schemas
- Generation of executable DDL statements (not just comments)
- Proper operation ordering for safe migrations
- Complete support for all schema objects: databases, tables, dictionaries, views
- Smart rename detection to avoid unnecessary DROP+CREATE operations
- Different migration strategies for different object types
- Error handling for unsupported operations (engine/cluster changes)
- Comprehensive testing with YAML fixtures and table-driven tests
Supported Operations:
- Database operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME DATABASE
- Table operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME TABLE
- Dictionary operations: CREATE OR REPLACE, ATTACH, DETACH, DROP, RENAME DICTIONARY
- View operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME for both regular and materialized views
Migration Strategies:
- Databases: Standard CREATE, ALTER, DROP operations
- Tables: Full DDL support including column modifications
- Dictionaries: CREATE OR REPLACE for modifications (since they can't be altered)
- Regular Views: CREATE OR REPLACE for modifications
- Materialized Views: DROP+CREATE for query changes (more reliable than ALTER TABLE MODIFY QUERY)
- Integration Engine Tables: DROP+CREATE for all modifications (required due to read-only nature)
The migration generation process:
- Parse current schema state (from ClickHouse or SQL files)
- Parse target schema state (from SQL files)
- Compare the two states using intelligent algorithms
- Generate appropriate DDL for each difference with correct strategies
- Order operations correctly (databases → tables → dictionaries → views; CREATE → ALTER → RENAME → DROP)
Example usage:
// Parse current schema (from ClickHouse or existing SQL files)
currentSQL, _ := parser.ParseString("CREATE DATABASE analytics;")
// Parse target schema (from SQL string or file)
targetSQL, _ := parser.ParseString(targetSchemaString)
// Generate diff
diff, err := schema.GenerateDiff(
currentSQL,
targetSQL
)
if err != nil {
// Handle error (e.g., unsupported operation like engine changes)
log.Fatalf("Migration generation failed: %v", err)
}
// Format and write timestamped migration file
timestamp := time.Now().Format("20060102150405")
migrationFile := fmt.Sprintf("%s_migration.sql", timestamp)
var buf bytes.Buffer
format.FormatSQL(&buf, format.Defaults, diff)
os.WriteFile(migrationFile, buf.Bytes(), consts.ModeFile)
The package will return errors for operations that cannot be safely automated, such as database engine changes or cluster modifications. For integration engines and materialized views, it automatically uses DROP+CREATE strategies instead of reporting errors, ensuring migrations are both safe and executable.
Index ¶
- Variables
- func Compile(path string, w io.Writer) error
- func GenerateDiff(current, target *parser.SQL) (*parser.SQL, error)
- func GenerateMigrationFile(migrationDir string, current, target *parser.SQL) (string, error)
- type ColumnDiff
- type ColumnDiffType
- type ColumnInfo
- type DatabaseDiff
- type DatabaseDiffType
- type DatabaseInfo
- type DictionaryDiff
- type DictionaryDiffType
- type DictionaryInfo
- type FunctionDiff
- type FunctionDiffType
- type FunctionInfo
- type GrantInfo
- type NamedCollectionDiff
- type NamedCollectionDiffType
- type NamedCollectionInfo
- type RoleDiff
- type RoleDiffType
- type RoleInfo
- type TableDiff
- type TableDiffType
- type TableInfo
- type ViewDiff
- type ViewDiffType
- type ViewInfo
Constants ¶
This section is empty.
Variables ¶
var ( // ErrUnsupported is returned for operations that are fundamentally unsupported by ClickHouse ErrUnsupported = errors.New("unsupported operation") // ErrDictionaryAlter is returned when attempting to use ALTER operations on dictionaries ErrDictionaryAlter = errors.New("dictionary ALTER operations not supported") // ErrClusterChange is returned when attempting to change cluster configuration ErrClusterChange = errors.New("cluster configuration changes not supported") // ErrEngineChange is returned when attempting to change engine types ErrEngineChange = errors.New("engine type changes not supported") // ErrSystemObject is returned when attempting to modify system objects ErrSystemObject = errors.New("system object modifications not supported") // ErrInvalidType is returned for invalid type combinations ErrInvalidType = errors.New("invalid type combination") )
Invalid migration operation errors
var ErrNoDiff = errors.New("no differences found")
ErrNoDiff is returned when no differences are found between current and target schemas
Functions ¶
func Compile ¶
Compile recursively compiles a schema file and its imports. It processes import directives (lines starting with "-- housekeeper:import") and includes the referenced files' contents in the output. Import paths are resolved relative to the current file's directory.
Example:
var buf bytes.Buffer
err := schema.Compile("db/main.sql", &buf)
if err != nil {
log.Fatal(err)
}
// The compiled schema is now in buf
compiledSQL := buf.String()
// Parse the compiled schema
sql, err := parser.ParseString(compiledSQL)
if err != nil {
log.Fatal(err)
}
func GenerateDiff ¶
GenerateDiff creates a diff by comparing current and target schema states. It analyzes the differences between the current schema and the desired target schema, then generates appropriate DDL statements.
The migration includes all schema objects (roles, functions, databases, tables, dictionaries, views), processing them in the correct order: Roles → Functions → Databases → Named Collections → Tables → Dictionaries → Views (CREATE → ALTER → RENAME → DROP)
Migration strategies for different object types:
- Roles: Standard DDL operations (CREATE, ALTER, DROP, RENAME, GRANT, REVOKE)
- Functions: DROP+CREATE for modifications (since they can't be altered)
- Databases: Standard DDL operations (CREATE, ALTER, DROP, RENAME)
- Named Collections: Standard DDL operations (CREATE, ALTER, DROP)
- Tables: Full DDL support including column modifications (CREATE, ALTER, DROP, RENAME)
- Dictionaries: CREATE OR REPLACE for modifications (since they can't be altered)
- Regular Views: CREATE OR REPLACE for modifications
- Materialized Views: DROP+CREATE for query modifications (more reliable than ALTER TABLE MODIFY QUERY)
The function returns a *parser.SQL containing the migration statements, or an error if:
- No differences are found between current and target schemas (returns ErrNoDiff)
- An unsupported operation is detected (e.g., engine or cluster changes)
- Schema comparison fails for any object type
- Generated SQL cannot be parsed back into statements
Example:
currentSQL := `
CREATE DATABASE analytics ENGINE = Atomic COMMENT 'Old comment';
CREATE TABLE analytics.events (id UInt64, name String) ENGINE = MergeTree() ORDER BY id;
`
targetSQL := `
CREATE DATABASE analytics ENGINE = Atomic COMMENT 'New comment';
CREATE TABLE analytics.events (id UInt64, name String, timestamp DateTime) ENGINE = MergeTree() ORDER BY id;
CREATE DICTIONARY analytics.users_dict (id UInt64) PRIMARY KEY id SOURCE(HTTP(url 'test')) LAYOUT(FLAT()) LIFETIME(600);
CREATE VIEW analytics.daily_stats AS SELECT date, count() FROM events GROUP BY date;
`
current, _ := parser.ParseString(currentSQL)
target, _ := parser.ParseString(targetSQL)
diff, err := GenerateDiff(current, target)
if err != nil {
log.Fatal(err)
}
// Format the migration SQL for output
var buf bytes.Buffer
format.FormatSQL(&buf, format.Defaults, diff)
fmt.Println(buf.String())
func GenerateMigrationFile ¶
GenerateMigrationFile creates a timestamped migration file by comparing current and target schemas. The migration file is named using UTC timestamp in yyyyMMddhhmmss format and written to the specified directory. Returns the generated filename and any error encountered.
Parameters:
- migrationDir: Directory where the migration file should be written
- current: Current schema state
- target: Target schema state
Returns:
- filename: The generated migration filename (e.g., "20240806143022.sql")
- error: Any error encountered during generation or file writing (returns ErrNoDiff if no differences found)
Example:
filename, err := GenerateMigrationFile("/path/to/migrations", currentSchema, targetSchema)
// Creates: /path/to/migrations/20240806143022.sql
Types ¶
type ColumnDiff ¶
type ColumnDiff struct {
Type ColumnDiffType // Type of column operation
ColumnName string // Name of the column
Current *ColumnInfo // Current column definition (nil for ADD)
Target *ColumnInfo // Target column definition (nil for DROP)
Description string // Human-readable description
}
ColumnDiff represents a difference in column definitions
type ColumnDiffType ¶
type ColumnDiffType string
ColumnDiffType represents the type of column difference
const ( // ColumnDiffAdd indicates a column needs to be added ColumnDiffAdd ColumnDiffType = "ADD" // ColumnDiffDrop indicates a column needs to be dropped ColumnDiffDrop ColumnDiffType = "DROP" // ColumnDiffModify indicates a column needs to be modified ColumnDiffModify ColumnDiffType = "MODIFY" )
type ColumnInfo ¶
type ColumnInfo struct {
Name string // Column name
DataType *parser.DataType // Data type AST
DefaultType string // Default type: DEFAULT, MATERIALIZED, EPHEMERAL, ALIAS
Default *parser.Expression // Default expression AST
Codec *parser.CodecClause // Codec AST
TTL *parser.TTLClause // TTL AST
Comment string // Column comment
}
ColumnInfo represents a single column definition
func (ColumnInfo) Equal ¶ added in v0.1.3
func (c ColumnInfo) Equal(other ColumnInfo) bool
Equal compares two ColumnInfo instances for equality using AST comparison
type DatabaseDiff ¶
type DatabaseDiff struct {
Type DatabaseDiffType // Type of operation (CREATE, ALTER, DROP, RENAME)
DatabaseName string // Name of the database being modified
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *DatabaseInfo // Current state (nil if database doesn't exist)
Target *DatabaseInfo // Target state (nil if database should be dropped)
NewDatabaseName string // For rename operations - the new name
}
DatabaseDiff represents a difference between current and target database states. It contains all information needed to generate migration SQL statements for database operations including CREATE, ALTER, DROP, and RENAME.
type DatabaseDiffType ¶
type DatabaseDiffType string
DatabaseDiffType represents the type of database difference
const ( // DatabaseDiffCreate indicates a database needs to be created DatabaseDiffCreate DatabaseDiffType = "CREATE" // DatabaseDiffDrop indicates a database needs to be dropped DatabaseDiffDrop DatabaseDiffType = "DROP" // DatabaseDiffAlter indicates a database needs to be altered DatabaseDiffAlter DatabaseDiffType = "ALTER" // DatabaseDiffRename indicates a database needs to be renamed DatabaseDiffRename DatabaseDiffType = "RENAME" )
type DatabaseInfo ¶
type DatabaseInfo struct {
Name string // Database name
Engine string // Engine type (e.g., "Atomic", "MySQL", "Memory")
Comment string // Database comment (without quotes)
Cluster string // Cluster name if specified (empty if not clustered)
}
DatabaseInfo represents parsed database information extracted from DDL statements. This structure contains all the properties needed for database comparison and migration generation, including metadata for cluster and engine configuration.
type DictionaryDiff ¶
type DictionaryDiff struct {
Type DictionaryDiffType // Type of operation (CREATE, DROP, REPLACE, RENAME)
DictionaryName string // Full name of the dictionary (database.name)
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *DictionaryInfo // Current state (nil if dictionary doesn't exist)
Target *DictionaryInfo // Target state (nil if dictionary should be dropped)
NewDictionaryName string // For rename operations - the new full name
}
DictionaryDiff represents a difference between current and target dictionary states. It contains all information needed to generate migration SQL statements for dictionary operations including CREATE, DROP, REPLACE, and RENAME.
type DictionaryDiffType ¶
type DictionaryDiffType string
DictionaryDiffType represents the type of dictionary difference
const ( // DictionaryDiffCreate indicates a dictionary needs to be created DictionaryDiffCreate DictionaryDiffType = "CREATE" // DictionaryDiffDrop indicates a dictionary needs to be dropped DictionaryDiffDrop DictionaryDiffType = "DROP" // DictionaryDiffReplace indicates a dictionary needs to be replaced (since dictionaries can't be altered) DictionaryDiffReplace DictionaryDiffType = "REPLACE" // DictionaryDiffRename indicates a dictionary needs to be renamed DictionaryDiffRename DictionaryDiffType = "RENAME" )
type DictionaryInfo ¶
type DictionaryInfo struct {
Name string // Dictionary name
Database string // Database name (empty for default database)
Cluster string // Cluster name if specified (empty if not clustered)
Comment string // Dictionary comment (without quotes)
Statement *parser.CreateDictionaryStmt // Full parsed CREATE DICTIONARY statement for deep comparison
}
DictionaryInfo represents parsed dictionary information extracted from DDL statements. This structure contains all the properties needed for dictionary comparison and migration generation, including the full parsed statement for deep comparison.
type FunctionDiff ¶ added in v0.1.2
type FunctionDiff struct {
Type FunctionDiffType // Type of operation (CREATE, DROP, REPLACE, RENAME)
FunctionName string // Name of the function being modified
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *FunctionInfo // Current state (nil if function doesn't exist)
Target *FunctionInfo // Target state (nil if function should be dropped)
NewFunctionName string // For rename operations - the new name
}
FunctionDiff represents a difference between current and target function states. It contains all information needed to generate migration SQL statements for function operations including CREATE, DROP, REPLACE, and RENAME.
type FunctionDiffType ¶ added in v0.1.2
type FunctionDiffType string
FunctionDiffType represents the type of function difference
const ( // FunctionDiffCreate indicates a function needs to be created FunctionDiffCreate FunctionDiffType = "CREATE" // FunctionDiffDrop indicates a function needs to be dropped FunctionDiffDrop FunctionDiffType = "DROP" // FunctionDiffReplace indicates a function needs to be replaced (functions use DROP+CREATE for modifications) FunctionDiffReplace FunctionDiffType = "REPLACE" // FunctionDiffRename indicates a function needs to be renamed FunctionDiffRename FunctionDiffType = "RENAME" )
type FunctionInfo ¶ added in v0.1.2
type FunctionInfo struct {
Name string // Function name
Parameters []string // Function parameter names
Expression *parser.Expression // Function expression/body AST
Cluster string // Cluster name if specified (empty if not clustered)
}
FunctionInfo represents parsed function information extracted from DDL statements. This structure contains all the properties needed for function comparison and migration generation, including parameters, expression, and cluster configuration.
type GrantInfo ¶ added in v0.1.2
type GrantInfo struct {
Grantee string // Role or user receiving the grant
Privileges []string // List of privileges or roles granted
OnTarget string // Target object (database.table or *.*)
WithGrant bool // WITH GRANT OPTION
WithAdmin bool // WITH ADMIN OPTION
Cluster string // Cluster name if specified
}
GrantInfo represents parsed grant/privilege information
type NamedCollectionDiff ¶ added in v0.1.1
type NamedCollectionDiff struct {
Type NamedCollectionDiffType // Type of operation (CREATE, DROP, ALTER, REPLACE, RENAME)
CollectionName string // Name of the named collection
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *NamedCollectionInfo // Current state (nil if collection doesn't exist)
Target *NamedCollectionInfo // Target state (nil if collection should be dropped)
NewCollectionName string // For rename operations - the new name
}
NamedCollectionDiff represents a difference between current and target named collection states. It contains all information needed to generate migration SQL statements for named collection operations including CREATE, DROP, ALTER, REPLACE, and RENAME.
type NamedCollectionDiffType ¶ added in v0.1.1
type NamedCollectionDiffType string
NamedCollectionDiffType represents the type of named collection difference
const ( // NamedCollectionDiffCreate indicates a named collection needs to be created NamedCollectionDiffCreate NamedCollectionDiffType = "CREATE" // NamedCollectionDiffDrop indicates a named collection needs to be dropped NamedCollectionDiffDrop NamedCollectionDiffType = "DROP" // NamedCollectionDiffAlter indicates a named collection needs to be altered NamedCollectionDiffAlter NamedCollectionDiffType = "ALTER" // NamedCollectionDiffReplace indicates a named collection needs to be replaced NamedCollectionDiffReplace NamedCollectionDiffType = "REPLACE" // NamedCollectionDiffRename indicates a named collection needs to be renamed NamedCollectionDiffRename NamedCollectionDiffType = "RENAME" )
type NamedCollectionInfo ¶ added in v0.1.1
type NamedCollectionInfo struct {
Name string // Collection name
Cluster string // Cluster name if specified (empty if not clustered)
Comment string // Collection comment (without quotes)
Parameters map[string]string // Key-value parameters
Overridable *bool // Global override setting (nil if not specified)
Statement *parser.CreateNamedCollectionStmt // Full parsed CREATE NAMED COLLECTION statement
}
NamedCollectionInfo represents parsed named collection information extracted from DDL statements. This structure contains all the properties needed for named collection comparison and migration generation, including the full parsed statement for deep comparison.
type RoleDiff ¶ added in v0.1.2
type RoleDiff struct {
Type RoleDiffType // Type of operation (CREATE, ALTER, DROP, RENAME, GRANT, REVOKE)
RoleName string // Name of the role being modified
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *RoleInfo // Current state (nil if role doesn't exist)
Target *RoleInfo // Target state (nil if role should be dropped)
NewRoleName string // For rename operations - the new name
}
RoleDiff represents a difference between current and target role states. It contains all information needed to generate migration SQL statements for role operations including CREATE, ALTER, DROP, RENAME, GRANT, and REVOKE.
type RoleDiffType ¶ added in v0.1.2
type RoleDiffType string
RoleDiffType represents the type of role difference
const ( // RoleDiffCreate indicates a role needs to be created RoleDiffCreate RoleDiffType = "CREATE" // RoleDiffDrop indicates a role needs to be dropped RoleDiffDrop RoleDiffType = "DROP" // RoleDiffAlter indicates a role needs to be altered RoleDiffAlter RoleDiffType = "ALTER" // RoleDiffRename indicates a role needs to be renamed RoleDiffRename RoleDiffType = "RENAME" // RoleDiffGrant indicates privileges need to be granted RoleDiffGrant RoleDiffType = "GRANT" // RoleDiffRevoke indicates privileges need to be revoked RoleDiffRevoke RoleDiffType = "REVOKE" )
type RoleInfo ¶ added in v0.1.2
type RoleInfo struct {
Name string // Role name
Settings map[string]string // Role settings (key-value pairs)
Cluster string // Cluster name if specified (empty if not clustered)
}
RoleInfo represents parsed role information extracted from DDL statements. This structure contains all the properties needed for role comparison and migration generation, including settings and cluster configuration.
type TableDiff ¶
type TableDiff struct {
Type TableDiffType // Type of operation (CREATE, ALTER, DROP, RENAME)
TableName string // Name of the table being modified (with database prefix if needed)
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *TableInfo // Current state (nil if table doesn't exist)
Target *TableInfo // Target state (nil if table should be dropped)
NewTableName string // For rename operations - the new name
ColumnChanges []ColumnDiff // For ALTER operations - specific column changes
}
TableDiff represents a difference between current and target table states. It contains all information needed to generate migration SQL statements for table operations including CREATE, ALTER, DROP, and RENAME.
type TableDiffType ¶
type TableDiffType string
TableDiffType represents the type of table difference
const ( // TableDiffCreate indicates a table needs to be created TableDiffCreate TableDiffType = "CREATE" // TableDiffDrop indicates a table needs to be dropped TableDiffDrop TableDiffType = "DROP" // TableDiffAlter indicates a table needs to be altered TableDiffAlter TableDiffType = "ALTER" // TableDiffRename indicates a table needs to be renamed TableDiffRename TableDiffType = "RENAME" )
type TableInfo ¶
type TableInfo struct {
Name string // Table name (without database prefix)
Database string // Database name (empty if not specified)
Engine *parser.TableEngine // Engine AST
Cluster string // Cluster name for distributed tables
Comment string // Table comment
OrderBy *parser.Expression // ORDER BY expression AST
PartitionBy *parser.Expression // PARTITION BY expression AST
PrimaryKey *parser.Expression // PRIMARY KEY expression AST
SampleBy *parser.Expression // SAMPLE BY expression AST
TTL *parser.Expression // Table-level TTL expression AST
Settings map[string]string // Table settings
Columns []ColumnInfo // Column definitions
OrReplace bool // Whether CREATE OR REPLACE was used
IfNotExists bool // Whether IF NOT EXISTS was used
AsSourceTable *string // If this table uses AS, the source table name (qualified)
AsDependents map[string]bool // Tables that use AS to reference this table
}
TableInfo represents parsed table information extracted from DDL statements. This structure contains all the properties needed for table comparison and migration generation, including columns, engine, and other table options.
type ViewDiff ¶
type ViewDiff struct {
Type ViewDiffType // Type of operation (CREATE, DROP, ALTER, RENAME)
ViewName string // Full name of the view (database.name)
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *ViewInfo // Current state (nil if view doesn't exist)
Target *ViewInfo // Target state (nil if view should be dropped)
NewViewName string // For rename operations - the new full name
IsMaterialized bool // True if this is a materialized view
}
ViewDiff represents a difference between current and target view states. It handles both regular views and materialized views, with special handling for materialized views which can only be altered using ALTER TABLE MODIFY QUERY.
type ViewDiffType ¶
type ViewDiffType string
ViewDiffType represents the type of view difference
const ( // ViewDiffCreate indicates a view needs to be created ViewDiffCreate ViewDiffType = "CREATE" // ViewDiffDrop indicates a view needs to be dropped ViewDiffDrop ViewDiffType = "DROP" // ViewDiffAlter indicates a view needs to be altered (only for materialized views using ALTER TABLE MODIFY QUERY) ViewDiffAlter ViewDiffType = "ALTER" // ViewDiffRename indicates a view needs to be renamed (uses RENAME TABLE for both regular and materialized views) ViewDiffRename ViewDiffType = "RENAME" )
type ViewInfo ¶
type ViewInfo struct {
Name string // View name
Database string // Database name (empty for default database)
Cluster string // Cluster name if specified (empty if not clustered)
IsMaterialized bool // True if this is a materialized view
OrReplace bool // True if created with OR REPLACE
Query string // Query string for validation compatibility
Statement *parser.CreateViewStmt // Full parsed CREATE VIEW statement for deep comparison
}
ViewInfo represents parsed view information extracted from DDL statements. This structure contains all the properties needed for view comparison and migration generation, including whether it's a materialized view.