Documentation
¶
Overview ¶
Package schema provides schema migration generation for ClickHouse.
This package compares current and target schemas to generate executable migration files with SQL statements. It supports all major ClickHouse schema objects including databases, tables, dictionaries, and views, ensuring safe and predictable migrations for ClickHouse deployments.
Key features:
- Intelligent diff detection between current and target schemas
- Generation of executable DDL statements (not just comments)
- Proper operation ordering for safe migrations
- Complete support for all schema objects: databases, tables, dictionaries, views
- Smart rename detection to avoid unnecessary DROP+CREATE operations
- Different migration strategies for different object types
- Error handling for unsupported operations (engine/cluster changes)
- Comprehensive testing with YAML fixtures and table-driven tests
Supported Operations:
- Database operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME DATABASE
- Table operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME TABLE
- Dictionary operations: CREATE OR REPLACE, ATTACH, DETACH, DROP, RENAME DICTIONARY
- View operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME for both regular and materialized views
Migration Strategies:
- Databases: Standard CREATE, ALTER, DROP operations
- Tables: Full DDL support including column modifications
- Dictionaries: CREATE OR REPLACE for modifications (since they can't be altered)
- Regular Views: CREATE OR REPLACE for modifications
- Materialized Views: DROP+CREATE for query changes (more reliable than ALTER TABLE MODIFY QUERY)
- Integration Engine Tables: DROP+CREATE for all modifications (required due to read-only nature)
The migration generation process:
- Parse current schema state (from ClickHouse or SQL files)
- Parse target schema state (from SQL files)
- Compare the two states using intelligent algorithms
- Generate appropriate DDL for each difference with correct strategies
- Order operations correctly (databases → tables → dictionaries → views; CREATE → ALTER → RENAME → DROP)
Example usage:
// Parse current schema (from ClickHouse or existing SQL files)
currentSQL, _ := parser.ParseString("CREATE DATABASE analytics;")
// Parse target schema (from SQL string or file)
targetSQL, _ := parser.ParseString(targetSchemaString)
// Generate diff
diff, err := schema.GenerateDiff(
currentSQL,
targetSQL
)
if err != nil {
// Handle error (e.g., unsupported operation like engine changes)
log.Fatalf("Migration generation failed: %v", err)
}
// Format and write timestamped migration file
timestamp := time.Now().Format("20060102150405")
migrationFile := fmt.Sprintf("%s_migration.sql", timestamp)
var buf bytes.Buffer
format.FormatSQL(&buf, format.Defaults, diff)
os.WriteFile(migrationFile, buf.Bytes(), consts.ModeFile)
The package will return errors for operations that cannot be safely automated, such as database engine changes or cluster modifications. For integration engines and materialized views, it automatically uses DROP+CREATE strategies instead of reporting errors, ensuring migrations are both safe and executable.
Index ¶
- Variables
- func Compile(path string, w io.Writer) error
- func GenerateDiff(current, target *parser.SQL) (*parser.SQL, error)
- func GenerateMigrationFile(migrationDir string, current, target *parser.SQL) (string, error)
- type ColumnDiff
- type ColumnDiffType
- type ColumnInfo
- type DatabaseDiff
- type DatabaseDiffType
- type DatabaseInfo
- type DictionaryDiff
- type DictionaryDiffType
- type DictionaryInfo
- type TableDiff
- type TableDiffType
- type TableInfo
- type ViewDiff
- type ViewDiffType
- type ViewInfo
Constants ¶
This section is empty.
Variables ¶
var ( // ErrUnsupported is returned for operations that are fundamentally unsupported by ClickHouse ErrUnsupported = errors.New("unsupported operation") // ErrDictionaryAlter is returned when attempting to use ALTER operations on dictionaries ErrDictionaryAlter = errors.New("dictionary ALTER operations not supported") // ErrClusterChange is returned when attempting to change cluster configuration ErrClusterChange = errors.New("cluster configuration changes not supported") // ErrEngineChange is returned when attempting to change engine types ErrEngineChange = errors.New("engine type changes not supported") // ErrSystemObject is returned when attempting to modify system objects ErrSystemObject = errors.New("system object modifications not supported") // ErrInvalidType is returned for invalid type combinations ErrInvalidType = errors.New("invalid type combination") )
Invalid migration operation errors
var ErrNoDiff = errors.New("no differences found")
ErrNoDiff is returned when no differences are found between current and target schemas
Functions ¶
func Compile ¶
Compile recursively compiles a schema file and its imports. It processes import directives (lines starting with "-- housekeeper:import") and includes the referenced files' contents in the output. Import paths are resolved relative to the current file's directory.
Example:
var buf bytes.Buffer
err := schema.Compile("db/main.sql", &buf)
if err != nil {
log.Fatal(err)
}
// The compiled schema is now in buf
compiledSQL := buf.String()
// Parse the compiled schema
sql, err := parser.ParseString(compiledSQL)
if err != nil {
log.Fatal(err)
}
func GenerateDiff ¶
GenerateDiff creates a diff by comparing current and target schema states. It analyzes the differences between the current schema and the desired target schema, then generates appropriate DDL statements.
The migration includes all schema objects (databases, tables, dictionaries, views), processing them in the correct order: Databases → Tables → Dictionaries → Views (CREATE → ALTER → RENAME → DROP)
Migration strategies for different object types:
- Databases: Standard DDL operations (CREATE, ALTER, DROP, RENAME)
- Tables: Full DDL support including column modifications (CREATE, ALTER, DROP, RENAME)
- Dictionaries: CREATE OR REPLACE for modifications (since they can't be altered)
- Regular Views: CREATE OR REPLACE for modifications
- Materialized Views: ALTER TABLE MODIFY QUERY for query changes
The function returns a *parser.SQL containing the migration statements, or an error if:
- No differences are found between current and target schemas (returns ErrNoDiff)
- An unsupported operation is detected (e.g., engine or cluster changes)
- Schema comparison fails for any object type
- Generated SQL cannot be parsed back into statements
Example:
currentSQL := `
CREATE DATABASE analytics ENGINE = Atomic COMMENT 'Old comment';
CREATE TABLE analytics.events (id UInt64, name String) ENGINE = MergeTree() ORDER BY id;
`
targetSQL := `
CREATE DATABASE analytics ENGINE = Atomic COMMENT 'New comment';
CREATE TABLE analytics.events (id UInt64, name String, timestamp DateTime) ENGINE = MergeTree() ORDER BY id;
CREATE DICTIONARY analytics.users_dict (id UInt64) PRIMARY KEY id SOURCE(HTTP(url 'test')) LAYOUT(FLAT()) LIFETIME(600);
CREATE VIEW analytics.daily_stats AS SELECT date, count() FROM events GROUP BY date;
`
current, _ := parser.ParseString(currentSQL)
target, _ := parser.ParseString(targetSQL)
diff, err := GenerateDiff(current, target)
if err != nil {
log.Fatal(err)
}
// Format the migration SQL for output
var buf bytes.Buffer
format.FormatSQL(&buf, format.Defaults, diff)
fmt.Println(buf.String())
func GenerateMigrationFile ¶
GenerateMigrationFile creates a timestamped migration file by comparing current and target schemas. The migration file is named using UTC timestamp in yyyyMMddhhmmss format and written to the specified directory. Returns the generated filename and any error encountered.
Parameters:
- migrationDir: Directory where the migration file should be written
- current: Current schema state
- target: Target schema state
Returns:
- filename: The generated migration filename (e.g., "20240806143022.sql")
- error: Any error encountered during generation or file writing (returns ErrNoDiff if no differences found)
Example:
filename, err := GenerateMigrationFile("/path/to/migrations", currentSchema, targetSchema)
// Creates: /path/to/migrations/20240806143022.sql
Types ¶
type ColumnDiff ¶
type ColumnDiff struct {
Type ColumnDiffType // Type of column operation
ColumnName string // Name of the column
Current *ColumnInfo // Current column definition (nil for ADD)
Target *ColumnInfo // Target column definition (nil for DROP)
Description string // Human-readable description
}
ColumnDiff represents a difference in column definitions
type ColumnDiffType ¶
type ColumnDiffType string
ColumnDiffType represents the type of column difference
const ( // ColumnDiffAdd indicates a column needs to be added ColumnDiffAdd ColumnDiffType = "ADD" // ColumnDiffDrop indicates a column needs to be dropped ColumnDiffDrop ColumnDiffType = "DROP" // ColumnDiffModify indicates a column needs to be modified ColumnDiffModify ColumnDiffType = "MODIFY" )
type ColumnInfo ¶
type ColumnInfo struct {
Name string // Column name
DataType string // Full data type specification (e.g., "Nullable(String)", "Array(UInt64)")
Default string // Default value specification (e.g., "DEFAULT 'value'", "MATERIALIZED expr")
Codec string // Compression codec (e.g., "CODEC(ZSTD)")
TTL string // Column-level TTL
Comment string // Column comment
}
ColumnInfo represents a single column definition
type DatabaseDiff ¶
type DatabaseDiff struct {
Type DatabaseDiffType // Type of operation (CREATE, ALTER, DROP, RENAME)
DatabaseName string // Name of the database being modified
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *DatabaseInfo // Current state (nil if database doesn't exist)
Target *DatabaseInfo // Target state (nil if database should be dropped)
NewDatabaseName string // For rename operations - the new name
}
DatabaseDiff represents a difference between current and target database states. It contains all information needed to generate migration SQL statements for database operations including CREATE, ALTER, DROP, and RENAME.
type DatabaseDiffType ¶
type DatabaseDiffType string
DatabaseDiffType represents the type of database difference
const ( // DatabaseDiffCreate indicates a database needs to be created DatabaseDiffCreate DatabaseDiffType = "CREATE" // DatabaseDiffDrop indicates a database needs to be dropped DatabaseDiffDrop DatabaseDiffType = "DROP" // DatabaseDiffAlter indicates a database needs to be altered DatabaseDiffAlter DatabaseDiffType = "ALTER" // DatabaseDiffRename indicates a database needs to be renamed DatabaseDiffRename DatabaseDiffType = "RENAME" )
type DatabaseInfo ¶
type DatabaseInfo struct {
Name string // Database name
Engine string // Engine type (e.g., "Atomic", "MySQL", "Memory")
Comment string // Database comment (without quotes)
Cluster string // Cluster name if specified (empty if not clustered)
}
DatabaseInfo represents parsed database information extracted from DDL statements. This structure contains all the properties needed for database comparison and migration generation, including metadata for cluster and engine configuration.
type DictionaryDiff ¶
type DictionaryDiff struct {
Type DictionaryDiffType // Type of operation (CREATE, DROP, REPLACE, RENAME)
DictionaryName string // Full name of the dictionary (database.name)
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *DictionaryInfo // Current state (nil if dictionary doesn't exist)
Target *DictionaryInfo // Target state (nil if dictionary should be dropped)
NewDictionaryName string // For rename operations - the new full name
}
DictionaryDiff represents a difference between current and target dictionary states. It contains all information needed to generate migration SQL statements for dictionary operations including CREATE, DROP, REPLACE, and RENAME.
type DictionaryDiffType ¶
type DictionaryDiffType string
DictionaryDiffType represents the type of dictionary difference
const ( // DictionaryDiffCreate indicates a dictionary needs to be created DictionaryDiffCreate DictionaryDiffType = "CREATE" // DictionaryDiffDrop indicates a dictionary needs to be dropped DictionaryDiffDrop DictionaryDiffType = "DROP" // DictionaryDiffReplace indicates a dictionary needs to be replaced (since dictionaries can't be altered) DictionaryDiffReplace DictionaryDiffType = "REPLACE" // DictionaryDiffRename indicates a dictionary needs to be renamed DictionaryDiffRename DictionaryDiffType = "RENAME" )
type DictionaryInfo ¶
type DictionaryInfo struct {
Name string // Dictionary name
Database string // Database name (empty for default database)
Cluster string // Cluster name if specified (empty if not clustered)
Comment string // Dictionary comment (without quotes)
Statement *parser.CreateDictionaryStmt // Full parsed CREATE DICTIONARY statement for deep comparison
}
DictionaryInfo represents parsed dictionary information extracted from DDL statements. This structure contains all the properties needed for dictionary comparison and migration generation, including the full parsed statement for deep comparison.
type TableDiff ¶
type TableDiff struct {
Type TableDiffType // Type of operation (CREATE, ALTER, DROP, RENAME)
TableName string // Name of the table being modified (with database prefix if needed)
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *TableInfo // Current state (nil if table doesn't exist)
Target *TableInfo // Target state (nil if table should be dropped)
NewTableName string // For rename operations - the new name
ColumnChanges []ColumnDiff // For ALTER operations - specific column changes
}
TableDiff represents a difference between current and target table states. It contains all information needed to generate migration SQL statements for table operations including CREATE, ALTER, DROP, and RENAME.
type TableDiffType ¶
type TableDiffType string
TableDiffType represents the type of table difference
const ( // TableDiffCreate indicates a table needs to be created TableDiffCreate TableDiffType = "CREATE" // TableDiffDrop indicates a table needs to be dropped TableDiffDrop TableDiffType = "DROP" // TableDiffAlter indicates a table needs to be altered TableDiffAlter TableDiffType = "ALTER" // TableDiffRename indicates a table needs to be renamed TableDiffRename TableDiffType = "RENAME" )
type TableInfo ¶
type TableInfo struct {
Name string // Table name (without database prefix)
Database string // Database name (empty if not specified)
Engine string // Engine type and parameters (e.g., "MergeTree()", "ReplicatedMergeTree('/path', 'replica')")
Cluster string // Cluster name for distributed tables
Comment string // Table comment
OrderBy string // ORDER BY expression
PartitionBy string // PARTITION BY expression
PrimaryKey string // PRIMARY KEY expression
SampleBy string // SAMPLE BY expression
TTL string // Table-level TTL expression
Settings map[string]string // Table settings
Columns []ColumnInfo // Column definitions
OrReplace bool // Whether CREATE OR REPLACE was used
IfNotExists bool // Whether IF NOT EXISTS was used
}
TableInfo represents parsed table information extracted from DDL statements. This structure contains all the properties needed for table comparison and migration generation, including columns, engine, and other table options.
type ViewDiff ¶
type ViewDiff struct {
Type ViewDiffType // Type of operation (CREATE, DROP, ALTER, RENAME)
ViewName string // Full name of the view (database.name)
Description string // Human-readable description of the change
UpSQL string // SQL to apply the change (forward migration)
DownSQL string // SQL to rollback the change (reverse migration)
Current *ViewInfo // Current state (nil if view doesn't exist)
Target *ViewInfo // Target state (nil if view should be dropped)
NewViewName string // For rename operations - the new full name
IsMaterialized bool // True if this is a materialized view
}
ViewDiff represents a difference between current and target view states. It handles both regular views and materialized views, with special handling for materialized views which can only be altered using ALTER TABLE MODIFY QUERY.
type ViewDiffType ¶
type ViewDiffType string
ViewDiffType represents the type of view difference
const ( // ViewDiffCreate indicates a view needs to be created ViewDiffCreate ViewDiffType = "CREATE" // ViewDiffDrop indicates a view needs to be dropped ViewDiffDrop ViewDiffType = "DROP" // ViewDiffAlter indicates a view needs to be altered (only for materialized views using ALTER TABLE MODIFY QUERY) ViewDiffAlter ViewDiffType = "ALTER" // ViewDiffRename indicates a view needs to be renamed (uses RENAME TABLE for both regular and materialized views) ViewDiffRename ViewDiffType = "RENAME" )
type ViewInfo ¶
type ViewInfo struct {
Name string // View name
Database string // Database name (empty for default database)
Cluster string // Cluster name if specified (empty if not clustered)
IsMaterialized bool // True if this is a materialized view
OrReplace bool // True if created with OR REPLACE
Query string // Query string for validation compatibility
Statement *parser.CreateViewStmt // Full parsed CREATE VIEW statement for deep comparison
}
ViewInfo represents parsed view information extracted from DDL statements. This structure contains all the properties needed for view comparison and migration generation, including whether it's a materialized view.