schema

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2025 License: GPL-3.0 Imports: 15 Imported by: 0

Documentation

Overview

Package schema provides schema migration generation for ClickHouse.

This package compares current and target schemas to generate executable migration files with SQL statements. It supports all major ClickHouse schema objects including databases, tables, dictionaries, and views, ensuring safe and predictable migrations for ClickHouse deployments.

Key features:

  • Intelligent diff detection between current and target schemas
  • Generation of executable DDL statements (not just comments)
  • Proper operation ordering for safe migrations
  • Complete support for all schema objects: databases, tables, dictionaries, views
  • Smart rename detection to avoid unnecessary DROP+CREATE operations
  • Different migration strategies for different object types
  • Error handling for unsupported operations (engine/cluster changes)
  • Comprehensive testing with YAML fixtures and table-driven tests

Supported Operations:

  • Database operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME DATABASE
  • Table operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME TABLE
  • Dictionary operations: CREATE OR REPLACE, ATTACH, DETACH, DROP, RENAME DICTIONARY
  • View operations: CREATE, ALTER, ATTACH, DETACH, DROP, RENAME for both regular and materialized views

Migration Strategies:

  • Databases: Standard CREATE, ALTER, DROP operations
  • Tables: Full DDL support including column modifications
  • Dictionaries: CREATE OR REPLACE for modifications (since they can't be altered)
  • Regular Views: CREATE OR REPLACE for modifications
  • Materialized Views: DROP+CREATE for query changes (more reliable than ALTER TABLE MODIFY QUERY)
  • Integration Engine Tables: DROP+CREATE for all modifications (required due to read-only nature)

The migration generation process:

  1. Parse current schema state (from ClickHouse or SQL files)
  2. Parse target schema state (from SQL files)
  3. Compare the two states using intelligent algorithms
  4. Generate appropriate DDL for each difference with correct strategies
  5. Order operations correctly (databases → tables → dictionaries → views; CREATE → ALTER → RENAME → DROP)

Example usage:

// Parse current schema (from ClickHouse or existing SQL files)
currentSQL, _ := parser.ParseString("CREATE DATABASE analytics;")

// Parse target schema (from SQL string or file)
targetSQL, _ := parser.ParseString(targetSchemaString)

// Generate diff
diff, err := schema.GenerateDiff(
    currentSQL,
    targetSQL
)
if err != nil {
    // Handle error (e.g., unsupported operation like engine changes)
    log.Fatalf("Migration generation failed: %v", err)
}

// Format and write timestamped migration file
timestamp := time.Now().Format("20060102150405")
migrationFile := fmt.Sprintf("%s_migration.sql", timestamp)

var buf bytes.Buffer
format.FormatSQL(&buf, format.Defaults, diff)
os.WriteFile(migrationFile, buf.Bytes(), consts.ModeFile)

The package will return errors for operations that cannot be safely automated, such as database engine changes or cluster modifications. For integration engines and materialized views, it automatically uses DROP+CREATE strategies instead of reporting errors, ensuring migrations are both safe and executable.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrUnsupported is returned for operations that are fundamentally unsupported by ClickHouse
	ErrUnsupported = errors.New("unsupported operation")
	// ErrDictionaryAlter is returned when attempting to use ALTER operations on dictionaries
	ErrDictionaryAlter = errors.New("dictionary ALTER operations not supported")
	// ErrClusterChange is returned when attempting to change cluster configuration
	ErrClusterChange = errors.New("cluster configuration changes not supported")
	// ErrEngineChange is returned when attempting to change engine types
	ErrEngineChange = errors.New("engine type changes not supported")
	// ErrSystemObject is returned when attempting to modify system objects
	ErrSystemObject = errors.New("system object modifications not supported")
	// ErrInvalidType is returned for invalid type combinations
	ErrInvalidType = errors.New("invalid type combination")
)

Invalid migration operation errors

View Source
var ErrNoDiff = errors.New("no differences found")

ErrNoDiff is returned when no differences are found between current and target schemas

Functions

func Compile

func Compile(path string, w io.Writer) error

Compile recursively compiles a schema file and its imports. It processes import directives (lines starting with "-- housekeeper:import") and includes the referenced files' contents in the output. Import paths are resolved relative to the current file's directory.

Example:

var buf bytes.Buffer
err := schema.Compile("db/main.sql", &buf)
if err != nil {
	log.Fatal(err)
}

// The compiled schema is now in buf
compiledSQL := buf.String()

// Parse the compiled schema
sql, err := parser.ParseString(compiledSQL)
if err != nil {
	log.Fatal(err)
}

func GenerateDiff

func GenerateDiff(current, target *parser.SQL) (*parser.SQL, error)

GenerateDiff creates a diff by comparing current and target schema states. It analyzes the differences between the current schema and the desired target schema, then generates appropriate DDL statements.

The migration includes all schema objects (databases, tables, dictionaries, views), processing them in the correct order: Databases → Tables → Dictionaries → Views (CREATE → ALTER → RENAME → DROP)

Migration strategies for different object types:

  • Databases: Standard DDL operations (CREATE, ALTER, DROP, RENAME)
  • Tables: Full DDL support including column modifications (CREATE, ALTER, DROP, RENAME)
  • Dictionaries: CREATE OR REPLACE for modifications (since they can't be altered)
  • Regular Views: CREATE OR REPLACE for modifications
  • Materialized Views: ALTER TABLE MODIFY QUERY for query changes

The function returns a *parser.SQL containing the migration statements, or an error if:

  • No differences are found between current and target schemas (returns ErrNoDiff)
  • An unsupported operation is detected (e.g., engine or cluster changes)
  • Schema comparison fails for any object type
  • Generated SQL cannot be parsed back into statements

Example:

currentSQL := `
	CREATE DATABASE analytics ENGINE = Atomic COMMENT 'Old comment';
	CREATE TABLE analytics.events (id UInt64, name String) ENGINE = MergeTree() ORDER BY id;
`
targetSQL := `
	CREATE DATABASE analytics ENGINE = Atomic COMMENT 'New comment';
	CREATE TABLE analytics.events (id UInt64, name String, timestamp DateTime) ENGINE = MergeTree() ORDER BY id;
	CREATE DICTIONARY analytics.users_dict (id UInt64) PRIMARY KEY id SOURCE(HTTP(url 'test')) LAYOUT(FLAT()) LIFETIME(600);
	CREATE VIEW analytics.daily_stats AS SELECT date, count() FROM events GROUP BY date;
`

current, _ := parser.ParseString(currentSQL)
target, _ := parser.ParseString(targetSQL)

diff, err := GenerateDiff(current, target)
if err != nil {
	log.Fatal(err)
}

// Format the migration SQL for output
var buf bytes.Buffer
format.FormatSQL(&buf, format.Defaults, diff)
fmt.Println(buf.String())

func GenerateMigrationFile

func GenerateMigrationFile(migrationDir string, current, target *parser.SQL) (string, error)

GenerateMigrationFile creates a timestamped migration file by comparing current and target schemas. The migration file is named using UTC timestamp in yyyyMMddhhmmss format and written to the specified directory. Returns the generated filename and any error encountered.

Parameters:

  • migrationDir: Directory where the migration file should be written
  • current: Current schema state
  • target: Target schema state

Returns:

  • filename: The generated migration filename (e.g., "20240806143022.sql")
  • error: Any error encountered during generation or file writing (returns ErrNoDiff if no differences found)

Example:

filename, err := GenerateMigrationFile("/path/to/migrations", currentSchema, targetSchema)
// Creates: /path/to/migrations/20240806143022.sql

Types

type ColumnDiff

type ColumnDiff struct {
	Type        ColumnDiffType // Type of column operation
	ColumnName  string         // Name of the column
	Current     *ColumnInfo    // Current column definition (nil for ADD)
	Target      *ColumnInfo    // Target column definition (nil for DROP)
	Description string         // Human-readable description
}

ColumnDiff represents a difference in column definitions

type ColumnDiffType

type ColumnDiffType string

ColumnDiffType represents the type of column difference

const (
	// ColumnDiffAdd indicates a column needs to be added
	ColumnDiffAdd ColumnDiffType = "ADD"
	// ColumnDiffDrop indicates a column needs to be dropped
	ColumnDiffDrop ColumnDiffType = "DROP"
	// ColumnDiffModify indicates a column needs to be modified
	ColumnDiffModify ColumnDiffType = "MODIFY"
)

type ColumnInfo

type ColumnInfo struct {
	Name     string // Column name
	DataType string // Full data type specification (e.g., "Nullable(String)", "Array(UInt64)")
	Default  string // Default value specification (e.g., "DEFAULT 'value'", "MATERIALIZED expr")
	Codec    string // Compression codec (e.g., "CODEC(ZSTD)")
	TTL      string // Column-level TTL
	Comment  string // Column comment
}

ColumnInfo represents a single column definition

type DatabaseDiff

type DatabaseDiff struct {
	Type            DatabaseDiffType // Type of operation (CREATE, ALTER, DROP, RENAME)
	DatabaseName    string           // Name of the database being modified
	Description     string           // Human-readable description of the change
	UpSQL           string           // SQL to apply the change (forward migration)
	DownSQL         string           // SQL to rollback the change (reverse migration)
	Current         *DatabaseInfo    // Current state (nil if database doesn't exist)
	Target          *DatabaseInfo    // Target state (nil if database should be dropped)
	NewDatabaseName string           // For rename operations - the new name
}

DatabaseDiff represents a difference between current and target database states. It contains all information needed to generate migration SQL statements for database operations including CREATE, ALTER, DROP, and RENAME.

type DatabaseDiffType

type DatabaseDiffType string

DatabaseDiffType represents the type of database difference

const (
	// DatabaseDiffCreate indicates a database needs to be created
	DatabaseDiffCreate DatabaseDiffType = "CREATE"
	// DatabaseDiffDrop indicates a database needs to be dropped
	DatabaseDiffDrop DatabaseDiffType = "DROP"
	// DatabaseDiffAlter indicates a database needs to be altered
	DatabaseDiffAlter DatabaseDiffType = "ALTER"
	// DatabaseDiffRename indicates a database needs to be renamed
	DatabaseDiffRename DatabaseDiffType = "RENAME"
)

type DatabaseInfo

type DatabaseInfo struct {
	Name    string // Database name
	Engine  string // Engine type (e.g., "Atomic", "MySQL", "Memory")
	Comment string // Database comment (without quotes)
	Cluster string // Cluster name if specified (empty if not clustered)
}

DatabaseInfo represents parsed database information extracted from DDL statements. This structure contains all the properties needed for database comparison and migration generation, including metadata for cluster and engine configuration.

type DictionaryDiff

type DictionaryDiff struct {
	Type              DictionaryDiffType // Type of operation (CREATE, DROP, REPLACE, RENAME)
	DictionaryName    string             // Full name of the dictionary (database.name)
	Description       string             // Human-readable description of the change
	UpSQL             string             // SQL to apply the change (forward migration)
	DownSQL           string             // SQL to rollback the change (reverse migration)
	Current           *DictionaryInfo    // Current state (nil if dictionary doesn't exist)
	Target            *DictionaryInfo    // Target state (nil if dictionary should be dropped)
	NewDictionaryName string             // For rename operations - the new full name
}

DictionaryDiff represents a difference between current and target dictionary states. It contains all information needed to generate migration SQL statements for dictionary operations including CREATE, DROP, REPLACE, and RENAME.

type DictionaryDiffType

type DictionaryDiffType string

DictionaryDiffType represents the type of dictionary difference

const (
	// DictionaryDiffCreate indicates a dictionary needs to be created
	DictionaryDiffCreate DictionaryDiffType = "CREATE"
	// DictionaryDiffDrop indicates a dictionary needs to be dropped
	DictionaryDiffDrop DictionaryDiffType = "DROP"
	// DictionaryDiffReplace indicates a dictionary needs to be replaced (since dictionaries can't be altered)
	DictionaryDiffReplace DictionaryDiffType = "REPLACE"
	// DictionaryDiffRename indicates a dictionary needs to be renamed
	DictionaryDiffRename DictionaryDiffType = "RENAME"
)

type DictionaryInfo

type DictionaryInfo struct {
	Name      string                       // Dictionary name
	Database  string                       // Database name (empty for default database)
	Cluster   string                       // Cluster name if specified (empty if not clustered)
	Comment   string                       // Dictionary comment (without quotes)
	Statement *parser.CreateDictionaryStmt // Full parsed CREATE DICTIONARY statement for deep comparison
}

DictionaryInfo represents parsed dictionary information extracted from DDL statements. This structure contains all the properties needed for dictionary comparison and migration generation, including the full parsed statement for deep comparison.

type TableDiff

type TableDiff struct {
	Type          TableDiffType // Type of operation (CREATE, ALTER, DROP, RENAME)
	TableName     string        // Name of the table being modified (with database prefix if needed)
	Description   string        // Human-readable description of the change
	UpSQL         string        // SQL to apply the change (forward migration)
	DownSQL       string        // SQL to rollback the change (reverse migration)
	Current       *TableInfo    // Current state (nil if table doesn't exist)
	Target        *TableInfo    // Target state (nil if table should be dropped)
	NewTableName  string        // For rename operations - the new name
	ColumnChanges []ColumnDiff  // For ALTER operations - specific column changes
}

TableDiff represents a difference between current and target table states. It contains all information needed to generate migration SQL statements for table operations including CREATE, ALTER, DROP, and RENAME.

type TableDiffType

type TableDiffType string

TableDiffType represents the type of table difference

const (
	// TableDiffCreate indicates a table needs to be created
	TableDiffCreate TableDiffType = "CREATE"
	// TableDiffDrop indicates a table needs to be dropped
	TableDiffDrop TableDiffType = "DROP"
	// TableDiffAlter indicates a table needs to be altered
	TableDiffAlter TableDiffType = "ALTER"
	// TableDiffRename indicates a table needs to be renamed
	TableDiffRename TableDiffType = "RENAME"
)

type TableInfo

type TableInfo struct {
	Name        string            // Table name (without database prefix)
	Database    string            // Database name (empty if not specified)
	Engine      string            // Engine type and parameters (e.g., "MergeTree()", "ReplicatedMergeTree('/path', 'replica')")
	Cluster     string            // Cluster name for distributed tables
	Comment     string            // Table comment
	OrderBy     string            // ORDER BY expression
	PartitionBy string            // PARTITION BY expression
	PrimaryKey  string            // PRIMARY KEY expression
	SampleBy    string            // SAMPLE BY expression
	TTL         string            // Table-level TTL expression
	Settings    map[string]string // Table settings
	Columns     []ColumnInfo      // Column definitions
	OrReplace   bool              // Whether CREATE OR REPLACE was used
	IfNotExists bool              // Whether IF NOT EXISTS was used
}

TableInfo represents parsed table information extracted from DDL statements. This structure contains all the properties needed for table comparison and migration generation, including columns, engine, and other table options.

type ViewDiff

type ViewDiff struct {
	Type           ViewDiffType // Type of operation (CREATE, DROP, ALTER, RENAME)
	ViewName       string       // Full name of the view (database.name)
	Description    string       // Human-readable description of the change
	UpSQL          string       // SQL to apply the change (forward migration)
	DownSQL        string       // SQL to rollback the change (reverse migration)
	Current        *ViewInfo    // Current state (nil if view doesn't exist)
	Target         *ViewInfo    // Target state (nil if view should be dropped)
	NewViewName    string       // For rename operations - the new full name
	IsMaterialized bool         // True if this is a materialized view
}

ViewDiff represents a difference between current and target view states. It handles both regular views and materialized views, with special handling for materialized views which can only be altered using ALTER TABLE MODIFY QUERY.

type ViewDiffType

type ViewDiffType string

ViewDiffType represents the type of view difference

const (
	// ViewDiffCreate indicates a view needs to be created
	ViewDiffCreate ViewDiffType = "CREATE"
	// ViewDiffDrop indicates a view needs to be dropped
	ViewDiffDrop ViewDiffType = "DROP"
	// ViewDiffAlter indicates a view needs to be altered (only for materialized views using ALTER TABLE MODIFY QUERY)
	ViewDiffAlter ViewDiffType = "ALTER"
	// ViewDiffRename indicates a view needs to be renamed (uses RENAME TABLE for both regular and materialized views)
	ViewDiffRename ViewDiffType = "RENAME"
)

type ViewInfo

type ViewInfo struct {
	Name           string                 // View name
	Database       string                 // Database name (empty for default database)
	Cluster        string                 // Cluster name if specified (empty if not clustered)
	IsMaterialized bool                   // True if this is a materialized view
	OrReplace      bool                   // True if created with OR REPLACE
	Query          string                 // Query string for validation compatibility
	Statement      *parser.CreateViewStmt // Full parsed CREATE VIEW statement for deep comparison
}

ViewInfo represents parsed view information extracted from DDL statements. This structure contains all the properties needed for view comparison and migration generation, including whether it's a materialized view.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL