Documentation
¶
Overview ¶
Package executor provides migration execution functionality for ClickHouse databases.
The executor package handles the safe, atomic execution of database migrations with comprehensive error handling, progress tracking, and integrity verification. It integrates with the existing revision tracking system to maintain a complete audit trail of migration execution.
Core Components ¶
The package provides the Executor type for migration execution and supporting types for configuration and result handling:
- Executor: Main execution engine for applying migrations
- Config: Configuration options for executor creation
- ExecutionResult: Detailed results of migration execution
- ExecutionStatus: Status enumeration for migration outcomes
Key Features ¶
- Statement-by-statement execution with transaction safety
- Automatic bootstrap of housekeeper.revisions infrastructure
- Progress tracking and comprehensive error recovery
- Hash-based integrity verification
- Integration with existing revision and migration systems
- Cluster-aware execution with proper ON CLUSTER handling
Usage Example ¶
// Create executor with ClickHouse client
executor := executor.New(executor.Config{
ClickHouse: clickhouseClient,
Formatter: format.New(format.Defaults),
HousekeeperVersion: "1.0.0",
})
// Load migrations from directory
migrationDir, err := migrator.LoadMigrationDir(os.DirFS("./migrations"))
if err != nil {
log.Fatal(err)
}
// Execute pending migrations
results, err := executor.Execute(ctx, migrationDir.Migrations)
if err != nil {
log.Fatal(err)
}
// Process results
for _, result := range results {
switch result.Status {
case executor.StatusSuccess:
fmt.Printf("✓ %s completed in %v\n", result.Version, result.ExecutionTime)
case executor.StatusFailed:
fmt.Printf("✗ %s failed: %v\n", result.Version, result.Error)
case executor.StatusSkipped:
fmt.Printf("- %s already applied\n", result.Version)
}
}
Bootstrap Handling ¶
The executor automatically handles the bootstrap process for new ClickHouse instances that don't have the housekeeper migration tracking infrastructure:
- Detects if housekeeper database and revisions table exist
- Creates missing infrastructure automatically before migration execution
- Uses IF NOT EXISTS clauses for safe, idempotent bootstrap operations
- Handles the special case where revisions table doesn't exist on initial setup
Error Handling and Recovery ¶
The executor provides robust error handling with detailed context:
- Statement-level error reporting with exact SQL and position
- Partial execution tracking for migration recovery scenarios
- Comprehensive revision records for failed migrations
- Safe execution termination on first failure to prevent cascade issues
Integration with Revision System ¶
The executor seamlessly integrates with the existing pkg/migrator revision tracking system:
- Loads existing revisions to determine pending migrations
- Creates comprehensive revision records for all executions
- Records timing, error information, and integrity hashes
- Supports both StandardRevision and SnapshotRevision types
- Maintains compatibility with existing revision query methods
Testing and Development ¶
The package is designed with testing in mind and includes:
- Interface-based ClickHouse client for easy mocking
- Comprehensive test coverage with testcontainers integration
- Support for dry-run execution modes
- Integration with existing Housekeeper test infrastructure
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ClickHouse ¶
type ClickHouse interface {
Query(context.Context, string, ...any) (driver.Rows, error)
Exec(context.Context, string, ...any) error
}
ClickHouse defines the interface for ClickHouse database operations required by the migration executor.
type Config ¶
type Config struct {
// ClickHouse client for database operations
ClickHouse ClickHouse
// Formatter for generating clean SQL output
Formatter *format.Formatter
// HousekeeperVersion to record in revision entries
HousekeeperVersion string
}
Config contains configuration options for creating a new Executor.
type ExecutionResult ¶
type ExecutionResult struct {
// Version is the migration version that was executed
Version string
// Status indicates the outcome of the migration execution
Status ExecutionStatus
// Error contains any error that occurred during execution
Error error
// ExecutionTime records how long the migration took to execute
ExecutionTime time.Duration
// StatementsApplied indicates how many statements were successfully executed
StatementsApplied int
// TotalStatements is the total number of statements in the migration
TotalStatements int
// Revision contains the revision record that was created for this execution
Revision *migrator.Revision
}
ExecutionResult contains the result of executing a single migration.
Results provide detailed information about migration execution including timing, success/failure status, and any errors encountered. This information is essential for debugging failed migrations and tracking execution progress.
type ExecutionStatus ¶
type ExecutionStatus string
ExecutionStatus represents the outcome of a migration execution.
const ( // StatusSuccess indicates the migration was executed successfully StatusSuccess ExecutionStatus = "success" // StatusFailed indicates the migration execution failed StatusFailed ExecutionStatus = "failed" // StatusSkipped indicates the migration was skipped (already applied) StatusSkipped ExecutionStatus = "skipped" )
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor handles the execution of database migrations against ClickHouse.
The executor provides safe, atomic migration execution with comprehensive error handling, progress tracking, and integrity verification. It integrates with the existing revision tracking system to maintain a complete audit trail of migration execution.
Key features:
- Statement-by-statement execution with transaction safety
- Automatic bootstrap of housekeeper.revisions table
- Progress tracking and error recovery
- Hash-based integrity verification
- Integration with existing revision and migration systems
Example usage:
executor := executor.New(executor.Config{
ClickHouse: client,
Formatter: format.New(format.Defaults),
HousekeeperVersion: "1.0.0",
})
results, err := executor.Execute(ctx, []*migrator.Migration{migration})
if err != nil {
log.Fatal(err)
}
for _, result := range results {
fmt.Printf("Migration %s: %s\n", result.Version, result.Status)
}
func New ¶
New creates a new migration executor with the provided configuration.
The executor requires a ClickHouse client for database operations, a formatter for SQL generation, and the current Housekeeper version for revision tracking.
Example usage:
executor := executor.New(executor.Config{
ClickHouse: clickhouseClient,
Formatter: format.New(format.Defaults),
HousekeeperVersion: "1.0.0",
})
func (*Executor) ComputeHashes ¶
ComputeHashes computes the migration hash and partial hashes for each statement. This method is exported for testing purposes.
func (*Executor) Execute ¶
func (e *Executor) Execute(ctx context.Context, migrations []*migrator.Migration) ([]*ExecutionResult, error)
Execute applies a list of migrations to the ClickHouse database.
This method handles the complete migration execution lifecycle:
- Ensures housekeeper database and table exist (bootstrap)
- Loads existing revisions to determine what needs to be executed
- Executes each migration statement-by-statement with error handling
- Records revision entries for successful and failed executions
- Returns detailed execution results for each migration
The execution is atomic per migration - if any statement in a migration fails, the entire migration is marked as failed and execution stops. Previously executed migrations remain applied.
Example usage:
migrations := []*migrator.Migration{migration1, migration2}
results, err := executor.Execute(ctx, migrations)
if err != nil {
log.Fatal(err)
}
for _, result := range results {
switch result.Status {
case executor.StatusSuccess:
fmt.Printf("✓ %s completed in %v\n", result.Version, result.ExecutionTime)
case executor.StatusFailed:
fmt.Printf("✗ %s failed: %v\n", result.Version, result.Error)
case executor.StatusSkipped:
fmt.Printf("- %s already applied\n", result.Version)
}
}
func (*Executor) IsBootstrapped ¶
IsBootstrapped checks whether the housekeeper database and revisions table exist.
This method verifies that the migration tracking infrastructure is properly set up and ready for use. It checks for both the housekeeper database and the revisions table within that database.
Example usage:
bootstrapped, err := executor.IsBootstrapped(ctx)
if err != nil {
log.Fatal(err)
}
if !bootstrapped {
fmt.Println("Housekeeper infrastructure needs to be initialized")
}
Source Files