Documentation
¶
Overview ¶
Package clickhouse provides a comprehensive client for interacting with ClickHouse databases.
This package offers functionality to connect to ClickHouse instances, retrieve complete database schemas (databases, tables, views, dictionaries), and execute DDL migrations. It serves as the bridge between the Housekeeper tool and actual ClickHouse deployments.
Key features:
- Simple connection management with DSN-based configuration
- Comprehensive schema extraction for all ClickHouse object types
- Dedicated SchemaExtractor for granular control over schema operations
- Retrieval of current schemas as parser.SQL structures for migration generation
- Execution of migration SQL statements with validation
- Automatic filtering of system objects (system, information_schema databases)
- Support for clustered ClickHouse deployments (ON CLUSTER detection)
Schema Extraction:
- Databases: Complete CREATE DATABASE statements with engine and comments
- Tables: Full table definitions with columns, engines, and all table properties
- Views: Both regular and materialized views with their definitions
- Dictionaries: Dictionary definitions with source, layout, and lifetime configurations
The client is designed to work seamlessly with the parser and migrator packages, retrieving current database state in a format that can be directly compared with target schemas defined in SQL files for migration generation.
Example usage:
// Connect to ClickHouse
client, err := clickhouse.NewClient(ctx, "localhost:9000")
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Get complete schema (databases, tables, views, dictionaries)
currentSchema, err := client.GetSchema(ctx)
if err != nil {
log.Fatal(err)
}
// Or get specific object types
tables, err := client.GetTables(ctx)
if err != nil {
log.Fatal(err)
}
// Use schema extractor for more control
extractor := client.NewSchemaExtractor()
databases, err := extractor.ExtractDatabases(ctx)
if err != nil {
log.Fatal(err)
}
// Execute a migration
err = client.ExecuteMigration(ctx, migrationSQL)
if err != nil {
log.Fatal(err)
}
The client automatically handles:
- Connection pooling and management
- Proper reconstruction of DDL statements from system tables
- Validation of generated DDL using the built-in parser
- Cluster configuration detection and preservation
Index ¶
- func DumpSchema(ctx context.Context, client *Client) (*parser.SQL, error)
- type Client
- func (c *Client) Close() error
- func (c *Client) Exec(ctx context.Context, query string, args ...any) error
- func (c *Client) ExecuteMigration(ctx context.Context, sql string) error
- func (c *Client) GetDatabases(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetDictionaries(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetFunctions(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetNamedCollections(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetRoles(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetSchema(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetTables(ctx context.Context) (*parser.SQL, error)
- func (c *Client) GetVersion(ctx context.Context) (*VersionInfo, error)
- func (c *Client) GetViews(ctx context.Context) (*parser.SQL, error)
- func (c *Client) Query(ctx context.Context, query string, args ...any) (driver.Rows, error)
- type ClientOptions
- type VersionInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DumpSchema ¶
DumpSchema retrieves all schema objects (databases, tables, named collections, dictionaries, views, roles, functions) and returns them as a parsed SQL structure ready for use with migration generation.
This function combines all individual extraction functions to provide a complete view of the ClickHouse schema. It's the primary function for getting schema information for migration generation and schema comparison operations.
The extraction follows this order:
- Databases - extracted first as they define the namespace
- Tables - extracted with full DDL statements
- Named Collections - connection configurations that dictionaries might reference
- Dictionaries - dictionary definitions with source/layout/lifetime
- Views - both regular and materialized views (extracted last since they may depend on dictionaries)
- Roles - global role definitions and privilege grants
- Functions - user-defined function definitions (global objects)
All system objects are automatically excluded and all DDL statements are validated.
Example:
client, err := clickhouse.NewClient(ctx, "localhost:9000")
if err != nil {
log.Fatal(err)
}
defer client.Close()
schema, err := clickhouse.DumpSchema(ctx, client)
if err != nil {
log.Fatalf("Failed to get schema: %v", err)
}
// Use with diff generation
diff, err := schema.GenerateDiff(currentSchema, targetSchema)
if err != nil {
log.Fatal(err)
}
Returns a parser.SQL containing all schema objects or an error if extraction fails.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client represents a ClickHouse database connection
func NewClient ¶
NewClient creates a new ClickHouse client connection using a DSN. The DSN can be in various formats supported by ClickHouse, including:
- Simple host:port format: "localhost:9000"
- Full DSN: "clickhouse://username:password@host:port/database?param=value"
- Native protocol: "tcp://host:port?username=default&password=&database=default"
Example:
// Simple host:port format
client, err := clickhouse.NewClient(ctx, "localhost:9000")
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Full DSN with authentication and database
client, err := clickhouse.NewClient(ctx, "clickhouse://user:pass@localhost:9000/analytics?dial_timeout=10s")
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Get current schema (databases and dictionaries)
sql, err := client.GetSchema(context.Background())
if err != nil {
log.Fatal(err)
}
// Print all parsed statements
for _, stmt := range sql.Statements {
if stmt.CreateDatabase != nil {
fmt.Printf("Database: %s\n", stmt.CreateDatabase.Name)
}
if stmt.CreateDictionary != nil {
name := stmt.CreateDictionary.Name
if stmt.CreateDictionary.Database != nil {
name = *stmt.CreateDictionary.Database + "." + name
}
fmt.Printf("Dictionary: %s\n", name)
}
}
func NewClientWithOptions ¶
func NewClientWithOptions(ctx context.Context, dsn string, clientOpts ClientOptions) (*Client, error)
NewClientWithOptions creates a new ClickHouse client connection with custom options. This provides additional configuration options beyond the basic DSN connection.
Example:
// Client with cluster support for distributed deployments
client, err := clickhouse.NewClientWithOptions(ctx, "localhost:9000", clickhouse.ClientOptions{
Cluster: "production_cluster",
})
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Dumped schema will include ON CLUSTER clauses
schema, err := client.GetSchema(ctx)
// All DDL statements will have "ON CLUSTER production_cluster"
func (*Client) Close ¶
Close closes the ClickHouse connection and releases associated resources. This method should be called when the client is no longer needed to ensure proper cleanup of database connections.
Example:
client, err := clickhouse.NewClient(ctx, "localhost:9000")
if err != nil {
log.Fatal(err)
}
defer client.Close() // Ensure connection is closed
Returns an error if the connection cleanup fails.
func (*Client) Exec ¶
Exec executes a query without returning any rows, such as INSERT, UPDATE, DELETE, or DDL statements. This method is compatible with the executor.ClickHouse interface.
func (*Client) ExecuteMigration ¶
ExecuteMigration executes a migration SQL script against the ClickHouse database. The SQL can contain multiple DDL statements separated by semicolons, including database and dictionary operations such as CREATE, ALTER, DROP, and RENAME.
This method is typically used to apply up/down migration files generated by the migration system. All statements are executed within a single connection context for consistency.
Example:
migrationSQL := `
CREATE DATABASE analytics ENGINE = Atomic;
CREATE DICTIONARY analytics.users (id UInt64) PRIMARY KEY id
SOURCE(HTTP(url 'http://api.example.com/users')) LAYOUT(HASHED()) LIFETIME(3600);
RENAME DATABASE old_analytics TO analytics_backup;
`
err := client.ExecuteMigration(ctx, migrationSQL)
if err != nil {
log.Fatalf("Migration failed: %v", err)
}
Returns an error if any statement in the migration fails to execute.
func (*Client) GetDatabases ¶
GetDatabases retrieves all database definitions from the ClickHouse instance. This method extracts complete CREATE DATABASE statements including engine specifications, parameters, and comments. System databases are automatically excluded from the results. When the client is configured with a cluster, ON CLUSTER clauses are automatically injected into all extracted DDL statements.
Example:
databases, err := client.GetDatabases(ctx)
if err != nil {
log.Fatal(err)
}
for _, stmt := range databases.Statements {
if stmt.CreateDatabase != nil {
fmt.Printf("Database: %s", stmt.CreateDatabase.Name)
if stmt.CreateDatabase.Engine != nil {
fmt.Printf(" (Engine: %s)", stmt.CreateDatabase.Engine.Name)
}
fmt.Println()
}
}
Returns a *parser.SQL containing validated database CREATE statements or an error if extraction fails.
func (*Client) GetDictionaries ¶
GetDictionaries retrieves all dictionary definitions from the ClickHouse instance. This method extracts complete CREATE DICTIONARY statements including column definitions, source configurations, layout specifications, lifetime settings, and comments. When the client is configured with a cluster, ON CLUSTER clauses are automatically injected into all extracted DDL statements.
The extraction includes all dictionary features:
- Column attributes (IS_OBJECT_ID, HIERARCHICAL, INJECTIVE)
- Source configurations (MySQL, HTTP, ClickHouse, File, etc.)
- Layout types (FLAT, HASHED, COMPLEX_KEY_HASHED, etc.)
- Lifetime configurations (single values or MIN/MAX ranges)
- Settings and comments
Returns a *parser.SQL containing validated dictionary CREATE statements or an error if extraction fails.
func (*Client) GetFunctions ¶ added in v0.1.2
GetFunctions retrieves all user-defined function definitions from the ClickHouse instance. It queries the system.functions table and reconstructs CREATE FUNCTION statements for user-defined functions only (system functions are filtered out).
Uses the origin = 'SQLUserDefined' column for precise filtering, which is available in all supported ClickHouse versions (24.0+).
Returns a *parser.SQL containing all function CREATE statements, or an error if the query fails.
func (*Client) GetNamedCollections ¶ added in v0.1.1
GetNamedCollections retrieves all named collection definitions from the ClickHouse instance
func (*Client) GetRoles ¶ added in v0.1.2
GetRoles retrieves all role definitions from the ClickHouse instance. It queries the system.roles table and reconstructs CREATE ROLE statements with their settings and cluster configuration.
Returns a *parser.SQL containing all role CREATE statements, or an error if the query fails.
func (*Client) GetSchema ¶
GetSchema returns complete schema information including databases, tables, views, and dictionaries. This is the primary method for retrieving the current ClickHouse schema state, including all schema objects, parsed into a structured SQL object.
This method uses dedicated extraction functions for comprehensive schema extraction, providing better performance and more complete schema information than the legacy database+dictionary-only approach.
System objects are automatically excluded from results. All returned DDL statements are validated by the parser before being included.
Example:
schema, err := client.GetSchema(ctx)
if err != nil {
log.Fatalf("Failed to get schema: %v", err)
}
// Process all schema objects
for _, stmt := range schema.Statements {
if stmt.CreateDatabase != nil {
fmt.Printf("Database: %s\n", stmt.CreateDatabase.Name)
}
if stmt.CreateTable != nil {
name := stmt.CreateTable.Name
if stmt.CreateTable.Database != nil {
name = *stmt.CreateTable.Database + "." + name
}
fmt.Printf("Table: %s\n", name)
}
if stmt.CreateView != nil {
viewType := "VIEW"
if stmt.CreateView.Materialized {
viewType = "MATERIALIZED VIEW"
}
name := stmt.CreateView.Name
if stmt.CreateView.Database != nil {
name = *stmt.CreateView.Database + "." + name
}
fmt.Printf("%s: %s\n", viewType, name)
}
if stmt.CreateDictionary != nil {
name := stmt.CreateDictionary.Name
if stmt.CreateDictionary.Database != nil {
name = *stmt.CreateDictionary.Database + "." + name
}
fmt.Printf("Dictionary: %s\n", name)
}
}
Returns a parsed SQL containing all schema objects or an error if retrieval fails.
func (*Client) GetTables ¶
GetTables retrieves all table definitions from the ClickHouse instance. This method extracts complete CREATE TABLE statements for all non-system tables, including column definitions, engine specifications, and table-level settings. When the client is configured with a cluster, ON CLUSTER clauses are automatically injected into all extracted DDL statements.
The extraction excludes:
- System tables and databases
- Temporary tables
- Internal materialized view tables (.inner.* and .inner_id.*)
Example:
client, err := clickhouse.NewClientWithOptions(ctx, "localhost:9000", clickhouse.ClientOptions{
Cluster: "production",
})
if err != nil {
log.Fatal(err)
}
defer client.Close()
tables, err := client.GetTables(ctx)
if err != nil {
log.Fatal(err)
}
// All table statements will include "ON CLUSTER production"
for _, stmt := range tables.Statements {
if stmt.CreateTable != nil {
fmt.Printf("Table: %s\n", stmt.CreateTable.Name)
}
}
Returns a *parser.SQL containing validated table CREATE statements or an error if extraction fails.
func (*Client) GetVersion ¶ added in v0.1.2
func (c *Client) GetVersion(ctx context.Context) (*VersionInfo, error)
GetVersion retrieves and parses the ClickHouse version from the server
func (*Client) GetViews ¶
GetViews retrieves all view definitions (both regular and materialized) from the ClickHouse instance. This method extracts complete CREATE VIEW and CREATE MATERIALIZED VIEW statements, including SELECT queries, engine specifications for materialized views, and population options. When the client is configured with a cluster, ON CLUSTER clauses are automatically injected into all extracted DDL statements.
The extraction includes:
- Regular views (CREATE VIEW statements)
- Materialized views (CREATE MATERIALIZED VIEW statements)
- Both ENGINE-based and TO table materialized views
Example:
client, err := clickhouse.NewClient(ctx, "localhost:9000")
if err != nil {
log.Fatal(err)
}
defer client.Close()
views, err := client.GetViews(ctx)
if err != nil {
log.Fatal(err)
}
for _, stmt := range views.Statements {
if stmt.CreateView != nil {
viewType := "VIEW"
if stmt.CreateView.Materialized {
viewType = "MATERIALIZED VIEW"
}
fmt.Printf("%s: %s\n", viewType, stmt.CreateView.Name)
}
}
Returns a *parser.SQL containing validated view CREATE statements or an error if extraction fails.
type ClientOptions ¶
type ClientOptions struct {
// Cluster specifies the cluster name to inject into ON CLUSTER clauses
// in dumped DDL statements. When set, all extracted DDL will include
// "ON CLUSTER <cluster_name>" to support distributed ClickHouse deployments.
Cluster string
// IgnoreDatabases specifies a list of database names to exclude from schema operations.
// These databases will be ignored during GetSchema, GetDatabases, GetTables, GetViews,
// and GetDictionaries operations. This is useful for excluding test or temporary databases.
IgnoreDatabases []string
}
ClientOptions contains configuration options for the ClickHouse client
type VersionInfo ¶ added in v0.1.2
type VersionInfo struct {
Major int // Major version number (e.g., 21)
Minor int // Minor version number (e.g., 10)
Patch int // Patch version number (e.g., 3)
Raw string // Raw version string from ClickHouse
}
VersionInfo represents parsed ClickHouse version information
func (VersionInfo) IsAtLeast ¶ added in v0.1.2
func (v VersionInfo) IsAtLeast(major, minor int) bool
IsAtLeast checks if this version is at least the specified version
func (VersionInfo) String ¶ added in v0.1.2
func (v VersionInfo) String() string
String returns the version as a string in format "major.minor.patch"
func (VersionInfo) SupportsOriginColumn ¶ added in v0.1.2
func (v VersionInfo) SupportsOriginColumn() bool
SupportsOriginColumn returns true if this ClickHouse version supports the origin column in system.functions This feature was introduced in ClickHouse 21.10
Source Files