squasher

package
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 9, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

Package squasher provides provenance tracking for migration squashing operations. It generates .squashmap.json files that map original migrations to squashed output, enabling traceability and auditability.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CanMergeExtensions

func CanMergeExtensions(ext1, ext2 ExtensionRef) bool

CanMergeExtensions checks if two extension references can be safely merged

func Deparse

func Deparse(tree *pg_query.ParseResult) (string, error)

Deparse takes a modified pg_query.ParseResult and generates a SQL string. This is the primary interface for converting AST back to SQL. Formats deparsed SQL to ensure proper spacing and readability.

func DeparseWithStatement

func DeparseWithStatement(stmt *types.Statement) (string, error)

DeparseWithStatement takes a Statement and its ParseTree, and generates SQL. Clears implicit AccessMethod="btree" from indexes to preserve original semantics. Preserves function volatility markers (STABLE/VOLATILE/IMMUTABLE) during deparsing.

func IsValidSafetyLevel

func IsValidSafetyLevel(level SafetyLevel) bool

IsValidSafetyLevel checks if a SafetyLevel is valid

func NewSquasherRuleEngine

func NewSquasherRuleEngine(safetyLevel SafetyLevel, ruleOverrides map[string]bool) (*consolidation.ConsolidationRuleEngine, error)

NewSquasherRuleEngine creates a rule engine for the given safety level.

The rule sets form a strict subset ladder (stricter level = fewer rules):

paranoid ⊂ conservative ⊂ standard ⊂ aggressive

ruleOverrides force-enables (true) or force-disables (false) specific named rules relative to that baseline. Names must match the consolidation rule registry (see ValidateRuleOverrides); unknown names are rejected. A nil map applies the baseline unchanged.

Invalid safety levels are rejected with an error instead of silently producing an engine with no consolidation rules.

func ValidateRuleOverrides

func ValidateRuleOverrides(overrides map[string]bool) error

ValidateRuleOverrides checks that every override key names a rule known to the consolidation rule registry (the same catalog served by pkg/rules.GetRegistry). Unknown names are an error, never silently ignored.

func ValidateSafetyLevel

func ValidateSafetyLevel(level SafetyLevel) error

ValidateSafetyLevel returns an error if the SafetyLevel is invalid

Types

type AuthServiceType

type AuthServiceType string

AuthServiceType represents different authentication services

const (
	AuthServiceNone     AuthServiceType = "none"
	AuthServiceClerk    AuthServiceType = "clerk"
	AuthServiceSupabase AuthServiceType = "supabase"
	AuthServiceAuth0    AuthServiceType = "auth0"
)

type CircularFKHandler

type CircularFKHandler struct {
	// contains filtered or unexported fields
}

CircularFKHandler detects and handles circular foreign key dependencies

func NewCircularFKHandler

func NewCircularFKHandler() *CircularFKHandler

NewCircularFKHandler creates a new circular FK handler

func (*CircularFKHandler) DetectCircularDependencies

func (h *CircularFKHandler) DetectCircularDependencies(tables map[string]*types.Statement) [][]string

DetectCircularDependencies finds all circular FK dependencies in a set of tables

func (*CircularFKHandler) RemoveCircularFKsFromTables

func (h *CircularFKHandler) RemoveCircularFKsFromTables(
	tables map[string]*types.Statement,
	cycles [][]string,
) (map[string]*types.Statement, []*types.Statement, error)

RemoveCircularFKsFromTables removes FK constraints that are part of circular dependencies Returns modified table statements and a list of ALTER TABLE statements to add FKs later

type ConsolidationConflict

type ConsolidationConflict struct {
	ObjectName string
	ObjectType types.ObjectType
	Operations []string
	Reason     string
}

ConsolidationConflict represents operations that cannot be consolidated

type ConsolidationPlan

type ConsolidationPlan struct {
	TotalMigrations    int
	TotalOperations    int
	Consolidations     []PlannedConsolidation
	CannotConsolidate  []ConsolidationConflict
	EstimatedReduction ConsolidationStats
}

ConsolidationPlan represents a detailed plan of consolidations to be applied

func (*ConsolidationPlan) FormatPlan

func (p *ConsolidationPlan) FormatPlan() string

FormatPlan generates a human-readable consolidation plan

type ConsolidationStats

type ConsolidationStats struct {
	OriginalFiles      int
	OriginalOperations int
	FinalFiles         int
	FinalOperations    int
	FilesReduced       int
	OperationsReduced  int
	PercentageReduced  float64
}

ConsolidationStats provides summary statistics

type Edge

type Edge struct {
	// contains filtered or unexported fields
}

Edge represents a dependency edge

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine provides comprehensive PostgreSQL migration squashing with modern patterns and streaming support

func NewEngine

func NewEngine(cfg EngineConfig) (*Engine, error)

NewEngine creates an enhanced engine with the provided configuration.

Breaking change: constructor failures now return explicit errors instead of silently returning nil engine pointers.

func (*Engine) Close

func (e *Engine) Close() error

Close gracefully shuts down the engine

func (*Engine) GenerateConsolidationPlan

func (e *Engine) GenerateConsolidationPlan(migrationMap map[int]string) (*ConsolidationPlan, error)

GenerateConsolidationPlan analyzes migrations and creates a detailed consolidation plan

func (*Engine) GetAuthCompatibilitySQL

func (e *Engine) GetAuthCompatibilitySQL() string

GetAuthCompatibilitySQL returns the auth compatibility SQL for Docker validation

func (*Engine) GetConfig

func (e *Engine) GetConfig() *config.Config

GetConfig returns the configuration for use by consolidation rules GetConfig returns the configuration for use by consolidation rules

func (*Engine) GetMemoryStats

func (e *Engine) GetMemoryStats() performance.MemoryStats

GetMemoryStats returns current memory usage statistics (streaming mode only)

func (*Engine) GetSafetyLevel

func (e *Engine) GetSafetyLevel() string

GetSafetyLevel returns the current safety level

func (*Engine) GetStats

func (e *Engine) GetStats() SquashStats

GetStats returns current squashing statistics

func (*Engine) GetTracker

func (e *Engine) GetTracker() *tracking.Tracker

GetTracker returns the tracker for use by consolidation rules

func (*Engine) SetProgressCallback

func (e *Engine) SetProgressCallback(callback func(processed, total int64, phase string))

SetProgressCallback sets a progress callback for the engine

func (*Engine) Squash

func (e *Engine) Squash(migrations map[int]string) (*SquashResult, error)

Squash processes migrations using enhanced patterns and modern PostgreSQL conventions Squash processes migrations using enhanced patterns and modern PostgreSQL conventions

func (*Engine) SquashFromDirectory

func (e *Engine) SquashFromDirectory(dir string) (*SquashResult, error)

SquashFromDirectory processes migrations from a directory using streaming

func (*Engine) SquashStreaming

func (e *Engine) SquashStreaming(migrations map[int]string) (*SquashResult, error)

SquashStreaming processes migrations with streaming approach for large datasets SquashStreaming processes migrations with streaming approach for large datasets

func (*Engine) SquashWithSeparateFiles

func (e *Engine) SquashWithSeparateFiles(migrations map[int]string) (*SquashResult, error)

SquashWithSeparateFiles performs squashing and returns separate files for DDL and data operations

type EngineConfig

type EngineConfig struct {
	Config  *config.Config
	Context context.Context

	// Version is the tool version stamped into provenance metadata
	// (.squashmap.json). Callers set it from their release metadata (e.g. the
	// CLI passes its rootCmd version). Empty falls back to "dev" - never a
	// hardcoded release string.
	Version string

	EnableStreaming     bool
	BatchSize           int
	WorkerCount         int
	MemoryLimitMB       int
	EnableProgressTrack bool
	ProgressCallback    func(processed, total int64, phase string)

	// Transformation options
	EnableBackup         bool
	EnableRollback       bool
	EnableTransformation bool
	BackupConfig         *transformation.BackupConfig
	TransformationConfig *transformation.TransformationConfig
	RollbackPath         string // Directory for rollback scripts
	BackupPath           string // Directory for pg_dump backups (default: <output dir>/.backups)
	BackupRetentionDays  int    // Retention window for old backups (0 = keep forever)

	// Output options
	ExcludeDataFromBaseline bool // If true, filter data operations from baseline SQL (default: false, meaning include)

	EnableCycleDetection bool
	ShowCycleDetails     bool
	CycleDetectionDepth  int
}

EngineConfig configures the engine with optional streaming capabilities

type ExtensionAnalysis

type ExtensionAnalysis struct {
	RequiredExtensions    []string                 // List of extensions found
	ExtensionDetails      map[string]ExtensionInfo // Detailed info for each extension
	RecommendedDockerBase string                   // Best Docker image to use
	InstallationScript    string                   // Script to install extensions
	ValidationScript      string                   // Script to validate extensions
	MissingExtensions     []string                 // Extensions we don't know about
	AuthService           AuthServiceType          // Detected authentication service
	AuthCompatibilitySQL  string                   // SQL to create service compatibility
}

ExtensionAnalysis holds the results of extension detection

type ExtensionDetector

type ExtensionDetector struct {
	// contains filtered or unexported fields
}

ExtensionDetector analyzes migrations to detect required PostgreSQL extensions

func NewExtensionDetector

func NewExtensionDetector() *ExtensionDetector

NewExtensionDetector creates a new extension detector with known extensions

func (*ExtensionDetector) AnalyzeMigrations

func (ed *ExtensionDetector) AnalyzeMigrations(migrations map[int]string) *ExtensionAnalysis

AnalyzeMigrations scans migration content to detect required extensions

func (*ExtensionDetector) DetectExtensionRefs

func (ed *ExtensionDetector) DetectExtensionRefs(content string) []ExtensionRef

DetectExtensionRefs scans SQL content and returns detailed extension references with versions

func (*ExtensionDetector) GenerateDockerfile

func (ed *ExtensionDetector) GenerateDockerfile(analysis *ExtensionAnalysis) string

GenerateDockerfile creates a Dockerfile with required extensions

func (*ExtensionDetector) GenerateInitSQL

func (ed *ExtensionDetector) GenerateInitSQL(analysis *ExtensionAnalysis) string

GenerateInitSQL creates an SQL script to initialize extensions

type ExtensionInfo

type ExtensionInfo struct {
	Name            string   // Extension name
	PackageName     string   // APT package name for installation
	DockerImage     string   // Preferred Docker image that includes this extension
	Dependencies    []string // Other extensions this depends on
	InstallCommand  string   // Custom installation command if needed
	ValidationSQL   string   // SQL to test if extension is available
	RequiresCASCADE bool     // Whether this extension typically needs CASCADE
	RequiresPostGIS bool     // Whether this requires PostGIS base
}

ExtensionInfo holds information about a PostgreSQL extension

type ExtensionRef

type ExtensionRef struct {
	Name    string // Extension name
	Version string // Extension version (e.g., "0.6.0")
	Schema  string // Schema where extension is installed (e.g., "public")
	Line    int    // Line number where extension is defined
}

ExtensionRef represents a specific extension reference with version and schema

func (ExtensionRef) CanMergeWith

func (e ExtensionRef) CanMergeWith(other ExtensionRef) bool

CanMergeWith checks if this extension reference can be merged with another

func (ExtensionRef) Key

func (e ExtensionRef) Key() string

Key returns a unique key for this extension reference

type ForeignKeyConstraint

type ForeignKeyConstraint struct {
	ConstraintName string
	SourceTable    string
	SourceColumns  []string
	RefTable       string
	RefColumns     []string
	OnDelete       string
	OnUpdate       string
	Deferrable     bool
	Initially      string
	OriginalSQL    string // Original SQL for the constraint
}

ForeignKeyConstraint represents a foreign key constraint

type ModernPatternOptimizer

type ModernPatternOptimizer struct {
	// contains filtered or unexported fields
}

ModernPatternOptimizer handles JWT v2, storage, and dynamic SQL patterns

func NewModernPatternOptimizer

func NewModernPatternOptimizer() *ModernPatternOptimizer

NewModernPatternOptimizer creates an optimizer for modern PostgreSQL patterns NOTE: Many auth-specific rules are now handled by plugins (Clerk, Supabase, etc.) These rules provide fallback generic handling.

func (*ModernPatternOptimizer) ApplyModernOptimizations

func (m *ModernPatternOptimizer) ApplyModernOptimizations(statements []*types.Statement, safetyLevel SafetyLevel) []*types.Statement

ApplyModernOptimizations applies pattern-specific optimizations with priority and conflict handling

type ModernPatternRule

type ModernPatternRule struct {
	Name        string
	Priority    int // Higher number = higher priority
	Pattern     string
	AuthType    types.AuthPatternType
	Conflicts   []string    // Rule names that conflict with this one
	SafetyLevel SafetyLevel // Minimum safety level required
	Consolidate func([]*types.Statement) []*types.Statement
}

ModernPatternRule defines optimization rules for modern PostgreSQL patterns

type PlannedConsolidation

type PlannedConsolidation struct {
	ObjectName   string
	ObjectType   types.ObjectType
	ObjectSchema string
	Operations   []string // SQL snippets
	ResultSQL    string
	Rule         string
	Reason       string
	SafetyLevel  string
	RiskLevel    tracking.RiskLevel
}

PlannedConsolidation represents a single consolidation action

type ProvenanceTracker

type ProvenanceTracker struct {
	// contains filtered or unexported fields
}

ProvenanceTracker tracks provenance information during squashing

func NewProvenanceTracker

func NewProvenanceTracker(version, safetyMode, pgVersion string, extensions []string) *ProvenanceTracker

NewProvenanceTracker creates a new provenance tracker

func (*ProvenanceTracker) AddInputFile

func (pt *ProvenanceTracker) AddInputFile(filePath string)

AddInputFile registers an input migration file

func (*ProvenanceTracker) AddOutputFile

func (pt *ProvenanceTracker) AddOutputFile(filePath string)

AddOutputFile registers an output file

func (*ProvenanceTracker) AddWarning

func (pt *ProvenanceTracker) AddWarning(warning string)

AddWarning adds a warning to the provenance map

func (*ProvenanceTracker) ComputeContentHash

func (pt *ProvenanceTracker) ComputeContentHash(content string)

ComputeContentHash computes the SHA256 hash of the squashed output

func (*ProvenanceTracker) GetSquashMap

func (pt *ProvenanceTracker) GetSquashMap() *SquashMap

GetSquashMap returns the current squash map

func (*ProvenanceTracker) RecordMapping

func (pt *ProvenanceTracker) RecordMapping(mapping StatementMapping)

RecordMapping records a statement mapping

func (*ProvenanceTracker) SetCurrentOutput

func (pt *ProvenanceTracker) SetCurrentOutput(filePath string, line int)

SetCurrentOutput sets the current output location

func (*ProvenanceTracker) SetCurrentSource

func (pt *ProvenanceTracker) SetCurrentSource(filePath string, line int)

SetCurrentSource sets the current source file being processed

func (*ProvenanceTracker) SetStatistics

func (pt *ProvenanceTracker) SetStatistics(stats SquashStatistics)

SetStatistics sets the consolidation statistics

func (*ProvenanceTracker) WriteSquashMap

func (pt *ProvenanceTracker) WriteSquashMap(outputDir string) error

WriteSquashMap writes the .squashmap.json file to the specified directory

type SQLDependencyInfo

type SQLDependencyInfo struct {
	ObjectKey     string
	Result        *tracking.ConsolidationResult
	Dependencies  []string // Objects this depends on
	Provides      []string // Objects this provides/creates
	RequiredFirst bool     // Must be first in category (extensions, schemas)
	RequiredLast  bool     // Must be last in category (data operations)
}

SQLDependencyInfo holds dependency information for an object (renamed to avoid conflict)

type SafetyLevel

type SafetyLevel string
const (
	Conservative SafetyLevel = "conservative"
	Standard     SafetyLevel = "standard"
	Aggressive   SafetyLevel = "aggressive"
	Paranoid     SafetyLevel = "paranoid"
)

func ParseSafetyLevel

func ParseSafetyLevel(s string) (SafetyLevel, error)

ParseSafetyLevel converts a string to SafetyLevel with validation

func ValidSafetyLevels

func ValidSafetyLevels() []SafetyLevel

ValidSafetyLevels returns all valid SafetyLevel values

type SquashMap

type SquashMap struct {
	// Metadata
	Version     string    `json:"version"`      // pgsquash version
	Timestamp   time.Time `json:"timestamp"`    // When the squash was performed
	SafetyMode  string    `json:"safety_mode"`  // Safety level used (paranoid, conservative, standard, aggressive)
	PGVersion   string    `json:"pg_version"`   // PostgreSQL version targeted
	Extensions  []string  `json:"extensions"`   // Extensions detected/required
	ContentHash string    `json:"content_hash"` // SHA256 hash of squashed output

	// Input/Output
	Inputs  []string `json:"inputs"`  // List of original migration files
	Outputs []string `json:"outputs"` // List of generated output files

	// Detailed Mappings
	Mappings []StatementMapping `json:"mappings"` // Statement-level mappings

	// Statistics
	Stats SquashStatistics `json:"statistics"` // Consolidation statistics

	// Warnings
	Warnings []string `json:"warnings,omitempty"` // Any warnings during squash
}

SquashMap represents the complete mapping of a squash operation

func LoadSquashMap

func LoadSquashMap(filePath string) (*SquashMap, error)

LoadSquashMap loads a .squashmap.json file from disk

func (*SquashMap) FindMappingForOutput

func (sm *SquashMap) FindMappingForOutput(outputFile string, line int) []StatementMapping

FindMappingForOutput finds all mappings to a specific output location

func (*SquashMap) FindMappingForSource

func (sm *SquashMap) FindMappingForSource(sourceFile string) []StatementMapping

FindMappingForSource finds all mappings from a specific source file

func (*SquashMap) FormatSquashMap

func (sm *SquashMap) FormatSquashMap() string

FormatSquashMap formats the squash map for human-readable display

func (*SquashMap) VerifyContentHash

func (sm *SquashMap) VerifyContentHash(content string) bool

VerifyContentHash verifies the content hash matches the squashed output

type SquashResult

type SquashResult struct {
	BaselineSQL       string     // DDL-only SQL (000_baseline.sql)
	DataOperationsSQL string     // Data operations SQL (010_data.sql)
	Warnings          []string   // Warnings generated during squash
	ProvenanceMap     *SquashMap // Provenance tracking information
	Extensions        []string   // Extensions detected/required

	// Auth compatibility for validation
	AuthCompatibilitySQL string `json:"auth_compatibility_sql,omitempty"`
}

SquashResult represents the result of a squash operation with multiple output files

NOTE: detailed partner-integration metrics (DetailedMetrics/RecommendedActions) live exclusively in pkg/engine (detailed_metrics.go); the internal result carries only what the engine itself computes.

func OptimizedSquashForLargeDatasets

func OptimizedSquashForLargeDatasets(cfg *config.Config, migrations map[int]string, memoryLimitMB int) (*SquashResult, error)

OptimizedSquashForLargeDatasets provides a high-level interface for large migration sets

func OptimizedSquashFromDirectory

func OptimizedSquashFromDirectory(cfg *config.Config, dir string, memoryLimitMB int) (*SquashResult, error)

OptimizedSquashFromDirectory provides a high-level interface for directory processing

type SquashStatistics

type SquashStatistics struct {
	OriginalStatements     int     `json:"original_statements"`     // Total statements in original migrations
	ConsolidatedStatements int     `json:"consolidated_statements"` // Total statements in output
	ReductionRate          float64 `json:"reduction_rate"`          // Percentage reduction
	FilesProcessed         int     `json:"files_processed"`         // Number of input files
	FilesGenerated         int     `json:"files_generated"`         // Number of output files
	TotalLinesOriginal     int     `json:"total_lines_original"`    // Total lines in original
	TotalLinesSquashed     int     `json:"total_lines_squashed"`    // Total lines in output
}

SquashStatistics contains metrics about the squash operation

type SquashStats

type SquashStats struct {
	Phase                 string        `json:"current_phase"`
	MigrationsProcessed   int64         `json:"migrations_processed"`
	TotalMigrations       int64         `json:"total_migrations"`
	ObjectsTracked        int64         `json:"objects_tracked"`
	ConsolidationsApplied int64         `json:"consolidations_applied"`
	ProcessingTime        time.Duration `json:"processing_time"`
	PeakMemoryUsage       int64         `json:"peak_memory_usage"`
	ThroughputMPS         float64       `json:"throughput_migrations_per_sec"`
}

SquashStats tracks squashing statistics (both regular and streaming)

type StatementMapping

type StatementMapping struct {
	SourceFile      string `json:"source_file"`      // Original migration file
	SourceLine      int    `json:"source_line"`      // Line number in source file
	SourceStatement string `json:"source_statement"` // The original SQL statement (first 100 chars)

	OutputFile      string `json:"output_file"`       // Output file name
	OutputLineStart int    `json:"output_line_start"` // Starting line in output
	OutputLineEnd   int    `json:"output_line_end"`   // Ending line in output

	Action     string `json:"action"`      // What happened: "preserved", "merged", "eliminated", "reordered"
	ObjectType string `json:"object_type"` // Type of object (table, index, function, etc.)
	ObjectName string `json:"object_name"` // Name of the object
}

StatementMapping maps an original statement to its location in the output

type UnifiedDependencyResolver

type UnifiedDependencyResolver struct {
	// contains filtered or unexported fields
}

UnifiedDependencyResolver provides comprehensive dependency resolution for both object lifecycle analysis and SQL consolidation phases

func NewUnifiedDependencyResolver

func NewUnifiedDependencyResolver() *UnifiedDependencyResolver

NewUnifiedDependencyResolver creates a new unified dependency resolver

func (*UnifiedDependencyResolver) EnhanceExtensionSQL

func (udr *UnifiedDependencyResolver) EnhanceExtensionSQL(sql string) string

EnhanceExtensionSQL adds CASCADE to extension creation where needed

func (*UnifiedDependencyResolver) ResolveLifecycleDependencies

func (udr *UnifiedDependencyResolver) ResolveLifecycleDependencies(
	graph *tracking.DependencyGraph,
	lifecycles map[string]*tracking.ObjectLifecycle,
) ([]tracking.ObjectID, error)

ResolveLifecycleDependencies performs advanced dependency resolution for object lifecycles This replaces the functionality from EnhancedDependencyResolver

func (*UnifiedDependencyResolver) SortConsolidationResults

func (udr *UnifiedDependencyResolver) SortConsolidationResults(
	categoryObjects map[string]*tracking.ConsolidationResult,
	category types.Category,
	lifecycles map[string]*tracking.ObjectLifecycle,
) []*tracking.ConsolidationResult

SortConsolidationResults sorts consolidated SQL results by their dependencies within a category This replaces the functionality from DependencyResolver

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL