Documentation
¶
Overview ¶
Package faithful copies a SQL database between SQLite and PostgreSQL and keeps everything the two engines can both express: tables, columns, nullability, primary keys (composite too), foreign keys (composite too, with their ON DELETE / ON UPDATE actions), unique constraints, indexes, check constraints, defaults, auto-increment columns, and every value byte for byte.
What an engine cannot carry is never dropped silently: every object that is not carried, and every column whose type had to be widened, is a Warning of the Report (text for people, JSON for machines).
The package works on database/sql handles and reads each engine's own catalog (PRAGMA tables for SQLite, pg_catalog for PostgreSQL) because the engine-neutral DALgo dbschema model has no place for a declared column type, a check constraint or a default text, which is what a faithful copy needs.
Index ¶
- Constants
- Variables
- func MySQLNameFold(s string) string
- func WriteScript(ctx context.Context, w io.Writer, src Queryer, p *ScriptPlan, h Header) (int64, error)
- type Catalog
- type Check
- type Choice
- type Column
- type ColumnUse
- type Cond
- type ConflictError
- type Dialect
- type Diff
- type Endpoint
- type EndpointInfo
- type Engine
- type ForeignKey
- type Header
- type Identity
- type Index
- type IndexColumn
- type Kind
- type Marker
- type NameError
- type Options
- type Plan
- type Queryer
- type RawObject
- type ReloadMismatchError
- type Report
- type RowFilter
- type ScriptColumn
- type ScriptPlan
- type ScriptRequest
- type ScriptStats
- type ScriptTable
- type ScriptUsageError
- type Table
- type TableReport
- type TargetColumn
- type TargetTable
- type Totals
- type Unique
- type Warning
Constants ¶
const ( // WarnKeyTypeBounded: a text or blob column of a key got a bounded type (an engine cannot index an unbounded one). WarnKeyTypeBounded = "key-type-bounded" // WarnRowSizeBounded: VARCHAR columns became TEXT so a row fits the engine's row size limit. WarnRowSizeBounded = "row-size-bounded" // WarnFKActionNotCarried: a foreign key is carried, one of its actions or its deferral is not. WarnFKActionNotCarried = "fk-action-not-carried" // WarnNameRenamed: a table or column name the engine cannot store was written under another name. WarnNameRenamed = "name-renamed" // WarnNameCaseCollision: two names differ only by case and collide on an engine with case-insensitive names. WarnNameCaseCollision = "name-case-collision" )
Warning kinds that only the script export raises.
const ( WarnViewNotCarried = "view-not-carried" WarnTriggerNotCarried = "trigger-not-carried" WarnTypeWidened = "type-widened" WarnTypeApproximated = "type-approximated" WarnDefaultNotCarried = "default-not-carried" WarnDefaultTranslated = "default-translated" WarnIndexNotCarried = "index-not-carried" WarnCheckNotCarried = "check-not-carried" WarnFKNotValidated = "fk-not-validated" WarnFKNotCarried = "fk-not-carried" WarnIdentityNotCarried = "identity-not-carried" WarnGeneratedColumn = "generated-column" WarnCollationNotCarried = "collation-not-carried" WarnStorageOption = "storage-option-not-carried" WarnCommentNotCarried = "comment-not-carried" WarnObjectNotCarried = "object-not-carried" WarnConstraintNameLost = "constraint-name-not-carried" WarnIndexRenamed = "index-renamed" WarnValuesNormalized = "values-normalized" WarnForeignKeyViolations = "fk-violations-in-source" WarnNarrowedOnReturn = "type-narrowed" )
Warning kinds. Every feature that is not carried has its own kind so a machine can filter on it.
const MySQLMarkerTable = "_demodb_import"
MySQLMarkerTable is the table the script creates first (it fails when the table is there, so a second load into the same database stops at once) and fills last (a row means the load completed).
const WarnKeyValuesCollide = "key-values-collide"
WarnKeyValuesCollide: key values that SQLite keeps apart are equal to the target's collation, so a unique key refuses the second one and stops the load.
Variables ¶
var ErrNoTables = errors.New("source has no tables; nothing to copy")
ErrNoTables says the source holds no table to copy.
Functions ¶
func MySQLNameFold ¶ added in v0.97.1
MySQLNameFold is foldName for the verifier of the export: a name as MySQL 8.4 compares the names of columns, indexes and checks.
func WriteScript ¶ added in v0.95.0
func WriteScript(ctx context.Context, w io.Writer, src Queryer, p *ScriptPlan, h Header) (int64, error)
WriteScript streams the script of a plan: the header, the session settings, every table, its rows (in a total order, so the bytes are the same on every run), the counters of the auto-numbered columns, then indexes, checks and foreign keys (keys follow the data), and last the marker row. The rows are read from src one table at a time. It returns the number of rows written.
Every line starts with a token this function wrote; no value can start one (see Dialect.Literal).
Types ¶
type Catalog ¶
type Catalog struct {
Engine Engine
Schema string
Tables []*Table
// Skipped names objects the reader saw and cannot carry, as warnings.
Notes []Warning
// Objects are the stored definitions of the views, triggers and indexes of a SQLite database, in creation order.
Objects []RawObject
}
Catalog is everything read from a source database.
func ReadPostgres ¶
ReadPostgres reads the catalog of one PostgreSQL schema (the caller says which: Endpoint.Schema defaults to public).
func ReadSQLite ¶
ReadSQLite reads the catalog of a SQLite database: its tables with columns, primary keys, foreign keys, unique constraints, indexes and check constraints. Views, triggers, virtual tables and the like are recorded as notes (warnings) because they cannot be carried across engines.
type Choice ¶ added in v0.95.0
type Choice struct {
Kind Kind
// Family is the storage family: int, float, decimal, text, bytes, date, datetime, time.
Family string
Type string
// Bits is the width of an int (16, 32, 64). Len is the character length of a bounded text or the byte
// length of a bounded binary, 0 for an unbounded type. Prec and Scale of a decimal; Fsp of a time.
Bits, Len, Prec, Scale, Fsp int
// Note is why the declared type could not be carried, "" when it was.
Note string
// Normalized is how many text values are spelled differently by the target (decimal text), with the first as Example.
Normalized int64
Example string
// Bounded marks a key column given a bounded type that was not declared so.
Bounded bool
// Tight is the shortest length that holds every value (characters of a text, bytes of a binary), at least 1.
Tight int
// MaxSize is the longest value in bytes; it decides the TEXT or BLOB size class of an unbounded type.
MaxSize int
}
Choice is the target type of a column and what is needed to write its values.
type Column ¶
type Column struct {
Name string
// SourceType is the declared type text in the source engine, as the engine reports it.
SourceType string
Kind Kind
// Special is a PostgreSQL type with no logical kind of its own that is carried as text: json, jsonb, uuid.
Special string
// Width is the byte width of an integer (2, 4, 8) or float (4, 8) when the source declares one.
Width int
// Length is the character length of a varchar / char column, 0 when none.
Length int
// Fixed marks a blank-padded char(n).
Fixed bool
// Precision and Scale of a decimal, -1 when the source declares none.
Precision, Scale int
NotNull bool
// Default is the source default expression as the source reports it, "" for none.
Default string
Identity Identity
// Marker is what an earlier datatug copy recorded on this column (PostgreSQL comment).
Marker *Marker
// Comment is a PostgreSQL column comment that is not a datatug marker.
Comment string
// Generated is the expression of a generated column, "" for an ordinary column.
Generated string
// SequenceStart is the next value of an identity / autoincrement counter, 0 when unknown.
SequenceStart int64
}
Column is a column as the source engine declares it.
type ColumnUse ¶ added in v0.95.0
type ColumnUse struct{ PK, Unique, Index, FK bool }
ColumnUse tells Choose where a column is used.
type Cond ¶
type Cond struct{ Field, Op, Value string }
Cond is one --where predicate of a table: <field> <op> <value>. Op is a token of the filter vocabulary (=, <, <=, >, >=, in); Value is the text the user gave.
type ConflictError ¶
type ConflictError struct{ Tables []string }
ConflictError says the target already holds tables of the source.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
type Dialect ¶ added in v0.95.0
type Dialect interface {
// Name is the --dialect value ("mysql").
Name() string
// QuoteIdent quotes a name that StorableName accepted.
QuoteIdent(name string) string
// StorableName says why the engine cannot store the name, "" when it can.
StorableName(name string) string
// MaxIdentLen is the longest name in characters.
MaxIdentLen() int
// Choose picks the column type for the values the pre-scan saw.
Choose(c *Column, st *ScriptStats, use ColumnUse) Choice
// Adjust applies the rules that need the whole plan: unification of foreign key columns, key and row size
// budgets, foreign key and index support. It may add warnings and may refuse (an error).
Adjust(p *ScriptPlan) error
// Default translates a SQLite default expression for the column; note is a warning text ("" for none).
Default(expr string, ch Choice) (def string, note string, ok bool)
// Check translates a SQLite CHECK expression; ok is false when it cannot, with the reason in note.
Check(expr string, t *ScriptTable) (out string, note string, ok bool)
// Literal is the one place where a value becomes SQL text.
Literal(ch Choice, v any) (string, error)
// Statements.
CreateTable(t *ScriptTable) string
IndexSQL(t *ScriptTable, ix Index) string
CheckSQL(t *ScriptTable, ck Check) string
FKSQL(p *ScriptPlan, t *ScriptTable, fk ForeignKey) string
CounterSQL(t *ScriptTable, c *ScriptColumn, next int64) string
// Script framing.
Header(p *ScriptPlan, h Header) []string
Prelude(p *ScriptPlan, h Header) []string
Epilogue(p *ScriptPlan, h Header) []string
BeginData(t *ScriptTable) []string
EndData(t *ScriptTable) []string
// InsertHead is the first line of a multi-row insert; rows follow one per line, "," between and ";" after.
InsertHead(t *ScriptTable) string
// BatchLimits are the most rows and bytes of one insert statement.
BatchLimits() (rows, bytes int)
}
Dialect is a target engine of the script export. Every method is pure: a dialect holds no connection.
func DialectByName ¶ added in v0.95.0
DialectByName returns the dialect called name.
type Diff ¶
type Diff struct {
Object string `json:"object"`
Aspect string `json:"aspect"`
Left string `json:"left"`
Right string `json:"right"`
}
Diff is one difference between two databases of the same engine.
func Verify ¶
Verify compares two databases of the same engine at catalog level and data level: tables, columns (declared type, nullability, default), primary keys, foreign keys (columns, referenced table and columns, actions), unique constraints, indexes, check constraints, row counts, and a digest of every table's rows (so every value, and in SQLite its storage class, is compared).
type Endpoint ¶
type Endpoint struct {
Engine Engine
DB *sql.DB
Display string
// Schema is the PostgreSQL schema read from (source) or written to (target); "public" when empty.
Schema string
}
Endpoint is one side of a copy.
type EndpointInfo ¶
type EndpointInfo struct {
Engine Engine `json:"engine"`
Display string `json:"display"`
Schema string `json:"schema,omitempty"`
}
EndpointInfo describes one side of the copy without secrets.
type ForeignKey ¶
type ForeignKey struct {
Name string
Columns []string
RefTable string
RefColumns []string
OnUpdate string // NO ACTION, RESTRICT, CASCADE, SET NULL, SET DEFAULT
OnDelete string
Deferrable bool
Deferred bool
NotValid bool
Match string // "FULL" when the key is MATCH FULL (PostgreSQL), else ""
}
ForeignKey is a foreign key constraint.
type Header ¶ added in v0.95.0
type Header struct {
// Name is the dataset name, Edition the contract edition label, License and Attribution as the provider gives them.
Name, Edition, License, Attribution string
SourceSHA256 string
Version string
}
Header is what the script says about itself.
type Identity ¶
type Identity int
Identity is how a column generates its own value.
const ( IdentityNone Identity = iota // IdentityRowid is a SQLite INTEGER PRIMARY KEY (an alias of the rowid). IdentityRowid // IdentityAutoincrement is a SQLite INTEGER PRIMARY KEY AUTOINCREMENT. IdentityAutoincrement // IdentityGenerated is a PostgreSQL identity column or serial. IdentityGenerated )
The ways a column generates its value.
type Index ¶
type Index struct {
Name string
Unique bool
Columns []IndexColumn
// Where is the predicate of a partial index in source dialect, "" for a full index.
Where string
}
Index is a plain secondary index.
type IndexColumn ¶
IndexColumn is one key column of an index.
type Kind ¶
type Kind int
Kind is the logical kind of a column, the vocabulary both directions of the type mapping meet in.
type Marker ¶
type Marker struct {
Src string `json:"src"`
Type string `json:"type"`
Identity string `json:"identity,omitempty"` // "rowid" | "autoincrement"
// Carried is the PostgreSQL type text the column was given, so a later copy can tell a widened column.
Widened string `json:"widened,omitempty"`
}
Marker is what a copy records on a PostgreSQL column so the column can go back to SQLite as it was.
type NameError ¶
type NameError struct{ Object, Reason string }
NameError says a name cannot be written on the target.
type Options ¶
type Options struct {
// Overwrite is "" (the target must hold none of the source's tables), "recreate" (drop them first) or
// "reload" (empty them and load the rows again; their schema stays).
Overwrite string
Include, Exclude []string
// Filters is the --where / --limit of each table, by source table name.
Filters map[string]RowFilter
// DecimalsText keeps a DECIMAL_TEXT column whose texts numeric would spell differently as text
// (--decimals=text); by default such a column is numeric and the change of spelling is a warning.
DecimalsText bool
DryRun bool
Progress io.Writer
}
Options controls a copy.
type Plan ¶
type Plan struct {
From, To Engine
Schema string // target schema (PostgreSQL)
Tables []*TargetTable
Warnings []Warning
}
Plan is the whole copy as it will be written.
func PlanPGToSQLite ¶
PlanPGToSQLite decides how every table of a PostgreSQL catalog is created in SQLite. checkOK reports whether SQLite accepts a CHECK expression over the given columns.
func PlanSQLiteToPG ¶
func PlanSQLiteToPG(cat *Catalog, stats map[string][]*colStats, schema string, decimalsText bool) *Plan
PlanSQLiteToPG decides how every table, column and constraint of a SQLite catalog is created on PostgreSQL. stats holds the pre-scan of every column, by table.
func PlanSQLiteToSQLite ¶
PlanSQLiteToSQLite plans a copy between two SQLite databases: the tables are created from the very statements the source stores (so declared types, defaults, collations, checks, foreign keys and storage options are exactly the source's) and the data is copied value for value.
type Queryer ¶
type Queryer interface {
QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
}
Queryer is what the catalog readers need: a *sql.DB, a *sql.Conn or a *sql.Tx.
type RawObject ¶
type RawObject struct {
Type, Name, Table, SQL string
}
RawObject is a SQLite schema object as sqlite_master stores it.
type ReloadMismatchError ¶
ReloadMismatchError says a table the copy would reload has a schema the source rows do not fit.
func (*ReloadMismatchError) Error ¶
func (e *ReloadMismatchError) Error() string
type Report ¶
type Report struct {
Command string `json:"command"`
Status string `json:"status"` // ok | ok-with-warnings | dry-run | failed
DryRun bool `json:"dryRun"`
From EndpointInfo `json:"from"`
To EndpointInfo `json:"to"`
Tables []TableReport `json:"tables"`
Totals Totals `json:"totals"`
// Warnings lists every feature that was not carried exactly, per object.
Warnings []Warning `json:"warnings"`
// Plan is the DDL that is (dry-run: would be) executed.
Plan []string `json:"plan,omitempty"`
ExitCode int `json:"exitCode"`
Error string `json:"error,omitempty"`
}
Report is the machine-readable summary of a copy (the JSON of `db copy --json`).
func Copy ¶
Copy copies the source into the target and returns the report. The report is returned with ExitCode and Error set on failure as well. Nothing on the target changes unless the whole copy succeeds (it runs in one transaction on both engines), and the source is read as one snapshot (one transaction on both engines).
func ExportScript ¶ added in v0.95.0
ExportScript reads the SQLite database of the request in one read transaction, plans the script and, unless it is a dry run, writes it to w. The report has the same shape as the report of a copy.
type RowFilter ¶
RowFilter is what --where and --limit say about one table: every condition must hold (AND) and at most Limit rows are copied (0: no limit).
type ScriptColumn ¶ added in v0.95.0
type ScriptColumn struct {
// Object names the column in a warning: table.column.
Object string
Src *Column
Name string
Choice Choice
NotNull bool
Default string
AutoInc bool
Stats *ScriptStats
Use ColumnUse
}
ScriptColumn is a column as the script creates it.
type ScriptPlan ¶ added in v0.95.0
type ScriptPlan struct {
Dialect Dialect
Tables []*ScriptTable
Warnings []Warning
}
ScriptPlan is the whole script as it will be written.
func PlanSQLiteTo ¶ added in v0.95.0
func PlanSQLiteTo(ctx context.Context, d Dialect, db Queryer, cat *Catalog, stats map[string][]*ScriptStats) (*ScriptPlan, error)
PlanSQLiteTo decides how every table of a SQLite catalog is created by a script of the dialect. stats is the pre-scan of every column, by table. The plan is deterministic.
func PlanScript ¶ added in v0.95.0
PlanScript reads and plans without writing; the cross-engine verifier uses it to know what the script created.
func (*ScriptPlan) Table ¶ added in v0.95.0
func (p *ScriptPlan) Table(name string) *ScriptTable
Table returns the target table called name, or nil.
type ScriptRequest ¶ added in v0.95.0
type ScriptRequest struct {
Dialect string
DB *sql.DB
Display string
Header Header
// DryRun plans and reports, and writes nothing.
DryRun bool
// Include and Exclude name the tables to write (all when both are empty).
Include, Exclude []string
}
ScriptRequest is one SQL-script export.
type ScriptStats ¶ added in v0.95.0
type ScriptStats struct {
MaxBytes int
MaxBlob int
BadUTF8 int64 // text values that are not valid UTF-8
NonFinite int64 // infinite floats
LowYear int64 // dates and timestamps before year 1000
MaxFsp int // most decimals of a timestamp or time text
MaxIntDigits int // most integer digits of a value that is a decimal number
MaxFracDigits int
Empty int64 // empty texts
// RealExample is the first REAL value the column holds, as its shortest decimal ("" when it holds none).
RealExample string
// contains filtered or unexported fields
}
ScriptStats is what the pre-scan of a SQLite column learns for the script export: the statistics the live copy uses, plus the ones a script target needs (byte lengths, text that is not UTF-8, decimal digits, the precision of times). The statistics of the live copy are not changed.
type ScriptTable ¶ added in v0.95.0
type ScriptTable struct {
Src *Table
Name string
Columns []*ScriptColumn
PK []string // target column names
Uniques []Unique
Indexes []Index
Checks []Check
FKs []ForeignKey
Rows int64
// OrderBy is the ORDER BY of the source query: a total order.
OrderBy string
}
ScriptTable is a table as the script creates it.
func (*ScriptTable) Col ¶ added in v0.95.0
func (t *ScriptTable) Col(name string) *ScriptColumn
Col returns the target column called name, or nil.
type ScriptUsageError ¶ added in v0.97.1
type ScriptUsageError struct{ Err error }
ScriptUsageError is a request the user can fix by changing it (a table that does not exist, a header value the script cannot carry): the command exits with 2, not 1.
func (*ScriptUsageError) Error ¶ added in v0.97.1
func (e *ScriptUsageError) Error() string
func (*ScriptUsageError) Unwrap ¶ added in v0.97.1
func (e *ScriptUsageError) Unwrap() error
type Table ¶
type Table struct {
Name string
Columns []*Column
// PKName is the constraint name, "" when the engine does not report one.
PKName string
PK []string
Uniques []Unique
Indexes []Index
Checks []Check
FKs []ForeignKey
// Rows is the row count read from the source.
Rows int64
// CreateSQL is the CREATE TABLE statement of a SQLite table, as stored.
CreateSQL string
}
Table is a table and everything the source declares about it.
type TableReport ¶
type TableReport struct {
Name string `json:"name"`
Rows int64 `json:"rows"`
Columns int `json:"columns"`
PrimaryKey []string `json:"primaryKey"`
ForeignKeys int `json:"foreignKeys"`
Indexes int `json:"indexes"`
UniqueConstraints int `json:"uniqueConstraints"`
CheckConstraints int `json:"checkConstraints"`
}
TableReport is what happened to one table.
type TargetColumn ¶
type TargetColumn struct {
Src *Column
Name string
Type string // type text in the target dialect
NotNull bool
Default string // default expression in the target dialect, "" for none
Identity Identity
Marker *Marker
// Kind is the logical kind the column is written as.
Kind Kind
// Cast is the PostgreSQL type a text parameter is cast to on insert ("" for a parameter of the column's own type).
Cast string
// contains filtered or unexported fields
}
TargetColumn is a column as it is created on the target.
type TargetTable ¶
type TargetTable struct {
Src *Table
Name string
Columns []*TargetColumn
PKName string
PK []string
Uniques []Unique
Indexes []Index
Checks []Check
FKs []ForeignKey
// Copied is the number of rows written, set when the table has been loaded.
Copied int64
// Filter is the --where / --limit of the table, nil for none.
Filter *compiledFilter
// Loaded says the table's rows have been written (Copied is then the number).
Loaded bool
// Reload says the table exists on the target and keeps its schema: it is emptied and loaded again.
Reload bool
}
TargetTable is a table as it is created on the target.
type Totals ¶
type Totals struct {
Tables int `json:"tables"`
Rows int64 `json:"rows"`
Columns int `json:"columns"`
PrimaryKeys int `json:"primaryKeys"`
ForeignKeys int `json:"foreignKeys"`
Indexes int `json:"indexes"`
UniqueConstraints int `json:"uniqueConstraints"`
CheckConstraints int `json:"checkConstraints"`
}
Totals are the sums over all tables.
Source Files
¶
- copy.go
- copy_names.go
- decl.go
- dialect.go
- jsonb.go
- model.go
- mysql_ddl.go
- mysql_fkfold.go
- mysql_fold.go
- mysql_sql.go
- mysql_types.go
- pg_read.go
- plan.go
- plan_pg.go
- plan_same.go
- plan_sqlite.go
- report.go
- rowfilter.go
- script_collide.go
- script_names.go
- script_plan.go
- script_stats.go
- script_write.go
- sqlite_read.go
- sqlscan.go
- stats.go
- verify.go