Documentation
¶
Overview ¶
Package faithful copies a SQL database between SQLite and PostgreSQL and keeps everything the two engines can both express: tables, columns, nullability, primary keys (composite too), foreign keys (composite too, with their ON DELETE / ON UPDATE actions), unique constraints, indexes, check constraints, defaults, auto-increment columns, and every value byte for byte.
What an engine cannot carry is never dropped silently: every object that is not carried, and every column whose type had to be widened, is a Warning of the Report (text for people, JSON for machines).
The package works on database/sql handles and reads each engine's own catalog (PRAGMA tables for SQLite, pg_catalog for PostgreSQL) because the engine-neutral DALgo dbschema model has no place for a declared column type, a check constraint or a default text, which is what a faithful copy needs.
Index ¶
- Constants
- Variables
- type Catalog
- type Check
- type Column
- type Cond
- type ConflictError
- type Diff
- type Endpoint
- type EndpointInfo
- type Engine
- type ForeignKey
- type Identity
- type Index
- type IndexColumn
- type Kind
- type Marker
- type NameError
- type Options
- type Plan
- type Queryer
- type RawObject
- type ReloadMismatchError
- type Report
- type RowFilter
- type Table
- type TableReport
- type TargetColumn
- type TargetTable
- type Totals
- type Unique
- type Warning
Constants ¶
const ( WarnViewNotCarried = "view-not-carried" WarnTriggerNotCarried = "trigger-not-carried" WarnTypeWidened = "type-widened" WarnTypeApproximated = "type-approximated" WarnDefaultNotCarried = "default-not-carried" WarnDefaultTranslated = "default-translated" WarnIndexNotCarried = "index-not-carried" WarnCheckNotCarried = "check-not-carried" WarnFKNotValidated = "fk-not-validated" WarnFKNotCarried = "fk-not-carried" WarnIdentityNotCarried = "identity-not-carried" WarnGeneratedColumn = "generated-column" WarnCollationNotCarried = "collation-not-carried" WarnStorageOption = "storage-option-not-carried" WarnCommentNotCarried = "comment-not-carried" WarnObjectNotCarried = "object-not-carried" WarnConstraintNameLost = "constraint-name-not-carried" WarnIndexRenamed = "index-renamed" WarnValuesNormalized = "values-normalized" WarnForeignKeyViolations = "fk-violations-in-source" WarnNarrowedOnReturn = "type-narrowed" )
Warning kinds. Every feature that is not carried has its own kind so a machine can filter on it.
Variables ¶
var ErrNoTables = errors.New("source has no tables; nothing to copy")
ErrNoTables says the source holds no table to copy.
Functions ¶
This section is empty.
Types ¶
type Catalog ¶
type Catalog struct {
Engine Engine
Schema string
Tables []*Table
// Skipped names objects the reader saw and cannot carry, as warnings.
Notes []Warning
// Objects are the stored definitions of the views, triggers and indexes of a SQLite database, in creation order.
Objects []RawObject
}
Catalog is everything read from a source database.
func ReadPostgres ¶
ReadPostgres reads the catalog of one PostgreSQL schema (the caller says which: Endpoint.Schema defaults to public).
func ReadSQLite ¶
ReadSQLite reads the catalog of a SQLite database: its tables with columns, primary keys, foreign keys, unique constraints, indexes and check constraints. Views, triggers, virtual tables and the like are recorded as notes (warnings) because they cannot be carried across engines.
type Column ¶
type Column struct {
Name string
// SourceType is the declared type text in the source engine, as the engine reports it.
SourceType string
Kind Kind
// Special is a PostgreSQL type with no logical kind of its own that is carried as text: json, jsonb, uuid.
Special string
// Width is the byte width of an integer (2, 4, 8) or float (4, 8) when the source declares one.
Width int
// Length is the character length of a varchar / char column, 0 when none.
Length int
// Fixed marks a blank-padded char(n).
Fixed bool
// Precision and Scale of a decimal, -1 when the source declares none.
Precision, Scale int
NotNull bool
// Default is the source default expression as the source reports it, "" for none.
Default string
Identity Identity
// Marker is what an earlier datatug copy recorded on this column (PostgreSQL comment).
Marker *Marker
// Comment is a PostgreSQL column comment that is not a datatug marker.
Comment string
// Generated is the expression of a generated column, "" for an ordinary column.
Generated string
// SequenceStart is the next value of an identity / autoincrement counter, 0 when unknown.
SequenceStart int64
}
Column is a column as the source engine declares it.
type Cond ¶
type Cond struct{ Field, Op, Value string }
Cond is one --where predicate of a table: <field> <op> <value>. Op is a token of the filter vocabulary (=, <, <=, >, >=, in); Value is the text the user gave.
type ConflictError ¶
type ConflictError struct{ Tables []string }
ConflictError says the target already holds tables of the source.
func (*ConflictError) Error ¶
func (e *ConflictError) Error() string
type Diff ¶
type Diff struct {
Object string `json:"object"`
Aspect string `json:"aspect"`
Left string `json:"left"`
Right string `json:"right"`
}
Diff is one difference between two databases of the same engine.
func Verify ¶
Verify compares two databases of the same engine at catalog level and data level: tables, columns (declared type, nullability, default), primary keys, foreign keys (columns, referenced table and columns, actions), unique constraints, indexes, check constraints, row counts, and a digest of every table's rows (so every value, and in SQLite its storage class, is compared).
type Endpoint ¶
type Endpoint struct {
Engine Engine
DB *sql.DB
Display string
// Schema is the PostgreSQL schema read from (source) or written to (target); "public" when empty.
Schema string
}
Endpoint is one side of a copy.
type EndpointInfo ¶
type EndpointInfo struct {
Engine Engine `json:"engine"`
Display string `json:"display"`
Schema string `json:"schema,omitempty"`
}
EndpointInfo describes one side of the copy without secrets.
type ForeignKey ¶
type ForeignKey struct {
Name string
Columns []string
RefTable string
RefColumns []string
OnUpdate string // NO ACTION, RESTRICT, CASCADE, SET NULL, SET DEFAULT
OnDelete string
Deferrable bool
Deferred bool
NotValid bool
Match string // "FULL" when the key is MATCH FULL (PostgreSQL), else ""
}
ForeignKey is a foreign key constraint.
type Identity ¶
type Identity int
Identity is how a column generates its own value.
const ( IdentityNone Identity = iota // IdentityRowid is a SQLite INTEGER PRIMARY KEY (an alias of the rowid). IdentityRowid // IdentityAutoincrement is a SQLite INTEGER PRIMARY KEY AUTOINCREMENT. IdentityAutoincrement // IdentityGenerated is a PostgreSQL identity column or serial. IdentityGenerated )
The ways a column generates its value.
type Index ¶
type Index struct {
Name string
Unique bool
Columns []IndexColumn
// Where is the predicate of a partial index in source dialect, "" for a full index.
Where string
}
Index is a plain secondary index.
type IndexColumn ¶
IndexColumn is one key column of an index.
type Kind ¶
type Kind int
Kind is the logical kind of a column, the vocabulary both directions of the type mapping meet in.
type Marker ¶
type Marker struct {
Src string `json:"src"`
Type string `json:"type"`
Identity string `json:"identity,omitempty"` // "rowid" | "autoincrement"
// Carried is the PostgreSQL type text the column was given, so a later copy can tell a widened column.
Widened string `json:"widened,omitempty"`
}
Marker is what a copy records on a PostgreSQL column so the column can go back to SQLite as it was.
type NameError ¶
type NameError struct{ Object, Reason string }
NameError says a name cannot be written on the target.
type Options ¶
type Options struct {
// Overwrite is "" (the target must hold none of the source's tables), "recreate" (drop them first) or
// "reload" (empty them and load the rows again; their schema stays).
Overwrite string
Include, Exclude []string
// Filters is the --where / --limit of each table, by source table name.
Filters map[string]RowFilter
// DecimalsText keeps a DECIMAL_TEXT column whose texts numeric would spell differently as text
// (--decimals=text); by default such a column is numeric and the change of spelling is a warning.
DecimalsText bool
DryRun bool
Progress io.Writer
}
Options controls a copy.
type Plan ¶
type Plan struct {
From, To Engine
Schema string // target schema (PostgreSQL)
Tables []*TargetTable
Warnings []Warning
}
Plan is the whole copy as it will be written.
func PlanPGToSQLite ¶
PlanPGToSQLite decides how every table of a PostgreSQL catalog is created in SQLite. checkOK reports whether SQLite accepts a CHECK expression over the given columns.
func PlanSQLiteToPG ¶
func PlanSQLiteToPG(cat *Catalog, stats map[string][]*colStats, schema string, decimalsText bool) *Plan
PlanSQLiteToPG decides how every table, column and constraint of a SQLite catalog is created on PostgreSQL. stats holds the pre-scan of every column, by table.
func PlanSQLiteToSQLite ¶
PlanSQLiteToSQLite plans a copy between two SQLite databases: the tables are created from the very statements the source stores (so declared types, defaults, collations, checks, foreign keys and storage options are exactly the source's) and the data is copied value for value.
type Queryer ¶
type Queryer interface {
QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
}
Queryer is what the catalog readers need: a *sql.DB, a *sql.Conn or a *sql.Tx.
type RawObject ¶
type RawObject struct {
Type, Name, Table, SQL string
}
RawObject is a SQLite schema object as sqlite_master stores it.
type ReloadMismatchError ¶
ReloadMismatchError says a table the copy would reload has a schema the source rows do not fit.
func (*ReloadMismatchError) Error ¶
func (e *ReloadMismatchError) Error() string
type Report ¶
type Report struct {
Command string `json:"command"`
Status string `json:"status"` // ok | ok-with-warnings | dry-run | failed
DryRun bool `json:"dryRun"`
From EndpointInfo `json:"from"`
To EndpointInfo `json:"to"`
Tables []TableReport `json:"tables"`
Totals Totals `json:"totals"`
// Warnings lists every feature that was not carried exactly, per object.
Warnings []Warning `json:"warnings"`
// Plan is the DDL that is (dry-run: would be) executed.
Plan []string `json:"plan,omitempty"`
ExitCode int `json:"exitCode"`
Error string `json:"error,omitempty"`
}
Report is the machine-readable summary of a copy (the JSON of `db copy --json`).
func Copy ¶
Copy copies the source into the target and returns the report. The report is returned with ExitCode and Error set on failure as well. Nothing on the target changes unless the whole copy succeeds (it runs in one transaction on both engines), and the source is read as one snapshot (one transaction on both engines).
type RowFilter ¶
RowFilter is what --where and --limit say about one table: every condition must hold (AND) and at most Limit rows are copied (0: no limit).
type Table ¶
type Table struct {
Name string
Columns []*Column
// PKName is the constraint name, "" when the engine does not report one.
PKName string
PK []string
Uniques []Unique
Indexes []Index
Checks []Check
FKs []ForeignKey
// Rows is the row count read from the source.
Rows int64
// CreateSQL is the CREATE TABLE statement of a SQLite table, as stored.
CreateSQL string
}
Table is a table and everything the source declares about it.
type TableReport ¶
type TableReport struct {
Name string `json:"name"`
Rows int64 `json:"rows"`
Columns int `json:"columns"`
PrimaryKey []string `json:"primaryKey"`
ForeignKeys int `json:"foreignKeys"`
Indexes int `json:"indexes"`
UniqueConstraints int `json:"uniqueConstraints"`
CheckConstraints int `json:"checkConstraints"`
}
TableReport is what happened to one table.
type TargetColumn ¶
type TargetColumn struct {
Src *Column
Name string
Type string // type text in the target dialect
NotNull bool
Default string // default expression in the target dialect, "" for none
Identity Identity
Marker *Marker
// Kind is the logical kind the column is written as.
Kind Kind
// Cast is the PostgreSQL type a text parameter is cast to on insert ("" for a parameter of the column's own type).
Cast string
// contains filtered or unexported fields
}
TargetColumn is a column as it is created on the target.
type TargetTable ¶
type TargetTable struct {
Src *Table
Name string
Columns []*TargetColumn
PKName string
PK []string
Uniques []Unique
Indexes []Index
Checks []Check
FKs []ForeignKey
// Copied is the number of rows written, set when the table has been loaded.
Copied int64
// Filter is the --where / --limit of the table, nil for none.
Filter *compiledFilter
// Loaded says the table's rows have been written (Copied is then the number).
Loaded bool
// Reload says the table exists on the target and keeps its schema: it is emptied and loaded again.
Reload bool
}
TargetTable is a table as it is created on the target.
type Totals ¶
type Totals struct {
Tables int `json:"tables"`
Rows int64 `json:"rows"`
Columns int `json:"columns"`
PrimaryKeys int `json:"primaryKeys"`
ForeignKeys int `json:"foreignKeys"`
Indexes int `json:"indexes"`
UniqueConstraints int `json:"uniqueConstraints"`
CheckConstraints int `json:"checkConstraints"`
}
Totals are the sums over all tables.