faithful

package
v0.93.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Overview

Package faithful copies a SQL database between SQLite and PostgreSQL and keeps everything the two engines can both express: tables, columns, nullability, primary keys (composite too), foreign keys (composite too, with their ON DELETE / ON UPDATE actions), unique constraints, indexes, check constraints, defaults, auto-increment columns, and every value byte for byte.

What an engine cannot carry is never dropped silently: every object that is not carried, and every column whose type had to be widened, is a Warning of the Report (text for people, JSON for machines).

The package works on database/sql handles and reads each engine's own catalog (PRAGMA tables for SQLite, pg_catalog for PostgreSQL) because the engine-neutral DALgo dbschema model has no place for a declared column type, a check constraint or a default text, which is what a faithful copy needs.

Index

Constants

View Source
const (
	WarnViewNotCarried       = "view-not-carried"
	WarnTriggerNotCarried    = "trigger-not-carried"
	WarnTypeWidened          = "type-widened"
	WarnTypeApproximated     = "type-approximated"
	WarnDefaultNotCarried    = "default-not-carried"
	WarnDefaultTranslated    = "default-translated"
	WarnIndexNotCarried      = "index-not-carried"
	WarnCheckNotCarried      = "check-not-carried"
	WarnFKNotValidated       = "fk-not-validated"
	WarnFKNotCarried         = "fk-not-carried"
	WarnIdentityNotCarried   = "identity-not-carried"
	WarnGeneratedColumn      = "generated-column"
	WarnCollationNotCarried  = "collation-not-carried"
	WarnStorageOption        = "storage-option-not-carried"
	WarnCommentNotCarried    = "comment-not-carried"
	WarnObjectNotCarried     = "object-not-carried"
	WarnConstraintNameLost   = "constraint-name-not-carried"
	WarnIndexRenamed         = "index-renamed"
	WarnValuesNormalized     = "values-normalized"
	WarnForeignKeyViolations = "fk-violations-in-source"
	WarnNarrowedOnReturn     = "type-narrowed"
)

Warning kinds. Every feature that is not carried has its own kind so a machine can filter on it.

Variables

View Source
var ErrNoTables = errors.New("source has no tables; nothing to copy")

ErrNoTables says the source holds no table to copy.

Functions

This section is empty.

Types

type Catalog

type Catalog struct {
	Engine Engine
	Schema string
	Tables []*Table
	// Skipped names objects the reader saw and cannot carry, as warnings.
	Notes []Warning
	// Objects are the stored definitions of the views, triggers and indexes of a SQLite database, in creation order.
	Objects []RawObject
}

Catalog is everything read from a source database.

func ReadPostgres

func ReadPostgres(ctx context.Context, db Queryer, schema string) (*Catalog, error)

ReadPostgres reads the catalog of one PostgreSQL schema (the caller says which: Endpoint.Schema defaults to public).

func ReadSQLite

func ReadSQLite(ctx context.Context, db Queryer) (*Catalog, error)

ReadSQLite reads the catalog of a SQLite database: its tables with columns, primary keys, foreign keys, unique constraints, indexes and check constraints. Views, triggers, virtual tables and the like are recorded as notes (warnings) because they cannot be carried across engines.

func (*Catalog) Table

func (c *Catalog) Table(name string) *Table

Table returns the table called name, or nil.

type Check

type Check struct {
	Name string
	Expr string
}

Check is a CHECK constraint, its expression in the source dialect.

type Column

type Column struct {
	Name string
	// SourceType is the declared type text in the source engine, as the engine reports it.
	SourceType string
	Kind       Kind
	// Special is a PostgreSQL type with no logical kind of its own that is carried as text: json, jsonb, uuid.
	Special string
	// Width is the byte width of an integer (2, 4, 8) or float (4, 8) when the source declares one.
	Width int
	// Length is the character length of a varchar / char column, 0 when none.
	Length int
	// Fixed marks a blank-padded char(n).
	Fixed bool
	// Precision and Scale of a decimal, -1 when the source declares none.
	Precision, Scale int
	NotNull          bool
	// Default is the source default expression as the source reports it, "" for none.
	Default  string
	Identity Identity
	// Marker is what an earlier datatug copy recorded on this column (PostgreSQL comment).
	Marker *Marker
	// Comment is a PostgreSQL column comment that is not a datatug marker.
	Comment string
	// Generated is the expression of a generated column, "" for an ordinary column.
	Generated string
	// SequenceStart is the next value of an identity / autoincrement counter, 0 when unknown.
	SequenceStart int64
}

Column is a column as the source engine declares it.

type Cond

type Cond struct{ Field, Op, Value string }

Cond is one --where predicate of a table: <field> <op> <value>. Op is a token of the filter vocabulary (=, <, <=, >, >=, in); Value is the text the user gave.

type ConflictError

type ConflictError struct{ Tables []string }

ConflictError says the target already holds tables of the source.

func (*ConflictError) Error

func (e *ConflictError) Error() string

type Diff

type Diff struct {
	Object string `json:"object"`
	Aspect string `json:"aspect"`
	Left   string `json:"left"`
	Right  string `json:"right"`
}

Diff is one difference between two databases of the same engine.

func Verify

func Verify(ctx context.Context, left, right Endpoint) ([]Diff, error)

Verify compares two databases of the same engine at catalog level and data level: tables, columns (declared type, nullability, default), primary keys, foreign keys (columns, referenced table and columns, actions), unique constraints, indexes, check constraints, row counts, and a digest of every table's rows (so every value, and in SQLite its storage class, is compared).

func (Diff) String

func (d Diff) String() string

type Endpoint

type Endpoint struct {
	Engine  Engine
	DB      *sql.DB
	Display string
	// Schema is the PostgreSQL schema read from (source) or written to (target); "public" when empty.
	Schema string
}

Endpoint is one side of a copy.

type EndpointInfo

type EndpointInfo struct {
	Engine  Engine `json:"engine"`
	Display string `json:"display"`
	Schema  string `json:"schema,omitempty"`
}

EndpointInfo describes one side of the copy without secrets.

type Engine

type Engine string

Engine names a SQL engine this package copies between.

const (
	SQLite   Engine = "sqlite"
	Postgres Engine = "postgres"
)

The engines of phase 1.

type ForeignKey

type ForeignKey struct {
	Name       string
	Columns    []string
	RefTable   string
	RefColumns []string
	OnUpdate   string // NO ACTION, RESTRICT, CASCADE, SET NULL, SET DEFAULT
	OnDelete   string
	Deferrable bool
	Deferred   bool
	NotValid   bool
	Match      string // "FULL" when the key is MATCH FULL (PostgreSQL), else ""
}

ForeignKey is a foreign key constraint.

type Identity

type Identity int

Identity is how a column generates its own value.

const (
	IdentityNone Identity = iota
	// IdentityRowid is a SQLite INTEGER PRIMARY KEY (an alias of the rowid).
	IdentityRowid
	// IdentityAutoincrement is a SQLite INTEGER PRIMARY KEY AUTOINCREMENT.
	IdentityAutoincrement
	// IdentityGenerated is a PostgreSQL identity column or serial.
	IdentityGenerated
)

The ways a column generates its value.

type Index

type Index struct {
	Name    string
	Unique  bool
	Columns []IndexColumn
	// Where is the predicate of a partial index in source dialect, "" for a full index.
	Where string
}

Index is a plain secondary index.

type IndexColumn

type IndexColumn struct {
	Name string
	Desc bool
}

IndexColumn is one key column of an index.

type Kind

type Kind int

Kind is the logical kind of a column, the vocabulary both directions of the type mapping meet in.

const (
	KindOther Kind = iota
	KindInt
	KindFloat
	KindDecimal
	KindText
	KindBytes
	KindBool
	KindDate
	KindTimestamp
	KindTimestampTZ
	KindTime
)

The logical kinds of a column.

func (Kind) String

func (k Kind) String() string

type Marker

type Marker struct {
	Src      string `json:"src"`
	Type     string `json:"type"`
	Identity string `json:"identity,omitempty"` // "rowid" | "autoincrement"
	// Carried is the PostgreSQL type text the column was given, so a later copy can tell a widened column.
	Widened string `json:"widened,omitempty"`
}

Marker is what a copy records on a PostgreSQL column so the column can go back to SQLite as it was.

type NameError

type NameError struct{ Object, Reason string }

NameError says a name cannot be written on the target.

func (*NameError) Error

func (e *NameError) Error() string

type Options

type Options struct {
	// Overwrite is "" (the target must hold none of the source's tables), "recreate" (drop them first) or
	// "reload" (empty them and load the rows again; their schema stays).
	Overwrite        string
	Include, Exclude []string
	// Filters is the --where / --limit of each table, by source table name.
	Filters map[string]RowFilter
	// DecimalsText keeps a DECIMAL_TEXT column whose texts numeric would spell differently as text
	// (--decimals=text); by default such a column is numeric and the change of spelling is a warning.
	DecimalsText bool
	DryRun       bool
	Progress     io.Writer
}

Options controls a copy.

type Plan

type Plan struct {
	From, To Engine
	Schema   string // target schema (PostgreSQL)
	Tables   []*TargetTable
	Warnings []Warning
}

Plan is the whole copy as it will be written.

func PlanPGToSQLite

func PlanPGToSQLite(cat *Catalog, checkOK func(cols []string, expr string) error) *Plan

PlanPGToSQLite decides how every table of a PostgreSQL catalog is created in SQLite. checkOK reports whether SQLite accepts a CHECK expression over the given columns.

func PlanSQLiteToPG

func PlanSQLiteToPG(cat *Catalog, stats map[string][]*colStats, schema string, decimalsText bool) *Plan

PlanSQLiteToPG decides how every table, column and constraint of a SQLite catalog is created on PostgreSQL. stats holds the pre-scan of every column, by table.

func PlanSQLiteToSQLite

func PlanSQLiteToSQLite(cat *Catalog) *Plan

PlanSQLiteToSQLite plans a copy between two SQLite databases: the tables are created from the very statements the source stores (so declared types, defaults, collations, checks, foreign keys and storage options are exactly the source's) and the data is copied value for value.

type Queryer

type Queryer interface {
	QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error)
	QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row
}

Queryer is what the catalog readers need: a *sql.DB, a *sql.Conn or a *sql.Tx.

type RawObject

type RawObject struct {
	Type, Name, Table, SQL string
}

RawObject is a SQLite schema object as sqlite_master stores it.

type ReloadMismatchError

type ReloadMismatchError struct {
	Table    string
	Problems []string
}

ReloadMismatchError says a table the copy would reload has a schema the source rows do not fit.

func (*ReloadMismatchError) Error

func (e *ReloadMismatchError) Error() string

type Report

type Report struct {
	Command string        `json:"command"`
	Status  string        `json:"status"` // ok | ok-with-warnings | dry-run | failed
	DryRun  bool          `json:"dryRun"`
	From    EndpointInfo  `json:"from"`
	To      EndpointInfo  `json:"to"`
	Tables  []TableReport `json:"tables"`
	Totals  Totals        `json:"totals"`
	// Warnings lists every feature that was not carried exactly, per object.
	Warnings []Warning `json:"warnings"`
	// Plan is the DDL that is (dry-run: would be) executed.
	Plan     []string `json:"plan,omitempty"`
	ExitCode int      `json:"exitCode"`
	Error    string   `json:"error,omitempty"`
}

Report is the machine-readable summary of a copy (the JSON of `db copy --json`).

func Copy

func Copy(ctx context.Context, from, to Endpoint, opts Options) (*Report, error)

Copy copies the source into the target and returns the report. The report is returned with ExitCode and Error set on failure as well. Nothing on the target changes unless the whole copy succeeds (it runs in one transaction on both engines), and the source is read as one snapshot (one transaction on both engines).

func (*Report) WriteText

func (r *Report) WriteText(w io.Writer)

WriteText prints the report for people.

type RowFilter

type RowFilter struct {
	Where []Cond
	Limit int
}

RowFilter is what --where and --limit say about one table: every condition must hold (AND) and at most Limit rows are copied (0: no limit).

type Table

type Table struct {
	Name    string
	Columns []*Column
	// PKName is the constraint name, "" when the engine does not report one.
	PKName  string
	PK      []string
	Uniques []Unique
	Indexes []Index
	Checks  []Check
	FKs     []ForeignKey
	// Rows is the row count read from the source.
	Rows int64
	// CreateSQL is the CREATE TABLE statement of a SQLite table, as stored.
	CreateSQL string
}

Table is a table and everything the source declares about it.

func (*Table) Column

func (t *Table) Column(name string) *Column

Column returns the column called name, or nil.

type TableReport

type TableReport struct {
	Name              string   `json:"name"`
	Rows              int64    `json:"rows"`
	Columns           int      `json:"columns"`
	PrimaryKey        []string `json:"primaryKey"`
	ForeignKeys       int      `json:"foreignKeys"`
	Indexes           int      `json:"indexes"`
	UniqueConstraints int      `json:"uniqueConstraints"`
	CheckConstraints  int      `json:"checkConstraints"`
}

TableReport is what happened to one table.

type TargetColumn

type TargetColumn struct {
	Src      *Column
	Name     string
	Type     string // type text in the target dialect
	NotNull  bool
	Default  string // default expression in the target dialect, "" for none
	Identity Identity
	Marker   *Marker
	// Kind is the logical kind the column is written as.
	Kind Kind
	// Cast is the PostgreSQL type a text parameter is cast to on insert ("" for a parameter of the column's own type).
	Cast string
	// contains filtered or unexported fields
}

TargetColumn is a column as it is created on the target.

type TargetTable

type TargetTable struct {
	Src     *Table
	Name    string
	Columns []*TargetColumn
	PKName  string
	PK      []string
	Uniques []Unique
	Indexes []Index
	Checks  []Check
	FKs     []ForeignKey
	// Copied is the number of rows written, set when the table has been loaded.
	Copied int64
	// Filter is the --where / --limit of the table, nil for none.
	Filter *compiledFilter
	// Loaded says the table's rows have been written (Copied is then the number).
	Loaded bool
	// Reload says the table exists on the target and keeps its schema: it is emptied and loaded again.
	Reload bool
}

TargetTable is a table as it is created on the target.

type Totals

type Totals struct {
	Tables            int   `json:"tables"`
	Rows              int64 `json:"rows"`
	Columns           int   `json:"columns"`
	PrimaryKeys       int   `json:"primaryKeys"`
	ForeignKeys       int   `json:"foreignKeys"`
	Indexes           int   `json:"indexes"`
	UniqueConstraints int   `json:"uniqueConstraints"`
	CheckConstraints  int   `json:"checkConstraints"`
}

Totals are the sums over all tables.

type Unique

type Unique struct {
	Name    string
	Columns []string
}

Unique is a UNIQUE constraint.

type Warning

type Warning struct {
	Kind string `json:"kind"`
	// Object names the table, column, index or constraint the warning is about ("t", "t.c", "t.idx").
	Object  string `json:"object"`
	Message string `json:"message"`
}

Warning is one thing that was not carried exactly.

func (Warning) String

func (w Warning) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL