Documentation
¶
Overview ¶
Package dbcopy implements the `datatug db copy` cross-engine database copy primitive. See spec/features/cli/db/copy/ for the contract.
Package dbcopy implements the cross-engine database-copy primitive for `datatug db copy`. This file holds the type-mapping table between the MVP backends (SQLite via dalgo2sqlite and inGitDB via dalgo2ingitdb).
Coverage bar: every column type appearing in the canonical Chinook fixture MUST map cleanly in both directions. Types outside that closed set return a *UnsupportedTypeError naming the source type and target backend.
Mapping policy for the MVP:
- Both backends speak the engine-neutral dbschema.Type vocabulary, so the translation is identity for every type currently produced by either driver's SchemaReader.
- Chinook's *.db SQLite fixture contains columns of SQLite affinity INTEGER, NVARCHAR/TEXT, NUMERIC(10,2), and DATETIME. As of dalgo2sqlite v0.0.0-20260513182736-6886f34af097 the driver's DescribeCollection rejects NUMERIC(p,s) and DATETIME columns (driver gap upstream of dbcopy). For the columns that DO describe successfully (Album, Artist, Customer, Genre, MediaType, Playlist, PlaylistTrack), only dbschema.Int and dbschema.String are observed. The MVP type-map declares the full dbschema vocabulary supported (Bool, Int, Float, String, Bytes, Time, Decimal) so the coverage holds the day dalgo2sqlite ships DATETIME / NUMERIC support and Track/Invoice/InvoiceLine/Employee describe successfully.
- dbschema.Null is intentionally rejected: it is the zero value used for "unset" FieldDef.Type and never a meaningful column type for cross-engine copy.
Package dbcopy implements `datatug db copy --from <url> --to <url>`.
This file covers the URL scheme dispatcher: parsing --from/--to arguments into a typed BackendRef and opening the underlying DALgo dal.DB.
Scheme support per spec/features/cli/db/copy/README.md (REQ:supported-schemes) — that spec predates http/https and does not document them yet:
- sqlite:// fully wired via dalgo2sqlite
- ingitdb:// fully wired via dalgo2ingitdb; local-paths-only (REQ:ingitdb-url-local-only)
- postgres:// parses; Open returns ErrPostgresNotWired until a PostgreSQL DALgo driver implements the three capability interfaces (dbschema.SchemaReader, ddl.SchemaModifier, dal.ConcurrencyAware)
- env:NAME resolves the environment variable NAME, which holds any other supported URL (the way to give PostgreSQL its password without writing it in a project file or on a command line); see parseEnvSource
- http:// https:// fully wired via dal-go/dalgo2http (pkg/httpsource); local-paths-only, same convention as ingitdb:// — see parseHTTPSource
Index ¶
- Constants
- Variables
- func CheckDescriptorEnvName(name string, lookupEnv func(string) (string, bool)) error
- func CheckSourceFile(path string) error
- func LocalSourceURL(scheme, path string) string
- func MapType(t dbschema.Type, sourceBackend, targetBackend string) (dbschema.Type, error)
- func ProjectSourceURL(projectDir string) string
- func RedactError(err error) error
- func RedactErrorWithLiterals(err error, secrets ...string) error
- func RedactErrorWithSecrets(err error, source string) error
- func RedactSourceURL(raw string) string
- func RedactText(text string) string
- func RedactTextWithSecrets(text string, secrets ...string) string
- func SourceScopeIdentity(source string) string
- func SupportedSchemes() []string
- func ValidEnvName(name string) bool
- type BackendRef
- func (r BackendRef) GoString() string
- func (r BackendRef) Open(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenForTest(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenProtected(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenProtectedForTest(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenSchemaScan(_ context.Context) (SchemaScanDB, error)
- func (r BackendRef) String() string
- type CopyOpts
- type NonEmptyTargetError
- type PostgresDescriptor
- type PostgresTarget
- type ProgressWriter
- type ReloadSchemaMismatchError
- type SchemaScanDB
- type SourceSummary
- type UnsupportedTypeError
Constants ¶
const ( // DescriptorEnvPrefix is the name prefix a project descriptor's dsnEnv may // always carry: a variable the operator named for DataTug. DescriptorEnvPrefix = "DATATUG_" // DescriptorEnvAllowList names the operator-set variable that lists, comma or // space separated, the other variables a project descriptor may name. It // follows the shape of the OpenVaultDB descriptor, whose token is bound to a // destination the operator sets (pkg/openvaultdb/source.go): the project // file proposes, the operator's own environment decides. DescriptorEnvAllowList = "DATATUG_DSN_ENV_ALLOW" )
const ( BackendSQLite = "sqlite" BackendInGitDB = "ingitdb" )
Supported backend identifiers accepted by MapType.
const EnvSourceForm = envPrefix + "NAME"
EnvSourceForm is how the env source form is spelled in help text.
const PostgresDefaultSchema = dalgo2postgres.DefaultSchema
PostgresDefaultSchema is the PostgreSQL schema a schema scan reads: the one the DALgo reader inspects when it is not told another.
Variables ¶
var ErrNoPrimaryKey = errors.New("source collection has no primary key declared")
ErrNoPrimaryKey is returned when a source collection has no PK declared. The MVP row-copy path requires a declared PK to construct target record keys.
var ErrPostgresNotWired = errors.New("PostgreSQL backend not yet wired")
ErrPostgresNotWired is returned by BackendRef.Open for postgres:// URLs until a PostgreSQL DALgo driver implements the three capability interfaces (dbschema.SchemaReader, ddl.SchemaModifier, dal.ConcurrencyAware).
var ErrSourceFileMissing = errors.New("source file does not exist")
ErrSourceFileMissing is wrapped by CheckSourceFile (and, through it, by Open's sqlite/ingitdb branches) when a file-backed source does not exist on disk — e.g. `datatug serve --project` against a demo project before `datatug demo` has fetched ~/datatug/dbs/chinook-local.sqlite. Checked with errors.Is so a caller (pkg/secureread, pkg/server/endpoints) can map it to api-contract.md's SOURCE_UNAVAILABLE (503) instead of letting the driver's own opaque "unable to open database file" text reach an HTTP 500.
var ErrSourceHasNoTables = errors.New("source has no tables; nothing to copy")
ErrSourceHasNoTables signals that the source introspected cleanly but has zero collections. Callers should exit 0 with a stderr note per REQ:source-introspection-failure.
Functions ¶
func CheckDescriptorEnvName ¶ added in v0.54.0
CheckDescriptorEnvName reports whether a project descriptor may name the environment variable name. A project file is not trusted: a cloned project could otherwise select any database whose URL the operator keeps in the environment. So a descriptor may name only a variable whose name starts with DATATUG_, or one the operator lists in DATATUG_DSN_ENV_ALLOW. name must already match ValidEnvName, which keeps the error free to quote it.
func CheckSourceFile ¶ added in v0.20.1
CheckSourceFile reports ErrSourceFileMissing (wrapping path and a `datatug demo` recovery hint) when path does not exist on disk, and nil when it does or when the stat fails for any other reason (permissions, etc. — left for the underlying driver to report on its own terms). It is exported so callers that open a file-backed source through a path other than BackendRef.Open (pkg/secureread's read-only native-SQL connection) can perform the identical check.
func LocalSourceURL ¶ added in v0.53.0
LocalSourceURL returns the "scheme://path" source URL that names the local directory path for a scheme that takes one (http, https, ingitdb). A relative path whose first segment holds an "@" or that is shaped like "user:password@host" (a:b/c@d) is written "./path", because Parse refuses "scheme://my@proj" and "scheme://a:b/c@d" as URLs that carry credentials. Every caller that builds such a URL from a directory it was given verbatim goes through here, so a directory such as my@proj keeps working.
func MapType ¶
MapType translates a column type from sourceBackend to targetBackend. The returned Type is ready for use in a target ddl.CreateCollection call. For the MVP, sourceBackend and targetBackend MUST each be one of "sqlite" or "ingitdb". Types outside the Chinook coverage set (dbschema.Null and any unrecognized Type value) return a *UnsupportedTypeError naming the source type and target backend.
Both backends accept the same engine-neutral dbschema.Type vocabulary, so this function is the identity for every supported type today. The function exists as the seam where engine-specific widening / narrowing rules will land if and when the type-mapping matrix needs them.
func ProjectSourceURL ¶ added in v0.53.0
ProjectSourceURL returns the http:// source URL that names the datatug project directory projectDir, the way the commands and the server build one from a directory they were given verbatim. A relative directory whose first segment holds an "@" (my@proj, @acme/proj) is written "./dir", because Parse refuses "http://my@proj" as a URL that carries credentials.
func RedactError ¶ added in v0.53.0
RedactError returns err with a message passed through RedactText. It wraps err, so errors.Is and errors.As still see the original; only Error() text is redacted. A nil err stays nil.
func RedactErrorWithLiterals ¶ added in v0.53.0
RedactErrorWithLiterals is RedactError for an error written by code that was handed secrets as plain values (a password given as a flag): it also removes every literal in secrets, in the spellings secretSpellings lists.
func RedactErrorWithSecrets ¶ added in v0.53.0
RedactErrorWithSecrets is RedactError for an error written by code that was handed source, the real URL: besides the URL-shaped text RedactError finds, it removes every literal secret source holds, so a driver that formats the connection string in its own way still cannot put the password in a message.
func RedactSourceURL ¶ added in v0.53.0
RedactSourceURL returns raw with every secret replaced by "xxxxx": the password in "user:password@host", the value of any query parameter whose name says it is a secret (password, sslpassword, token, key ...), and the value of a libpq "password=..." keyword. A string that holds no secret comes back unchanged, byte for byte.
It works on the text and never parses, so it cannot fail and a malformed URL is still redacted. It prefers to over-redact: the userinfo ends at the last "@" in the string, so a password that holds an unescaped "@", "/" or "?" is still removed whole. The one exemption is an http or https URL whose authority is a plain host or host:port (see isHostOnly), whose "@" belongs to the path.
func RedactText ¶ added in v0.53.0
RedactText returns text with every URL inside it passed through RedactSourceURL and every "password=..." keyword value removed. Use it on any message, log line or stored string that could carry a source URL.
func RedactTextWithSecrets ¶ added in v0.53.0
RedactTextWithSecrets is RedactText for text that may hold secrets the caller knows as plain values (a password it was given as a flag): besides everything RedactText removes it replaces every literal in secrets, in the spellings secretSpellings lists.
func SourceScopeIdentity ¶ added in v0.54.0
SourceScopeIdentity returns the text that stands for source in a persisted scope identity, such as the key of a chat session store.
An "env:NAME" source names a variable, and the variable can be repointed at another database while its name stays: under the name alone, record sets read from the old database would stay reachable in the new one. So the identity of an env source is the name plus a hash of where the variable points now: for a PostgreSQL URL its host, port, database and user, never the password. The hash is the same for the same destination, whatever the password or the connection options, and differs for any other. A part the URL leaves out is what libpq's environment variables give it (PGHOST, PGPORT, PGDATABASE, PGUSER and PGSERVICE), and the database is the user when nothing names one, as pgx resolves them; the contents of a connection service file and the operating system's user name are not part of the identity. A variable that cannot be resolved is its own scope. Every other source is returned unchanged.
func SupportedSchemes ¶ added in v0.17.2
func SupportedSchemes() []string
SupportedSchemes returns the exact schemes Parse/Open dispatch, in dispatch order. It is the single source of truth other packages should build user-facing scheme lists from (e.g. a `--db` flag's help text) so that text cannot drift from what Open actually accepts the way it did when http/https were wired in here without every caller's help text being updated to match.
func ValidEnvName ¶ added in v0.53.0
ValidEnvName reports whether name is an acceptable environment variable name for an "env:NAME" source or a PostgreSQL descriptor's dsnEnv.
Types ¶
type BackendRef ¶
type BackendRef struct {
Scheme string
// Path holds the scheme-specific resource locator.
// - sqlite: filesystem path to the .db file (e.g. "/tmp/foo.db" or "./rel.db")
// - ingitdb: filesystem path to the project directory
// - postgres: full original URL (passed verbatim to the future driver).
// It can hold the password: never print or store Path; use
// Raw or String for any message.
// - http/https: filesystem path to the datatug project directory whose
// queries/ tree declares the HTTP QueryDefs to serve (see
// pkg/httpsource) — same local-path convention as ingitdb,
// NOT a literal remote endpoint; the project's own query
// definitions name the actual remote endpoints.
Path string
// Raw is the original input string, preserved for error messages, with
// every secret redacted (see RedactSourceURL): a postgres:// URL shows
// "user:xxxxx@host", and an "env:NAME" input stays "env:NAME" because the
// value of the variable is never copied here.
Raw string
}
BackendRef is a parsed --from/--to URL.
func Parse ¶
func Parse(rawURL string) (BackendRef, error)
Parse parses a CLI URL argument into a BackendRef. It returns an error for unknown schemes (REQ:unknown-scheme-rejected), malformed URLs, and remote ingitdb:// URLs (REQ:ingitdb-url-local-only).
The unknown-scheme error message names BOTH the unsupported scheme AND the supported list, as required by REQ:unknown-scheme-rejected.
An input of the form env:NAME is resolved first: NAME must match ^[A-Z][A-Z0-9_]*$ and name an environment variable holding any other supported URL. No error ever echoes the variable's value.
func ParseWithEnv ¶ added in v0.54.0
ParseWithEnv is Parse over an injected environment lookup, so a caller's tests never touch the process environment. Parse is ParseWithEnv with os.LookupEnv.
func (BackendRef) GoString ¶ added in v0.53.0
func (r BackendRef) GoString() string
GoString makes %#v print the same redacted text as String.
func (BackendRef) Open ¶
Open opens the underlying DALgo dal.DB for this BackendRef.
Dispatch:
- sqlite: opens via dalgo2sqlite.NewDatabase.
- ingitdb: opens via dalgo2ingitdb.NewDatabase with the default validator-backed CollectionsReader.
- postgres: returns ErrPostgresNotWired (no DALgo Postgres driver yet exposes the three capability interfaces).
- http/https: opens via httpsource.Open, translating every HTTP QueryDef under the project directory (r.Path) into a dalgo2http collection.
Open never applies the provider-side read hardening OpenProtected does (see its doc comment): every caller here — `datatug db copy`, and schema introspection in pkg/server/endpoints — is a trusted, operator-level caller with no pkg/accesspolicies wrapper above it, so a formula/computed column is returned exactly as the provider evaluates it, matching every dalgo2sql/dalgo2ingitdb release before Task 13 (S110).
The context is reserved for future use; today's driver constructors are synchronous and do not honor cancellation. That's acceptable for the MVP CLI verb.
func (BackendRef) OpenForTest ¶ added in v0.20.4
OpenForTest is Open, except that for an "http"/"https" BackendRef every dalgo2http.Collection it builds gets Collection.InsecureAllowLoopback set (dal-go/dalgo2http v0.2.0's TEST-ONLY escape hatch — see httpsource.AllowInsecureLoopback's doc comment). Every other scheme behaves identically to Open.
It exists so a test that drives the full sourceURL -> Parse -> Open pipeline in-process (e.g. apps/datatugapp/commands's cmd_query_http_provenance_test.go, pkg/secureread's executor_provenance_test.go via Executor.RunStructuredInsecureForTest) can point an HTTP QueryDef's .query.http file at a loopback httptest.Server or an intentionally-unreachable loopback address (e.g. 127.0.0.1:1, for a fast deterministic live-failure), without any project descriptor file ever requesting that itself — the field is set here, in Go code, only when a caller explicitly calls THIS method instead of Open. NEVER call this from production code.
func (BackendRef) OpenProtected ¶ added in v0.20.8
OpenProtected is Open, except an "ingitdb" BackendRef is opened with dalgo2ingitdb.WithStoredOnlyReads(). SQLite uses the validated, parameter-bound structured-query dialect on both paths. Other schemes behave like Open.
pkg/secureread.openSource uses this for policy-secured sessions; direct `query run` uses it for SQLite structured queries. A policy-secured session wraps the returned dal.DB with pkg/accesspolicies before any row reaches a caller. dalgo2ingitdb's own formula evaluator has no way to know which of a computed column's dependencies pkg/accesspolicies would have redacted — the adapter computes the value from the FULL underlying record and hands back the (correct) result, which can leak a hidden field's value through an allowed computed column (see dal-go/dalgo2ingitdb#8 / this repo's README "Owner access policies" section). WithStoredOnlyReads keeps dalgo2ingitdb from evaluating or returning any formula column at all under an outer policy wrapper, so pkg/accesspolicies' field allow-list is the only thing that can ever put a value on the wire — it stays the single enforcement point. This adapter never writes an .ingitdb/access/manifest.yaml file of its own, so dalgo2ingitdb's persisted owner-policy layer (also new in v0.4.0) never activates for a datatug-cli-opened project; policies do NOT layer under pkg/accesspolicies here — pkg/accesspolicies remains the sole enforcement layer for every source this CLI opens.
The sqlite scheme opts every read, protected or not, into dalgo2sql's DbOptions.StructuredQueryDialect: "sqlite" (bounded, parameter-bound structured-query compilation): dal-go/dalgo2sql#179 added FROM-source alias support to compileStructuredSQL, which used to unconditionally reject any structured query whose FROM source carried an alias — this project's own demo query (queries/customers/customer-invoices.query.dtql, `from: {name: Invoice, alias: i}`) uses exactly that shape, and used to turn into a hard error the moment this dialect was enabled. With #179 fixed, existing non-aggregate DTQL queries (see dalgo2sql's own dtql_datatug_inventory_test.go) compile cleanly, so protected reads now get the dialect's real guarantees: every dal.Constant value becomes a genuine `?` placeholder + bound arg (not a quoted-string literal), and unsupported shapes (such as joins and cursors) fail closed. GROUP BY and HAVING are compiled by dalgo2sql's aggregation path. The plain Open path (db copy / introspection) sets the same dialect, so it never reaches the legacy emitSQL renderer; only the ingitdb hardening differs between Open and OpenProtected.
func (BackendRef) OpenProtectedForTest ¶ added in v0.20.8
OpenProtectedForTest combines OpenProtected's provider-side read hardening with OpenForTest's http(s) loopback escape hatch. It exists for pkg/secureread's Executor.RunStructuredInsecureForTest, which must drive the exact same openSource -> BackendRef -> pkg/accesspolicies pipeline RunStructured uses in production, just against a loopback test server. NEVER call this from production code.
func (BackendRef) OpenSchemaScan ¶ added in v0.54.0
func (r BackendRef) OpenSchemaScan(_ context.Context) (SchemaScanDB, error)
OpenSchemaScan opens a postgres source for reading its schema and nothing else. It is not Open: Open keeps answering that PostgreSQL is not available for queries, and the handle returned here is for the scan only. DataTug opens it with exact identifiers, so a table is looked up under the name PostgreSQL reports (the driver's default folds every name to lower case).
The URL is checked before the driver sees it (ParsePostgresTarget): a URL that net/url and pgx read differently from how it was written, such as a password with an unescaped "/" after digits, is refused, because the driver would connect to the wrong host and print the rest of the password as the database.
The driver quotes the URL, password included, in its open and ping errors, so every error leaves here with every PostgreSQL URL in it, as given or as pgx writes it again in a parse error, replaced by the name the source was given (the "env:NAME" it was opened from, never the URL: it carries every connection option, not only the password) and with every secret the URL holds scrubbed, exactly as Open's errors are. What the driver says about the target in words of its own, such as pgx's "user=alice database=shop" after a failed connection, or the value of an option it cannot read ("parsing \"soon\""), is not a secret and stays. The context is reserved for future use, as in Open.
func (BackendRef) String ¶ added in v0.53.0
func (r BackendRef) String() string
String returns the redacted source, so printing a BackendRef with %v or %+v can never put a password in a log or message.
type CopyOpts ¶
type CopyOpts struct {
// Overwrite is "" (require empty target), "recreate" (drop source-named
// tables, then create from source), or "reload" (reserved — row-level
// semantics; behaves like "recreate" for schema-only copies until row
// CRUD lands).
Overwrite string
// Stderr receives per-table skip / row-error notes and the final
// summary note. Defaults to discard if nil.
Stderr io.Writer
// Progress, if non-nil, receives per-table progress lines.
Progress *ProgressWriter
// SchemaOnly, if true, skips row streaming entirely and only replicates
// the target schema. Useful for E2E-test scaffolding and for backends
// where row streaming isn't supported in this direction.
SchemaOnly bool
// ParallelStreams is the requested max number of source tables copied
// concurrently. 0 means "use the default": runtime.NumCPU()-1 (with a
// floor of 1). Negative values are normalized to 1. Capped to 1 when
// either source or target advertises SupportsConcurrentConnections()==false
// (REQ:concurrency-cap).
ParallelStreams int
// Filters carries resolved filtering directives (table include/exclude,
// row WHERE predicates, row limits). nil or empty means "no filtering
// — copy whole DB per parent Feature ACs". Subsequent tasks consume
// this at two seams (pre-worker table filter; engine_rows query builder).
// Spec: spec/features/cli/db/copy/filtering/README.md
Filters *filter.Directives
}
CopyOpts controls a Copy call.
type NonEmptyTargetError ¶
NonEmptyTargetError is returned by checkEmptyTarget when at least one source-named table exists on the target with >=1 row. The user should rerun with --overwrite=recreate or --overwrite=reload.
func (*NonEmptyTargetError) Error ¶
func (e *NonEmptyTargetError) Error() string
type PostgresDescriptor ¶ added in v0.53.0
type PostgresDescriptor struct {
// DSNEnv is the name of the environment variable that holds a postgres://
// or postgresql:// URL. It must match ^[A-Z][A-Z0-9_]*$.
DSNEnv string `json:"dsnEnv"`
}
PostgresDescriptor is the project-owned file that points a PostgreSQL source at its credentials. It names an environment variable and nothing else: the variable holds the whole connection URL (host, user and password together), so no project file can hold a password, and a project file cannot redirect a credential to a host of its own choosing.
func DecodePostgresDescriptor ¶ added in v0.53.0
func DecodePostgresDescriptor(data []byte) (PostgresDescriptor, error)
DecodePostgresDescriptor strictly decodes a descriptor: one JSON object, no duplicate keys, no field other than dsnEnv, and a valid variable name. No error it returns holds a value from the input; an unknown field is named only when the name is a plain identifier.
func ReadPostgresDescriptor ¶ added in v0.53.0
func ReadPostgresDescriptor(path string) (PostgresDescriptor, error)
ReadPostgresDescriptor reads and strictly decodes the descriptor at path, and refuses a dsnEnv that CheckDescriptorEnvName does not allow in the process environment.
func (PostgresDescriptor) SourceURL ¶ added in v0.53.0
func (d PostgresDescriptor) SourceURL() string
SourceURL returns the source URL that resolves this descriptor: "env:NAME". It names the variable and never holds its value, so it is safe to pass through the chat store, logs and error messages.
type PostgresTarget ¶ added in v0.54.0
type PostgresTarget struct {
Host string
// Port is 0 when the URL names none.
Port int
Database string
User string
}
PostgresTarget is the part of a PostgreSQL connection URL that says where it points. It never holds the password.
func ParsePostgresTarget ¶ added in v0.54.0
func ParsePostgresTarget(rawURL string) (PostgresTarget, error)
ParsePostgresTarget reads the destination a postgres:// or postgresql:// URL states: the host, port, user and database of the URL, each replaced by the query parameter of the same name (host, port, user, dbname or database) when there is one, as pgx applies them. It reads the URL only: what pgx takes from the PG* environment variables or a service file for a part the URL leaves out is not here.
It refuses a URL with a literal "@" after the authority (see atSignAfterAuthority): such a URL is not read the way it was written, and refusing it is the only way to keep the rest of a password out of the host, database and scope that are printed or hashed. It also refuses a user name that holds a colon, however the URL spells it (see errPostgresUserHoldsColon): "alice%3Apw@host" has no literal colon, so net/url reads the user "alice:pw" and no password, and the password would be printed as the user, where no redactor looks for a secret. No error quotes the URL, which holds the password.
type ProgressWriter ¶
type ProgressWriter struct {
// contains filtered or unexported fields
}
ProgressWriter emits per-table start/finish lines to a writer (intended to be os.Stderr in production). When enabled is false, all methods are no-ops. Safe for concurrent use across multiple worker goroutines.
Format is pinned by REQ:progress-reporting in spec/features/cli/db/copy/README.md.
func NewProgressWriter ¶
func NewProgressWriter(w io.Writer, enabled bool) *ProgressWriter
NewProgressWriter returns a ProgressWriter writing to w when enabled is true. If enabled is false (or w is nil), all methods are no-ops.
func (*ProgressWriter) FinishTable ¶
func (p *ProgressWriter) FinishTable(table string, rows int64, d time.Duration)
FinishTable announces that a copy of `table` has completed: `rows` inserted in `d`. Duration is rounded to millisecond per AC.
func (*ProgressWriter) StartTable ¶
func (p *ProgressWriter) StartTable(table string, estRows int64)
StartTable announces that a copy of `table` is starting. If estRows is negative, the start line prints `est. ? rows` instead of a number.
type ReloadSchemaMismatchError ¶
type ReloadSchemaMismatchError struct {
Table string
Column string
SourceValue string
TargetValue string
Reason string // short reason: "missing column", "type mismatch", "primary key mismatch"
}
ReloadSchemaMismatchError signals that the target's schema for a given table is not a superset of the source's, per REQ:reload-schema-match.
Column is either a column name (when a source column is missing or has an incompatible target type) or the literal "<primary key>" when the primary-key column sets differ.
SourceValue and TargetValue describe what differed (e.g. type names or PK column lists) in human-readable form, for the stderr diff.
func (*ReloadSchemaMismatchError) Error ¶
func (e *ReloadSchemaMismatchError) Error() string
type SchemaScanDB ¶ added in v0.54.0
SchemaScanDB is the handle a schema scan reads through: the DALgo database, its schema reader, and a Close that releases the connection pool.
type SourceSummary ¶
type SourceSummary struct {
Tables int
Created int
CreatedNames []string // names of collections created on the target, in source-iteration order
Skipped []string // tables skipped at DescribeCollection time
RowsCopied int64 // total rows inserted into the target across all tables
RowsByTable map[string]int64
RowSkips map[string]string // table → reason (e.g. composite PK, no PK)
TargetBackend string // adapter name of target, for error messages
}
SourceSummary is what Copy reports back to the caller.
func Copy ¶
Copy replicates the source database into the target — schema first, then row data per table.
Schema replication uses DALgo dbschema/ddl. Row streaming uses ExecuteQueryToRecordsReader on the source and RunReadwriteTransaction → InsertMulti on the target (see engine_rows.go).
If opts.SchemaOnly is true, only schema is replicated.
Tables the source can't describe (e.g. dalgo2sqlite rejecting DATETIME / NUMERIC) are appended to Skipped and processing continues. Tables with no PK get schema replicated but row copy is skipped with the reason recorded in RowSkips. Composite-PK tables are now copied (key encoded as `__`-joined PK values; see encodeRecordID in engine_rows.go).
Concurrency: opts.ParallelStreams governs how many tables are copied in parallel. The effective value is capped to 1 if either source or target advertises SupportsConcurrentConnections()==false. When the cap reduces an explicitly-requested value >1, one warning line is emitted on stderr.
Errors:
- ErrSourceHasNoTables — source introspects cleanly but has zero collections.
- any other error — wrapped with the failing operation and table name.
type UnsupportedTypeError ¶
type UnsupportedTypeError struct {
// SourceType is the dbschema.Type that could not be mapped.
SourceType dbschema.Type
// TargetBackend is the backend name supplied to MapType.
TargetBackend string
}
UnsupportedTypeError is returned by MapType when the source column type is outside the MVP coverage set.
func (*UnsupportedTypeError) Error ¶
func (e *UnsupportedTypeError) Error() string
Error implements the error interface. The message names both the source type and the target backend so the operator can act.
Source Files
Directories