Documentation
¶
Overview ¶
Package dbcopy implements the `datatug db copy` cross-engine database copy primitive. See spec/features/cli/db/copy/ for the contract.
Package dbcopy implements the cross-engine database-copy primitive for `datatug db copy`. This file holds the type-mapping table between the MVP backends (SQLite via dalgo2sqlite and inGitDB via dalgo2ingitdb).
Coverage bar: every column type appearing in the canonical Chinook fixture MUST map cleanly in both directions. Types outside that closed set return a *UnsupportedTypeError naming the source type and target backend.
Mapping policy for the MVP:
- Both backends speak the engine-neutral dbschema.Type vocabulary, so the translation is identity for every type currently produced by either driver's SchemaReader.
- Chinook's *.db SQLite fixture contains columns of SQLite affinity INTEGER, NVARCHAR/TEXT, NUMERIC(10,2), and DATETIME. As of dalgo2sqlite v0.0.0-20260513182736-6886f34af097 the driver's DescribeCollection rejects NUMERIC(p,s) and DATETIME columns (driver gap upstream of dbcopy). For the columns that DO describe successfully (Album, Artist, Customer, Genre, MediaType, Playlist, PlaylistTrack), only dbschema.Int and dbschema.String are observed. The MVP type-map declares the full dbschema vocabulary supported (Bool, Int, Float, String, Bytes, Time, Decimal) so the coverage holds the day dalgo2sqlite ships DATETIME / NUMERIC support and Track/Invoice/InvoiceLine/Employee describe successfully.
- dbschema.Null is intentionally rejected: it is the zero value used for "unset" FieldDef.Type and never a meaningful column type for cross-engine copy.
Package dbcopy implements `datatug db copy --from <url> --to <url>`.
This file covers the URL scheme dispatcher: parsing --from/--to arguments into a typed BackendRef and opening the underlying DALgo dal.DB.
Scheme support per spec/features/cli/db/copy/README.md (REQ:supported-schemes) — that spec predates http/https and does not document them yet:
- sqlite:// fully wired via dalgo2sqlite
- ingitdb:// fully wired via dalgo2ingitdb; local-paths-only (REQ:ingitdb-url-local-only)
- postgres:// parses; Open returns ErrPostgresNotWired until a PostgreSQL DALgo driver implements the three capability interfaces (dbschema.SchemaReader, ddl.SchemaModifier, dal.ConcurrencyAware)
- http:// https:// fully wired via dal-go/dalgo2http (pkg/httpsource); local-paths-only, same convention as ingitdb:// — see parseHTTPSource
Index ¶
- Constants
- Variables
- func CheckSourceFile(path string) error
- func MapType(t dbschema.Type, sourceBackend, targetBackend string) (dbschema.Type, error)
- func SupportedSchemes() []string
- type BackendRef
- type CopyOpts
- type NonEmptyTargetError
- type ProgressWriter
- type ReloadSchemaMismatchError
- type SourceSummary
- type UnsupportedTypeError
Constants ¶
const ( BackendSQLite = "sqlite" BackendInGitDB = "ingitdb" )
Supported backend identifiers accepted by MapType.
Variables ¶
var ErrNoPrimaryKey = errors.New("source collection has no primary key declared")
ErrNoPrimaryKey is returned when a source collection has no PK declared. The MVP row-copy path requires a declared PK to construct target record keys.
var ErrPostgresNotWired = errors.New("PostgreSQL backend not yet wired")
ErrPostgresNotWired is returned by BackendRef.Open for postgres:// URLs until a PostgreSQL DALgo driver implements the three capability interfaces (dbschema.SchemaReader, ddl.SchemaModifier, dal.ConcurrencyAware).
var ErrSourceFileMissing = errors.New("source file does not exist")
ErrSourceFileMissing is wrapped by CheckSourceFile (and, through it, by Open's sqlite/ingitdb branches) when a file-backed source does not exist on disk — e.g. `datatug serve --project` against a demo project before `datatug demo` has fetched ~/datatug/dbs/chinook-local.sqlite. Checked with errors.Is so a caller (pkg/secureread, pkg/server/endpoints) can map it to api-contract.md's SOURCE_UNAVAILABLE (503) instead of letting the driver's own opaque "unable to open database file" text reach an HTTP 500.
var ErrSourceHasNoTables = errors.New("source has no tables; nothing to copy")
ErrSourceHasNoTables signals that the source introspected cleanly but has zero collections. Callers should exit 0 with a stderr note per REQ:source-introspection-failure.
Functions ¶
func CheckSourceFile ¶ added in v0.20.1
CheckSourceFile reports ErrSourceFileMissing (wrapping path and a `datatug demo` recovery hint) when path does not exist on disk, and nil when it does or when the stat fails for any other reason (permissions, etc. — left for the underlying driver to report on its own terms). It is exported so callers that open a file-backed source through a path other than BackendRef.Open (pkg/secureread's read-only native-SQL connection) can perform the identical check.
func MapType ¶
MapType translates a column type from sourceBackend to targetBackend. The returned Type is ready for use in a target ddl.CreateCollection call. For the MVP, sourceBackend and targetBackend MUST each be one of "sqlite" or "ingitdb". Types outside the Chinook coverage set (dbschema.Null and any unrecognized Type value) return a *UnsupportedTypeError naming the source type and target backend.
Both backends accept the same engine-neutral dbschema.Type vocabulary, so this function is the identity for every supported type today. The function exists as the seam where engine-specific widening / narrowing rules will land if and when the type-mapping matrix needs them.
func SupportedSchemes ¶ added in v0.17.2
func SupportedSchemes() []string
SupportedSchemes returns the exact schemes Parse/Open dispatch, in dispatch order. It is the single source of truth other packages should build user-facing scheme lists from (e.g. a `--db` flag's help text) so that text cannot drift from what Open actually accepts the way it did when http/https were wired in here without every caller's help text being updated to match.
Types ¶
type BackendRef ¶
type BackendRef struct {
Scheme string
// Path holds the scheme-specific resource locator.
// - sqlite: filesystem path to the .db file (e.g. "/tmp/foo.db" or "./rel.db")
// - ingitdb: filesystem path to the project directory
// - postgres: full original URL (passed verbatim to the future driver)
// - http/https: filesystem path to the datatug project directory whose
// queries/ tree declares the HTTP QueryDefs to serve (see
// pkg/httpsource) — same local-path convention as ingitdb,
// NOT a literal remote endpoint; the project's own query
// definitions name the actual remote endpoints.
Path string
// Raw is the original input string, preserved for error messages.
Raw string
}
BackendRef is a parsed --from/--to URL.
func Parse ¶
func Parse(rawURL string) (BackendRef, error)
Parse parses a CLI URL argument into a BackendRef. It returns an error for unknown schemes (REQ:unknown-scheme-rejected), malformed URLs, and remote ingitdb:// URLs (REQ:ingitdb-url-local-only).
The unknown-scheme error message names BOTH the unsupported scheme AND the supported list, as required by REQ:unknown-scheme-rejected.
func (BackendRef) Open ¶
Open opens the underlying DALgo dal.DB for this BackendRef.
Dispatch:
- sqlite: opens via dalgo2sqlite.NewDatabase.
- ingitdb: opens via dalgo2ingitdb.NewDatabase with the default validator-backed CollectionsReader.
- postgres: returns ErrPostgresNotWired (no DALgo Postgres driver yet exposes the three capability interfaces).
- http/https: opens via httpsource.Open, translating every HTTP QueryDef under the project directory (r.Path) into a dalgo2http collection.
Open never applies the provider-side read hardening OpenProtected does (see its doc comment): every caller here — `datatug db copy`, and schema introspection in pkg/server/endpoints — is a trusted, operator-level caller with no pkg/accesspolicies wrapper above it, so a formula/computed column is returned exactly as the provider evaluates it, matching every dalgo2sql/dalgo2ingitdb release before Task 13 (S110).
The context is reserved for future use; today's driver constructors are synchronous and do not honor cancellation. That's acceptable for the MVP CLI verb.
func (BackendRef) OpenForTest ¶ added in v0.20.4
OpenForTest is Open, except that for an "http"/"https" BackendRef every dalgo2http.Collection it builds gets Collection.InsecureAllowLoopback set (dal-go/dalgo2http v0.2.0's TEST-ONLY escape hatch — see httpsource.AllowInsecureLoopback's doc comment). Every other scheme behaves identically to Open.
It exists so a test that drives the full sourceURL -> Parse -> Open pipeline in-process (e.g. apps/datatugapp/commands's cmd_query_http_provenance_test.go, pkg/secureread's executor_provenance_test.go via Executor.RunStructuredInsecureForTest) can point an HTTP QueryDef's .query.http file at a loopback httptest.Server or an intentionally-unreachable loopback address (e.g. 127.0.0.1:1, for a fast deterministic live-failure), without any project descriptor file ever requesting that itself — the field is set here, in Go code, only when a caller explicitly calls THIS method instead of Open. NEVER call this from production code.
func (BackendRef) OpenProtected ¶ added in v0.20.8
OpenProtected is Open, except an "ingitdb" BackendRef is opened with dalgo2ingitdb.WithStoredOnlyReads() and SQLite uses the validated, parameter-bound structured-query dialect. Other schemes behave like Open.
pkg/secureread.openSource uses this for policy-secured sessions; direct `query run` uses it for SQLite structured queries. A policy-secured session wraps the returned dal.DB with pkg/accesspolicies before any row reaches a caller. dalgo2ingitdb's own formula evaluator has no way to know which of a computed column's dependencies pkg/accesspolicies would have redacted — the adapter computes the value from the FULL underlying record and hands back the (correct) result, which can leak a hidden field's value through an allowed computed column (see dal-go/dalgo2ingitdb#8 / this repo's README "Owner access policies" section). WithStoredOnlyReads keeps dalgo2ingitdb from evaluating or returning any formula column at all under an outer policy wrapper, so pkg/accesspolicies' field allow-list is the only thing that can ever put a value on the wire — it stays the single enforcement point. This adapter never writes an .ingitdb/access/manifest.yaml file of its own, so dalgo2ingitdb's persisted owner-policy layer (also new in v0.4.0) never activates for a datatug-cli-opened project; policies do NOT layer under pkg/accesspolicies here — pkg/accesspolicies remains the sole enforcement layer for every source this CLI opens.
The sqlite scheme opts protected reads into dalgo2sql's DbOptions.StructuredQueryDialect: "sqlite" (bounded, parameter-bound structured-query compilation): dal-go/dalgo2sql#179 added FROM-source alias support to compileStructuredSQL, which used to unconditionally reject any structured query whose FROM source carried an alias — this project's own demo query (queries/customers/customer-invoices.query.dtql, `from: {name: Invoice, alias: i}`) uses exactly that shape, and used to turn into a hard error the moment this dialect was enabled. With #179 fixed, existing non-aggregate DTQL queries (see dalgo2sql's own dtql_datatug_inventory_test.go) compile cleanly, so protected reads now get the dialect's real guarantees: every dal.Constant value becomes a genuine `?` placeholder + bound arg (not a quoted-string literal), and unsupported shapes (such as joins and cursors) fail closed. GROUP BY and HAVING are compiled by dalgo2sql's aggregation path. The plain Open path (db copy / introspection) is unaffected — it never sets StructuredQueryDialect, so it keeps using the legacy emitSQL rendering.
func (BackendRef) OpenProtectedForTest ¶ added in v0.20.8
OpenProtectedForTest combines OpenProtected's provider-side read hardening with OpenForTest's http(s) loopback escape hatch. It exists for pkg/secureread's Executor.RunStructuredInsecureForTest, which must drive the exact same openSource -> BackendRef -> pkg/accesspolicies pipeline RunStructured uses in production, just against a loopback test server. NEVER call this from production code.
type CopyOpts ¶
type CopyOpts struct {
// Overwrite is "" (require empty target), "recreate" (drop source-named
// tables, then create from source), or "reload" (reserved — row-level
// semantics; behaves like "recreate" for schema-only copies until row
// CRUD lands).
Overwrite string
// Stderr receives per-table skip / row-error notes and the final
// summary note. Defaults to discard if nil.
Stderr io.Writer
// Progress, if non-nil, receives per-table progress lines.
Progress *ProgressWriter
// SchemaOnly, if true, skips row streaming entirely and only replicates
// the target schema. Useful for E2E-test scaffolding and for backends
// where row streaming isn't supported in this direction.
SchemaOnly bool
// ParallelStreams is the requested max number of source tables copied
// concurrently. 0 means "use the default": runtime.NumCPU()-1 (with a
// floor of 1). Negative values are normalized to 1. Capped to 1 when
// either source or target advertises SupportsConcurrentConnections()==false
// (REQ:concurrency-cap).
ParallelStreams int
// Filters carries resolved filtering directives (table include/exclude,
// row WHERE predicates, row limits). nil or empty means "no filtering
// — copy whole DB per parent Feature ACs". Subsequent tasks consume
// this at two seams (pre-worker table filter; engine_rows query builder).
// Spec: spec/features/cli/db/copy/filtering/README.md
Filters *filter.Directives
}
CopyOpts controls a Copy call.
type NonEmptyTargetError ¶
NonEmptyTargetError is returned by checkEmptyTarget when at least one source-named table exists on the target with >=1 row. The user should rerun with --overwrite=recreate or --overwrite=reload.
func (*NonEmptyTargetError) Error ¶
func (e *NonEmptyTargetError) Error() string
type ProgressWriter ¶
type ProgressWriter struct {
// contains filtered or unexported fields
}
ProgressWriter emits per-table start/finish lines to a writer (intended to be os.Stderr in production). When enabled is false, all methods are no-ops. Safe for concurrent use across multiple worker goroutines.
Format is pinned by REQ:progress-reporting in spec/features/cli/db/copy/README.md.
func NewProgressWriter ¶
func NewProgressWriter(w io.Writer, enabled bool) *ProgressWriter
NewProgressWriter returns a ProgressWriter writing to w when enabled is true. If enabled is false (or w is nil), all methods are no-ops.
func (*ProgressWriter) FinishTable ¶
func (p *ProgressWriter) FinishTable(table string, rows int64, d time.Duration)
FinishTable announces that a copy of `table` has completed: `rows` inserted in `d`. Duration is rounded to millisecond per AC.
func (*ProgressWriter) StartTable ¶
func (p *ProgressWriter) StartTable(table string, estRows int64)
StartTable announces that a copy of `table` is starting. If estRows is negative, the start line prints `est. ? rows` instead of a number.
type ReloadSchemaMismatchError ¶
type ReloadSchemaMismatchError struct {
Table string
Column string
SourceValue string
TargetValue string
Reason string // short reason: "missing column", "type mismatch", "primary key mismatch"
}
ReloadSchemaMismatchError signals that the target's schema for a given table is not a superset of the source's, per REQ:reload-schema-match.
Column is either a column name (when a source column is missing or has an incompatible target type) or the literal "<primary key>" when the primary-key column sets differ.
SourceValue and TargetValue describe what differed (e.g. type names or PK column lists) in human-readable form, for the stderr diff.
func (*ReloadSchemaMismatchError) Error ¶
func (e *ReloadSchemaMismatchError) Error() string
type SourceSummary ¶
type SourceSummary struct {
Tables int
Created int
CreatedNames []string // names of collections created on the target, in source-iteration order
Skipped []string // tables skipped at DescribeCollection time
RowsCopied int64 // total rows inserted into the target across all tables
RowsByTable map[string]int64
RowSkips map[string]string // table → reason (e.g. composite PK, no PK)
TargetBackend string // adapter name of target, for error messages
}
SourceSummary is what Copy reports back to the caller.
func Copy ¶
Copy replicates the source database into the target — schema first, then row data per table.
Schema replication uses DALgo dbschema/ddl. Row streaming uses ExecuteQueryToRecordsReader on the source and RunReadwriteTransaction → InsertMulti on the target (see engine_rows.go).
If opts.SchemaOnly is true, only schema is replicated.
Tables the source can't describe (e.g. dalgo2sqlite rejecting DATETIME / NUMERIC) are appended to Skipped and processing continues. Tables with no PK get schema replicated but row copy is skipped with the reason recorded in RowSkips. Composite-PK tables are now copied (key encoded as `__`-joined PK values; see encodeRecordID in engine_rows.go).
Concurrency: opts.ParallelStreams governs how many tables are copied in parallel. The effective value is capped to 1 if either source or target advertises SupportsConcurrentConnections()==false. When the cap reduces an explicitly-requested value >1, one warning line is emitted on stderr.
Errors:
- ErrSourceHasNoTables — source introspects cleanly but has zero collections.
- any other error — wrapped with the failing operation and table name.
type UnsupportedTypeError ¶
type UnsupportedTypeError struct {
// SourceType is the dbschema.Type that could not be mapped.
SourceType dbschema.Type
// TargetBackend is the backend name supplied to MapType.
TargetBackend string
}
UnsupportedTypeError is returned by MapType when the source column type is outside the MVP coverage set.
func (*UnsupportedTypeError) Error ¶
func (e *UnsupportedTypeError) Error() string
Error implements the error interface. The message names both the source type and the target backend so the operator can act.
Source Files
Directories