Documentation
¶
Overview ¶
Package dbcopy implements the `datatug db copy` cross-engine database copy primitive. See spec/features/cli/db/copy/ for the contract.
Package dbcopy implements the cross-engine database-copy primitive for `datatug db copy`. This file holds the type-mapping table between the MVP backends (SQLite via dalgo2sqlite and inGitDB via dalgo2ingitdb).
Coverage bar: every column type appearing in the canonical Chinook fixture MUST map cleanly in both directions. Types outside that closed set return a *UnsupportedTypeError naming the source type and target backend.
Mapping policy for the MVP:
- Both backends speak the engine-neutral dbschema.Type vocabulary, so the translation is identity for every type currently produced by either driver's SchemaReader.
- Chinook's *.db SQLite fixture contains columns of SQLite affinity INTEGER, NVARCHAR/TEXT, NUMERIC(10,2), and DATETIME. As of dalgo2sqlite v0.0.0-20260513182736-6886f34af097 the driver's DescribeCollection rejects NUMERIC(p,s) and DATETIME columns (driver gap upstream of dbcopy). For the columns that DO describe successfully (Album, Artist, Customer, Genre, MediaType, Playlist, PlaylistTrack), only dbschema.Int and dbschema.String are observed. The MVP type-map declares the full dbschema vocabulary supported (Bool, Int, Float, String, Bytes, Time, Decimal) so the coverage holds the day dalgo2sqlite ships DATETIME / NUMERIC support and Track/Invoice/InvoiceLine/Employee describe successfully.
- dbschema.Null is intentionally rejected: it is the zero value used for "unset" FieldDef.Type and never a meaningful column type for cross-engine copy.
Package dbcopy implements `datatug db copy --from <url> --to <url>`.
This file covers the URL scheme dispatcher: parsing --from/--to arguments into a typed BackendRef and opening the underlying DALgo dal.DB.
Scheme support per spec/features/cli/db/copy/README.md (REQ:supported-schemes) — that spec predates http/https and does not document them yet:
- sqlite:// fully wired via dalgo2sqlite
- ingitdb:// fully wired via dalgo2ingitdb; local-paths-only (REQ:ingitdb-url-local-only)
- postgres:// parses; Open returns ErrPostgresNotWired until a PostgreSQL DALgo driver implements the three capability interfaces (dbschema.SchemaReader, ddl.SchemaModifier, dal.ConcurrencyAware)
- env:NAME resolves the environment variable NAME, which holds any other supported URL (the way to give PostgreSQL its password without writing it in a project file or on a command line); see parseEnvSource
- http:// https:// fully wired via dal-go/dalgo2http (pkg/httpsource); local-paths-only, same convention as ingitdb:// — see parseHTTPSource
Index ¶
- Constants
- Variables
- func CheckDescriptorEnvName(name string, lookupEnv func(string) (string, bool)) error
- func CheckSourceFile(path string) error
- func IsPlainSourceID(id string) bool
- func LocalSourceURL(scheme, path string) string
- func MapType(t dbschema.Type, sourceBackend, targetBackend string) (dbschema.Type, error)
- func PathDisplay(path string) string
- func PathHoldsURL(path string) bool
- func ProjectSourceURL(projectDir string) string
- func QueryIDDisplay(id string) string
- func RedactError(err error) error
- func RedactErrorWithLiterals(err error, secrets ...string) error
- func RedactErrorWithSecrets(err error, source string) error
- func RedactSourceURL(raw string) string
- func RedactText(text string) string
- func RedactTextWithSecrets(text string, secrets ...string) string
- func SourceDisplay(raw string) string
- func SourceIDDisplay(id string) string
- func SourceScopeIdentity(source string) string
- func SupportedSchemes() []string
- func ValidEnvName(name string) bool
- type BackendRef
- func (r BackendRef) CheckFile() error
- func (r BackendRef) Display() string
- func (r BackendRef) GoString() string
- func (r BackendRef) Open(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenFailure(err error) error
- func (r BackendRef) OpenForTest(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenProtected(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenProtectedForTest(ctx context.Context) (dal.DB, error)
- func (r BackendRef) OpenSchemaScan(_ context.Context) (SchemaScanDB, error)
- func (r BackendRef) String() string
- type CopyOpts
- type NonEmptyTargetError
- type PostgresDescriptor
- type PostgresTarget
- type ProgressWriter
- type ReloadSchemaMismatchError
- type SchemaScanDB
- type SourceSummary
- type UnsupportedTypeError
Constants ¶
const ( // DescriptorEnvPrefix is the name prefix a project descriptor's dsnEnv may // always carry: a variable the operator named for DataTug. DescriptorEnvPrefix = "DATATUG_" // DescriptorEnvAllowList names the operator-set variable that lists, comma or // space separated, the other variables a project descriptor may name. It // follows the shape of the OpenVaultDB descriptor, whose token is bound to a // destination the operator sets (pkg/openvaultdb/source.go): the project // file proposes, the operator's own environment decides. DescriptorEnvAllowList = "DATATUG_DSN_ENV_ALLOW" )
const ( BackendSQLite = "sqlite" BackendInGitDB = "ingitdb" )
Supported backend identifiers accepted by MapType.
const EnvSourceForm = envPrefix + "NAME"
EnvSourceForm is how the env source form is spelled in help text.
const PostgresDefaultSchema = dalgo2postgres.DefaultSchema
PostgresDefaultSchema is the PostgreSQL schema a schema scan reads: the one the DALgo reader inspects when it is not told another.
const SourceIDNotShown = "<source id not shown>"
SourceIDNotShown is what SourceIDDisplay returns for a source ID that is not a plain name. It does not say the ID is wrong: a valid ID may hold a space or a slash, and a client may send a source string where an ID belongs; either way the text is not echoed.
const UnparsableSource = "<unparsable source>"
UnparsableSource is what SourceDisplay returns for a string that does not read as a source. Nothing from the input is in it.
Variables ¶
var ErrNoPrimaryKey = errors.New("source collection has no primary key declared")
ErrNoPrimaryKey is returned when a source collection has no PK declared. The MVP row-copy path requires a declared PK to construct target record keys.
var ErrPostgresNotWired = errors.New("PostgreSQL backend not yet wired")
ErrPostgresNotWired is returned by BackendRef.Open for postgres:// URLs until a PostgreSQL DALgo driver implements the three capability interfaces (dbschema.SchemaReader, ddl.SchemaModifier, dal.ConcurrencyAware).
var ErrSourceFileMissing = errors.New("source file does not exist")
ErrSourceFileMissing is wrapped by CheckSourceFile (and, through it, by Open's sqlite/ingitdb branches) when a file-backed source does not exist on disk — e.g. `datatug serve --project` against a demo project before `datatug demo` has fetched ~/datatug/dbs/chinook-local.sqlite. Checked with errors.Is so a caller (pkg/secureread, pkg/server/endpoints) can map it to api-contract.md's SOURCE_UNAVAILABLE (503) instead of letting the driver's own opaque "unable to open database file" text reach an HTTP 500.
var ErrSourceHasNoTables = errors.New("source has no tables; nothing to copy")
ErrSourceHasNoTables signals that the source introspected cleanly but has zero collections. Callers should exit 0 with a stderr note per REQ:source-introspection-failure.
Functions ¶
func CheckDescriptorEnvName ¶ added in v0.54.0
CheckDescriptorEnvName reports whether a project descriptor may name the environment variable name. A project file is not trusted: a cloned project could otherwise select any database whose URL the operator keeps in the environment. So a descriptor may name only a variable whose name starts with DATATUG_, or one the operator lists in DATATUG_DSN_ENV_ALLOW. name must already match ValidEnvName, which keeps the error free to quote it.
func CheckSourceFile ¶ added in v0.20.1
CheckSourceFile reports ErrSourceFileMissing (wrapping the path and a `datatug demo` recovery hint) when path does not exist on disk, and nil when it does or when the stat fails for any other reason (permissions, etc. — left for the underlying driver to report on its own terms). It is exported so callers that open a file-backed source through a path other than BackendRef.Open (pkg/secureread's read-only native-SQL connection) can perform the identical check.
The path in the message is the display form of path (see PathDisplay): the query string and the fragment are cut off, a path that starts like userinfo is shown without it, and a URL handed in by mistake is shown as SourceDisplay shows it, so no caller can put a secret in the message by passing what the source string held.
func IsPlainSourceID ¶ added in v0.54.4
IsPlainSourceID reports whether id is a plain name for a source in a project: letters and digits of any script, "." "_" and "-", at most 128 characters, starting with a letter or a digit. It is the one definition of a plain name: SourceIDDisplay shows an ID only when it holds, and an ID that becomes a folder or file name must satisfy it. Compare with it, not with SourceIDDisplay(id) == id, because SourceIDNotShown is itself text a client can send.
func LocalSourceURL ¶ added in v0.53.0
LocalSourceURL returns the "scheme://path" source URL that names the local file or directory path for a scheme that takes one (sqlite, openvaultdb, ingitdb, http, https). A path that holds an "@" and does not start like a path (see readsAsPath): a relative directory such as my@proj or team/a@b, or one shaped like "user:password@host" (a:b/c@d), is written "./path", because Parse refuses "scheme://my@proj" and "scheme://a@b.db" as URLs that carry credentials (and dburl reads the second as user a and opens b.db). Every caller that builds such a URL from a path it was given verbatim goes through here, so a directory such as my@proj keeps working. A path that starts with a UNC start ("\\server\share") is written as it is, never with the dot: it is not a relative directory, and "./" in front of two backslashes names a path that does not exist. A WebDAV path ("\\host@SSL\share") is a path and Parse takes it; one that holds credentials after the UNC start is left for Parse to refuse.
func MapType ¶
MapType translates a column type from sourceBackend to targetBackend. The returned Type is ready for use in a target ddl.CreateCollection call. For the MVP, sourceBackend and targetBackend MUST each be one of "sqlite" or "ingitdb". Types outside the Chinook coverage set (dbschema.Null and any unrecognized Type value) return a *UnsupportedTypeError naming the source type and target backend.
Both backends accept the same engine-neutral dbschema.Type vocabulary, so this function is the identity for every supported type today. The function exists as the seam where engine-specific widening / narrowing rules will land if and when the type-mapping matrix needs them.
func PathDisplay ¶ added in v0.54.1
PathDisplay returns the text to show for path, a file or directory path a source resolved to (BackendRef.Path), a path given with no scheme, or a URL when a caller hands in one.
func PathHoldsURL ¶ added in v0.54.4
PathHoldsURL reports whether path, the path field of a catalog, is a URL ("https://tok@host/x") or holds one in front of its last "@" ("./https://tok@host"): it is then not a file or a directory, and what follows its scheme may be credentials, so a caller refuses it before it is joined to a folder, which would turn it into a relative path that a message shows whole. A drive path ("C://work/a@b.db") is a path: its drive letter is not a scheme.
func ProjectSourceURL ¶ added in v0.53.0
ProjectSourceURL returns the http:// source URL that names the datatug project directory projectDir, the way the commands and the server build one from a directory they were given verbatim. A relative directory whose first segment holds an "@" (my@proj, @acme/proj) is written "./dir", because Parse refuses "http://my@proj" as a URL that carries credentials.
func QueryIDDisplay ¶ added in v0.54.4
QueryIDDisplay returns id, a saved query's ID, when every "/"-separated part of it is a plain name (see IsPlainSourceID) and SourceIDNotShown otherwise: a query ID is the folders of the query and its name, and a client may send a whole source string where one belongs. Use it to name a query ID a client sent in a message.
func RedactError ¶ added in v0.53.0
RedactError returns err with a message passed through RedactText. It wraps err, so errors.Is and errors.As still see the original; only Error() text is redacted. A nil err stays nil. Like RedactText it is a last line of defence for text that code outside this package wrote; no message that names a source may rely on it (see SourceDisplay and BackendRef.OpenFailure).
func RedactErrorWithLiterals ¶ added in v0.53.0
RedactErrorWithLiterals is RedactError for an error written by code that was handed secrets as plain values (a password given as a flag): it also removes every literal in secrets, in the spellings secretSpellings lists.
func RedactErrorWithSecrets ¶ added in v0.53.0
RedactErrorWithSecrets is RedactError for an error written by code that was handed source, the real URL: besides the URL-shaped text RedactError finds, it removes every literal secret source holds, so a driver that formats the connection string in its own way still cannot put the password in a message.
func RedactSourceURL ¶ added in v0.53.0
RedactSourceURL returns raw with every secret replaced by "xxxxx": the password in "user:password@host", the value of any query parameter whose name says it is a secret (password, sslpassword, token, key ...), and the value of a libpq "password=..." keyword. A string that holds no secret comes back unchanged, byte for byte.
It works on the text and never parses, so it cannot fail and a malformed URL is still redacted. It prefers to over-redact: the userinfo ends at the last "@" in the string, so a password that holds an unescaped "@", "/" or "?" is still removed whole. There are two exemptions, both for http and https only, whose URLs name a local project directory: an authority that is a plain host or host:port (see isHostOnly), whose "@" belongs to the path, and a Windows drive path (see redactUserinfo).
Nothing that is shown may depend on this function: a recogniser of secrets can always be beaten by one more shape of string. Use SourceDisplay to show a source.
func RedactText ¶ added in v0.53.0
RedactText returns text with every URL inside it passed through RedactSourceURL and every "password=..." keyword value removed.
It is a last line of defence and nothing else: the top-level error and panic handlers (main.go) and the sinks that print text written by code that does not build it (an HTTP error body, a chat error) run text through it in case something slipped by. No message path relies on it. Every message that names a source builds the name with SourceDisplay from parts that passed a strict check, and every parser error and every driver error of opening a source is classified, never quoted (see BackendRef.OpenFailure), so there is no secret in the text for this to find. A driver error after a source is open still passes through RedactError: it is the one place left that relies on a recogniser. A recogniser of secrets in text cannot be finished (it was patched three times and each round found another shape of string that got through), so do not add a message that needs it.
func RedactTextWithSecrets ¶ added in v0.53.0
RedactTextWithSecrets is RedactText for text that may hold secrets the caller knows as plain values (a password it was given as a flag): besides everything RedactText removes it replaces every literal in secrets, in the spellings secretSpellings lists.
func SourceDisplay ¶ added in v0.54.1
SourceDisplay returns the text to show for the source string raw. It is the only way the CLI names a source in a message.
The result is the scheme (lower-cased, and on the known list), then, for a database URL, the host, the port and a one-segment path, or, for a file or a directory, its path with the query string and the fragment cut off. A "host:port/path" that follows a user name and a password is shown without them: whatever precedes the last "@" is userinfo, because a password may hold an "@", a "/", a "?" or only digits and a parser may read it as a host and a port. The same holds for a file or a directory unless the text is certainly a path (see readsAsPath): an explicit path ("/abs/a@b", "./a@b", "../a@b", "~/a@b", a UNC path, a drive path) is shown as typed, unless it holds a second "scheme://" in front of its last "@", or is a UNC start with a ":" in front of it, or one whose server name (the first segment) is empty or holds an "@" (a Windows WebDAV server name, "host@SSL" or "host@8080", excepted): that is a URL or credentials, whatever the text starts with. Any other text that holds an "@" is userinfo and a host, even when a slash comes before the colon or the "@". Two shapes stay paths by design: an explicit path, and "X:/..." or "X:\..." (read as a Windows drive path even when it was meant as user X with a password that starts with a slash). An "env:NAME" source is shown as it is, because it names a variable and holds no value. A string that is not led by a known scheme yields UnparsableSource.
func SourceIDDisplay ¶ added in v0.54.1
SourceIDDisplay returns id when it is a plain name (letters and digits of any script, "." "_" and "-", at most 128 characters, starting with a letter or a digit) and SourceIDNotShown otherwise. Use it to name a source ID a client sent in a message: a client may send a whole source string where an ID belongs.
func SourceScopeIdentity ¶ added in v0.54.0
SourceScopeIdentity returns the text that stands for source in a persisted scope identity, such as the key of a chat session store. It is never the display form alone (SourceDisplay): that drops the user name and the query of a URL and cuts a path at its "?" and "#", so two sources that are different places would share a scope. And it never holds a password.
An "env:NAME" source names a variable, and the variable can be repointed at another database while its name stays: under the name alone, record sets read from the old database would stay reachable in the new one. So the identity of an env source is the name plus a hash of where the variable points now: for a PostgreSQL URL its host, port, database and user, never the password. The hash is the same for the same destination, whatever the password or the connection options, and differs for any other. A part the URL leaves out is what libpq's environment variables give it (PGHOST, PGPORT, PGDATABASE, PGUSER and PGSERVICE), and the database is the user when nothing names one, as pgx resolves them; the contents of a connection service file and the operating system's user name are not part of the identity. A variable that cannot be resolved is its own scope.
A literal PostgreSQL URL is identified the same way: "postgres" and the hash of where it points and as whom, so two roles on one host and database are two scopes, and the spelling of the URL (the postgresql alias, the case of the host, the default port) is not part of it.
A file or a directory is identified by its whole path, with the "?" and "#" the display form cuts off (a sqlite URL's query holds driver options, which can include a key, and is not part of its path). When that is not the display form, the identity is the display form and a hash of the whole path, so the text holds nothing the display form does not. A source that Parse refuses can hold credentials: its identity is its display form.
func SupportedSchemes ¶ added in v0.17.2
func SupportedSchemes() []string
SupportedSchemes returns the exact schemes Parse/Open dispatch, in dispatch order. It is the single source of truth other packages should build user-facing scheme lists from (e.g. a `--db` flag's help text) so that text cannot drift from what Open actually accepts the way it did when http/https were wired in here without every caller's help text being updated to match.
func ValidEnvName ¶ added in v0.53.0
ValidEnvName reports whether name is an acceptable environment variable name for an "env:NAME" source or a PostgreSQL descriptor's dsnEnv.
Types ¶
type BackendRef ¶
type BackendRef struct {
Scheme string
// Path holds the scheme-specific resource locator.
// - sqlite: filesystem path to the .db file (e.g. "/tmp/foo.db" or "./rel.db")
// - ingitdb: filesystem path to the project directory
// - postgres: full original URL (passed verbatim to the future driver).
// It can hold the password: never print or store Path; use
// Raw or String for any message.
// - http/https: filesystem path to the datatug project directory whose
// queries/ tree declares the HTTP QueryDefs to serve (see
// pkg/httpsource) — same local-path convention as ingitdb,
// NOT a literal remote endpoint; the project's own query
// definitions name the actual remote endpoints.
Path string
// Raw is the text to show for the source in a message: the display form of
// the input (see SourceDisplay), built from its scheme, host, port and path
// and never holding userinfo, a query string or a fragment. An "env:NAME"
// input stays "env:NAME" because the value of the variable is never copied
// here. The name is historical: Raw is never the input as typed.
Raw string
}
BackendRef is a parsed --from/--to URL.
func Parse ¶
func Parse(rawURL string) (BackendRef, error)
Parse parses a CLI URL argument into a BackendRef. It returns an error for unknown schemes (REQ:unknown-scheme-rejected), malformed URLs, and remote ingitdb:// URLs (REQ:ingitdb-url-local-only).
The unknown-scheme error message names BOTH the unsupported scheme AND the supported list, as required by REQ:unknown-scheme-rejected.
An input of the form env:NAME is resolved first: NAME must match ^[A-Z][A-Z0-9_]*$ and name an environment variable holding any other supported URL. No error ever echoes the variable's value.
func ParseWithEnv ¶ added in v0.54.0
ParseWithEnv is Parse over an injected environment lookup, so a caller's tests never touch the process environment. Parse is ParseWithEnv with os.LookupEnv.
func (BackendRef) CheckFile ¶ added in v0.54.1
func (r BackendRef) CheckFile() error
CheckFile reports ErrSourceFileMissing when this source is backed by a file or directory (sqlite, ingitdb, or the connection descriptor of openvaultdb) and that does not exist; for every other source it returns nil. It is CheckSourceFile for a caller that holds the ref: a PostgreSQL ref's Path is its URL and is not a path to stat. Open itself calls it for sqlite and ingitdb only: openvaultdb reports its own missing descriptor.
func (BackendRef) Display ¶ added in v0.54.1
func (r BackendRef) Display() string
Display returns the text to show for this source in a message. It is built by SourceDisplay and never holds userinfo, a query string or a fragment.
func (BackendRef) GoString ¶ added in v0.53.0
func (r BackendRef) GoString() string
GoString makes %#v print the same display text as String.
func (BackendRef) Open ¶
Open opens the underlying DALgo dal.DB for this BackendRef.
Dispatch:
- sqlite: opens via dalgo2sqlite.NewDatabase.
- ingitdb: opens via dalgo2ingitdb.NewDatabase with the default validator-backed CollectionsReader.
- postgres: returns ErrPostgresNotWired (no DALgo Postgres driver yet exposes the three capability interfaces).
- http/https: opens via httpsource.Open, translating every HTTP QueryDef under the project directory (r.Path) into a dalgo2http collection.
Open never applies the provider-side read hardening OpenProtected does (see its doc comment): every caller here — `datatug db copy`, and schema introspection in pkg/server/endpoints — is a trusted, operator-level caller with no pkg/accesspolicies wrapper above it, so a formula/computed column is returned exactly as the provider evaluates it, matching every dalgo2sql/dalgo2ingitdb release before Task 13 (S110).
The context is reserved for future use; today's driver constructors are synchronous and do not honor cancellation. That's acceptable for the MVP CLI verb.
func (BackendRef) OpenFailure ¶ added in v0.54.1
func (r BackendRef) OpenFailure(err error) error
OpenFailure turns err, what a driver returned when it opened this source, into an error that is safe to show. A driver formats the connection string it was given into its open and ping errors in whatever shape it likes (dalgo2postgres quotes the whole DSN), so a driver's message is never shown, whatever it holds: the error says which source could not be opened and, for the few causes that can be told apart without reading the message (a missing file, a refused permission, a timeout, a cancelled attempt, a refused connection), why. errors.Is and errors.As still see the driver's own error.
The errors this package wrote itself (the missing-file error CheckSourceFile returns, an error OpenFailure returned before, ErrPostgresNotWired) carry text built from the display form of the source and pass through unchanged. They are recognised by what they are, never by errors.Is: a driver can wrap one of them around its own text. A nil err stays nil.
func (BackendRef) OpenForTest ¶ added in v0.20.4
OpenForTest is Open, except that for an "http"/"https" BackendRef every dalgo2http.Collection it builds gets Collection.InsecureAllowLoopback set (dal-go/dalgo2http v0.2.0's TEST-ONLY escape hatch — see httpsource.AllowInsecureLoopback's doc comment). Every other scheme behaves identically to Open.
It exists so a test that drives the full sourceURL -> Parse -> Open pipeline in-process (e.g. apps/datatugapp/commands's cmd_query_http_provenance_test.go, pkg/secureread's executor_provenance_test.go via Executor.RunStructuredInsecureForTest) can point an HTTP QueryDef's .query.http file at a loopback httptest.Server or an intentionally-unreachable loopback address (e.g. 127.0.0.1:1, for a fast deterministic live-failure), without any project descriptor file ever requesting that itself — the field is set here, in Go code, only when a caller explicitly calls THIS method instead of Open. NEVER call this from production code.
func (BackendRef) OpenProtected ¶ added in v0.20.8
OpenProtected is Open, except an "ingitdb" BackendRef is opened with dalgo2ingitdb.WithStoredOnlyReads(). SQLite uses the validated, parameter-bound structured-query dialect on both paths. Other schemes behave like Open.
pkg/secureread.openSource uses this for policy-secured sessions; direct `query run` uses it for SQLite structured queries. A policy-secured session wraps the returned dal.DB with pkg/accesspolicies before any row reaches a caller. dalgo2ingitdb's own formula evaluator has no way to know which of a computed column's dependencies pkg/accesspolicies would have redacted — the adapter computes the value from the FULL underlying record and hands back the (correct) result, which can leak a hidden field's value through an allowed computed column (see dal-go/dalgo2ingitdb#8 / this repo's README "Owner access policies" section). WithStoredOnlyReads keeps dalgo2ingitdb from evaluating or returning any formula column at all under an outer policy wrapper, so pkg/accesspolicies' field allow-list is the only thing that can ever put a value on the wire — it stays the single enforcement point. This adapter never writes an .ingitdb/access/manifest.yaml file of its own, so dalgo2ingitdb's persisted owner-policy layer (also new in v0.4.0) never activates for a datatug-cli-opened project; policies do NOT layer under pkg/accesspolicies here — pkg/accesspolicies remains the sole enforcement layer for every source this CLI opens.
The sqlite scheme opts every read, protected or not, into dalgo2sql's DbOptions.StructuredQueryDialect: "sqlite" (bounded, parameter-bound structured-query compilation): dal-go/dalgo2sql#179 added FROM-source alias support to compileStructuredSQL, which used to unconditionally reject any structured query whose FROM source carried an alias — this project's own demo query (queries/customers/customer-invoices.query.dtql, `from: {name: Invoice, alias: i}`) uses exactly that shape, and used to turn into a hard error the moment this dialect was enabled. With #179 fixed, existing non-aggregate DTQL queries (see dalgo2sql's own dtql_datatug_inventory_test.go) compile cleanly, so protected reads now get the dialect's real guarantees: every dal.Constant value becomes a genuine `?` placeholder + bound arg (not a quoted-string literal), and unsupported shapes (such as joins and cursors) fail closed. GROUP BY and HAVING are compiled by dalgo2sql's aggregation path. The plain Open path (db copy / introspection) sets the same dialect, so it never reaches the legacy emitSQL renderer; only the ingitdb hardening differs between Open and OpenProtected.
func (BackendRef) OpenProtectedForTest ¶ added in v0.20.8
OpenProtectedForTest combines OpenProtected's provider-side read hardening with OpenForTest's http(s) loopback escape hatch. It exists for pkg/secureread's Executor.RunStructuredInsecureForTest, which must drive the exact same openSource -> BackendRef -> pkg/accesspolicies pipeline RunStructured uses in production, just against a loopback test server. NEVER call this from production code.
func (BackendRef) OpenSchemaScan ¶ added in v0.54.0
func (r BackendRef) OpenSchemaScan(_ context.Context) (SchemaScanDB, error)
OpenSchemaScan opens a postgres source for reading its schema and nothing else. It is not Open: Open keeps answering that PostgreSQL is not available for queries, and the handle returned here is for the scan only. DataTug opens it with exact identifiers, so a table is looked up under the name PostgreSQL reports (the driver's default folds every name to lower case).
The URL is checked before the driver sees it (ParsePostgresTarget): a URL that net/url and pgx read differently from how it was written, such as a password with an unescaped "/" after digits, is refused, because the driver would connect to the wrong host and print the rest of the password as the database.
The driver quotes the URL, password included, in its open and ping errors, so its message is never shown: the error is OpenFailure's, a fixed sentence built from the display form of the source (the "env:NAME" it was opened from, never the URL: it carries every connection option, not only the password) that says why only for a cause that can be told apart without reading the message. The driver's own error stays reachable through errors.Is and errors.As, exactly as it does for Open. The context is reserved for future use, as in Open.
func (BackendRef) String ¶ added in v0.53.0
func (r BackendRef) String() string
String returns the display form of the source (see Display), so printing a BackendRef with %v or %+v can never put a password in a log or message.
type CopyOpts ¶
type CopyOpts struct {
// Overwrite is "" (require empty target), "recreate" (drop source-named
// tables, then create from source), or "reload" (reserved — row-level
// semantics; behaves like "recreate" for schema-only copies until row
// CRUD lands).
Overwrite string
// Stderr receives per-table skip / row-error notes and the final
// summary note. Defaults to discard if nil.
Stderr io.Writer
// Progress, if non-nil, receives per-table progress lines.
Progress *ProgressWriter
// SchemaOnly, if true, skips row streaming entirely and only replicates
// the target schema. Useful for E2E-test scaffolding and for backends
// where row streaming isn't supported in this direction.
SchemaOnly bool
// ParallelStreams is the requested max number of source tables copied
// concurrently. 0 means "use the default": runtime.NumCPU()-1 (with a
// floor of 1). Negative values are normalized to 1. Capped to 1 when
// either source or target advertises SupportsConcurrentConnections()==false
// (REQ:concurrency-cap).
ParallelStreams int
// Filters carries resolved filtering directives (table include/exclude,
// row WHERE predicates, row limits). nil or empty means "no filtering
// — copy whole DB per parent Feature ACs". Subsequent tasks consume
// this at two seams (pre-worker table filter; engine_rows query builder).
// Spec: spec/features/cli/db/copy/filtering/README.md
Filters *filter.Directives
}
CopyOpts controls a Copy call.
type NonEmptyTargetError ¶
NonEmptyTargetError is returned by checkEmptyTarget when at least one source-named table exists on the target with >=1 row. The user should rerun with --overwrite=recreate or --overwrite=reload.
func (*NonEmptyTargetError) Error ¶
func (e *NonEmptyTargetError) Error() string
type PostgresDescriptor ¶ added in v0.53.0
type PostgresDescriptor struct {
// DSNEnv is the name of the environment variable that holds a postgres://
// or postgresql:// URL. It must match ^[A-Z][A-Z0-9_]*$.
DSNEnv string `json:"dsnEnv"`
}
PostgresDescriptor is the project-owned file that points a PostgreSQL source at its credentials. It names an environment variable and nothing else: the variable holds the whole connection URL (host, user and password together), so no project file can hold a password, and a project file cannot redirect a credential to a host of its own choosing.
func DecodePostgresDescriptor ¶ added in v0.53.0
func DecodePostgresDescriptor(data []byte) (PostgresDescriptor, error)
DecodePostgresDescriptor strictly decodes a descriptor: one JSON object, no duplicate keys, no field other than dsnEnv, and a valid variable name. No error it returns holds a value from the input; an unknown field is named only when the name is a plain identifier.
func ReadPostgresDescriptor ¶ added in v0.53.0
func ReadPostgresDescriptor(path string) (PostgresDescriptor, error)
ReadPostgresDescriptor reads and strictly decodes the descriptor at path, and refuses a dsnEnv that CheckDescriptorEnvName does not allow in the process environment.
func (PostgresDescriptor) SourceURL ¶ added in v0.53.0
func (d PostgresDescriptor) SourceURL() string
SourceURL returns the source URL that resolves this descriptor: "env:NAME". It names the variable and never holds its value, so it is safe to pass through the chat store, logs and error messages.
type PostgresTarget ¶ added in v0.54.0
type PostgresTarget struct {
Host string
// Port is 0 when the URL names none.
Port int
Database string
User string
}
PostgresTarget is the part of a PostgreSQL connection URL that says where it points. It never holds the password.
func ParsePostgresTarget ¶ added in v0.54.0
func ParsePostgresTarget(rawURL string) (PostgresTarget, error)
ParsePostgresTarget reads the destination a postgres:// or postgresql:// URL states: the host, port, user and database of the URL, each replaced by the query parameter of the same name (host, port, user, dbname or database) when there is one, as pgx applies them. It reads the URL only: what pgx takes from the PG* environment variables or a service file for a part the URL leaves out is not here.
It refuses a URL with a literal "@" after the authority (see atSignAfterAuthority): such a URL is not read the way it was written, and refusing it is the only way to keep the rest of a password out of the host, database and scope that are printed or hashed. It also refuses a user name that holds a colon, however the URL spells it (see errPostgresUserHoldsColon): "alice%3Apw@host" has no literal colon, so net/url reads the user "alice:pw" and no password, and the password would be printed as the user, where no redactor looks for a secret. No error quotes the URL, which holds the password.
type ProgressWriter ¶
type ProgressWriter struct {
// contains filtered or unexported fields
}
ProgressWriter emits per-table start/finish lines to a writer (intended to be os.Stderr in production). When enabled is false, all methods are no-ops. Safe for concurrent use across multiple worker goroutines.
Format is pinned by REQ:progress-reporting in spec/features/cli/db/copy/README.md.
func NewProgressWriter ¶
func NewProgressWriter(w io.Writer, enabled bool) *ProgressWriter
NewProgressWriter returns a ProgressWriter writing to w when enabled is true. If enabled is false (or w is nil), all methods are no-ops.
func (*ProgressWriter) FinishTable ¶
func (p *ProgressWriter) FinishTable(table string, rows int64, d time.Duration)
FinishTable announces that a copy of `table` has completed: `rows` inserted in `d`. Duration is rounded to millisecond per AC.
func (*ProgressWriter) StartTable ¶
func (p *ProgressWriter) StartTable(table string, estRows int64)
StartTable announces that a copy of `table` is starting. If estRows is negative, the start line prints `est. ? rows` instead of a number.
type ReloadSchemaMismatchError ¶
type ReloadSchemaMismatchError struct {
Table string
Column string
SourceValue string
TargetValue string
Reason string // short reason: "missing column", "type mismatch", "primary key mismatch"
}
ReloadSchemaMismatchError signals that the target's schema for a given table is not a superset of the source's, per REQ:reload-schema-match.
Column is either a column name (when a source column is missing or has an incompatible target type) or the literal "<primary key>" when the primary-key column sets differ.
SourceValue and TargetValue describe what differed (e.g. type names or PK column lists) in human-readable form, for the stderr diff.
func (*ReloadSchemaMismatchError) Error ¶
func (e *ReloadSchemaMismatchError) Error() string
type SchemaScanDB ¶ added in v0.54.0
SchemaScanDB is the handle a schema scan reads through: the DALgo database, its schema reader, and a Close that releases the connection pool.
type SourceSummary ¶
type SourceSummary struct {
Tables int
Created int
CreatedNames []string // names of collections created on the target, in source-iteration order
Skipped []string // tables skipped at DescribeCollection time
RowsCopied int64 // total rows inserted into the target across all tables
RowsByTable map[string]int64
RowSkips map[string]string // table → reason (e.g. composite PK, no PK)
TargetBackend string // adapter name of target, for error messages
}
SourceSummary is what Copy reports back to the caller.
func Copy ¶
Copy replicates the source database into the target — schema first, then row data per table.
Schema replication uses DALgo dbschema/ddl. Row streaming uses ExecuteQueryToRecordsReader on the source and RunReadwriteTransaction → InsertMulti on the target (see engine_rows.go).
If opts.SchemaOnly is true, only schema is replicated.
Tables the source can't describe (e.g. dalgo2sqlite rejecting DATETIME / NUMERIC) are appended to Skipped and processing continues. Tables with no PK get schema replicated but row copy is skipped with the reason recorded in RowSkips. Composite-PK tables are now copied (key encoded as `__`-joined PK values; see encodeRecordID in engine_rows.go).
Concurrency: opts.ParallelStreams governs how many tables are copied in parallel. The effective value is capped to 1 if either source or target advertises SupportsConcurrentConnections()==false. When the cap reduces an explicitly-requested value >1, one warning line is emitted on stderr.
Errors:
- ErrSourceHasNoTables — source introspects cleanly but has zero collections.
- any other error — wrapped with the failing operation and table name.
type UnsupportedTypeError ¶
type UnsupportedTypeError struct {
// SourceType is the dbschema.Type that could not be mapped.
SourceType dbschema.Type
// TargetBackend is the backend name supplied to MapType.
TargetBackend string
}
UnsupportedTypeError is returned by MapType when the source column type is outside the MVP coverage set.
func (*UnsupportedTypeError) Error ¶
func (e *UnsupportedTypeError) Error() string
Error implements the error interface. The message names both the source type and the target backend so the operator can act.
Source Files
Directories