dalgo2sqlite

package module
v0.6.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 12, 2026 License: MIT Imports: 22 Imported by: 2

README

dalgo2sqlite

SQLite-specific DALgo driver. Wraps github.com/dal-go/dalgo2sql to provide the dal.DB surface, and adds SQLite-native implementations of:

  • dbschema.Adapter — schema introspection via sqlite_master and pragma_table_info
  • ddl.Applier — SQLite-flavored CREATE TABLE / CREATE INDEX / DROP TABLE etc.
  • dal.ConcurrencyAware — advertises Concurrency() = 1 for write paths (SQLite is single-writer)

Used by consumers (e.g. datatug-cli's db copy) that need schema-modification and concurrency hints through the unified DALgo abstraction without hand-rolling engine-specific SQL.

Our approach to development

We build with our own tooling:

  • SpecScore — specify requirements as SpecScore.md artifacts
  • SpecStudio — author & manage specs across their lifecycle
  • inGitDB — store structured data in Git where applicable
  • DALgo — data access layer for Go
  • cover100.dev — drive toward 100% test coverage
  • DataTug — query & explore data

SQLite driver: modernc.org/sqlite (pure Go, CGO_ENABLED=0)

This package uses modernc.org/sqlite — a pure-Go transpilation of the SQLite C library — instead of the former github.com/mattn/go-sqlite3 cgo binding. The migration was done because:

  • The cgo driver required a C toolchain in every build environment (CI, containers, cross-compilation targets).
  • It prevented single static binaries for downstream consumers such as datatug-cli.
  • modernc.org/sqlite exposes the same database/sql interface (driver name "sqlite" instead of "sqlite3"), so dalgo2sql continues to work unchanged.
  • All existing SQLite features relied on here — sqlite_master introspection, PRAGMA table/index queries, transactional DDL — behave identically under the pure-Go driver.

The cgo_enabled: true flag can be removed from this repo's CI workflow and from any downstream workflow that no longer needs cgo for other reasons.

TugQL JOIN acceptance

The checked-in testdata/joins directory contains a small Chinook-shaped SQLite seed, the canonical nested Invoice/Customer/Employee TugQL query, and expected rows. The test loads the seed into a temporary database, runs the query through both the database and read-transaction entry points, and checks ordered results and LEFT JOIN null extension. A second query carries independent algorithm preferences on outer, nested, and sibling JOINs; SQLite's native execution may ignore those physical preferences and must return the same rows. The tests also check the canonical fixture hashes, their DALgo source commit, and the expected diagnostics for malformed JOIN references.

Run the complete acceptance journey from this repository's root:

go test ./end2end -run 'TestTugQL(NestedJoinChinook|JoinFixtureManifest|JoinNegativeFixtures)' -count=1

Optional pinned table reads

OpenTableRead returns an owned TableReadSpan for one existing absolute SQLite file. Its first supported profile requires the exact native file route, UTF-8 encoding, explicit main, and a simple ASCII ordinary-table name. Views, virtual/shadow/generated tables, SQLite internal tables, attachments, temp databases and implicit or rewritten identifiers are refused. Existing NewDatabase behavior is unchanged.

The span privately retains one mode=ro connection and a rollback-only transaction. It establishes query_only, native limits, exact catalog spelling and ordinary columns before a data reader is available. Facts subsequently returns cloned immutable metadata and checks liveness without issuing SQL. Session exposes only DALgo read methods with a fixed SQLite compiler and native values; every call remains linked to the original span cancellation and the earlier caller/span deadline. Readers retain that context until close, EOF or error. Always close readers and the span and inspect their errors. Observed cleanup errors are returned; database/sql may consume an error from its automatic rollback after cancellation, so universal rollback diagnostics are not promised.

MaxBytes is used to derive a native per-cell limit, with a separate MaxColumns ceiling. After discovering the selected table's actual column count N, the span sets and reads back SQLITE_LIMIT_LENGTH as floor(MaxBytes/N), requiring at least eight bytes per column. This native per-cell bound is reported as MaxCellBytes. The sum of accepted native value bytes is bounded by MaxBytes only when the host enforces a frozen one-table query returning exactly those original N table columns. Session is a generic read surface: it does not enforce query shape or permissions, and arbitrary projections/expressions do not inherit that combined-value bound.

UTF-8 admission avoids native UTF-16-to-UTF-8 expansion. The per-cell checks are conservative and all-or-nothing: a cell below MaxBytes can still exceed MaxCellBytes and refuse the operation. There is no filtering or truncation. Row count, serialized output, supported value classes, finite numbers and valid UTF-8 still need independent host bounds/validation. SQLite row headers, pager/cache, opaque driver storage and Go heap overhead are outside the bound.

These facts are neither permissions nor authenticated source identity. The host must authorize before opening, secure the returned DALgo session, and independently bind its owned/fixed private file route to the authenticated source. Filename describes SQLite's logical opened-file route; it does not attest OS inode identity or protect against external filesystem replacement. Caller-owned records and opaque driver buffers are not covered by a purge promise. The owned temporary-file tests establish this optional API profile; they do not certify every SQLite configuration or activate any application security routes.

Documentation

Overview

Package dalgo2sqlite is the SQLite-specific DALgo driver.

It composes github.com/dal-go/dalgo2sql for the dal.DB read/write surface (transactions, recordset reader, Get/Set/Insert/Delete) and adds SQLite-native implementations of:

Index

Constants

View Source 
const Version = "0.1.0"

Version is the dalgo2sqlite package version. Updated by hand on each release; consumed by Adapter.Version().

Variables

This section is empty.

Functions

func IsAlreadyExists added in v0.1.6 

func IsAlreadyExists(err error) bool

IsAlreadyExists reports whether err — the raw error modernc.org/sqlite (the pure-Go SQLite driver this adapter registers; see database.go) returns for a failed INSERT — represents a duplicate-key violation: a duplicate primary key or a duplicate value in a UNIQUE index.

It is dalgo2sqlite's implementation of the github.com/dal-go/dalgo2sql.DbOptions.IsAlreadyExists hook, which dalgo2sql itself cannot supply because detecting a duplicate key is driver-specific. NewDatabase and NewDatabaseWithOptions wire it in by default whenever the caller-supplied DbOptions leaves IsAlreadyExists nil (see database.go), so ordinary use needs no extra configuration. It is exported so a caller assembling a custom dalgo2sql.DbOptions can still reference it directly — e.g. to compose it with another classifier, or to confirm what it matches.

It matches only on errors.As(err, *sqlite.Error) and that error's extended result code — never on message text, which is not a stable contract across modernc.org/sqlite versions or SQLite builds. The primary result code SQLITE_CONSTRAINT (19) is deliberately not matched: it also covers NOT NULL, CHECK, and FOREIGN KEY violations, none of which are duplicate keys, so matching it would misclassify them.

func NewLenientSchemaReader added in v0.5.0 

func NewLenientSchemaReader(sqlDB *sql.DB) (dbschema.SchemaReader, error)

NewLenientSchemaReader is like NewSchemaReader but its schema reads describe the database the way SQLite lists it, with the same distinctions Database.LenientSchema makes: views are describable, a column whose declared type has no portable type is dbschema.Null instead of an error, and generated or hidden columns are left out instead of refused. It borrows sqlDB on the same terms: the caller owns the pool and its teardown.

func NewSchemaReader added in v0.5.0 

func NewSchemaReader(sqlDB *sql.DB) (dbschema.SchemaReader, error)

NewSchemaReader returns a read-only dbschema.SchemaReader that introspects the SQLite schema reachable through sqlDB — a pool the caller opened and owns. It opens no file, creates no pool, and does not ping: every SQLite query runs on the caller's pool under the caller's context, and a fault is reported by the read that hits it. The reader borrows the pool and never closes it; the caller must keep sqlDB open while the reader is in use.

The returned value is narrowed to dbschema.SchemaReader: it is not a dal.DB, exposes no DDL, row, or Close surface, so a runtime capability assertion cannot reach a write or a teardown. Its metadata is the strict kind a Database reports, including refusal (a *dbschema.NotSupportedError) of a schema a lossless export cannot carry. Use NewLenientSchemaReader for the Database.LenientSchema view.

sqlDB is typically a pool to a database this package did not open, such as an in-memory SQLite database or a remotely brokered connection.

Types

type Database 

type Database struct {
	dal.NoConcurrency // SupportsConcurrentConnections() = false

	dal.DB // delegate for the dal.DB surface
	// contains filtered or unexported fields
}

Database is the dalgo2sqlite driver instance. It implements dal.DB by embedding a dal.DB obtained from dalgo2sql.NewDatabase, and adds SQLite-specific dbschema, ddl, and concurrency surfaces.

The embedded dal.DB (rather than a named field) is what lets Database satisfy dal.DB itself: dal.DB is sealed by an unexported marker method, and embedding is the only way for that method to be promoted onto a decorating type — see dal.NewDB's doc comment.

Construct via NewDatabase. Database values are safe for concurrent use only insofar as SQLite itself is — readers can be concurrent under WAL mode; writers serialize.

func NewDatabase 

func NewDatabase(dbPath string) (*Database, error)

NewDatabase opens (or creates) the SQLite file at dbPath using modernc.org/sqlite (pure Go, CGO_ENABLED=0), pings to surface malformed-file errors at construction time, wraps the *sql.DB via dalgo2sql.NewDatabase for the dal.DB surface, and returns a *Database that satisfies dal.DB + dal.ConcurrencyAware.

Use NewDatabaseWithOptions when you need to supply per-collection primary-key metadata (required for Insert/Get/Delete with map[string]any data).

func NewDatabaseWithOptions added in v0.0.17 

func NewDatabaseWithOptions(dbPath string, schema dal.Schema, opts dalgo2sql.DbOptions) (*Database, error)

func (*Database) Adapter 

func (d *Database) Adapter() dal.Adapter

Adapter returns the driver/version identifier.

func (*Database) AlterCollection 

func (d *Database) AlterCollection(ctx context.Context, name string, ops ...ddl.AlterOp) error

AlterCollection applies ops in order inside a single transaction. Partial failures roll back and leave the collection untouched.

func (*Database) CheckSourceConstraints added in v0.3.0 

func (d *Database) CheckSourceConstraints(ctx context.Context) (resultErr error)

CheckSourceConstraints asks SQLite to verify the source's table/index/CHECK integrity and foreign keys with SQLite's own affinity and collation rules. It checks stored rows; it does not claim the export target enforces them.

func (*Database) Close 

func (d *Database) Close() error

Close closes the underlying *sql.DB. After Close the Database value is unusable; further method calls will fail with an error from database/sql.

func (*Database) CreateCollection 

func (d *Database) CreateCollection(ctx context.Context, c dbschema.CollectionDef, opts ...ddl.Option) error

CreateCollection creates a table and its inline indexes transactionally. On any error, the transaction rolls back and no schema state remains.

func (*Database) Delete 

func (d *Database) Delete(ctx context.Context, key *dalrecord.Key) error

func (*Database) DeleteMulti 

func (d *Database) DeleteMulti(ctx context.Context, keys []*dalrecord.Key) error

func (*Database) DescribeCollection 

func (d *Database) DescribeCollection(ctx context.Context, ref *dal.CollectionRef) (*dbschema.CollectionDef, error)

DescribeCollection is implemented in T15.

func (*Database) DropCollection 

func (d *Database) DropCollection(ctx context.Context, name string, opts ...ddl.Option) error

DropCollection drops the table; SQLite cascades to its indexes.

func (*Database) ExecuteQueryToRecordsReader 

func (d *Database) ExecuteQueryToRecordsReader(ctx context.Context, query dal.Query) (dal.RecordsReader, error)

func (*Database) ExecuteQueryToRecordsetReader 

func (d *Database) ExecuteQueryToRecordsetReader(ctx context.Context, query dal.Query, opts ...recordset.Option) (dal.RecordsetReader, error)

func (*Database) Exists 

func (d *Database) Exists(ctx context.Context, key *dalrecord.Key) (bool, error)

func (*Database) Get 

func (d *Database) Get(ctx context.Context, record dalrecord.Record) error

func (*Database) GetMulti 

func (d *Database) GetMulti(ctx context.Context, records []dalrecord.Record) error

func (*Database) ID 

func (d *Database) ID() string

ID returns the driver-issued database ID (delegated to dalgo2sql).

func (*Database) Insert 

func (d *Database) Insert(ctx context.Context, record dalrecord.Record, opts ...dal.InsertOption) error

func (*Database) LenientSchema added in v0.4.8 

func (d *Database) LenientSchema() *Database

LenientSchema returns a Database over the same connection whose schema reads describe a file the way SQLite itself lists it, for a reader that shows a database to a person (a scan, a viewer) rather than copies it:

  • DescribeCollection describes a view too, by the columns SQLite lists for it;
  • a column whose declared type has no portable type (none at all, or one such as JSON or UUID) is dbschema.Null instead of failing the description, and its declared type stays in SourceDefinition;
  • generated columns and the hidden columns of a virtual table are left out of the description, as PRAGMA table_info leaves them out, instead of failing it;
  • a table or a view with an empty name is left out of ListCollections and ListViews (UnnamedObjects lists them), where the strict reader refuses it;
  • an index on a column the description leaves out is left out of Indexes and ListIndexes; SourceDefinition.Indexes keeps it.

The Database it is called on is unchanged: its schema reads still refuse what a lossless export cannot represent. The two share one connection pool, so closing either closes both: close the Database that owns the pool, not the lenient one.

func (*Database) ListCollections 

func (d *Database) ListCollections(ctx context.Context, parent *record.Key) ([]dal.CollectionRef, error)

ListCollections returns the user-defined tables in alphabetical order. The parent *record.Key is ignored — SQLite has no catalog/schema hierarchy.

func (*Database) ListConstraints 

func (d *Database) ListConstraints(ctx context.Context, ref *dal.CollectionRef) ([]dbschema.ConstraintDef, error)

ListConstraints returns a best-effort survey of constraints on the table:

  • The primary-key constraint (one row if any PK columns exist)
  • Foreign-key constraints from PRAGMA foreign_key_list
  • UNIQUE constraints, each named as the index SQLite made for it (origin "u" in PRAGMA index_list): ConstraintDef holds no columns, so a constraint's columns are those of the index of its name in ListIndexes

CHECK clauses and inline NOT NULL constraints are NOT enumerated (SQLite doesn't expose CHECK source portably). Callers read those from DescribeCollection.Fields.

func (*Database) ListIndexes 

func (d *Database) ListIndexes(ctx context.Context, ref *dal.CollectionRef) ([]dbschema.IndexDef, error)

ListIndexes is implemented in T16.

func (*Database) ListReferrers 

func (d *Database) ListReferrers(ctx context.Context, ref *dal.CollectionRef) ([]dbschema.Referrer, error)

ListReferrers performs an O(N) scan: for each other user-defined table, query PRAGMA foreign_key_list and check whether any row references ref.Name.

func (*Database) ListSourceViews added in v0.3.0 

func (d *Database) ListSourceViews(ctx context.Context) ([]dbschema.SourceViewDef, error)

ListSourceViews lists view definitions once per database. View SQL is descriptive metadata; views are not materialized as collection records.

func (*Database) ListViews added in v0.4.8 

func (d *Database) ListViews(ctx context.Context) ([]dal.CollectionRef, error)

ListViews returns the views of the database in alphabetical order; a view with an empty name is treated as ListCollections treats a table with one. A view is not among the collections ListCollections returns; on a Database made by LenientSchema, DescribeCollection describes one.

func (*Database) OpenSourceRows added in v0.3.0 

func (d *Database) OpenSourceRows(ctx context.Context, ref *dal.CollectionRef) (dbschema.SourceRowCursor, error)

OpenSourceRows streams every native table row with its SQLite storage classes. A declared NUMERIC/DECIMAL value stored as REAL is rendered as the shortest decimal string that round-trips to its stored binary64 value; SQLite cannot recover decimal digits that were lost on initial insertion.

func (*Database) RunReadonlyTransaction 

func (d *Database) RunReadonlyTransaction(ctx context.Context, f dal.ROTxWorker, opts ...dal.TransactionOption) error

func (*Database) RunReadwriteTransaction 

func (d *Database) RunReadwriteTransaction(ctx context.Context, f dal.RWTxWorker, opts ...dal.TransactionOption) error

func (*Database) Schema 

func (d *Database) Schema() dal.Schema

Schema returns the dal-level Schema (delegated to dalgo2sql).

func (*Database) Set 

func (d *Database) Set(ctx context.Context, record dalrecord.Record) error

func (*Database) SetMulti 

func (d *Database) SetMulti(ctx context.Context, records []dalrecord.Record) error

func (*Database) SupportsConcurrentConnections added in v0.1.0 

func (d *Database) SupportsConcurrentConnections() bool

SupportsConcurrentConnections reports SQLite's own concurrency behaviour (always false — see dal.NoConcurrency), not dalgo2sql's. An explicit method is required here: dal.NoConcurrency and the embedded dal.DB (whose Backend requirement embeds dal.ConcurrencyAware) both declare this method at the same promotion depth, which Go otherwise treats as an ambiguous selector.

func (*Database) SupportsTransactionalDDL 

func (d *Database) SupportsTransactionalDDL() bool

SupportsTransactionalDDL reports that SQLite supports transactional DDL — every CREATE / DROP / ALTER statement can be wrapped in a BEGIN/COMMIT and is rolled back atomically on commit failure.

func (*Database) UnnamedObjects added in v0.4.8 

func (d *Database) UnnamedObjects(ctx context.Context) ([]string, error)

UnnamedObjects lists the kind ("table" or "view") of each table and view of the database whose name is empty, which SQLite allows and no reference can hold: the lenient reader leaves them out of its listings, and its caller can say so.

func (*Database) Update 

func (d *Database) Update(ctx context.Context, key *dalrecord.Key, updates []update.Update, preconditions ...dal.Precondition) error

func (*Database) UpdateMulti 

func (d *Database) UpdateMulti(ctx context.Context, keys []*dalrecord.Key, updates []update.Update, preconditions ...dal.Precondition) error

func (*Database) UpdateRecord 

func (d *Database) UpdateRecord(ctx context.Context, record dalrecord.Record, updates []update.Update, preconditions ...dal.Precondition) error

UpdateRecord is not supported at the database level by dalgo2sql; use Update with an explicit key instead, or call UpdateRecord inside a RunReadwriteTransaction where the transaction object does support it.

func (*Database) Upsert 

func (d *Database) Upsert(ctx context.Context, record dalrecord.Record) error

type TableReadFacts added in v0.6.0 

type TableReadFacts struct {
	Filename, Namespace, Table string
	Columns                    []string
	MaxBytes, MaxColumns       uint32
	// MaxCellBytes is verified on the same native Conn after column discovery.
	MaxCellBytes uint32
	// Encoding is exactly UTF-8 in the only admitted first profile.
	Encoding string
}

TableReadFacts describes one pinned main-table read. These are native naming and allocation facts, never permissions or proof of an authenticated source. Filename is SQLite's logical opened-file path; it is not OS inode/replacement attestation. A trusted host must own/fix its private file route and bind it to an independently authenticated source identity before using these facts.

type TableReadLimits added in v0.6.0 

type TableReadLimits struct{ MaxBytes, MaxColumns uint32 }

TableReadLimits configures a derived per-cell limit and column ceiling before data SQL. A combined accepted-value bound of MaxBytes applies only when the host enforces a frozen query returning exactly the selected table's original N columns. Session is a generic read surface, not a query-shape or permission enforcer; arbitrary projections/expressions do not inherit that combined-value bound. The derived per-cell bound can refuse cells below MaxBytes; no truncation or guarantee of accepting every smaller row is implied. Encoded-row/driver/Go overhead is excluded. The host must also bound row count and serialized output. MaxBytes must be 1..256KiB and MaxColumns 1..128.

type TableReadSpan added in v0.6.0 

type TableReadSpan struct {
	// contains filtered or unexported fields
}

TableReadSpan owns a private mode=ro connection and rollback-only transaction. Its immutable session/compiler and facts remain on that handle until Close. Ingress cancellation invalidates the transaction; callers must still Close. It exposes no native/write/commit handle. Permission evaluation belongs to the host's secured DALgo session, not this provider.

func OpenTableRead added in v0.6.0 

func OpenTableRead(ctx context.Context, filename, namespace, table string, limits TableReadLimits) (*TableReadSpan, error)

OpenTableRead opens an existing absolute file through a provider-built mode=ro URI. Only a UTF-8 file, explicit main namespace and exact simple ASCII ordinary table are supported; views, virtual/shadow/generated tables, attachments, implicit namespaces and case/quote/dot rewrites are refused. It does not authorize the table or authenticate filename. See TableReadFacts for the host's source duty.

All metadata, query_only and native limits are established before data SQL. Facts is SQL-free thereafter. Close invalidates/cancels first, then rolls back and closes native storage; it never commits. The first Close caller owns any cleanup errors it observes (database/sql can consume auto-rollback errors). Concurrent/repeated Close calls return nil without waiting for that caller's native I/O. Hosts must report cleanup errors in their lifecycle channel, not append them to a completed response.

func (*TableReadSpan) Close added in v0.6.0 

func (s *TableReadSpan) Close() error

Close invalidates before native cleanup. No ownership mutex spans native I/O.

func (*TableReadSpan) Facts added in v0.6.0 

func (s *TableReadSpan) Facts(ctx context.Context) (TableReadFacts, error)

Facts returns a defensive metadata copy and SQL-free liveness check. It does not acquire a lock used by a reader or native Close, and never queries SQLite.

func (*TableReadSpan) Session added in v0.6.0 

func (s *TableReadSpan) Session() dal.ReadSession

Session returns the same immutable DALgo read surface. It is nil after invalidation. The host must wrap it with its own central access policy.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.