sqlite3

go-sqlite3

Sponsors

This project is proudly sponsored by:

Latest stable version is v1.14 or later, not v2.

Description

A sqlite3 driver that conforms to the built-in database/sql interface.

Supported Golang version: See .github/workflows/go.yaml.

0xCarbon fork: SQLCipher

This fork of mattn/go-sqlite3 bundles the SQLCipher amalgamation instead of plain SQLite, so every build provides at-rest encryption through OpenSSL (libcrypto). It tracks upstream master and is consumed directly under its own module path — import and require github.com/0xCarbon/go-sqlite3; no replace directive is needed:

import _ "github.com/0xCarbon/go-sqlite3"

require github.com/0xCarbon/go-sqlite3 <version>

Keys are configured per driver or per DSN, in this precedence order:

• SQLiteDriver.EncryptionKeyBytes []byte — raw key; the driver hex-encodes it into a zeroized mutable PRAGMA key = "x'...'" buffer (key derivation is skipped for raw keys). SQLCipher only honors the raw form for exactly 32 bytes, or 48/80 bytes with salt; other lengths are rejected.
• SQLiteDriver.EncryptionKey string — interpolated into PRAGMA key as-is; pass a SQL-literal value such as "x'0123...'" or a passphrase literal.
• _key=<hex> DSN parameter — raw key for DSNs opened through the globally registered driver; invalid or empty values fail the open.

The key runs as the first statement after sqlite3_open_v2, before all other pragmas. Note that _key DSN values and EncryptionKey strings are immutable Go strings held by the caller and cannot be scrubbed from process memory; prefer EncryptionKeyBytes for sensitive keys (the driver wipes its own buffers, including the C copies). Keyed opens verify PRAGMA cipher_version and fail loudly when the linked build has no codec (for example -tags libsqlite3 against a plain system libsqlite3) instead of silently operating unencrypted.

To upgrade the bundled amalgamation, run upgrade/sqlcipher.sh (see upgrade/check.sh for the current vs. latest SQLCipher version). The libsqlite3 build tag links the system library instead of the amalgamation; encryption then requires a SQLCipher build of that system library.

The canonical branch for fork work is master; feat/sqlcipher-encryption-key is frozen history that master already contains.

Opening encrypted databases: rules and caveats

• Key before open, not after. The key must be known at open time (EncryptionKeyBytes/EncryptionKey/_key). Opening an existing encrypted database and then issuing PRAGMA key yourself is unsupported: the driver applies setup pragmas at open (notably PRAGMA synchronous, which reads page 1) and SQLCipher fails the open with SQLITE_NOTADB before your statement runs. The same mechanism is what makes a wrong driver-level key fail fast at Ping instead of poisoning the pool with unusable connections.
• cache=shared keys once per pager. In shared-cache mode the codec belongs to the shared pager: every connection in the pool must use the same key, and wrong-key detection covers only the first opener. Later connections with a different key silently join under the first key.
• ATTACH ... KEY needs the quoted raw-key string. SQLCipher only treats x'<hex>' as a raw key when it arrives as text: use ATTACH DATABASE ? AS x KEY ? binding "x'<hex>'" (as sqlcipher_export flows in the wild do). A bare blob literal KEY x'<hex>' silently runs key derivation over the literal bytes.
• Passphrases must be quoted SQL literals. EncryptionKey is interpolated as-is into PRAGMA key = <value>;: pass "'my pass phrase '"-style quoted values, and never user input (values containing ; would execute as multiple statements).
• Serialize() returns plaintext. On a keyed connection Serialize() returns the fully decrypted database image (and Deserialize() writes to the in-memory image, not the encrypted file). Handle serialized bytes as plaintext.
• Never set PRAGMA temp_store=FILE on keyed databases. SQLCipher does not encrypt sorter/temp spill files; the build default (SQLITE_TEMP_STORE=2, memory) keeps them off disk at the cost of the RSS ceiling below.

Performance characteristics

Measured on Linux/amd64 (see sqlite3_encryption_bench_test.go):

Operation	Plain	Keyed (32-byte raw key)
Open+Close	~49µs	~83µs (1.7×)
Open+Close, WAL + busy_timeout	~128µs	~188µs
Writes (WAL, 256B rows)	1×	+56%
Reads, warm page cache	1×	parity
Reads, cold page cache	1×	+158%
Statement-cache hit (warm)	1×	parity

• Passphrase keys cost ~0.2–0.4 s per open: SQLCipher runs PBKDF2-HMAC-SHA512 with 256,000 iterations on every connection. Use raw keys (EncryptionKeyBytes/_key) under load; the driver rejects raw-key lengths other than 32/48/80 bytes precisely because anything else would silently take this slow, derived path.
• SQLITE_TEMP_STORE=2 trade-off: large sorts and index builds stay in memory instead of spilling to (plaintext) temp files — size memory-limited containers for worst-case sort volumes.
• Every PRAGMA pays a small fixed overhead (~0.2µs) from SQLCipher's pragma hook; irrelevant except in pragma-heavy micro-loops.

This package follows the official Golang Release Policy.

Overview

• go-sqlite3
• Description
  • Overview
• Installation
• API Reference
• Connection String
  • DSN Examples
• Features
  • Usage
  • Feature / Extension List
• Compilation
  • Android
• ARM
• Cross Compile
• Compiling
  • Linux
    • Alpine
    • Fedora
    • Ubuntu
  • macOS
  • Windows
  • Errors
• User Authentication
  • Compile
  • Usage
    • Create protected database
    • Password Encoding
      • Available Encoders
    • Restrictions
    • Support
    • User Management
      • SQL
        • Examples
      • *SQLiteConn
    • Attached database
• Extensions
  • Spatialite
• FAQ
• License
• Author

Installation

This package can be installed with the go get command:

go get github.com/mattn/go-sqlite3

go-sqlite3 is cgo package. If you want to build your app using go-sqlite3, you need gcc.

Important: because this is a CGO enabled package, you are required to set the environment variable CGO_ENABLED=1 and have a gcc compiler present within your path.

API Reference

API documentation can be found here.

Examples can be found under the examples directory.

Connection String

When creating a new SQLite database or connection to an existing one, with the file name additional options can be given. This is also known as a DSN (Data Source Name) string.

Options are append after the filename of the SQLite database. The database filename and options are separated by an ? (Question Mark). Options should be URL-encoded (see url.QueryEscape).

This also applies when using an in-memory database instead of a file.

Options can be given using the following format: KEYWORD=VALUE and multiple options can be combined with the & ampersand.

This library supports DSN options of SQLite itself and provides additional options.

Boolean values can be one of:

• 0 no false off
• 1 yes true on

Name	Key	Value(s)	Description
UA - Create	_auth	-	Create User Authentication, for more information see User Authentication
UA - Username	_auth_user	string	Username for User Authentication, for more information see User Authentication
UA - Password	_auth_pass	string	Password for User Authentication, for more information see User Authentication
UA - Crypt	_auth_crypt	SHA1 SSHA1 SHA256 SSHA256 SHA384 SSHA384 SHA512 SSHA512	Password encoder to use for User Authentication, for more information see User Authentication
UA - Salt	_auth_salt	string	Salt to use if the configure password encoder requires a salt, for User Authentication, for more information see User Authentication
Auto Vacuum	_auto_vacuum | _vacuum	0 | none 1 | full 2 | incremental	For more information see PRAGMA auto_vacuum
Busy Timeout	_busy_timeout | _timeout	int	Specify value for sqlite3_busy_timeout. For more information see PRAGMA busy_timeout
Case Sensitive LIKE	_case_sensitive_like | _cslike	boolean	For more information see PRAGMA case_sensitive_like
Defer Foreign Keys	_defer_foreign_keys | _defer_fk	boolean	For more information see PRAGMA defer_foreign_keys
Encryption Key	_key	hex string	0xCarbon fork: raw SQLCipher key, hex-encoded; must decode to 32, 48 or 80 bytes (PRAGMA key = "x'...'" form, key derivation skipped). Lower precedence than the EncryptionKeyBytes/EncryptionKey driver fields. Invalid, empty or duplicate values fail the open. Keys embedded in the DSN cannot be scrubbed from memory; prefer EncryptionKeyBytes for sensitive keys.
Foreign Keys	_foreign_keys | _fk	boolean	For more information see PRAGMA foreign_keys
Ignore CHECK Constraints	_ignore_check_constraints	boolean	For more information see PRAGMA ignore_check_constraints
Immutable	immutable	boolean	For more information see Immutable
Journal Mode	_journal_mode | _journal	DELETE TRUNCATE PERSIST MEMORY WAL OFF	For more information see PRAGMA journal_mode
Locking Mode	_locking_mode | _locking	NORMAL EXCLUSIVE	For more information see PRAGMA locking_mode
Mode	mode	ro rw rwc memory	Access Mode of the database. For more information see SQLite Open
Mutex Locking	_mutex	no full	Specify mutex mode.
Query Only	_query_only	boolean	For more information see PRAGMA query_only
Recursive Triggers	_recursive_triggers | _rt	boolean	For more information see PRAGMA recursive_triggers
Secure Delete	_secure_delete	boolean | FAST	For more information see PRAGMA secure_delete
Shared-Cache Mode	cache	shared private	Set cache mode for more information see sqlite.org
Synchronous	_synchronous | _sync	0 | OFF 1 | NORMAL 2 | FULL 3 | EXTRA	For more information see PRAGMA synchronous
Time Zone Location	_loc	auto	Specify location of time format.
Transaction Lock	_txlock	immediate deferred exclusive	Specify locking behavior for transactions.
Writable Schema	_writable_schema	Boolean	When this pragma is on, the SQLITE_MASTER tables in which database can be changed using ordinary UPDATE, INSERT, and DELETE statements. Warning: misuse of this pragma can easily result in a corrupt database file.
Cache Size	_cache_size	int	Maximum cache size; default is 2000K (2M). See PRAGMA cache_size
Statement Cache Size	_stmt_cache_size	int	Maximum number of prepared statements cached per connection; default is 0 (disabled). Note that sql.DB is a connection pool, so each connection maintains its own independent cache.

DSN Examples

file:test.db?cache=shared&mode=memory

Features

This package allows additional configuration of features available within SQLite3 to be enabled or disabled by golang build constraints also known as build tags.

Click here for more information about build tags / constraints.

Usage

If you wish to build this library with additional extensions / features, use the following command:

go build -tags "<FEATURE>"

For available features, see the extension list. When using multiple build tags, all the different tags should be space delimited.

Example:

go build -tags "icu json1 fts5 secure_delete"

Feature / Extension List

Extension	Build Tag	Description
Additional Statistics	sqlite_stat4	This option adds additional logic to the ANALYZE command and to the query planner that can help SQLite to chose a better query plan under certain situations. The ANALYZE command is enhanced to collect histogram data from all columns of every index and store that data in the sqlite_stat4 table. The query planner will then use the histogram data to help it make better index choices. The downside of this compile-time option is that it violates the query planner stability guarantee making it more difficult to ensure consistent performance in mass-produced applications. SQLITE_ENABLE_STAT4 is an enhancement of SQLITE_ENABLE_STAT3. STAT3 only recorded histogram data for the left-most column of each index whereas the STAT4 enhancement records histogram data from all columns of each index. The SQLITE_ENABLE_STAT3 compile-time option is a no-op and is ignored if the SQLITE_ENABLE_STAT4 compile-time option is used
Allow URI Authority	sqlite_allow_uri_authority	URI filenames normally throws an error if the authority section is not either empty or "localhost". However, if SQLite is compiled with the SQLITE_ALLOW_URI_AUTHORITY compile-time option, then the URI is converted into a Uniform Naming Convention (UNC) filename and passed down to the underlying operating system that way
App Armor	sqlite_app_armor	When defined, this C-preprocessor macro activates extra code that attempts to detect misuse of the SQLite API, such as passing in NULL pointers to required parameters or using objects after they have been destroyed. App Armor is not available under Windows.
Disable Load Extensions	sqlite_omit_load_extension	Loading of external extensions is enabled by default. To disable extension loading add the build tag sqlite_omit_load_extension.
Enable Serialization with libsqlite3	sqlite_serialize	Serialization and deserialization of a SQLite database is available by default, unless the build tag libsqlite3 is set. To enable this functionality even if libsqlite3 is set, add the build tag sqlite_serialize.
Foreign Keys	sqlite_foreign_keys	This macro determines whether enforcement of foreign key constraints is enabled or disabled by default for new database connections. Each database connection can always turn enforcement of foreign key constraints on and off and run-time using the foreign_keys pragma. Enforcement of foreign key constraints is normally off by default, but if this compile-time parameter is set to 1, enforcement of foreign key constraints will be on by default
Full Auto Vacuum	sqlite_vacuum_full	Set the default auto vacuum to full
Incremental Auto Vacuum	sqlite_vacuum_incr	Set the default auto vacuum to incremental
Full Text Search Engine	sqlite_fts5	When this option is defined in the amalgamation, versions 5 of the full-text search engine (fts5) is added to the build automatically
International Components for Unicode	sqlite_icu	This option causes the International Components for Unicode or "ICU" extension to SQLite to be added to the build
Introspect PRAGMAS	sqlite_introspect	This option adds some extra PRAGMA statements. PRAGMA function_list PRAGMA module_list PRAGMA pragma_list
JSON SQL Functions	sqlite_json	When this option is defined in the amalgamation, the JSON SQL functions are added to the build automatically
Math Functions	sqlite_math_functions	This compile-time option enables built-in scalar math functions. For more information see Built-In Mathematical SQL Functions
OS Trace	sqlite_os_trace	This option enables OSTRACE() debug logging. This can be verbose and should not be used in production.
Percentile	sqlite_percentile	This option enables The Percentile Extension.
Pre Update Hook	sqlite_preupdate_hook	Registers a callback function that is invoked prior to each INSERT, UPDATE, and DELETE operation on a database table.
Secure Delete	sqlite_secure_delete	This compile-time option changes the default setting of the secure_delete pragma. When this option is not used, secure_delete defaults to off. When this option is present, secure_delete defaults to on. The secure_delete setting causes deleted content to be overwritten with zeros. There is a small performance penalty since additional I/O must occur. On the other hand, secure_delete can prevent fragments of sensitive information from lingering in unused parts of the database file after it has been deleted. See the documentation on the secure_delete pragma for additional information
Secure Delete (FAST)	sqlite_secure_delete_fast	For more information see PRAGMA secure_delete
Tracing / Debug	sqlite_trace	Activate trace functions
User Authentication	sqlite_userauth	SQLite User Authentication see User Authentication for more information.
Virtual Tables	sqlite_vtable	SQLite Virtual Tables see SQLite Official VTABLE Documentation for more information, and a full example here
The DBSTAT Virtual Table	sqlite_dbstat	The DBSTAT virtual table is a read-only virtual table that returns information about the amount of disk space used to store the content of an SQLite database. See SQLite Official Documentation for more information.

Compilation

This package requires the CGO_ENABLED=1 environment variable if not set by default, and the presence of the gcc compiler.

If you need to add additional CFLAGS or LDFLAGS to the build command, and do not want to modify this package, then this can be achieved by using the CGO_CFLAGS and CGO_LDFLAGS environment variables.

Android

This package can be compiled for android. Compile with:

go build -tags "android"

For more information see #201

ARM

To compile for ARM use the following environment:

env CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ \
    CGO_ENABLED=1 GOOS=linux GOARCH=arm GOARM=7 \
    go build -v 

Additional information:

• #242
• #504

Cross Compile

This library can be cross-compiled.

In some cases you are required to the CC environment variable with the cross compiler.

Cross Compiling from macOS

The simplest way to cross compile from macOS is to use xgo.

Steps:

• Install musl-cross (brew install FiloSottile/musl-cross/musl-cross).
• Run CC=x86_64-linux-musl-gcc CXX=x86_64-linux-musl-g++ GOARCH=amd64 GOOS=linux CGO_ENABLED=1 go build -ldflags "-linkmode external -extldflags -static".

Please refer to the project's README for further information.

Compiling

Linux

To compile this package on Linux, you must install the development tools for your linux distribution.

To compile under linux use the build tag linux.

go build -tags "linux"

If you wish to link directly to libsqlite3 then you can use the libsqlite3 build tag.

go build -tags "libsqlite3 linux"

Alpine

When building in an alpine container run the following command before building:

apk add --update gcc musl-dev

Fedora

sudo yum groupinstall "Development Tools" "Development Libraries"

Ubuntu

sudo apt-get install build-essential

macOS

macOS should have all the tools present to compile this package. If not, install XCode to add all the developers tools.

Required dependency:

brew install sqlite3

For macOS, there is an additional package to install which is required if you wish to build the icu extension.

This additional package can be installed with homebrew:

brew upgrade icu4c

To compile for macOS on x86:

go build -tags "darwin amd64"

To compile for macOS on ARM chips:

go build -tags "darwin arm64"

If you wish to link directly to libsqlite3, use the libsqlite3 build tag:

# x86 
go build -tags "libsqlite3 darwin amd64"
# ARM
go build -tags "libsqlite3 darwin arm64"

Additional information:

• #206
• #404

Windows

To compile this package on Windows, you must have the gcc compiler installed.

1. Install a Windows gcc toolchain.
2. Add the bin folder to the Windows path, if the installer did not do this by default.
3. Open a terminal for the TDM-GCC toolchain, which can be found in the Windows Start menu.
4. Navigate to your project folder and run the go build ... command for this package.

For example the TDM-GCC Toolchain can be found here.

Errors

• Compile error: can not be used when making a shared object; recompile with -fPIC

  When receiving a compile time error referencing recompile with -FPIC then you are probably using a hardend system.

  You can compile the library on a hardend system with the following command.

  go build -ldflags '-extldflags=-fno-PIC'
  

  More details see #120

• Can't build go-sqlite3 on windows 64bit.

  Probably, you are using go 1.0, go1.0 has a problem when it comes to compiling/linking on windows 64bit. See: #27

• go get github.com/mattn/go-sqlite3 throws compilation error.

  gcc throws: internal compiler error

  Remove the download repository from your disk and try re-install with:

  go install github.com/mattn/go-sqlite3
  

User Authentication

This is deprecated

This package supports the SQLite User Authentication module.

Compile

To use the User authentication module, the package has to be compiled with the tag sqlite_userauth. See Features.

Usage

Create protected database

To create a database protected by user authentication, provide the following argument to the connection string _auth. This will enable user authentication within the database. This option however requires two additional arguments:

• _auth_user
• _auth_pass

When _auth is present in the connection string user authentication will be enabled and the provided user will be created as an admin user. After initial creation, the parameter _auth has no effect anymore and can be omitted from the connection string.

Example connection strings:

Create an user authentication database with user admin and password admin:

file:test.s3db?_auth&_auth_user=admin&_auth_pass=admin

Create an user authentication database with user admin and password admin and use SHA1 for the password encoding:

file:test.s3db?_auth&_auth_user=admin&_auth_pass=admin&_auth_crypt=sha1

Password Encoding

The passwords within the user authentication module of SQLite are encoded with the SQLite function sqlite_cryp. This function uses a ceasar-cypher which is quite insecure. This library provides several additional password encoders which can be configured through the connection string.

The password cypher can be configured with the key _auth_crypt. And if the configured password encoder also requires an salt this can be configured with _auth_salt.

Available Encoders

• SHA1
• SSHA1 (Salted SHA1)
• SHA256
• SSHA256 (salted SHA256)
• SHA384
• SSHA384 (salted SHA384)
• SHA512
• SSHA512 (salted SHA512)

Restrictions

Operations on the database regarding user management can only be preformed by an administrator user.

Support

The user authentication supports two kinds of users:

• administrators
• regular users

User Management

User management can be done by directly using the *SQLiteConn or by SQL.

SQL

The following sql functions are available for user management:

Function	Arguments	Description
authenticate	username string, password string	Will authenticate an user, this is done by the connection; and should not be used manually.
auth_user_add	username string, password string, admin int	This function will add an user to the database. if the database is not protected by user authentication it will enable it. Argument admin is an integer identifying if the added user should be an administrator. Only Administrators can add administrators.
auth_user_change	username string, password string, admin int	Function to modify an user. Users can change their own password, but only an administrator can change the administrator flag.
authUserDelete	username string	Delete an user from the database. Can only be used by an administrator. The current logged in administrator cannot be deleted. This is to make sure their is always an administrator remaining.

These functions will return an integer:

• 0 (SQLITE_OK)
• 23 (SQLITE_AUTH) Failed to perform due to authentication or insufficient privileges

// Autheticate user
// Create Admin User
SELECT auth_user_add('admin2', 'admin2', 1);

// Change password for user
SELECT auth_user_change('user', 'userpassword', 0);

// Delete user
SELECT user_delete('user');

*SQLiteConn

The following functions are available for User authentication from the *SQLiteConn:

Function	Description
Authenticate(username, password string) error	Authenticate user
AuthUserAdd(username, password string, admin bool) error	Add user
AuthUserChange(username, password string, admin bool) error	Modify user
AuthUserDelete(username string) error	Delete user

Attached database

When using attached databases, SQLite will use the authentication from the main database for the attached database(s).

Extensions

If you want your own extension to be listed here, or you want to add a reference to an extension; please submit an Issue for this.

Spatialite

Spatialite is available as an extension to SQLite, and can be used in combination with this repository. For an example, see shaxbee/go-spatialite.

extension-functions.c from SQLite3 Contrib

extension-functions.c is available as an extension to SQLite, and provides the following functions:

• Math: acos, asin, atan, atn2, atan2, acosh, asinh, atanh, difference, degrees, radians, cos, sin, tan, cot, cosh, sinh, tanh, coth, exp, log, log10, power, sign, sqrt, square, ceil, floor, pi.
• String: replicate, charindex, leftstr, rightstr, ltrim, rtrim, trim, replace, reverse, proper, padl, padr, padc, strfilter.
• Aggregate: stdev, variance, mode, median, lower_quartile, upper_quartile

For an example, see dinedal/go-sqlite3-extension-functions.

FAQ

• Getting insert error while query is opened.

  You can pass some arguments into the connection string, for example, a URI. See: #39

• Do you want to cross compile? mingw on Linux or Mac?

  See: #106 See also: http://www.limitlessfx.com/cross-compile-golang-app-for-windows-from-linux.html

• Want to get time.Time with current locale

  Use _loc=auto in SQLite3 filename schema like file:foo.db?_loc=auto.

• Can I use this in multiple routines concurrently?

  Yes for readonly. But not for writable. See #50, #51, #209, #274.

• Why I'm getting no such table error?

  Why is it racy if I use a sql.Open("sqlite3", ":memory:") database?

  Each connection to ":memory:" opens a brand new in-memory sql database, so if the stdlib's sql engine happens to open another connection and you've only specified ":memory:", that connection will see a brand new database. A workaround is to use "file::memory:?cache=shared" (or "file:foobar?mode=memory&cache=shared"). Every connection to this string will point to the same in-memory database.

  Note that if the last database connection in the pool closes, the in-memory database is deleted. Make sure the max idle connection limit is > 0, and the connection lifetime is infinite.

  For more information see:

  • #204
  • #511
  • https://www.sqlite.org/sharedcache.html#shared_cache_and_in_memory_databases
  • https://www.sqlite.org/inmemorydb.html#sharedmemdb

• Reading from database with large amount of goroutines fails on OSX.

  OS X limits OS-wide to not have more than 1000 files open simultaneously by default.

  For more information, see #289

• Trying to execute a . (dot) command throws an error.

  Error: Error: near ".": syntax error Dot command are part of SQLite3 CLI, not of this library.

  You need to implement the feature or call the sqlite3 cli.

  More information see #305.

• Error: database is locked

  When you get a database is locked, please use the following options.

  Add to DSN: cache=shared

  Example:

  db, err := sql.Open("sqlite3", "file:locked.sqlite?cache=shared")
  

  Next, please set the database connections of the SQL package to 1:

  db.SetMaxOpenConns(1)
  

  For more information, see #209.

Contributors

Code Contributors

This project exists thanks to all the people who [contribute].

Financial Contributors

Become a financial contributor and help us sustain our community. [Contribute here].

Individuals

Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]

License

MIT: http://mattn.mit-license.org/2018

sqlite3-binding.c, sqlite3-binding.h, sqlite3ext.h

The -binding suffix was added to avoid build failures under gccgo.

In this repository, those files are an amalgamation of code that was copied from SQLite3. The license of that code is the same as the license of SQLite3.

Author

Yasuhiro Matsumoto (a.k.a mattn)

G.J.R. Timmer