sqlite

package module
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2026 License: BSD-3-Clause Imports: 21 Imported by: 0

README

sqlite

Hanzo SQLite

Dual-backend SQLite driver for the Hanzo ecosystem. Registers the database/sql driver name sqlite under both build configurations and exposes the same API either way:

Build Backend Encryption Use
CGO_ENABLED=1 + -tags libsqlite3 + libsqlcipher mattn/go-sqlite3 → SQLCipher AES-256 page-level, at rest production
CGO_ENABLED=0 modernc.org/sqlite (pure Go) none CI tests / lint / local dev

One import, one driver name, two backends:

import _ "github.com/hanzoai/sqlite" // registers "sqlite" under both tags

db, _ := sql.Open("sqlite", dsn)     // mattn+SQLCipher (cgo) or modernc (!cgo)

The pure-Go backend cannot encrypt. Demanding a key on it (Open(path, WithKey(...)), OpenDB(path, key)) returns ErrEncryptionUnavailable and writes nothing — it never silently persists plaintext.

Building the encrypted (production) backend — READ THIS

mainline mattn/go-sqlite3 has no sqlcipher build tag and no sqlite3_key() binding. SQLCipher works only when you:

  1. link the system sqlite (the libsqlite3 tag) against libsqlcipher, and
  2. enable the codec + URI keying via CGO flags, and
  3. supply the key as SQLCipher's native URI key parameter so it is applied inside sqlite3_open_v2 — before mattn's pragma battery runs.
CGO_ENABLED=1 \
CGO_CFLAGS="-DSQLITE_HAS_CODEC -DSQLITE_USE_URI=1 -I<sqlcipher>/include/sqlcipher" \
CGO_LDFLAGS="-L<sqlcipher>/lib -lsqlcipher" \
go build -tags "libsqlite3 sqlite_fts5" ./...

Alpine: apk add gcc musl-dev sqlcipher-dev pkgconfig.

Why not -tags sqlcipher + PRAGMA key in a ConnectHook?

Both are traps that ship plaintext:

  • -tags sqlcipher is inert in mainline mattn (no such tag) → links plain sqlite → PRAGMA key is a silent no-op.
  • mattn runs PRAGMA busy_timeout/journal_mode/foreign_keys/... via sqlite3_exec before the ConnectHook fires. On an existing encrypted file that touches the header before the key is set → file is not a database on reopen. So a ConnectHook can create but never reopen a SQLCipher DB.

The URI key parameter sidesteps both: SQLCipher's VFS keys the connection at open time. TestEncryptionProof asserts real ciphertext on disk and a working keyed reopen, so a mis-linked build fails CI instead of shipping plaintext.

The key rides the DSN (file:PATH?...&key=x'HEX'). Never log the DSN. IAM keeps showSql=false and does not log it.

Encryption

  • Algorithm: AES-256 (SQLCipher 4 defaults: 4096-byte pages, PBKDF2-HMAC-SHA512, 256000 iters, per-page HMAC-SHA512).
  • Key: raw 256-bit (no passphrase KDF) via WithRawKey / the key=x'HEX' DSN param, or derived per principal:
// CEK = HKDF-SHA256(masterKey, "{org|user}:{id}")
db, _ := sqlite.Open("data.db", sqlite.WithPrincipalKey(masterKey, sqlite.PrincipalOrg, "acme"))

// Or get a *sql.DB to hand to xorm.NewEngineWithDB:
cek, _ := sqlite.DeriveKey(masterKey, sqlite.PrincipalOrg, "acme")
sqldb, _ := sqlite.OpenDB(dbPath, cek)
eng, _ := xorm.NewEngineWithDB("sqlite", "", core.FromDB(sqldb))

Different orgs/users get different CEKs (domain-separated info); destroying the master key renders every derived CEK irrecoverable.

Per-principal CEK

DeriveKey(masterKey, principalType, principalID) → 32-byte CEK via HKDF-SHA256. Master key is 32 bytes, sourced from KMS. Used for per-org and per-user database isolation in IAM, KMS, and other Hanzo services.

Threshold write attestation

ThresholdManager coordinates t-of-n Ed25519 attestations for writes (MPC shard storage, multi-sig). Pure Go; available under both backends.

License

Apache-2.0

Documentation

Overview

Package sqlite provides an at-rest-encrypted SQLite driver for Hanzo.

ONE WAY, ALWAYS ENCRYPTED. A keyed store is encrypted on EVERY build — there is no build tag, environment switch, or missing C library under which it falls back to plaintext. EncryptionAvailable() is always true. The SQLCipher 4 page format is the single at-rest format, produced two byte-compatible ways:

  • The pure-Go hanzoai/sqlcipher codec (the default, works everywhere): a keyed open decrypts the file to a RAM-backed plaintext copy, the engine reads and writes that copy, and Close/Checkpoint re-encrypts it to the real path (envelope.go). This needs no C toolchain and no libsqlcipher, so native Go (CGO_ENABLED=0) ALWAYS encrypts.
  • The live libsqlcipher codec (an optional acceleration on cgo builds that link it): page-level AES-256 keyed inside sqlite3_open_v2, with per-commit durability. A cgo build links it with: CGO_ENABLED=1 \ CGO_CFLAGS="-DSQLITE_HAS_CODEC -DSQLITE_USE_URI=1 -I<sqlcipher>/include/sqlcipher" \ CGO_LDFLAGS="-lsqlcipher" \ go build -tags "libsqlite3 sqlite_fts5"

On a cgo build openDB uses the live codec when a runtime probe proves it encrypts (CodecLinked), and otherwise FALLS BACK to the pure-Go codec envelope — so a mis-linked libsqlcipher degrades to the pure-Go codec, never to plaintext. On the pure-Go build a key always routes to the envelope. Either way the file on disk is SQLCipher ciphertext, readable by the C library and by this package alike.

The database/sql driver name "sqlite" is registered under both build tags (hanzoai/csqlite on cgo, modernc.org/sqlite on pure-Go), so any code doing `_ "github.com/hanzoai/sqlite"` + `sql.Open("sqlite", dsn)` compiles and runs on both — one import, one driver name, one encryption format.

OPT-OUT BUILD TAG `sqlite_purego`: forces the pure-Go (modernc) backend even when CGO_ENABLED=1. It exists for one reason: a binary that links this fork AND another package importing modernc.org/sqlite directly would, under a plain CGO_ENABLED=1 build, register the "sqlite" driver TWICE and panic at init. Such a service builds with `-tags sqlite_purego` so the whole binary registers "sqlite" exactly once. It does NOT change the encryption guarantee: the pure-Go backend also always encrypts a keyed store, via the same codec.

The encryption key, replication mode, threshold config, and per-principal CEK derivation (cek.go, threshold.go) and the codec envelope (envelope.go) are pure Go and tag-neutral; only the driver registration and the DSN pragma syntax differ by build tag (driver_cgo.go / driver_nocgo.go).

Threshold write attestation for multi-party SQLite.

Each party runs a node with a full replica. Write operations require t-of-n parties to sign the write before it's committed. Reads are local.

Use case: MPC wallet shard storage, DEX trade approvals, multi-sig transaction authorization in trading platforms.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrThresholdNotMet = errors.New("sqlite: threshold attestations not met")
	ErrDuplicateSigner = errors.New("sqlite: duplicate signer")
	ErrInvalidSig      = errors.New("sqlite: invalid attestation signature")
	ErrSessionExpired  = errors.New("sqlite: attestation session expired")
)
View Source 
var DefaultPragmas = []Pragma{
	{Name: "busy_timeout", Value: "10000"},
	{Name: "journal_mode", Value: "WAL"},
	{Name: "journal_size_limit", Value: "200000000"},
	{Name: "synchronous", Value: "NORMAL"},
	{Name: "foreign_keys", Value: "ON"},
	{Name: "temp_store", Value: "MEMORY"},
	{Name: "cache_size", Value: "-32000"},
}

DefaultPragmas is the canonical Hanzo embedded-SQLite connection tuning. Order matters: busy_timeout MUST lead so a connection blocks on a busy database before journal_mode=WAL is set (WAL cannot be enabled while another connection holds the database). This is the single source of truth for the unencrypted default-connect DSN used across Hanzo Base (core + multitenant store) — treat it as read-only.

Functions

func Checkpoint added in v0.4.0 

func Checkpoint(db *sql.DB) error

Checkpoint re-encrypts an open envelope-backed database to its encrypted file WITHOUT closing it, so the ciphertext on disk reflects committed writes up to this point. It bounds the crash-loss window between the coarse encrypt-on-close events. On a database that is not envelope-backed (the live-libsqlcipher or unencrypted paths, which persist per-commit) it is a successful no-op.

func CodecLinked added in v0.1.3 

func CodecLinked() bool

CodecLinked reports whether the SQLCipher C codec is linked and ACTUALLY encrypting, proven by a cached one-time runtime probe — NOT a compile-time constant.

The distinction is the whole point. A cgo build only has a live C codec if the SQLite it links is SQLCipher. A cgo build that links plain sqlite (the vendored amalgamation, or libsqlcipher absent) SILENTLY no-ops PRAGMA key and writes PLAINTEXT. A compile-time `return true` could not tell those apart, so it once reported the codec linked for a build that ships plaintext.

This probes the linked engine at runtime (codecEncrypts): open a throwaway keyed database, write a sentinel, and confirm the bytes on disk are real ciphertext that round-trips under the key. The result is cached. When it is true, openDB uses the live libsqlcipher codec (per-commit durability); when it is FALSE, openDB falls back to the pure-Go SQLCipher codec envelope (envelope.go), which encrypts without libsqlcipher — so a keyed store is encrypted either way, never plaintext.

func DSN added in v0.1.2 

func DSN(path string, rawKey []byte) string

DSN builds a canonical IAM SQLite DSN for the active (CGO/csqlite/SQLCipher) backend.

When rawKey is non-nil it is emitted as SQLCipher's native `key=x'HEX'` URI parameter, which SQLCipher applies inside sqlite3_open_v2 — before mattn's pragma battery — making the connection reopen-safe. The DSN consequently CONTAINS the key; callers MUST NOT log it.

rawKey must be exactly 32 bytes when non-nil (validated by Open; OpenDB trusts its caller). A nil key yields an unencrypted DSN for the global/dev/test engine.

The path is percent-escaped (escapeDBPath): a path containing '?' or '#' would otherwise prematurely terminate the file portion of the DSN and DROP the trailing query params — including `key=` — which would silently open the database UNENCRYPTED. Escaping makes that impossible regardless of caller input.

func EncryptionAvailable added in v0.1.2 

func EncryptionAvailable() bool

EncryptionAvailable reports whether this build can encrypt a keyed store at rest. It is ALWAYS true: the pure-Go hanzoai/sqlcipher codec is compiled into every build, so a keyed open always encrypts — through the live libsqlcipher codec when it is linked and proven (CodecLinked), otherwise through the pure-Go codec envelope (envelope.go). There is no build, and no build-tag or environment switch, under which a keyed store falls back to plaintext.

It is a value, not a capability probe: "can a keyed store be encrypted here?" is unconditionally yes. Which mechanism does it — live C codec vs pure-Go envelope — is CodecLinked's concern, and a broken libsqlcipher is caught by that runtime probe (never silent plaintext), which is orthogonal to whether encryption is available at all.

func IsConstraintForeignKey added in v0.2.2 

func IsConstraintForeignKey(err error) bool

IsConstraintForeignKey reports whether err is a FOREIGN KEY-constraint violation from the active SQLite backend.

func IsConstraintPrimaryKey added in v0.2.2 

func IsConstraintPrimaryKey(err error) bool

IsConstraintPrimaryKey reports whether err is a PRIMARY KEY-constraint violation from the active SQLite backend.

func IsConstraintUnique added in v0.2.2 

func IsConstraintUnique(err error) bool

IsConstraintUnique reports whether err is a UNIQUE-constraint violation from the active SQLite backend. It unwraps via errors.As, so a wrapped error is still classified. Returns false for a nil or non-sqlite error.

func OpenDB added in v0.1.2 

func OpenDB(path string, rawKey []byte) (*sql.DB, error)

OpenDB opens *path* as a database/sql DB on the CGO/SQLCipher backend with the given raw 256-bit key. It is the entry point for callers that need a *sql.DB (e.g. to hand to xorm.NewEngineWithDB). A nil key opens unencrypted. The returned DB is NOT pinged here; the caller (or Open) pings.

func OpenPragma added in v0.2.3 

func OpenPragma(dsn string, pragmas []Pragma) (*sql.DB, error)

OpenPragma opens a *sql.DB on the registered "sqlite" backend where EVERY pooled connection has the given pragmas applied, in order, via PRAGMA statements run at connect time.

Use it for pragmas that must hold on every connection but that the two backends do NOT accept uniformly in the DSN. The load-bearing case is wal_autocheckpoint: modernc honors `?_pragma=wal_autocheckpoint(0)` in the DSN, but mattn silently DROPS it (mattn's DSN grammar has no such param), so a binary built CGO=1 would re-enable auto-checkpoint and truncate the WAL out from under a WAL-shipping replicator — losing committed frames. Running the pragma per connection via ExecContext is backend-neutral (both mattn and modernc conns implement driver.ExecerContext), so it takes effect either way.

Order matters: put busy_timeout first so a connection blocks on a busy database before journal_mode=WAL is attempted (WAL can't be set while another connection holds the db).

dsn is a plain DSN — typically "file:"+path, or DSN(path, key) when the file is SQLCipher-encrypted (the key must ride the DSN; pragmas can't carry it). Do NOT also put these pragmas in the DSN; they're applied here.

func PragmaDSN added in v0.2.1 

func PragmaDSN(path string, pragmas []Pragma) string

PragmaDSN builds an unencrypted `file:` DSN for path that applies pragmas in mattn's `_NAME=VALUE` syntax. mattn recognises _busy_timeout, _journal_mode, _synchronous, _foreign_keys and _cache_size and silently ignores any other parameter — so pragmas it cannot express as a DSN param (journal_size_limit, temp_store) are inert here but harmless, while the correctness-critical ones (busy_timeout, WAL) DO apply. The path is percent-escaped (escapeDBPath) so a '?'/'#' in it cannot strip the query. No key is added — keyed/encrypted opens go through OpenDB/DSN.

func SetPersistWAL added in v0.2.3 

func SetPersistWAL(rawConn any, on bool) error

SetPersistWAL sets SQLite's SQLITE_FCNTL_PERSIST_WAL file control on rawConn — the raw driver connection handed to the callback of (*sql.Conn).Raw. When on, the -wal file is kept across the last connection close (SQLite otherwise deletes it), which WAL-shipping replication (hanzoai/replicate) requires so a crash mid-sync doesn't lose committed frames.

It is the one backend-neutral way to set PERSIST_WAL; consumers never touch the csqlite/modernc conn type directly. Returns an error if rawConn is not this build's SQLite connection type.

CGO backend: csqlite exposes SetFileControlInt; the op constant is csqlite's own SQLITE_FCNTL_PERSIST_WAL (= 10).

Types

type CommitHookFn added in v0.2.1 

type CommitHookFn func() int32

CommitHookFn is a commit-hook callback. Returning a non-zero value ABORTS the commit (SQLite rolls back and reports SQLITE_BUSY); returning zero lets it proceed. This is the one callback type consumers use regardless of backend — the build-tagged adapter converts it to the backend's native signature.

type Config 

type Config struct {
	// Encryption
	RawKey []byte // raw 256-bit key (skips KDF); nil means unencrypted

	// Replication
	Mode   Mode
	NodeID string
	Listen string   // bind address for replication
	Peers  []string // peer addresses

	// Threshold mode
	Threshold  int                // t value (signatures required)
	Parties    int                // n value (total parties)
	SigningKey ed25519.PrivateKey // this node's signing key

}

Config for opening a database.

type DB 

type DB struct {
	*sql.DB
	// contains filtered or unexported fields
}

DB wraps sql.DB with replication and encryption.

func Open 

func Open(path string, opts ...Option) (*DB, error)

Open opens an at-rest-encrypted (when keyed), optionally distributed SQLite database. Passing an encryption key (WithKey/WithRawKey) always encrypts, on every build: the SQLCipher codec (live libsqlcipher when linked, otherwise the pure-Go codec envelope) keys the file. It never writes a keyed store as plaintext.

func (*DB) Encrypted added in v0.1.2 

func (db *DB) Encrypted() bool

Encrypted reports whether this DB is encrypted at rest — true exactly when it was opened with a key. A keyed store always encrypts (EncryptionAvailable is always true), so the key alone decides it.

type HookRegisterer added in v0.2.1 

type HookRegisterer interface {
	RegisterCommitHook(CommitHookFn)
}

HookRegisterer installs a commit hook on a driver connection. Values satisfying it are produced by CommitHookRegisterer, never by asserting a raw backend conn directly (see the note above). Passing a nil CommitHookFn clears any installed hook.

func CommitHookRegisterer added in v0.2.1 

func CommitHookRegisterer(driverConn any) (HookRegisterer, bool)

CommitHookRegisterer adapts a raw csqlite driver connection (obtained from (*sql.Conn).Raw) to HookRegisterer. csqlite's *SQLiteConn commit-hook type is `func() int` (not int32), so the adapter widens our CommitHookFn's int32 result to csqlite's int. This is the SAME engine driver_cgo.go registers as the "sqlite" driver name. Returns (nil, false) if driverConn is not a csqlite connection.

type Mode 

type Mode string

Mode determines the replication strategy. 

const (
	ModeSingle    Mode = "single"    // Local only
	ModeRaft      Mode = "raft"      // Strong consistency (leader writes)
	ModeCRDT      Mode = "crdt"      // Eventual consistency (all write)
	ModeThreshold Mode = "threshold" // t-of-n attestation for writes
)

type Option 

type Option func(*Config)

Option configures a database.

func WithCRDT 

func WithCRDT(nodeID, listen string, peers []string) Option

WithCRDT enables CRDT eventual consistency replication.

func WithKey 

func WithKey(passphrase string) Option

WithKey derives the raw 256-bit page key from a passphrase with a SINGLE, UNSALTED SHA-256 — NOT a slow, salted KDF. It is therefore safe ONLY for a high-entropy passphrase (e.g. a base64-encoded random secret); a human-memorable passphrase is brute-forceable because there is no PBKDF2 stretching and no salt.

A slow KDF is deliberately not applied here: the SQLCipher raw-key form takes the 32 bytes directly, and the file salt (which would seed a KDF) is not known when an Option is constructed. Production keys come from KMS already uniformly random — use WithRawKey / OpenDB(path, key) with those. For a low-entropy passphrase, derive a key out of band with a salted KDF (argon2id / PBKDF2) and pass it via WithRawKey.

func WithPeers 

func WithPeers(peers []string) Option

WithPeers sets replication peers.

func WithRaft 

func WithRaft(nodeID, listen string, peers []string) Option

WithRaft enables Raft consensus replication.

func WithRawKey 

func WithRawKey(key []byte) Option

WithRawKey sets a raw 256-bit encryption key (skips KDF).

func WithThreshold 

func WithThreshold(t, n int, signingKey ed25519.PrivateKey) Option

WithThreshold enables multi-party threshold attestation for writes.

type Pragma added in v0.2.1 

type Pragma struct{ Name, Value string }

Pragma is one SQLite PRAGMA (name + value) applied at connection-open time via the DSN.

type ThresholdManager 

type ThresholdManager struct {
	// contains filtered or unexported fields
}

ThresholdManager coordinates multi-party write attestation.

func NewThresholdManager 

func NewThresholdManager(threshold, parties int, nodeID string, signingKey ed25519.PrivateKey) *ThresholdManager

NewThresholdManager creates a threshold write coordinator.

func (*ThresholdManager) Attest 

func (tm *ThresholdManager) Attest(proposalID [32]byte, nodeID string, signature []byte) error

Attest adds a peer's attestation to a pending proposal.

func (*ThresholdManager) CleanExpired 

func (tm *ThresholdManager) CleanExpired() int

CleanExpired removes expired proposals.

func (*ThresholdManager) Pending 

func (tm *ThresholdManager) Pending() int

Pending returns the number of pending proposals.

func (*ThresholdManager) Propose 

func (tm *ThresholdManager) Propose(sql string, params []any) ([32]byte, error)

Propose creates a new write proposal. Returns the proposal ID.

func (*ThresholdManager) RegisterPeer 

func (tm *ThresholdManager) RegisterPeer(nodeID string, pubKey ed25519.PublicKey)

RegisterPeer adds a peer's public key for attestation verification.

func (*ThresholdManager) SetCommitFunc 

func (tm *ThresholdManager) SetCommitFunc(fn func(string, []any) error)

SetCommitFunc sets the function called when threshold is met.

type WriteProposal 

type WriteProposal struct {
	ID        [32]byte // SHA-256 of the SQL + params
	SQL       string
	Params    []any
	Proposer  string // node ID of proposer
	CreatedAt time.Time
	ExpiresAt time.Time
	// contains filtered or unexported fields
}

WriteProposal is a proposed write that needs t-of-n attestations.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.