Documentation
¶
Overview ¶
Package sqlite implements storage.Storage on top of a SQLCipher-encrypted SQLite database. Per-queue databases live in their own files; their encryption keys are derived from a single master seed plus a per-queue random salt via HKDF, so deleting the master row crypto-shreds the queue even if the file is later recovered from a backup or snapshot.
Index ¶
- Constants
- Variables
- type MasterSeed
- type PoolStats
- type Store
- func (s *Store) Close() error
- func (s *Store) CreateQueue(ctx context.Context, keys queue.OwnerKeys) (*queue.Queue, error)
- func (s *Store) DeleteMessage(ctx context.Context, queueID, messageID string) error
- func (s *Store) DeleteQueue(ctx context.Context, id string) error
- func (s *Store) ExpiredQueueIDs(ctx context.Context, t time.Time) ([]string, error)
- func (s *Store) GetQueue(ctx context.Context, id string) (*queue.Queue, error)
- func (s *Store) ListMessages(ctx context.Context, queueID string, limit int) ([]*queue.Message, bool, error)
- func (s *Store) PutMessage(ctx context.Context, queueID string, blob []byte) (*queue.Message, error)
- func (s *Store) Snapshot(ctx context.Context, dst string) error
Constants ¶
const MasterSeedBytes = 32
MasterSeedBytes is the length of the master seed in bytes.
const QueueKeySaltBytes = 32
QueueKeySaltBytes is the length of the per-queue salt stored in the master database alongside each queue row.
Variables ¶
var ( ErrMasterSeedMissing = errors.New("master seed missing: set " + envVarMasterSeed) ErrMasterSeedInvalid = fmt.Errorf("master seed must be base64-encoded %d bytes", MasterSeedBytes) ErrInvalidQueueSalt = fmt.Errorf("queue salt must be %d bytes", QueueKeySaltBytes) )
Sentinel errors related to seed loading.
Functions ¶
This section is empty.
Types ¶
type MasterSeed ¶
type MasterSeed [MasterSeedBytes]byte
MasterSeed is a high-entropy 32-byte secret used to derive per-database SQLCipher keys. It is loaded once at startup; production deployments should fetch it from an HSM or KMS rather than from an environment variable.
func LoadMasterSeed ¶
func LoadMasterSeed() (MasterSeed, error)
LoadMasterSeed reads the seed from the environment variable.
The value must be a base64 encoding (standard, with or without padding) of exactly MasterSeedBytes bytes.
func (MasterSeed) DeriveQueueKey ¶
func (m MasterSeed) DeriveQueueKey(queueID string, salt []byte) (string, error)
DeriveQueueKey returns the SQLCipher hex key (without the 'x' wrapper) for the per-queue database identified by queueID and bound to the supplied per-queue salt. Once the salt is overwritten or deleted, the queue's database file becomes unrecoverable even when the master seed is intact — this is the crypto-shred property.
func (MasterSeed) MasterKey ¶
func (m MasterSeed) MasterKey() (string, error)
MasterKey returns the SQLCipher hex key (without the 'x' wrapper) for the master metadata database. The error is reserved for any future HKDF-side failure; SHA-256 single-block expansion never fails today.
func (MasterSeed) QueueFilename ¶
func (m MasterSeed) QueueFilename(queueID string) string
QueueFilename returns the on-disk basename for the queue's database file. The basename is HMAC(seed, queueID) so a directory listing of the data dir does not reveal which queue identifiers exist; mapping a queueID to its file requires the master seed.
type PoolStats ¶
PoolStats is a point-in-time snapshot of the pool's counters, suitable for exporting to the L8 aggregate-counter pipeline.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a SQLCipher-backed implementation of storage.Storage.
One master database holds queue metadata (including the per-queue salt); each queue's messages live in their own encrypted file whose key is derived from the master seed, the queue id, and that salt. Deleting a queue's master row therefore crypto-shreds the queue: the file may persist on disk or in a backup, but without the salt the HKDF derivation cannot be reproduced.
Threat model the crypto-shred property covers:
- File-only attacker (backup leak, filesystem snapshot, forensic image): cannot recover deleted queues because the salt is gone from the master row, and we PRAGMA wal_checkpoint(FULL) after every DeleteQueue so nothing meaningful sits in master.db-wal.
- Cold seed leak: leaking only the master seed without any of the storage files yields no plaintext.
Threat model it does NOT cover:
- Live memory dump: master seed and active per-queue salts both live in the process heap as Go values; an attacker with /proc/<pid> access on a live process can still recover them.
- Operational metadata: directory shard count, file mtime/size, and master.db-wal size leak coarse activity signals to anyone with filesystem read access. Mitigate via encrypted block storage and `mount -o noatime`; tracked operationally.
- WAL bytes between commit and checkpoint on per-queue files (per-queue WAL is currently disabled, so this is a no-op today, but flip the bit and the same caveat applies).
func New ¶
func New(seed MasterSeed, dataDir string) (*Store, error)
New opens (or creates) the master database and initialises its schema. dataDir is created with 0700 permissions if it does not exist.
func (*Store) Close ¶
Close releases resources held by the store: it drains the per-queue pool first, then closes both master DB handles (reader and writer). The first close error is reported; subsequent ones are best-effort.
func (*Store) CreateQueue ¶
CreateQueue inserts a new queue row and creates its per-queue database file. A fresh per-queue salt is generated and persisted alongside the metadata; the salt is the load-bearing secret for the crypto-shred property when the queue is later deleted.
func (*Store) DeleteMessage ¶
DeleteMessage removes a single message from the queue.
func (*Store) DeleteQueue ¶
DeleteQueue removes a queue's database file first, then the metadata row, then forces a WAL checkpoint so the row's salt cannot survive in master.db-wal beyond the call.
Removing the file before the row means a partial failure leaves the master row pointing at a missing file — subsequent operations surface that as ErrQueueNotFound, which is the appropriate semantic (the queue has effectively been destroyed). Deleting the row first would risk leaving an orphan file with recoverable content; with crypto-shred the salt also vanishes when the row goes, but the file itself should not survive a successful delete.
The wal_checkpoint(FULL) call is the load-bearing piece for the crypto-shred guarantee against backup-style adversaries. Without it, a snapshot taken between DELETE and the next autocheckpoint (default 1000 pages, which a small master.db rarely hits) would still see the salt in the WAL and could decrypt the file.
func (*Store) ExpiredQueueIDs ¶
ExpiredQueueIDs returns queue identifiers whose expires_at is before t. Read-only — uses the reader pool so the periodic cleanup tick does not contend with in-flight writes on the master DB.
func (*Store) ListMessages ¶
func (s *Store) ListMessages(ctx context.Context, queueID string, limit int) ([]*queue.Message, bool, error)
ListMessages returns up to limit messages in arrival order plus a hasMore flag.
func (*Store) PutMessage ¶
func (s *Store) PutMessage(ctx context.Context, queueID string, blob []byte) (*queue.Message, error)
PutMessage appends an opaque blob to the queue.
func (*Store) Snapshot ¶
Snapshot writes a self-consistent backup of all encrypted databases to dst. The destination directory is created with the same 0700 permissions as the live data directory. Existing snapshot files at the target paths are overwritten.
Layout mirrors the live data directory: <dst>/master.db plus <dst>/<shard>/<hash>.db for each queue. The snapshot files retain the original SQLCipher encryption — opening them requires the same master seed and per-queue salts as the source.
The snapshot is taken online: writers continue to make progress against the source. Each queue file is captured under the read side of the per-queue lock, which is concurrent with PutMessage, ListMessages and DeleteMessage and only blocks DeleteQueue. Master is captured first; the queue set is then enumerated from live master and each shard captured one at a time. A queue created between master capture and enumeration is included; a queue created after enumeration is omitted from this snapshot — both cases are safe under the disaster-recovery use case.
Source Files