Documentation
¶
Overview ¶
Package memory — FTS5 lexical shadow index (change 0021).
ftsIndex is a derived full-text index over the chunks table's (id, text), backed by modernc.org/sqlite (transpiled C→Go, cgo-free, FTS5-capable). It exists because Turso's libturso binary does not ship the fts5 module (verified 2026-07-25 across both turso-go v0.2.2 and turso.tech/database/tursogo v0.7.1). The shadow is a pure derived index: reconstructable from chunks at any time via Store.RebuildFTS.
The shadow registers under the database/sql driver name "sqlite", which does not collide with the primary "turso" driver — both engines coexist in one binary. No core package imports modernc.org/sqlite; this adapter is an internal collaborator of Store behind the existing MemoryStore port.
Package memory — hybrid retrieval (T-042; change 0020 vector-only; change 0021 restored the lexical half via an FTS5 shadow index; change 0024 made the RRF fusion tunable).
Search fuses two ranked lists via weighted Reciprocal Rank Fusion:
- vector KNN: Turso's vector_distance_cos against the embedding column.
- lexical bm25: the modernc.org/sqlite FTS5 shadow index (when present).
Each half contributes w/(K+rank) where w is the half's weight and K the rank-damping constant (standard 60), both from Store.rrf (DefaultRRF = {60, 1, 1} — the change-0021 equal-weight behavior). When the shadow is absent (Open without WithFTS), Search is vector-only; LexicalWeight has no effect. See WithRRF / RRFConfig.
Package memory is the persistent memory store (ADR-0013/0014; changes 0020/0021/0022). It runs on the cgo-free turso.tech/database/tursogo v0.7.1 driver (libturso via purego), with the native vector column (vector32() + vector_distance_cos()) on the primary Turso database. The lexical half of hybrid retrieval runs on a separate cgo-free modernc.org/sqlite FTS5 shadow index (change 0021); Turso's libturso ships no fts5 module, so the two engines are composed behind this package's Store. Memory + embeddings stay local; chunk text never leaves the host.
Change 0020 replaced the prior ncruces/go-sqlite3 + sqlite-vec-go-bindings stack with Turso. Change 0022 re-targeted from the archived github.com/tursodatabase/turso-go to the canonical turso.tech/database/ tursogo path (ADR-0015).
Index ¶
- type OpenOption
- type RRFConfig
- type Result
- type Store
- func (s *Store) Close() error
- func (s *Store) DB() *sql.DB
- func (s *Store) RebuildFTS(ctx context.Context) error
- func (s *Store) Search(ctx context.Context, query string, k int) ([]Result, error)
- func (s *Store) SetMaxEntries(n int)
- func (s *Store) VecVersion() (string, error)
- func (s *Store) Write(ctx context.Context, text, source string) (int64, error)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type OpenOption ¶
type OpenOption func(*Store)
OpenOption configures an Open call. The zero-value option set gives the change 0020 vector-only behavior; WithFTS opts into hybrid retrieval.
func WithFTS ¶
func WithFTS() OpenOption
WithFTS enables the FTS5 lexical shadow index for hybrid retrieval. The shadow lives at a path derived from the primary (":memory:" → ":memory:", or "<base>.fts.db" beside the primary file). It is a derived index, reconstructable via RebuildFTS.
func WithRRF ¶
func WithRRF(cfg RRFConfig) OpenOption
WithRRF overrides the store's Reciprocal Rank Fusion config (change 0024). The config is applied as-is — there is no zero-value magic, so a caller passing WithRRF(RRFConfig{}) gets {0,0,0} (their explicit choice to zero out the fusion). The neutral default comes from DefaultRRF(), which Open applies before options; most callers either omit WithRRF entirely or pass DefaultRRF() with a field or two adjusted.
type RRFConfig ¶
type RRFConfig struct {
K float64 // rank-damping constant; standard 60. Must be > 0 for a physical fusion.
VectorWeight float64 // weight on the vector-KNN half. 0 = disable the vector contribution.
LexicalWeight float64 // weight on the lexical-bm25 half. 0 = disable the lexical contribution.
}
RRFConfig tunes Reciprocal Rank Fusion of the two retrieval halves in Search. The fused score for a chunk is:
VectorWeight/(K + vectorRank) + LexicalWeight·(1/(K + lexicalRank))
where each rank is 0-indexed and a half that does not return the chunk contributes 0 for that half. K is the rank-damping constant (the standard RRF value is 60); lower K makes top ranks dominate more. DefaultRRF() returns the proven-neutral {60, 1, 1} — equal weights, standard K — which reproduces the change-0021 fusion exactly.
func DefaultRRF ¶
func DefaultRRF() RRFConfig
DefaultRRF returns the neutral RRF config: K=60, both weights 1.0. This is the change-0021 equal-weight behavior and the value Open applies when no WithRRF option is passed. Tests that assert exact hybrid scores (e.g. the +1/60 lexical boost) rely on these defaults.
type Result ¶
type Result struct {
ID int64
Text string
Source string
Score float64 // fused weighted-RRF score (higher = more relevant)
}
Result is one ranked retrieval hit.
type Store ¶
type Store struct {
// Embedder turns written text into vectors (T-042+). Nil → Write errors.
Embedder embed.Embedder
// contains filtered or unexported fields
}
Store is an open memory database. Methods are safe for concurrent use as the underlying *sql.DB manages a connection pool.
func Open ¶
func Open(path string, opts ...OpenOption) (*Store, error)
Open opens (or creates) the database at path. Use ":memory:" for an ephemeral in-memory store. Recommended pragmas (WAL, sane busy timeout) are applied.
Options: pass WithFTS() to enable the FTS5 lexical shadow index for hybrid (lexical+vector) retrieval. Without it, the store is vector-only (the change 0020 default). The shadow lives at a derived sibling path.
func (*Store) DB ¶
DB exposes the underlying handle for store-internal migrations (T-042+). Callers must not close it; use Store.Close.
func (*Store) RebuildFTS ¶
RebuildFTS reconstructs the lexical shadow index from the primary chunks table. It clears chunks_fts and re-indexes every (id, text) row. Use this to recover from a corrupt or missing shadow file, or after bulk-loading chunks while the shadow was disabled. No-op when the shadow is absent (vector-only store); returns nil in that case.
func (*Store) Search ¶
Search runs hybrid vector+lexical retrieval, returning the top-k chunks ranked by weighted Reciprocal Rank Fusion of the two halves. Returns nil (no error) if nothing has been written yet.
The fusion weights and K come from Store.rrf (DefaultRRF = {60, 1, 1}, applied by Open). Override with the WithRRF option. When the FTS shadow is absent, Search is the change-0020 vector-only path.
func (*Store) SetMaxEntries ¶
SetMaxEntries caps the stored chunks at n. Oldest chunks (lowest id) are pruned on each Write so the cap is enforced even mid-session. n == 0 disables the cap. Calling SetMaxEntries(n) with n < 0 is a programming error and treated as 0.
func (*Store) VecVersion ¶
VecVersion reports the libturso version baked into the Go bindings. This is the canary test that change 0020's migration actually loaded the Turso-backed driver — if this query returns empty, the libturso load failed at the import time. (Turso v0.2.2's libturso does not expose sqlite-vec's `vec_version()` SQL function, so we read the SQLite version instead, which is always present.)
Source Files