Documentation
¶
Overview ¶
Package sqlitevec implements goagent.VectorStore over SQLite with the sqlite-vec extension (https://github.com/asg017/sqlite-vec).
Build requirements ¶
This package requires CGO and a C compiler. The sqlite-vec extension needs sqlite3.h, which mattn/go-sqlite3 ships as sqlite3-binding.h. A shim header at csrc/sqlite3.h bridges the two. Run once from the repository root:
go env -w CGO_CFLAGS="-I$(pwd)/memory/vector/sqlitevec/csrc -I$(go env GOMODCACHE)/github.com/mattn/go-sqlite3@v1.14.40"
The setting persists in your Go environment file (~/.config/go/env). Update the mattn/go-sqlite3 version suffix if that dependency is upgraded.
Schema ¶
This package uses two tables (created by Migrate or supplied by the caller):
goagent_embeddings — regular table: id, content, metadata, created_at goagent_embeddings_vec — vec0 virtual table: rowid (FK to main table), embedding
The vec0 table provides an indexed KNN search via sqlite-vec's MATCH operator.
Usage ¶
Call Register (or use Open) before opening any SQLite connection:
sqlitevec.Register()
db, err := sql.Open("sqlite3", "path/to/db.sqlite")
Or use the convenience wrapper:
db, err := sqlitevec.Open("path/to/db.sqlite")
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Migrate ¶
Migrate creates the data table and the vec0 virtual table if they do not exist. It is idempotent — safe to call on every application start.
Two tables are created:
- TableName: id TEXT PK, content TEXT, metadata TEXT (JSON), created_at INTEGER
- TableName+"_vec": vec0 virtual table with an embedding float[Dims] column
To use the created tables with New:
cfg := sqlitevec.MigrateConfig{TableName: "goagent_embeddings", Dims: 1536}
if err := sqlitevec.Migrate(ctx, db, cfg); err != nil { ... }
store, err := sqlitevec.New(db, sqlitevec.TableConfig{
Table: cfg.TableName,
IDColumn: "id",
VectorColumn: "embedding",
TextColumn: "content",
MetadataColumn: "metadata",
})
Example ¶
package main
import (
"context"
"fmt"
"log"
"github.com/Germanblandin1/goagent/memory/vector/sqlitevec"
)
func main() {
ctx := context.Background()
db, err := sqlitevec.Open("/path/to/mydb.sqlite")
if err != nil {
log.Fatal(err)
}
defer db.Close()
cfg := sqlitevec.MigrateConfig{
TableName: "goagent_embeddings",
Dims: 1536, // match your embedding model (e.g. text-embedding-3-small)
}
if err := sqlitevec.Migrate(ctx, db, cfg); err != nil {
log.Fatal(err)
}
store, err := sqlitevec.New(db, sqlitevec.TableConfig{
Table: cfg.TableName,
IDColumn: "id",
VectorColumn: "embedding",
TextColumn: "content",
MetadataColumn: "metadata",
})
if err != nil {
log.Fatal(err)
}
_ = store
fmt.Println("store ready")
}
Output:
Types ¶
type DistanceMetric ¶
type DistanceMetric string
DistanceMetric selects the vector similarity function used in Search.
const ( // L2 uses Euclidean distance via the sqlite-vec KNN index (MATCH … AND k = ?). // Score is 1 / (1 + distance), in (0, 1]. // This is the default. Queries are index-accelerated. L2 DistanceMetric = "l2" // Cosine uses cosine distance via the vec_distance_cosine SQL function. // Score is 1 - distance; for unit-normalised vectors the range is [0, 1]. // Queries perform a full scan — suitable for datasets up to tens of thousands // of rows. For larger datasets, normalise vectors before inserting and use L2 // (L2 on unit vectors is equivalent to cosine similarity). Cosine DistanceMetric = "cosine" )
type MigrateConfig ¶
type MigrateConfig struct {
// TableName is the base name for the two tables to create:
// TableName — regular data table
// TableName+"_vec" — vec0 virtual table for vector search
// Default: "goagent_embeddings".
TableName string
// Dims is the vector dimension. Required. Must match the embedding model
// (e.g. 768 for nomic-embed-text, 1024 for voyage-3, 1536 for
// text-embedding-3-small).
Dims int
// Metric is the distance metric that will be used with New.
// Stored only for documentation — has no effect on the schema.
// Default: L2.
Metric DistanceMetric
}
MigrateConfig configures the tables that Migrate will create. Use this when the caller has no existing schema and wants a quick start.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store implements goagent.VectorStore over SQLite with the sqlite-vec extension. It satisfies the goagent.VectorStore interface directly.
func New ¶
func New(db *sql.DB, cfg TableConfig, opts ...StoreOption) (*Store, error)
New creates a Store backed by db using the given TableConfig and options. Returns an error if any required field is missing or contains invalid characters.
Example ¶
package main
import (
"fmt"
"log"
"github.com/Germanblandin1/goagent/memory/vector/sqlitevec"
)
func main() {
db, err := sqlitevec.Open("/path/to/mydb.sqlite")
if err != nil {
log.Fatal(err)
}
defer db.Close()
store, err := sqlitevec.New(db, sqlitevec.TableConfig{
Table: "embeddings",
IDColumn: "id",
VectorColumn: "embedding",
TextColumn: "content",
MetadataColumn: "metadata", // optional; omit if your table has no metadata column
})
if err != nil {
log.Fatal(err)
}
_ = store
fmt.Println("store created")
}
Output:
func (*Store) Delete ¶
Delete removes the entry with the given id from both the data table and the vec0 virtual table. It is a no-op if id does not exist.
func (*Store) Search ¶
func (s *Store) Search(ctx context.Context, vec []float32, topK int) ([]goagent.ScoredMessage, error)
Search returns the topK messages most similar to vec, ordered by similarity descending. Each returned Message has RoleDocument so it is never forwarded to a provider.
L2 search uses sqlite-vec's indexed KNN (MATCH … AND k = ?). Cosine search uses vec_distance_cosine and performs a full scan.
type StoreOption ¶
type StoreOption func(*storeOptions)
StoreOption configures the behaviour of a Store.
func WithDistanceMetric ¶
func WithDistanceMetric(m DistanceMetric) StoreOption
WithDistanceMetric sets the similarity metric used in Search. Default: L2.
type TableConfig ¶
type TableConfig struct {
// Table is the regular SQLite table name (e.g. "goagent_embeddings").
Table string
// IDColumn is the PRIMARY KEY column of type TEXT in the data table.
IDColumn string
// VectorColumn is the vector column defined in the vec0 virtual table
// (the table named Table+"_vec").
VectorColumn string
// TextColumn is the TEXT column in the data table containing the chunk text.
TextColumn string
// MetadataColumn is optional. If non-empty, it must be a TEXT column holding
// a JSON object. Its content is deserialized into Message.Metadata.
MetadataColumn string
}
TableConfig describes the schema used by this package. The vec0 virtual table is inferred by appending "_vec" to Table. All fields are required except MetadataColumn.
Source Files