Documentation
¶
Overview ¶
Package vec gives SQLite vector functions, for similarity search over vectors kept in ordinary tables:
sqlite.NewConnector(dsn, vec.Option())
A vector is a BLOB of float32 values in little-endian order, the bytes sqlite-vec and libSQL use for a float32 vector too. Each function also takes a vector written as a JSON array, such as '[0.1, -2, 3e4]'.
vec_f32(v) the vector as a BLOB
vec_to_json(v) the vector as a JSON array
vec_length(v) its number of dimensions
vec_normalize(v) the vector of length 1 in its direction
vec_distance_l2(a, b) the Euclidean distance
vec_distance_l1(a, b) the sum of the absolute differences
vec_distance_cosine(a, b) 1 minus the cosine of the angle between them:
0 for the same direction, 2 for opposite ones
vec_dot(a, b) the dot product
The k rows nearest to a vector come from a full scan:
SELECT id FROM doc ORDER BY vec_distance_cosine(embedding, vec_f32(?)) LIMIT 10
SQLite computes vec_f32(?) once per query. The time grows with the number of rows: 100000 vectors of 768 dimensions take about 0.1 s on an Apple M4 (BenchmarkNearest). On vectors stored normalized, vec_dot ordered DESC gives the cosine order.
The distances are summed in float64, so float32 values never overflow, and every platform gives the same bits. A NULL argument gives NULL. Vectors of different dimensions, an empty vector, an element that is NaN or infinite, and the cosine or the direction of a zero vector are errors. One such vector in a table fails every query that measures it, so insert vectors through vec_f32(?), which refuses elements that are not finite, or vec_normalize(?), which also refuses a zero vector. vec_length takes any vector, to find them.
Example ¶
package main
import (
"database/sql"
"fmt"
"github.com/liustack/sqlite"
"github.com/liustack/sqlite/ext/vec"
)
func main() {
c, err := sqlite.NewConnector(":memory:", vec.Option())
if err != nil {
panic(err)
}
db := sql.OpenDB(c)
defer db.Close()
db.SetMaxOpenConns(1)
_, err = db.Exec(`CREATE TABLE doc(title TEXT, embedding BLOB);
INSERT INTO doc VALUES
('cats', vec_f32('[0.9, 0.1, 0.0]')),
('dogs', vec_f32('[0.8, 0.3, 0.1]')),
('tax law', vec_f32('[0.0, 0.2, 0.9]'))`)
if err != nil {
panic(err)
}
rows, err := db.Query(`SELECT title, vec_distance_cosine(embedding, vec_f32(?)) AS d
FROM doc ORDER BY d LIMIT 2`, "[1, 0.2, 0]")
if err != nil {
panic(err)
}
defer rows.Close()
for rows.Next() {
var title string
var d float64
if err := rows.Scan(&title, &d); err != nil {
panic(err)
}
fmt.Printf("%s %.3f\n", title, d)
}
}
Output: cats 0.004 dogs 0.020
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
This section is empty.