Documentation
¶
Overview ¶
Package store defines the storage provider interface for knowledge bases.
The boundary is semantic ("index this document", "run this hybrid query"), not SQL: each provider resolves hybrid search with whatever mechanism its backend offers. The v0.1 provider is SQLite (FTS5 + sqlite-vec + RRF in Go); pgvector, OpenSearch and S3 Vectors can plug in later through Register without touching the engine.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNotFound = fmt.Errorf("document not found")
ErrNotFound is returned when a requested document does not exist.
Functions ¶
Types ¶
type Chunk ¶
type Chunk struct {
Seq int
HeadingPath string // "H1 > H2 > H3", empty for flat documents
StartLine int
EndLine int
Text string
Context string // LLM-generated situating context (contextual retrieval)
TokenEst int
}
Chunk is a retrievable slice of a document.
type Document ¶
type Document struct {
ID int64 `json:"-"`
SourceName string `json:"source"`
RelPath string `json:"rel_path"` // path within the source, or generated id for managed docs
URI string `json:"uri"` // file://..., git URL#branch:path, managed://<kb>/<id>
Title string `json:"title"`
SHA256 string `json:"content_sha256"`
SizeBytes int64 `json:"size_bytes"`
MtimeUnix int64 `json:"mtime_unix"`
// Metadata is a flat string map attached to the document (agent writes,
// source annotations). Filterable at search time.
Metadata map[string]string `json:"metadata,omitempty"`
}
Document is the indexed unit: one file, one managed memory, one page.
type DocumentContent ¶
type DocumentContent struct {
Document Document
Text string // reassembled from chunks in seq order
}
DocumentContent is a full document as stored, for get_document.
type Hit ¶
type Hit struct {
DocumentID int64 `json:"document_id"`
SourceName string `json:"source"`
RelPath string `json:"rel_path"`
URI string `json:"uri"`
Title string `json:"title"`
HeadingPath string `json:"heading_path,omitempty"`
StartLine int `json:"start_line"`
EndLine int `json:"end_line"`
Text string `json:"text"`
Context string `json:"context,omitempty"`
Metadata map[string]string `json:"metadata,omitempty"`
Score float64 `json:"score"` // fused (RRF) or reranked score, higher is better
FTSRank int `json:"fts_rank,omitempty"` // 1-based rank in the keyword result list, 0 if absent
VecRank int `json:"vec_rank,omitempty"` // 1-based rank in the vector result list, 0 if absent
}
Hit is one hybrid search result.
type Options ¶
type Options struct {
KBName string
ModelName string
Dimensions int
// ProviderConfig is the raw per-provider config block from the YAML.
ProviderConfig map[string]any
}
Options carries provider-independent identity for a knowledge base store. Providers must record ModelName and Dimensions at creation time and refuse to open (hard error, never wipe) when an existing store disagrees.
type Store ¶
type Store interface {
// UpsertDocument replaces the document at (doc.SourceName, doc.RelPath)
// with the given chunks and their embeddings, atomically.
// len(chunks) must equal len(embeddings).
UpsertDocument(ctx context.Context, doc Document, chunks []Chunk, embeddings [][]float32) error
// DeleteDocument removes a document and its chunks. Deleting a document
// that does not exist is not an error.
DeleteDocument(ctx context.Context, sourceName, relPath string) error
// Manifest returns the sync state of every tracked file for a source.
Manifest(ctx context.Context, sourceName string) (map[string]FileState, error)
// TouchManifest updates a file's manifest entry without reindexing, for
// files whose size/mtime changed but whose content hash did not.
TouchManifest(ctx context.Context, sourceName, relPath string, fs FileState) error
// HybridSearch runs keyword and vector retrieval and fuses the results.
// queryVec dimension must match the store's configured dimension.
HybridSearch(ctx context.Context, query string, queryVec []float32, k int) ([]Hit, error)
// GetDocument fetches one document by (sourceName, relPath). Returns
// ErrNotFound if it does not exist. sourceName may be empty to search
// across all sources of the KB (first match wins; rel paths are unique
// per source).
GetDocument(ctx context.Context, sourceName, relPath string) (*DocumentContent, error)
// ListDocuments pages through documents ordered by rel_path. A prefix
// filters by rel_path prefix; cursor is the last rel_path of the
// previous page ("" for the first page).
ListDocuments(ctx context.Context, prefix, cursor string, limit int) ([]Document, error)
Stats(ctx context.Context) (Stats, error)
Close() error
}
Store is one knowledge base's storage. Implementations must make UpsertDocument and DeleteDocument atomic: a failure leaves the previous version of the document fully intact.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package pgvector implements store.Store on PostgreSQL with the pgvector extension: tsvector full-text search for keyword retrieval, a pgvector (HNSW) index for dense retrieval, and Reciprocal Rank Fusion in Go.
|
Package pgvector implements store.Store on PostgreSQL with the pgvector extension: tsvector full-text search for keyword retrieval, a pgvector (HNSW) index for dense retrieval, and Reciprocal Rank Fusion in Go. |
|
Package sqlite implements the store.Store interface on a single SQLite file per knowledge base: FTS5 for keyword retrieval, and embeddings persisted as BLOBs served by an in-memory cosine KNN index (see vec.go).
|
Package sqlite implements the store.Store interface on a single SQLite file per knowledge base: FTS5 for keyword retrieval, and embeddings persisted as BLOBs served by an in-memory cosine KNN index (see vec.go). |
|
Package storetest is a reusable conformance harness for store.Store implementations.
|
Package storetest is a reusable conformance harness for store.Store implementations. |