store

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 25, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Overview

Package store defines the storage provider interface for knowledge bases.

The boundary is semantic ("index this document", "run this hybrid query"), not SQL: each provider resolves hybrid search with whatever mechanism its backend offers. The v0.1 provider is SQLite (FTS5 + sqlite-vec + RRF in Go); pgvector, OpenSearch and S3 Vectors can plug in later through Register without touching the engine.

Index

Constants

This section is empty.

Variables

View Source
var ErrNotFound = fmt.Errorf("document not found")

ErrNotFound is returned when a requested document does not exist.

Functions

func Providers

func Providers() []string

Providers lists registered provider names, sorted.

func Register

func Register(name string, f Factory)

Register makes a provider available under a name. It panics on duplicate registration, mirroring database/sql.

Types

type Chunk

type Chunk struct {
	Seq         int
	HeadingPath string // "H1 > H2 > H3", empty for flat documents
	StartLine   int
	EndLine     int
	Text        string
	Context     string // LLM-generated situating context (contextual retrieval)
	TokenEst    int
}

Chunk is a retrievable slice of a document.

type Document

type Document struct {
	ID         int64  `json:"-"`
	SourceName string `json:"source"`
	RelPath    string `json:"rel_path"` // path within the source, or generated id for managed docs
	URI        string `json:"uri"`      // file://..., git URL#branch:path, managed://<kb>/<id>
	Title      string `json:"title"`
	SHA256     string `json:"content_sha256"`
	SizeBytes  int64  `json:"size_bytes"`
	MtimeUnix  int64  `json:"mtime_unix"`
	// Metadata is a flat string map attached to the document (agent writes,
	// source annotations). Filterable at search time.
	Metadata map[string]string `json:"metadata,omitempty"`
}

Document is the indexed unit: one file, one managed memory, one page.

type DocumentContent

type DocumentContent struct {
	Document Document
	Text     string // reassembled from chunks in seq order
}

DocumentContent is a full document as stored, for get_document.

type Factory

type Factory func(ctx context.Context, opts Options) (Store, error)

Factory opens (creating if needed) the store for one knowledge base.

type FileState

type FileState struct {
	SHA256    string
	SizeBytes int64
	MtimeUnix int64
}

FileState is what incremental sync compares against a source listing.

type Hit

type Hit struct {
	DocumentID  int64             `json:"document_id"`
	SourceName  string            `json:"source"`
	RelPath     string            `json:"rel_path"`
	URI         string            `json:"uri"`
	Title       string            `json:"title"`
	HeadingPath string            `json:"heading_path,omitempty"`
	StartLine   int               `json:"start_line"`
	EndLine     int               `json:"end_line"`
	Text        string            `json:"text"`
	Context     string            `json:"context,omitempty"`
	Metadata    map[string]string `json:"metadata,omitempty"`
	Score       float64           `json:"score"`              // fused (RRF) or reranked score, higher is better
	FTSRank     int               `json:"fts_rank,omitempty"` // 1-based rank in the keyword result list, 0 if absent
	VecRank     int               `json:"vec_rank,omitempty"` // 1-based rank in the vector result list, 0 if absent
}

Hit is one hybrid search result.

type Options

type Options struct {
	KBName     string
	ModelName  string
	Dimensions int
	// ProviderConfig is the raw per-provider config block from the YAML.
	ProviderConfig map[string]any
}

Options carries provider-independent identity for a knowledge base store. Providers must record ModelName and Dimensions at creation time and refuse to open (hard error, never wipe) when an existing store disagrees.

type Stats

type Stats struct {
	Documents     int64
	Chunks        int64
	LastIndexedAt int64
}

Stats summarizes a knowledge base.

type Store

type Store interface {
	// UpsertDocument replaces the document at (doc.SourceName, doc.RelPath)
	// with the given chunks and their embeddings, atomically.
	// len(chunks) must equal len(embeddings).
	UpsertDocument(ctx context.Context, doc Document, chunks []Chunk, embeddings [][]float32) error

	// DeleteDocument removes a document and its chunks. Deleting a document
	// that does not exist is not an error.
	DeleteDocument(ctx context.Context, sourceName, relPath string) error

	// Manifest returns the sync state of every tracked file for a source.
	Manifest(ctx context.Context, sourceName string) (map[string]FileState, error)

	// TouchManifest updates a file's manifest entry without reindexing, for
	// files whose size/mtime changed but whose content hash did not.
	TouchManifest(ctx context.Context, sourceName, relPath string, fs FileState) error

	// HybridSearch runs keyword and vector retrieval and fuses the results.
	// queryVec dimension must match the store's configured dimension.
	HybridSearch(ctx context.Context, query string, queryVec []float32, k int) ([]Hit, error)

	// GetDocument fetches one document by (sourceName, relPath). Returns
	// ErrNotFound if it does not exist. sourceName may be empty to search
	// across all sources of the KB (first match wins; rel paths are unique
	// per source).
	GetDocument(ctx context.Context, sourceName, relPath string) (*DocumentContent, error)

	// ListDocuments pages through documents ordered by rel_path. A prefix
	// filters by rel_path prefix; cursor is the last rel_path of the
	// previous page ("" for the first page).
	ListDocuments(ctx context.Context, prefix, cursor string, limit int) ([]Document, error)

	Stats(ctx context.Context) (Stats, error)

	Close() error
}

Store is one knowledge base's storage. Implementations must make UpsertDocument and DeleteDocument atomic: a failure leaves the previous version of the document fully intact.

func Open

func Open(ctx context.Context, provider string, opts Options) (Store, error)

Open opens a knowledge base store with the named provider.

Directories

Path Synopsis
Package pgvector implements store.Store on PostgreSQL with the pgvector extension: tsvector full-text search for keyword retrieval, a pgvector (HNSW) index for dense retrieval, and Reciprocal Rank Fusion in Go.
Package pgvector implements store.Store on PostgreSQL with the pgvector extension: tsvector full-text search for keyword retrieval, a pgvector (HNSW) index for dense retrieval, and Reciprocal Rank Fusion in Go.
Package sqlite implements the store.Store interface on a single SQLite file per knowledge base: FTS5 for keyword retrieval, and embeddings persisted as BLOBs served by an in-memory cosine KNN index (see vec.go).
Package sqlite implements the store.Store interface on a single SQLite file per knowledge base: FTS5 for keyword retrieval, and embeddings persisted as BLOBs served by an in-memory cosine KNN index (see vec.go).
Package storetest is a reusable conformance harness for store.Store implementations.
Package storetest is a reusable conformance harness for store.Store implementations.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL