vectorstore

package
v0.21.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 12, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package vectorstore defines provider-neutral semantic indexing and search. Six independent interfaces split the surface by capability:

  • Indexer indexes documents.
  • Searcher finds similar documents by query + metadata filter.
  • IDDeleter removes documents by identifier.
  • FilterDeleter removes documents matching a metadata filter.
  • Closer releases the resources a store created for itself.
  • Batcher supplies an order-preserving ingestion partition policy.

There is deliberately no aggregate Store interface: consumers depend only on the capabilities they call, and providers implement only what they can support. Batching remains an injected capability rather than a framework dependency. IndexRequest owns the shared validation and batching boundary before provider I/O. SearchRequest validates search input, while SearchResponse validates ranked provider output against that request.

Indexed documents have a caller-assigned, non-empty ID and non-empty text. Providers preserve both values so every successful SearchResult is immediately usable by retrieval pipelines. Providers never generate IDs, and a vector-index-plus-external-document-store architecture must hydrate results explicitly outside these capabilities rather than return partial Documents. Provider distances and similarities are converted to the common Score contract with the ScoreFrom functions in this package.

Metadata filtering uses the filter mini-language: build predicates with typed constructors or parse them from text with filter.Parse. See github.com/Tangerg/scope/core/vectorstore/filter. SearchOptions JSON stores the canonical DSL string so serializing a request preserves its predicate.

Quick start: 

expr, _ := filter.Parse(`category == 'tech' AND year >= 2020`)
req := &vectorstore.SearchRequest{
	Query: "attention",
	Options: vectorstore.SearchOptions{TopK: 5, MinScore: 0.7, Filter: expr},
}
response, err := searcher.Search(ctx, req)
Example 
package main

import (
	"fmt"

	"github.com/Tangerg/scope/core/vectorstore"
	"github.com/Tangerg/scope/core/vectorstore/filter"
)

func main() {
	expr := filter.EQ("category", "wildlife")
	request := &vectorstore.SearchRequest{
		Query: "scope habitat",
		Options: vectorstore.SearchOptions{
			TopK: 5, MinScore: 0.7, Filter: expr,
		},
	}
	if err := request.Validate(); err != nil {
		panic(err)
	}

	fmt.Println(request.Query, request.Options.TopK, request.Options.MinScore, expr.Operator())
}

Output:
scope habitat 5 0.7 ==

Index

Examples

Constants

View Source 
const (
	// DefaultTopK is used when [SearchOptions.TopK] is zero.
	DefaultTopK = 5

	// MinRelevanceScore is the lowest valid score.
	MinRelevanceScore = 0.0

	// MaxRelevanceScore is the highest valid score.
	MaxRelevanceScore = 1.0
)

Relevance-score range for SearchOptions.MinScore and search defaults.

Variables

View Source 
var (
	ErrInvalidOptions = errors.New("vectorstore: invalid options")

	ErrInvalidRequest = errors.New("vectorstore: invalid request")

	ErrInvalidResponse = errors.New("vectorstore: invalid response")

	ErrInvalidScore = errors.New("vectorstore: invalid score")

	ErrEmptyDocuments = errors.New("vectorstore: documents must not be empty")

	ErrInvalidDocument = errors.New("vectorstore: invalid document")

	ErrMissingDocumentID = errors.New("vectorstore: document ID is required")

	ErrDuplicateDocumentID = errors.New("vectorstore: duplicate document ID")

	ErrMissingFilter = errors.New("vectorstore: filter is required")

	ErrUnsupportedSearchMode = errors.New("vectorstore: unsupported search mode")
)
View Source 
var ErrInvalidBatcherOutput = errors.New("vectorstore: invalid batcher output")

Functions

This section is empty.

Types

type Batcher 

type Batcher interface {
	// Batch partitions the supplied pointers without cloning or retaining them.
	// Every input pointer must occur exactly once in the output, global order is
	// preserved, and empty batches are invalid. Context cancellation remains
	// identifiable through errors.Is.
	Batch(ctx context.Context, documents []*document.Document) ([][]*document.Document, error)
}

Batcher partitions documents for ingestion. It must preserve every document pointer exactly once and in input order, and it must not return empty batches. Implementations commonly come from document pipelines; stores depend only on this narrow capability contract.

type Closer added in v0.16.0 

type Closer interface {
	// Close releases what the store created. The store is unusable afterward,
	// and a second call is not a no-op: the underlying resource decides what a
	// repeat close means, and a gRPC connection reports an error for it.
	Close() error
}

Closer releases the resources a store created for itself.

It is a capability rather than a method every store carries, because most stores create nothing: they are handed a client, session, pool or collection and hold it for as long as the caller keeps it open. Closing a caller-owned resource is not cleanup but sabotage — one client normally serves several collections, so a store that closed it would take down every other store sharing it. A store in that position implements nothing here, and a caller detects the difference the way it detects every other capability: 

if closer, ok := store.(vectorstore.Closer); ok {
	defer closer.Close()
}

Answering with a no-op Close instead is worse than answering with nothing. It tells a caller that this store has resources to release and that calling Close released them, and both are false; the caller then cannot tell the stores that need cleanup from the stores that do not, which is the only thing it asked.

type FilterDeleter 

type FilterDeleter interface {
	// DeleteWhere removes every document matching predicate. Implementations return
	// [ErrMissingFilter] for nil and reject invalid expressions.
	DeleteWhere(ctx context.Context, predicate filter.Predicate) error
}

FilterDeleter removes documents selected by a metadata expression. It is a separate capability because some providers can search but cannot mutate their managed index.

type IDDeleter 

type IDDeleter interface {
	// DeleteIDs removes the documents with the given ids. Unknown ids
	// are ignored (idempotent); an empty slice is a no-op.
	DeleteIDs(ctx context.Context, ids []string) error
}

IDDeleter removes documents by identifier. It is independent from FilterDeleter: providers frequently expose only one of the two paths.

type IndexRequest 

type IndexRequest struct {
	Documents []*document.Document `json:"documents"`
}

IndexRequest describes documents for one indexing call. It owns their provider-independent validation and batching lifecycle. Validation covers the complete request; it does not make backend writes atomic across batches.

func NewIndexRequest 

func NewIndexRequest(documents []*document.Document) (*IndexRequest, error)

NewIndexRequest validates every document up front. Stores validate again at entry because callers can modify the request or its documents.

func (*IndexRequest) Batch 

func (i *IndexRequest) Batch(ctx context.Context, batcher Batcher) ([]*IndexRequest, error)

Batch delegates to batcher and returns validated, order-preserving child requests. The receiver itself must be valid.

func (IndexRequest) MarshalJSON 

func (i IndexRequest) MarshalJSON() ([]byte, error)

func (*IndexRequest) Texts added in v0.13.0 

func (i *IndexRequest) Texts() ([]string, error)

Texts returns an owned, order-preserving projection of document text.

func (*IndexRequest) UnmarshalJSON 

func (i *IndexRequest) UnmarshalJSON(data []byte) error

func (*IndexRequest) Validate 

func (i *IndexRequest) Validate() error

type Indexer 

type Indexer interface {
	// Index persists request documents using caller-assigned IDs. Existing IDs
	// are replaced according to the backend's upsert semantics. Implementations
	// validate the complete request before external I/O. Index does not promise
	// atomic writes across batches: an error may leave earlier documents stored.
	// Backend-specific transaction guarantees belong to the implementation.
	// Accepted document content, including Media, must survive retrieval;
	// implementations reject unsupported media before external I/O.
	//
	// Index never invents document IDs: its error-only result has no channel for
	// returning generated identities to the caller.
	Index(ctx context.Context, request *IndexRequest) error
}

Indexer embeds and indexes documents in the vector store. The store runs:

  1. Embedding (text → vector)
  2. Indexing (vector + metadata → searchable record)
  3. Storage (record → durable backend)

type Score 

type Score float64

Score is a provider-neutral, query-relative relevance value in [0, 1]. Scores preserve ordering but are not comparable across providers or search modes.

func ScoreFromCosineDistance 

func ScoreFromCosineDistance(distance float64) Score

ScoreFromCosineDistance maps 1-cosine-similarity from [0, 2] to [0, 1].

func ScoreFromCosineSimilarity 

func ScoreFromCosineSimilarity(similarity float64) Score

ScoreFromCosineSimilarity maps cosine similarity from [-1, 1] to [0, 1].

func ScoreFromDistance 

func ScoreFromDistance(distance float64) Score

ScoreFromDistance maps a non-negative, unbounded distance to (0, 1], where zero is an exact match. Tiny negative values caused by floating-point error are treated as zero.

func ScoreFromInnerProduct 

func ScoreFromInnerProduct(product float64) Score

ScoreFromInnerProduct maps an unbounded dot product monotonically into (0, 1).

func ScoreFromNegativeInnerProductDistance 

func ScoreFromNegativeInnerProductDistance(distance float64) Score

ScoreFromNegativeInnerProductDistance maps a provider distance defined as the negative dot product into the similarity range.

func ScoreFromOneMinusInnerProductDistance 

func ScoreFromOneMinusInnerProductDistance(distance float64) Score

ScoreFromOneMinusInnerProductDistance maps a provider distance defined as 1-dot-product into the similarity range.

func ScoreFromValue 

func ScoreFromValue(value float64) Score

ScoreFromValue clamps a finite provider score to the common range. Non-finite input becomes NaN so result validation reports the contract breach.

func (Score) Float64 

func (s Score) Float64() float64

func (Score) MarshalJSON 

func (s Score) MarshalJSON() ([]byte, error)

func (*Score) UnmarshalJSON 

func (s *Score) UnmarshalJSON(data []byte) error

func (Score) Validate 

func (s Score) Validate() error

type SearchMode added in v0.12.0 

type SearchMode string

SearchMode selects the retrieval evidence used by a search operation. 

const (
	SearchModeSemantic SearchMode = "semantic"
	SearchModeHybrid   SearchMode = "hybrid"
)

Search modes are explicit so a caller can require one and be refused by a store that cannot honor it. Silently downgrading a hybrid request to semantic would return plausible results that answer a different query than the one asked.

func (SearchMode) String added in v0.12.0 

func (s SearchMode) String() string

func (SearchMode) Valid added in v0.12.0 

func (s SearchMode) Valid() bool

type SearchOptions 

type SearchOptions struct {
	// TopK limits the result count. Zero uses DefaultTopK.
	TopK     int   `json:"top_k,omitempty"`
	MinScore Score `json:"min_score,omitempty"`
	// Filter is encoded as its canonical filter DSL string. An omitted or null
	// JSON filter means no predicate; a present string must parse successfully.
	Filter filter.Predicate `json:"filter,omitempty"`
	Mode   SearchMode       `json:"mode,omitempty"`
}

SearchOptions owns the policies applied to a relevance search. Semantic is the zero-value mode; hybrid combines semantic and lexical evidence.

func (SearchOptions) EffectiveMode added in v0.12.0 

func (s SearchOptions) EffectiveMode() SearchMode

EffectiveMode returns semantic for the zero-value mode.

func (SearchOptions) MarshalJSON 

func (s SearchOptions) MarshalJSON() ([]byte, error)

func (SearchOptions) RequireMode added in v0.12.0 

func (s SearchOptions) RequireMode(supported ...SearchMode) error

RequireMode rejects a valid request mode before provider I/O when the store cannot implement it without changing its semantics.

func (SearchOptions) ResultLimit 

func (s SearchOptions) ResultLimit() int

ResultLimit returns the explicit TopK or DefaultTopK when it is omitted.

func (*SearchOptions) UnmarshalJSON 

func (s *SearchOptions) UnmarshalJSON(data []byte) error

func (SearchOptions) Validate 

func (s SearchOptions) Validate() error

type SearchRequest 

type SearchRequest struct {
	Query   string        `json:"query,omitempty"`
	Options SearchOptions `json:"options"`
}

SearchRequest describes one relevance search and owns its input validation.

func NewSearchRequest 

func NewSearchRequest(query string) (*SearchRequest, error)

NewSearchRequest starts from the query alone because everything else — top-k, filters, mode — has a defined default. Requiring them would push the same boilerplate into every call site.

func (SearchRequest) MarshalJSON 

func (s SearchRequest) MarshalJSON() ([]byte, error)

func (*SearchRequest) UnmarshalJSON 

func (s *SearchRequest) UnmarshalJSON(data []byte) error

func (*SearchRequest) Validate 

func (s *SearchRequest) Validate() error

type SearchResponse 

type SearchResponse struct {
	Results []*SearchResult `json:"results"`
}

SearchResponse owns a complete ranked result set.

func NewSearchResponse 

func NewSearchResponse(results []*SearchResult) (*SearchResponse, error)

NewSearchResponse validates the result set as a whole, which is where an adapter's ordering or scoring mistakes become visible; an individually valid result says nothing about the ranking it sits in.

func (*SearchResponse) Documents 

func (s *SearchResponse) Documents() []*document.Document

func (*SearchResponse) First 

func (s *SearchResponse) First() *SearchResult

func (SearchResponse) MarshalJSON 

func (s SearchResponse) MarshalJSON() ([]byte, error)

func (*SearchResponse) UnmarshalJSON 

func (s *SearchResponse) UnmarshalJSON(data []byte) error

func (*SearchResponse) Validate 

func (s *SearchResponse) Validate() error

func (*SearchResponse) ValidateFor 

func (s *SearchResponse) ValidateFor(request *SearchRequest) error

type SearchResult 

type SearchResult struct {
	Document *document.Document `json:"document"`
	Score    Score              `json:"score"`
}

SearchResult relates a document to one search operation. Score is deliberately kept outside document.Document: relevance belongs to a query/result pair, not to the indexed content itself.

func NewSearchResult 

func NewSearchResult(matched *document.Document, score Score) (*SearchResult, error)

NewSearchResult pairs a document with a validated Score, so a backend cannot emit a NaN or out-of-range relevance that would silently corrupt ranking and fusion downstream.

func (SearchResult) MarshalJSON 

func (s SearchResult) MarshalJSON() ([]byte, error)

func (*SearchResult) UnmarshalJSON 

func (s *SearchResult) UnmarshalJSON(data []byte) error

func (*SearchResult) Validate 

func (s *SearchResult) Validate() error

type Searcher 

type Searcher interface {
	// Search returns a response honoring the mode, semantic score threshold,
	// metadata filter, and result cap owned by [SearchRequest.Options].
	Search(ctx context.Context, request *SearchRequest) (*SearchResponse, error)
}

Searcher retrieves documents ranked by query relevance in descending order.

Source Files

View all Source files

Directories

Path Synopsis
Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
 Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
Package inmemory provides an in-process vector store backed by a map and a configurable similarity function.
 Package inmemory provides an in-process vector store backed by a map and a configurable similarity function.
Package storetest contains provider-independent contract tests for vector-store implementations and their filter visitors.
 Package storetest contains provider-independent contract tests for vector-store implementations and their filter visitors.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.