vectorstore

package
v0.53.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

Package vectorstore defines provider-neutral semantic indexing and search. Six independent interfaces split the surface by capability:

  • Indexer indexes documents.
  • Searcher finds similar documents by query + metadata filter.
  • IDDeleter removes documents by identifier.
  • FilterDeleter removes documents matching a metadata filter.
  • Closer releases the resources a store created for itself.
  • Batcher supplies an order-preserving ingestion partition policy.

There is deliberately no aggregate Store interface: consumers depend only on the capabilities they call, and providers implement only what they can support. Batching remains an injected capability rather than a framework dependency. IndexRequest owns the shared validation and batching boundary before provider I/O. SearchRequest validates search input, while SearchResponse validates ranked provider output against that request.

Indexed documents have a caller-assigned, non-empty ID and non-empty text. Providers preserve both values so every successful SearchResult is immediately usable by retrieval pipelines. Providers never generate IDs, and a vector-index-plus-external-document-store architecture must hydrate results explicitly outside these capabilities rather than return partial Documents. Provider distances and similarities are converted to the common Score contract with the ScoreFrom functions in this package.

Metadata filtering uses the filter mini-language: build predicates with typed constructors or parse them from text with filter.Parse. See github.com/Tangerg/scope/core/vectorstore/filter. SearchOptions JSON stores the canonical DSL string so serializing a request preserves its predicate.

Quick start:

expr, _ := filter.Parse(`category == 'tech' AND year >= 2020`)
req := &vectorstore.SearchRequest{
	Query: "attention",
	Options: vectorstore.SearchOptions{TopK: 5, MinScore: 0.7, Filter: expr},
}
response, err := searcher.Search(ctx, req)
Example
package main

import (
	"fmt"

	"github.com/Tangerg/scope/core/vectorstore"
	"github.com/Tangerg/scope/core/vectorstore/filter"
)

func main() {
	expr := filter.EQ("category", "wildlife")
	request := &vectorstore.SearchRequest{
		Query: "scope habitat",
		Options: vectorstore.SearchOptions{
			TopK: 5, MinScore: 0.7, Filter: expr,
		},
	}
	if err := request.Validate(); err != nil {
		panic(err)
	}

	fmt.Println(request.Query, request.Options.TopK, request.Options.MinScore, expr.Operator())
}
Output:
scope habitat 5 0.7 ==

Index

Examples

Constants

View Source
const (
	DefaultTopK = 5

	MinRelevanceScore = 0.0

	MaxRelevanceScore = 1.0
)

Variables

View Source
var (
	ErrInvalidOptions = errors.New("vectorstore: invalid options")

	ErrInvalidRequest = errors.New("vectorstore: invalid request")

	ErrInvalidResponse = errors.New("vectorstore: invalid response")

	ErrInvalidScore = errors.New("vectorstore: invalid score")

	ErrEmptyDocuments = errors.New("vectorstore: documents must not be empty")

	ErrInvalidDocument = errors.New("vectorstore: invalid document")

	ErrMissingDocumentID = errors.New("vectorstore: document ID is required")

	ErrDuplicateDocumentID = errors.New("vectorstore: duplicate document ID")

	ErrMissingFilter = errors.New("vectorstore: filter is required")

	ErrUnsupportedSearchMode = errors.New("vectorstore: unsupported search mode")
)
View Source
var ErrInvalidBatcherOutput = errors.New("vectorstore: invalid batcher output")

Functions

This section is empty.

Types

type Batcher

type Batcher interface {
	// Batch partitions the supplied pointers without cloning or retaining them.
	// Every input pointer must occur exactly once in the output, global order is
	// preserved, and empty batches are invalid. Context cancellation remains
	// identifiable through errors.Is.
	Batch(ctx context.Context, documents []*document.Document) ([][]*document.Document, error)
}

type Closer added in v0.16.0

type Closer interface {
	// Close makes the store unusable. Repeated calls follow the underlying
	// resource's close semantics and are not guaranteed to be idempotent.
	Close() error
}

Closer is implemented only by stores that create resources they must release. A store must not close a borrowed client, session, pool, or collection.

type FilterDeleter

type FilterDeleter interface {
	// DeleteWhere removes every document matching predicate. Implementations return
	// [ErrMissingFilter] for nil and reject invalid expressions.
	DeleteWhere(ctx context.Context, predicate filter.Predicate) error
}

type IDDeleter

type IDDeleter interface {
	// DeleteIDs removes the documents with the given ids. Unknown ids
	// are ignored (idempotent); an empty slice is a no-op.
	DeleteIDs(ctx context.Context, ids []string) error
}

type IndexRequest

type IndexRequest struct {
	Documents []*document.Document `json:"documents"`
}

IndexRequest validates the complete input and batching result; backend writes need not be atomic across batches.

func NewIndexRequest

func NewIndexRequest(documents []*document.Document) (*IndexRequest, error)

NewIndexRequest validates every document up front. Stores validate again at entry because callers can modify the request or its documents.

func (*IndexRequest) Batch

func (i *IndexRequest) Batch(ctx context.Context, batcher Batcher) ([]*IndexRequest, error)

Batch delegates to batcher and returns validated, order-preserving child requests. The receiver itself must be valid.

func (IndexRequest) MarshalJSON

func (i IndexRequest) MarshalJSON() ([]byte, error)

func (*IndexRequest) Texts added in v0.13.0

func (i *IndexRequest) Texts() ([]string, error)

Texts returns an owned, order-preserving projection of document text.

func (*IndexRequest) UnmarshalJSON

func (i *IndexRequest) UnmarshalJSON(data []byte) error

func (*IndexRequest) Validate

func (i *IndexRequest) Validate() error

type Indexer

type Indexer interface {
	// Index persists request documents using caller-assigned IDs. Existing IDs
	// are replaced according to the backend's upsert semantics. Implementations
	// validate the complete request before external I/O. Index does not promise
	// atomic writes across batches: an error may leave earlier documents stored.
	// Backend-specific transaction guarantees belong to the implementation.
	// Accepted document content, including Media, must survive retrieval;
	// implementations reject unsupported media before external I/O.
	//
	// Index never invents document IDs: its error-only result has no channel for
	// returning generated identities to the caller.
	Index(ctx context.Context, request *IndexRequest) error
}

type Score

type Score float64

Score is a provider-neutral, query-relative relevance value in [0, 1]. Scores preserve ordering but are not comparable across providers or search modes.

func ScoreFromCosineDistance

func ScoreFromCosineDistance(distance float64) Score

ScoreFromCosineDistance maps 1-cosine-similarity from [0, 2] to [0, 1].

func ScoreFromCosineSimilarity

func ScoreFromCosineSimilarity(similarity float64) Score

ScoreFromCosineSimilarity maps cosine similarity from [-1, 1] to [0, 1].

func ScoreFromDistance

func ScoreFromDistance(distance float64) Score

ScoreFromDistance maps a non-negative, unbounded distance to (0, 1], where zero is an exact match. Negative values, which only floating-point error produces, are treated as zero.

func ScoreFromInnerProduct

func ScoreFromInnerProduct(product float64) Score

ScoreFromInnerProduct maps an unbounded dot product monotonically into (0, 1).

func ScoreFromNegativeInnerProductDistance

func ScoreFromNegativeInnerProductDistance(distance float64) Score

ScoreFromNegativeInnerProductDistance maps a provider distance defined as the negative dot product into the similarity range.

func ScoreFromOneMinusInnerProductDistance

func ScoreFromOneMinusInnerProductDistance(distance float64) Score

ScoreFromOneMinusInnerProductDistance maps a provider distance defined as 1-dot-product into the similarity range.

func ScoreFromValue

func ScoreFromValue(value float64) Score

ScoreFromValue clamps a finite provider score to the common range. Non-finite input becomes NaN so result validation reports the contract breach.

func (Score) Float64

func (s Score) Float64() float64

func (Score) MarshalJSON

func (s Score) MarshalJSON() ([]byte, error)

func (*Score) UnmarshalJSON

func (s *Score) UnmarshalJSON(data []byte) error

func (Score) Validate

func (s Score) Validate() error

type SearchMode added in v0.12.0

type SearchMode string
const (
	SearchModeSemantic SearchMode = "semantic"
	SearchModeHybrid   SearchMode = "hybrid"
)

Stores must reject unsupported search modes rather than silently substitute another.

func (SearchMode) String added in v0.12.0

func (s SearchMode) String() string

func (SearchMode) Valid added in v0.12.0

func (s SearchMode) Valid() bool

type SearchOptions

type SearchOptions struct {
	// TopK limits the result count. Zero uses DefaultTopK.
	TopK     int   `json:"top_k,omitzero"`
	MinScore Score `json:"min_score,omitzero"`
	// Filter is encoded as its canonical filter DSL string. An omitted or null
	// JSON filter means no predicate; a present string must parse successfully.
	Filter filter.Predicate `json:"filter,omitempty"`
	Mode   SearchMode       `json:"mode,omitempty"`
}

SearchOptions owns the policies applied to a relevance search. Semantic is the zero-value mode; hybrid combines semantic and lexical evidence.

func (SearchOptions) EffectiveMode added in v0.12.0

func (s SearchOptions) EffectiveMode() SearchMode

func (SearchOptions) MarshalJSON

func (s SearchOptions) MarshalJSON() ([]byte, error)

func (SearchOptions) RequireMode added in v0.12.0

func (s SearchOptions) RequireMode(supported ...SearchMode) error

RequireMode rejects a valid request mode before provider I/O when the store cannot implement it without changing its semantics.

func (SearchOptions) ResultLimit

func (s SearchOptions) ResultLimit() int

func (*SearchOptions) UnmarshalJSON

func (s *SearchOptions) UnmarshalJSON(data []byte) error

func (SearchOptions) Validate

func (s SearchOptions) Validate() error

type SearchRequest

type SearchRequest struct {
	Query   string        `json:"query,omitempty"`
	Options SearchOptions `json:"options"`
}

func NewSearchRequest

func NewSearchRequest(query string) (*SearchRequest, error)

func (SearchRequest) MarshalJSON

func (s SearchRequest) MarshalJSON() ([]byte, error)

func (*SearchRequest) UnmarshalJSON

func (s *SearchRequest) UnmarshalJSON(data []byte) error

func (*SearchRequest) Validate

func (s *SearchRequest) Validate() error

type SearchResponse

type SearchResponse struct {
	Results []*SearchResult `json:"results"`
}

func NewSearchResponse

func NewSearchResponse(results []*SearchResult) (*SearchResponse, error)

func (*SearchResponse) Documents

func (s *SearchResponse) Documents() []*document.Document

func (*SearchResponse) First

func (s *SearchResponse) First() *SearchResult

func (SearchResponse) MarshalJSON

func (s SearchResponse) MarshalJSON() ([]byte, error)

func (*SearchResponse) UnmarshalJSON

func (s *SearchResponse) UnmarshalJSON(data []byte) error

func (*SearchResponse) Validate

func (s *SearchResponse) Validate() error

func (*SearchResponse) ValidateFor

func (s *SearchResponse) ValidateFor(request *SearchRequest) error

type SearchResult

type SearchResult struct {
	Document *document.Document `json:"document"`
	Score    Score              `json:"score"`
}

SearchResult keeps query-relative relevance separate from the indexed document.

func NewSearchResult

func NewSearchResult(matched *document.Document, score Score) (*SearchResult, error)

func (SearchResult) MarshalJSON

func (s SearchResult) MarshalJSON() ([]byte, error)

func (*SearchResult) UnmarshalJSON

func (s *SearchResult) UnmarshalJSON(data []byte) error

func (*SearchResult) Validate

func (s *SearchResult) Validate() error

type Searcher

type Searcher interface {
	// Search returns a response honoring the mode, semantic score threshold,
	// metadata filter, and result cap owned by [SearchRequest.Options].
	Search(ctx context.Context, request *SearchRequest) (*SearchResponse, error)
}

Searcher retrieves documents ranked by query relevance in descending order.

Directories

Path Synopsis
Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
Package inmemory provides an in-process vector store backed by a map and a configurable similarity function.
Package inmemory provides an in-process vector store backed by a map and a configurable similarity function.
Package storetest provides conformance suites for vector stores and filters.
Package storetest provides conformance suites for vector stores and filters.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL