vectorstore

package
v0.60.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 4, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package vectorstore defines the raw, space-bound database capabilities. Upserter writes prepared records and Querier ranks prepared coordinates. Semantic text encoding and ingestion belong to rag/vector and rag/index. IDDeleter and FilterDeleter are independent optional operations. Unsupported native predicates must fail without enumerating source documents. ScanBudget grants an explicit, finite exception on backends that support it. Scan enumerates documents under ScanRequest; QueryByScan ranks a Query under an explicit ScanBudget. Consumers declare the narrow capability they use.

Example 
package main

import (
	"fmt"

	"github.com/Tangerg/scope/core/vectorstore"
	"github.com/Tangerg/scope/core/vectorstore/filter"
)

func main() {
	expr := filter.EQ("category", "wildlife")
	request := &vectorstore.Query{
		Space: "habitat-v1", Values: []float64{1, 0},
		Options: vectorstore.SearchOptions{
			TopK: 5, MinScore: 0.7, Filter: expr,
		},
	}
	if err := request.ValidateFor("habitat-v1"); err != nil {
		panic(err)
	}

	fmt.Println(request.Space, request.Options.TopK, request.Options.MinScore, expr.Operator())
}

Output:
habitat-v1 5 0.7 ==

Index

Examples

Constants

View Source 
const (
	DefaultTopK = 5

	MinRelevanceScore = 0.0

	MaxRelevanceScore = 1.0
)
View Source 
const SpaceStorageKeyPrefix = "sv_"

SpaceStorageKeyPrefix reserves a namespace so adapters can detect every declared binding, including a second constraint for a conflicting space.

Variables

View Source 
var (
	ErrInvalidOptions = errors.New("vectorstore: invalid options")

	ErrInvalidRequest = errors.New("vectorstore: invalid request")

	ErrInvalidResponse = errors.New("vectorstore: invalid response")

	ErrInvalidScore = errors.New("vectorstore: invalid score")

	ErrEmptyDocuments = errors.New("vectorstore: documents must not be empty")

	ErrInvalidDocument = errors.New("vectorstore: invalid document")

	ErrMissingDocumentID = errors.New("vectorstore: document ID is required")

	ErrDuplicateDocumentID = errors.New("vectorstore: duplicate document ID")

	ErrMissingFilter = errors.New("vectorstore: filter is required")

	ErrUnsupportedSearchMode = errors.New("vectorstore: unsupported search mode")
)
View Source 
var ErrIncompatibleSpace = errors.New("vectorstore: incompatible vector space")
View Source 
var ErrScanBudget = errors.New("vectorstore: complete scan exceeds the authorized budget")
View Source 
var ErrUnsupportedFilter = errors.New("vectorstore: metadata predicate cannot be executed natively")

Functions

func ValidateDocument added in v0.59.0 

func ValidateDocument(doc *document.Document) error

ValidateDocument checks stored identity and content. A prepared vector may describe media without text; individual stores can reject media they cannot preserve.

func ValidateDocuments added in v0.59.0 

func ValidateDocuments(docs []*document.Document) error

ValidateDocuments additionally rejects empty input and duplicate identities.

Types

type Batch added in v0.59.0 

type Batch struct {
	Space   SpaceID  `json:"space"`
	Records []Record `json:"records"`
}

Batch contains records from exactly one encoder space. Upsert borrows these records for the duration of the call; it must not retain mutable input.

func (Batch) MarshalJSON added in v0.59.0 

func (b Batch) MarshalJSON() ([]byte, error)

func (*Batch) UnmarshalJSON added in v0.59.0 

func (b *Batch) UnmarshalJSON(data []byte) error

func (*Batch) ValidateFor added in v0.59.0 

func (b *Batch) ValidateFor(space SpaceID) error

type Closer added in v0.16.0 

type Closer interface {
	// Close makes the store unusable. Repeated calls follow the underlying
	// resource's close semantics and are not guaranteed to be idempotent.
	Close() error
}

Closer is implemented only by stores that create resources they must release. A store must not close a borrowed client, session, pool, or collection.

type FilterDeleter 

type FilterDeleter interface {
	// DeleteWhere removes every document matching predicate. Implementations return
	// [ErrMissingFilter] for nil and reject invalid expressions.
	DeleteWhere(ctx context.Context, predicate filter.Predicate) error
}

type IDDeleter 

type IDDeleter interface {
	// DeleteIDs removes the documents with the given ids. Unknown ids
	// are ignored (idempotent); an empty slice is a no-op.
	DeleteIDs(ctx context.Context, ids []string) error
}

type Querier added in v0.59.0 

type Querier interface {
	Query(context.Context, *Query) (*SearchResponse, error)
}

Querier ranks precomputed vectors without invoking an embedding model.

type Query added in v0.59.0 

type Query struct {
	Space       SpaceID       `json:"space"`
	Values      []float64     `json:"values"`
	LexicalText string        `json:"lexical_text,omitzero"`
	Options     SearchOptions `json:"options"`
}

Query carries precomputed coordinates. LexicalText is only meaningful in native hybrid mode: stores must not silently run a different search.

func (Query) MarshalJSON added in v0.59.0 

func (q Query) MarshalJSON() ([]byte, error)

func (*Query) UnmarshalJSON added in v0.59.0 

func (q *Query) UnmarshalJSON(data []byte) error

func (*Query) ValidateFor added in v0.59.0 

func (q *Query) ValidateFor(space SpaceID) error

func (*Query) ValidateResponse added in v0.59.0 

func (q *Query) ValidateResponse(result *SearchResponse) error

ValidateResponse detects protocol violations; it never fixes incorrect membership by filtering a truncated top-K result.

type Record added in v0.59.0 

type Record struct {
	Document *document.Document `json:"document"`
	Values   []float64          `json:"values"`
}

Record associates an indexed document with already-computed coordinates.

func (Record) MarshalJSON added in v0.59.0 

func (r Record) MarshalJSON() ([]byte, error)

func (*Record) UnmarshalJSON added in v0.59.0 

func (r *Record) UnmarshalJSON(data []byte) error

func (Record) Validate added in v0.59.0 

func (r Record) Validate() error

type ScanBudget added in v0.59.0 

type ScanBudget struct {
	MaxRecords int
	MaxBytes   int64
	Timeout    time.Duration
}

ScanBudget is an explicit Host grant for a complete source read. Native query must never infer this grant from a filter. Each adapter documents what it measures as source bytes and the consistency of its scan.

func (ScanBudget) Begin added in v0.59.0 

func (s ScanBudget) Begin(ctx context.Context) (*ScanMeter, error)

Begin starts a fresh meter and deadline; it does not modify the grant.

func (ScanBudget) Validate added in v0.59.0 

func (s ScanBudget) Validate() error

type ScanMeter added in v0.59.0 

type ScanMeter struct {
	// contains filtered or unexported fields
}

ScanMeter tracks the host's remaining grant for one scan, including across concurrent native readers. The owner must Close it to release its deadline. Charge records one observed source record; adapters must bound native pages separately when their protocol can fetch more than one record at a time.

func (*ScanMeter) Charge added in v0.59.0 

func (s *ScanMeter) Charge(n int64) error

Charge checks the deadline and atomically charges one record and its native bytes before decoding, filtering or retaining it. No partial debit occurs.

func (*ScanMeter) ChargePage added in v0.59.0 

func (s *ScanMeter) ChargePage(records int, n int64) error

ChargePage debits native rows and bytes together. Streaming protocols can first debit bounded wire bytes, then the decoded page's record count.

func (*ScanMeter) Close added in v0.59.0 

func (s *ScanMeter) Close()

func (*ScanMeter) Context added in v0.59.0 

func (s *ScanMeter) Context() context.Context

func (*ScanMeter) RemainingBytes added in v0.59.0 

func (s *ScanMeter) RemainingBytes() int64

RemainingBytes is an observation, not a reservation. A sequential reader can cap its next response with it; ChargePage still performs the atomic debit.

type ScanRequest added in v0.60.0 

type ScanRequest struct {
	Budget ScanBudget
	Filter filter.Predicate
}

ScanRequest grants document enumeration, without a ranking vector or TopK. It never authorizes an implicit fallback from a native query. Backends must return no partial matches when the source exceeds the budget.

func (ScanRequest) Validate added in v0.60.0 

func (s ScanRequest) Validate() error

type Score 

type Score float64

Score is a provider-neutral, query-relative relevance value in [0, 1]. Scores preserve ordering but are not comparable across providers or search modes. Its JSON value must be a number.

func ScoreFromCosineDistance 

func ScoreFromCosineDistance(distance float64) Score

ScoreFromCosineDistance maps 1-cosine-similarity from [0, 2] to [0, 1].

func ScoreFromCosineSimilarity 

func ScoreFromCosineSimilarity(similarity float64) Score

ScoreFromCosineSimilarity maps cosine similarity from [-1, 1] to [0, 1].

func ScoreFromDistance 

func ScoreFromDistance(distance float64) Score

ScoreFromDistance maps a non-negative, unbounded distance to (0, 1], where zero is an exact match. An invalid native distance cannot establish a match.

func ScoreFromInnerProduct 

func ScoreFromInnerProduct(product float64) Score

ScoreFromInnerProduct maps an unbounded dot product monotonically into [0, 1]. Finite precision saturates the logistic function at both endpoints.

func ScoreFromNegativeInnerProductDistance 

func ScoreFromNegativeInnerProductDistance(distance float64) Score

ScoreFromNegativeInnerProductDistance maps a provider distance defined as the negative dot product into the similarity range.

func ScoreFromOneMinusInnerProductDistance 

func ScoreFromOneMinusInnerProductDistance(distance float64) Score

ScoreFromOneMinusInnerProductDistance maps a provider distance defined as 1-dot-product into the similarity range.

func ScoreFromValue 

func ScoreFromValue(value float64) Score

ScoreFromValue clamps a finite provider score to the common range. Non-finite input becomes NaN so result validation reports the contract breach.

func (Score) Float64 

func (s Score) Float64() float64

func (Score) MarshalJSON 

func (s Score) MarshalJSON() ([]byte, error)

func (*Score) UnmarshalJSON 

func (s *Score) UnmarshalJSON(data []byte) error

func (Score) Validate 

func (s Score) Validate() error

type SearchMode added in v0.12.0 

type SearchMode string

SearchMode selects how a store ranks a query. Semantic is the zero value, so it has exactly one spelling. Stores must reject an unsupported mode rather than silently substitute another. 

const (
	SearchModeSemantic SearchMode = ""
	SearchModeHybrid   SearchMode = "hybrid"
)

func (SearchMode) String added in v0.12.0 

func (s SearchMode) String() string

func (SearchMode) Valid added in v0.12.0 

func (s SearchMode) Valid() bool

type SearchOptions 

type SearchOptions struct {
	// TopK limits the result count. Zero uses DefaultTopK.
	TopK     int   `json:"top_k,omitzero"`
	MinScore Score `json:"min_score,omitzero"`
	// Filter is encoded as its canonical filter DSL string. An omitted or null
	// JSON filter means no predicate; a present string must parse successfully.
	Filter filter.Predicate `json:"filter,omitempty"`
	Mode   SearchMode       `json:"mode,omitempty"`
}

SearchOptions owns the policies applied to a relevance search. Semantic is the zero-value mode; hybrid combines semantic and lexical evidence.

func (SearchOptions) Clone added in v0.60.0 

func (s SearchOptions) Clone() SearchOptions

func (SearchOptions) MarshalJSON 

func (s SearchOptions) MarshalJSON() ([]byte, error)

func (SearchOptions) RequireMode added in v0.12.0 

func (s SearchOptions) RequireMode(supported ...SearchMode) error

RequireMode rejects a valid request mode before provider I/O when the store cannot implement it without changing its semantics.

func (SearchOptions) ResultLimit 

func (s SearchOptions) ResultLimit() int

func (*SearchOptions) UnmarshalJSON 

func (s *SearchOptions) UnmarshalJSON(data []byte) error

func (SearchOptions) Validate 

func (s SearchOptions) Validate() error

type SearchResponse 

type SearchResponse struct {
	Results []*SearchResult `json:"results"`
}

SearchResponse reports a completed search. JSON requires a results array; an explicit empty array reports no matches.

func NewSearchResponse 

func NewSearchResponse(results []*SearchResult) (*SearchResponse, error)

func (*SearchResponse) Documents 

func (s *SearchResponse) Documents() []*document.Document

func (*SearchResponse) First 

func (s *SearchResponse) First() *SearchResult

func (SearchResponse) MarshalJSON 

func (s SearchResponse) MarshalJSON() ([]byte, error)

func (*SearchResponse) UnmarshalJSON 

func (s *SearchResponse) UnmarshalJSON(data []byte) error

func (*SearchResponse) Validate 

func (s *SearchResponse) Validate() error

type SearchResult 

type SearchResult struct {
	Document *document.Document `json:"document"`
	Score    Score              `json:"score"`
}

func NewSearchResult 

func NewSearchResult(matched *document.Document, score Score) (*SearchResult, error)

func (SearchResult) MarshalJSON 

func (s SearchResult) MarshalJSON() ([]byte, error)

func (*SearchResult) UnmarshalJSON 

func (s *SearchResult) UnmarshalJSON(data []byte) error

func (*SearchResult) Validate 

func (s *SearchResult) Validate() error

type SpaceID added in v0.59.0 

type SpaceID string

SpaceID identifies the Host's immutable pairing of document and query encoders (including preprocessing and model revisions). Coordinates do not reveal this identity, and equal dimensions do not establish compatibility.

func (SpaceID) Require added in v0.59.0 

func (s SpaceID) Require(expected SpaceID) error

Require rejects requests for a different physical index binding.

func (SpaceID) StorageKey added in v0.59.0 

func (s SpaceID) StorageKey() string

StorageKey is a compact SQL-identifier-safe projection for physical schemas with length or character restrictions. Its input is the declared Host binding, not the model name, embedding coordinates, or executable code.

func (SpaceID) Validate added in v0.59.0 

func (s SpaceID) Validate() error

type Upserter added in v0.59.0 

type Upserter interface {
	Upsert(context.Context, *Batch) error
}

Upserter writes precomputed vectors without invoking an embedding model.

type WriteError added in v0.59.0 

type WriteError struct {
	// contains filtered or unexported fields
}

WriteError reports native acknowledgments received before an indexing operation failed. An acknowledgment is not a guarantee of search visibility. Unconfirmed IDs must not be replayed merely because the caller received an error. The error owns its outcome: adapter slices and reader copies never alias it.

func NewWriteError added in v0.60.0 

func NewWriteError(cause error, outcome WriteOutcome) *WriteError

func (*WriteError) Error added in v0.59.0 

func (w *WriteError) Error() string

func (*WriteError) Outcome added in v0.60.0 

func (w *WriteError) Outcome() WriteOutcome

Outcome returns a copy the caller owns.

func (*WriteError) Unwrap added in v0.59.0 

func (w *WriteError) Unwrap() error

type WriteOutcome added in v0.60.0 

type WriteOutcome struct {
	// Acknowledged contains IDs for which the backend acknowledged every item
	// in its write group.
	Acknowledged []string
	// Rejected contains IDs for which the backend explicitly rejected the write.
	Rejected []string
	// Unconfirmed is the attempted group whose individual outcomes are unknown.
	Unconfirmed []string
	// AcknowledgedCount is a usable aggregate acknowledgment for Unconfirmed,
	// when available. It does not identify particular records in that group.
	AcknowledgedCount *int
	// Unattempted contains IDs in groups the adapter never submitted.
	Unattempted []string
}

WriteOutcome partitions the IDs of one failed indexing operation by what the backend acknowledged. It is plain data an adapter assembles while writing; a WriteError owns its own copy.

func (WriteOutcome) Clone added in v0.60.0 

func (w WriteOutcome) Clone() WriteOutcome

Clone separates each partition from the source and from the other partitions, so appending to one outcome cannot rewrite another outcome.

Source Files

View all Source files

Directories

Path Synopsis
Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
 Package filter defines the stable metadata-filter expression vocabulary used by vector stores.
keyword
Package keyword derives exact, indexable string terms from canonical document metadata for native indexes that support keyword membership.
 Package keyword derives exact, indexable string terms from canonical document metadata for native indexes that support keyword membership.
Package inmemory implements exact vector search over process-local records.
 Package inmemory implements exact vector search over process-local records.
Package storetest provides conformance suites for vector stores and filters.
 Package storetest provides conformance suites for vector stores and filters.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.