Documentation
¶
Overview ¶
Package s3vectors exposes AWS S3 Vectors through the Core vector-store capability interfaces. S3 Vectors is a purpose-built, fully managed vector storage tier that lives next to regular S3 buckets — vectors live in a *vector bucket* under a typed *vector index*. Documents containing media are rejected before indexing I/O because this adapter persists document text and metadata only.
Requirements: an AWS account with S3 Vectors enabled (currently available in a subset of regions), a vector bucket + index provisioned out of band (ARM / Terraform / CloudFormation / SDK control plane), and an aws-sdk-go-v2 s3vectors client. The store does NOT create indexes — index dimensionality / metric / metadata schema are declared at index creation.
Distance metrics. S3 Vectors indexes are registered with one of `cosine` or `euclidean` at creation, and StoreConfig.DistanceMetric is what maps QueryVectors' raw distance into a higher-is-better score in [0, 1]. Because the response carries the distance and nothing that identifies the metric behind it, a mismatch would rescale every score and leave MinScore filtering by a threshold in the wrong scale, so NewStore reads the index's registered metric and refuses a configured value that disagrees with ErrIncompatibleIndex.
That read is GetIndex, and s3vectors:GetIndex is its own action: granting PutVectors and QueryVectors does not imply it, so an IAM policy scoped to the data plane alone has to add it at the index ARN. Dimensionality is not compared: this store declares none, and S3 Vectors rejects a wrong-width vector on write.
Filter visitor produces S3 Vectors' Mongo-flavored JSON filter document — `{"author": {"$eq": "Alice"}}`, `{"year": {"$gte": 2020}}`, `{"$and": [...]}`, `{"$not": {...}}`. Metadata keys are addressed flat (no nested-path support).
Batching. PutVectors caps at 500 vectors per request; the document batcher should produce shards smaller than that. The store passes each shard through as one PutVectors call.
Delete. S3 Vectors has no filter-based DeleteVectors, and QueryVectors is an approximate nearest-neighbor search that answers with up to topK candidates rather than every match — it cannot enumerate a filter. The store therefore walks the index with ListVectors, which is exhaustive and key-paginated, and decides membership with the shared client-side evaluation in github.com/Tangerg/scope/core/vectorstore/filter.Match. Listing completes before anything is deleted, so pagination never observes its own mutations. Filtered deletion needs s3vectors:GetVectors alongside s3vectors:ListVectors, because membership reads each vector's metadata.
Null tests emit $exists, which S3 Vectors documents as checking whether the key is present "regardless of the value that's stored". Filterable metadata holds strings, numbers, booleans and lists and cannot hold null, so an absent key is the only null-ish state.
LIKE is refused by the query filter, whose documented operator set has no pattern match. Filtered deletion is unaffected: it enumerates with ListVectors and decides membership with filter.Match, so the same filter that Search rejects will delete correctly.
Operation limits. A query ranks at most MaxTopK results and one QueryVectors response carries at most MaxResultsPerQueryPage of them, so Search follows the continuation token until the ranked run is complete rather than treating one page as the answer. PutVectors and DeleteVectors each carry at most MaxVectorsPerWrite vectors, so Index and DeleteIDs split at that bound instead of leaving a documented provider limit to the caller's batcher.
See https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-vectors.html. Metadata numbers are encoded with the SDK's arbitrary-precision Smithy number representation. The service remains responsible for its storage limits.
Index ¶
- Constants
- Variables
- type DistanceMetric
- type Store
- func (s *Store) DeleteIDs(ctx context.Context, ids []string) error
- func (s *Store) DeleteWhere(ctx context.Context, expr filter.Predicate) (err error)
- func (s *Store) Index(ctx context.Context, request *vectorstore.IndexRequest) (err error)
- func (s *Store) Search(ctx context.Context, req *vectorstore.SearchRequest) (response *vectorstore.SearchResponse, err error)
- type StoreConfig
- type VectorClient
Constants ¶
const ( // MaxTopK is the largest number of results one query may rank. MaxTopK = 10_000 // MaxResultsPerQueryPage is the largest number of hits one QueryVectors // response carries; the rest arrive under a continuation token. MaxResultsPerQueryPage = 100 // MaxVectorsPerWrite is the largest number of vectors one PutVectors or // DeleteVectors call may carry. MaxVectorsPerWrite = 500 )
Documented S3 Vectors operation limits. A request past any of them is rejected by the service, so the store either splits the work or refuses locally instead of sending one that cannot succeed.
const Provider = "S3Vectors"
Provider is the stable backend name for host-side attribution.
Variables ¶
var ErrIncompatibleIndex = errors.New("s3vectors: index is incompatible")
ErrIncompatibleIndex reports an index that is not the one the store was configured for: it was registered with a different distance metric.
Functions ¶
This section is empty.
Types ¶
type DistanceMetric ¶
type DistanceMetric string
DistanceMetric is the metric registered with the S3 Vectors index. The query response carries a raw distance and nothing that identifies the metric behind it, so the value is declared here and checked against the index at construction.
const ( DistanceCosine DistanceMetric = "cosine" DistanceEuclidean DistanceMetric = "euclidean" )
The metric is a closed vocabulary because score direction and threshold semantics depend on it: the same raw number means "near" under one metric and "far" under another, so an unrecognized value must be rejected rather than guessed.
func (DistanceMetric) String ¶
func (d DistanceMetric) String() string
func (DistanceMetric) Valid ¶
func (d DistanceMetric) Valid() bool
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store implements vector-store capabilities with Amazon S3 Vectors.
func NewStore ¶
func NewStore(ctx context.Context, config StoreConfig) (*Store, error)
NewStore confirms the index agrees with the configured metric during construction, which is why it takes a context: a store returned with the wrong metric would go on returning scores that are wrong rather than absent, and the misconfiguration is at wiring.
func (*Store) DeleteIDs ¶
DeleteIDs removes vectors by key. An empty slice is a no-op; unknown keys are ignored by S3 Vectors.
func (*Store) DeleteWhere ¶
DeleteWhere removes every document matching expr. S3 Vectors has no filter-based deletion, and QueryVectors is an approximate nearest-neighbor search that answers with up to topK candidates rather than every match, so it cannot enumerate a filter exhaustively. The store therefore lists the index with ListVectors — exhaustive and key-paginated — and decides membership with filter.Match, the same evaluation the in-memory store uses. Listing completes before anything is deleted so pagination never observes its own mutations. Requires s3vectors:GetVectors alongside s3vectors:ListVectors, because membership needs each vector's metadata. Implements vectorstore.FilterDeleter.
func (*Store) Index ¶
func (s *Store) Index(ctx context.Context, request *vectorstore.IndexRequest) (err error)
Index embeds documents and PUTs them, splitting each batch at MaxVectorsPerWrite. Leaving that to the caller's batcher would make a documented provider limit their problem, and a larger shard is a request certain to be rejected.
func (*Store) Search ¶
func (s *Store) Search(ctx context.Context, req *vectorstore.SearchRequest) (response *vectorstore.SearchResponse, err error)
Search runs QueryVectors with the configured filter.
type StoreConfig ¶
type StoreConfig struct {
// Client is the S3 Vectors API surface, normally an *s3vectors.Client.
// Required.
Client VectorClient
// VectorBucketName names the S3 Vectors bucket. Required.
VectorBucketName string
// IndexName names the vector index inside the bucket. Required.
IndexName string
// EmbeddingModel produces vectors for the documents. Required.
EmbeddingModel embedding.Model
// DocumentBatcher batches documents before upload. Required.
DocumentBatcher vectorstore.Batcher
// DistanceMetric records the metric the index was created with —
// the store uses this only to map the raw distance returned by
// QueryVectors into a `higher = more similar` [0, 1] score. The
// actual metric is set on the index out of band.
DistanceMetric DistanceMetric
}
StoreConfig contains configuration options for the AWS S3 Vectors vector store.
func (StoreConfig) Validate ¶
func (s StoreConfig) Validate() error
type VectorClient ¶ added in v0.16.0
type VectorClient interface {
GetIndex(context.Context, *s3vectors.GetIndexInput, ...func(*s3vectors.Options)) (*s3vectors.GetIndexOutput, error)
PutVectors(context.Context, *s3vectors.PutVectorsInput, ...func(*s3vectors.Options)) (*s3vectors.PutVectorsOutput, error)
QueryVectors(context.Context, *s3vectors.QueryVectorsInput, ...func(*s3vectors.Options)) (*s3vectors.QueryVectorsOutput, error)
ListVectors(context.Context, *s3vectors.ListVectorsInput, ...func(*s3vectors.Options)) (*s3vectors.ListVectorsOutput, error)
DeleteVectors(context.Context, *s3vectors.DeleteVectorsInput, ...func(*s3vectors.Options)) (*s3vectors.DeleteVectorsOutput, error)
}
VectorClient is the narrow S3 Vectors surface the store depends on. It is declared here rather than accepting the whole SDK client so the store's requirements stay visible and a caller can supply a decorated or recorded implementation. *s3vectors.Client satisfies it.
Source Files