Documentation
¶
Overview ¶
Package vespa exposes Yahoo Vespa's vector search through the Core vector-store capability interfaces. Documents are regular Vespa documents in a schema with id / content / embedding (tensor) fields plus any metadata attributes — reached over the HTTP Document / Search REST APIs. Documents containing media are rejected before indexing I/O because this adapter persists document text and metadata only.
Requirements: a Vespa application (Vespa Cloud or self-hosted) with a schema (.sd file) declaring the embedding tensor field and any metadata attributes the filter visitor will address. The store does NOT create the schema — Vespa schemas are part of the application package, not a runtime API.
Authentication. Talk to Vespa over HTTPS with mTLS (Vespa Cloud) or plain HTTP (self-hosted). Inject credentials by passing a configured http.Client via StoreConfig.HTTPClient.
Schema agreement is the caller's. StoreConfig.RankingProfile must rank by closeness, and the schema must declare the configured fields; both live in an application package this store cannot read, and Vespa's documentation does not establish that a status path answers on the container endpoint this store is configured with. So unlike its siblings, NewStore confirms nothing — a profile ranking by something else reports scores on a scale this store reads as closeness, and only the deployment can prevent that.
Search shape. The store issues a `nearestNeighbor` YQL search:
POST /search/
{
"yql": "select * from <schema> where {targetHits:K}nearestNeighbor(<vec_field>, q) and <filter>",
"hits": K,
"input.query(q)": {"values": [...]},
"ranking": "default"
}
The result's `relevance` is taken as-is — for the default cosine configuration this is already a [0, 1] similarity score.
Result ceiling. Vespa documents that "hits is capped at maxHits, default 400", and applies the cap by trimming the result rather than answering an error — a search for more would come back short with nothing to distinguish a capped result from an exhausted one. Search refuses a TopK above StoreConfig.MaxHits, which defaults to Vespa's own DefaultMaxHits; a deployment whose query profile raised the value says so there, the same way it declares its schema and rank profile. Filtered deletion pages within the same ceiling.
Query completeness. Vespa enables soft timeout by default and answers a partially evaluated query with 200 plus a degraded `root.coverage` report. Both search and filtered deletion require full coverage and no reported `root.errors`, because a shortened hit list is indistinguishable from a smaller result and would silently skip documents during deletion.
Filter visitor produces YQL where-clause fragments — `author contains "Alice"` (equality on string fields uses `contains`), `year >= 2020`, `tag in ("a", "b")`, `!(...)`, ` and ` / ` or `. Every filterable metadata key must exist as a top-level attribute in the schema.
Delete. Vespa selection expressions live under their own mini language; rather than translate the AST a second way, the store enumerates ids via a YQL search and then issues per-id deletes against the Document API (`DELETE /document/v1/<ns>/<schema>/docid/<id>`).
Null tests are refused. The query language reference states that "there is no way to query for a field that is not set / equals null or NaN" and suggests a magic sentinel value as a workaround, which this store will not invent on a caller's behalf.
Filterable keys. A metadata key is written into the query language as text, and that language cannot quote a field name, so a filter can only name a key that is a plain identifier. An indexed key is a string literal in the filter DSL, so without that limit a caller's key was read as syntax. A document whose metadata key is anything at all still stores and reads back fine; this is only about which keys a filter can name.
Index ¶
Constants ¶
const ( Provider = "Vespa" // DefaultContentField names the document field that stores the // raw text. DefaultContentField = "content" // DefaultEmbeddingField names the document field that stores the // vector tensor. DefaultEmbeddingField = "embedding" // DefaultIDField names the field used for the Scope document id. DefaultIDField = "doc_id" // DefaultQueryTensorName names the rank-profile query tensor. DefaultQueryTensorName = "q" DefaultMaxResponseBytes = int64(16 * 1024 * 1024) // DefaultMaxHits is the value Vespa documents for the query profile's // maxHits: "hits is capped at maxHits, default 400". The cap is applied // silently, so a search asking for more would come back short with nothing // to say it had been truncated. DefaultMaxHits = 400 )
Exported identifiers keep provider-owned names and defaults out of caller literals.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store implements vector-store capabilities through Vespa's REST API.
func NewStore ¶
func NewStore(_ context.Context, config StoreConfig) (*Store, error)
NewStore performs no I/O. Everything this store depends on — the schema, the fields, the rank profile's distance function — lives in an application package deployed out of band, and Vespa's documentation does not establish that a status path answers on the container endpoint this store is configured with, so there is nothing here it can confirm without guessing.
The context is still taken, because every store in this family is constructed the same way and a caller should not have to remember which backend happens to be checkable.
func (*Store) DeleteWhere ¶
func (*Store) Index ¶
func (s *Store) Index(ctx context.Context, request *vectorstore.IndexRequest) (err error)
Index embeds documents and writes them through the Vespa Document API.
func (*Store) Search ¶
func (s *Store) Search(ctx context.Context, req *vectorstore.SearchRequest) (response *vectorstore.SearchResponse, err error)
Search runs a nearestNeighbor YQL query.
type StoreConfig ¶
type StoreConfig struct {
// Endpoint is the Vespa container endpoint (Document API + search
// API), e.g. "https://my-app.aws-us-east-1c.z.vespa-app.cloud" or
// "http://localhost:8080". Required.
Endpoint string
// SchemaName is the document type name (matches the schema name
// in the .sd file). Required.
SchemaName string
// Namespace is the document-id namespace component. Required by
// the Vespa document-id grammar but commonly defaults to the
// schema name.
Namespace string
// EmbeddingField / ContentField / IDField name the well-known
// schema fields the store writes to. Optional defaults apply.
EmbeddingField string
ContentField string
IDField string
// QueryTensorName is the query tensor declared by RankingProfile. Optional:
// defaults to [DefaultQueryTensorName].
QueryTensorName string
// RankingProfile is the Vespa rank profile used for nearest-neighbor
// scoring. It must rank by closeness(field, <EmbeddingField>), whose
// relevance is in [0, 1]. Required: Vespa's built-in default profile uses
// nativeRank and does not represent vector similarity.
RankingProfile string
// EmbeddingModel produces vectors for the documents. Required.
EmbeddingModel embedding.Model
// DocumentBatcher batches documents before upload. Required.
DocumentBatcher vectorstore.Batcher
// HTTPClient lets callers override transport (timeouts,
// proxies, mTLS for Vespa Cloud). Optional: defaults to
// http.DefaultClient.
HTTPClient *http.Client
// MaxResponseBytes bounds every buffered HTTP response. Zero selects
// [DefaultMaxResponseBytes].
MaxResponseBytes int64
// MaxHits is the query profile's maxHits for this application. Optional:
// defaults to Vespa's own [DefaultMaxHits]. It belongs here for the same
// reason SchemaName and RankingProfile do — it is a fact about the deployed
// application package that this store cannot read, and Vespa caps hits
// against it without saying so.
MaxHits int
}
StoreConfig contains configuration options for the Vespa vector store. Vespa uses an HTTP REST surface; the store assumes the schema (the .sd file) is provisioned out of band — Vespa schema management is YAML/SDL and lives in the application package.
func (StoreConfig) Validate ¶
func (s StoreConfig) Validate() error
Source Files