external

package
v2.3.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 2, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package external: provider health aggregation (W8.4, REQ-VEC-002).

This file implements the EXPLICIT health/fallback semantics for vector providers. The design invariant is NO SILENT FALLBACK: when an operator configures an external provider (qdrant, pgvector), Cortex MUST NOT silently substitute sqlite_blob when that provider is unhealthy. Doing so would:

  1. Serve STALE results from a different index (the external index may have a different vector set, different model version, or be ahead/behind the local store).
  2. Violate the no-dual-source-of-truth invariant (ADR-05): SQLite is the authoritative observation store, but the EXTERNAL index is the authoritative VECTOR store for the configured deployment. Switching indices silently changes which vectors answer a query.
  3. Mask a configuration or operational problem the operator needs to see.

Instead, ResolveProviderHealth reports the REAL health (degraded/unhealthy) of the configured provider, and SelectForSearch returns the configured provider's index (even if unhealthy) so the caller sees the real error. The caller's policy (retry, alert, or operator-initiated switchover) decides what to do — the engine does not decide for it.

This is the opposite of a "graceful degradation" that hides problems. The spec scenario (REQ-VEC-002 edge: "external adapter outage with fallback") describes an OPERATOR-APPROVED fallback, not an automatic one. Automatic fallback is a separate, explicit code path (ApplyApprovedFallback) that the caller invokes only when it has decided to switch.

Package external: reindex implementation (W8.4 — replay external vector replica).

Reindex replays observations from the authoritative SQLite store into a configured external VectorIndex (qdrant, pgvector). It is the recovery / consistency path for an external vector replica that has drifted or is being initialized fresh. SQLite remains the single source of truth for observation data (ADR-05); the external index is a read-optional dense candidate source.

DESIGN — narrow port, no raw BLOB access:

The ReindexSource port exposes ONLY List (observation iteration) and GetEmbedding (single-vector retrieval via the existing VectorRepository contract). It deliberately does NOT reach into the SQLite vector BLOB table directly. When GetEmbedding returns ErrVectorSearchDisabled (the zero-CGO stub) or ErrNotFound, the reindex falls back to re-embedding via the provided EmbeddingProvider — regenerating fresh vectors rather than reading a source whose contract is insufficient.

IDEMPOTENCE: upsert is by observation ID (the external adapter's PK). Running Reindex multiple times produces the same replica state — no duplicates, no growth. The model-version namespace on each VectorPoint lets the adapter enforce dimension consistency (REQ-VEC-001).

BATCHING: observations are processed in batches of BatchSize. Each batch is upserted in one call to VectorIndex.Upsert, respecting the adapter's MaxBatchSize (the adapter chunks further if needed).

PROGRESS: an optional OnProgress callback is invoked after each batch with cumulative counts, enabling CLI/UI progress reporting.

FAILURE: target Upsert errors are returned explicitly (not silently swallowed). The result reflects counts up to the failure point. Individual observations that cannot be embedded (no vector, no provider, embed error) are counted as Skipped — the reindex is best-effort and continues.

Package external is the SERVER-TRACK VectorIndex factory (W8.4, ADR-05, REQ-VEC-001/002).

It is the ONLY place in the codebase that selects a concrete external vector adapter (qdrant, pgvector) based on config. The local composition path (internal/app) wires the sqlite_blob zero-CGO default DIRECTLY and MUST NOT import this package — the architecture gate (TestNoLocalToServerImport) bans it. This preserves REQ-FOUND-001: CGO_ENABLED=0 local build with zero external vector dependencies.

Provider selection is EXPLICIT and FAIL-CLOSED:

  • "" / "sqlite_blob" → sqlite_blob adapter over the caller's *sql.DB
  • "qdrant" → qdrant adapter over the official gRPC client
  • "pgvector" → pgvector adapter over the pgx pure-Go driver
  • "none" → nil (vector search disabled)
  • unknown → error (NO silent fallback to sqlite_blob)

There is NO graceful degradation to sqlite_blob when an EXTERNAL provider is CONFIGURED but unhealthy. Doing so would silently serve STALE results from a different index, violating the no-dual-source-of-truth invariant (REQ-VEC-002). The caller receives the unhealthy adapter (or a degraded health surface) and decides policy explicitly.

Index

Constants

This section is empty.

Variables

View Source 
var ErrReindexCorpusChanged = errors.New("external: reindex corpus changed during run")
View Source 
var ErrServerScopedDeleteUnsupported = errors.New("external: server-scoped vector delete requires a scoped adapter contract")

Functions

func IsProviderUnhealthy 

func IsProviderUnhealthy(err error) bool

IsProviderUnhealthy reports whether err is an *ErrProviderUnhealthy.

func NewRequestScopedVectorIndex added in v2.3.0 

func NewRequestScopedVectorIndex(inner domain.VectorIndex) (domain.VectorIndex, error)

NewRequestScopedVectorIndex enforces the request's verified scope at the final vector boundary. It is used by SaaS server mode; a missing context fails closed rather than falling back to an unscoped collection query.

func NewServerScopedVectorIndex added in v2.3.0 

func NewServerScopedVectorIndex(inner domain.VectorIndex, tenantID, workspaceID string) (domain.VectorIndex, error)

func NewVectorIndex 

func NewVectorIndex(ctx context.Context, cfg config.VectorConfig, in FactoryInput) (domain.VectorIndex, error)

NewVectorIndex selects and constructs the concrete domain.VectorIndex for the configured provider. It is the SERVER composition entry point.

SELECTION IS EXPLICIT AND FAIL-CLOSED:

  • cfg.Provider == "" or "sqlite_blob": the sqlite_blob adapter over in.DB. The local composition wires this directly; calling the factory here is allowed for symmetry/testing.
  • cfg.Provider == "qdrant": the qdrant adapter, configured from cfg.Qdrant + in.ModelInfo. The adapter owns its gRPC client.
  • cfg.Provider == "pgvector": the pgvector adapter, configured from cfg.Pgvector + in.ModelInfo. The adapter owns its pgxpool.
  • cfg.Provider == "none": returns (nil, nil) — vector search disabled by explicit operator choice. Callers gate on Health / nil-check.
  • cfg.Provider is anything else: returns an error. There is NO silent fallback to sqlite_blob — an unknown provider is a configuration error the operator must fix explicitly.

REQUIRED INPUT VALIDATION (fail-closed BEFORE constructing any adapter):

  • sqlite_blob: in.DB MUST be non-nil.
  • qdrant: in.ModelInfo.Dimension MUST be > 0 (the adapter needs it for collection creation and namespace enforcement). The QdrantConfig is already validated by config.Load, but the model dimension is runtime data not known at config-load time, so it is validated here.
  • pgvector: in.ModelInfo.Dimension MUST be > 0 (same reason).

SECRET SAFETY: the factory NEVER echoes APIKey or DSN passwords in errors. Per-adapter redaction (qdrant.redact, pgvector.redactDSN) is the defense-in- depth layer; the factory adds none of its own surface.

func ReindexCode added in v2.3.0 

func ReindexCode(ctx context.Context, source CodeGraphSource, target CodeGraphTarget, project string) error

ReindexCode copies one trusted project corpus. The target replacement is checksum-idempotent; this coordinator rejects source scope drift first.

func WithRequestVectorScope added in v2.3.0 

func WithRequestVectorScope(ctx context.Context, tenantID, workspaceID string) context.Context

WithRequestVectorScope carries an already-authorized tenant/workspace boundary from server authentication to vector operations. It is package private at the storage boundary: callers cannot pass vector filters as an alternative authority source.

Types

type CodeGraphSource added in v2.3.0 

type CodeGraphSource interface {
	GetCodeGraph(context.Context, string) (*code.CodeGraph, error)
}

CodeGraphSource is the read-only side of an AST reindex operation.

type CodeGraphTarget added in v2.3.0 

type CodeGraphTarget interface {
	SaveCodeGraph(context.Context, *code.CodeGraph) error
}

CodeGraphTarget atomically replaces a scoped AST corpus.

type ErrProviderUnhealthy 

type ErrProviderUnhealthy struct {
	Provider string
	Status   string
	Message  string
}

ErrProviderUnhealthy is returned by callers that choose fail-closed when the configured provider reports unhealthy. It wraps the health surface so the caller can inspect status and message without re-querying.

func (*ErrProviderUnhealthy) Error 

func (e *ErrProviderUnhealthy) Error() string

type FactoryInput 

type FactoryInput struct {
	// DB is the shared SQLite *sql.DB. REQUIRED for the sqlite_blob provider
	// (the adapter wraps the existing concrete store). Ignored by external
	// providers (qdrant, pgvector) — they hold their own connection pools.
	DB *sql.DB

	// ModelInfo is the resolved embedding model identity. REQUIRED for the
	// qdrant and pgvector providers: the adapters stamp this on every upsert
	// for namespace enforcement (model-version namespacing, REQ-VEC-001
	// dim-mismatch pin) and use Dimension for collection/index sizing. For
	// sqlite_blob it is OPTIONAL — the adapter validates dimensions against
	// VectorPoint.ModelInfo at upsert time, so the factory does not need to
	// pre-declare it.
	ModelInfo domain.ModelInfo
}

FactoryInput carries the runtime handles the factory needs to construct a concrete adapter. Not every field is used by every provider — the per-field doc explains when each is required.

type ProviderHealth 

type ProviderHealth struct {
	// Provider is the configured provider name ("sqlite_blob", "qdrant",
	// "pgvector", "none").
	Provider string

	// IndexType is the adapter's declared Capabilities.IndexType. May differ
	// from Provider if a fallback was applied (it should not, under the
	// no-silent-fallback invariant).
	IndexType string

	// Status is the aggregated health: healthy, degraded, or unhealthy.
	Status string

	// Message is a human-readable diagnostic.
	Message string

	// FallbackUsed reports whether a fallback index was substituted. Under
	// the no-silent-fallback invariant this is ALWAYS false for a configured
	// external provider. It is true only when ApplyApprovedFallback has been
	// explicitly invoked by caller policy.
	FallbackUsed bool
}

ProviderHealth is the aggregated health surface for a configured vector provider. It carries the provider name, the adapter's declared IndexType, the current health status, and whether a fallback is in use.

func ApplyApprovedFallback 

func ApplyApprovedFallback(ctx context.Context, provider string, primary domain.VectorIndex, fallback domain.VectorIndex) (domain.VectorIndex, ProviderHealth)

ApplyApprovedFallback is the EXPLICIT fallback path. It is invoked by caller policy (CLI, operator command, monitoring hook) when the operator has decided to switch from an unhealthy external provider to the local sqlite_blob index. It is NEVER called automatically by the search path.

Returns the fallback index and a ProviderHealth marked FallbackUsed=true. If the fallback is nil, returns the primary (unhealthy) index with a fail-closed message.

func ResolveProviderHealth 

func ResolveProviderHealth(ctx context.Context, provider string, idx domain.VectorIndex) ProviderHealth

ResolveProviderHealth reads the health of the configured provider's adapter and returns the aggregated surface. It does NOT perform any fallback — it reports the REAL health of the REAL configured index.

For a nil index (provider=none), it reports unhealthy with a clear message.

func SelectForSearch 

func SelectForSearch(ctx context.Context, provider string, primary domain.VectorIndex, fallback domain.VectorIndex) (domain.VectorIndex, ProviderHealth)

SelectForSearch returns the VectorIndex to use for a search operation and its health surface. Under the NO SILENT FALLBACK invariant, it returns the CONFIGURED provider's index directly — even if that index is unhealthy.

The optional fallback parameter is the sqlite_blob local index. It is IGNORED for external providers (qdrant, pgvector) regardless of health, because substituting it silently would serve stale results from a different index. For the local provider (sqlite_blob), the primary IS the local index and fallback is unused.

Callers that want explicit, operator-approved fallback MUST call ApplyApprovedFallback separately — this function does not perform it.

type ReindexCorpusDescriptor added in v2.3.0 

type ReindexCorpusDescriptor struct {
	Generation string
	Checksum   string
	Count      int
}

ReindexCorpusDescriptor is a stable identity for the exact visible corpus. Generation is source-defined monotonic/change metadata; Checksum binds the rows and fields that can affect embeddings or vector metadata.

type ReindexOptions 

type ReindexOptions struct {
	// TenantID and WorkspaceID form the immutable trusted boundary applied to
	// every vector produced by this run. Both are required.
	TenantID    string
	WorkspaceID string
	// ProjectID is the canonical public UUID resolved by trusted server
	// composition. Project remains optional display metadata only.
	ProjectID string

	// BatchSize is the number of observations processed per Upsert call to
	// the target. Default 64. Larger batches reduce round-trips; smaller
	// batches reduce memory and per-batch failure blast radius.
	BatchSize int

	// OnProgress is invoked after each batch with cumulative counts. Optional;
	// nil means no progress reporting.
	OnProgress func(p ReindexProgress)
}

ReindexOptions configures a Reindex run.

type ReindexProgress 

type ReindexProgress struct {
	TenantID    string
	WorkspaceID string
	Processed   int // observations examined (including skipped)
	Upserted    int // vectors successfully upserted into the target
	ReEmbedded  int // vectors regenerated via EmbeddingProvider (no source vector)
	Skipped     int // observations with no vector and no way to produce one
}

ReindexProgress is the cumulative state at a progress checkpoint.

type ReindexResult 

type ReindexResult struct {
	TenantID    string
	WorkspaceID string
	Total       int // total observations examined
	Upserted    int // vectors successfully upserted
	ReEmbedded  int // vectors regenerated via EmbeddingProvider
	Skipped     int // observations skipped (no vector available)
	Batches     int // number of Upsert calls made to the target
}

ReindexResult is the final outcome of a Reindex run.

func Reindex 

func Reindex(ctx context.Context, src ReindexSource, provider domain.EmbeddingProvider, target domain.VectorIndex, opts ReindexOptions) (*ReindexResult, error)

Reindex replays observations from src into target, copying existing embeddings where available and re-embedding via provider where not.

The run is idempotent: the target's upsert is by observation ID, so running Reindex multiple times converges the replica to the source state without duplicates.

Returns a ReindexResult with final counts. If the target returns an Upsert error, it is returned wrapped (with counts visible in the result up to the failure batch).

type ReindexScope added in v2.3.0 

type ReindexScope struct {
	TenantID    string
	WorkspaceID string
	ProjectID   string
}

ReindexScope is the trusted corpus boundary resolved by server composition. ProjectID is the public project UUID; Project labels are display-only.

type ReindexSource 

type ReindexSource interface {
	// DescribeCorpus returns the identity of the active, principal-visible
	// corpus. Reindex compares it before preflight and after all target writes;
	// any insert, update, delete, or visibility change fails the run.
	DescribeCorpus(ctx context.Context, scope ReindexScope) (ReindexCorpusDescriptor, error)

	// List retrieves observations matching the filter, paginated by
	// Limit/Offset. Used by the reindex to iterate active observations in
	// deterministic ID-ascending order.
	List(ctx context.Context, scope ReindexScope, filter domain.ObservationFilter) ([]*domain.Observation, error)

	// Scope returns the immutable authoritative boundary for one observation.
	// Reindex verifies it before reading or upserting an embedding.
	Scope(ctx context.Context, observationID int64) (ReindexScope, error)

	// GetEmbedding retrieves an existing embedding for an observation.
	// Returns ErrNotFound if no embedding exists, or ErrVectorSearchDisabled
	// if the source's vector backend is unavailable (zero-CGO stub). Both
	// trigger re-embedding via the EmbeddingProvider.
	GetEmbedding(ctx context.Context, scope ReindexScope, observationID int64) ([]float32, string, error)
}

ReindexSource is the narrow port for the authoritative observation store. It exposes ONLY the two operations the reindex needs:

  • List: page through active observations (for text + metadata)
  • GetEmbedding: retrieve a single existing embedding vector

It deliberately does NOT embed the full ObservationRepository (which would drag in Save/Update/Delete/etc. that the reindex never calls) and does NOT expose raw vector BLOB access or bulk vector reads. If the source cannot provide embeddings (zero-CGO stub returns ErrVectorSearchDisabled), the reindex falls back to the EmbeddingProvider. This keeps the port minimal and avoids coupling to SQLite-internal schema.

The concrete authoritative store satisfies this structurally: *sqlite.Store has List, and *sqlite.VectorStore has GetEmbedding. A composite wiring in the server composition root combines both into a single value satisfying this interface; test fakes implement it directly.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.