external

package
v2.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 23, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package external: provider health aggregation (W8.4, REQ-VEC-002).

This file implements the EXPLICIT health/fallback semantics for vector providers. The design invariant is NO SILENT FALLBACK: when an operator configures an external provider (qdrant, pgvector), Cortex MUST NOT silently substitute sqlite_blob when that provider is unhealthy. Doing so would:

  1. Serve STALE results from a different index (the external index may have a different vector set, different model version, or be ahead/behind the local store).
  2. Violate the no-dual-source-of-truth invariant (ADR-05): SQLite is the authoritative observation store, but the EXTERNAL index is the authoritative VECTOR store for the configured deployment. Switching indices silently changes which vectors answer a query.
  3. Mask a configuration or operational problem the operator needs to see.

Instead, ResolveProviderHealth reports the REAL health (degraded/unhealthy) of the configured provider, and SelectForSearch returns the configured provider's index (even if unhealthy) so the caller sees the real error. The caller's policy (retry, alert, or operator-initiated switchover) decides what to do — the engine does not decide for it.

This is the opposite of a "graceful degradation" that hides problems. The spec scenario (REQ-VEC-002 edge: "external adapter outage with fallback") describes an OPERATOR-APPROVED fallback, not an automatic one. Automatic fallback is a separate, explicit code path (ApplyApprovedFallback) that the caller invokes only when it has decided to switch.

Package external: reindex implementation (W8.4 — replay external vector replica).

Reindex replays observations from the authoritative SQLite store into a configured external VectorIndex (qdrant, pgvector). It is the recovery / consistency path for an external vector replica that has drifted or is being initialized fresh. SQLite remains the single source of truth for observation data (ADR-05); the external index is a read-optional dense candidate source.

DESIGN — narrow port, no raw BLOB access:

The ReindexSource port exposes ONLY List (observation iteration) and GetEmbedding (single-vector retrieval via the existing VectorRepository contract). It deliberately does NOT reach into the SQLite vector BLOB table directly. When GetEmbedding returns ErrVectorSearchDisabled (the zero-CGO stub) or ErrNotFound, the reindex falls back to re-embedding via the provided EmbeddingProvider — regenerating fresh vectors rather than reading a source whose contract is insufficient.

IDEMPOTENCE: upsert is by observation ID (the external adapter's PK). Running Reindex multiple times produces the same replica state — no duplicates, no growth. The model-version namespace on each VectorPoint lets the adapter enforce dimension consistency (REQ-VEC-001).

BATCHING: observations are processed in batches of BatchSize. Each batch is upserted in one call to VectorIndex.Upsert, respecting the adapter's MaxBatchSize (the adapter chunks further if needed).

PROGRESS: an optional OnProgress callback is invoked after each batch with cumulative counts, enabling CLI/UI progress reporting.

FAILURE: target Upsert errors are returned explicitly (not silently swallowed). The result reflects counts up to the failure point. Individual observations that cannot be embedded (no vector, no provider, embed error) are counted as Skipped — the reindex is best-effort and continues.

Package external is the SERVER-TRACK VectorIndex factory (W8.4, ADR-05, REQ-VEC-001/002).

It is the ONLY place in the codebase that selects a concrete external vector adapter (qdrant, pgvector) based on config. The local composition path (internal/app) wires the sqlite_blob zero-CGO default DIRECTLY and MUST NOT import this package — the architecture gate (TestNoLocalToServerImport) bans it. This preserves REQ-FOUND-001: CGO_ENABLED=0 local build with zero external vector dependencies.

Provider selection is EXPLICIT and FAIL-CLOSED:

  • "" / "sqlite_blob" → sqlite_blob adapter over the caller's *sql.DB
  • "qdrant" → qdrant adapter over the official gRPC client
  • "pgvector" → pgvector adapter over the pgx pure-Go driver
  • "none" → nil (vector search disabled)
  • unknown → error (NO silent fallback to sqlite_blob)

There is NO graceful degradation to sqlite_blob when an EXTERNAL provider is CONFIGURED but unhealthy. Doing so would silently serve STALE results from a different index, violating the no-dual-source-of-truth invariant (REQ-VEC-002). The caller receives the unhealthy adapter (or a degraded health surface) and decides policy explicitly.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsProviderUnhealthy 

func IsProviderUnhealthy(err error) bool

IsProviderUnhealthy reports whether err is an *ErrProviderUnhealthy.

func NewVectorIndex 

func NewVectorIndex(ctx context.Context, cfg config.VectorConfig, in FactoryInput) (domain.VectorIndex, error)

NewVectorIndex selects and constructs the concrete domain.VectorIndex for the configured provider. It is the SERVER composition entry point.

SELECTION IS EXPLICIT AND FAIL-CLOSED:

  • cfg.Provider == "" or "sqlite_blob": the sqlite_blob adapter over in.DB. The local composition wires this directly; calling the factory here is allowed for symmetry/testing.
  • cfg.Provider == "qdrant": the qdrant adapter, configured from cfg.Qdrant + in.ModelInfo. The adapter owns its gRPC client.
  • cfg.Provider == "pgvector": the pgvector adapter, configured from cfg.Pgvector + in.ModelInfo. The adapter owns its pgxpool.
  • cfg.Provider == "none": returns (nil, nil) — vector search disabled by explicit operator choice. Callers gate on Health / nil-check.
  • cfg.Provider is anything else: returns an error. There is NO silent fallback to sqlite_blob — an unknown provider is a configuration error the operator must fix explicitly.

REQUIRED INPUT VALIDATION (fail-closed BEFORE constructing any adapter):

  • sqlite_blob: in.DB MUST be non-nil.
  • qdrant: in.ModelInfo.Dimension MUST be > 0 (the adapter needs it for collection creation and namespace enforcement). The QdrantConfig is already validated by config.Load, but the model dimension is runtime data not known at config-load time, so it is validated here.
  • pgvector: in.ModelInfo.Dimension MUST be > 0 (same reason).

SECRET SAFETY: the factory NEVER echoes APIKey or DSN passwords in errors. Per-adapter redaction (qdrant.redact, pgvector.redactDSN) is the defense-in- depth layer; the factory adds none of its own surface.

Types

type ErrProviderUnhealthy 

type ErrProviderUnhealthy struct {
	Provider string
	Status   string
	Message  string
}

ErrProviderUnhealthy is returned by callers that choose fail-closed when the configured provider reports unhealthy. It wraps the health surface so the caller can inspect status and message without re-querying.

func (*ErrProviderUnhealthy) Error 

func (e *ErrProviderUnhealthy) Error() string

type FactoryInput 

type FactoryInput struct {
	// DB is the shared SQLite *sql.DB. REQUIRED for the sqlite_blob provider
	// (the adapter wraps the existing concrete store). Ignored by external
	// providers (qdrant, pgvector) — they hold their own connection pools.
	DB *sql.DB

	// ModelInfo is the resolved embedding model identity. REQUIRED for the
	// qdrant and pgvector providers: the adapters stamp this on every upsert
	// for namespace enforcement (model-version namespacing, REQ-VEC-001
	// dim-mismatch pin) and use Dimension for collection/index sizing. For
	// sqlite_blob it is OPTIONAL — the adapter validates dimensions against
	// VectorPoint.ModelInfo at upsert time, so the factory does not need to
	// pre-declare it.
	ModelInfo domain.ModelInfo
}

FactoryInput carries the runtime handles the factory needs to construct a concrete adapter. Not every field is used by every provider — the per-field doc explains when each is required.

type ProviderHealth 

type ProviderHealth struct {
	// Provider is the configured provider name ("sqlite_blob", "qdrant",
	// "pgvector", "none").
	Provider string

	// IndexType is the adapter's declared Capabilities.IndexType. May differ
	// from Provider if a fallback was applied (it should not, under the
	// no-silent-fallback invariant).
	IndexType string

	// Status is the aggregated health: healthy, degraded, or unhealthy.
	Status string

	// Message is a human-readable diagnostic.
	Message string

	// FallbackUsed reports whether a fallback index was substituted. Under
	// the no-silent-fallback invariant this is ALWAYS false for a configured
	// external provider. It is true only when ApplyApprovedFallback has been
	// explicitly invoked by caller policy.
	FallbackUsed bool
}

ProviderHealth is the aggregated health surface for a configured vector provider. It carries the provider name, the adapter's declared IndexType, the current health status, and whether a fallback is in use.

func ApplyApprovedFallback 

func ApplyApprovedFallback(ctx context.Context, provider string, primary domain.VectorIndex, fallback domain.VectorIndex) (domain.VectorIndex, ProviderHealth)

ApplyApprovedFallback is the EXPLICIT fallback path. It is invoked by caller policy (CLI, operator command, monitoring hook) when the operator has decided to switch from an unhealthy external provider to the local sqlite_blob index. It is NEVER called automatically by the search path.

Returns the fallback index and a ProviderHealth marked FallbackUsed=true. If the fallback is nil, returns the primary (unhealthy) index with a fail-closed message.

func ResolveProviderHealth 

func ResolveProviderHealth(ctx context.Context, provider string, idx domain.VectorIndex) ProviderHealth

ResolveProviderHealth reads the health of the configured provider's adapter and returns the aggregated surface. It does NOT perform any fallback — it reports the REAL health of the REAL configured index.

For a nil index (provider=none), it reports unhealthy with a clear message.

func SelectForSearch 

func SelectForSearch(ctx context.Context, provider string, primary domain.VectorIndex, fallback domain.VectorIndex) (domain.VectorIndex, ProviderHealth)

SelectForSearch returns the VectorIndex to use for a search operation and its health surface. Under the NO SILENT FALLBACK invariant, it returns the CONFIGURED provider's index directly — even if that index is unhealthy.

The optional fallback parameter is the sqlite_blob local index. It is IGNORED for external providers (qdrant, pgvector) regardless of health, because substituting it silently would serve stale results from a different index. For the local provider (sqlite_blob), the primary IS the local index and fallback is unused.

Callers that want explicit, operator-approved fallback MUST call ApplyApprovedFallback separately — this function does not perform it.

type ReindexOptions 

type ReindexOptions struct {
	// BatchSize is the number of observations processed per Upsert call to
	// the target. Default 64. Larger batches reduce round-trips; smaller
	// batches reduce memory and per-batch failure blast radius.
	BatchSize int

	// OnProgress is invoked after each batch with cumulative counts. Optional;
	// nil means no progress reporting.
	OnProgress func(p ReindexProgress)
}

ReindexOptions configures a Reindex run.

type ReindexProgress 

type ReindexProgress struct {
	Processed  int // observations examined (including skipped)
	Upserted   int // vectors successfully upserted into the target
	ReEmbedded int // vectors regenerated via EmbeddingProvider (no source vector)
	Skipped    int // observations with no vector and no way to produce one
}

ReindexProgress is the cumulative state at a progress checkpoint.

type ReindexResult 

type ReindexResult struct {
	Total      int // total observations examined
	Upserted   int // vectors successfully upserted
	ReEmbedded int // vectors regenerated via EmbeddingProvider
	Skipped    int // observations skipped (no vector available)
	Batches    int // number of Upsert calls made to the target
}

ReindexResult is the final outcome of a Reindex run.

func Reindex 

func Reindex(ctx context.Context, src ReindexSource, provider domain.EmbeddingProvider, target domain.VectorIndex, opts ReindexOptions) (*ReindexResult, error)

Reindex replays observations from src into target, copying existing embeddings where available and re-embedding via provider where not.

The run is idempotent: the target's upsert is by observation ID, so running Reindex multiple times converges the replica to the source state without duplicates.

Returns a ReindexResult with final counts. If the target returns an Upsert error, it is returned wrapped (with counts visible in the result up to the failure batch).

type ReindexSource 

type ReindexSource interface {
	// List retrieves observations matching the filter, paginated by
	// Limit/Offset. Used by the reindex to iterate active observations in
	// deterministic ID-ascending order.
	List(ctx context.Context, filter domain.ObservationFilter) ([]*domain.Observation, error)

	// GetEmbedding retrieves an existing embedding for an observation.
	// Returns ErrNotFound if no embedding exists, or ErrVectorSearchDisabled
	// if the source's vector backend is unavailable (zero-CGO stub). Both
	// trigger re-embedding via the EmbeddingProvider.
	GetEmbedding(ctx context.Context, observationID int64) ([]float32, string, error)
}

ReindexSource is the narrow port for the authoritative observation store. It exposes ONLY the two operations the reindex needs:

  • List: page through active observations (for text + metadata)
  • GetEmbedding: retrieve a single existing embedding vector

It deliberately does NOT embed the full ObservationRepository (which would drag in Save/Update/Delete/etc. that the reindex never calls) and does NOT expose raw vector BLOB access or bulk vector reads. If the source cannot provide embeddings (zero-CGO stub returns ErrVectorSearchDisabled), the reindex falls back to the EmbeddingProvider. This keeps the port minimal and avoids coupling to SQLite-internal schema.

The concrete authoritative store satisfies this structurally: *sqlite.Store has List, and *sqlite.VectorStore has GetEmbedding. A composite wiring in the server composition root combines both into a single value satisfying this interface; test fakes implement it directly.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.