vectorstore

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 1, 2026 License: GPL-3.0 Imports: 32 Imported by: 0

README

vectorstore

Interface-based vector storage package for contextd with multi-tenant isolation.

Overview

The vectorstore package provides an abstraction layer for vector storage operations with multiple provider implementations:

  • ChromemStore (default) - Embedded chromem-go storage with local embeddings
  • QdrantStore - External Qdrant service via gRPC

Both providers implement the Store interface and support multi-tenant isolation via payload filtering or filesystem isolation.

Multi-Tenant Isolation

The package provides three isolation modes for multi-tenant deployments:

Mode Description Use Case
PayloadIsolation Single collection with metadata filtering Default, recommended
FilesystemIsolation Separate database per tenant/project Legacy, migration path available
NoIsolation No tenant filtering Testing only
PayloadIsolation (Default)

All documents stored in shared collections with automatic tenant filtering:

import "github.com/fyrsmithlabs/contextd/internal/vectorstore"

// Configure with PayloadIsolation (default if not specified)
config := vectorstore.ChromemConfig{
    Path:              "/path/to/data",
    DefaultCollection: "memories",
    VectorSize:        384,
    Isolation:         vectorstore.NewPayloadIsolation(), // Optional - this is the default
}

store, err := vectorstore.NewChromemStore(config, embedder, logger)
if err != nil {
    return err
}
defer store.Close()

// REQUIRED: Add tenant context to all operations
ctx := vectorstore.ContextWithTenant(ctx, &vectorstore.TenantInfo{
    TenantID:  "org-123",      // Required
    TeamID:    "platform",     // Optional
    ProjectID: "contextd",     // Optional
})

// Documents automatically tagged with tenant metadata
docs := []vectorstore.Document{{ID: "doc1", Content: "example"}}
ids, err := store.AddDocuments(ctx, docs)

// Searches automatically filtered by tenant
results, err := store.Search(ctx, "query", 10)
Security Model
Aspect Behavior
Missing tenant context Returns ErrMissingTenant (fail-closed)
User filters with tenant fields Rejected with ErrTenantFilterInUserFilters
Document metadata Tenant fields always overwritten from context
Default isolation PayloadIsolation for production safety
Threat Model

The isolation system defends against:

  1. Cross-Tenant Data Access - Attacker injects tenant_id in query filters

    • Defense: ApplyTenantFilters() rejects user-provided tenant fields

  2. Metadata Poisoning - Attacker sets tenant_id in document metadata

    • Defense: InjectMetadata() always overwrites tenant fields from context

  3. Context Bypass - Code executes without tenant context

    • Defense: Fail-closed behavior returns error, not empty results

  4. Race Conditions - Isolation mode changed during operation

    • Defense: Config-based isolation is immutable after construction

Migration Guide

From FilesystemIsolation to PayloadIsolation

If you're using StoreProvider with database-per-project isolation:

// BEFORE: FilesystemIsolation (database per project)
provider, _ := vectorstore.NewChromemStoreProvider(providerConfig, embedder, logger)
store, _ := provider.GetProjectStore(ctx, "platform", "contextd")
// Each project gets separate database files

// AFTER: PayloadIsolation (shared database, metadata filtering)
config := vectorstore.ChromemConfig{
    Path:              "/path/to/shared/data",
    DefaultCollection: "memories",
    VectorSize:        384,
    Isolation:         vectorstore.NewPayloadIsolation(),
}
store, _ := vectorstore.NewChromemStore(config, embedder, logger)

// Add tenant context to all operations
ctx = vectorstore.ContextWithTenant(ctx, &vectorstore.TenantInfo{
    TenantID:  "org-123",
    TeamID:    "platform",
    ProjectID: "contextd",
})

Migration steps:

  1. Export data from per-project databases
  2. Create new shared database with PayloadIsolation
  3. Import data with tenant metadata added
  4. Update all callers to use ContextWithTenant()
  5. Verify tenant filtering works correctly

No data format changes - only metadata is added to documents.

Provider Selection

vectorstore:
  provider: chromem  # "chromem" (default) or "qdrant"
Provider Use Case Dependencies
Chromem Local dev, simple setups None (embedded)
Qdrant Production, high scale External Qdrant + embedder

Collection Naming Convention

Collections follow a hierarchical naming pattern:

Scope Pattern Example
Organization org_{type} org_memories
Team {team}_{type} platform_memories
Project {team}_{project}_{type} platform_contextd_memories

Note: With PayloadIsolation, you can use a single collection (e.g., memories) and rely on metadata filtering for tenant separation. Collection naming is optional.

Interface

The Store interface provides:

  • Document Operations: AddDocuments, DeleteDocuments, DeleteDocumentsFromCollection
  • Search Operations: Search, SearchWithFilters, SearchInCollection, ExactSearch
  • Collection Management: CreateCollection, DeleteCollection, CollectionExists, ListCollections, GetCollectionInfo
  • Isolation: SetIsolationMode (deprecated), IsolationMode
  • Resource Management: Close

Usage Examples

ChromemStore with PayloadIsolation
import "github.com/fyrsmithlabs/contextd/internal/vectorstore"

// Configure store with isolation
config := vectorstore.ChromemConfig{
    Path:              "/data/vectorstore",
    DefaultCollection: "memories",
    VectorSize:        384,
    Compress:          true,
    Isolation:         vectorstore.NewPayloadIsolation(),
}

store, err := vectorstore.NewChromemStore(config, embedder, logger)
if err != nil {
    return err
}
defer store.Close()

// Create tenant context (required for all operations)
ctx := vectorstore.ContextWithTenant(context.Background(), &vectorstore.TenantInfo{
    TenantID:  "org-123",
    TeamID:    "platform",
    ProjectID: "contextd",
})

// Add documents (tenant metadata injected automatically)
docs := []vectorstore.Document{
    {
        ID:      "mem-1",
        Content: "User prefers dark mode",
        Metadata: map[string]interface{}{
            "category": "preference",
        },
    },
}
ids, err := store.AddDocuments(ctx, docs)

// Search (tenant filter injected automatically)
results, err := store.Search(ctx, "user preferences", 10)

// Search with additional filters
results, err := store.SearchWithFilters(ctx, "preferences", 10,
    map[string]interface{}{"category": "preference"})
QdrantStore with PayloadIsolation
config := vectorstore.QdrantConfig{
    Host:           "localhost",
    Port:           6334,
    CollectionName: "memories",
    VectorSize:     384,
    UseTLS:         false,
    Isolation:      vectorstore.NewPayloadIsolation(),
}

store, err := vectorstore.NewQdrantStore(config, embedder)
if err != nil {
    return err
}
defer store.Close()

// Same tenant context pattern
ctx := vectorstore.ContextWithTenant(ctx, &vectorstore.TenantInfo{
    TenantID: "org-123",
})

results, err := store.Search(ctx, "query", 10)
Testing with NoIsolation
// For tests where tenant isolation is not relevant
config := vectorstore.ChromemConfig{
    Path:              t.TempDir(),
    DefaultCollection: "test",
    VectorSize:        384,
    Isolation:         vectorstore.NewNoIsolation(), // WARNING: Testing only!
}

store, _ := vectorstore.NewChromemStore(config, embedder, nil)
// No tenant context required
results, _ := store.Search(context.Background(), "query", 10)

Configuration

ChromemConfig
type ChromemConfig struct {
    Path              string         // Database directory path
    Compress          bool           // Enable compression
    DefaultCollection string         // Default collection name
    VectorSize        int            // Embedding dimensions (e.g., 384)
    Isolation         IsolationMode  // PayloadIsolation (default), FilesystemIsolation, or NoIsolation
}
QdrantConfig
type QdrantConfig struct {
    Host                    string         // Qdrant server hostname
    Port                    int            // gRPC port (default: 6334)
    CollectionName          string         // Default collection
    VectorSize              uint64         // Embedding dimensions
    Distance                qdrant.Distance // Similarity metric (default: Cosine)
    UseTLS                  bool           // Enable TLS
    MaxRetries              int            // Retry attempts (default: 3)
    RetryBackoff            time.Duration  // Initial backoff (default: 1s)
    MaxMessageSize          int            // gRPC message size (default: 50MB)
    CircuitBreakerThreshold int            // Failures before open (default: 5)
    Isolation               IsolationMode  // PayloadIsolation (default), etc.
}

Security

Input Validation
  • Collection names: ^[a-z0-9_]{1,64}$ (prevents path traversal)
  • Query length: Max 10,000 characters (QdrantStore)
  • Result limits: Capped at collection size or 10,000
Tenant Isolation
  • Fail-closed: Missing context returns error, never empty results
  • Filter injection blocked: User cannot provide tenant_id/team_id/project_id
  • Metadata enforced: Tenant fields always set from authenticated context

Testing

# Unit tests only
go test ./internal/vectorstore/... -short

# Integration tests (requires Qdrant on localhost:6334)
go test ./internal/vectorstore/...

Dependencies

ChromemStore
  • github.com/philippgille/chromem-go - Embedded vector store
  • go.opentelemetry.io/otel - Observability
QdrantStore
  • github.com/qdrant/go-client - Official Qdrant gRPC client
  • github.com/google/uuid - UUID generation
  • go.opentelemetry.io/otel - Observability

Documentation

Overview

Package vectorstore provides vector storage implementations.

Package vectorstore provides periodic background health scanning.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage abstraction with multi-tenant isolation.

The package offers a unified interface for vector storage operations with multiple provider implementations (chromem embedded, Qdrant external). It enables semantic search over embeddings while enforcing strict tenant isolation for multi-tenant deployments.

Security

The package implements defense-in-depth security with fail-closed behavior:

  • Multi-tenant isolation via payload filtering (default) or filesystem separation
  • Mandatory tenant context for all operations (ErrMissingTenant if missing)
  • Filter injection prevention (rejects user-provided tenant_id/team_id/project_id)
  • Metadata poisoning protection (tenant fields always overwritten from context)
  • Collection name validation (prevents path traversal)
  • Query length limits (10,000 chars max)
  • Result limits (capped at collection size or 10,000)

Usage

Basic usage with PayloadIsolation (default): 

import "github.com/fyrsmithlabs/contextd/internal/vectorstore"

// Configure store (PayloadIsolation is default)
config := vectorstore.ChromemConfig{
    Path:              "/data/vectorstore",
    DefaultCollection: "memories",
    VectorSize:        384,
    Compress:          true,
}

store, err := vectorstore.NewChromemStore(config, embedder, logger)
if err != nil {
    return err
}
defer store.Close()

// REQUIRED: Add tenant context to all operations
ctx := vectorstore.ContextWithTenant(ctx, &vectorstore.TenantInfo{
    TenantID:  "org-123",      // Required
    TeamID:    "platform",     // Optional
    ProjectID: "contextd",     // Optional
})

// Documents automatically tagged with tenant metadata
docs := []vectorstore.Document{
    {
        ID:      "mem-1",
        Content: "User prefers dark mode",
        Metadata: map[string]interface{}{"category": "preference"},
    },
}
ids, err := store.AddDocuments(ctx, docs)

// Searches automatically filtered by tenant
results, err := store.Search(ctx, "user preferences", 10)

Isolation Modes

The package supports three isolation strategies:

PayloadIsolation (default, recommended):

  • All documents in shared collections with metadata filtering
  • tenant_id, team_id, project_id stored as document metadata
  • All queries automatically filtered by tenant context
  • Fail-closed: missing tenant context returns error, not empty results

FilesystemIsolation (legacy):

  • Separate database per tenant/team/project
  • Physical filesystem isolation provides security boundary
  • Use StoreProvider to manage per-tenant stores
  • Migration path available to PayloadIsolation

NoIsolation (testing only):

  • No tenant filtering or validation
  • WARNING: Provides no security guarantees
  • Use only in tests where isolation is not relevant

Provider Selection

The package supports multiple vector store providers:

ChromemStore (default):

  • Embedded chromem-go storage (no external dependencies)
  • Local ONNX embeddings via FastEmbed
  • Perfect for local dev and simple setups
  • Just works: brew install contextd

QdrantStore (optional):

  • External Qdrant service via gRPC
  • Requires external Qdrant server + embedder
  • Recommended for production, high scale

Provider selection via config: 

vectorstore:
  provider: chromem  # "chromem" (default) or "qdrant"

Collection Naming Convention

Collections follow a hierarchical naming pattern (with PayloadIsolation, these are optional - use a single collection with metadata filtering):

  • Organization: org_{type} (e.g., org_memories)
  • Team: {team}_{type} (e.g., platform_memories)
  • Project: {team}_{project}_{type} (e.g., platform_contextd_memories)

Performance

Current implementation optimizations:

  • Batch embedding generation for multiple documents
  • Concurrent search operations across collections
  • Optional compression for storage efficiency
  • HNSW index for fast approximate nearest neighbor search

Future optimization opportunities:

  • Connection pooling for Qdrant gRPC
  • Result caching with TTL
  • Adaptive batch sizing based on load

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage implementations.

Package vectorstore defines the interface for vector storage operations.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage with metrics instrumentation.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage implementations.

Package vectorstore provides startup validation for metadata integrity.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage implementations.

Package vectorstore provides vector storage implementations.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrCollectionNotFound is returned when a collection does not exist.
	ErrCollectionNotFound = errors.New("collection not found")

	// ErrCollectionExists is returned when attempting to create an existing collection.
	ErrCollectionExists = errors.New("collection already exists")

	// ErrInvalidConfig indicates invalid configuration.
	ErrInvalidConfig = errors.New("invalid configuration")

	// ErrEmptyDocuments indicates empty or nil documents.
	ErrEmptyDocuments = errors.New("empty or nil documents")

	// ErrConnectionFailed indicates gRPC connection issues.
	ErrConnectionFailed = errors.New("failed to connect to Qdrant")

	// ErrEmbeddingFailed indicates embedding generation failure.
	ErrEmbeddingFailed = errors.New("failed to generate embeddings")

	// ErrInvalidCollectionName indicates collection name validation failure.
	ErrInvalidCollectionName = errors.New("invalid collection name")
)

Sentinel errors for vector store operations.

View Source 
var (
	// ErrMissingTenant is returned when tenant info is missing from context.
	// This triggers "fail closed" behavior - no empty results, just errors.
	ErrMissingTenant = errors.New("tenant info missing from context")

	// ErrInvalidTenant is returned when tenant identifier is invalid.
	ErrInvalidTenant = errors.New("invalid tenant identifier")
)

Tenant isolation error types - fail closed security model.

View Source 
var ErrTenantFilterInUserFilters = errors.New("user filters cannot contain tenant fields")

ErrTenantFilterInUserFilters indicates user tried to inject tenant fields.

View Source 
var OptionalTenantFields = []string{"team_id", "project_id"}

OptionalTenantFields are additional scope fields that may be present.

View Source 
var RequiredTenantFields = []string{"tenant_id"}

RequiredTenantFields are the fields that must be present for tenant isolation.

Functions

func ApplyTenantFilters added in v0.3.0 

func ApplyTenantFilters(userFilters, tenantFilters map[string]interface{}) (map[string]interface{}, error)

ApplyTenantFilters merges user filters with tenant filters, enforcing security.

This function ensures tenant filters (from isolation layer) always win and rejects attempts to inject tenant fields via user filters.

Parameters:

  • userFilters: User-provided query filters (may be nil)
  • tenantFilters: Tenant filters from isolation layer (may be nil)

Returns error if userFilters contains tenant_id, team_id, or project_id.

func CalculateMRR added in v0.4.0 

func CalculateMRR(results []SearchResult, relevantDocs []string) float64

CalculateMRR computes Mean Reciprocal Rank.

MRR measures where the first relevant document appears in the results. It's particularly useful for tasks where users only need one good result.

Parameters:

  • results: Search results ordered by the system
  • relevantDocs: Set of document IDs that are considered relevant

Returns:

  • MRR score in range [0.0, 1.0]
  • 1.0 if first result is relevant
  • 1/position for first relevant result at position (1-indexed)
  • 0.0 if no relevant documents found

Example:

  • First result relevant: MRR = 1.0
  • Third result relevant: MRR = 1/3 = 0.333
  • No relevant results: MRR = 0.0

func CalculateNDCG added in v0.4.0 

func CalculateNDCG(results []SearchResult, expectedRanking []string, k int) float64

CalculateNDCG computes Normalized Discounted Cumulative Gain at rank K.

NDCG measures ranking quality by comparing actual ranking to ideal ranking. It considers both relevance and position, with higher weight for top positions.

Parameters:

  • results: Search results ordered by the system
  • expectedRanking: Ideal order of document IDs by relevance (most relevant first)
  • k: Number of top results to consider

Returns:

  • NDCG score in range [0.0, 1.0] where 1.0 is perfect ranking
  • Returns 0.0 if results are empty or k <= 0

Algorithm:

  • DCG (Discounted Cumulative Gain) = sum of (relevance_score / log2(position+1))
  • IDCG (Ideal DCG) = DCG with perfect ranking
  • NDCG = DCG / IDCG (normalized to handle different result set sizes)

func CalculatePrecisionAtK added in v0.4.0 

func CalculatePrecisionAtK(results []SearchResult, relevantDocs []string, k int) float64

CalculatePrecisionAtK computes Precision at rank K.

Precision@K measures the proportion of relevant documents in the top K results. It answers: "Of the K documents I retrieved, how many are actually relevant?"

Parameters:

  • results: Search results ordered by the system
  • relevantDocs: Set of document IDs that are considered relevant
  • k: Number of top results to consider

Returns:

  • Precision score in range [0.0, 1.0]
  • 1.0 if all top K results are relevant
  • 0.0 if no relevant documents in top K
  • Returns 0.0 if results are empty or k <= 0

Example:

  • 3 relevant docs in top 5: P@5 = 3/5 = 0.6
  • 5 relevant docs in top 5: P@5 = 5/5 = 1.0
  • 0 relevant docs in top 5: P@5 = 0/5 = 0.0

func ContextWithTenant added in v0.3.0 

func ContextWithTenant(ctx context.Context, tenant *TenantInfo) context.Context

ContextWithTenant adds TenantInfo to a context.

func HasTenant added in v0.3.0 

func HasTenant(ctx context.Context) bool

HasTenant checks if TenantInfo is present in context without error.

func IsTransientError 

func IsTransientError(err error) bool

IsTransientError checks if an error is transient (should retry). Returns true for network timeouts, temporary unavailability. Returns false for invalid config, not found, permission denied.

func MergeFilters deprecated added in v0.3.0 

func MergeFilters(base, override map[string]interface{}) map[string]interface{}

MergeFilters combines two filter maps, with override taking precedence.

Deprecated: Use ApplyTenantFilters for tenant-aware merging with security validation. This function is kept for backward compatibility but does not enforce tenant security.

func NewResilientChromemDB added in v0.4.0 

func NewResilientChromemDB(path string, compress bool, logger *zap.Logger) (*chromem.DB, error)

NewResilientChromemDB creates a chromem DB with graceful degradation for corrupt collections. If a collection is missing its metadata file, it will be quarantined and the DB will load successfully.

func RecordHealthCheckResult added in v0.4.0 

func RecordHealthCheckResult(success bool)

RecordHealthCheckResult records whether a health check succeeded or failed.

TODO(#94): Implement health check metrics when health monitoring is added.

Expected implementation:

  • Metric: contextd.vectorstore.health_check_total{status}
  • Labels: status="success" or status="failure"
  • Use case: Alert on health_check_total{status="failure"} > 0 for 5m

Interface contract:

  • success=true: Vectorstore responded within timeout
  • success=false: Connection failed, timeout, or unhealthy response

func RecordQuarantineResult added in v0.4.0 

func RecordQuarantineResult(success bool)

RecordQuarantineResult records whether a quarantine operation succeeded or failed.

TODO(#94): Implement when document quarantine feature is added.

Expected implementation:

  • Metric: contextd.vectorstore.quarantine_total{status,reason}
  • Labels: status="success"/"failure", reason="metadata_mismatch"/"tenant_violation"
  • Use case: Track documents flagged for manual review due to inconsistencies

Interface contract:

  • success=true: Document successfully quarantined for review
  • success=false: Quarantine operation failed (log details separately)

func UpdateHealthMetrics added in v0.4.0 

func UpdateHealthMetrics(health *MetadataHealth)

UpdateHealthMetrics updates metrics based on health check results.

TODO(#94): Implement when MetadataHealth tracking is production-ready.

Expected implementation:

  • contextd.vectorstore.collection_size{collection}: Document count per collection
  • contextd.vectorstore.tenant_count: Unique tenants in the store
  • contextd.vectorstore.storage_bytes: Total storage utilization

Interface contract:

  • Called periodically (e.g., every 30s) with latest health snapshot
  • MetadataHealth struct provides collection stats, tenant count, storage usage

func ValidateCollectionName 

func ValidateCollectionName(name string) error

ValidateCollectionName validates a collection name against security rules. Pattern: ^[a-z0-9_]{1,64}$ Rejects: uppercase, special chars, path traversal, spaces.

func ValidateFilterHasTenant added in v0.3.0 

func ValidateFilterHasTenant(filters map[string]interface{}) error

ValidateFilterHasTenant checks that a filter map contains required tenant fields. Returns ErrMissingTenant if tenant_id is not present or not a string. Returns ErrInvalidTenant if tenant_id is empty.

Types

type BackgroundScanner added in v0.4.0 

type BackgroundScanner struct {
	// contains filtered or unexported fields
}

BackgroundScanner performs periodic health checks in the background.

func NewBackgroundScanner added in v0.4.0 

func NewBackgroundScanner(checker *MetadataHealthChecker, config *BackgroundScannerConfig, logger *zap.Logger) *BackgroundScanner

NewBackgroundScanner creates a new background health scanner.

func (*BackgroundScanner) IsRunning added in v0.4.0 

func (s *BackgroundScanner) IsRunning() bool

IsRunning returns true if the scanner is actively running.

func (*BackgroundScanner) LastError added in v0.4.0 

func (s *BackgroundScanner) LastError() error

LastError returns the most recent health check error (if any).

func (*BackgroundScanner) LastHealth added in v0.4.0 

func (s *BackgroundScanner) LastHealth() *MetadataHealth

LastHealth returns the most recent health check result.

func (*BackgroundScanner) Start added in v0.4.0 

func (s *BackgroundScanner) Start(ctx context.Context)

Start begins periodic health scanning in the background. Returns immediately; scanning happens in a goroutine.

func (*BackgroundScanner) Stop added in v0.4.0 

func (s *BackgroundScanner) Stop()

Stop halts the background scanner and waits for it to finish.

type BackgroundScannerConfig added in v0.4.0 

type BackgroundScannerConfig struct {
	// Interval between health scans. Default: 5 minutes.
	Interval time.Duration

	// OnDegraded is called when degraded state is detected.
	// Receives the health result for alerting/notification.
	OnDegraded func(health *MetadataHealth)

	// OnRecovered is called when system recovers from degraded state.
	OnRecovered func(health *MetadataHealth)

	// OnError is called when health check fails.
	OnError func(err error)
}

BackgroundScannerConfig configures periodic health scanning.

type ChromemConfig added in v0.3.0 

type ChromemConfig struct {
	// Path is the directory for persistent storage.
	// Default: "~/.config/contextd/vectorstore"
	Path string

	// Compress enables gzip compression for stored data.
	// Note: This defaults to false (Go zero value). Set explicitly if compression is desired.
	Compress bool

	// DefaultCollection is the default collection name.
	// Default: "contextd_default"
	DefaultCollection string

	// VectorSize is the expected embedding dimension.
	// Must match the embedder's output dimension.
	// Default: 384 (for FastEmbed bge-small-en-v1.5)
	VectorSize int

	// Isolation is the tenant isolation mode.
	// Default: PayloadIsolation for fail-closed security.
	// Set at construction time; immutable afterward to prevent race conditions.
	Isolation IsolationMode
}

ChromemConfig holds configuration for chromem-go embedded vector database.

func (*ChromemConfig) ApplyDefaults added in v0.3.0 

func (c *ChromemConfig) ApplyDefaults()

ApplyDefaults sets default values for unset fields.

func (*ChromemConfig) Validate added in v0.3.0 

func (c *ChromemConfig) Validate() error

Validate validates the configuration.

type ChromemStore added in v0.3.0 

type ChromemStore struct {
	// contains filtered or unexported fields
}

ChromemStore implements the Store interface using chromem-go.

chromem-go is an embeddable vector database with zero third-party dependencies. It provides in-memory storage with optional persistence to gob files.

Key features:

  • Pure Go, no CGO required
  • No external database service needed
  • Fast similarity search (1000 docs in 0.3ms)
  • Automatic persistence to disk
  • Tenant isolation via payload filtering or filesystem isolation

func NewChromemStore added in v0.3.0 

func NewChromemStore(config ChromemConfig, embedder Embedder, logger *zap.Logger) (*ChromemStore, error)

NewChromemStore creates a new ChromemStore with the given configuration.

func (*ChromemStore) AddDocuments added in v0.3.0 

func (s *ChromemStore) AddDocuments(ctx context.Context, docs []Document) ([]string, error)

AddDocuments adds documents to the vector store. If isolation mode is set, tenant metadata is automatically injected.

func (*ChromemStore) Close added in v0.3.0 

func (s *ChromemStore) Close() error

Close closes the ChromemStore. Note: chromem-go handles persistence automatically, no explicit close needed.

func (*ChromemStore) CollectionExists added in v0.3.0 

func (s *ChromemStore) CollectionExists(ctx context.Context, collectionName string) (bool, error)

CollectionExists checks if a collection exists.

func (*ChromemStore) CreateCollection added in v0.3.0 

func (s *ChromemStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error

CreateCollection creates a new collection with the specified configuration.

func (*ChromemStore) DeleteCollection added in v0.3.0 

func (s *ChromemStore) DeleteCollection(ctx context.Context, collectionName string) error

DeleteCollection deletes a collection and all its documents.

func (*ChromemStore) DeleteDocuments added in v0.3.0 

func (s *ChromemStore) DeleteDocuments(ctx context.Context, ids []string) error

DeleteDocuments deletes documents by their IDs from the default collection.

func (*ChromemStore) DeleteDocumentsFromCollection added in v0.3.0 

func (s *ChromemStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error

DeleteDocumentsFromCollection deletes documents by their IDs from a specific collection.

func (*ChromemStore) ExactSearch added in v0.3.0 

func (s *ChromemStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)

ExactSearch performs brute-force similarity search. Note: chromem-go always uses exact search (no HNSW), so this is the same as regular search.

func (*ChromemStore) GetCollectionInfo added in v0.3.0 

func (s *ChromemStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)

GetCollectionInfo returns metadata about a collection.

func (*ChromemStore) IsolationMode added in v0.3.0 

func (s *ChromemStore) IsolationMode() IsolationMode

IsolationMode returns the current isolation mode.

func (*ChromemStore) ListCollections added in v0.3.0 

func (s *ChromemStore) ListCollections(ctx context.Context) ([]string, error)

ListCollections returns a list of all collection names.

func (*ChromemStore) Search added in v0.3.0 

func (s *ChromemStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)

Search performs similarity search in the default collection.

func (*ChromemStore) SearchInCollection added in v0.3.0 

func (s *ChromemStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchInCollection performs similarity search in a specific collection. If isolation mode is set, tenant filters are automatically injected.

func (*ChromemStore) SearchWithFilters added in v0.3.0 

func (s *ChromemStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchWithFilters performs similarity search with metadata filters.

func (*ChromemStore) SetIsolationMode added in v0.3.0 

func (s *ChromemStore) SetIsolationMode(mode IsolationMode)

SetIsolationMode sets the tenant isolation mode for this store. Use NewPayloadIsolation() for multi-tenant payload filtering, or NewNoIsolation() for testing (default).

type ChromemStoreProvider added in v0.3.0 

type ChromemStoreProvider struct {
	// contains filtered or unexported fields
}

ChromemStoreProvider implements StoreProvider using chromem-go.

func NewChromemStoreProvider added in v0.3.0 

func NewChromemStoreProvider(config ProviderConfig, embedder Embedder, logger *zap.Logger) (*ChromemStoreProvider, error)

NewChromemStoreProvider creates a new StoreProvider backed by chromem-go.

SECURITY: This provider has NO authorization checks. It is suitable for:

  • Local development (single-user, localhost)
  • CLI tools (trusted environment)
  • Testing environments

For production multi-tenant use, wrap with AuthorizedStoreProvider.

func (*ChromemStoreProvider) Close added in v0.3.0 

func (p *ChromemStoreProvider) Close() error

Close closes all managed stores.

func (*ChromemStoreProvider) GetOrgStore added in v0.3.0 

func (p *ChromemStoreProvider) GetOrgStore(ctx context.Context, tenant string) (Store, error)

GetOrgStore returns a store scoped to an org (for org-level shared collections).

func (*ChromemStoreProvider) GetProjectStore added in v0.3.0 

func (p *ChromemStoreProvider) GetProjectStore(ctx context.Context, tenant, team, project string) (Store, error)

GetProjectStore returns a store scoped to a specific project.

func (*ChromemStoreProvider) GetTeamStore added in v0.3.0 

func (p *ChromemStoreProvider) GetTeamStore(ctx context.Context, tenant, team string) (Store, error)

GetTeamStore returns a store scoped to a team (for shared collections).

func (*ChromemStoreProvider) Registry added in v0.3.0 

func (p *ChromemStoreProvider) Registry() *registry.Registry

Registry returns the underlying registry for direct access if needed.

type CircuitBreaker added in v0.4.0 

type CircuitBreaker struct {
	// contains filtered or unexported fields
}

CircuitBreaker protects against repeated sync failures.

func NewCircuitBreaker added in v0.4.0 

func NewCircuitBreaker(threshold int32, resetAfter time.Duration) *CircuitBreaker

NewCircuitBreaker creates a new circuit breaker.

func (*CircuitBreaker) Allow added in v0.4.0 

func (cb *CircuitBreaker) Allow() bool

Allow returns true if the operation is allowed.

func (*CircuitBreaker) RecordFailure added in v0.4.0 

func (cb *CircuitBreaker) RecordFailure()

RecordFailure records a failed operation.

func (*CircuitBreaker) RecordSuccess added in v0.4.0 

func (cb *CircuitBreaker) RecordSuccess()

RecordSuccess records a successful operation.

func (*CircuitBreaker) State added in v0.4.0 

func (cb *CircuitBreaker) State() string

State returns the current circuit state.

type CollectionInfo 

type CollectionInfo struct {
	// Name is the collection name.
	Name string `json:"name"`

	// PointCount is the number of vectors in the collection.
	PointCount int `json:"point_count"`

	// VectorSize is the dimensionality of vectors in this collection.
	VectorSize int `json:"vector_size"`
}

CollectionInfo contains metadata about a vector collection.

type CompressionLevel 

type CompressionLevel string

CompressionLevel represents the compression state of content. 

const (
	// CompressionLevelNone represents uncompressed original content.
	CompressionLevelNone CompressionLevel = "none"
	// CompressionLevelFolded represents context-folded content (partial compression).
	CompressionLevelFolded CompressionLevel = "folded"
	// CompressionLevelSummary represents summarized content (high compression).
	CompressionLevelSummary CompressionLevel = "summary"
)

type CompressionMetadata 

type CompressionMetadata struct {
	Level            CompressionLevel `json:"compression_level"`               // Current compression level
	Algorithm        string           `json:"compression_algorithm,omitempty"` // Compression algorithm used
	OriginalSize     int              `json:"original_size"`                   // Original content size (tokens/chars)
	CompressedSize   int              `json:"compressed_size"`                 // Compressed content size
	CompressionRatio float64          `json:"compression_ratio"`               // Compression ratio (original/compressed)
	CompressedAt     *time.Time       `json:"compressed_at,omitempty"`         // When compression was applied
}

CompressionMetadata tracks compression state and metrics.

type Document 

type Document struct {
	// ID is the unique identifier for the document
	ID string

	// Content is the text content of the document
	Content string

	// Metadata contains additional key-value pairs for filtering
	// Common fields: owner, project, file, branch, timestamp
	Metadata map[string]interface{}

	// Collection is the target collection name for this document.
	// If empty, uses the service's default collection.
	//
	// Collection naming convention:
	//   - Organization: org_{type} (e.g., org_memories)
	//   - Team: {team}_{type} (e.g., platform_memories)
	//   - Project: {team}_{project}_{type} (e.g., platform_contextd_memories)
	Collection string
}

Document represents a document to be stored in the vector store.

type Embedder 

type Embedder interface {
	// EmbedDocuments generates embeddings for multiple texts.
	// Returns a slice of embeddings (one per input text) or an error.
	EmbedDocuments(ctx context.Context, texts []string) ([][]float32, error)

	// EmbedQuery generates an embedding for a single query.
	// Some models optimize differently for queries vs documents.
	EmbedQuery(ctx context.Context, text string) ([]float32, error)
}

Embedder generates vector embeddings from text.

Embeddings are dense numerical representations that capture semantic meaning, enabling similarity search. Implementations can use local models (TEI) or cloud APIs (OpenAI, Cohere).

type FallbackConfig added in v0.4.0 

type FallbackConfig struct {
	// Enabled enables fallback storage (default: false).
	Enabled bool

	// LocalPath is the path for local fallback storage.
	// Default: .claude/contextd/store
	LocalPath string

	// SyncOnConnect triggers immediate sync when remote becomes available (default: true).
	SyncOnConnect bool

	// HealthCheckInterval is the interval for periodic health checks (default: 30s).
	HealthCheckInterval string

	// WALPath is the directory for write-ahead log.
	// Default: .claude/contextd/wal
	WALPath string

	// WALRetentionDays is how long to keep synced entries in WAL (default: 7).
	WALRetentionDays int
}

FallbackConfig holds configuration for fallback storage.

func (*FallbackConfig) ApplyDefaults added in v0.4.0 

func (c *FallbackConfig) ApplyDefaults()

ApplyDefaults sets default values for unset fields.

func (*FallbackConfig) Validate added in v0.4.0 

func (c *FallbackConfig) Validate() error

Validate validates the fallback configuration.

type FallbackStore added in v0.4.0 

type FallbackStore struct {
	// contains filtered or unexported fields
}

FallbackStore implements the Store interface with graceful fallback to local storage.

When the remote store (Qdrant) is unavailable, writes go to local storage and a write-ahead log (WAL). When connectivity is restored, pending operations are automatically synced to the remote store.

Thread-safe: All operations are protected by internal mutexes.

func NewFallbackStore added in v0.4.0 

func NewFallbackStore(
	ctx context.Context,
	remote, local Store,
	health *HealthMonitor,
	wal *WAL,
	config FallbackConfig,
	logger *zap.Logger,
) (*FallbackStore, error)

NewFallbackStore creates a new FallbackStore wrapping remote and local stores.

func (*FallbackStore) AddDocuments added in v0.4.0 

func (fs *FallbackStore) AddDocuments(ctx context.Context, docs []Document) ([]string, error)

AddDocuments adds documents with fallback support.

Write path (atomic with rollback):

  1. Scrub document content (handled by WAL)
  2. Check remote health
  3. IF HEALTHY: a. Write to REMOTE first b. Write to LOCAL (for query consistency) c. Record in WAL as SYNCED d. Return success
  4. IF UNHEALTHY: a. Record in WAL as PENDING (with checksum) b. Write to LOCAL c. Return success
  5. ON ANY FAILURE: a. Rollback: Delete from stores where written b. Remove incomplete WAL entry c. Return error

func (*FallbackStore) Close added in v0.4.0 

func (fs *FallbackStore) Close() error

Close closes the fallback store and releases resources.

func (*FallbackStore) CollectionExists added in v0.4.0 

func (fs *FallbackStore) CollectionExists(ctx context.Context, collectionName string) (bool, error)

CollectionExists checks if a collection exists.

func (*FallbackStore) CreateCollection added in v0.4.0 

func (fs *FallbackStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error

CreateCollection creates a new collection.

func (*FallbackStore) DeleteCollection added in v0.4.0 

func (fs *FallbackStore) DeleteCollection(ctx context.Context, collectionName string) error

DeleteCollection deletes a collection and all its documents.

func (*FallbackStore) DeleteDocuments added in v0.4.0 

func (fs *FallbackStore) DeleteDocuments(ctx context.Context, ids []string) error

DeleteDocuments deletes documents by their IDs.

func (*FallbackStore) DeleteDocumentsFromCollection added in v0.4.0 

func (fs *FallbackStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error

DeleteDocumentsFromCollection deletes documents from a specific collection.

func (*FallbackStore) ExactSearch added in v0.4.0 

func (fs *FallbackStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)

ExactSearch performs brute-force similarity search.

func (*FallbackStore) GetCollectionInfo added in v0.4.0 

func (fs *FallbackStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)

GetCollectionInfo returns metadata about a collection.

func (*FallbackStore) IsolationMode added in v0.4.0 

func (fs *FallbackStore) IsolationMode() IsolationMode

IsolationMode returns the current isolation mode.

func (*FallbackStore) ListCollections added in v0.4.0 

func (fs *FallbackStore) ListCollections(ctx context.Context) ([]string, error)

ListCollections returns a list of all collection names.

func (*FallbackStore) Search added in v0.4.0 

func (fs *FallbackStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)

Search performs similarity search with merge strategy.

Read path (merge strategy):

  1. Check remote health
  2. IF HEALTHY: a. Search REMOTE (authoritative) b. Search LOCAL for pending (unsynced) documents c. Merge results (local wins for conflicts) d. Add metadata: {source: "merged", pending_count: N}
  3. IF UNHEALTHY: a. Search LOCAL only b. Add metadata: {source: "local", last_sync: timestamp, stale_warning: true}
  4. Return results with metadata

func (*FallbackStore) SearchInCollection added in v0.4.0 

func (fs *FallbackStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchInCollection performs similarity search in a specific collection.

func (*FallbackStore) SearchWithFilters added in v0.4.0 

func (fs *FallbackStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchWithFilters performs similarity search with metadata filters.

func (*FallbackStore) SetIsolationMode added in v0.4.0 

func (fs *FallbackStore) SetIsolationMode(mode IsolationMode)

SetIsolationMode sets the tenant isolation mode for both stores.

type FilesystemIsolation added in v0.3.0 

type FilesystemIsolation struct{}

FilesystemIsolation implements IsolationMode using separate stores.

In this mode:

  • Each tenant/team/project gets its own database directory
  • Physical filesystem isolation provides security boundary
  • No metadata filtering needed (isolation is structural)

Note: This is the legacy mode used by ChromemStoreProvider. Consider migration to PayloadIsolation for simpler operations.

func NewFilesystemIsolation added in v0.3.0 

func NewFilesystemIsolation() *FilesystemIsolation

NewFilesystemIsolation creates a new FilesystemIsolation mode.

func (*FilesystemIsolation) InjectFilter added in v0.3.0 

func (f *FilesystemIsolation) InjectFilter(ctx context.Context, filters map[string]interface{}) (map[string]interface{}, error)

InjectFilter is a no-op for filesystem isolation (isolation is structural).

func (*FilesystemIsolation) InjectMetadata added in v0.3.0 

func (f *FilesystemIsolation) InjectMetadata(ctx context.Context, docs []Document) error

InjectMetadata is optional for filesystem isolation but adds tenant info for auditability.

func (*FilesystemIsolation) Mode added in v0.3.0 

func (f *FilesystemIsolation) Mode() string

Mode returns "filesystem" for this isolation mode.

func (*FilesystemIsolation) ValidateTenant added in v0.3.0 

func (f *FilesystemIsolation) ValidateTenant(ctx context.Context) error

ValidateTenant checks tenant context is present and valid.

type FilterBuilder added in v0.3.0 

type FilterBuilder struct {
	// contains filtered or unexported fields
}

FilterBuilder provides a fluent interface for building query filters.

func NewFilterBuilder added in v0.3.0 

func NewFilterBuilder() *FilterBuilder

NewFilterBuilder creates a new FilterBuilder.

func (*FilterBuilder) Build added in v0.3.0 

func (b *FilterBuilder) Build() map[string]interface{}

Build returns the constructed filter map.

func (*FilterBuilder) With added in v0.3.0 

func (b *FilterBuilder) With(key string, value interface{}) *FilterBuilder

With adds a key-value pair to the filter.

func (*FilterBuilder) WithMap added in v0.3.0 

func (b *FilterBuilder) WithMap(m map[string]interface{}) *FilterBuilder

WithMap merges an existing filter map.

func (*FilterBuilder) WithTenant added in v0.3.0 

func (b *FilterBuilder) WithTenant(tenant *TenantInfo) *FilterBuilder

WithTenant adds tenant filters from TenantInfo.

type GRPCHealthChecker added in v0.4.0 

type GRPCHealthChecker struct {
	// contains filtered or unexported fields
}

GRPCHealthChecker implements HealthChecker for Qdrant gRPC connections.

func NewGRPCHealthChecker added in v0.4.0 

func NewGRPCHealthChecker(conn *grpc.ClientConn, logger *zap.Logger) *GRPCHealthChecker

NewGRPCHealthChecker creates a new gRPC health checker.

func (*GRPCHealthChecker) IsHealthy added in v0.4.0 

func (g *GRPCHealthChecker) IsHealthy(ctx context.Context) bool

IsHealthy returns true if the gRPC connection is in Ready state.

func (*GRPCHealthChecker) WatchState added in v0.4.0 

func (g *GRPCHealthChecker) WatchState(ctx context.Context, callback func(healthy bool)) error

WatchState watches for gRPC connectivity state changes.

type HealthChecker added in v0.4.0 

type HealthChecker interface {
	// IsHealthy returns true if the remote store is healthy.
	IsHealthy(ctx context.Context) bool

	// WatchState watches for connectivity state changes.
	// The callback is invoked whenever health status changes.
	WatchState(ctx context.Context, callback func(healthy bool)) error
}

HealthChecker interface for dependency injection and testability.

type HealthMonitor added in v0.4.0 

type HealthMonitor struct {
	// contains filtered or unexported fields
}

HealthMonitor monitors remote store connectivity.

func NewHealthMonitor added in v0.4.0 

func NewHealthMonitor(ctx context.Context, checker HealthChecker, checkInterval time.Duration, logger *zap.Logger) *HealthMonitor

NewHealthMonitor creates a new health monitor.

func (*HealthMonitor) IsHealthy added in v0.4.0 

func (hm *HealthMonitor) IsHealthy() bool

IsHealthy returns the current health status.

func (*HealthMonitor) LastCheck added in v0.4.0 

func (hm *HealthMonitor) LastCheck() time.Time

LastCheck returns the time of the last health check.

func (*HealthMonitor) RegisterCallback added in v0.4.0 

func (hm *HealthMonitor) RegisterCallback(cb func(bool)) error

RegisterCallback adds a callback with mutex protection. Returns an error if the callback is nil.

func (*HealthMonitor) SetCallbackTimeout added in v0.4.0 

func (hm *HealthMonitor) SetCallbackTimeout(timeout time.Duration)

SetCallbackTimeout configures the timeout for callback executions.

func (*HealthMonitor) Start added in v0.4.0 

func (hm *HealthMonitor) Start()

Start begins health monitoring.

func (*HealthMonitor) Stop added in v0.4.0 

func (hm *HealthMonitor) Stop()

Stop gracefully shuts down the health monitor.

type IsolationMode added in v0.3.0 

type IsolationMode interface {
	// InjectFilter adds tenant filtering to search options.
	// Must fail with ErrMissingTenant if tenant context is absent.
	InjectFilter(ctx context.Context, filters map[string]interface{}) (map[string]interface{}, error)

	// InjectMetadata adds tenant metadata to documents before storage.
	// Must fail with ErrMissingTenant if tenant context is absent.
	InjectMetadata(ctx context.Context, docs []Document) error

	// ValidateTenant checks that tenant context is present and valid.
	// Returns nil if valid, ErrMissingTenant or ErrInvalidTenant otherwise.
	ValidateTenant(ctx context.Context) error

	// Mode returns the isolation mode name for logging/debugging.
	Mode() string
}

IsolationMode defines how tenant isolation is enforced in vector stores.

Implementations determine whether isolation is via:

  • Payload filtering (single collection, metadata-based filtering)
  • Filesystem isolation (separate databases per tenant/project)
  • Hybrid approaches (tiered isolation)

Security: All implementations must enforce fail-closed behavior.

func IsolationModeFromString added in v0.3.0 

func IsolationModeFromString(mode string) (IsolationMode, error)

IsolationModeFromString creates an IsolationMode from a string name.

type MetadataBuilder added in v0.3.0 

type MetadataBuilder struct {
	// contains filtered or unexported fields
}

MetadataBuilder provides a fluent interface for building document metadata.

func NewMetadataBuilder added in v0.3.0 

func NewMetadataBuilder() *MetadataBuilder

NewMetadataBuilder creates a new MetadataBuilder.

func (*MetadataBuilder) Build added in v0.3.0 

func (b *MetadataBuilder) Build() map[string]interface{}

Build returns the constructed metadata map.

func (*MetadataBuilder) With added in v0.3.0 

func (b *MetadataBuilder) With(key string, value interface{}) *MetadataBuilder

With adds a key-value pair to the metadata.

func (*MetadataBuilder) WithMap added in v0.3.0 

func (b *MetadataBuilder) WithMap(m map[string]interface{}) *MetadataBuilder

WithMap merges an existing metadata map.

func (*MetadataBuilder) WithTenant added in v0.3.0 

func (b *MetadataBuilder) WithTenant(tenant *TenantInfo) *MetadataBuilder

WithTenant adds tenant metadata from TenantInfo.

type MetadataHealth added in v0.4.0 

type MetadataHealth struct {
	Healthy       []string          `json:"healthy"`         // Collections with valid metadata
	Corrupt       []string          `json:"corrupt"`         // Collections missing metadata but have documents
	Empty         []string          `json:"empty"`           // Collections with no documents
	Total         int               `json:"total"`           // Total collections found
	HealthyCount  int               `json:"healthy_count"`   // Count of healthy collections
	CorruptCount  int               `json:"corrupt_count"`   // Count of corrupt collections
	LastCheckTime time.Time         `json:"last_check_time"` // When the check was performed
	CheckDuration time.Duration     `json:"check_duration"`  // How long the check took
	Details       map[string]string `json:"details"`         // Per-collection status details
}

MetadataHealth represents the health status of collection metadata files.

func (*MetadataHealth) IsHealthy added in v0.4.0 

func (h *MetadataHealth) IsHealthy() bool

IsHealthy returns true if all collections have valid metadata.

func (*MetadataHealth) Status added in v0.4.0 

func (h *MetadataHealth) Status() string

Status returns a simple status string.

type MetadataHealthChecker added in v0.4.0 

type MetadataHealthChecker struct {
	// contains filtered or unexported fields
}

MetadataHealthChecker provides metadata integrity verification.

func NewMetadataHealthChecker added in v0.4.0 

func NewMetadataHealthChecker(path string, logger *zap.Logger) *MetadataHealthChecker

NewMetadataHealthChecker creates a new metadata health checker.

func (*MetadataHealthChecker) Check added in v0.4.0 

func (c *MetadataHealthChecker) Check(ctx context.Context) (*MetadataHealth, error)

Check performs a metadata integrity check on all collections.

type Metrics added in v0.4.0 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics holds all vectorstore-related metrics.

func NewMetrics added in v0.4.0 

func NewMetrics(logger *zap.Logger) *Metrics

NewMetrics creates a new Metrics instance for vectorstore.

func (*Metrics) RecordDocuments added in v0.4.0 

func (m *Metrics) RecordDocuments(ctx context.Context, op, collection string, count int)

RecordDocuments records document count for add/delete operations.

func (*Metrics) RecordOperation added in v0.4.0 

func (m *Metrics) RecordOperation(ctx context.Context, op, collection string, duration time.Duration, err error)

RecordOperation records a vectorstore operation metric.

func (*Metrics) RecordSearchResults added in v0.4.0 

func (m *Metrics) RecordSearchResults(ctx context.Context, collection string, count int)

RecordSearchResults records the number of search results returned.

type MockHealthChecker added in v0.4.0 

type MockHealthChecker struct {
	// contains filtered or unexported fields
}

MockHealthChecker for testing.

func NewMockHealthChecker added in v0.4.0 

func NewMockHealthChecker() *MockHealthChecker

NewMockHealthChecker creates a new mock health checker.

func (*MockHealthChecker) IsHealthy added in v0.4.0 

func (m *MockHealthChecker) IsHealthy(ctx context.Context) bool

IsHealthy returns the mock health status.

func (*MockHealthChecker) SetHealthy added in v0.4.0 

func (m *MockHealthChecker) SetHealthy(healthy bool)

SetHealthy sets the mock health status and does not trigger callbacks.

func (*MockHealthChecker) WatchState added in v0.4.0 

func (m *MockHealthChecker) WatchState(ctx context.Context, callback func(healthy bool)) error

WatchState does nothing for mock (no state changes to watch).

type NoIsolation added in v0.3.0 

type NoIsolation struct{}

NoIsolation provides no tenant isolation - for testing only.

WARNING: This mode provides no security guarantees. Use only in tests where tenant isolation is not relevant.

func NewNoIsolation added in v0.3.0 

func NewNoIsolation() *NoIsolation

NewNoIsolation creates a new NoIsolation mode (testing only).

func (*NoIsolation) InjectFilter added in v0.3.0 

func (n *NoIsolation) InjectFilter(ctx context.Context, filters map[string]interface{}) (map[string]interface{}, error)

InjectFilter passes through filters unchanged.

func (*NoIsolation) InjectMetadata added in v0.3.0 

func (n *NoIsolation) InjectMetadata(ctx context.Context, docs []Document) error

InjectMetadata is a no-op.

func (*NoIsolation) Mode added in v0.3.0 

func (n *NoIsolation) Mode() string

Mode returns "none" for this isolation mode.

func (*NoIsolation) ValidateTenant added in v0.3.0 

func (n *NoIsolation) ValidateTenant(ctx context.Context) error

ValidateTenant always succeeds.

type PayloadIsolation added in v0.3.0 

type PayloadIsolation struct{}

PayloadIsolation implements IsolationMode using metadata filtering.

In this mode:

  • All documents in a single collection per type (e.g., "memories")
  • tenant_id, team_id, project_id stored as document metadata
  • All queries automatically filtered by tenant context
  • Missing tenant context = error (fail closed)

Security guarantees:

  • Mandatory filter injection on all queries
  • No bypass possible - private methods enforce filtering
  • Audit-friendly - tenant always in context

func NewPayloadIsolation added in v0.3.0 

func NewPayloadIsolation() *PayloadIsolation

NewPayloadIsolation creates a new PayloadIsolation mode.

func (*PayloadIsolation) InjectFilter added in v0.3.0 

func (p *PayloadIsolation) InjectFilter(ctx context.Context, filters map[string]interface{}) (map[string]interface{}, error)

InjectFilter adds tenant filters to existing query filters.

func (*PayloadIsolation) InjectMetadata added in v0.3.0 

func (p *PayloadIsolation) InjectMetadata(ctx context.Context, docs []Document) error

InjectMetadata adds tenant metadata to all documents.

func (*PayloadIsolation) Mode added in v0.3.0 

func (p *PayloadIsolation) Mode() string

Mode returns "payload" for this isolation mode.

func (*PayloadIsolation) ValidateTenant added in v0.3.0 

func (p *PayloadIsolation) ValidateTenant(ctx context.Context) error

ValidateTenant checks tenant context is present and valid.

type ProviderConfig added in v0.3.0 

type ProviderConfig struct {
	// BasePath is the root directory for all vectorstore data.
	// Default: ~/.config/contextd/vectorstore
	BasePath string

	// Compress enables gzip compression for stored data.
	Compress bool

	// VectorSize is the expected embedding dimension.
	// Default: 384 (for FastEmbed bge-small-en-v1.5)
	VectorSize int

	// LocalModeAcknowledged suppresses security warnings about missing authorization.
	// Set to true when you understand this provider has no auth and is for local use only.
	// Alternative: Set CONTEXTD_LOCAL_MODE=1 environment variable.
	LocalModeAcknowledged bool
}

ProviderConfig holds configuration for ChromemStoreProvider.

func (*ProviderConfig) ApplyDefaults added in v0.3.0 

func (c *ProviderConfig) ApplyDefaults()

ApplyDefaults sets default values for unset fields.

type QdrantConfig 

type QdrantConfig struct {
	// Host is the Qdrant server hostname or IP address.
	// Default: "localhost"
	Host string

	// Port is the Qdrant gRPC port (NOT HTTP REST port).
	// Default: 6334 (gRPC), not 6333 (HTTP)
	Port int

	// CollectionName is the default collection for operations.
	// Format: {scope}_{type} for multi-tenancy
	// Examples: org_memories, platform_memories, platform_contextd_memories
	CollectionName string

	// VectorSize is the dimensionality of embeddings.
	// Examples: 384 (BAAI/bge-small-en-v1.5), 768 (BERT), 1536 (OpenAI)
	// MUST match Embedder output dimensions.
	VectorSize uint64

	// Distance is the similarity metric for vector search.
	// Options: Cosine (default), Euclid, Dot
	Distance qdrant.Distance

	// UseTLS enables TLS encryption for gRPC connection.
	// Default: false (MVP), true (production)
	UseTLS bool

	// MaxRetries is the maximum number of retry attempts for transient failures.
	// Default: 3
	MaxRetries int

	// RetryBackoff is the initial backoff duration for retries.
	// Doubles on each retry (exponential backoff).
	// Default: 1 second
	RetryBackoff time.Duration

	// MaxMessageSize is the maximum gRPC message size in bytes.
	// Default: 50MB (to handle large documents)
	MaxMessageSize int

	// CircuitBreakerThreshold is the number of failures before opening circuit.
	// Default: 5
	CircuitBreakerThreshold int

	// Isolation is the tenant isolation mode.
	// Default: PayloadIsolation for fail-closed security.
	// Set at construction time; immutable afterward to prevent race conditions.
	Isolation IsolationMode
}

QdrantConfig holds configuration for Qdrant gRPC client.

func (*QdrantConfig) ApplyDefaults 

func (c *QdrantConfig) ApplyDefaults()

ApplyDefaults sets default values for unset fields.

func (QdrantConfig) Validate 

func (c QdrantConfig) Validate() error

Validate validates the configuration.

type QdrantStore 

type QdrantStore struct {
	// contains filtered or unexported fields
}

QdrantStore is a Store implementation using Qdrant's native gRPC client.

This implementation bypasses Qdrant's actix-web HTTP layer, eliminating the 256kB payload limit that causes 413 errors during repository indexing.

Key features:

  • Native gRPC transport (port 6334)
  • Binary protobuf encoding (no JSON size limits)
  • Better performance than HTTP REST
  • Full Qdrant feature access
  • Collection-per-project isolation
  • Tenant isolation via payload filtering

func NewQdrantStore 

func NewQdrantStore(config QdrantConfig, embedder Embedder) (*QdrantStore, error)

NewQdrantStore creates a new QdrantStore with the given configuration.

The constructor performs the following steps:

  1. Validates configuration
  2. Creates Qdrant gRPC client
  3. Performs health check
  4. Returns ready-to-use store

Returns an error if:

  • Configuration is invalid
  • Connection to Qdrant fails
  • Health check fails

func (*QdrantStore) AddDocuments 

func (s *QdrantStore) AddDocuments(ctx context.Context, docs []Document) ([]string, error)

AddDocuments adds documents to the vector store. If isolation mode is set, tenant metadata is automatically injected.

func (*QdrantStore) Close 

func (s *QdrantStore) Close() error

Close closes the Qdrant gRPC connection.

func (*QdrantStore) CollectionExists 

func (s *QdrantStore) CollectionExists(ctx context.Context, collectionName string) (bool, error)

CollectionExists checks if a collection exists.

func (*QdrantStore) CreateCollection 

func (s *QdrantStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error

CreateCollection creates a new collection with the specified configuration.

func (*QdrantStore) DeleteCollection 

func (s *QdrantStore) DeleteCollection(ctx context.Context, collectionName string) error

DeleteCollection deletes a collection and all its documents.

func (*QdrantStore) DeleteDocuments 

func (s *QdrantStore) DeleteDocuments(ctx context.Context, ids []string) error

DeleteDocuments deletes documents by their IDs from the default collection.

func (*QdrantStore) DeleteDocumentsFromCollection 

func (s *QdrantStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error

DeleteDocumentsFromCollection deletes documents by their IDs from a specific collection.

func (*QdrantStore) ExactSearch 

func (s *QdrantStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)

ExactSearch performs brute-force similarity search without using HNSW index. This is a fallback for small datasets (<10 vectors) where HNSW index may not be built.

func (*QdrantStore) GetCollectionInfo 

func (s *QdrantStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)

GetCollectionInfo returns metadata about a collection.

func (*QdrantStore) IsolationMode added in v0.3.0 

func (s *QdrantStore) IsolationMode() IsolationMode

IsolationMode returns the current isolation mode.

func (*QdrantStore) ListCollections 

func (s *QdrantStore) ListCollections(ctx context.Context) ([]string, error)

ListCollections returns a list of all collection names.

func (*QdrantStore) Search 

func (s *QdrantStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)

Search performs similarity search in the default collection.

func (*QdrantStore) SearchInCollection 

func (s *QdrantStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchInCollection performs similarity search in a specific collection. If isolation mode is set, tenant filters are automatically injected.

func (*QdrantStore) SearchWithFilters 

func (s *QdrantStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchWithFilters performs similarity search with metadata filters.

func (*QdrantStore) SetIsolationMode added in v0.3.0 

func (s *QdrantStore) SetIsolationMode(mode IsolationMode)

SetIsolationMode sets the tenant isolation mode for this store.

DEPRECATED: Prefer setting isolation via config at construction time (e.g., QdrantConfig.Isolation) for thread-safety. This method exists for backward compatibility but should only be called once before any operations. Calling SetIsolationMode concurrently with operations may cause race conditions.

Use NewPayloadIsolation() for multi-tenant payload filtering, NewFilesystemIsolation() for database-per-project isolation, or NewNoIsolation() for testing only.

Default is PayloadIsolation for fail-closed security.

type QualityMetrics added in v0.4.0 

type QualityMetrics struct {
	// NDCG is Normalized Discounted Cumulative Gain (0.0-1.0)
	// Measures ranking quality considering both relevance and position.
	// 1.0 = perfect ranking, 0.0 = worst possible ranking.
	NDCG float64

	// MRR is Mean Reciprocal Rank (0.0-1.0)
	// Measures where the first relevant document appears.
	// 1.0 = first result is relevant, 0.0 = no relevant results.
	MRR float64

	// PrecisionAtK is the proportion of relevant documents in top K results (0.0-1.0)
	// Measures how many retrieved documents are actually relevant.
	PrecisionAtK float64

	// K is the cutoff used for these metrics
	K int
}

QualityMetrics contains retrieval quality measurements. These metrics help track search effectiveness over time and detect regressions.

func CalculateAllMetrics added in v0.4.0 

func CalculateAllMetrics(results []SearchResult, expectedRanking []string, relevantDocs []string, k int) QualityMetrics

CalculateAllMetrics computes NDCG, MRR, and Precision@K in a single pass.

This is more efficient than calling each metric function separately when you need all three metrics, as it only builds the relevance lookups once.

Parameters:

  • results: Search results ordered by the system
  • expectedRanking: Ideal order of document IDs by relevance (for NDCG)
  • relevantDocs: Set of document IDs that are considered relevant (for MRR and P@K)
  • k: Number of top results to consider

Returns:

  • QualityMetrics struct containing all three metrics

type SearchResult 

type SearchResult struct {
	// ID is the document identifier
	ID string

	// Content is the document text content
	Content string

	// Score is the similarity score (higher = more similar)
	Score float32

	// Metadata contains the document metadata
	Metadata map[string]interface{}
}

SearchResult represents a search result from the vector store.

type StartupValidationConfig added in v0.4.0 

type StartupValidationConfig struct {
	// FailOnCorruption blocks startup if corrupt collections are detected.
	// Default: false (log warning but continue with graceful degradation)
	FailOnCorruption bool

	// FailOnDegraded blocks startup if any health issues are detected.
	// This is stricter than FailOnCorruption (includes empty collections, etc.)
	// Default: false
	FailOnDegraded bool
}

StartupValidationConfig configures pre-flight health checks.

type StartupValidationResult added in v0.4.0 

type StartupValidationResult struct {
	Passed       bool
	Health       *MetadataHealth
	WarningCount int
	ErrorCount   int
	Messages     []string
}

StartupValidationResult contains the outcome of pre-flight checks.

func ValidateStartup added in v0.4.0 

func ValidateStartup(ctx context.Context, checker *MetadataHealthChecker, cfg *StartupValidationConfig, logger *zap.Logger) (*StartupValidationResult, error)

ValidateStartup performs pre-flight health checks before services start. Returns nil if validation passes, error if startup should be blocked.

type Store 

type Store interface {
	// AddDocuments adds documents to the vector store.
	//
	// Documents are embedded and stored with their metadata. The document ID
	// is used as the unique identifier in the vector store.
	//
	// If Document.Collection is specified, the document is added to that collection.
	// Otherwise, the implementation's default collection is used.
	//
	// Returns the IDs of added documents and an error if the operation fails.
	AddDocuments(ctx context.Context, docs []Document) ([]string, error)

	// Search performs similarity search in the default collection.
	//
	// It searches for documents similar to the query and returns up to k results
	// ordered by similarity score (highest first).
	//
	// Returns search results with scores and metadata, or an error if search fails.
	Search(ctx context.Context, query string, k int) ([]SearchResult, error)

	// SearchWithFilters performs similarity search with metadata filters.
	//
	// Filters are applied to document metadata (e.g., {"owner": "alice"}).
	// Only documents matching ALL filter conditions are returned.
	//
	// Returns filtered search results or an error if search fails.
	SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

	// SearchInCollection performs similarity search in a specific collection.
	//
	// This supports the hierarchical collection architecture by allowing searches
	// in scope-specific collections (e.g., "org_memories", "platform_contextd_memories").
	//
	// Returns filtered search results from the specified collection, or an error.
	SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

	// DeleteDocuments deletes documents by their IDs from the default collection.
	//
	// Returns an error if deletion fails.
	DeleteDocuments(ctx context.Context, ids []string) error

	// DeleteDocumentsFromCollection deletes documents by their IDs from a specific collection.
	//
	// Returns an error if deletion fails.
	DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error

	// CreateCollection creates a new collection with the specified configuration.
	//
	// Collections are namespaces for documents (e.g., project-specific collections).
	// The vectorSize parameter specifies the dimensionality of embeddings.
	//
	// Returns an error if collection creation fails or collection already exists.
	CreateCollection(ctx context.Context, collectionName string, vectorSize int) error

	// DeleteCollection deletes a collection and all its documents.
	//
	// This is a destructive operation that cannot be undone.
	//
	// Returns an error if deletion fails or collection doesn't exist.
	DeleteCollection(ctx context.Context, collectionName string) error

	// CollectionExists checks if a collection exists.
	//
	// Returns true if the collection exists, false otherwise.
	// Returns an error only if the check operation itself fails.
	CollectionExists(ctx context.Context, collectionName string) (bool, error)

	// ListCollections returns a list of all collection names.
	//
	// Returns collection names or an error if listing fails.
	ListCollections(ctx context.Context) ([]string, error)

	// GetCollectionInfo returns metadata about a collection.
	//
	// Returns collection info including point count and vector size.
	// Returns ErrCollectionNotFound if the collection doesn't exist.
	GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)

	// ExactSearch performs brute-force similarity search without using HNSW index.
	//
	// This is a fallback for small datasets (<10 vectors) where HNSW index
	// may not be built. It performs exact cosine similarity on all vectors.
	//
	// Returns search results ordered by similarity score (highest first).
	ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)

	// SetIsolationMode sets the tenant isolation mode for this store.
	//
	// DEPRECATED: Prefer setting isolation via config at construction time
	// (e.g., ChromemConfig.Isolation) for thread-safety. This method exists
	// for backward compatibility but should only be called once before any
	// operations. Calling SetIsolationMode concurrently with operations may
	// cause race conditions.
	//
	// Use NewPayloadIsolation() for multi-tenant payload filtering,
	// NewFilesystemIsolation() for database-per-project isolation,
	// or NewNoIsolation() for testing only.
	//
	// Default is PayloadIsolation for fail-closed security.
	SetIsolationMode(mode IsolationMode)

	// IsolationMode returns the current isolation mode.
	IsolationMode() IsolationMode

	// Close closes the vector store connection and releases resources.
	Close() error
}

Store is the interface for vector storage operations.

This interface is transport-agnostic - implementations can use HTTP REST, gRPC, or any other protocol. The interface focuses on contextd's specific needs for document storage, search, and collection management.

Collection Naming Convention:

  • Organization: org_{type} (e.g., org_memories)
  • Team: {team}_{type} (e.g., platform_memories)
  • Project: {team}_{project}_{type} (e.g., platform_contextd_memories)

Tenant Isolation:

Stores support two isolation modes. The preferred pattern is to set isolation via config at construction time (e.g., ChromemConfig.Isolation) for thread-safety:

  • PayloadIsolation: Single collection per type with metadata-based filtering. All documents include tenant_id, team_id, project_id in metadata. Queries automatically filter by tenant context from ctx. Requires: TenantInfo in context (see ContextWithTenant). Security: Fail-closed - missing tenant context returns ErrMissingTenant.

  • FilesystemIsolation: Database-per-project isolation (legacy). Uses StoreProvider to create separate stores per tenant/project path. Physical filesystem isolation provides security boundary.

When using PayloadIsolation, callers MUST provide tenant context: 

ctx = vectorstore.ContextWithTenant(ctx, &vectorstore.TenantInfo{
    TenantID:  "org-123",
    TeamID:    "team-1",    // optional
    ProjectID: "proj-1",    // optional
})
results, err := store.Search(ctx, query, k)

Implementations:

  • ChromemStore: Embedded chromem-go (default)
  • QdrantStore: External Qdrant gRPC client

func NewStore added in v0.3.0 

func NewStore(cfg *config.Config, embedder Embedder, logger *zap.Logger, opts ...StoreOption) (Store, error)

NewStore creates a new Store based on the configuration.

This factory function examines the VectorStoreConfig.Provider field and creates the appropriate store implementation:

  • "chromem" (default): Creates an embedded ChromemStore (no external deps)
  • "qdrant": Creates a QdrantStore (requires external Qdrant server)

If fallback is enabled, the store is wrapped with FallbackStore for graceful degradation when the remote store (Qdrant) is unavailable.

Tenant Isolation:

By default, stores use PayloadIsolation mode for fail-closed security. All operations require tenant context (TenantInfo in ctx) or return ErrMissingTenant. To disable isolation for testing, use the WithIsolation option: 

store, err := vectorstore.NewStore(cfg, embedder, logger,
    vectorstore.WithIsolation(vectorstore.NewNoIsolation()))  // Testing only!

The chromem provider is recommended for most users as it requires no setup: 

brew install contextd  # Just works!

Example usage: 

cfg := config.Load()
store, err := vectorstore.NewStore(cfg, embedder, logger)
if err != nil {
    log.Fatal(err)
}
defer store.Close()

func NewStoreFromProvider added in v0.3.0 

func NewStoreFromProvider(provider string, chromemCfg *ChromemConfig, qdrantCfg *QdrantConfig, embedder Embedder, logger *zap.Logger, opts ...StoreOption) (Store, error)

NewStoreFromProvider creates a store directly from provider name and specific config. This is useful when you need more control over configuration.

type StoreOption added in v0.3.0 

type StoreOption func(store Store)

StoreOption configures a Store after creation.

func WithIsolation added in v0.3.0 

func WithIsolation(mode IsolationMode) StoreOption

WithIsolation sets the isolation mode for a store. Use NewPayloadIsolation() for multi-tenant payload filtering, NewFilesystemIsolation() for database-per-project isolation, or NewNoIsolation() for testing only.

type StoreProvider added in v0.3.0 

type StoreProvider interface {
	// GetProjectStore returns a store for project-level collections.
	// Path: {basePath}/{tenant}/{project}/ (direct)
	// Path: {basePath}/{tenant}/{team}/{project}/ (team-scoped)
	GetProjectStore(ctx context.Context, tenant, team, project string) (Store, error)

	// GetTeamStore returns a store for team-level shared collections.
	// Path: {basePath}/{tenant}/{team}/
	GetTeamStore(ctx context.Context, tenant, team string) (Store, error)

	// GetOrgStore returns a store for org-level shared collections.
	// Path: {basePath}/{tenant}/
	GetOrgStore(ctx context.Context, tenant string) (Store, error)

	// Close closes all managed stores.
	Close() error
}

StoreProvider manages chromem.DB instances per scope path.

This enables database-per-project isolation where:

  • Each project gets its own chromem.DB at a unique filesystem path
  • Collection names are simple ("checkpoints", "memories") not prefixed
  • Physical filesystem isolation prevents data leakage

Path hierarchy:

  • Project (direct): {basePath}/{tenant}/{project}/
  • Project (team-scoped): {basePath}/{tenant}/{team}/{project}/
  • Team shared: {basePath}/{tenant}/{team}/
  • Org shared: {basePath}/{tenant}/

SECURITY NOTE: LOCAL DEVELOPMENT ONLY

The current implementation does NOT include authorization checks. Any caller can request any tenant/team/project store without verification. This is acceptable for:

  • Local development (single-user, localhost)
  • CLI tools (trusted environment)
  • Testing environments

For multi-tenant production deployments, you MUST:

  1. Wrap with AuthorizedStoreProvider with session-based authentication
  2. Implement tenant membership verification before granting store access
  3. Enable audit logging for all store access

To acknowledge local-only mode and suppress warnings:

  • Set CONTEXTD_LOCAL_MODE=1 environment variable, OR
  • Set LocalModeAcknowledged=true in ProviderConfig

PRODUCTION MODE ENFORCEMENT:

When CONTEXTD_PRODUCTION_MODE=1 is set, the provider will FAIL FAST if authorization is not explicitly acknowledged. This prevents accidental deployment without security review.

Production mode behavior:

  • Without acknowledgment: Returns error, server fails to start
  • With CONTEXTD_LOCAL_MODE=1: Starts with warning (explicit override)
  • With LocalModeAcknowledged=true: Starts with warning (explicit override)

Recommended production setup:

  1. Set CONTEXTD_PRODUCTION_MODE=1 in your deployment
  2. Implement AuthorizedStoreProvider wrapper (see example below)
  3. Never set CONTEXTD_LOCAL_MODE=1 in production

Example AuthorizedStoreProvider pattern (NOT IMPLEMENTED - reference only): 

type AuthorizedStoreProvider struct {
    inner   StoreProvider
    session *AuthSession  // Contains tenant, user, permissions
}

func (a *AuthorizedStoreProvider) GetProjectStore(ctx context.Context, tenant, team, project string) (Store, error) {
    if !a.session.CanAccess(tenant, project) {
        return nil, fmt.Errorf("unauthorized: user cannot access tenant %s", tenant)
    }
    return a.inner.GetProjectStore(ctx, tenant, team, project)
}

type SyncManager added in v0.4.0 

type SyncManager struct {
	// contains filtered or unexported fields
}

SyncManager manages background synchronization from local to remote store.

func NewSyncManager added in v0.4.0 

func NewSyncManager(ctx context.Context, wal *WAL, local, remote Store, health *HealthMonitor, logger *zap.Logger) *SyncManager

NewSyncManager creates a SyncManager with bounded channels and shutdown support.

func (*SyncManager) Start added in v0.4.0 

func (s *SyncManager) Start()

Start begins background sync goroutine.

func (*SyncManager) Stop added in v0.4.0 

func (s *SyncManager) Stop() error

Stop gracefully shuts down the sync manager.

func (*SyncManager) TriggerSync added in v0.4.0 

func (s *SyncManager) TriggerSync()

TriggerSync requests a sync operation (non-blocking).

type TenantInfo added in v0.3.0 

type TenantInfo struct {
	// TenantID is the organization/user identifier (required).
	TenantID string

	// TeamID is the team identifier (optional).
	TeamID string

	// ProjectID is the project identifier (optional).
	ProjectID string
}

TenantInfo holds tenant context for filtering and isolation.

Multi-tenancy hierarchy:

  • TenantID (required): Organization or user identifier
  • TeamID (optional): Team scope within tenant
  • ProjectID (optional): Project scope within team

Security: All fields are validated before use in queries.

func ExtractTenantFromFilters added in v0.3.0 

func ExtractTenantFromFilters(filters map[string]interface{}) (*TenantInfo, error)

ExtractTenantFromFilters creates a TenantInfo from filter map. Returns ErrMissingTenant if required fields are missing. Returns ErrInvalidTenant if tenant_id is not a valid string.

func MustTenantFromContext added in v0.3.0 

func MustTenantFromContext(ctx context.Context) *TenantInfo

MustTenantFromContext extracts TenantInfo from context or panics. Use only when tenant presence is guaranteed by middleware.

func TenantFromContext added in v0.3.0 

func TenantFromContext(ctx context.Context) (*TenantInfo, error)

TenantFromContext extracts TenantInfo from a context. Returns ErrMissingTenant if not present - fail closed.

func (*TenantInfo) TenantFilter added in v0.3.0 

func (t *TenantInfo) TenantFilter() map[string]interface{}

TenantFilter returns filter conditions for queries. Returns conditions that match this tenant's scope.

func (*TenantInfo) TenantMetadata added in v0.3.0 

func (t *TenantInfo) TenantMetadata() map[string]interface{}

TenantMetadata returns tenant info as a metadata map for document storage.

func (*TenantInfo) Validate added in v0.3.0 

func (t *TenantInfo) Validate() error

Validate checks that required fields are present and valid.

type TroubleshootAdapter 

type TroubleshootAdapter struct {
	// contains filtered or unexported fields
}

TroubleshootAdapter adapts Store to implement troubleshoot.VectorStore interface.

func NewTroubleshootAdapter 

func NewTroubleshootAdapter(store Store) *TroubleshootAdapter

NewTroubleshootAdapter creates an adapter for troubleshoot service.

func (*TroubleshootAdapter) AddDocuments 

func (a *TroubleshootAdapter) AddDocuments(ctx context.Context, docs []Document) error

AddDocuments adds documents to the vector store. Returns nil on success (discards the returned IDs since troubleshoot doesn't need them).

func (*TroubleshootAdapter) SearchWithFilters 

func (a *TroubleshootAdapter) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)

SearchWithFilters performs similarity search with metadata filters.

type WAL added in v0.4.0 

type WAL struct {
	// contains filtered or unexported fields
}

WAL manages a write-ahead log for pending sync operations.

func NewWAL added in v0.4.0 

func NewWAL(path string, scrubber secrets.Scrubber, logger *zap.Logger) (*WAL, error)

NewWAL creates a new Write-Ahead Log.

func (*WAL) Close added in v0.4.0 

func (w *WAL) Close() error

Close closes the WAL (currently a no-op, but future-proof).

func (*WAL) Compact added in v0.4.0 

func (w *WAL) Compact(retentionDays int) error

Compact removes synced entries older than retention period.

func (*WAL) MarkSynced added in v0.4.0 

func (w *WAL) MarkSynced(id string) error

MarkSynced marks an entry as synced.

func (*WAL) PendingEntries added in v0.4.0 

func (w *WAL) PendingEntries() []WALEntry

PendingEntries returns all unsynced entries.

func (*WAL) RecordSyncAttempt added in v0.4.0 

func (w *WAL) RecordSyncAttempt(id string, err error) error

RecordSyncAttempt records a failed sync attempt.

func (*WAL) WriteEntry added in v0.4.0 

func (w *WAL) WriteEntry(ctx context.Context, entry WALEntry) error

WriteEntry writes a new entry to the WAL with security controls.

type WALEntry added in v0.4.0 

type WALEntry struct {
	ID           string
	Operation    string     // "add", "delete" - validated against whitelist
	Docs         []Document // For add operations (scrubbed before write)
	IDs          []string   // For delete operations
	Timestamp    time.Time
	Synced       bool
	Checksum     []byte    // HMAC-SHA256 of entry content
	RemoteState  string    // "unknown", "exists", "deleted"
	SyncAttempts int       // Number of sync attempts
	LastAttempt  time.Time // When last sync attempted
	SyncError    string    // Last error message (for debugging)
}

WALEntry represents a single write-ahead log entry.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.