Documentation
¶
Overview ¶
Package vectorstore provides vector storage implementations.
Package vectorstore provides vector storage implementations.
Package vectorstore provides vector storage implementations.
Package vectorstore defines the interface for vector storage operations.
Package vectorstore provides vector storage implementations.
Index ¶
- Variables
- func IsTransientError(err error) bool
- func ValidateCollectionName(name string) error
- type ChromemConfig
- type ChromemStore
- func (s *ChromemStore) AddDocuments(ctx context.Context, docs []Document) ([]string, error)
- func (s *ChromemStore) Close() error
- func (s *ChromemStore) CollectionExists(ctx context.Context, collectionName string) (bool, error)
- func (s *ChromemStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error
- func (s *ChromemStore) DeleteCollection(ctx context.Context, collectionName string) error
- func (s *ChromemStore) DeleteDocuments(ctx context.Context, ids []string) error
- func (s *ChromemStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error
- func (s *ChromemStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)
- func (s *ChromemStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)
- func (s *ChromemStore) ListCollections(ctx context.Context) ([]string, error)
- func (s *ChromemStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)
- func (s *ChromemStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, ...) ([]SearchResult, error)
- func (s *ChromemStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
- type CollectionInfo
- type CompressionLevel
- type CompressionMetadata
- type Document
- type Embedder
- type QdrantConfig
- type QdrantStore
- func (s *QdrantStore) AddDocuments(ctx context.Context, docs []Document) ([]string, error)
- func (s *QdrantStore) Close() error
- func (s *QdrantStore) CollectionExists(ctx context.Context, collectionName string) (bool, error)
- func (s *QdrantStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error
- func (s *QdrantStore) DeleteCollection(ctx context.Context, collectionName string) error
- func (s *QdrantStore) DeleteDocuments(ctx context.Context, ids []string) error
- func (s *QdrantStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error
- func (s *QdrantStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)
- func (s *QdrantStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)
- func (s *QdrantStore) ListCollections(ctx context.Context) ([]string, error)
- func (s *QdrantStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)
- func (s *QdrantStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, ...) ([]SearchResult, error)
- func (s *QdrantStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
- type SearchResult
- type Store
- type TroubleshootAdapter
Constants ¶
This section is empty.
Variables ¶
var ( // ErrCollectionNotFound is returned when a collection does not exist. ErrCollectionNotFound = errors.New("collection not found") // ErrCollectionExists is returned when attempting to create an existing collection. ErrCollectionExists = errors.New("collection already exists") // ErrInvalidConfig indicates invalid configuration. ErrInvalidConfig = errors.New("invalid configuration") // ErrEmptyDocuments indicates empty or nil documents. ErrEmptyDocuments = errors.New("empty or nil documents") // ErrConnectionFailed indicates gRPC connection issues. ErrConnectionFailed = errors.New("failed to connect to Qdrant") // ErrEmbeddingFailed indicates embedding generation failure. ErrEmbeddingFailed = errors.New("failed to generate embeddings") // ErrInvalidCollectionName indicates collection name validation failure. ErrInvalidCollectionName = errors.New("invalid collection name") )
Sentinel errors for vector store operations.
Functions ¶
func IsTransientError ¶
IsTransientError checks if an error is transient (should retry). Returns true for network timeouts, temporary unavailability. Returns false for invalid config, not found, permission denied.
func ValidateCollectionName ¶
ValidateCollectionName validates a collection name against security rules. Pattern: ^[a-z0-9_]{1,64}$ Rejects: uppercase, special chars, path traversal, spaces.
Types ¶
type ChromemConfig ¶ added in v0.3.0
type ChromemConfig struct {
// Path is the directory for persistent storage.
// Default: "~/.config/contextd/vectorstore"
Path string
// Compress enables gzip compression for stored data.
// Note: This defaults to false (Go zero value). Set explicitly if compression is desired.
Compress bool
// DefaultCollection is the default collection name.
// Default: "contextd_default"
DefaultCollection string
// VectorSize is the expected embedding dimension.
// Must match the embedder's output dimension.
// Default: 384 (for FastEmbed bge-small-en-v1.5)
VectorSize int
}
ChromemConfig holds configuration for chromem-go embedded vector database.
func (*ChromemConfig) ApplyDefaults ¶ added in v0.3.0
func (c *ChromemConfig) ApplyDefaults()
ApplyDefaults sets default values for unset fields.
func (*ChromemConfig) Validate ¶ added in v0.3.0
func (c *ChromemConfig) Validate() error
Validate validates the configuration.
type ChromemStore ¶ added in v0.3.0
type ChromemStore struct {
// contains filtered or unexported fields
}
ChromemStore implements the Store interface using chromem-go.
chromem-go is an embeddable vector database with zero third-party dependencies. It provides in-memory storage with optional persistence to gob files.
Key features:
- Pure Go, no CGO required
- No external database service needed
- Fast similarity search (1000 docs in 0.3ms)
- Automatic persistence to disk
func NewChromemStore ¶ added in v0.3.0
func NewChromemStore(config ChromemConfig, embedder Embedder, logger *zap.Logger) (*ChromemStore, error)
NewChromemStore creates a new ChromemStore with the given configuration.
func (*ChromemStore) AddDocuments ¶ added in v0.3.0
AddDocuments adds documents to the vector store.
func (*ChromemStore) Close ¶ added in v0.3.0
func (s *ChromemStore) Close() error
Close closes the ChromemStore. Note: chromem-go handles persistence automatically, no explicit close needed.
func (*ChromemStore) CollectionExists ¶ added in v0.3.0
CollectionExists checks if a collection exists.
func (*ChromemStore) CreateCollection ¶ added in v0.3.0
func (s *ChromemStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error
CreateCollection creates a new collection with the specified configuration.
func (*ChromemStore) DeleteCollection ¶ added in v0.3.0
func (s *ChromemStore) DeleteCollection(ctx context.Context, collectionName string) error
DeleteCollection deletes a collection and all its documents.
func (*ChromemStore) DeleteDocuments ¶ added in v0.3.0
func (s *ChromemStore) DeleteDocuments(ctx context.Context, ids []string) error
DeleteDocuments deletes documents by their IDs from the default collection.
func (*ChromemStore) DeleteDocumentsFromCollection ¶ added in v0.3.0
func (s *ChromemStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error
DeleteDocumentsFromCollection deletes documents by their IDs from a specific collection.
func (*ChromemStore) ExactSearch ¶ added in v0.3.0
func (s *ChromemStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)
ExactSearch performs brute-force similarity search. Note: chromem-go always uses exact search (no HNSW), so this is the same as regular search.
func (*ChromemStore) GetCollectionInfo ¶ added in v0.3.0
func (s *ChromemStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)
GetCollectionInfo returns metadata about a collection.
func (*ChromemStore) ListCollections ¶ added in v0.3.0
func (s *ChromemStore) ListCollections(ctx context.Context) ([]string, error)
ListCollections returns a list of all collection names.
func (*ChromemStore) Search ¶ added in v0.3.0
func (s *ChromemStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)
Search performs similarity search in the default collection.
func (*ChromemStore) SearchInCollection ¶ added in v0.3.0
func (s *ChromemStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
SearchInCollection performs similarity search in a specific collection.
func (*ChromemStore) SearchWithFilters ¶ added in v0.3.0
func (s *ChromemStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
SearchWithFilters performs similarity search with metadata filters.
type CollectionInfo ¶
type CollectionInfo struct {
// Name is the collection name.
Name string `json:"name"`
// PointCount is the number of vectors in the collection.
PointCount int `json:"point_count"`
// VectorSize is the dimensionality of vectors in this collection.
VectorSize int `json:"vector_size"`
}
CollectionInfo contains metadata about a vector collection.
type CompressionLevel ¶
type CompressionLevel string
CompressionLevel represents the compression state of content.
const ( // CompressionLevelNone represents uncompressed original content. CompressionLevelNone CompressionLevel = "none" // CompressionLevelFolded represents context-folded content (partial compression). CompressionLevelFolded CompressionLevel = "folded" // CompressionLevelSummary represents summarized content (high compression). CompressionLevelSummary CompressionLevel = "summary" )
type CompressionMetadata ¶
type CompressionMetadata struct {
Level CompressionLevel `json:"compression_level"` // Current compression level
Algorithm string `json:"compression_algorithm,omitempty"` // Compression algorithm used
OriginalSize int `json:"original_size"` // Original content size (tokens/chars)
CompressedSize int `json:"compressed_size"` // Compressed content size
CompressionRatio float64 `json:"compression_ratio"` // Compression ratio (original/compressed)
CompressedAt *time.Time `json:"compressed_at,omitempty"` // When compression was applied
}
CompressionMetadata tracks compression state and metrics.
type Document ¶
type Document struct {
// ID is the unique identifier for the document
ID string
// Content is the text content of the document
Content string
// Metadata contains additional key-value pairs for filtering
// Common fields: owner, project, file, branch, timestamp
Metadata map[string]interface{}
// Collection is the target collection name for this document.
// If empty, uses the service's default collection.
//
// Collection naming convention:
// - Organization: org_{type} (e.g., org_memories)
// - Team: {team}_{type} (e.g., platform_memories)
// - Project: {team}_{project}_{type} (e.g., platform_contextd_memories)
Collection string
}
Document represents a document to be stored in the vector store.
type Embedder ¶
type Embedder interface {
// EmbedDocuments generates embeddings for multiple texts.
// Returns a slice of embeddings (one per input text) or an error.
EmbedDocuments(ctx context.Context, texts []string) ([][]float32, error)
// EmbedQuery generates an embedding for a single query.
// Some models optimize differently for queries vs documents.
EmbedQuery(ctx context.Context, text string) ([]float32, error)
}
Embedder generates vector embeddings from text.
Embeddings are dense numerical representations that capture semantic meaning, enabling similarity search. Implementations can use local models (TEI) or cloud APIs (OpenAI, Cohere).
type QdrantConfig ¶
type QdrantConfig struct {
// Host is the Qdrant server hostname or IP address.
// Default: "localhost"
Host string
// Port is the Qdrant gRPC port (NOT HTTP REST port).
// Default: 6334 (gRPC), not 6333 (HTTP)
Port int
// CollectionName is the default collection for operations.
// Format: {scope}_{type} for multi-tenancy
// Examples: org_memories, platform_memories, platform_contextd_memories
CollectionName string
// VectorSize is the dimensionality of embeddings.
// Examples: 384 (BAAI/bge-small-en-v1.5), 768 (BERT), 1536 (OpenAI)
// MUST match Embedder output dimensions.
VectorSize uint64
// Distance is the similarity metric for vector search.
// Options: Cosine (default), Euclid, Dot
Distance qdrant.Distance
// UseTLS enables TLS encryption for gRPC connection.
// Default: false (MVP), true (production)
UseTLS bool
// MaxRetries is the maximum number of retry attempts for transient failures.
// Default: 3
MaxRetries int
// RetryBackoff is the initial backoff duration for retries.
// Doubles on each retry (exponential backoff).
// Default: 1 second
RetryBackoff time.Duration
// MaxMessageSize is the maximum gRPC message size in bytes.
// Default: 50MB (to handle large documents)
MaxMessageSize int
// CircuitBreakerThreshold is the number of failures before opening circuit.
// Default: 5
CircuitBreakerThreshold int
}
QdrantConfig holds configuration for Qdrant gRPC client.
func (*QdrantConfig) ApplyDefaults ¶
func (c *QdrantConfig) ApplyDefaults()
ApplyDefaults sets default values for unset fields.
func (QdrantConfig) Validate ¶
func (c QdrantConfig) Validate() error
Validate validates the configuration.
type QdrantStore ¶
type QdrantStore struct {
// contains filtered or unexported fields
}
QdrantStore is a Store implementation using Qdrant's native gRPC client.
This implementation bypasses Qdrant's actix-web HTTP layer, eliminating the 256kB payload limit that causes 413 errors during repository indexing.
Key features:
- Native gRPC transport (port 6334)
- Binary protobuf encoding (no JSON size limits)
- Better performance than HTTP REST
- Full Qdrant feature access
- Collection-per-project isolation
func NewQdrantStore ¶
func NewQdrantStore(config QdrantConfig, embedder Embedder) (*QdrantStore, error)
NewQdrantStore creates a new QdrantStore with the given configuration.
The constructor performs the following steps:
- Validates configuration
- Creates Qdrant gRPC client
- Performs health check
- Returns ready-to-use store
Returns an error if:
- Configuration is invalid
- Connection to Qdrant fails
- Health check fails
func (*QdrantStore) AddDocuments ¶
AddDocuments adds documents to the vector store.
func (*QdrantStore) Close ¶
func (s *QdrantStore) Close() error
Close closes the Qdrant gRPC connection.
func (*QdrantStore) CollectionExists ¶
CollectionExists checks if a collection exists.
func (*QdrantStore) CreateCollection ¶
func (s *QdrantStore) CreateCollection(ctx context.Context, collectionName string, vectorSize int) error
CreateCollection creates a new collection with the specified configuration.
func (*QdrantStore) DeleteCollection ¶
func (s *QdrantStore) DeleteCollection(ctx context.Context, collectionName string) error
DeleteCollection deletes a collection and all its documents.
func (*QdrantStore) DeleteDocuments ¶
func (s *QdrantStore) DeleteDocuments(ctx context.Context, ids []string) error
DeleteDocuments deletes documents by their IDs from the default collection.
func (*QdrantStore) DeleteDocumentsFromCollection ¶
func (s *QdrantStore) DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error
DeleteDocumentsFromCollection deletes documents by their IDs from a specific collection.
func (*QdrantStore) ExactSearch ¶
func (s *QdrantStore) ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)
ExactSearch performs brute-force similarity search without using HNSW index. This is a fallback for small datasets (<10 vectors) where HNSW index may not be built.
func (*QdrantStore) GetCollectionInfo ¶
func (s *QdrantStore) GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)
GetCollectionInfo returns metadata about a collection.
func (*QdrantStore) ListCollections ¶
func (s *QdrantStore) ListCollections(ctx context.Context) ([]string, error)
ListCollections returns a list of all collection names.
func (*QdrantStore) Search ¶
func (s *QdrantStore) Search(ctx context.Context, query string, k int) ([]SearchResult, error)
Search performs similarity search in the default collection.
func (*QdrantStore) SearchInCollection ¶
func (s *QdrantStore) SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
SearchInCollection performs similarity search in a specific collection.
func (*QdrantStore) SearchWithFilters ¶
func (s *QdrantStore) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
SearchWithFilters performs similarity search with metadata filters.
type SearchResult ¶
type SearchResult struct {
// ID is the document identifier
ID string
// Content is the document text content
Content string
// Score is the similarity score (higher = more similar)
Score float32
// Metadata contains the document metadata
Metadata map[string]interface{}
}
SearchResult represents a search result from the vector store.
type Store ¶
type Store interface {
// AddDocuments adds documents to the vector store.
//
// Documents are embedded and stored with their metadata. The document ID
// is used as the unique identifier in the vector store.
//
// If Document.Collection is specified, the document is added to that collection.
// Otherwise, the implementation's default collection is used.
//
// Returns the IDs of added documents and an error if the operation fails.
AddDocuments(ctx context.Context, docs []Document) ([]string, error)
// Search performs similarity search in the default collection.
//
// It searches for documents similar to the query and returns up to k results
// ordered by similarity score (highest first).
//
// Returns search results with scores and metadata, or an error if search fails.
Search(ctx context.Context, query string, k int) ([]SearchResult, error)
// SearchWithFilters performs similarity search with metadata filters.
//
// Filters are applied to document metadata (e.g., {"owner": "alice"}).
// Only documents matching ALL filter conditions are returned.
//
// Returns filtered search results or an error if search fails.
SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
// SearchInCollection performs similarity search in a specific collection.
//
// This supports the hierarchical collection architecture by allowing searches
// in scope-specific collections (e.g., "org_memories", "platform_contextd_memories").
//
// Returns filtered search results from the specified collection, or an error.
SearchInCollection(ctx context.Context, collectionName string, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
// DeleteDocuments deletes documents by their IDs from the default collection.
//
// Returns an error if deletion fails.
DeleteDocuments(ctx context.Context, ids []string) error
// DeleteDocumentsFromCollection deletes documents by their IDs from a specific collection.
//
// Returns an error if deletion fails.
DeleteDocumentsFromCollection(ctx context.Context, collectionName string, ids []string) error
// CreateCollection creates a new collection with the specified configuration.
//
// Collections are namespaces for documents (e.g., project-specific collections).
// The vectorSize parameter specifies the dimensionality of embeddings.
//
// Returns an error if collection creation fails or collection already exists.
CreateCollection(ctx context.Context, collectionName string, vectorSize int) error
// DeleteCollection deletes a collection and all its documents.
//
// This is a destructive operation that cannot be undone.
//
// Returns an error if deletion fails or collection doesn't exist.
DeleteCollection(ctx context.Context, collectionName string) error
// CollectionExists checks if a collection exists.
//
// Returns true if the collection exists, false otherwise.
// Returns an error only if the check operation itself fails.
CollectionExists(ctx context.Context, collectionName string) (bool, error)
// ListCollections returns a list of all collection names.
//
// Returns collection names or an error if listing fails.
ListCollections(ctx context.Context) ([]string, error)
// GetCollectionInfo returns metadata about a collection.
//
// Returns collection info including point count and vector size.
// Returns ErrCollectionNotFound if the collection doesn't exist.
GetCollectionInfo(ctx context.Context, collectionName string) (*CollectionInfo, error)
// ExactSearch performs brute-force similarity search without using HNSW index.
//
// This is a fallback for small datasets (<10 vectors) where HNSW index
// may not be built. It performs exact cosine similarity on all vectors.
//
// Returns search results ordered by similarity score (highest first).
ExactSearch(ctx context.Context, collectionName string, query string, k int) ([]SearchResult, error)
// Close closes the vector store connection and releases resources.
Close() error
}
Store is the interface for vector storage operations.
This interface is transport-agnostic - implementations can use HTTP REST, gRPC, or any other protocol. The interface focuses on contextd's specific needs for document storage, search, and collection management.
Collection Naming Convention:
- Organization: org_{type} (e.g., org_memories)
- Team: {team}_{type} (e.g., platform_memories)
- Project: {team}_{project}_{type} (e.g., platform_contextd_memories)
Implementations:
- QdrantStore: Uses official Qdrant gRPC client (built-in)
- Future providers via plugins
func NewStore ¶ added in v0.3.0
NewStore creates a new Store based on the configuration.
This factory function examines the VectorStoreConfig.Provider field and creates the appropriate store implementation:
- "chromem" (default): Creates an embedded ChromemStore (no external deps)
- "qdrant": Creates a QdrantStore (requires external Qdrant server)
The chromem provider is recommended for most users as it requires no setup:
brew install contextd # Just works!
Example usage:
cfg := config.Load()
store, err := vectorstore.NewStore(cfg, embedder, logger)
if err != nil {
log.Fatal(err)
}
defer store.Close()
func NewStoreFromProvider ¶ added in v0.3.0
func NewStoreFromProvider(provider string, chromemCfg *ChromemConfig, qdrantCfg *QdrantConfig, embedder Embedder, logger *zap.Logger) (Store, error)
NewStoreFromProvider creates a store directly from provider name and specific config. This is useful when you need more control over configuration.
type TroubleshootAdapter ¶
type TroubleshootAdapter struct {
// contains filtered or unexported fields
}
TroubleshootAdapter adapts Store to implement troubleshoot.VectorStore interface.
func NewTroubleshootAdapter ¶
func NewTroubleshootAdapter(store Store) *TroubleshootAdapter
NewTroubleshootAdapter creates an adapter for troubleshoot service.
func (*TroubleshootAdapter) AddDocuments ¶
func (a *TroubleshootAdapter) AddDocuments(ctx context.Context, docs []Document) error
AddDocuments adds documents to the vector store. Returns nil on success (discards the returned IDs since troubleshoot doesn't need them).
func (*TroubleshootAdapter) SearchWithFilters ¶
func (a *TroubleshootAdapter) SearchWithFilters(ctx context.Context, query string, k int, filters map[string]interface{}) ([]SearchResult, error)
SearchWithFilters performs similarity search with metadata filters.