Documentation
¶
Index ¶
- Constants
- func NormalizeVectorElementType(elementType string) string
- type BulkIndexer
- type BulkIndexerProvider
- type CompositeSearch
- func (c *CompositeSearch) BulkIndexer() BulkIndexer
- func (c *CompositeSearch) Clear(ctx context.Context) error
- func (c *CompositeSearch) Delete(ctx context.Context, postIDs []string) error
- func (c *CompositeSearch) DeleteOrphaned(ctx context.Context, nowTime, batchSize int64) (int64, error)
- func (c *CompositeSearch) Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)
- func (c *CompositeSearch) Store(ctx context.Context, docs []PostDocument) error
- type EmbeddingProvider
- type EmbeddingSearch
- type EmbeddingSearchConfig
- func (c *EmbeddingSearchConfig) EffectiveReindexIndexStrategy() string
- func (c *EmbeddingSearchConfig) GetHNSWM() int
- func (c *EmbeddingSearchConfig) GetIndexRetentionDays() int
- func (c *EmbeddingSearchConfig) GetModelName() string
- func (c *EmbeddingSearchConfig) GetProviderType() string
- func (c *EmbeddingSearchConfig) GetRecencyBiasSettings() RecencyBiasSettings
- func (c *EmbeddingSearchConfig) GetReindexBatchSize() int
- func (c *EmbeddingSearchConfig) GetReindexWorkers() int
- func (c *EmbeddingSearchConfig) GetVectorElementType() string
- func (c *EmbeddingSearchConfig) IndexRetentionFloor(nowMillis int64) int64
- type PostDocument
- type RecencyBiasSettings
- type SchemaChecker
- type SearchOptions
- type SearchResult
- type UpstreamConfig
- type VectorStore
Constants ¶
const ( ProviderTypeOpenAI = "openai" ProviderTypeOpenAICompatible = "openai-compatible" ProviderTypeBifrost = "bifrost" ProviderTypeMock = "mock" )
Provider types
const ( DefaultReindexWorkers = 4 MaxReindexWorkers = 32 DefaultReindexBatchSize = 200 MaxReindexBatchSize = 1000 )
Reindex throughput defaults and bounds. Workers are concurrent embedding pipelines; batch size is posts fetched/embedded per request. Defaults sit comfortably inside OpenAI Tier 1 rate limits.
const ( DefaultHNSWM = 8 MinHNSWM = 2 MaxHNSWM = 100 )
HNSW m (graph connections per row). pgvector accepts 2–100; 8 is a lower-RAM default than pgvector's own 16.
const ( VectorElementTypeVector = "vector" VectorElementTypeHalfvec = "halfvec" )
Embedding column element types. "vector" is float32; "halfvec" is float16 (pgvector 0.7+). Unset or unknown values normalize to vector.
const ( ReindexIndexStrategyMaintain = "maintain" ReindexIndexStrategyDefer = "defer" )
ReindexIndexStrategy*: maintain updates ANN during load; defer rebuilds after.
const ( DefaultRecencyHalfLifeDays = 7.0 DefaultRecencyFloor = 0.7 )
Recency bias defaults, chosen to match common time-decay ranking practice (Elasticsearch function_score exp decay, Azure AI Search freshness boosting). A 7-day half-life suits workplace chat, where this week's messages matter far more than last quarter's. The 0.7 floor bounds the worst-case demotion so a strong old match (raw 0.9 -> 0.63) still outranks a weak fresh one (raw <0.63), keeping old canonical answers findable.
const MaxSearchResults = 1000
MaxSearchResults is the hard cap on rows a vector store returns for a single search; stores clamp unset (<=0) or larger limits to this value.
const MillisPerDay int64 = 24 * 60 * 60 * 1000
MillisPerDay is the Unix-millis length of a 24-hour day.
const (
SearchTypeComposite = "composite"
)
Search types
const (
VectorStoreTypePGVector = "pgvector"
)
Vector store types
Variables ¶
This section is empty.
Functions ¶
func NormalizeVectorElementType ¶ added in v2.5.2
NormalizeVectorElementType keeps only "halfvec"; anything else (including unset) is "vector".
Types ¶
type BulkIndexer ¶
type BulkIndexer interface {
PrepareBulkIndex(ctx context.Context) error
FinalizeBulkIndex(ctx context.Context) error
VectorIndexExists(ctx context.Context) (bool, error)
}
BulkIndexer drops/rebuilds the ANN index around bulk loads.
type BulkIndexerProvider ¶
type BulkIndexerProvider interface {
// BulkIndexer returns control, or nil if unsupported.
BulkIndexer() BulkIndexer
}
BulkIndexerProvider exposes bulk index control from a search service.
type CompositeSearch ¶
type CompositeSearch struct {
// contains filtered or unexported fields
}
CompositeSearch implements EmbeddingSearch using separate vector store and embedding provider
func NewCompositeSearch ¶
func NewCompositeSearch(store VectorStore, provider EmbeddingProvider, options chunking.Options, recency RecencyBiasSettings) *CompositeSearch
NewCompositeSearch creates a new CompositeSearch with required chunking options
func (*CompositeSearch) BulkIndexer ¶
func (c *CompositeSearch) BulkIndexer() BulkIndexer
BulkIndexer returns the store's bulk control, or nil.
func (*CompositeSearch) Clear ¶
func (c *CompositeSearch) Clear(ctx context.Context) error
Clear removes all documents and chunks
func (*CompositeSearch) Delete ¶
func (c *CompositeSearch) Delete(ctx context.Context, postIDs []string) error
Delete removes documents and their chunks
func (*CompositeSearch) DeleteOrphaned ¶
func (c *CompositeSearch) DeleteOrphaned(ctx context.Context, nowTime, batchSize int64) (int64, error)
DeleteOrphaned removes embeddings whose posts no longer exist or are past retention.
func (*CompositeSearch) Search ¶
func (c *CompositeSearch) Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)
Search performs a semantic search and merges results from chunks of the same document
func (*CompositeSearch) Store ¶
func (c *CompositeSearch) Store(ctx context.Context, docs []PostDocument) error
Store chunks documents, generates embeddings, and stores them
type EmbeddingProvider ¶
type EmbeddingProvider interface {
// CreateEmbedding generates embedding for the given text
CreateEmbedding(ctx context.Context, text string) ([]float32, error)
// BatchCreateEmbeddings generates embeddings for any number of texts;
// implementations are responsible for splitting the batch to respect
// their provider's per-request limits
BatchCreateEmbeddings(ctx context.Context, texts []string) ([][]float32, error)
// Dimensions returns the dimensionality of the embeddings
Dimensions() int
}
EmbeddingProvider defines the interface for embedding generation
func NewMockEmbeddingProvider ¶
func NewMockEmbeddingProvider(dimensions int) EmbeddingProvider
NewMockEmbeddingProvider creates a new mock embedding provider that produces repeatable vectors.
type EmbeddingSearch ¶
type EmbeddingSearch interface {
// Store stores documents and handles embedding generation internally
Store(ctx context.Context, docs []PostDocument) error
// Search performs a similarity search using the query text
Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)
// Delete removes documents
Delete(ctx context.Context, postIDs []string) error
// Clear removes all documents
Clear(ctx context.Context) error
// DeleteOrphaned removes embeddings whose posts no longer exist or are past retention.
// nowTime is the retention cutoff (Unix millis), batchSize limits rows deleted per call.
// Returns the number of rows deleted.
DeleteOrphaned(ctx context.Context, nowTime, batchSize int64) (int64, error)
}
EmbeddingSearch defines the high-level interface for storing and searching using embeddings
type EmbeddingSearchConfig ¶
type EmbeddingSearchConfig struct {
Type string `json:"type"`
VectorStore UpstreamConfig `json:"vectorStore"`
EmbeddingProvider UpstreamConfig `json:"embeddingProvider"`
Parameters json.RawMessage `json:"parameters"`
Dimensions int `json:"dimensions"`
HNSWM int `json:"hnswM,omitempty"`
VectorElementType string `json:"vectorElementType,omitempty"`
ChunkingOptions chunking.Options `json:"chunkingOptions"`
ReindexWorkers int `json:"reindexWorkers,omitempty"`
ReindexBatchSize int `json:"reindexBatchSize,omitempty"`
ReindexIndexStrategy string `json:"reindexIndexStrategy,omitempty"`
RecencyBiasEnabled bool `json:"recencyBiasEnabled,omitempty"`
RecencyHalfLifeDays float64 `json:"recencyHalfLifeDays,omitempty"`
RecencyFloor float64 `json:"recencyFloor,omitempty"`
IndexRetentionDays int `json:"indexRetentionDays,omitempty"`
}
ServiceConfig holds configuration for the embedding search service
func (*EmbeddingSearchConfig) EffectiveReindexIndexStrategy ¶
func (c *EmbeddingSearchConfig) EffectiveReindexIndexStrategy() string
EffectiveReindexIndexStrategy: defer if set, otherwise maintain.
func (*EmbeddingSearchConfig) GetHNSWM ¶ added in v2.5.2
func (c *EmbeddingSearchConfig) GetHNSWM() int
GetHNSWM returns the configured HNSW m, clamped to pgvector's [2, 100] range, with unset (<=0) falling back to the default.
func (*EmbeddingSearchConfig) GetIndexRetentionDays ¶ added in v2.5.2
func (c *EmbeddingSearchConfig) GetIndexRetentionDays() int
GetIndexRetentionDays returns the configured retention window in days. Negative values are treated as 0 (index all posts).
func (*EmbeddingSearchConfig) GetModelName ¶
func (c *EmbeddingSearchConfig) GetModelName() string
GetModelName extracts the model name from the embedding provider parameters
func (*EmbeddingSearchConfig) GetProviderType ¶
func (c *EmbeddingSearchConfig) GetProviderType() string
GetProviderType returns the embedding provider type
func (*EmbeddingSearchConfig) GetRecencyBiasSettings ¶ added in v2.6.0
func (c *EmbeddingSearchConfig) GetRecencyBiasSettings() RecencyBiasSettings
GetRecencyBiasSettings resolves the recency bias config: unset (<=0) half-life and floor fall back to defaults; the floor is clamped to [0, 1]. These are query-time-only settings; changing them never requires a reindex.
func (*EmbeddingSearchConfig) GetReindexBatchSize ¶
func (c *EmbeddingSearchConfig) GetReindexBatchSize() int
GetReindexBatchSize returns the configured reindex batch size, clamped to valid bounds, with unset (<=0) falling back to the default.
func (*EmbeddingSearchConfig) GetReindexWorkers ¶
func (c *EmbeddingSearchConfig) GetReindexWorkers() int
GetReindexWorkers returns the configured reindex worker count, clamped to valid bounds, with unset (<=0) falling back to the default.
func (*EmbeddingSearchConfig) GetVectorElementType ¶ added in v2.5.2
func (c *EmbeddingSearchConfig) GetVectorElementType() string
GetVectorElementType returns the configured embedding column type.
func (*EmbeddingSearchConfig) IndexRetentionFloor ¶ added in v2.5.2
func (c *EmbeddingSearchConfig) IndexRetentionFloor(nowMillis int64) int64
IndexRetentionFloor is the inclusive CreateAt lower bound for indexing writes. 0 days (all posts) returns 0, meaning no lower bound. The result is never negative.
type PostDocument ¶
type PostDocument struct {
PostID string // ID of the Mattermost post
CreateAt int64 // Creation timestamp of the referenced post, not when this was indexed
TeamID string
ChannelID string
UserID string
Content string
// Embed chunk info to track if this is a chunk
chunking.ChunkInfo
}
PostDocument represents a Mattermost post with its metadata
type RecencyBiasSettings ¶ added in v2.6.0
type RecencyBiasSettings struct {
Enabled bool
HalfLifeDays float64
// Floor is the minimum decay multiplier in [0, 1]; it bounds the
// worst-case demotion of old results to Floor x their raw score.
Floor float64
}
RecencyBiasSettings holds resolved (defaulted and clamped) recency reranking parameters used by CompositeSearch.
type SchemaChecker ¶ added in v2.5.2
SchemaChecker is an optional VectorStore preflight. CompositeSearch calls it before generating embeddings so a column-type mismatch does not bill the provider. Clear must still succeed so Full Reindex can repair the schema.
type SearchOptions ¶
type SearchOptions struct {
Limit int
Offset int
MinScore float32
TeamID string
ChannelID string
UserID string // User ID for permission checks
CreatedAfter int64
CreatedBefore int64
}
SearchOptions contains parameters for search operations
type SearchResult ¶
type SearchResult struct {
Document PostDocument
Score float32
}
SearchResult represents a single search result with its similarity score
type UpstreamConfig ¶
type UpstreamConfig struct {
Type string `json:"type"`
Parameters json.RawMessage `json:"parameters"`
}
UpstreamConfig holds configuration for the upstream service
type VectorStore ¶
type VectorStore interface {
// Store stores documents and their embeddings
Store(ctx context.Context, docs []PostDocument, embeddings [][]float32) error
// Search performs a similarity search using the provided embedding
Search(ctx context.Context, embedding []float32, opts SearchOptions) ([]SearchResult, error)
// Delete removes documents from the vector store
Delete(ctx context.Context, postIDs []string) error
// Clear removes all documents from the vector store
Clear(ctx context.Context) error
// DeleteOrphaned removes embeddings whose posts no longer exist or are past retention.
// nowTime is the retention cutoff (Unix millis), batchSize limits rows deleted per call.
// Returns the number of rows deleted.
DeleteOrphaned(ctx context.Context, nowTime, batchSize int64) (int64, error)
}
VectorStore defines the interface for vector storage and search operations