Documentation
¶
Overview ¶
Package vector provides the vector store used for Genie's semantic memory: embedding and searching over documents (runbooks, synced data from Drive, Gmail, Slack, etc.) so the agent can retrieve relevant context via memory_search.
It solves the problem of giving the agent access to large, heterogeneous corpora: items are embedded, stored in an IStore (in-memory or Qdrant), and queried by semantic similarity. The sync pipeline upserts NormalizedItems from data sources; tools expose search and (when configured) delete. Without this package, the agent would have no persistent, searchable memory across runs.
Index ¶
- Constants
- func ChunkTextForEmbedding(text string) []string
- func NewMemorySearchTool(store IStore, cfg *Config) tool.Tool
- func NewMemoryStoreTool(store IStore, cfg *Config) tool.Tool
- type BatchItem
- type Config
- type IStore
- type MemorySearchRequest
- type MemorySearchResponse
- type MemorySearchResultItem
- type MemoryStoreRequest
- type MemoryStoreResponse
- type SearchResult
- type SearchResults
- type Store
- func (s *Store) Add(ctx context.Context, items ...BatchItem) error
- func (s *Store) Close(ctx context.Context) error
- func (s *Store) Delete(ctx context.Context, ids ...string) error
- func (s *Store) Search(ctx context.Context, query string, limit int) ([]SearchResult, error)
- func (s *Store) SearchWithFilter(ctx context.Context, query string, limit int, filter map[string]string) ([]SearchResult, error)
- func (s *Store) Upsert(ctx context.Context, items ...BatchItem) error
- type ToolProvider
Constants ¶
const ( MemoryStoreToolName = "memory_store" MemorySearchToolName = "memory_search" )
Tool name constants for the vector memory tools. Use these instead of magic strings when referencing memory tools elsewhere (e.g. retrieval tool classification, empty-memory guard, loop detection).
const MaxCharsPerEmbeddingChunk = 16_000
MaxCharsPerEmbeddingChunk is a safe character limit so that chunked text stays under embedding model token limits (e.g. OpenAI text-embedding-3-small 8191 tokens). Email/HTML can be ~2 chars per token; 8000 tokens ≈ 16000 chars. Used with trpc-agent-go's knowledge/chunking FixedSizeChunking strategy.
Variables ¶
This section is empty.
Functions ¶
func ChunkTextForEmbedding ¶
ChunkTextForEmbedding splits text into chunks of at most MaxCharsPerEmbeddingChunk characters using trpc-agent-go's knowledge/chunking FixedSizeChunking (UTF-8 safe, rune-based). Returns a single chunk if text is short enough; otherwise multiple chunks suitable for embedding one at a time. See: https://github.com/trpc-group/trpc-agent-go/tree/main/examples/knowledge/sources/fixed-chunking
func NewMemorySearchTool ¶
NewMemorySearchTool creates a tool that searches the vector memory. When cfg.AllowedMetadataKeys is set, only those keys may be used in filter.
func NewMemoryStoreTool ¶
NewMemoryStoreTool creates a tool that stores text into the vector memory. When cfg.AllowedMetadataKeys is set, only those keys are accepted in metadata. If req.ID is set, the store is upserted (existing document with that ID is replaced).
Types ¶
type BatchItem ¶
BatchItem represents a single document to be stored via Add.
func ChunkContentToBatchItems ¶
func ChunkContentToBatchItems(itemID, content string, baseMeta map[string]string) ([]BatchItem, error)
ChunkContentToBatchItems uses trpc-agent-go's chunking to split content and returns BatchItems ready for Upsert (IDs, chunk metadata). Use this when you need both chunk text and stable chunk IDs/metadata from the library.
type Config ¶
type Config struct {
// PersistenceDir is the directory where the vector store snapshot is
// saved as a JSON file. If empty, the store is ephemeral (in-memory only).
// Note: PersistenceDir is ignored when using an external store (Qdrant),
// as those handle persistence internally.
PersistenceDir string `yaml:"persistence_dir,omitempty" toml:"persistence_dir,omitempty"`
EmbeddingProvider string `yaml:"embedding_provider,omitempty" toml:"embedding_provider,omitempty"` // "openai", "ollama", "huggingface", "gemini"
APIKey string `yaml:"api_key,omitempty" toml:"api_key,omitempty"`
OllamaURL string `yaml:"ollama_url,omitempty" toml:"ollama_url,omitempty"`
OllamaModel string `yaml:"ollama_model,omitempty" toml:"ollama_model,omitempty"`
HuggingFaceURL string `yaml:"huggingface_url,omitempty" toml:"huggingface_url,omitempty"`
GeminiAPIKey string `yaml:"gemini_api_key,omitempty" toml:"gemini_api_key,omitempty"`
GeminiModel string `yaml:"gemini_model,omitempty" toml:"gemini_model,omitempty"`
// VectorStoreProvider specifies the vector store backend to use.
// Options: "inmemory" (default), "qdrant"
VectorStoreProvider string `yaml:"vector_store_provider,omitempty" toml:"vector_store_provider,omitempty"`
// Qdrant configuration (only used when VectorStoreProvider is "qdrant")
Qdrant qdrantstore.Config `yaml:"qdrant,omitempty" toml:"qdrant,omitempty"`
// AllowedMetadataKeys optionally restricts which metadata keys may be used in
// memory_store and memory_search. If non-empty, only these keys are accepted
// for metadata (store) and filter (search), enabling product/category buckets.
AllowedMetadataKeys []string `yaml:"allowed_metadata_keys,omitempty" toml:"allowed_metadata_keys,omitempty"`
}
Config holds the configuration for the vector store. It supports OpenAI, Ollama (via OpenAI-compatible endpoint), HuggingFace Text-Embeddings-Inference, Gemini, and a deterministic dummy embedder for development and testing.
func DefaultConfig ¶
func DefaultConfig(ctx context.Context, sp security.SecretProvider) Config
DefaultConfig builds the default vector store configuration by resolving API keys and endpoints through the given SecretProvider. Without a SecretProvider, callers can pass security.NewEnvProvider() to preserve the legacy os.Getenv behavior.
type IStore ¶
type IStore interface {
Search(ctx context.Context, query string, limit int) ([]SearchResult, error)
SearchWithFilter(ctx context.Context, query string, limit int, filter map[string]string) ([]SearchResult, error)
Add(ctx context.Context, items ...BatchItem) error
// Upsert replaces existing documents with the same ID, or inserts if not present.
// Use a stable ID (e.g. source:external_id) to overwrite memory when appropriate.
Upsert(ctx context.Context, items ...BatchItem) error
Delete(ctx context.Context, ids ...string) error
Close(ctx context.Context) error
}
type MemorySearchRequest ¶
type MemorySearchRequest struct {
Query string `json:"query" jsonschema:"description=The search query to find relevant memories,required"`
Limit int `json:"limit,omitempty" jsonschema:"description=Maximum number of results to return (default 5)"`
Filter map[string]string `` /* 153-byte string literal not displayed */
}
MemorySearchRequest is the input for the memory_search tool.
type MemorySearchResponse ¶
type MemorySearchResponse struct {
Results []MemorySearchResultItem `json:"results"`
Count int `json:"count"`
}
MemorySearchResponse is the output for the memory_search tool.
func (MemorySearchResponse) MarshalJSON ¶
func (r MemorySearchResponse) MarshalJSON() ([]byte, error)
MarshalJSON implements custom JSON marshaling for tool responses.
type MemorySearchResultItem ¶
type MemorySearchResultItem struct {
Content string `json:"content"`
Metadata map[string]string `json:"metadata,omitempty"`
Similarity float64 `json:"similarity"`
}
MemorySearchResultItem represents a single search result from the memory_search tool.
type MemoryStoreRequest ¶
type MemoryStoreRequest struct {
Text string `json:"text" jsonschema:"description=The text content to store in memory,required"`
Metadata map[string]string `` /* 144-byte string literal not displayed */
ID string `` /* 133-byte string literal not displayed */
}
MemoryStoreRequest is the input for the memory_store tool.
type MemoryStoreResponse ¶
MemoryStoreResponse is the output for the memory_store tool.
func (MemoryStoreResponse) MarshalJSON ¶
func (r MemoryStoreResponse) MarshalJSON() ([]byte, error)
MarshalJSON implements custom JSON marshaling for tool responses.
type SearchResult ¶
type SearchResult struct {
ID string `json:"id"`
Content string `json:"content"`
Metadata map[string]string `json:"metadata,omitempty"`
Score float64 `json:"score"`
}
SearchResult represents a single result returned by Store.Search. It contains the matched document content, its metadata and the cosine similarity score (0.0–1.0, higher is more similar).
func (SearchResult) String ¶
func (s SearchResult) String() string
type SearchResults ¶
type SearchResults []SearchResult
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store wraps a trpc-agent-go vector store and embedder to provide simple add/search operations for agent memory. When PersistenceDir is set and using in-memory store, the store snapshots its state to disk after every Add and restores it on startup. When using Qdrant, persistence is handled by the external store itself.
func (*Store) Add ¶
Add stores one or more documents in the vector store. When multiple items are provided, their embeddings are generated concurrently via errgroup, reducing wall-clock latency from N×round-trip to max(round-trip). A single disk snapshot is taken at the end.
func (*Store) Close ¶
Close flushes any pending state to disk (if persistence is configured). For Qdrant stores, it closes the client connection. It is safe to call multiple times.
func (*Store) Delete ¶
Delete removes one or more documents by their IDs from the vector store. A single snapshot is taken at the end. Errors from individual deletes are collected but do not stop processing of remaining items.
func (*Store) Search ¶
Search finds the most semantically similar documents to the query text. It returns up to limit results ordered by descending similarity.
func (*Store) SearchWithFilter ¶
func (s *Store) SearchWithFilter(ctx context.Context, query string, limit int, filter map[string]string) ([]SearchResult, error)
SearchWithFilter finds semantically similar documents, optionally filtered by metadata key-value pairs. Only documents whose metadata contains ALL specified filter entries are returned. Pass nil for unfiltered search. This enables source-based memory isolation (e.g. per-sender, per-channel).
type ToolProvider ¶
type ToolProvider struct {
// contains filtered or unexported fields
}
ToolProvider wraps an IStore and optional Config, and satisfies the tools.ToolProviders interface so vector memory tools can be passed directly to tools.NewRegistry. When cfg is non-nil and AllowedMetadataKeys is set, memory_store and memory_search only accept those keys for metadata and filter (product/category buckets).
func NewToolProvider ¶
func NewToolProvider(store IStore, cfg *Config) *ToolProvider
NewToolProvider creates a ToolProvider for the vector memory tools (memory_store and memory_search). cfg may be nil; when set with AllowedMetadataKeys, only those metadata keys are allowed.
func (*ToolProvider) GetTools ¶
func (p *ToolProvider) GetTools() []tool.Tool
GetTools returns the memory store and memory search tools.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package qdrantstore provides the Qdrant vector store backend for Genie's semantic memory.
|
Package qdrantstore provides the Qdrant vector store backend for Genie's semantic memory. |
|
Code generated by counterfeiter.
|
Code generated by counterfeiter. |