Documentation
¶
Overview ¶
Package memory – embeddings.go implements embedding generation for semantic search. Supports multiple providers: OpenAI, Gemini, Voyage, Mistral, and a zero-cost null fallback. Embeddings are cached by content hash + provider + model to avoid redundant API calls.
Package memory – embeddings_gemini.go implements Google Gemini embedding provider. Uses the Gemini REST API with support for single and batch embedding requests.
Package memory – embeddings_mistral.go implements Mistral embedding provider. Uses the OpenAI-compatible /embeddings endpoint.
Package memory – embeddings_voyage.go implements Voyage AI embedding provider. Uses the OpenAI-compatible /embeddings endpoint with Voyage-specific input_type field.
Package memory – indexer.go implements markdown chunking and hash-based delta sync for the memory index. Files are split into chunks, each chunk gets a SHA-256 hash, and only changed chunks are re-embedded.
Package memory – query_expansion.go handles keyword extraction and FTS5 query building. Supports stop words in English, Portuguese, Spanish, and French. Filters out pure numbers, all-punctuation tokens, and very short tokens.
Package memory – sqlite_store.go implements a SQLite-backed memory store with FTS5 (BM25 ranking) and in-process vector search (cosine similarity). Embeddings are stored as JSON-encoded float32 arrays in the chunks table. This avoids the need for the sqlite-vec extension while still providing hybrid semantic + keyword search.
Package memory implements persistent memory for DevClaw. Provides long-term fact storage and daily conversation logs using the filesystem (MEMORY.md + daily markdown files).
Architecture:
- MEMORY.md: Long-term facts (append-only, curated by the agent)
- memory/YYYY-MM-DD.md: Daily conversation summaries (append-only)
- Search uses simple substring matching (future: BM25 / embeddings)
Index ¶
- func FileHash(path string) (string, error)
- func IndexDirectory(dir string, cfg ChunkConfig) (map[string][]Chunk, error)
- type Chunk
- type ChunkConfig
- type EmbeddingConfig
- type EmbeddingProvider
- type Entry
- type FallbackEmbedder
- type FileStore
- func (fs *FileStore) GetAll() ([]Entry, error)
- func (fs *FileStore) GetByDate(date time.Time) ([]Entry, error)
- func (fs *FileStore) GetRecent(limit int) ([]Entry, error)
- func (fs *FileStore) ListDailyLogs() ([]string, error)
- func (fs *FileStore) RecentFacts(maxFacts int, query string) string
- func (fs *FileStore) Save(entry Entry) error
- func (fs *FileStore) SaveDailyLog(date time.Time, content string) error
- func (fs *FileStore) Search(query string, maxResults int) ([]Entry, error)
- type GeminiEmbedder
- type MMRConfig
- type MistralEmbedder
- type NullEmbedder
- type OpenAIEmbedder
- type SQLiteStore
- func (s *SQLiteStore) ApplyMMR(results []SearchResult, cfg MMRConfig, maxResults int) []SearchResult
- func (s *SQLiteStore) ApplyTemporalDecay(results []SearchResult, cfg TemporalDecayConfig) []SearchResult
- func (s *SQLiteStore) ChunkCount() int
- func (s *SQLiteStore) Close() error
- func (s *SQLiteStore) FileCount() int
- func (s *SQLiteStore) HybridSearch(ctx context.Context, query string, maxResults int, minScore float64, ...) ([]SearchResult, error)
- func (s *SQLiteStore) HybridSearchWithOptions(ctx context.Context, query string, maxResults int, minScore float64, ...) ([]SearchResult, error)
- func (s *SQLiteStore) IndexChunks(ctx context.Context, fileID string, chunks []Chunk, fileHash string) error
- func (s *SQLiteStore) IndexMemoryDir(ctx context.Context, memDir string, chunkCfg ChunkConfig) error
- func (s *SQLiteStore) IndexTranscript(ctx context.Context, sessionID string, entries []TranscriptEntry) error
- func (s *SQLiteStore) PruneEmbeddingCache(maxEntries int)
- func (s *SQLiteStore) SearchBM25(query string, maxResults int) ([]SearchResult, error)
- func (s *SQLiteStore) SearchVector(ctx context.Context, query string, maxResults int) ([]SearchResult, error)
- type SearchResult
- type Store
- type TemporalDecayConfig
- type TranscriptEntry
- type VoyageEmbedder
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func IndexDirectory ¶
func IndexDirectory(dir string, cfg ChunkConfig) (map[string][]Chunk, error)
IndexDirectory scans a directory for .md files and chunks them. Returns a map of fileID → []Chunk. The fileID is the relative path.
Types ¶
type Chunk ¶
type Chunk struct {
// FileID identifies the source file (relative path).
FileID string
// Index is the chunk position within the file (0-based).
Index int
// Text is the chunk content.
Text string
// Hash is the SHA-256 hex digest of the chunk text.
Hash string
}
Chunk represents an indexed text fragment from a memory file.
func ChunkMarkdown ¶
func ChunkMarkdown(text string, cfg ChunkConfig) []Chunk
ChunkMarkdown splits markdown text into semantic chunks. Prefers splitting at heading boundaries, then paragraph boundaries, then sentence boundaries, respecting MaxTokens.
type ChunkConfig ¶
type ChunkConfig struct {
// MaxTokens is the approximate max tokens per chunk (default: 500).
// Uses ~4 chars/token heuristic.
MaxTokens int
// Overlap is the number of characters to overlap between chunks (default: 100).
Overlap int
}
ChunkConfig controls the chunking behavior.
func DefaultChunkConfig ¶
func DefaultChunkConfig() ChunkConfig
DefaultChunkConfig returns sensible defaults.
type EmbeddingConfig ¶
type EmbeddingConfig struct {
// Provider is the embedding provider ("openai", "gemini", "voyage", "mistral", "auto", "none").
Provider string `yaml:"provider"`
// Model is the embedding model name (e.g. "text-embedding-3-small").
Model string `yaml:"model"`
// Dimensions is the output vector dimensionality (default: auto from model).
Dimensions int `yaml:"dimensions"`
// APIKey is the API key for the embedding provider. If empty, falls back to
// the main LLM API key or provider-specific env vars.
APIKey string `yaml:"api_key"`
// BaseURL is the API base URL. If empty, uses the provider default.
BaseURL string `yaml:"base_url"`
// Cache enables embedding caching in SQLite (default: true).
Cache bool `yaml:"cache"`
// Fallback is the fallback provider when the primary fails ("openai", "gemini", etc., or "none").
Fallback string `yaml:"fallback"`
// FallbackAPIKey is the API key for the fallback provider.
FallbackAPIKey string `yaml:"fallback_api_key"`
// FallbackBaseURL is the base URL for the fallback provider.
FallbackBaseURL string `yaml:"fallback_base_url"`
// FallbackModel is the model for the fallback provider.
FallbackModel string `yaml:"fallback_model"`
}
EmbeddingConfig configures the embedding provider.
func DefaultEmbeddingConfig ¶
func DefaultEmbeddingConfig() EmbeddingConfig
DefaultEmbeddingConfig returns sensible defaults.
type EmbeddingProvider ¶
type EmbeddingProvider interface {
// Embed generates embeddings for a batch of texts.
// Returns one float32 vector per input text.
Embed(ctx context.Context, texts []string) ([][]float32, error)
// Dimensions returns the dimensionality of the output vectors.
Dimensions() int
// Name returns the provider name (for cache key derivation).
Name() string
// Model returns the model name (for cache key derivation).
Model() string
}
EmbeddingProvider generates vector embeddings from text.
func NewEmbeddingProvider ¶
func NewEmbeddingProvider(cfg EmbeddingConfig) EmbeddingProvider
NewEmbeddingProvider creates an embedding provider from config. When a fallback is configured, wraps with FallbackEmbedder for automatic failover.
type Entry ¶
type Entry struct {
Content string `json:"content"`
Source string `json:"source"` // "user", "agent", "system"
Category string `json:"category"` // "fact", "preference", "event", "summary"
Timestamp time.Time `json:"timestamp"`
}
Entry represents a single memory fact or event.
type FallbackEmbedder ¶ added in v1.13.0
type FallbackEmbedder struct {
// contains filtered or unexported fields
}
FallbackEmbedder wraps a primary and fallback provider. On primary failure, automatically retries with the fallback. If both fail, returns an error; callers should degrade to FTS-only search.
func NewFallbackEmbedder ¶ added in v1.13.0
func NewFallbackEmbedder(primary, fallback EmbeddingProvider, logger *slog.Logger) *FallbackEmbedder
NewFallbackEmbedder creates a fallback-enabled embedder.
func (*FallbackEmbedder) Dimensions ¶ added in v1.13.0
func (f *FallbackEmbedder) Dimensions() int
Dimensions returns the primary provider's dimensions.
func (*FallbackEmbedder) Embed ¶ added in v1.13.0
Embed tries the primary provider, falling back on error.
func (*FallbackEmbedder) Model ¶ added in v1.13.0
func (f *FallbackEmbedder) Model() string
Model returns the primary provider's model.
func (*FallbackEmbedder) Name ¶ added in v1.13.0
func (f *FallbackEmbedder) Name() string
Name returns "fallback:{primary}" — only the primary is used for cache keys since embeddings from different models are not compatible.
type FileStore ¶
type FileStore struct {
// contains filtered or unexported fields
}
FileStore implements Store using the filesystem. Uses MEMORY.md for long-term facts and daily markdown files for logs.
func NewFileStore ¶
NewFileStore creates a file-based memory store at the given directory.
func (*FileStore) ListDailyLogs ¶
ListDailyLogs returns the dates of all daily log files, sorted newest first.
func (*FileStore) RecentFacts ¶
RecentFacts returns a formatted string of recent facts suitable for injection into the system prompt.
func (*FileStore) SaveDailyLog ¶
SaveDailyLog appends a conversation summary to the daily log file.
type GeminiEmbedder ¶ added in v1.13.0
type GeminiEmbedder struct {
// contains filtered or unexported fields
}
GeminiEmbedder generates embeddings using the Google Gemini API.
func NewGeminiEmbedder ¶ added in v1.13.0
func NewGeminiEmbedder(cfg EmbeddingConfig) *GeminiEmbedder
NewGeminiEmbedder creates a Gemini embedding provider.
func (*GeminiEmbedder) Dimensions ¶ added in v1.13.0
func (e *GeminiEmbedder) Dimensions() int
Dimensions returns the output vector dimensionality.
func (*GeminiEmbedder) Embed ¶ added in v1.13.0
Embed generates embeddings for a batch of texts. Uses batchEmbedContents for multiple texts, embedContent for a single text.
func (*GeminiEmbedder) Model ¶ added in v1.13.0
func (e *GeminiEmbedder) Model() string
Model returns the model name.
func (*GeminiEmbedder) Name ¶ added in v1.13.0
func (e *GeminiEmbedder) Name() string
Name returns the provider name.
type MMRConfig ¶ added in v1.12.0
MMRConfig configures Maximal Marginal Relevance for search diversification.
type MistralEmbedder ¶ added in v1.13.0
type MistralEmbedder struct {
// contains filtered or unexported fields
}
MistralEmbedder generates embeddings using the Mistral API. Mistral uses a fully OpenAI-compatible request/response format.
func NewMistralEmbedder ¶ added in v1.13.0
func NewMistralEmbedder(cfg EmbeddingConfig) *MistralEmbedder
NewMistralEmbedder creates a Mistral embedding provider.
func (*MistralEmbedder) Dimensions ¶ added in v1.13.0
func (e *MistralEmbedder) Dimensions() int
Dimensions returns the output vector dimensionality.
func (*MistralEmbedder) Model ¶ added in v1.13.0
func (e *MistralEmbedder) Model() string
Model returns the model name.
func (*MistralEmbedder) Name ¶ added in v1.13.0
func (e *MistralEmbedder) Name() string
Name returns the provider name.
type NullEmbedder ¶
type NullEmbedder struct{}
NullEmbedder is a no-op provider that disables semantic search. Used when no embedding provider is configured.
type OpenAIEmbedder ¶
type OpenAIEmbedder struct {
// contains filtered or unexported fields
}
OpenAIEmbedder generates embeddings using the OpenAI Embeddings API.
func NewOpenAIEmbedder ¶
func NewOpenAIEmbedder(cfg EmbeddingConfig) *OpenAIEmbedder
NewOpenAIEmbedder creates an OpenAI embedding provider.
func (*OpenAIEmbedder) Dimensions ¶
func (e *OpenAIEmbedder) Dimensions() int
Dimensions returns the output vector dimensionality.
func (*OpenAIEmbedder) Model ¶
func (e *OpenAIEmbedder) Model() string
Model returns the model name.
func (*OpenAIEmbedder) Name ¶
func (e *OpenAIEmbedder) Name() string
Name returns the provider name.
type SQLiteStore ¶
type SQLiteStore struct {
// contains filtered or unexported fields
}
SQLiteStore provides persistent memory storage with hybrid search.
func NewSQLiteStore ¶
func NewSQLiteStore(dbPath string, embedder EmbeddingProvider, logger *slog.Logger) (*SQLiteStore, error)
NewSQLiteStore opens or creates a SQLite memory database with FTS5.
func (*SQLiteStore) ApplyMMR ¶ added in v1.12.0
func (s *SQLiteStore) ApplyMMR(results []SearchResult, cfg MMRConfig, maxResults int) []SearchResult
ApplyMMR applies Maximal Marginal Relevance re-ranking to diversify results. Lambda controls the balance: 0 = max diversity, 1 = max relevance.
func (*SQLiteStore) ApplyTemporalDecay ¶ added in v1.12.0
func (s *SQLiteStore) ApplyTemporalDecay(results []SearchResult, cfg TemporalDecayConfig) []SearchResult
ApplyTemporalDecay applies exponential decay to search results based on file age. Files matching the pattern "memory/YYYY-MM-DD.md" are decayed; evergreen files (MEMORY.md or non-dated) are not decayed.
func (*SQLiteStore) ChunkCount ¶
func (s *SQLiteStore) ChunkCount() int
ChunkCount returns the total number of indexed chunks.
func (*SQLiteStore) Close ¶
func (s *SQLiteStore) Close() error
Close closes the database connection.
func (*SQLiteStore) FileCount ¶
func (s *SQLiteStore) FileCount() int
FileCount returns the total number of indexed files.
func (*SQLiteStore) HybridSearch ¶
func (s *SQLiteStore) HybridSearch(ctx context.Context, query string, maxResults int, minScore float64, vectorWeight, bm25Weight float64) ([]SearchResult, error)
HybridSearch performs a combined vector + BM25 search with configurable weights.
func (*SQLiteStore) HybridSearchWithOptions ¶ added in v1.12.0
func (s *SQLiteStore) HybridSearchWithOptions(ctx context.Context, query string, maxResults int, minScore float64, vectorWeight, bm25Weight float64, decayCfg TemporalDecayConfig, mmrCfg MMRConfig) ([]SearchResult, error)
HybridSearchWithOptions performs hybrid search with optional temporal decay and MMR.
func (*SQLiteStore) IndexChunks ¶
func (s *SQLiteStore) IndexChunks(ctx context.Context, fileID string, chunks []Chunk, fileHash string) error
IndexChunks indexes a set of chunks for a file. Uses delta sync: only re-embeds chunks whose hash has changed.
func (*SQLiteStore) IndexMemoryDir ¶
func (s *SQLiteStore) IndexMemoryDir(ctx context.Context, memDir string, chunkCfg ChunkConfig) error
IndexMemoryDir indexes all .md files in the memory directory and MEMORY.md.
func (*SQLiteStore) IndexTranscript ¶ added in v1.13.0
func (s *SQLiteStore) IndexTranscript(ctx context.Context, sessionID string, entries []TranscriptEntry) error
IndexTranscript indexes conversation transcript entries as searchable chunks. Each entry is stored as a chunk with file_id "session:<sessionID>". Uses content hashing to avoid re-indexing identical content.
func (*SQLiteStore) PruneEmbeddingCache ¶
func (s *SQLiteStore) PruneEmbeddingCache(maxEntries int)
PruneEmbeddingCache removes old cache entries exceeding maxEntries.
func (*SQLiteStore) SearchBM25 ¶
func (s *SQLiteStore) SearchBM25(query string, maxResults int) ([]SearchResult, error)
SearchBM25 performs a keyword search using FTS5 BM25 ranking. Falls back to LIKE-based search when FTS5 is not available. Applies query expansion to handle conversational queries.
func (*SQLiteStore) SearchVector ¶
func (s *SQLiteStore) SearchVector(ctx context.Context, query string, maxResults int) ([]SearchResult, error)
SearchVector performs a vector similarity search using in-memory cosine similarity.
type SearchResult ¶
SearchResult represents a single search hit with score.
type Store ¶
type Store interface {
// Save persists a new memory entry.
Save(entry Entry) error
// Search returns entries matching the query, limited to maxResults.
Search(query string, maxResults int) ([]Entry, error)
// GetRecent returns the most recent entries up to limit.
GetRecent(limit int) ([]Entry, error)
// GetByDate returns entries for a specific date.
GetByDate(date time.Time) ([]Entry, error)
// GetAll returns all stored entries.
GetAll() ([]Entry, error)
// SaveDailyLog appends a summary to the daily log file.
SaveDailyLog(date time.Time, content string) error
}
Store defines the interface for memory persistence.
type TemporalDecayConfig ¶ added in v1.12.0
TemporalDecayConfig configures exponential score decay based on memory age.
type TranscriptEntry ¶ added in v1.13.0
TranscriptEntry is a conversation entry for transcript indexing.
type VoyageEmbedder ¶ added in v1.13.0
type VoyageEmbedder struct {
// contains filtered or unexported fields
}
VoyageEmbedder generates embeddings using the Voyage AI API. Voyage uses an OpenAI-compatible format with an additional input_type field that improves relevance by distinguishing queries from documents.
func NewVoyageEmbedder ¶ added in v1.13.0
func NewVoyageEmbedder(cfg EmbeddingConfig) *VoyageEmbedder
NewVoyageEmbedder creates a Voyage AI embedding provider.
func (*VoyageEmbedder) Dimensions ¶ added in v1.13.0
func (e *VoyageEmbedder) Dimensions() int
Dimensions returns the output vector dimensionality.
func (*VoyageEmbedder) Model ¶ added in v1.13.0
func (e *VoyageEmbedder) Model() string
Model returns the model name.
func (*VoyageEmbedder) Name ¶ added in v1.13.0
func (e *VoyageEmbedder) Name() string
Name returns the provider name.