memory

package
v1.16.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 23, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package memory – embeddings.go implements embedding generation for semantic search. Supports multiple providers: OpenAI, Gemini, Voyage, Mistral, and a zero-cost null fallback. Embeddings are cached by content hash + provider + model to avoid redundant API calls.

Package memory – embeddings_gemini.go implements Google Gemini embedding provider. Uses the Gemini REST API with support for single and batch embedding requests.

Package memory – embeddings_mistral.go implements Mistral embedding provider. Uses the OpenAI-compatible /embeddings endpoint.

Package memory – embeddings_voyage.go implements Voyage AI embedding provider. Uses the OpenAI-compatible /embeddings endpoint with Voyage-specific input_type field.

Package memory – indexer.go implements markdown chunking and hash-based delta sync for the memory index. Files are split into chunks, each chunk gets a SHA-256 hash, and only changed chunks are re-embedded.

Package memory – query_expansion.go handles keyword extraction and FTS5 query building. Supports stop words in English, Portuguese, Spanish, and French. Filters out pure numbers, all-punctuation tokens, and very short tokens.

Package memory – sqlite_store.go implements a SQLite-backed memory store with FTS5 (BM25 ranking) and in-process vector search (cosine similarity). Embeddings are stored as JSON-encoded float32 arrays in the chunks table. This avoids the need for the sqlite-vec extension while still providing hybrid semantic + keyword search.

Package memory implements persistent memory for DevClaw. Provides long-term fact storage and daily conversation logs using the filesystem (MEMORY.md + daily markdown files).

Architecture:

  • MEMORY.md: Long-term facts (append-only, curated by the agent)
  • memory/YYYY-MM-DD.md: Daily conversation summaries (append-only)
  • Search uses simple substring matching (future: BM25 / embeddings)

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FileHash

func FileHash(path string) (string, error)

FileHash computes the SHA-256 hash of a file's content.

func IndexDirectory

func IndexDirectory(dir string, cfg ChunkConfig) (map[string][]Chunk, error)

IndexDirectory scans a directory for .md files and chunks them. Returns a map of fileID → []Chunk. The fileID is the relative path.

Types

type Chunk

type Chunk struct {
	// FileID identifies the source file (relative path).
	FileID string

	// Index is the chunk position within the file (0-based).
	Index int

	// Text is the chunk content.
	Text string

	// Hash is the SHA-256 hex digest of the chunk text.
	Hash string
}

Chunk represents an indexed text fragment from a memory file.

func ChunkMarkdown

func ChunkMarkdown(text string, cfg ChunkConfig) []Chunk

ChunkMarkdown splits markdown text into semantic chunks. Prefers splitting at heading boundaries, then paragraph boundaries, then sentence boundaries, respecting MaxTokens.

type ChunkConfig

type ChunkConfig struct {
	// MaxTokens is the approximate max tokens per chunk (default: 500).
	// Uses ~4 chars/token heuristic.
	MaxTokens int

	// Overlap is the number of characters to overlap between chunks (default: 100).
	Overlap int
}

ChunkConfig controls the chunking behavior.

func DefaultChunkConfig

func DefaultChunkConfig() ChunkConfig

DefaultChunkConfig returns sensible defaults.

type EmbeddingConfig

type EmbeddingConfig struct {
	// Provider is the embedding provider ("openai", "gemini", "voyage", "mistral", "auto", "none").
	Provider string `yaml:"provider"`

	// Model is the embedding model name (e.g. "text-embedding-3-small").
	Model string `yaml:"model"`

	// Dimensions is the output vector dimensionality (default: auto from model).
	Dimensions int `yaml:"dimensions"`

	// APIKey is the API key for the embedding provider. If empty, falls back to
	// the main LLM API key or provider-specific env vars.
	APIKey string `yaml:"api_key"`

	// BaseURL is the API base URL. If empty, uses the provider default.
	BaseURL string `yaml:"base_url"`

	// Cache enables embedding caching in SQLite (default: true).
	Cache bool `yaml:"cache"`

	// Fallback is the fallback provider when the primary fails ("openai", "gemini", etc., or "none").
	Fallback string `yaml:"fallback"`

	// FallbackAPIKey is the API key for the fallback provider.
	FallbackAPIKey string `yaml:"fallback_api_key"`

	// FallbackBaseURL is the base URL for the fallback provider.
	FallbackBaseURL string `yaml:"fallback_base_url"`

	// FallbackModel is the model for the fallback provider.
	FallbackModel string `yaml:"fallback_model"`
}

EmbeddingConfig configures the embedding provider.

func DefaultEmbeddingConfig

func DefaultEmbeddingConfig() EmbeddingConfig

DefaultEmbeddingConfig returns sensible defaults.

type EmbeddingProvider

type EmbeddingProvider interface {
	// Embed generates embeddings for a batch of texts.
	// Returns one float32 vector per input text.
	Embed(ctx context.Context, texts []string) ([][]float32, error)

	// Dimensions returns the dimensionality of the output vectors.
	Dimensions() int

	// Name returns the provider name (for cache key derivation).
	Name() string

	// Model returns the model name (for cache key derivation).
	Model() string
}

EmbeddingProvider generates vector embeddings from text.

func NewEmbeddingProvider

func NewEmbeddingProvider(cfg EmbeddingConfig) EmbeddingProvider

NewEmbeddingProvider creates an embedding provider from config. When a fallback is configured, wraps with FallbackEmbedder for automatic failover.

type Entry

type Entry struct {
	Content   string    `json:"content"`
	Source    string    `json:"source"`   // "user", "agent", "system"
	Category  string    `json:"category"` // "fact", "preference", "event", "summary"
	Timestamp time.Time `json:"timestamp"`
}

Entry represents a single memory fact or event.

type FallbackEmbedder added in v1.13.0

type FallbackEmbedder struct {
	// contains filtered or unexported fields
}

FallbackEmbedder wraps a primary and fallback provider. On primary failure, automatically retries with the fallback. If both fail, returns an error; callers should degrade to FTS-only search.

func NewFallbackEmbedder added in v1.13.0

func NewFallbackEmbedder(primary, fallback EmbeddingProvider, logger *slog.Logger) *FallbackEmbedder

NewFallbackEmbedder creates a fallback-enabled embedder.

func (*FallbackEmbedder) Dimensions added in v1.13.0

func (f *FallbackEmbedder) Dimensions() int

Dimensions returns the primary provider's dimensions.

func (*FallbackEmbedder) Embed added in v1.13.0

func (f *FallbackEmbedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed tries the primary provider, falling back on error.

func (*FallbackEmbedder) Model added in v1.13.0

func (f *FallbackEmbedder) Model() string

Model returns the primary provider's model.

func (*FallbackEmbedder) Name added in v1.13.0

func (f *FallbackEmbedder) Name() string

Name returns "fallback:{primary}" — only the primary is used for cache keys since embeddings from different models are not compatible.

type FileStore

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore implements Store using the filesystem. Uses MEMORY.md for long-term facts and daily markdown files for logs.

func NewFileStore

func NewFileStore(baseDir string) (*FileStore, error)

NewFileStore creates a file-based memory store at the given directory.

func (*FileStore) GetAll

func (fs *FileStore) GetAll() ([]Entry, error)

GetAll reads and parses all entries from MEMORY.md.

func (*FileStore) GetByDate

func (fs *FileStore) GetByDate(date time.Time) ([]Entry, error)

GetByDate returns entries from the daily log for the given date.

func (*FileStore) GetRecent

func (fs *FileStore) GetRecent(limit int) ([]Entry, error)

GetRecent returns the most recent entries up to the limit.

func (*FileStore) ListDailyLogs

func (fs *FileStore) ListDailyLogs() ([]string, error)

ListDailyLogs returns the dates of all daily log files, sorted newest first.

func (*FileStore) RecentFacts

func (fs *FileStore) RecentFacts(maxFacts int, query string) string

RecentFacts returns a formatted string of recent facts suitable for injection into the system prompt.

func (*FileStore) Save

func (fs *FileStore) Save(entry Entry) error

Save appends a memory entry to MEMORY.md.

func (*FileStore) SaveDailyLog

func (fs *FileStore) SaveDailyLog(date time.Time, content string) error

SaveDailyLog appends a conversation summary to the daily log file.

func (*FileStore) Search

func (fs *FileStore) Search(query string, maxResults int) ([]Entry, error)

Search returns entries whose content matches the query (case-insensitive). The query is split into words; an entry matches if ALL words appear in its content.

type GeminiEmbedder added in v1.13.0

type GeminiEmbedder struct {
	// contains filtered or unexported fields
}

GeminiEmbedder generates embeddings using the Google Gemini API.

func NewGeminiEmbedder added in v1.13.0

func NewGeminiEmbedder(cfg EmbeddingConfig) *GeminiEmbedder

NewGeminiEmbedder creates a Gemini embedding provider.

func (*GeminiEmbedder) Dimensions added in v1.13.0

func (e *GeminiEmbedder) Dimensions() int

Dimensions returns the output vector dimensionality.

func (*GeminiEmbedder) Embed added in v1.13.0

func (e *GeminiEmbedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed generates embeddings for a batch of texts. Uses batchEmbedContents for multiple texts, embedContent for a single text.

func (*GeminiEmbedder) Model added in v1.13.0

func (e *GeminiEmbedder) Model() string

Model returns the model name.

func (*GeminiEmbedder) Name added in v1.13.0

func (e *GeminiEmbedder) Name() string

Name returns the provider name.

type MMRConfig added in v1.12.0

type MMRConfig struct {
	Enabled bool
	Lambda  float64
}

MMRConfig configures Maximal Marginal Relevance for search diversification.

type MistralEmbedder added in v1.13.0

type MistralEmbedder struct {
	// contains filtered or unexported fields
}

MistralEmbedder generates embeddings using the Mistral API. Mistral uses a fully OpenAI-compatible request/response format.

func NewMistralEmbedder added in v1.13.0

func NewMistralEmbedder(cfg EmbeddingConfig) *MistralEmbedder

NewMistralEmbedder creates a Mistral embedding provider.

func (*MistralEmbedder) Dimensions added in v1.13.0

func (e *MistralEmbedder) Dimensions() int

Dimensions returns the output vector dimensionality.

func (*MistralEmbedder) Embed added in v1.13.0

func (e *MistralEmbedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed generates embeddings for a batch of texts.

func (*MistralEmbedder) Model added in v1.13.0

func (e *MistralEmbedder) Model() string

Model returns the model name.

func (*MistralEmbedder) Name added in v1.13.0

func (e *MistralEmbedder) Name() string

Name returns the provider name.

type NullEmbedder

type NullEmbedder struct{}

NullEmbedder is a no-op provider that disables semantic search. Used when no embedding provider is configured.

func (*NullEmbedder) Dimensions

func (e *NullEmbedder) Dimensions() int

Dimensions returns 0.

func (*NullEmbedder) Embed

func (e *NullEmbedder) Embed(_ context.Context, _ []string) ([][]float32, error)

Embed returns nil (no embeddings).

func (*NullEmbedder) Model

func (e *NullEmbedder) Model() string

Model returns "none".

func (*NullEmbedder) Name

func (e *NullEmbedder) Name() string

Name returns "none".

type OpenAIEmbedder

type OpenAIEmbedder struct {
	// contains filtered or unexported fields
}

OpenAIEmbedder generates embeddings using the OpenAI Embeddings API.

func NewOpenAIEmbedder

func NewOpenAIEmbedder(cfg EmbeddingConfig) *OpenAIEmbedder

NewOpenAIEmbedder creates an OpenAI embedding provider.

func (*OpenAIEmbedder) Dimensions

func (e *OpenAIEmbedder) Dimensions() int

Dimensions returns the output vector dimensionality.

func (*OpenAIEmbedder) Embed

func (e *OpenAIEmbedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed generates embeddings for a batch of texts.

func (*OpenAIEmbedder) Model

func (e *OpenAIEmbedder) Model() string

Model returns the model name.

func (*OpenAIEmbedder) Name

func (e *OpenAIEmbedder) Name() string

Name returns the provider name.

type SQLiteStore

type SQLiteStore struct {
	// contains filtered or unexported fields
}

SQLiteStore provides persistent memory storage with hybrid search.

func NewSQLiteStore

func NewSQLiteStore(dbPath string, embedder EmbeddingProvider, logger *slog.Logger) (*SQLiteStore, error)

NewSQLiteStore opens or creates a SQLite memory database with FTS5.

func (*SQLiteStore) ApplyMMR added in v1.12.0

func (s *SQLiteStore) ApplyMMR(results []SearchResult, cfg MMRConfig, maxResults int) []SearchResult

ApplyMMR applies Maximal Marginal Relevance re-ranking to diversify results. Lambda controls the balance: 0 = max diversity, 1 = max relevance.

func (*SQLiteStore) ApplyTemporalDecay added in v1.12.0

func (s *SQLiteStore) ApplyTemporalDecay(results []SearchResult, cfg TemporalDecayConfig) []SearchResult

ApplyTemporalDecay applies exponential decay to search results based on file age. Files matching the pattern "memory/YYYY-MM-DD.md" are decayed; evergreen files (MEMORY.md or non-dated) are not decayed.

func (*SQLiteStore) ChunkCount

func (s *SQLiteStore) ChunkCount() int

ChunkCount returns the total number of indexed chunks.

func (*SQLiteStore) Close

func (s *SQLiteStore) Close() error

Close closes the database connection.

func (*SQLiteStore) FileCount

func (s *SQLiteStore) FileCount() int

FileCount returns the total number of indexed files.

func (*SQLiteStore) HybridSearch

func (s *SQLiteStore) HybridSearch(ctx context.Context, query string, maxResults int, minScore float64, vectorWeight, bm25Weight float64) ([]SearchResult, error)

HybridSearch performs a combined vector + BM25 search with configurable weights.

func (*SQLiteStore) HybridSearchWithOptions added in v1.12.0

func (s *SQLiteStore) HybridSearchWithOptions(ctx context.Context, query string, maxResults int, minScore float64, vectorWeight, bm25Weight float64, decayCfg TemporalDecayConfig, mmrCfg MMRConfig) ([]SearchResult, error)

HybridSearchWithOptions performs hybrid search with optional temporal decay and MMR.

func (*SQLiteStore) IndexChunks

func (s *SQLiteStore) IndexChunks(ctx context.Context, fileID string, chunks []Chunk, fileHash string) error

IndexChunks indexes a set of chunks for a file. Uses delta sync: only re-embeds chunks whose hash has changed.

func (*SQLiteStore) IndexMemoryDir

func (s *SQLiteStore) IndexMemoryDir(ctx context.Context, memDir string, chunkCfg ChunkConfig) error

IndexMemoryDir indexes all .md files in the memory directory and MEMORY.md.

func (*SQLiteStore) IndexTranscript added in v1.13.0

func (s *SQLiteStore) IndexTranscript(ctx context.Context, sessionID string, entries []TranscriptEntry) error

IndexTranscript indexes conversation transcript entries as searchable chunks. Each entry is stored as a chunk with file_id "session:<sessionID>". Uses content hashing to avoid re-indexing identical content.

func (*SQLiteStore) PruneEmbeddingCache

func (s *SQLiteStore) PruneEmbeddingCache(maxEntries int)

PruneEmbeddingCache removes old cache entries exceeding maxEntries.

func (*SQLiteStore) SearchBM25

func (s *SQLiteStore) SearchBM25(query string, maxResults int) ([]SearchResult, error)

SearchBM25 performs a keyword search using FTS5 BM25 ranking. Falls back to LIKE-based search when FTS5 is not available. Applies query expansion to handle conversational queries.

func (*SQLiteStore) SearchVector

func (s *SQLiteStore) SearchVector(ctx context.Context, query string, maxResults int) ([]SearchResult, error)

SearchVector performs a vector similarity search using in-memory cosine similarity.

type SearchResult

type SearchResult struct {
	FileID string
	Text   string
	Score  float64
}

SearchResult represents a single search hit with score.

type Store

type Store interface {
	// Save persists a new memory entry.
	Save(entry Entry) error

	// Search returns entries matching the query, limited to maxResults.
	Search(query string, maxResults int) ([]Entry, error)

	// GetRecent returns the most recent entries up to limit.
	GetRecent(limit int) ([]Entry, error)

	// GetByDate returns entries for a specific date.
	GetByDate(date time.Time) ([]Entry, error)

	// GetAll returns all stored entries.
	GetAll() ([]Entry, error)

	// SaveDailyLog appends a summary to the daily log file.
	SaveDailyLog(date time.Time, content string) error
}

Store defines the interface for memory persistence.

type TemporalDecayConfig added in v1.12.0

type TemporalDecayConfig struct {
	Enabled      bool
	HalfLifeDays float64
}

TemporalDecayConfig configures exponential score decay based on memory age.

type TranscriptEntry added in v1.13.0

type TranscriptEntry struct {
	Role    string // "user" or "assistant"
	Content string
}

TranscriptEntry is a conversation entry for transcript indexing.

type VoyageEmbedder added in v1.13.0

type VoyageEmbedder struct {
	// contains filtered or unexported fields
}

VoyageEmbedder generates embeddings using the Voyage AI API. Voyage uses an OpenAI-compatible format with an additional input_type field that improves relevance by distinguishing queries from documents.

func NewVoyageEmbedder added in v1.13.0

func NewVoyageEmbedder(cfg EmbeddingConfig) *VoyageEmbedder

NewVoyageEmbedder creates a Voyage AI embedding provider.

func (*VoyageEmbedder) Dimensions added in v1.13.0

func (e *VoyageEmbedder) Dimensions() int

Dimensions returns the output vector dimensionality.

func (*VoyageEmbedder) Embed added in v1.13.0

func (e *VoyageEmbedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed generates embeddings for a batch of texts.

func (*VoyageEmbedder) Model added in v1.13.0

func (e *VoyageEmbedder) Model() string

Model returns the model name.

func (*VoyageEmbedder) Name added in v1.13.0

func (e *VoyageEmbedder) Name() string

Name returns the provider name.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL