memory

package
v1.11.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 22, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package memory – embeddings.go implements embedding generation for semantic search. Supports multiple providers: OpenAI, and a zero-cost local fallback. Embeddings are cached by content hash + provider + model to avoid redundant API calls.

Package memory – indexer.go implements markdown chunking and hash-based delta sync for the memory index. Files are split into chunks, each chunk gets a SHA-256 hash, and only changed chunks are re-embedded.

Package memory – sqlite_store.go implements a SQLite-backed memory store with FTS5 (BM25 ranking) and in-process vector search (cosine similarity). Embeddings are stored as JSON-encoded float32 arrays in the chunks table. This avoids the need for the sqlite-vec extension while still providing hybrid semantic + keyword search.

Package memory implements persistent memory for DevClaw. Provides long-term fact storage and daily conversation logs using the filesystem (MEMORY.md + daily markdown files).

Architecture:

  • MEMORY.md: Long-term facts (append-only, curated by the agent)
  • memory/YYYY-MM-DD.md: Daily conversation summaries (append-only)
  • Search uses simple substring matching (future: BM25 / embeddings)

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FileHash

func FileHash(path string) (string, error)

FileHash computes the SHA-256 hash of a file's content.

func IndexDirectory

func IndexDirectory(dir string, cfg ChunkConfig) (map[string][]Chunk, error)

IndexDirectory scans a directory for .md files and chunks them. Returns a map of fileID → []Chunk. The fileID is the relative path.

Types

type Chunk

type Chunk struct {
	// FileID identifies the source file (relative path).
	FileID string

	// Index is the chunk position within the file (0-based).
	Index int

	// Text is the chunk content.
	Text string

	// Hash is the SHA-256 hex digest of the chunk text.
	Hash string
}

Chunk represents an indexed text fragment from a memory file.

func ChunkMarkdown

func ChunkMarkdown(text string, cfg ChunkConfig) []Chunk

ChunkMarkdown splits markdown text into semantic chunks. Prefers splitting at heading boundaries, then paragraph boundaries, then sentence boundaries, respecting MaxTokens.

type ChunkConfig

type ChunkConfig struct {
	// MaxTokens is the approximate max tokens per chunk (default: 500).
	// Uses ~4 chars/token heuristic.
	MaxTokens int

	// Overlap is the number of characters to overlap between chunks (default: 100).
	Overlap int
}

ChunkConfig controls the chunking behavior.

func DefaultChunkConfig

func DefaultChunkConfig() ChunkConfig

DefaultChunkConfig returns sensible defaults.

type EmbeddingConfig

type EmbeddingConfig struct {
	// Provider is the embedding provider ("openai", "none").
	Provider string `yaml:"provider"`

	// Model is the embedding model name (e.g. "text-embedding-3-small").
	Model string `yaml:"model"`

	// Dimensions is the output vector dimensionality (default: auto from model).
	Dimensions int `yaml:"dimensions"`

	// APIKey is the API key for the embedding provider. If empty, falls back to
	// the main LLM API key.
	APIKey string `yaml:"api_key"`

	// BaseURL is the API base URL. If empty, uses the provider default.
	BaseURL string `yaml:"base_url"`

	// Cache enables embedding caching in SQLite (default: true).
	Cache bool `yaml:"cache"`
}

EmbeddingConfig configures the embedding provider.

func DefaultEmbeddingConfig

func DefaultEmbeddingConfig() EmbeddingConfig

DefaultEmbeddingConfig returns sensible defaults.

type EmbeddingProvider

type EmbeddingProvider interface {
	// Embed generates embeddings for a batch of texts.
	// Returns one float32 vector per input text.
	Embed(ctx context.Context, texts []string) ([][]float32, error)

	// Dimensions returns the dimensionality of the output vectors.
	Dimensions() int

	// Name returns the provider name (for cache key derivation).
	Name() string

	// Model returns the model name (for cache key derivation).
	Model() string
}

EmbeddingProvider generates vector embeddings from text.

func NewEmbeddingProvider

func NewEmbeddingProvider(cfg EmbeddingConfig) EmbeddingProvider

NewEmbeddingProvider creates an embedding provider from config.

type Entry

type Entry struct {
	Content   string    `json:"content"`
	Source    string    `json:"source"`   // "user", "agent", "system"
	Category  string    `json:"category"` // "fact", "preference", "event", "summary"
	Timestamp time.Time `json:"timestamp"`
}

Entry represents a single memory fact or event.

type FileStore

type FileStore struct {
	// contains filtered or unexported fields
}

FileStore implements Store using the filesystem. Uses MEMORY.md for long-term facts and daily markdown files for logs.

func NewFileStore

func NewFileStore(baseDir string) (*FileStore, error)

NewFileStore creates a file-based memory store at the given directory.

func (*FileStore) GetAll

func (fs *FileStore) GetAll() ([]Entry, error)

GetAll reads and parses all entries from MEMORY.md.

func (*FileStore) GetByDate

func (fs *FileStore) GetByDate(date time.Time) ([]Entry, error)

GetByDate returns entries from the daily log for the given date.

func (*FileStore) GetRecent

func (fs *FileStore) GetRecent(limit int) ([]Entry, error)

GetRecent returns the most recent entries up to the limit.

func (*FileStore) ListDailyLogs

func (fs *FileStore) ListDailyLogs() ([]string, error)

ListDailyLogs returns the dates of all daily log files, sorted newest first.

func (*FileStore) RecentFacts

func (fs *FileStore) RecentFacts(maxFacts int, query string) string

RecentFacts returns a formatted string of recent facts suitable for injection into the system prompt.

func (*FileStore) Save

func (fs *FileStore) Save(entry Entry) error

Save appends a memory entry to MEMORY.md.

func (*FileStore) SaveDailyLog

func (fs *FileStore) SaveDailyLog(date time.Time, content string) error

SaveDailyLog appends a conversation summary to the daily log file.

func (*FileStore) Search

func (fs *FileStore) Search(query string, maxResults int) ([]Entry, error)

Search returns entries whose content matches the query (case-insensitive substring).

type NullEmbedder

type NullEmbedder struct{}

NullEmbedder is a no-op provider that disables semantic search. Used when no embedding provider is configured.

func (*NullEmbedder) Dimensions

func (e *NullEmbedder) Dimensions() int

Dimensions returns 0.

func (*NullEmbedder) Embed

func (e *NullEmbedder) Embed(_ context.Context, _ []string) ([][]float32, error)

Embed returns nil (no embeddings).

func (*NullEmbedder) Model

func (e *NullEmbedder) Model() string

Model returns "none".

func (*NullEmbedder) Name

func (e *NullEmbedder) Name() string

Name returns "none".

type OpenAIEmbedder

type OpenAIEmbedder struct {
	// contains filtered or unexported fields
}

OpenAIEmbedder generates embeddings using the OpenAI Embeddings API.

func NewOpenAIEmbedder

func NewOpenAIEmbedder(cfg EmbeddingConfig) *OpenAIEmbedder

NewOpenAIEmbedder creates an OpenAI embedding provider.

func (*OpenAIEmbedder) Dimensions

func (e *OpenAIEmbedder) Dimensions() int

Dimensions returns the output vector dimensionality.

func (*OpenAIEmbedder) Embed

func (e *OpenAIEmbedder) Embed(ctx context.Context, texts []string) ([][]float32, error)

Embed generates embeddings for a batch of texts.

func (*OpenAIEmbedder) Model

func (e *OpenAIEmbedder) Model() string

Model returns the model name.

func (*OpenAIEmbedder) Name

func (e *OpenAIEmbedder) Name() string

Name returns the provider name.

type SQLiteStore

type SQLiteStore struct {
	// contains filtered or unexported fields
}

SQLiteStore provides persistent memory storage with hybrid search.

func NewSQLiteStore

func NewSQLiteStore(dbPath string, embedder EmbeddingProvider, logger *slog.Logger) (*SQLiteStore, error)

NewSQLiteStore opens or creates a SQLite memory database with FTS5.

func (*SQLiteStore) ChunkCount

func (s *SQLiteStore) ChunkCount() int

ChunkCount returns the total number of indexed chunks.

func (*SQLiteStore) Close

func (s *SQLiteStore) Close() error

Close closes the database connection.

func (*SQLiteStore) FileCount

func (s *SQLiteStore) FileCount() int

FileCount returns the total number of indexed files.

func (*SQLiteStore) HybridSearch

func (s *SQLiteStore) HybridSearch(ctx context.Context, query string, maxResults int, minScore float64, vectorWeight, bm25Weight float64) ([]SearchResult, error)

HybridSearch performs a combined vector + BM25 search with configurable weights.

func (*SQLiteStore) IndexChunks

func (s *SQLiteStore) IndexChunks(ctx context.Context, fileID string, chunks []Chunk, fileHash string) error

IndexChunks indexes a set of chunks for a file. Uses delta sync: only re-embeds chunks whose hash has changed.

func (*SQLiteStore) IndexMemoryDir

func (s *SQLiteStore) IndexMemoryDir(ctx context.Context, memDir string, chunkCfg ChunkConfig) error

IndexMemoryDir indexes all .md files in the memory directory and MEMORY.md.

func (*SQLiteStore) PruneEmbeddingCache

func (s *SQLiteStore) PruneEmbeddingCache(maxEntries int)

PruneEmbeddingCache removes old cache entries exceeding maxEntries.

func (*SQLiteStore) SearchBM25

func (s *SQLiteStore) SearchBM25(query string, maxResults int) ([]SearchResult, error)

SearchBM25 performs a keyword search using FTS5 BM25 ranking. Falls back to LIKE-based search when FTS5 is not available. Applies query expansion to handle conversational queries.

func (*SQLiteStore) SearchVector

func (s *SQLiteStore) SearchVector(ctx context.Context, query string, maxResults int) ([]SearchResult, error)

SearchVector performs a vector similarity search using in-memory cosine similarity.

type SearchResult

type SearchResult struct {
	FileID string
	Text   string
	Score  float64
}

SearchResult represents a single search hit with score.

type Store

type Store interface {
	// Save persists a new memory entry.
	Save(entry Entry) error

	// Search returns entries matching the query, limited to maxResults.
	Search(query string, maxResults int) ([]Entry, error)

	// GetRecent returns the most recent entries up to limit.
	GetRecent(limit int) ([]Entry, error)

	// GetByDate returns entries for a specific date.
	GetByDate(date time.Time) ([]Entry, error)

	// GetAll returns all stored entries.
	GetAll() ([]Entry, error)

	// SaveDailyLog appends a summary to the daily log file.
	SaveDailyLog(date time.Time, content string) error
}

Store defines the interface for memory persistence.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL