storage

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 15, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package storage holds LadyM's persistence layer: the SQLite store, pluggable embedding providers, and the vector index.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AssertDimMatches 

func AssertDimMatches(stored, configured int) error

AssertDimMatches raises EmbeddingDimensionMismatch when dims differ.

func CosineSimilarity 

func CosineSimilarity(a, b []float32) float64

CosineSimilarity returns the cosine of two vectors (assumed or forced to be normalised). Returns 0 when dimensions differ or either vector is zero.

func RegisterCallable 

func RegisterCallable(name string, fn func(string) ([]float32, error))

RegisterCallable registers a Go function as a named embedding provider.

func Tokenize 

func Tokenize(text string) []string

Tokenize is the lightweight tokenizer: words + punctuation as separate tokens, with camelCase and snake_case splitting so getUserName and get_user_name tokenize similarly.

Types

type CachedEmbedding 

type CachedEmbedding struct {
	// contains filtered or unexported fields
}

CachedEmbedding wraps an inner EmbeddingProvider with an LRU cache for Embed() calls. EmbedBatch always delegates straight through.

func NewCachedEmbedding 

func NewCachedEmbedding(inner EmbeddingProvider, size int) *CachedEmbedding

NewCachedEmbedding wraps inner with an LRU cache of the given size.

func (*CachedEmbedding) Dim 

func (c *CachedEmbedding) Dim() int

func (*CachedEmbedding) Embed 

func (c *CachedEmbedding) Embed(text string) ([]float32, error)

func (*CachedEmbedding) EmbedBatch 

func (c *CachedEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*CachedEmbedding) HealthCheck 

func (c *CachedEmbedding) HealthCheck() (bool, string)

type CallableEmbedding 

type CallableEmbedding struct {
	// contains filtered or unexported fields
}

CallableEmbedding wraps a Go function as an embedding provider.

func NewCallableEmbedding 

func NewCallableEmbedding(fn func(string) ([]float32, error), dim int) *CallableEmbedding

NewCallableEmbedding wraps fn as an embedding provider.

func (*CallableEmbedding) Dim 

func (c *CallableEmbedding) Dim() int

func (*CallableEmbedding) Embed 

func (c *CallableEmbedding) Embed(text string) ([]float32, error)

func (*CallableEmbedding) EmbedBatch 

func (c *CallableEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*CallableEmbedding) HealthCheck 

func (c *CallableEmbedding) HealthCheck() (bool, string)

type EmbeddingDimensionMismatch 

type EmbeddingDimensionMismatch struct {
	Stored     int
	Configured int
}

EmbeddingDimensionMismatch is raised when a reopened DB holds vectors of a different dim than the live provider.

func (*EmbeddingDimensionMismatch) Error 

func (e *EmbeddingDimensionMismatch) Error() string

type EmbeddingProvider 

type EmbeddingProvider interface {
	// Embed returns a float32 vector for text.
	Embed(text string) ([]float32, error)
	// EmbedBatch embeds a batch of texts (defaults to looping Embed).
	EmbedBatch(texts []string) ([][]float32, error)
	// Dim returns the vector dimension, or 0 when unknown until the first
	// embed call (deferred-dim providers such as Ollama).
	Dim() int
	// HealthCheck performs a one-shot probe for the web UI "test embedding".
	HealthCheck() (bool, string)
}

EmbeddingProvider is the contract every provider implements.

func MakeProvider 

func MakeProvider(cfg *config.Config) (EmbeddingProvider, error)

MakeProvider resolves the configured embedding provider.

type EmbeddingProviderError 

type EmbeddingProviderError struct {
	Msg string
}

EmbeddingProviderError is the base error for provider-side failures.

func (*EmbeddingProviderError) Error 

func (e *EmbeddingProviderError) Error() string

type FakeHTTPPoster 

type FakeHTTPPoster struct {
	Responder    func(payload any) (any, error)
	ExpectedPath string
	LastPayload  any
	LastURL      string
}

FakeHTTPPoster is a test double. Responder maps payload → JSON-able result.

func (*FakeHTTPPoster) Post 

func (f *FakeHTTPPoster) Post(url string, payload any, headers map[string]string) (any, error)

type HTTPEmbedding 

type HTTPEmbedding struct {
	// contains filtered or unexported fields
}

HTTPEmbedding is a generic, template-driven embedding provider.

func NewHTTPEmbedding 

func NewHTTPEmbedding(opts HTTPEmbeddingOptions) *HTTPEmbedding

NewHTTPEmbedding builds a generic HTTP embedding provider.

func (*HTTPEmbedding) Dim 

func (h *HTTPEmbedding) Dim() int

func (*HTTPEmbedding) Embed 

func (h *HTTPEmbedding) Embed(text string) ([]float32, error)

func (*HTTPEmbedding) EmbedBatch 

func (h *HTTPEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*HTTPEmbedding) HealthCheck 

func (h *HTTPEmbedding) HealthCheck() (bool, string)

type HTTPEmbeddingOptions 

type HTTPEmbeddingOptions struct {
	BaseURL      string
	Request      string
	ResponsePath string
	Dim          int
	Model        string
	TimeoutS     float64
	// Poster, when non-nil, overrides the default real HTTP client (tests).
	Poster HTTPPoster
}

HTTPEmbeddingOptions configures the generic HTTP embedding provider.

type HTTPPoster 

type HTTPPoster interface {
	Post(url string, payload any, headers map[string]string) (any, error)
}

HTTPPoster is the minimal HTTP contract used by embedding providers so tests can inject a fake and never touch the network.

type HashingEmbedding 

type HashingEmbedding struct {
	// contains filtered or unexported fields
}

HashingEmbedding is the deterministic, offline, dependency-free embedding via feature hashing (unigram + bigram features, L2-normalised).

func NewHashingEmbedding 

func NewHashingEmbedding(dim int) *HashingEmbedding

NewHashingEmbedding returns a HashingEmbedding with the given dim.

func (*HashingEmbedding) Dim 

func (h *HashingEmbedding) Dim() int

func (*HashingEmbedding) Embed 

func (h *HashingEmbedding) Embed(text string) ([]float32, error)

func (*HashingEmbedding) EmbedBatch 

func (h *HashingEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*HashingEmbedding) HealthCheck 

func (h *HashingEmbedding) HealthCheck() (bool, string)

type InMemoryVectorIndex 

type InMemoryVectorIndex struct {
	// contains filtered or unexported fields
}

InMemoryVectorIndex is a brute-force cosine index (deterministic, perfect for tests and small workspaces). It replaces the sqlite-vec extension used by the Python port with a pure-Go equivalent; vectors are still persisted as BLOBs in the store so a reopened store can rebuild the index.

func NewInMemoryVectorIndex 

func NewInMemoryVectorIndex(dim int) *InMemoryVectorIndex

NewInMemoryVectorIndex returns an empty index of the given dim.

func (*InMemoryVectorIndex) Delete 

func (ix *InMemoryVectorIndex) Delete(itemID string)

func (*InMemoryVectorIndex) Len 

func (ix *InMemoryVectorIndex) Len() int

func (*InMemoryVectorIndex) Search 

func (ix *InMemoryVectorIndex) Search(query []float32, topK int) []SearchHit

func (*InMemoryVectorIndex) Upsert 

func (ix *InMemoryVectorIndex) Upsert(itemID string, vector []float32) error

type OllamaEmbedding 

type OllamaEmbedding struct {
	// contains filtered or unexported fields
}

OllamaEmbedding targets the Ollama /api/embeddings endpoint. Dim is deferred until the first embed call.

func NewOllamaEmbedding 

func NewOllamaEmbedding(baseURL, model string, timeoutS float64, client HTTPPoster) *OllamaEmbedding

NewOllamaEmbedding builds an OllamaEmbedding (client may be nil for real HTTP).

func (*OllamaEmbedding) Dim 

func (o *OllamaEmbedding) Dim() int

func (*OllamaEmbedding) Embed 

func (o *OllamaEmbedding) Embed(text string) ([]float32, error)

func (*OllamaEmbedding) EmbedBatch 

func (o *OllamaEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*OllamaEmbedding) HealthCheck 

func (o *OllamaEmbedding) HealthCheck() (bool, string)

type OpenAIEmbedding 

type OpenAIEmbedding struct {
	// contains filtered or unexported fields
}

OpenAIEmbedding targets the OpenAI (or OpenAI-compatible) embeddings API.

func NewOpenAIEmbedding 

func NewOpenAIEmbedding(model, baseURL, apiKey string, timeoutS float64) *OpenAIEmbedding

NewOpenAIEmbedding builds an OpenAIEmbedding. Empty baseURL defaults to https://api.openai.com/v1.

func (*OpenAIEmbedding) Dim 

func (o *OpenAIEmbedding) Dim() int

func (*OpenAIEmbedding) Embed 

func (o *OpenAIEmbedding) Embed(text string) ([]float32, error)

func (*OpenAIEmbedding) EmbedBatch 

func (o *OpenAIEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*OpenAIEmbedding) HealthCheck 

func (o *OpenAIEmbedding) HealthCheck() (bool, string)

type RealHTTPPoster 

type RealHTTPPoster struct {
	Timeout time.Duration
}

RealHTTPPoster is the net/http-backed client.

func NewRealHTTPPoster 

func NewRealHTTPPoster(timeoutS float64) *RealHTTPPoster

NewRealHTTPPoster returns a RealHTTPPoster with the given timeout.

func (*RealHTTPPoster) Post 

func (c *RealHTTPPoster) Post(url string, payload any, headers map[string]string) (any, error)

type SQLiteStore 

type SQLiteStore struct {
	DBPath string
	Dim    int
	// contains filtered or unexported fields
}

SQLiteStore is the single persistence layer for LadyM. It owns the SQLite connection and the (in-memory) vector index.

func NewStore 

func NewStore(dbPath string, dim int, preferSQLiteVec bool, enableWAL bool) (*SQLiteStore, error)

NewStore opens (or creates) a SQLite store at dbPath.

NOTE: the Python port used the sqlite-vec loadable extension for persistent ANN search. The Go port always uses the pure-Go InMemoryVectorIndex (the sqlite-vec C extension has no Go binding); embeddings are still persisted as BLOBs and warmed into the index on reopen, so behaviour is identical. The preferSQLiteVec flag is accepted for config parity but has no effect.

func (*SQLiteStore) Close 

func (s *SQLiteStore) Close() error

Close commits and closes the underlying connection.

func (*SQLiteStore) Count 

func (s *SQLiteStore) Count(workspace string) (map[string]int, error)

Count returns a "layer/type" → count map (optionally scoped to workspace).

func (*SQLiteStore) CountEdges 

func (s *SQLiteStore) CountEdges() (int, error)

CountEdges returns the number of edges.

func (*SQLiteStore) DB 

func (s *SQLiteStore) DB() *sql.DB

DB exposes the underlying *sql.DB for low-level queries (layers, operations).

func (*SQLiteStore) DeleteMemory 

func (s *SQLiteStore) DeleteMemory(id string) error

DeleteMemory deletes a memory and its vector entry.

func (*SQLiteStore) EpisodicContentsSince 

func (s *SQLiteStore) EpisodicContentsSince(workspace string, since float64) ([]string, error)

EpisodicContentsSince returns the content strings of episodic events in workspace created at or after since (used by the attention gate's recent-duplicate scan; pushes the time-window cut into SQL).

func (*SQLiteStore) FindByHash 

func (s *SQLiteStore) FindByHash(contentHash, workspace string) (*schema.Memory, error)

FindByHash returns the memory with the given content hash (optionally scoped to workspace), or nil.

func (*SQLiteStore) GetIndexedHash 

func (s *SQLiteStore) GetIndexedHash(filePath string) (string, error)

GetIndexedHash returns the recorded body hash for a file, or "".

func (*SQLiteStore) GetMemory 

func (s *SQLiteStore) GetMemory(id string) (*schema.Memory, error)

GetMemory returns the memory with the given id, or nil.

func (*SQLiteStore) GetMeta 

func (s *SQLiteStore) GetMeta(key string) (string, error)

GetMeta returns the value for a meta key, or "".

func (*SQLiteStore) IterMemories 

func (s *SQLiteStore) IterMemories(workspace, layer, typ string) ([]*schema.Memory, error)

IterMemories returns memories matching the optional filters.

func (*SQLiteStore) NeighborCounts 

func (s *SQLiteStore) NeighborCounts() (map[string]int, error)

NeighborCounts returns {memory_id: neighbour_count} for spreading activation.

func (*SQLiteStore) Neighbors 

func (s *SQLiteStore) Neighbors(memID, relation string) ([]*schema.Edge, error)

Neighbors returns valid (valid_to IS NULL) edges touching memID.

func (*SQLiteStore) PutCodeRefs 

func (s *SQLiteStore) PutCodeRefs(refs []*schema.CodeRef) error

PutCodeRefs bulk-inserts cross references atomically: one transaction, so a failure rolls back the whole batch (Python: executemany in a single commit).

func (*SQLiteStore) PutCodeSymbol 

func (s *SQLiteStore) PutCodeSymbol(sym *schema.CodeSymbol) error

PutCodeSymbol inserts or updates a code symbol projection.

func (*SQLiteStore) PutEdge 

func (s *SQLiteStore) PutEdge(e *schema.Edge) error

PutEdge inserts or updates an edge.

func (*SQLiteStore) PutMemory 

func (s *SQLiteStore) PutMemory(mem *schema.Memory, vector []float32) error

func (*SQLiteStore) RebuildVectorIndex 

func (s *SQLiteStore) RebuildVectorIndex(newDim int)

RebuildVectorIndex resets the index at a new dim (used on dim change).

func (*SQLiteStore) RefsForSymbol 

func (s *SQLiteStore) RefsForSymbol(qualifiedName, direction string) ([]*schema.CodeRef, error)

RefsForSymbol returns cross references for a qualified name.

func (*SQLiteStore) SetIndexed 

func (s *SQLiteStore) SetIndexed(filePath, bodyHash string, now float64) error

SetIndexed records the body hash for a file.

func (*SQLiteStore) SetMeta 

func (s *SQLiteStore) SetMeta(key, value string) error

SetMeta upserts a meta key/value.

func (*SQLiteStore) SymbolsForFile 

func (s *SQLiteStore) SymbolsForFile(filePath string) ([]*schema.CodeSymbol, error)

SymbolsForFile returns code symbols for a file, ordered by line.

func (*SQLiteStore) TouchMemory 

func (s *SQLiteStore) TouchMemory(id string, now float64) error

TouchMemory bumps access_count / last_access_at.

func (*SQLiteStore) UsingSQLiteVec 

func (s *SQLiteStore) UsingSQLiteVec() bool

UsingSQLiteVec always returns false (see NewStore).

func (*SQLiteStore) VectorIndex 

func (s *SQLiteStore) VectorIndex() VectorIndex

VectorIndex exposes the vector index.

func (*SQLiteStore) Workspaces 

func (s *SQLiteStore) Workspaces() ([]string, error)

Workspaces lists distinct workspaces.

type SearchHit 

type SearchHit struct {
	ID         string
	Similarity float64
}

SearchHit is one (id, similarity) pair returned by Search.

type VectorIndex 

type VectorIndex interface {
	Upsert(itemID string, vector []float32) error
	Search(query []float32, topK int) []SearchHit
	Delete(itemID string)
	Len() int
}

VectorIndex inserts/queries/deletes vectors keyed by an arbitrary id.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.