Help us shape the future of Go! Take the 2026 Go Developer Survey

storage

package
v0.8.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 23, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package storage holds LadyM's persistence layer: the SQLite store, pluggable embedding providers, and the vector index.

Index

Constants

View Source 
const (
	// CJKTokenizerDict segments covered scripts with the gse dictionary.
	// It is the default and the mode every existing database was written
	// under, so switching away from it invalidates stored CJK vectors.
	CJKTokenizerDict = "dict"
	// CJKTokenizerChars emits one token per rune for every CJK script and
	// never loads the dictionary. features() adds adjacent-pair bigrams on
	// top, so retrieval quality holds up: on a 40-doc / 40-query Chinese
	// probe it scored MRR 1.000 against 0.988 for the dictionary path.
	// The win is latency — see SetCJKTokenizerMode.
	CJKTokenizerChars = "chars"
)

CJK tokenizer modes.

These change what storage.Tokenize emits, which reaches more than the embedding: the same tokens feed recall's coverage scoring (operations/recall.go reflect) and the per-memory token statistics. Under chars a CJK query yields per-rune tokens, so those numbers move too — but only for CJK text; ASCII tokenization is identical in both modes.

View Source 
const Edition = "personal"

Edition marks the build flavour of this binary ("personal" here). The CLI surfaces it via --version; the enterprise variant lives in edition_enterprise.go behind the enterprise build tag.

Variables

View Source 
var ErrIndexLockHeld = errors.New("code indexing is already running")

ErrIndexLockHeld is returned by TryAcquireIndexLock when another process already holds the code-index lock for this store's database. The user-facing IndexInProgressError type lives in the code package (code imports storage, so the type cannot live here); callers translate ErrIndexLockHeld into it.

View Source 
var ErrWorkerLockHeld = errors.New("a system2 worker cycle is already running")

ErrWorkerLockHeld is returned by TryAcquireWorkerLock when another process already holds the System2 worker lock for this store's database. Callers treat it as "standby": skip the cycle, do not report a failure.

Functions

func AssertDimMatches 

func AssertDimMatches(stored, configured int) error

AssertDimMatches raises EmbeddingDimensionMismatch when dims differ.

func CJKTokenizerMode added in v0.8.2 

func CJKTokenizerMode() string

CJKTokenizerMode reports the active CJK tokenizer mode.

func CJKTokenizerVersion added in v0.8.2 

func CJKTokenizerVersion() string

CJKTokenizerVersion tags the vectors the active mode produces. It is persisted under the "embedding_tokenizer" meta key so a mode switch is detectable: vectors produced by different versions are not comparable, and silently mixing them would make older CJK memories unsearchable.

func CosineSimilarity 

func CosineSimilarity(a, b []float32) float64

CosineSimilarity returns the cosine of two vectors (assumed or forced to be normalised). Returns 0 when dimensions differ or either vector is zero.

func RegisterCallable 

func RegisterCallable(name string, fn func(string) ([]float32, error))

RegisterCallable registers a Go function as a named embedding provider.

func RemoveCJKDict added in v0.5.1 

func RemoveCJKDict() error

RemoveCJKDict deletes the downloaded file dict and reloads, falling back to the embedded dict (fulldict builds) or per-character tokenization. It is a no-op when nothing was downloaded.

func SetCJKDictDir added in v0.5.1 

func SetCJKDictDir(dir string)

SetCJKDictDir overrides where the downloadable dictionary lives (default ~/.ladyM/dict). Intended for startup-time configuration by tools and tests; the active segmenter is reloaded immediately.

func SetCJKTokenizerMode added in v0.8.2 

func SetCJKTokenizerMode(mode string)

SetCJKTokenizerMode picks the CJK tokenizer mode. Unknown values fall back to CJKTokenizerDict so a config typo cannot silently change what every stored vector means.

The dictionary load is the reason this exists: building gse's cedar trie costs ~520ms and happens once per process, on the first CJK run any call touches. The hook forks a process per tool call, so a Chinese prompt costs ~450ms per invocation under dict mode and ~35ms under chars mode.

func SetEmbeddedCJKDict added in v0.5.1 

func SetEmbeddedCJKDict(fn func() *gse.Segmenter)

SetEmbeddedCJKDict registers a segmenter provider used when no file dictionary is present (pass nil to unregister). It exists for the storage/fulldict side-effect package — importing that package is the import-path equivalent of building with -tags fulldict, letting library consumers embed the dictionary without touching their build scripts. The active segmenter is reloaded immediately.

func Tokenize 

func Tokenize(text string) []string

Tokenize is the lightweight tokenizer: words + punctuation as separate tokens, with camelCase and snake_case splitting so getUserName and get_user_name tokenize similarly. CJK script runs are segmented into words by gse (jieba dictionary) for Chinese and per character for kana/hangul.

Types

type CJKDictName added in v0.5.1 

type CJKDictName string

CJKDictName identifies a downloadable dictionary variant. 

const (
	// CJKDictZH is Chinese simplified + traditional (the default).
	CJKDictZH CJKDictName = "zh"
	// CJKDictZHS is Chinese simplified only (~40% smaller download).
	CJKDictZHS CJKDictName = "zh_s"
	// CJKDictZHT is Chinese traditional only.
	CJKDictZHT CJKDictName = "zh_t"
	// CJKDictJP is Japanese (kanji + kana; ~23.7MB).
	CJKDictJP CJKDictName = "jp"
)

type CJKDictStatus added in v0.5.1 

type CJKDictStatus struct {
	Available bool        `json:"available"`
	Source    string      `json:"source"`  // file | embedded | none
	Variant   CJKDictName `json:"variant"` // active variant, "" when none
	Dir       string      `json:"dir"`
	Version   string      `json:"version"`
	Bytes     int64       `json:"bytes"` // on-disk size when source == file
}

CJKDictStatus is the console/API view of the dictionary state.

func CJKDictStatusNow added in v0.5.1 

func CJKDictStatusNow() CJKDictStatus

CJKDictStatusNow reports the active dictionary state, triggering the lazy load if tokenization has not run yet.

func DownloadCJKDict added in v0.5.1 

func DownloadCJKDict() (CJKDictStatus, error)

DownloadCJKDict downloads the default (zh) dictionary to the default dir.

func DownloadCJKDictTo added in v0.5.1 

func DownloadCJKDictTo(dict CJKDictName, dir string, mirrorBase string) (CJKDictStatus, error)

DownloadCJKDictTo downloads the named dictionary variant into dir (defaults to ~/.ladyM/dict) and reloads the segmenter so the new dict takes effect immediately. It is the ONE place that touches the network — called only by the admin-triggered download endpoint; no LadyM command, startup path, or background loop ever invokes it. dict "" selects the default (zh); mirrorBase, when non-empty, replaces the default mirror list so air-gapped installs can point at an internal mirror serving the same layout (<base>/<RelPath>).

Every file is fully downloaded and sha256-verified before anything is written into place, and files of the previously installed variant that the new one does not use are cleaned up, so a failed or variant-switching download leaves a coherent dictionary behind.

type CJKVariantInfo added in v0.5.1 

type CJKVariantInfo struct {
	Name  CJKDictName `json:"name"`
	Desc  string      `json:"desc"`
	Bytes int64       `json:"bytes"`
}

CJKVariantInfo is the console/API view of one registry entry.

func CJKDictVariants added in v0.5.1 

func CJKDictVariants() []CJKVariantInfo

CJKDictVariants lists the downloadable dictionary variants for UI/API enumeration, in canonical order.

type CachedEmbedding 

type CachedEmbedding struct {
	// contains filtered or unexported fields
}

CachedEmbedding wraps an inner EmbeddingProvider with an LRU cache for Embed() calls. EmbedBatch always delegates straight through.

func NewCachedEmbedding 

func NewCachedEmbedding(inner EmbeddingProvider, size int) *CachedEmbedding

NewCachedEmbedding wraps inner with an LRU cache of the given size.

func (*CachedEmbedding) Dim 

func (c *CachedEmbedding) Dim() int

func (*CachedEmbedding) Embed 

func (c *CachedEmbedding) Embed(text string) ([]float32, error)

func (*CachedEmbedding) EmbedBatch 

func (c *CachedEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*CachedEmbedding) HealthCheck 

func (c *CachedEmbedding) HealthCheck() (bool, string)

type CallableEmbedding 

type CallableEmbedding struct {
	// contains filtered or unexported fields
}

CallableEmbedding wraps a Go function as an embedding provider.

func NewCallableEmbedding 

func NewCallableEmbedding(fn func(string) ([]float32, error), dim int) *CallableEmbedding

NewCallableEmbedding wraps fn as an embedding provider.

func (*CallableEmbedding) Dim 

func (c *CallableEmbedding) Dim() int

func (*CallableEmbedding) Embed 

func (c *CallableEmbedding) Embed(text string) ([]float32, error)

func (*CallableEmbedding) EmbedBatch 

func (c *CallableEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*CallableEmbedding) HealthCheck 

func (c *CallableEmbedding) HealthCheck() (bool, string)

type EmbeddingDimensionMismatch 

type EmbeddingDimensionMismatch struct {
	Stored     int
	Configured int
}

EmbeddingDimensionMismatch is raised when a reopened DB holds vectors of a different dim than the live provider.

func (*EmbeddingDimensionMismatch) Error 

func (e *EmbeddingDimensionMismatch) Error() string

type EmbeddingProvider 

type EmbeddingProvider interface {
	// Embed returns a float32 vector for text.
	Embed(text string) ([]float32, error)
	// EmbedBatch embeds a batch of texts (defaults to looping Embed).
	EmbedBatch(texts []string) ([][]float32, error)
	// Dim returns the vector dimension, or 0 when unknown until the first
	// embed call (deferred-dim providers such as Ollama).
	Dim() int
	// HealthCheck performs a one-shot probe for the web UI "test embedding".
	HealthCheck() (bool, string)
}

EmbeddingProvider is the contract every provider implements.

func MakeProvider 

func MakeProvider(cfg *config.Config) (EmbeddingProvider, error)

MakeProvider resolves the configured embedding provider.

type EmbeddingProviderError 

type EmbeddingProviderError struct {
	Msg string
}

EmbeddingProviderError is the base error for provider-side failures.

func (*EmbeddingProviderError) Error 

func (e *EmbeddingProviderError) Error() string

type EmbeddingTokenizerMismatch added in v0.8.2 

type EmbeddingTokenizerMismatch struct {
	Stored     string
	Configured string
}

EmbeddingTokenizerMismatch is raised when a reopened database holds vectors produced by a different CJK tokenizer than the one now configured. Unlike a dim mismatch the vectors still have the right shape, so mixing them degrades recall silently instead of failing — hence the explicit check.

func (*EmbeddingTokenizerMismatch) Error added in v0.8.2 

func (e *EmbeddingTokenizerMismatch) Error() string

type FakeHTTPPoster 

type FakeHTTPPoster struct {
	Responder    func(payload any) (any, error)
	ExpectedPath string
	LastPayload  any
	LastURL      string
}

FakeHTTPPoster is a test double. Responder maps payload → JSON-able result.

func (*FakeHTTPPoster) Post 

func (f *FakeHTTPPoster) Post(url string, payload any, headers map[string]string) (any, error)

type HTTPEmbedding 

type HTTPEmbedding struct {
	// contains filtered or unexported fields
}

HTTPEmbedding is a generic, template-driven embedding provider.

func NewHTTPEmbedding 

func NewHTTPEmbedding(opts HTTPEmbeddingOptions) *HTTPEmbedding

NewHTTPEmbedding builds a generic HTTP embedding provider.

func (*HTTPEmbedding) Dim 

func (h *HTTPEmbedding) Dim() int

func (*HTTPEmbedding) Embed 

func (h *HTTPEmbedding) Embed(text string) ([]float32, error)

func (*HTTPEmbedding) EmbedBatch 

func (h *HTTPEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*HTTPEmbedding) HealthCheck 

func (h *HTTPEmbedding) HealthCheck() (bool, string)

type HTTPEmbeddingOptions 

type HTTPEmbeddingOptions struct {
	BaseURL      string
	Request      string
	ResponsePath string
	Dim          int
	Model        string
	TimeoutS     float64
	// Poster, when non-nil, overrides the default real HTTP client (tests).
	Poster HTTPPoster
}

HTTPEmbeddingOptions configures the generic HTTP embedding provider.

type HTTPPoster 

type HTTPPoster interface {
	Post(url string, payload any, headers map[string]string) (any, error)
}

HTTPPoster is the minimal HTTP contract used by embedding providers so tests can inject a fake and never touch the network.

type HashingEmbedding 

type HashingEmbedding struct {
	// contains filtered or unexported fields
}

HashingEmbedding is the deterministic, offline embedding via feature hashing (unigram + bigram features, L2-normalised). Tokenization covers ASCII plus CJK scripts (dictionary-backed word segmentation for Chinese; per-character for kana/hangul), so Chinese, Japanese, and Korean text embed without any network or model download.

func NewHashingEmbedding 

func NewHashingEmbedding(dim int) *HashingEmbedding

NewHashingEmbedding returns a HashingEmbedding with the given dim.

func (*HashingEmbedding) Dim 

func (h *HashingEmbedding) Dim() int

func (*HashingEmbedding) Embed 

func (h *HashingEmbedding) Embed(text string) ([]float32, error)

func (*HashingEmbedding) EmbedBatch 

func (h *HashingEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*HashingEmbedding) HealthCheck 

func (h *HashingEmbedding) HealthCheck() (bool, string)

type InMemoryVectorIndex 

type InMemoryVectorIndex struct {
	// contains filtered or unexported fields
}

InMemoryVectorIndex is a brute-force cosine index (deterministic, perfect for tests and small workspaces). It replaces the sqlite-vec extension used by the Python port with a pure-Go equivalent; vectors are still persisted as BLOBs in the store so a reopened store can rebuild the index.

func NewInMemoryVectorIndex 

func NewInMemoryVectorIndex(dim int) *InMemoryVectorIndex

NewInMemoryVectorIndex returns an empty index of the given dim.

func (*InMemoryVectorIndex) Delete 

func (ix *InMemoryVectorIndex) Delete(itemID string)

func (*InMemoryVectorIndex) Len 

func (ix *InMemoryVectorIndex) Len() int

func (*InMemoryVectorIndex) Search 

func (ix *InMemoryVectorIndex) Search(query []float32, topK int) []SearchHit

func (*InMemoryVectorIndex) Upsert 

func (ix *InMemoryVectorIndex) Upsert(itemID string, vector []float32) error

type OllamaEmbedding 

type OllamaEmbedding struct {
	// contains filtered or unexported fields
}

OllamaEmbedding targets the Ollama /api/embeddings endpoint. Dim is deferred until the first embed call.

func NewOllamaEmbedding 

func NewOllamaEmbedding(baseURL, model string, timeoutS float64, client HTTPPoster) *OllamaEmbedding

NewOllamaEmbedding builds an OllamaEmbedding (client may be nil for real HTTP).

func (*OllamaEmbedding) Dim 

func (o *OllamaEmbedding) Dim() int

func (*OllamaEmbedding) Embed 

func (o *OllamaEmbedding) Embed(text string) ([]float32, error)

func (*OllamaEmbedding) EmbedBatch 

func (o *OllamaEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*OllamaEmbedding) HealthCheck 

func (o *OllamaEmbedding) HealthCheck() (bool, string)

type OpenAIEmbedding 

type OpenAIEmbedding struct {
	// contains filtered or unexported fields
}

OpenAIEmbedding targets the OpenAI (or OpenAI-compatible) embeddings API.

func NewOpenAIEmbedding 

func NewOpenAIEmbedding(model, baseURL, apiKey string, timeoutS float64) *OpenAIEmbedding

NewOpenAIEmbedding builds an OpenAIEmbedding. Empty baseURL defaults to https://api.openai.com/v1.

func (*OpenAIEmbedding) Dim 

func (o *OpenAIEmbedding) Dim() int

func (*OpenAIEmbedding) Embed 

func (o *OpenAIEmbedding) Embed(text string) ([]float32, error)

func (*OpenAIEmbedding) EmbedBatch 

func (o *OpenAIEmbedding) EmbedBatch(texts []string) ([][]float32, error)

func (*OpenAIEmbedding) HealthCheck 

func (o *OpenAIEmbedding) HealthCheck() (bool, string)

type PostgresStore added in v0.4.0 

type PostgresStore struct {
	// contains filtered or unexported fields
}

PostgresStore is the PostgreSQL+pgvector Store implementation. It mirrors SQLiteStore method-by-method; differences are confined to SQL dialect ($N placeholders, ON CONFLICT ... (col), JSONB, vector(dim)) and to the lock/index mechanisms (pg advisory lock, pgvector HNSW index instead of the process-local InMemoryVectorIndex + flock file).

func NewPostgresStore added in v0.4.0 

func NewPostgresStore(dsn string, dim int) (*PostgresStore, error)

NewPostgresStore connects to the PostgreSQL database at dsn (pgxpool) and applies the schema idempotently. dim sizes the pgvector embedding column.

func (*PostgresStore) Close added in v0.4.0 

func (s *PostgresStore) Close() error

Close releases the connection pool.

func (*PostgresStore) Count added in v0.4.0 

func (s *PostgresStore) Count(workspace string) (map[string]int, error)

Count returns a "layer/type" → count map (optionally scoped to workspace).

func (*PostgresStore) CountCodeSymbols added in v0.4.0 

func (s *PostgresStore) CountCodeSymbols() (int, error)

CountCodeSymbols returns the number of code symbol projections.

func (*PostgresStore) CountEdges added in v0.4.0 

func (s *PostgresStore) CountEdges() (int, error)

CountEdges returns the number of edges.

func (*PostgresStore) DeleteCodeArtifactsByFile added in v0.7.0 

func (s *PostgresStore) DeleteCodeArtifactsByFile(workspace, filePath string) error

DeleteCodeArtifactsByFile removes every code artifact of one file (re-index cleanup, spec §6.4): code_refs touching the file's symbols in either direction, the code_symbols projection rows, and the code_symbol memories. The refs delete must run before the code_symbols delete it subqueries. No vector index sync is needed: VectorSearch reads the table.

func (*PostgresStore) DeleteMemoriesByTypeSource added in v0.4.0 

func (s *PostgresStore) DeleteMemoriesByTypeSource(typ, source, workspace string) error

DeleteMemoriesByTypeSource deletes memories matching type+source+workspace (code indexer re-index write path).

func (*PostgresStore) DeleteMemory added in v0.4.0 

func (s *PostgresStore) DeleteMemory(id string) error

DeleteMemory deletes a memory (the HNSW index entry goes with the row).

func (*PostgresStore) DeleteSymbolMemories added in v0.4.0 

func (s *PostgresStore) DeleteSymbolMemories(qualifiedName, workspace string) error

DeleteSymbolMemories deletes the code_symbol memories projecting the given qualified name (code indexer re-index write path).

func (*PostgresStore) DeleteUser added in v0.4.0 

func (s *PostgresStore) DeleteUser(username string) error

DeleteUser deletes a user; a missing username is a no-op.

func (*PostgresStore) EpisodicContentsSince added in v0.4.0 

func (s *PostgresStore) EpisodicContentsSince(workspace string, since float64) ([]string, error)

EpisodicContentsSince returns the content strings of episodic events in workspace created at or after since (attention gate recent-duplicate scan).

func (*PostgresStore) FindByHash added in v0.4.0 

func (s *PostgresStore) FindByHash(contentHash, workspace string) (*schema.Memory, error)

FindByHash returns the memory with the given content hash (optionally scoped to workspace), or nil.

func (*PostgresStore) GetIndexedHash added in v0.4.0 

func (s *PostgresStore) GetIndexedHash(filePath string) (string, error)

GetIndexedHash returns the recorded body hash for a file, or "".

func (*PostgresStore) GetMemory added in v0.4.0 

func (s *PostgresStore) GetMemory(id string) (*schema.Memory, error)

GetMemory returns the memory with the given id, or nil.

func (*PostgresStore) GetMeta added in v0.4.0 

func (s *PostgresStore) GetMeta(key string) (string, error)

GetMeta returns the value for a meta key, or "".

func (*PostgresStore) GetUser added in v0.4.0 

func (s *PostgresStore) GetUser(username string) (*schema.User, error)

GetUser returns the user with the given username, or nil.

func (*PostgresStore) InvalidateEdge added in v0.4.0 

func (s *PostgresStore) InvalidateEdge(edgeID string, t float64) error

InvalidateEdge marks an edge no longer current (sets valid_to).

func (*PostgresStore) IterMemories added in v0.4.0 

func (s *PostgresStore) IterMemories(workspace, layer, typ string) ([]*schema.Memory, error)

IterMemories returns memories matching the optional filters.

func (*PostgresStore) ListUsers added in v0.4.0 

func (s *PostgresStore) ListUsers() ([]*schema.User, error)

ListUsers returns all users sorted by username.

func (*PostgresStore) NeighborCounts added in v0.4.0 

func (s *PostgresStore) NeighborCounts() (map[string]int, error)

NeighborCounts returns {memory_id: neighbour_count} for spreading activation.

func (*PostgresStore) Neighbors added in v0.4.0 

func (s *PostgresStore) Neighbors(memID, relation string) ([]*schema.Edge, error)

Neighbors returns valid (valid_to IS NULL) edges touching memID.

func (*PostgresStore) Ping added in v0.4.0 

func (s *PostgresStore) Ping() error

Ping checks storage connectivity (health probe).

func (*PostgresStore) PutCodeRefs added in v0.4.0 

func (s *PostgresStore) PutCodeRefs(refs []*schema.CodeRef) error

PutCodeRefs bulk-inserts cross references atomically: one transaction, so a failure rolls back the whole batch.

func (*PostgresStore) PutCodeSymbol added in v0.4.0 

func (s *PostgresStore) PutCodeSymbol(sym *schema.CodeSymbol) error

PutCodeSymbol inserts or updates a code symbol projection.

func (*PostgresStore) PutEdge added in v0.4.0 

func (s *PostgresStore) PutEdge(e *schema.Edge) error

PutEdge inserts or updates an edge.

func (*PostgresStore) PutMemory added in v0.4.0 

func (s *PostgresStore) PutMemory(mem *schema.Memory, vector []float32) error

func (*PostgresStore) PutUser added in v0.4.0 

func (s *PostgresStore) PutUser(u *schema.User) error

PutUser inserts or updates a user (upsert by username).

func (*PostgresStore) RebuildVectorIndex added in v0.4.0 

func (s *PostgresStore) RebuildVectorIndex(newDim int)

RebuildVectorIndex resets the vector index at a new dim: drops the HNSW index, nulls all embeddings, alters the column to vector(newDim) and recreates the index — one transaction. Mirrors SQLiteStore's reset of the in-memory index; the interface returns no error, so failures are warned.

func (*PostgresStore) RefsForSymbol added in v0.4.0 

func (s *PostgresStore) RefsForSymbol(qualifiedName, direction string) ([]*schema.CodeRef, error)

RefsForSymbol returns cross references for a qualified name.

func (*PostgresStore) SetIndexed added in v0.4.0 

func (s *PostgresStore) SetIndexed(filePath, bodyHash string, now float64) error

SetIndexed records the body hash for a file.

func (*PostgresStore) SetMeta added in v0.4.0 

func (s *PostgresStore) SetMeta(key, value string) error

SetMeta upserts a meta key/value.

func (*PostgresStore) SweepDanglingEdges added in v0.7.0 

func (s *PostgresStore) SweepDanglingEdges() (int, error)

SweepDanglingEdges deletes valid edges whose endpoint memories no longer exist (legacy pre-FK databases keep them around) and returns the count.

func (*PostgresStore) SymbolsForFile added in v0.4.0 

func (s *PostgresStore) SymbolsForFile(filePath string) ([]*schema.CodeSymbol, error)

SymbolsForFile returns code symbols for a file, ordered by line.

func (*PostgresStore) TouchMemories added in v0.4.0 

func (s *PostgresStore) TouchMemories(ids []string, now float64) error

TouchMemories bumps access_count / last_access_at for every listed id in a single UPDATE (recall's access bookkeeping used to issue one UPDATE per id). An empty slice is a no-op.

func (*PostgresStore) TryAcquireIndexLock added in v0.4.0 

func (s *PostgresStore) TryAcquireIndexLock() (func(), error)

TryAcquireIndexLock takes the cross-process code-index lock as a pg advisory lock. Contention fails fast with ErrIndexLockHeld — callers do not queue.

func (*PostgresStore) TryAcquireWorkerLock added in v0.6.0 

func (s *PostgresStore) TryAcquireWorkerLock() (func(), error)

TryAcquireWorkerLock takes the cross-process System2 worker-cycle lock as a pg advisory lock. Contention fails fast with ErrWorkerLockHeld — callers skip the cycle instead of queueing.

func (*PostgresStore) UpdateMemoryContent added in v0.4.0 

func (s *PostgresStore) UpdateMemoryContent(id, content, summary string, tags []string, vector []float32, now float64) error

UpdateMemoryContent patches content/summary/tags/updated_at in place. A nil vector leaves the embedding column and content_hash alone (unlike the PutMemory upsert, which would NULL them); a non-nil vector rewrites the embedding and recomputes content_hash. A missing id is a no-op.

func (*PostgresStore) VectorSearch added in v0.4.0 

func (s *PostgresStore) VectorSearch(queryVec []float32, topK int) []SearchHit

VectorSearch runs a cosine top-k search via pgvector. Ordering matches InMemoryVectorIndex.Search: similarity desc, id asc as deterministic tiebreak; the vector index is global (workspace filtering happens in the recall layer above). Query failures degrade to nil, mirroring the in-memory index's error-free Search signature (same tolerance as warmIndexFromBlobs).

Degenerate vectors are handled outside the ANN path: pgvector's cosine distance is NaN for zero-norm vectors, and the HNSW index omits them entirely, while the in-memory baseline assigns them similarity 0. A zero-norm query therefore takes a plain id-ordered scan, and zero-norm stored vectors are appended via UNION ALL with a literal 0 similarity.

func (*PostgresStore) Workspaces added in v0.4.0 

func (s *PostgresStore) Workspaces() ([]string, error)

Workspaces lists distinct workspaces.

type RealHTTPPoster 

type RealHTTPPoster struct {
	Timeout time.Duration
}

RealHTTPPoster is the net/http-backed client.

func NewRealHTTPPoster 

func NewRealHTTPPoster(timeoutS float64) *RealHTTPPoster

NewRealHTTPPoster returns a RealHTTPPoster with the given timeout.

func (*RealHTTPPoster) Post 

func (c *RealHTTPPoster) Post(url string, payload any, headers map[string]string) (any, error)

type SQLiteStore 

type SQLiteStore struct {
	DBPath string
	Dim    int
	// contains filtered or unexported fields
}

SQLiteStore is the single persistence layer for LadyM. It owns the SQLite connection and the (in-memory) vector index.

The vector index is warm-loaded lazily: it is either cold (nothing read from the database yet) or hot (it covers every stored embedding). Writes may land at any time in either state. Warming happens once, on the first VectorSearch, and reloads every row with Upsert, so it is idempotent with respect to any write that already reached the index. The hook pays NewStore once per tool call, so an eager O(n) warm is an O(n) tax on every event.

func NewStore 

func NewStore(dbPath string, dim int, preferSQLiteVec bool, enableWAL bool) (*SQLiteStore, error)

NewStore opens (or creates) a SQLite store at dbPath.

NOTE: the Python port used the sqlite-vec loadable extension for persistent ANN search. The Go port always uses the pure-Go InMemoryVectorIndex (the sqlite-vec C extension has no Go binding); embeddings are still persisted as BLOBs and warmed into the index on reopen, so behaviour is identical. The preferSQLiteVec flag is accepted for config parity but has no effect.

func (*SQLiteStore) Close 

func (s *SQLiteStore) Close() error

Close commits and closes the underlying connection.

func (*SQLiteStore) Count 

func (s *SQLiteStore) Count(workspace string) (map[string]int, error)

Count returns a "layer/type" → count map (optionally scoped to workspace).

func (*SQLiteStore) CountCodeSymbols added in v0.4.0 

func (s *SQLiteStore) CountCodeSymbols() (int, error)

CountCodeSymbols returns the number of code symbol projections.

func (*SQLiteStore) CountEdges 

func (s *SQLiteStore) CountEdges() (int, error)

CountEdges returns the number of edges.

func (*SQLiteStore) DeleteCodeArtifactsByFile added in v0.7.0 

func (s *SQLiteStore) DeleteCodeArtifactsByFile(workspace, filePath string) error

DeleteCodeArtifactsByFile removes every code artifact of one file (re-index cleanup, spec §6.4): code_refs touching the file's symbols in either direction, the code_symbols projection rows, and the code_symbol memories, then syncs the in-process vector index for exactly the deleted memories. The refs delete must run before the code_symbols delete it subqueries.

func (*SQLiteStore) DeleteMemoriesByTypeSource added in v0.4.0 

func (s *SQLiteStore) DeleteMemoriesByTypeSource(typ, source, workspace string) error

DeleteMemoriesByTypeSource deletes memories matching type+source+workspace (code indexer re-index write path).

func (*SQLiteStore) DeleteMemory 

func (s *SQLiteStore) DeleteMemory(id string) error

DeleteMemory deletes a memory and its vector entry.

func (*SQLiteStore) DeleteSymbolMemories added in v0.4.0 

func (s *SQLiteStore) DeleteSymbolMemories(qualifiedName, workspace string) error

DeleteSymbolMemories deletes the code_symbol memories projecting the given qualified name (code indexer re-index write path).

func (*SQLiteStore) DeleteUser added in v0.4.0 

func (s *SQLiteStore) DeleteUser(username string) error

DeleteUser deletes a user; a missing username is a no-op.

func (*SQLiteStore) EpisodicContentsSince 

func (s *SQLiteStore) EpisodicContentsSince(workspace string, since float64) ([]string, error)

EpisodicContentsSince returns the content strings of episodic events in workspace created at or after since (used by the attention gate's recent-duplicate scan; pushes the time-window cut into SQL).

func (*SQLiteStore) FindByHash 

func (s *SQLiteStore) FindByHash(contentHash, workspace string) (*schema.Memory, error)

FindByHash returns the memory with the given content hash (optionally scoped to workspace), or nil.

func (*SQLiteStore) GetIndexedHash 

func (s *SQLiteStore) GetIndexedHash(filePath string) (string, error)

GetIndexedHash returns the recorded body hash for a file, or "".

func (*SQLiteStore) GetMemory 

func (s *SQLiteStore) GetMemory(id string) (*schema.Memory, error)

GetMemory returns the memory with the given id, or nil.

func (*SQLiteStore) GetMeta 

func (s *SQLiteStore) GetMeta(key string) (string, error)

GetMeta returns the value for a meta key, or "".

func (*SQLiteStore) GetUser added in v0.4.0 

func (s *SQLiteStore) GetUser(username string) (*schema.User, error)

GetUser returns the user with the given username, or nil.

func (*SQLiteStore) InvalidateEdge added in v0.4.0 

func (s *SQLiteStore) InvalidateEdge(edgeID string, t float64) error

InvalidateEdge marks an edge no longer current (sets valid_to).

func (*SQLiteStore) IterMemories 

func (s *SQLiteStore) IterMemories(workspace, layer, typ string) ([]*schema.Memory, error)

IterMemories returns memories matching the optional filters.

func (*SQLiteStore) ListUsers added in v0.4.0 

func (s *SQLiteStore) ListUsers() ([]*schema.User, error)

ListUsers returns all users sorted by username.

func (*SQLiteStore) NeighborCounts 

func (s *SQLiteStore) NeighborCounts() (map[string]int, error)

NeighborCounts returns {memory_id: neighbour_count} for spreading activation.

func (*SQLiteStore) Neighbors 

func (s *SQLiteStore) Neighbors(memID, relation string) ([]*schema.Edge, error)

Neighbors returns valid (valid_to IS NULL) edges touching memID.

func (*SQLiteStore) Ping added in v0.4.0 

func (s *SQLiteStore) Ping() error

Ping checks storage connectivity (health probe).

func (*SQLiteStore) PutCodeRefs 

func (s *SQLiteStore) PutCodeRefs(refs []*schema.CodeRef) error

PutCodeRefs bulk-inserts cross references atomically: one transaction, so a failure rolls back the whole batch (Python: executemany in a single commit).

func (*SQLiteStore) PutCodeSymbol 

func (s *SQLiteStore) PutCodeSymbol(sym *schema.CodeSymbol) error

PutCodeSymbol inserts or updates a code symbol projection.

func (*SQLiteStore) PutEdge 

func (s *SQLiteStore) PutEdge(e *schema.Edge) error

PutEdge inserts or updates an edge.

func (*SQLiteStore) PutMemory 

func (s *SQLiteStore) PutMemory(mem *schema.Memory, vector []float32) error

func (*SQLiteStore) PutUser added in v0.4.0 

func (s *SQLiteStore) PutUser(u *schema.User) error

PutUser inserts or updates a user (upsert by username).

func (*SQLiteStore) RebuildVectorIndex 

func (s *SQLiteStore) RebuildVectorIndex(newDim int)

RebuildVectorIndex resets the index at a new dim (used on dim change). The index goes cold again: the next VectorSearch reloads at the new dim.

func (*SQLiteStore) RefsForSymbol 

func (s *SQLiteStore) RefsForSymbol(qualifiedName, direction string) ([]*schema.CodeRef, error)

RefsForSymbol returns cross references for a qualified name.

func (*SQLiteStore) SetIndexed 

func (s *SQLiteStore) SetIndexed(filePath, bodyHash string, now float64) error

SetIndexed records the body hash for a file.

func (*SQLiteStore) SetMeta 

func (s *SQLiteStore) SetMeta(key, value string) error

SetMeta upserts a meta key/value.

func (*SQLiteStore) SweepDanglingEdges added in v0.7.0 

func (s *SQLiteStore) SweepDanglingEdges() (int, error)

SweepDanglingEdges deletes valid edges whose endpoint memories no longer exist (legacy pre-FK databases keep them around) and returns the count. Edges are not part of the vector index, so no index maintenance is needed.

func (*SQLiteStore) SymbolsForFile 

func (s *SQLiteStore) SymbolsForFile(filePath string) ([]*schema.CodeSymbol, error)

SymbolsForFile returns code symbols for a file, ordered by line.

func (*SQLiteStore) TouchMemories added in v0.4.0 

func (s *SQLiteStore) TouchMemories(ids []string, now float64) error

TouchMemories bumps access_count / last_access_at for every listed id in a single UPDATE (recall's access bookkeeping used to issue one UPDATE per id). An empty slice is a no-op.

func (*SQLiteStore) TryAcquireIndexLock added in v0.4.0 

func (s *SQLiteStore) TryAcquireIndexLock() (func(), error)

TryAcquireIndexLock takes the cross-process code-index lock on <db>.index.lock and returns the release function. Contention fails fast with ErrIndexLockHeld — callers do not queue.

func (*SQLiteStore) TryAcquireWorkerLock added in v0.6.0 

func (s *SQLiteStore) TryAcquireWorkerLock() (func(), error)

TryAcquireWorkerLock takes the cross-process System2 worker-cycle lock on <db>.worker.lock and returns the release function. Contention fails fast with ErrWorkerLockHeld — callers skip the cycle instead of queueing.

func (*SQLiteStore) UpdateMemoryContent added in v0.4.0 

func (s *SQLiteStore) UpdateMemoryContent(id, content, summary string, tags []string, vector []float32, now float64) error

UpdateMemoryContent patches content/summary/tags/updated_at in place. A nil vector leaves the embedding column and content_hash alone (unlike the PutMemory upsert, which would NULL them); a non-nil vector rewrites the embedding, recomputes content_hash and re-indexes the vector. A missing id is a no-op.

func (*SQLiteStore) UsingSQLiteVec 

func (s *SQLiteStore) UsingSQLiteVec() bool

UsingSQLiteVec always returns false (see NewStore).

func (*SQLiteStore) VectorSearch added in v0.4.0 

func (s *SQLiteStore) VectorSearch(queryVec []float32, topK int) []SearchHit

VectorSearch runs a cosine top-k search over the in-memory vector index, warming it from the database on first use.

func (*SQLiteStore) Workspaces 

func (s *SQLiteStore) Workspaces() ([]string, error)

Workspaces lists distinct workspaces.

type SearchHit 

type SearchHit struct {
	ID         string
	Similarity float64
}

SearchHit is one (id, similarity) pair returned by Search.

type Store added in v0.4.0 

type Store interface {
	// lifecycle
	Close() error

	// Ping checks storage connectivity (health probe): SQLite runs SELECT 1,
	// Postgres pings the pool.
	Ping() error

	// memory CRUD
	PutMemory(mem *schema.Memory, vector []float32) error
	GetMemory(id string) (*schema.Memory, error)
	DeleteMemory(id string) error
	// UpdateMemoryContent patches one memory's content/summary/tags and bumps
	// updated_at (now==0 falls back to the wall clock). Unlike PutMemory's
	// upsert — where a nil vector NULLs the embedding column — a nil vector
	// here leaves embedding and content_hash untouched; a non-nil vector
	// rewrites the embedding and recomputes content_hash from content. A
	// missing id is a no-op (callers 404 beforehand).
	UpdateMemoryContent(id, content, summary string, tags []string, vector []float32, now float64) error
	TouchMemories(ids []string, now float64) error // one batched UPDATE; empty slice is a no-op
	IterMemories(workspace, layer, typ string) ([]*schema.Memory, error)
	FindByHash(contentHash, workspace string) (*schema.Memory, error)
	EpisodicContentsSince(workspace string, since float64) ([]string, error)
	Count(workspace string) (map[string]int, error)

	// vector retrieval (cosine similarity, deterministic top-k; semantics
	// anchored to InMemoryVectorIndex.Search)
	VectorSearch(queryVec []float32, topK int) []SearchHit

	// L4 associative graph
	PutEdge(e *schema.Edge) error
	Neighbors(memID, relation string) ([]*schema.Edge, error)
	CountEdges() (int, error)
	NeighborCounts() (map[string]int, error)
	InvalidateEdge(edgeID string, t float64) error // UPDATE edges SET valid_to=t WHERE id=edgeID
	// SweepDanglingEdges deletes valid edges whose endpoint memories no
	// longer exist (legacy pre-FK databases) and returns how many went.
	SweepDanglingEdges() (int, error)

	// code projections
	PutCodeSymbol(sym *schema.CodeSymbol) error
	PutCodeRefs(refs []*schema.CodeRef) error
	SymbolsForFile(filePath string) ([]*schema.CodeSymbol, error)
	RefsForSymbol(qualifiedName, direction string) ([]*schema.CodeRef, error)
	CountCodeSymbols() (int, error)

	// index state / metadata
	GetIndexedHash(filePath string) (string, error)
	SetIndexed(filePath, bodyHash string, now float64) error
	Workspaces() ([]string, error)
	GetMeta(key string) (string, error)
	SetMeta(key, value string) error

	// users (HTTP data-plane Basic auth accounts)
	PutUser(u *schema.User) error                  // upsert by username
	GetUser(username string) (*schema.User, error) // nil when absent
	DeleteUser(username string) error              // missing username is a no-op
	ListUsers() ([]*schema.User, error)            // sorted by username

	// Cross-process mutex for code indexing. The SQLite implementation is the
	// existing flock(<db>.index.lock); it returns the release function. When
	// the lock is already held it returns an error matching ErrIndexLockHeld
	// — the user-facing IndexInProgressError type lives in the code package
	// (code imports storage, so storage cannot reference it) and the caller
	// performs the translation.
	TryAcquireIndexLock() (func(), error)

	// Cross-process mutex for System2 worker cycles, so redundant worker
	// replicas never run the same cycle concurrently (repeat LLM calls, write
	// conflicts). Callers acquire per cycle and release afterwards; when the
	// lock is already held it returns an error matching ErrWorkerLockHeld —
	// callers skip the cycle instead of queueing.
	TryAcquireWorkerLock() (func(), error)

	// RebuildVectorIndex resets the vector index at a new dim (engine's
	// enforceEmbeddingDim on dim change).
	RebuildVectorIndex(newDim int)

	// The two deletes the code/indexer.go write path needs.
	DeleteMemoriesByTypeSource(typ, source, workspace string) error // DELETE FROM memories WHERE type=? AND source=? AND workspace=?
	DeleteSymbolMemories(qualifiedName, workspace string) error     // DELETE FROM memories WHERE type='code_symbol' AND workspace=? AND id IN (SELECT memory_id FROM code_symbols WHERE qualified_name=?)
	// DeleteCodeArtifactsByFile removes every code artifact of one file
	// (re-index cleanup, spec §6.4): code_refs touching the file's symbols in
	// either direction, the code_symbols projection rows, and the code_symbol
	// memories. SQLite also syncs its in-process vector index.
	DeleteCodeArtifactsByFile(workspace, filePath string) error
}

Store is the persistence contract used by engine / operations / layers / code. SQLiteStore is the reference implementation; the interface exists so no package outside storage depends on the concrete type or the raw *sql.DB.

func OpenStore added in v0.4.0 

func OpenStore(cfg *config.Config, dim int) (Store, error)

OpenStore builds the Store implementation selected by cfg.StoreBackend. "sqlite" (the default) is the personal-edition path and behaves exactly as before; "postgres" requires a DSN (store.dsn / LADYM_STORE_DSN, or store.dsn_env indirection resolved by the config loader).

type VectorIndex 

type VectorIndex interface {
	Upsert(itemID string, vector []float32) error
	Search(query []float32, topK int) []SearchHit
	Delete(itemID string)
	Len() int
}

VectorIndex inserts/queries/deletes vectors keyed by an arbitrary id.

Source Files

View all Source files

Directories

Path Synopsis
Package fulldict embeds the gse Chinese dictionary (simplified + traditional, ~+31MB binary) so CJK word segmentation works with zero downloads and zero build flags.
 Package fulldict embeds the gse Chinese dictionary (simplified + traditional, ~+31MB binary) so CJK word segmentation works with zero downloads and zero build flags.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.