semantic

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 17, 2026 License: MIT Imports: 35 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// BoostFactor is the score boost per additional query match beyond the first
	BoostFactor = 0.05
	// MaxScore is the maximum allowed boosted score
	MaxScore = 1.0
)
View Source 
const (
	// MaxQueries is the maximum number of queries allowed in a single Multisearch call
	MaxQueries = 10
)

Variables

View Source 
var (
	// ErrNotFound indicates that a chunk was not found in storage
	ErrNotFound = errors.New("chunk not found")

	// ErrMemoryNotFound indicates that a memory entry was not found in storage
	ErrMemoryNotFound = errors.New("memory entry not found")

	// ErrStorageClosed indicates that the storage has been closed
	ErrStorageClosed = errors.New("storage is closed")
)
View Source 
var ErrEmptyIndex = errors.New("cannot calibrate: index is empty")

ErrEmptyIndex indicates calibration cannot proceed with an empty index

Functions

func ApplyPercentileRelevanceLabels added in v1.6.0 

func ApplyPercentileRelevanceLabels(results []SearchResult)

ApplyPercentileRelevanceLabels applies percentile-based relevance labels to results. This is useful for multi-profile search where calibration data is not available.

func CalculateAgeDays added in v1.6.0 

func CalculateAgeDays(mtime, now time.Time) float64

CalculateAgeDays calculates the age in days between a file's modification time and now. Returns 0 if mtime is zero (unset) or in the future (clamped).

func CalculateBoostedScore added in v1.6.0 

func CalculateBoostedScore(baseScore float32, matchCount int) float32

CalculateBoostedScore computes the boosted score based on match count. Formula: BoostedScore = BaseScore + (0.05 * (MatchCount - 1)) Score is capped at 1.0

func CalculateRecencyBoost added in v1.6.0 

func CalculateRecencyBoost(ageDays float64, cfg RecencyConfig) float64

CalculateRecencyBoost calculates the recency boost multiplier for a file. Formula: boost = 1 + factor * exp(-age_days / half_life_days)

The boost is always >= 1.0 (no penalty for old files). Maximum boost is 1 + factor (when age_days = 0). Boost approaches 1.0 as age_days approaches infinity.

Negative age_days (future dates) are clamped to 0 (max boost).

func EscapeFTS5Query added in v1.6.0 

func EscapeFTS5Query(query string) string

EscapeFTS5Query escapes special FTS5 characters in a query string. This prevents FTS5 syntax errors from user input containing operators. Note: This converts FTS5 operators to literal text - if you want to allow users to use FTS5 operators (AND, OR, NOT, *), don't escape.

func FormatError 

func FormatError(err error) string

FormatError formats an error with user-friendly output

func GetChunkBuffer 

func GetChunkBuffer() *chunkBuffer

GetChunkBuffer gets a chunk buffer from the pool

func GetEmbeddingBuffer 

func GetEmbeddingBuffer(size int) *[]byte

GetEmbeddingBuffer gets a buffer from the pool, sized appropriately

func IndexPath 

func IndexPath(repoRoot string) string

IndexPath returns the default index path for a repository

func IsValidOutputFormat added in v1.6.0 

func IsValidOutputFormat(format string) bool

IsValidOutputFormat checks if the given format string is valid

func LabelAllByPercentile added in v1.6.0 

func LabelAllByPercentile(scores []float32) []string

LabelAllByPercentile assigns relevance labels to all scores in a single pass. This is O(n log n) vs O(n² log n) when calling LabelByPercentile in a loop. Returns a slice of labels corresponding to each score in the input slice.

func LabelByPercentile added in v1.6.0 

func LabelByPercentile(score float32, allScores []float32) string

LabelByPercentile assigns relevance labels based on the score's position within the result set. Used as fallback when no calibration data is available. Distribution: top 20% = "high", middle 50% = "medium", bottom 30% = "low"

func LabelRelevance added in v1.6.0 

func LabelRelevance(score float32, cal *CalibrationMetadata) string

LabelRelevance returns a human-readable relevance label based on calibrated thresholds. Returns "high", "medium", or "low" based on score comparison to calibration thresholds. Returns empty string if calibration is nil (caller should use LabelByPercentile as fallback).

func LanguageFromExtension 

func LanguageFromExtension(filename string) string

LanguageFromExtension extracts the language identifier from a filename

func PutChunkBuffer 

func PutChunkBuffer(buf *chunkBuffer)

PutChunkBuffer returns a chunk buffer to the pool

func PutEmbeddingBuffer 

func PutEmbeddingBuffer(buf *[]byte)

PutEmbeddingBuffer returns a buffer to the pool

func ValidOutputFormats added in v1.6.0 

func ValidOutputFormats() []string

ValidOutputFormats returns all valid output format values

func ValidateRecencyConfig added in v1.6.0 

func ValidateRecencyConfig(cfg RecencyConfig) error

ValidateRecencyConfig validates the recency configuration parameters. Returns an error if Factor < 0 or HalfLifeDays <= 0.

Types

type CalibrationMetadata added in v1.6.0 

type CalibrationMetadata struct {
	EmbeddingModel    string    `json:"embedding_model"`
	CalibrationDate   time.Time `json:"calibration_date"`
	PerfectMatchScore float32   `json:"perfect_match_score"`
	BaselineScore     float32   `json:"baseline_score"`
	ScoreRange        float32   `json:"score_range"`
	HighThreshold     float32   `json:"high_threshold"`
	MediumThreshold   float32   `json:"medium_threshold"`
	LowThreshold      float32   `json:"low_threshold"`
}

CalibrationMetadata stores calibration results for score normalization. Different embedding models produce vastly different score distributions. This metadata allows normalizing scores to comparable thresholds.

func RunCalibration added in v1.6.0 

func RunCalibration(ctx context.Context, storage Storage, embedder EmbedderInterface, modelName string) (*CalibrationMetadata, error)

RunCalibration executes the calibration workflow to determine score thresholds. It runs self-match probes to find the "perfect match" score and unrelated probes to find the baseline, then calculates normalized thresholds.

Returns ErrEmptyIndex if the index has no chunks to probe.

type Chunk 

type Chunk struct {
	ID        string    `json:"id"`
	FilePath  string    `json:"file_path"`
	Type      ChunkType `json:"type"`
	Name      string    `json:"name"`
	Signature string    `json:"signature,omitempty"`
	Content   string    `json:"content"`
	StartLine int       `json:"start_line"`
	EndLine   int       `json:"end_line"`
	Language  string    `json:"language"`
	Domain    string    `json:"domain,omitempty"`     // Source profile/collection (e.g., "code", "docs") - set during indexing based on profile
	FileMtime int64     `json:"file_mtime,omitempty"` // Unix timestamp of file modification time
}

Chunk represents a semantic unit of code (function, struct, etc.)

func (*Chunk) GenerateID 

func (c *Chunk) GenerateID() string

GenerateID creates a deterministic ID for the chunk based on its content

func (*Chunk) Preview added in v1.6.0 

func (c *Chunk) Preview() string

Preview returns a short preview of the chunk content for display purposes. Prefers Signature if available, otherwise uses truncated Content. Replaces newlines with spaces and truncates to 150 characters with "..." suffix.

type ChunkType 

type ChunkType int

ChunkType represents the type of code chunk 

const (
	ChunkFunction ChunkType = iota
	ChunkMethod
	ChunkStruct
	ChunkInterface
	ChunkFile
)

func ParseChunkType 

func ParseChunkType(s string) (ChunkType, error)

ParseChunkType parses a string into a ChunkType

func (ChunkType) MarshalJSON 

func (ct ChunkType) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler for ChunkType

func (ChunkType) String 

func (ct ChunkType) String() string

String returns the string representation of ChunkType

func (*ChunkType) UnmarshalJSON 

func (ct *ChunkType) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler for ChunkType

type ChunkWithEmbedding added in v1.6.0 

type ChunkWithEmbedding struct {
	Chunk     Chunk
	Embedding []float32
}

ChunkWithEmbedding pairs a chunk with its embedding for batch operations

type Chunker 

type Chunker interface {
	// Chunk breaks source code into semantic chunks
	Chunk(path string, content []byte) ([]Chunk, error)

	// SupportedExtensions returns the file extensions this chunker handles
	SupportedExtensions() []string
}

Chunker is the interface for language-specific code chunking

type ChunkerFactory 

type ChunkerFactory struct {
	// contains filtered or unexported fields
}

ChunkerFactory manages language-specific chunkers

func NewChunkerFactory 

func NewChunkerFactory() *ChunkerFactory

NewChunkerFactory creates a new chunker factory

func (*ChunkerFactory) GetByExtension 

func (f *ChunkerFactory) GetByExtension(filename string) (Chunker, bool)

GetByExtension returns the appropriate chunker based on filename extension

func (*ChunkerFactory) GetChunker 

func (f *ChunkerFactory) GetChunker(ext string) (Chunker, bool)

GetChunker returns a chunker for the specified language/extension

func (*ChunkerFactory) Register 

func (f *ChunkerFactory) Register(ext string, chunker Chunker) bool

Register adds a chunker for a specific language/extension. Returns true if the extension was newly registered, false if it overwrote an existing registration. Callers can use this to detect and handle registration conflicts.

func (*ChunkerFactory) SupportedExtensions 

func (f *ChunkerFactory) SupportedExtensions() []string

SupportedExtensions returns all registered extensions

type CohereConfig 

type CohereConfig struct {
	APIKey    string        // Cohere API key (required)
	Model     string        // Model name (default: embed-english-v3.0)
	BaseURL   string        // Base URL (default: https://api.cohere.com)
	InputType string        // Input type: search_document, search_query, classification, clustering
	Timeout   time.Duration // Request timeout
}

CohereConfig holds configuration for the Cohere embedding client

func (*CohereConfig) Validate 

func (c *CohereConfig) Validate() error

Validate checks if the config is valid

type CohereEmbedder 

type CohereEmbedder struct {
	// contains filtered or unexported fields
}

CohereEmbedder generates embeddings using the Cohere API

func NewCohereEmbedder 

func NewCohereEmbedder(cfg CohereConfig) (*CohereEmbedder, error)

NewCohereEmbedder creates a new Cohere embedding client

func NewCohereEmbedderFromEnv 

func NewCohereEmbedderFromEnv() (*CohereEmbedder, error)

NewCohereEmbedderFromEnv creates a CohereEmbedder from environment variables

func (*CohereEmbedder) Dimensions 

func (e *CohereEmbedder) Dimensions() int

Dimensions returns the embedding dimension (0 if unknown)

func (*CohereEmbedder) Embed 

func (e *CohereEmbedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed generates an embedding for a single text

func (*CohereEmbedder) EmbedBatch 

func (e *CohereEmbedder) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)

EmbedBatch generates embeddings for multiple texts

func (*CohereEmbedder) Model 

func (e *CohereEmbedder) Model() string

Model returns the model name

func (*CohereEmbedder) SetInputType 

func (e *CohereEmbedder) SetInputType(inputType string)

SetInputType allows changing the input type for different use cases Use "search_query" for queries, "search_document" for documents to be searched

type CompactEmbedding 

type CompactEmbedding struct {
	Data   []int8
	Scale  float32
	Offset float32
}

CompactEmbedding reduces precision for storage (optional memory savings) Converts float32 embeddings to int8 quantized values (-127 to 127) This reduces memory by 75% but loses some precision

func Quantize 

func Quantize(embedding []float32) *CompactEmbedding

Quantize converts a float32 embedding to a quantized form

func (*CompactEmbedding) Dequantize 

func (c *CompactEmbedding) Dequantize() []float32

Dequantize converts a quantized embedding back to float32

type Embedder 

type Embedder struct {
	// contains filtered or unexported fields
}

Embedder generates embeddings using an OpenAI-compatible API

func NewEmbedder 

func NewEmbedder(cfg EmbedderConfig) (*Embedder, error)

NewEmbedder creates a new Embedder with the given configuration

func (*Embedder) Dimensions 

func (e *Embedder) Dimensions() int

Dimensions returns the embedding dimension (0 if unknown)

func (*Embedder) Embed 

func (e *Embedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed generates an embedding for a single text

func (*Embedder) EmbedBatch 

func (e *Embedder) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)

EmbedBatch generates embeddings for multiple texts

func (*Embedder) Model 

func (e *Embedder) Model() string

Model returns the model name

type EmbedderConfig 

type EmbedderConfig struct {
	APIURL     string        // Base URL for embedding API (OpenAI-compatible)
	Model      string        // Model name (e.g., "nomic-embed-text", "mxbai-embed-large", "text-embedding-ada-002")
	APIKey     string        // API key (optional for local models)
	Timeout    time.Duration // Request timeout
	MaxRetries int           // Maximum retry attempts
}

EmbedderConfig holds configuration for the embedding client

func DefaultEmbedderConfig 

func DefaultEmbedderConfig() EmbedderConfig

DefaultEmbedderConfig returns sensible defaults

type EmbedderInterface 

type EmbedderInterface interface {
	Embed(ctx context.Context, text string) ([]float32, error)
	EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
	Dimensions() int
	Model() string
}

EmbedderInterface defines the interface for embedding generation (separate from the concrete Embedder to allow mocking)

type EnhancedResult added in v1.6.0 

type EnhancedResult struct {
	SearchResult            // Embedded base result
	MatchedQueries []string `json:"matched_queries"` // Which queries matched this result
	BoostedScore   float32  `json:"boosted_score"`   // Score after multi-match boosting
}

EnhancedResult extends SearchResult with multi-match tracking and boosting.

type ErrorType 

type ErrorType int

Error types for better error handling 

const (
	ErrTypeUnknown ErrorType = iota
	ErrTypeNotFound
	ErrTypeConnection
	ErrTypeInvalidInput
	ErrTypeConfiguration
)

type GenericChunker 

type GenericChunker struct {
	// contains filtered or unexported fields
}

GenericChunker implements a simple text-based chunker for unsupported file types

func NewGenericChunker 

func NewGenericChunker(maxChunkSize int) *GenericChunker

NewGenericChunker creates a new generic text chunker

func (*GenericChunker) Chunk 

func (c *GenericChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk splits content into chunks based on paragraphs/size

func (*GenericChunker) SupportedExtensions 

func (c *GenericChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles. Extensions NOT listed here: md, markdown, html, htm (handled by specialized chunkers)

type GoChunker 

type GoChunker struct{}

GoChunker implements the Chunker interface for Go source code

func NewGoChunker 

func NewGoChunker() *GoChunker

NewGoChunker creates a new Go AST-based chunker

func (*GoChunker) Chunk 

func (c *GoChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk parses Go source code and extracts semantic chunks

func (*GoChunker) SupportedExtensions 

func (c *GoChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type HTMLChunker added in v1.6.0 

type HTMLChunker struct {
	// contains filtered or unexported fields
}

HTMLChunker implements structure-aware chunking for HTML files. It chunks by semantic elements (section, article, main, aside) and falls back to div boundaries if no semantic elements are found.

func NewHTMLChunker added in v1.6.0 

func NewHTMLChunker(maxChunkSize int) *HTMLChunker

NewHTMLChunker creates a new HTML chunker. The maxChunkSize parameter is accepted for API consistency with other chunkers but is not currently used (HTML chunks by semantic structure, not size).

func (*HTMLChunker) Chunk added in v1.6.0 

func (c *HTMLChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk breaks HTML content into semantic chunks based on element boundaries.

func (*HTMLChunker) SupportedExtensions added in v1.6.0 

func (c *HTMLChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type HuggingFaceConfig 

type HuggingFaceConfig struct {
	APIKey       string        // HuggingFace API key (required)
	Model        string        // Model name (default: sentence-transformers/all-MiniLM-L6-v2)
	BaseURL      string        // Base URL (default: https://api-inference.huggingface.co)
	Timeout      time.Duration // Request timeout
	WaitForModel bool          // Wait for model to load if not ready
}

HuggingFaceConfig holds configuration for the HuggingFace embedding client

func (*HuggingFaceConfig) Validate 

func (c *HuggingFaceConfig) Validate() error

Validate checks if the config is valid

type HuggingFaceEmbedder 

type HuggingFaceEmbedder struct {
	// contains filtered or unexported fields
}

HuggingFaceEmbedder generates embeddings using the HuggingFace Inference API

func NewHuggingFaceEmbedder 

func NewHuggingFaceEmbedder(cfg HuggingFaceConfig) (*HuggingFaceEmbedder, error)

NewHuggingFaceEmbedder creates a new HuggingFace embedding client

func NewHuggingFaceEmbedderFromEnv 

func NewHuggingFaceEmbedderFromEnv() (*HuggingFaceEmbedder, error)

NewHuggingFaceEmbedderFromEnv creates a HuggingFaceEmbedder from environment variables

func (*HuggingFaceEmbedder) Dimensions 

func (e *HuggingFaceEmbedder) Dimensions() int

Dimensions returns the embedding dimension (0 if unknown)

func (*HuggingFaceEmbedder) Embed 

func (e *HuggingFaceEmbedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed generates an embedding for a single text

func (*HuggingFaceEmbedder) EmbedBatch 

func (e *HuggingFaceEmbedder) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)

EmbedBatch generates embeddings for multiple texts

func (*HuggingFaceEmbedder) Model 

func (e *HuggingFaceEmbedder) Model() string

Model returns the model name

type HybridSearchOptions added in v1.6.0 

type HybridSearchOptions struct {
	SearchOptions         // Embedded search options
	FusionK       int     // RRF k parameter (default: 60)
	FusionAlpha   float64 // Weighted fusion alpha (default: 0.7)
	UseWeighted   bool    // Use weighted fusion instead of RRF
}

HybridSearchOptions configures hybrid search behavior.

type IndexHealth 

type IndexHealth struct {
	Status        string     `json:"status"` // "healthy", "stale", "missing"
	Stats         IndexStats `json:"stats"`
	StaleFiles    int        `json:"stale_files"`
	NewFiles      int        `json:"new_files"`
	ModifiedFiles int        `json:"modified_files"`
}

IndexHealth represents the health status of the index

type IndexManager 

type IndexManager struct {
	// contains filtered or unexported fields
}

IndexManager handles indexing operations

func NewIndexManager 

func NewIndexManager(storage Storage, embedder EmbedderInterface, factory *ChunkerFactory) *IndexManager

NewIndexManager creates a new IndexManager

func (*IndexManager) Index 

func (m *IndexManager) Index(ctx context.Context, rootPath string, opts IndexOptions) (*IndexResult, error)

Index builds or rebuilds the semantic index for a directory

func (*IndexManager) Status 

func (m *IndexManager) Status(ctx context.Context) (*IndexStats, error)

Status returns index statistics

func (*IndexManager) Update 

func (m *IndexManager) Update(ctx context.Context, rootPath string, opts UpdateOptions) (*UpdateResult, error)

Update performs incremental index update

type IndexOptions 

type IndexOptions struct {
	Includes   []string         // Glob patterns to include (e.g., "*.go")
	Excludes   []string         // Directory/pattern names to exclude (e.g., "vendor", "node_modules")
	Force      bool             // Re-index all files even if unchanged
	OnProgress ProgressCallback // Optional callback for progress updates
}

IndexOptions configures indexing behavior

type IndexResult 

type IndexResult struct {
	FilesProcessed int      // Files that were indexed
	FilesSkipped   int      // Files skipped due to errors
	FilesUnchanged int      // Files skipped because already indexed and unchanged
	ChunksCreated  int      // Total chunks created
	Errors         []string // Error messages
}

IndexResult contains statistics from an indexing operation

type IndexStats 

type IndexStats struct {
	FilesIndexed   int    `json:"files_indexed"`
	ChunksTotal    int    `json:"chunks_total"`
	EmbeddingModel string `json:"embedding_model"`
	IndexSizeBytes int64  `json:"index_size_bytes"`
	LastUpdated    string `json:"last_updated"`
}

IndexStats holds statistics about the semantic index

type JSChunker 

type JSChunker struct {
	// contains filtered or unexported fields
}

JSChunker implements the Chunker interface for JavaScript/TypeScript

func NewJSChunker 

func NewJSChunker() *JSChunker

NewJSChunker creates a new JavaScript/TypeScript chunker

func (*JSChunker) Chunk 

func (c *JSChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk parses JavaScript/TypeScript source code and extracts semantic chunks

func (*JSChunker) SupportedExtensions 

func (c *JSChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type LexicalIndex added in v1.6.0 

type LexicalIndex struct {
	// contains filtered or unexported fields
}

LexicalIndex provides a standalone FTS5 full-text search index. It can be used independently or as a parallel index for vector storage backends.

func NewLexicalIndex added in v1.6.0 

func NewLexicalIndex(dbPath string, embeddingDim int) (*LexicalIndex, error)

NewLexicalIndex creates a new lexical index at the specified path. Use ":memory:" for an in-memory index (useful for testing). Parent directories are created if they don't exist.

func (*LexicalIndex) Clear added in v1.6.0 

func (idx *LexicalIndex) Clear(ctx context.Context) error

Clear removes all chunks from the index.

func (*LexicalIndex) Close added in v1.6.0 

func (idx *LexicalIndex) Close() error

Close closes the database connection.

func (*LexicalIndex) DeleteByFilePath added in v1.6.0 

func (idx *LexicalIndex) DeleteByFilePath(ctx context.Context, filePath string) (int, error)

DeleteByFilePath removes all chunks with the given file path. Returns the number of chunks deleted.

func (*LexicalIndex) DeleteChunk added in v1.6.0 

func (idx *LexicalIndex) DeleteChunk(ctx context.Context, id string) error

DeleteChunk removes a chunk from the index by ID.

func (*LexicalIndex) GetAllMemoryStats added in v1.6.0 

func (idx *LexicalIndex) GetAllMemoryStats() ([]RetrievalStats, error)

GetAllMemoryStats returns stats for all tracked memories.

func (*LexicalIndex) GetMemoryStats added in v1.6.0 

func (idx *LexicalIndex) GetMemoryStats(memoryID string) (*RetrievalStats, error)

GetMemoryStats returns stats for a specific memory entry.

func (*LexicalIndex) GetRetrievalHistory added in v1.6.0 

func (idx *LexicalIndex) GetRetrievalHistory(memoryID string, limit int) ([]RetrievalLogEntry, error)

GetRetrievalHistory returns recent retrieval log entries for a memory.

func (*LexicalIndex) IndexBatch added in v1.6.0 

func (idx *LexicalIndex) IndexBatch(ctx context.Context, chunks []Chunk) error

IndexBatch adds multiple chunks to the index in a single transaction.

func (*LexicalIndex) IndexChunk added in v1.6.0 

func (idx *LexicalIndex) IndexChunk(ctx context.Context, chunk Chunk) error

IndexChunk adds a single chunk to the index.

func (*LexicalIndex) PruneRetrievalLog added in v1.6.0 

func (idx *LexicalIndex) PruneRetrievalLog(olderThanDays int) (int64, error)

PruneRetrievalLog removes retrieval log entries older than the specified duration.

func (*LexicalIndex) Search added in v1.6.0 

func (idx *LexicalIndex) Search(ctx context.Context, query string, opts LexicalSearchOptions) ([]SearchResult, error)

Search performs a full-text search and returns matching chunks.

func (*LexicalIndex) TrackRetrieval added in v1.6.0 

func (idx *LexicalIndex) TrackRetrieval(memoryID string, query string, score float32) error

TrackRetrieval records a memory retrieval event. This should be called after search results are returned for profile=memory.

func (*LexicalIndex) TrackRetrievalBatch added in v1.6.0 

func (idx *LexicalIndex) TrackRetrievalBatch(retrievals []struct {
	MemoryID string
	Score    float32
}, query string) error

TrackRetrievalBatch records multiple memory retrieval events in a single transaction. More efficient than calling TrackRetrieval multiple times.

func (*LexicalIndex) UpdateMemoryStatus added in v1.6.0 

func (idx *LexicalIndex) UpdateMemoryStatus(memoryID string, status string) error

UpdateMemoryStatus updates the status of a memory entry (active, archived, etc.).

type LexicalSearchOptions added in v1.6.0 

type LexicalSearchOptions struct {
	TopK       int     // Maximum results to return (0 = unlimited/all results, default: 10)
	Type       string  // Filter by chunk type
	PathFilter string  // Filter by file path prefix
	Threshold  float64 // Minimum BM25 score (more negative = more relevant)
}

LexicalSearchOptions configures lexical search parameters. This is defined here to avoid circular imports with fts_sqlite.go.

type LexicalSearcher added in v1.6.0 

type LexicalSearcher interface {
	// LexicalSearch performs full-text search using FTS5.
	// Returns results ranked by BM25 relevance score.
	LexicalSearch(ctx context.Context, query string, opts LexicalSearchOptions) ([]SearchResult, error)
}

LexicalSearcher is an optional interface for storage backends that support full-text search. Backends implementing this interface can be used with hybrid search (dense + lexical fusion).

type ListOptions 

type ListOptions struct {
	Limit    int    // Maximum number of chunks to return (0 = no limit)
	Offset   int    // Number of chunks to skip
	FilePath string // Filter by file path
	Type     string // Filter by chunk type
	Language string // Filter by language
}

ListOptions configures how chunks are listed

type MarkdownChunker added in v1.6.0 

type MarkdownChunker struct {
	// contains filtered or unexported fields
}

MarkdownChunker implements structure-aware chunking for markdown files. It chunks by header boundaries, preserves code blocks, and maintains header hierarchy for meaningful chunk names.

func NewMarkdownChunker added in v1.6.0 

func NewMarkdownChunker(maxChunkSize int) *MarkdownChunker

NewMarkdownChunker creates a new markdown chunker with the specified max chunk size. If maxChunkSize is 0 or negative, defaults to 4000.

func (*MarkdownChunker) Chunk added in v1.6.0 

func (c *MarkdownChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk breaks markdown content into semantic chunks based on header boundaries. It preserves code blocks intact and tracks header hierarchy for chunk naming.

func (*MarkdownChunker) SupportedExtensions added in v1.6.0 

func (c *MarkdownChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type MemoryEntry added in v1.6.0 

type MemoryEntry struct {
	ID          string       `json:"id"`
	Question    string       `json:"question"`
	Answer      string       `json:"answer"`
	Tags        []string     `json:"tags,omitempty"`
	Source      string       `json:"source,omitempty"`
	Status      MemoryStatus `json:"status"`
	Occurrences int          `json:"occurrences"`
	CreatedAt   string       `json:"created_at"`
	UpdatedAt   string       `json:"updated_at"`
}

MemoryEntry represents a stored Q&A memory for semantic search

func NewMemoryEntry added in v1.6.0 

func NewMemoryEntry(question, answer string) *MemoryEntry

NewMemoryEntry creates a new MemoryEntry with default values

func (*MemoryEntry) EmbeddingText added in v1.6.0 

func (m *MemoryEntry) EmbeddingText() string

EmbeddingText returns the formatted text used for generating embeddings Uses structured format: "Question: {question}\nAnswer: {answer}"

func (*MemoryEntry) GenerateID added in v1.6.0 

func (m *MemoryEntry) GenerateID() string

GenerateID creates a deterministic ID for the memory entry based on its content

type MemoryListOptions added in v1.6.0 

type MemoryListOptions struct {
	Limit  int          // Maximum number of entries to return (0 = no limit)
	Offset int          // Number of entries to skip
	Tags   []string     // Filter by tags (any match)
	Source string       // Filter by source
	Status MemoryStatus // Filter by status
}

MemoryListOptions configures how memory entries are listed

type MemoryRetrieval added in v1.6.0 

type MemoryRetrieval struct {
	MemoryID string
	Score    float32
}

MemoryRetrieval represents a single retrieval event for batch tracking.

type MemorySearchOptions added in v1.6.0 

type MemorySearchOptions struct {
	TopK      int          // Maximum results to return (0 = unlimited/all results, default: 10)
	Threshold float32      // Minimum similarity score (0.0-1.0)
	Tags      []string     // Filter by tags (any match)
	Source    string       // Filter by source
	Status    MemoryStatus // Filter by status
}

MemorySearchOptions configures memory search behavior

type MemorySearchResult added in v1.6.0 

type MemorySearchResult struct {
	Entry     MemoryEntry `json:"entry"`
	Score     float32     `json:"score"`
	Embedding []float32   `json:"-"` // Not included in JSON output
}

MemorySearchResult pairs a memory entry with its similarity score

func (MemorySearchResult) MinimalJSON added in v1.6.0 

func (mr MemorySearchResult) MinimalJSON() string

MinimalJSON returns a compact JSON representation for --min output

type MemoryStats 

type MemoryStats struct {
	ChunksInMemory   int
	EmbeddingBytes   int64
	EstimatedTotalMB float64
}

MemoryStats holds memory usage statistics

func EstimateMemory 

func EstimateMemory(chunks int, embeddingDim int) MemoryStats

EstimateMemory estimates memory usage for a set of chunks

type MemoryStatsTracker added in v1.6.0 

type MemoryStatsTracker interface {
	// TrackMemoryRetrieval records a single memory retrieval event.
	TrackMemoryRetrieval(ctx context.Context, memoryID string, query string, score float32) error

	// TrackMemoryRetrievalBatch records multiple memory retrieval events in a single transaction.
	TrackMemoryRetrievalBatch(ctx context.Context, retrievals []MemoryRetrieval, query string) error

	// GetMemoryStats returns stats for a specific memory entry.
	GetMemoryStats(ctx context.Context, memoryID string) (*RetrievalStats, error)

	// GetAllMemoryStats returns stats for all tracked memories.
	GetAllMemoryStats(ctx context.Context) ([]RetrievalStats, error)

	// GetMemoryRetrievalHistory returns recent retrieval log entries for a memory.
	GetMemoryRetrievalHistory(ctx context.Context, memoryID string, limit int) ([]RetrievalLogEntry, error)

	// PruneMemoryRetrievalLog removes retrieval log entries older than the specified duration.
	PruneMemoryRetrievalLog(ctx context.Context, olderThanDays int) (int64, error)

	// UpdateMemoryStatsStatus updates the status of a memory entry.
	UpdateMemoryStatsStatus(ctx context.Context, memoryID string, status string) error
}

MemoryStatsTracker is an optional interface for tracking memory retrieval statistics. Storage implementations that support memory stats should implement this interface.

type MemoryStatus added in v1.6.0 

type MemoryStatus string

MemoryStatus represents the status of a memory entry 

const (
	// MemoryStatusPending is the default status for new entries
	MemoryStatusPending MemoryStatus = "pending"
	// MemoryStatusPromoted indicates the entry has been graduated to CLAUDE.md
	MemoryStatusPromoted MemoryStatus = "promoted"
)

type MemoryWithEmbedding added in v1.6.0 

type MemoryWithEmbedding struct {
	Entry     MemoryEntry
	Embedding []float32
}

MemoryWithEmbedding pairs a memory entry with its embedding for batch operations

type MultiProfileSearcher added in v1.6.0 

type MultiProfileSearcher struct {
	// contains filtered or unexported fields
}

MultiProfileSearcher orchestrates search across multiple storage profiles (collections). It executes queries in parallel across configured profiles and merges results.

Thread Safety: The storageMap is treated as immutable after construction and must not be modified after NewMultiProfileSearcher returns. Only defaultProfile is protected by mutex and can be changed via SetDefaultProfile.

func NewMultiProfileSearcher added in v1.6.0 

func NewMultiProfileSearcher(embedder EmbedderInterface, storageMap map[string]Storage) *MultiProfileSearcher

NewMultiProfileSearcher creates a new MultiProfileSearcher with the given storage mapping. storageMap maps profile names (e.g., "code", "docs") to their Storage instances.

IMPORTANT: The storageMap is stored by reference and must not be modified after this call. The caller should not modify the map after passing it to this function.

func (*MultiProfileSearcher) GetDefaultProfile added in v1.6.0 

func (mps *MultiProfileSearcher) GetDefaultProfile() string

GetDefaultProfile returns the current default profile.

func (*MultiProfileSearcher) Multisearch added in v1.6.0 

func (mps *MultiProfileSearcher) Multisearch(ctx context.Context, opts MultisearchOptions) (*MultisearchResult, error)

Multisearch performs batch search across multiple queries and profiles. Uses the underlying Multisearch implementation with multi-profile support.

func (*MultiProfileSearcher) Search added in v1.6.0 

func (mps *MultiProfileSearcher) Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)

Search performs semantic search across specified profiles in parallel. If opts.Profiles is nil or empty, the default profile is used. Results are deduplicated by Chunk.ID (keeping highest score), merged, and sorted by score.

func (*MultiProfileSearcher) SetDefaultProfile added in v1.6.0 

func (mps *MultiProfileSearcher) SetDefaultProfile(profile string)

SetDefaultProfile sets the profile to use when Profiles is nil or empty.

type MultisearchOptions added in v1.6.0 

type MultisearchOptions struct {
	Queries         []string     `json:"queries"`                     // 1-10 search queries to execute
	TopK            int          `json:"top_k,omitempty"`             // Maximum total results to return (0 = unlimited)
	Threshold       float32      `json:"threshold,omitempty"`         // Minimum similarity score (0.0 - 1.0)
	Profiles        []string     `json:"profiles,omitempty"`          // Profiles to search across (nil = default)
	BoostMultiMatch *bool        `json:"boost_multi_match,omitempty"` // Boost scores for multi-match results (default: true)
	Output          OutputFormat `json:"output,omitempty"`            // Output format: blended (default), by_query, by_collection
}

MultisearchOptions configures batch multisearch behavior

func (*MultisearchOptions) IsBoostEnabled added in v1.6.0 

func (o *MultisearchOptions) IsBoostEnabled() bool

IsBoostEnabled returns whether multi-match boosting is enabled. Defaults to true if not explicitly set.

func (*MultisearchOptions) Validate added in v1.6.0 

func (o *MultisearchOptions) Validate() error

Validate validates MultisearchOptions and returns an error if invalid

type MultisearchResult added in v1.6.0 

type MultisearchResult struct {
	// Results is the deduplicated, sorted results with boosting (used for blended output)
	Results []EnhancedResult `json:"results,omitempty"`
	// ByQuery groups results by which query they matched (used for by_query output)
	ByQuery map[string][]EnhancedResult `json:"by_query,omitempty"`
	// ByCollection groups results by which collection/profile they came from (used for by_collection output)
	ByCollection map[string][]EnhancedResult `json:"by_collection,omitempty"`
	// Format indicates which output format was used
	Format         OutputFormat   `json:"format,omitempty"`
	TotalQueries   int            `json:"total_queries"`   // Number of queries executed
	TotalResults   int            `json:"total_results"`   // Number of results after deduplication
	QueriesMatched map[string]int `json:"queries_matched"` // How many results each query contributed
}

MultisearchResult contains the results of a batch multisearch operation

func (*MultisearchResult) FormatAs added in v1.6.0 

func (r *MultisearchResult) FormatAs(format OutputFormat) *MultisearchResult

FormatAs formats the result according to the specified output format.

func (*MultisearchResult) FormatBlended added in v1.6.0 

func (r *MultisearchResult) FormatBlended() *MultisearchResult

FormatBlended returns the result in blended format (already the default). This is primarily for consistency when explicitly requesting blended output.

func (*MultisearchResult) FormatByCollection added in v1.6.0 

func (r *MultisearchResult) FormatByCollection() *MultisearchResult

FormatByCollection converts a blended MultisearchResult to by_collection format. Results are grouped by their source collection/profile (using Chunk.Domain). Results are sorted by score within each group.

func (*MultisearchResult) FormatByQuery added in v1.6.0 

func (r *MultisearchResult) FormatByQuery() *MultisearchResult

FormatByQuery converts a blended MultisearchResult to by_query format. Results are grouped by which queries they matched, and sorted by score within each group. A result that matches multiple queries will appear in multiple groups.

type OfflineEmbedder 

type OfflineEmbedder struct {
	// contains filtered or unexported fields
}

OfflineEmbedder wraps an embedder with offline mode support

func NewOfflineEmbedder 

func NewOfflineEmbedder(embedder EmbedderInterface, dimensions int) *OfflineEmbedder

NewOfflineEmbedder creates an embedder with offline fallback support

func (*OfflineEmbedder) Dimensions 

func (o *OfflineEmbedder) Dimensions() int

Dimensions returns the embedding dimensions

func (*OfflineEmbedder) Embed 

func (o *OfflineEmbedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed tries to use the embedder, falls back to keyword embedding if unavailable

func (*OfflineEmbedder) EmbedBatch 

func (o *OfflineEmbedder) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)

EmbedBatch tries to use the embedder batch, falls back if unavailable

func (*OfflineEmbedder) IsOffline 

func (o *OfflineEmbedder) IsOffline() bool

IsOffline returns true if operating in offline mode

type OfflineMode 

type OfflineMode int

OfflineMode represents the current offline mode state 

const (
	// OnlineMode indicates the embedder is working normally
	OnlineMode OfflineMode = iota
	// OfflineFallback indicates we're using keyword fallback
	OfflineFallback
)

type OfflineSearcher 

type OfflineSearcher struct {
	// contains filtered or unexported fields
}

OfflineSearcher provides search capabilities with offline fallback

func NewOfflineSearcher 

func NewOfflineSearcher(storage Storage, embedder *OfflineEmbedder) *OfflineSearcher

NewOfflineSearcher creates a searcher with offline support

func (*OfflineSearcher) IsOffline 

func (s *OfflineSearcher) IsOffline() bool

IsOffline returns true if the searcher is operating in offline mode

func (*OfflineSearcher) Search 

func (s *OfflineSearcher) Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)

Search performs semantic or keyword-based search

type OpenRouterConfig 

type OpenRouterConfig struct {
	APIKey     string        // OpenRouter API key (required)
	Model      string        // Model name (default: mistralai/codestral-embed-2505)
	BaseURL    string        // Base URL (default: https://openrouter.ai)
	Timeout    time.Duration // Request timeout
	MaxRetries int           // Maximum retry attempts for rate limiting
}

OpenRouterConfig holds configuration for the OpenRouter embedding client

func (*OpenRouterConfig) Validate 

func (c *OpenRouterConfig) Validate() error

Validate checks if the config is valid

type OpenRouterEmbedder 

type OpenRouterEmbedder struct {
	// contains filtered or unexported fields
}

OpenRouterEmbedder generates embeddings using the OpenRouter API

func NewOpenRouterEmbedder 

func NewOpenRouterEmbedder(cfg OpenRouterConfig) (*OpenRouterEmbedder, error)

NewOpenRouterEmbedder creates a new OpenRouter embedding client

func NewOpenRouterEmbedderFromEnv 

func NewOpenRouterEmbedderFromEnv() (*OpenRouterEmbedder, error)

NewOpenRouterEmbedderFromEnv creates an OpenRouterEmbedder from environment variables

func (*OpenRouterEmbedder) Dimensions 

func (e *OpenRouterEmbedder) Dimensions() int

Dimensions returns the embedding dimension (0 if unknown)

func (*OpenRouterEmbedder) Embed 

func (e *OpenRouterEmbedder) Embed(ctx context.Context, text string) ([]float32, error)

Embed generates an embedding for a single text

func (*OpenRouterEmbedder) EmbedBatch 

func (e *OpenRouterEmbedder) EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)

EmbedBatch generates embeddings for multiple texts

func (*OpenRouterEmbedder) Model 

func (e *OpenRouterEmbedder) Model() string

Model returns the model name

type OutputFormat added in v1.6.0 

type OutputFormat string

OutputFormat specifies how multisearch results should be organized 

const (
	// OutputBlended returns all results in a flat array sorted by score (default)
	OutputBlended OutputFormat = "blended"
	// OutputByQuery groups results by which query they matched
	OutputByQuery OutputFormat = "by_query"
	// OutputByCollection groups results by which collection/profile they came from
	OutputByCollection OutputFormat = "by_collection"
)

type PHPChunker 

type PHPChunker struct {
	// contains filtered or unexported fields
}

PHPChunker implements the Chunker interface for PHP

func NewPHPChunker 

func NewPHPChunker() *PHPChunker

NewPHPChunker creates a new PHP chunker

func (*PHPChunker) Chunk 

func (c *PHPChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk parses PHP source code and extracts semantic chunks

func (*PHPChunker) SupportedExtensions 

func (c *PHPChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type ProgressCallback added in v1.6.0 

type ProgressCallback func(event ProgressEvent)

ProgressCallback is called during indexing to report progress

type ProgressEvent added in v1.6.0 

type ProgressEvent struct {
	Current     int    // Current file number (1-based)
	Total       int    // Total files to process
	FilePath    string // Current file being processed
	ChunksTotal int    // Total chunks created so far
	Skipped     bool   // True if this file was skipped (already indexed)
}

ProgressEvent represents a progress update during indexing

type PythonChunker 

type PythonChunker struct {
	// contains filtered or unexported fields
}

PythonChunker implements the Chunker interface for Python

func NewPythonChunker 

func NewPythonChunker() *PythonChunker

NewPythonChunker creates a new Python chunker

func (*PythonChunker) Chunk 

func (c *PythonChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk parses Python source code and extracts semantic chunks

func (*PythonChunker) SupportedExtensions 

func (c *PythonChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type QdrantConfig 

type QdrantConfig struct {
	APIKey         string
	URL            string // Full URL like https://abc123.qdrant.io:6334
	CollectionName string
	EmbeddingDim   int
	FTSDataDir     string // Directory for parallel FTS database (should be project's .index/)
	InMemoryFTS    bool   // Use in-memory FTS (for testing)
}

QdrantConfig holds configuration for Qdrant storage

func (*QdrantConfig) Validate 

func (c *QdrantConfig) Validate() error

Validate checks if the config is valid

type QdrantStorage 

type QdrantStorage struct {
	// contains filtered or unexported fields
}

QdrantStorage implements the Storage interface using Qdrant cloud

func NewQdrantStorage 

func NewQdrantStorage(config QdrantConfig) (*QdrantStorage, error)

NewQdrantStorage creates a new Qdrant-based storage

func NewQdrantStorageFromEnv 

func NewQdrantStorageFromEnv(embeddingDim int) (*QdrantStorage, error)

NewQdrantStorageFromEnv creates a QdrantStorage from environment variables Env vars: QDRANT_API_KEY, QDRANT_API_URL, QDRANT_COLLECTION (optional, default: llm_semantic)

func (*QdrantStorage) Clear added in v1.6.0 

func (s *QdrantStorage) Clear(ctx context.Context) error

Clear removes all points from the Qdrant collection (for force re-index)

func (*QdrantStorage) Close 

func (s *QdrantStorage) Close() error

func (*QdrantStorage) Create 

func (s *QdrantStorage) Create(ctx context.Context, chunk Chunk, embedding []float32) error

func (*QdrantStorage) CreateBatch added in v1.6.0 

func (s *QdrantStorage) CreateBatch(ctx context.Context, chunks []ChunkWithEmbedding) error

func (*QdrantStorage) Delete 

func (s *QdrantStorage) Delete(ctx context.Context, id string) error

func (*QdrantStorage) DeleteByFilePath 

func (s *QdrantStorage) DeleteByFilePath(ctx context.Context, filePath string) (int, error)

func (*QdrantStorage) DeleteCollection 

func (s *QdrantStorage) DeleteCollection() error

DeleteCollection removes the collection (used for testing cleanup)

func (*QdrantStorage) DeleteMemory added in v1.6.0 

func (s *QdrantStorage) DeleteMemory(ctx context.Context, id string) error

DeleteMemory removes a memory entry by ID

func (*QdrantStorage) GetCalibrationMetadata added in v1.6.0 

func (s *QdrantStorage) GetCalibrationMetadata(ctx context.Context) (*CalibrationMetadata, error)

GetCalibrationMetadata retrieves stored calibration data. Returns (nil, nil) if no calibration has been performed yet.

func (*QdrantStorage) GetFileHash added in v1.6.0 

func (s *QdrantStorage) GetFileHash(ctx context.Context, filePath string) (string, error)

GetFileHash retrieves the stored content hash for a file path

func (*QdrantStorage) GetMemory added in v1.6.0 

func (s *QdrantStorage) GetMemory(ctx context.Context, id string) (*MemoryEntry, error)

GetMemory retrieves a memory entry by ID

func (*QdrantStorage) LexicalSearch added in v1.6.0 

func (s *QdrantStorage) LexicalSearch(ctx context.Context, query string, opts LexicalSearchOptions) ([]SearchResult, error)

LexicalSearch performs a full-text search using the parallel FTS5 index. Returns results ranked by BM25 relevance score.

func (*QdrantStorage) List 

func (s *QdrantStorage) List(ctx context.Context, opts ListOptions) ([]Chunk, error)

func (*QdrantStorage) ListMemory added in v1.6.0 

func (s *QdrantStorage) ListMemory(ctx context.Context, opts MemoryListOptions) ([]MemoryEntry, error)

ListMemory retrieves memory entries based on filter options

func (*QdrantStorage) Read 

func (s *QdrantStorage) Read(ctx context.Context, id string) (*Chunk, error)

func (*QdrantStorage) Search 

func (s *QdrantStorage) Search(ctx context.Context, queryEmbedding []float32, opts SearchOptions) ([]SearchResult, error)

func (*QdrantStorage) SearchMemory added in v1.6.0 

func (s *QdrantStorage) SearchMemory(ctx context.Context, queryEmbedding []float32, opts MemorySearchOptions) ([]MemorySearchResult, error)

SearchMemory finds memory entries similar to the query embedding

func (*QdrantStorage) SetCalibrationMetadata added in v1.6.0 

func (s *QdrantStorage) SetCalibrationMetadata(ctx context.Context, meta *CalibrationMetadata) error

SetCalibrationMetadata stores calibration data. Overwrites any existing calibration data.

func (*QdrantStorage) SetFileHash added in v1.6.0 

func (s *QdrantStorage) SetFileHash(ctx context.Context, filePath string, hash string) error

SetFileHash stores the content hash for a file path

func (*QdrantStorage) Stats 

func (s *QdrantStorage) Stats(ctx context.Context) (*IndexStats, error)

func (*QdrantStorage) StoreMemory added in v1.6.0 

func (s *QdrantStorage) StoreMemory(ctx context.Context, entry MemoryEntry, embedding []float32) error

StoreMemory stores a memory entry with its embedding

func (*QdrantStorage) StoreMemoryBatch added in v1.6.0 

func (s *QdrantStorage) StoreMemoryBatch(ctx context.Context, entries []MemoryWithEmbedding) error

StoreMemoryBatch stores multiple memory entries with their embeddings

func (*QdrantStorage) Update 

func (s *QdrantStorage) Update(ctx context.Context, chunk Chunk, embedding []float32) error

type RecencyConfig added in v1.6.0 

type RecencyConfig struct {
	// Factor controls the maximum boost magnitude.
	// When age=0, boost = 1 + Factor.
	// Default: 0.5 (max 50% boost for newest files)
	Factor float64

	// HalfLifeDays controls how quickly the boost decays.
	// After HalfLifeDays, the boost is reduced by ~63% (1-1/e).
	// Default: 7.0 (one week)
	HalfLifeDays float64
}

RecencyConfig configures recency boost calculation. The recency boost formula is: boost = 1 + factor * exp(-age_days / half_life_days) This gives a multiplicative boost to more recently modified files.

func DefaultRecencyConfig added in v1.6.0 

func DefaultRecencyConfig() RecencyConfig

DefaultRecencyConfig returns the default recency configuration. Factor: 0.5 (max 50% boost), HalfLife: 7 days.

type RetrievalLogEntry added in v1.6.0 

type RetrievalLogEntry struct {
	ID        int64   `json:"id"`
	MemoryID  string  `json:"memory_id"`
	Query     string  `json:"query"`
	Score     float32 `json:"score"`
	Timestamp string  `json:"timestamp"`
}

RetrievalLogEntry represents a single retrieval event log.

type RetrievalStats added in v1.6.0 

type RetrievalStats struct {
	MemoryID       string  `json:"memory_id"`
	RetrievalCount int     `json:"retrieval_count"`
	LastRetrieved  string  `json:"last_retrieved,omitempty"`
	Status         string  `json:"status"`
	AvgScore       float32 `json:"avg_score,omitempty"`
}

RetrievalStats holds retrieval statistics for a memory entry.

type RustChunker added in v1.6.0 

type RustChunker struct {
	// contains filtered or unexported fields
}

RustChunker implements the Chunker interface for Rust

func NewRustChunker added in v1.6.0 

func NewRustChunker() *RustChunker

NewRustChunker creates a new Rust chunker

func (*RustChunker) Chunk added in v1.6.0 

func (c *RustChunker) Chunk(path string, content []byte) ([]Chunk, error)

Chunk parses Rust source code and extracts semantic chunks

func (*RustChunker) SupportedExtensions added in v1.6.0 

func (c *RustChunker) SupportedExtensions() []string

SupportedExtensions returns the file extensions this chunker handles

type SQLiteStorage 

type SQLiteStorage struct {
	// contains filtered or unexported fields
}

SQLiteStorage implements the Storage interface using SQLite

func NewSQLiteStorage 

func NewSQLiteStorage(dbPath string, embeddingDim int) (*SQLiteStorage, error)

NewSQLiteStorage creates a new SQLite-based storage

func (*SQLiteStorage) Clear added in v1.6.0 

func (s *SQLiteStorage) Clear(ctx context.Context) error

Clear removes all chunks from storage (for force re-index)

func (*SQLiteStorage) Close 

func (s *SQLiteStorage) Close() error

func (*SQLiteStorage) Create 

func (s *SQLiteStorage) Create(ctx context.Context, chunk Chunk, embedding []float32) error

func (*SQLiteStorage) CreateBatch added in v1.6.0 

func (s *SQLiteStorage) CreateBatch(ctx context.Context, chunks []ChunkWithEmbedding) error

func (*SQLiteStorage) Delete 

func (s *SQLiteStorage) Delete(ctx context.Context, id string) error

func (*SQLiteStorage) DeleteByFilePath 

func (s *SQLiteStorage) DeleteByFilePath(ctx context.Context, filePath string) (int, error)

func (*SQLiteStorage) DeleteMemory added in v1.6.0 

func (s *SQLiteStorage) DeleteMemory(ctx context.Context, id string) error

DeleteMemory removes a memory entry by ID

func (*SQLiteStorage) FTSIntegrityCheck added in v1.6.0 

func (s *SQLiteStorage) FTSIntegrityCheck() error

FTSIntegrityCheck verifies the FTS5 index integrity. Returns an error if the index is corrupted.

func (*SQLiteStorage) GetAllMemoryStats added in v1.6.0 

func (s *SQLiteStorage) GetAllMemoryStats(ctx context.Context) ([]RetrievalStats, error)

GetAllMemoryStats returns stats for all tracked memories.

func (*SQLiteStorage) GetCalibrationMetadata added in v1.6.0 

func (s *SQLiteStorage) GetCalibrationMetadata(ctx context.Context) (*CalibrationMetadata, error)

GetCalibrationMetadata retrieves stored calibration data. Returns (nil, nil) if no calibration has been performed yet.

func (*SQLiteStorage) GetFileHash added in v1.6.0 

func (s *SQLiteStorage) GetFileHash(ctx context.Context, filePath string) (string, error)

GetFileHash retrieves the stored content hash for a file path

func (*SQLiteStorage) GetMemory added in v1.6.0 

func (s *SQLiteStorage) GetMemory(ctx context.Context, id string) (*MemoryEntry, error)

GetMemory retrieves a memory entry by ID

func (*SQLiteStorage) GetMemoryRetrievalHistory added in v1.6.0 

func (s *SQLiteStorage) GetMemoryRetrievalHistory(ctx context.Context, memoryID string, limit int) ([]RetrievalLogEntry, error)

GetMemoryRetrievalHistory returns recent retrieval log entries for a memory.

func (*SQLiteStorage) GetMemoryStats added in v1.6.0 

func (s *SQLiteStorage) GetMemoryStats(ctx context.Context, memoryID string) (*RetrievalStats, error)

GetMemoryStats returns stats for a specific memory entry.

func (*SQLiteStorage) LexicalSearch added in v1.6.0 

func (s *SQLiteStorage) LexicalSearch(ctx context.Context, query string, opts LexicalSearchOptions) ([]SearchResult, error)

LexicalSearch performs a full-text search using FTS5. Returns results ranked by BM25 relevance score. The score is converted from BM25 (where more negative is better) to a positive scale where higher is better for consistency with vector search.

func (*SQLiteStorage) List 

func (s *SQLiteStorage) List(ctx context.Context, opts ListOptions) ([]Chunk, error)

func (*SQLiteStorage) ListMemory added in v1.6.0 

func (s *SQLiteStorage) ListMemory(ctx context.Context, opts MemoryListOptions) ([]MemoryEntry, error)

ListMemory retrieves memory entries based on filter options

func (*SQLiteStorage) PruneMemoryRetrievalLog added in v1.6.0 

func (s *SQLiteStorage) PruneMemoryRetrievalLog(ctx context.Context, olderThanDays int) (int64, error)

PruneMemoryRetrievalLog removes retrieval log entries older than the specified duration.

func (*SQLiteStorage) Read 

func (s *SQLiteStorage) Read(ctx context.Context, id string) (*Chunk, error)

func (*SQLiteStorage) RebuildFTSIndex added in v1.6.0 

func (s *SQLiteStorage) RebuildFTSIndex() error

RebuildFTSIndex rebuilds the FTS5 index from scratch. Use this if the FTS index becomes corrupted or out of sync.

func (*SQLiteStorage) SafeLexicalSearch added in v1.6.0 

func (s *SQLiteStorage) SafeLexicalSearch(ctx context.Context, query string, opts LexicalSearchOptions) ([]SearchResult, error)

SafeLexicalSearch performs a lexical search with automatic query escaping. Use this when the query comes from untrusted user input. For advanced users who want FTS5 operators, use LexicalSearch directly.

func (*SQLiteStorage) Search 

func (s *SQLiteStorage) Search(ctx context.Context, queryEmbedding []float32, opts SearchOptions) ([]SearchResult, error)

func (*SQLiteStorage) SearchMemory added in v1.6.0 

func (s *SQLiteStorage) SearchMemory(ctx context.Context, queryEmbedding []float32, opts MemorySearchOptions) ([]MemorySearchResult, error)

SearchMemory finds memory entries similar to the query embedding

func (*SQLiteStorage) SetCalibrationMetadata added in v1.6.0 

func (s *SQLiteStorage) SetCalibrationMetadata(ctx context.Context, meta *CalibrationMetadata) error

SetCalibrationMetadata stores calibration data. Overwrites any existing calibration data.

func (*SQLiteStorage) SetFileHash added in v1.6.0 

func (s *SQLiteStorage) SetFileHash(ctx context.Context, filePath string, hash string) error

SetFileHash stores the content hash for a file path

func (*SQLiteStorage) Stats 

func (s *SQLiteStorage) Stats(ctx context.Context) (*IndexStats, error)

func (*SQLiteStorage) StoreMemory added in v1.6.0 

func (s *SQLiteStorage) StoreMemory(ctx context.Context, entry MemoryEntry, embedding []float32) error

StoreMemory stores a memory entry with its embedding

func (*SQLiteStorage) StoreMemoryBatch added in v1.6.0 

func (s *SQLiteStorage) StoreMemoryBatch(ctx context.Context, entries []MemoryWithEmbedding) error

StoreMemoryBatch stores multiple memory entries with their embeddings

func (*SQLiteStorage) TrackMemoryRetrieval added in v1.6.0 

func (s *SQLiteStorage) TrackMemoryRetrieval(ctx context.Context, memoryID string, query string, score float32) error

TrackMemoryRetrieval records a memory retrieval event. This should be called after search results are returned for memory search.

func (*SQLiteStorage) TrackMemoryRetrievalBatch added in v1.6.0 

func (s *SQLiteStorage) TrackMemoryRetrievalBatch(ctx context.Context, retrievals []MemoryRetrieval, query string) error

TrackMemoryRetrievalBatch records multiple memory retrieval events in a single transaction.

func (*SQLiteStorage) Update 

func (s *SQLiteStorage) Update(ctx context.Context, chunk Chunk, embedding []float32) error

func (*SQLiteStorage) UpdateMemoryStatsStatus added in v1.6.0 

func (s *SQLiteStorage) UpdateMemoryStatsStatus(ctx context.Context, memoryID string, status string) error

UpdateMemoryStatsStatus updates the status of a memory entry (active, archived, etc.).

type SearchOptions 

type SearchOptions struct {
	TopK       int     `json:"top_k,omitempty"`       // Maximum number of results to return (0 = unlimited/all results)
	Threshold  float32 `json:"threshold,omitempty"`   // Minimum similarity score (0.0 - 1.0)
	Type       string  `json:"type,omitempty"`        // Filter by chunk type
	PathFilter string  `json:"path_filter,omitempty"` // Filter by file path pattern (glob)
	// Profiles specifies which profiles (collections) to search across.
	// NOTE: This field is NOT used by Storage.Search directly. It is used by
	// higher-level APIs (Searcher, MultiProfileSearcher) to coordinate multi-profile
	// search and is included here to allow unified option passing through the stack.
	Profiles []string `json:"profiles,omitempty"`
}

SearchOptions configures how vector search is performed

type SearchResult 

type SearchResult struct {
	Chunk     Chunk     `json:"chunk"`
	Score     float32   `json:"score"`
	Relevance string    `json:"relevance,omitempty"` // "high", "medium", or "low"
	Preview   string    `json:"preview,omitempty"`   // Short preview of chunk content
	Embedding []float32 `json:"-"`                   // Not included in JSON output
}

SearchResult represents a chunk with its similarity score

func ApplyRecencyBoost added in v1.6.0 

func ApplyRecencyBoost(results []SearchResult, mtimes map[string]time.Time, cfg RecencyConfig, now time.Time) []SearchResult

ApplyRecencyBoost applies recency boost to search results. Results are modified in place and returned (for chaining). Files without mtime entries receive neutral boost (1.0).

func FuseRRF added in v1.6.0 

func FuseRRF(denseResults, lexicalResults []SearchResult, k int) []SearchResult

FuseRRF combines dense and lexical search results using Reciprocal Rank Fusion. The formula is: score = sum(1 / (k + rank)) for each result list. Parameter k controls the smoothing - larger k reduces the impact of rank differences. Default k=60 is commonly used in literature. Returns results sorted by fused score in descending order.

func FuseRRFWithError added in v1.6.0 

func FuseRRFWithError(denseResults, lexicalResults []SearchResult, k int) ([]SearchResult, error)

FuseRRFWithError is like FuseRRF but returns an error for invalid parameters.

func FuseRRFWithTopK added in v1.6.0 

func FuseRRFWithTopK(denseResults, lexicalResults []SearchResult, k int, topK int) []SearchResult

FuseRRFWithTopK combines results using RRF and limits to top K results. If topK is 0 or negative, all results are returned.

func FuseWeighted added in v1.6.0 

func FuseWeighted(denseResults, lexicalResults []SearchResult, alpha float64) ([]SearchResult, error)

FuseWeighted combines dense and lexical results using weighted score fusion. The formula is: score = alpha * dense_score + (1-alpha) * lexical_score Alpha controls the balance: alpha=1.0 means 100% dense, alpha=0.0 means 100% lexical. Note: Scores are combined directly without normalization. For best results, ensure both score sources use comparable scales.

func (SearchResult) MinimalJSON 

func (sr SearchResult) MinimalJSON() string

MinimalJSON returns a compact JSON representation for --min output

type Searcher 

type Searcher struct {
	// contains filtered or unexported fields
}

Searcher orchestrates semantic search across the index

func NewSearcher 

func NewSearcher(storage Storage, embedder EmbedderInterface) *Searcher

NewSearcher creates a new Searcher with the given storage and embedder

func (*Searcher) HybridSearch added in v1.6.0 

func (s *Searcher) HybridSearch(ctx context.Context, query string, opts HybridSearchOptions) ([]SearchResult, error)

HybridSearch performs combined dense (vector) and lexical (FTS5) search. Results are fused using Reciprocal Rank Fusion (RRF) for improved recall. Returns an error if the storage doesn't support lexical search.

func (*Searcher) Multisearch added in v1.6.0 

func (s *Searcher) Multisearch(ctx context.Context, opts MultisearchOptions) (*MultisearchResult, error)

Multisearch performs batch search across multiple queries with deduplication and boosting. Results are deduplicated by Chunk.ID, keeping the highest score for each chunk. Multi-match boosting adds 0.05 to the score for each additional query match. Final results are sorted by boosted score descending.

func (*Searcher) Search 

func (s *Searcher) Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)

Search performs semantic search using the query text

func (*Searcher) SearchMultiple 

func (s *Searcher) SearchMultiple(ctx context.Context, queries []string, opts SearchOptions) ([]SearchResult, error)

SearchMultiple performs search with multiple query terms and combines results

type SemanticError 

type SemanticError struct {
	Type    ErrorType
	Message string
	Cause   error
	Hint    string
}

SemanticError provides structured error information

func ErrChunkingFailure 

func ErrChunkingFailure(filePath string, cause error) *SemanticError

ErrChunkingFailure creates an error for code chunking failures

func ErrConnectionFailed 

func ErrConnectionFailed(cause error) *SemanticError

ErrConnectionFailed creates an error for API connection failures

func ErrEmbeddingFailure 

func ErrEmbeddingFailure(cause error) *SemanticError

ErrEmbeddingFailure creates an error for embedding generation failures

func ErrIndexNotFound 

func ErrIndexNotFound() *SemanticError

ErrIndexNotFound creates an error for when the index doesn't exist

func ErrInvalidPath 

func ErrInvalidPath(path string, cause error) *SemanticError

ErrInvalidPath creates an error for invalid file paths

func ErrInvalidQuery 

func ErrInvalidQuery(query string) *SemanticError

ErrInvalidQuery creates an error for invalid search queries

func ErrStorageFailure 

func ErrStorageFailure(operation string, cause error) *SemanticError

ErrStorageFailure creates an error for storage-related issues

func (*SemanticError) Error 

func (e *SemanticError) Error() string

func (*SemanticError) FormatWithHint 

func (e *SemanticError) FormatWithHint() string

FormatWithHint returns the error message with hint if available

func (*SemanticError) Unwrap 

func (e *SemanticError) Unwrap() error

func (*SemanticError) WithHint 

func (e *SemanticError) WithHint(hint string) *SemanticError

WithHint adds a helpful hint to the error

type Storage 

type Storage interface {
	// Create stores a new chunk with its embedding
	Create(ctx context.Context, chunk Chunk, embedding []float32) error

	// CreateBatch stores multiple chunks with their embeddings in a single operation
	// This is more efficient than multiple Create calls, especially for network-based storage
	CreateBatch(ctx context.Context, chunks []ChunkWithEmbedding) error

	// Read retrieves a chunk by its ID
	Read(ctx context.Context, id string) (*Chunk, error)

	// Update replaces an existing chunk and its embedding
	Update(ctx context.Context, chunk Chunk, embedding []float32) error

	// Delete removes a chunk by its ID
	Delete(ctx context.Context, id string) error

	// DeleteByFilePath removes all chunks for a given file path
	DeleteByFilePath(ctx context.Context, filePath string) (int, error)

	// List retrieves chunks based on filter options
	List(ctx context.Context, opts ListOptions) ([]Chunk, error)

	// Search finds chunks similar to the query embedding
	Search(ctx context.Context, queryEmbedding []float32, opts SearchOptions) ([]SearchResult, error)

	// Stats returns statistics about the stored index
	Stats(ctx context.Context) (*IndexStats, error)

	// Clear removes all chunks from storage (for force re-index)
	Clear(ctx context.Context) error

	// GetFileHash retrieves the stored content hash for a file path
	// Returns empty string if file is not indexed
	GetFileHash(ctx context.Context, filePath string) (string, error)

	// SetFileHash stores the content hash for a file path
	SetFileHash(ctx context.Context, filePath string, hash string) error

	// Close releases any resources held by the storage
	Close() error

	// StoreMemory stores a memory entry with its embedding
	StoreMemory(ctx context.Context, entry MemoryEntry, embedding []float32) error

	// StoreMemoryBatch stores multiple memory entries with their embeddings in a single operation
	StoreMemoryBatch(ctx context.Context, entries []MemoryWithEmbedding) error

	// SearchMemory finds memory entries similar to the query embedding
	SearchMemory(ctx context.Context, queryEmbedding []float32, opts MemorySearchOptions) ([]MemorySearchResult, error)

	// GetMemory retrieves a memory entry by ID
	GetMemory(ctx context.Context, id string) (*MemoryEntry, error)

	// DeleteMemory removes a memory entry by ID
	DeleteMemory(ctx context.Context, id string) error

	// ListMemory retrieves memory entries based on filter options
	ListMemory(ctx context.Context, opts MemoryListOptions) ([]MemoryEntry, error)

	// GetCalibrationMetadata retrieves stored calibration data.
	// Returns (nil, nil) if no calibration has been performed yet.
	// Returns (nil, error) on storage errors.
	GetCalibrationMetadata(ctx context.Context) (*CalibrationMetadata, error)

	// SetCalibrationMetadata stores calibration data.
	// Overwrites any existing calibration data.
	SetCalibrationMetadata(ctx context.Context, meta *CalibrationMetadata) error
}

Storage defines the interface for persisting and querying chunks with embeddings

type StreamingChunkProcessor 

type StreamingChunkProcessor struct {
	// contains filtered or unexported fields
}

StreamingChunkProcessor processes chunks without loading all into memory

func NewStreamingChunkProcessor 

func NewStreamingChunkProcessor(batchSize int, processor func([]Chunk) error) *StreamingChunkProcessor

NewStreamingChunkProcessor creates a new streaming processor

func (*StreamingChunkProcessor) Process 

func (p *StreamingChunkProcessor) Process(chunks <-chan Chunk) error

Process processes chunks in batches

type UpdateOptions 

type UpdateOptions struct {
	Includes []string // Glob patterns to include
	Excludes []string // Patterns to exclude
}

UpdateOptions configures incremental update behavior

type UpdateResult 

type UpdateResult struct {
	FilesUpdated  int
	FilesRemoved  int
	ChunksCreated int
	ChunksRemoved int
}

UpdateResult contains statistics from an update operation

Source Files

View all Source files

Directories

Path Synopsis
Package config provides configuration file support for llm-semantic commands.
 Package config provides configuration file support for llm-semantic commands.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.