Documentation
¶
Overview ¶
Package kb provides a knowledge base system for storing and searching documents.
Documents are chunked and embedded for hybrid search combining vector similarity and full-text search using Reciprocal Rank Fusion (RRF).
Index ¶
- func ExtractTagsFromMetadata(meta map[string]interface{}) []string
- func InjectTagsIntoMetadata(meta map[string]interface{}, tags []string) map[string]interface{}
- func SerializeFrontMatter(fm FrontMatter, body string) string
- func StripFrontMatter(raw string) string
- type Document
- type DocumentRef
- type FrontMatter
- type KBStore
- func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
- func (s *KBStore) AddDocumentWithEmbeddings(ctx context.Context, filePath, content string, metadata map[string]interface{}, ...) error
- func (s *KBStore) ConfigureFilesystemMirror(dirPath string) error
- func (s *KBStore) CountDocuments(ctx context.Context) (int64, error)
- func (s *KBStore) DeleteDocument(ctx context.Context, filePath string) error
- func (s *KBStore) DeleteDocumentFromFilesystem(filePath string) error
- func (s *KBStore) FilesystemMirrorPath() string
- func (s *KBStore) GetDocument(ctx context.Context, filePath string) (*Document, error)
- func (s *KBStore) GetExpiredDocuments(ctx context.Context) ([]DocumentRef, error)
- func (s *KBStore) GetMemoriesForInjection(ctx context.Context, query string, limit int, maxChars int, ...) ([]MemoryResult, error)
- func (s *KBStore) GetMemoryByKey(ctx context.Context, key string) (*Document, error)
- func (s *KBStore) IncrementMemoryHits(ctx context.Context, docID int64, defaultTTLDays int) error
- func (s *KBStore) ListDocuments(ctx context.Context, limit, offset int) ([]Document, error)
- func (s *KBStore) MarkDocumentOutdated(ctx context.Context, filePath string) error
- func (s *KBStore) RebuildFTS(ctx context.Context) error
- func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)
- func (s *KBStore) SearchDocumentsWithOptions(ctx context.Context, query string, limit int, opts SearchOptions) ([]SearchResult, error)
- func (s *KBStore) SetSyncWorkers(workers int)
- func (s *KBStore) SetWriteProxy(proxy *dbproxy.DBProxy)
- func (s *KBStore) SyncDirectory(ctx context.Context, dirPath string) error
- func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)
- func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
- func (s *KBStore) UpsertMemory(ctx context.Context, opts MemoryUpsertOptions) (created bool, err error)
- func (s *KBStore) WatchDirectory(ctx context.Context, dirPath string) error
- func (s *KBStore) WriteDocumentToFilesystem(filePath, content string) error
- type MemoryGCService
- type MemoryOptions
- type MemoryResult
- type MemoryUpsertOptions
- type SearchOptions
- type SearchResult
- type SyncStats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ExtractTagsFromMetadata ¶ added in v0.413.0
ExtractTagsFromMetadata reads the "tags" key from a metadata map and returns them as a string slice. Returns nil if no tags are present.
func InjectTagsIntoMetadata ¶ added in v0.413.0
InjectTagsIntoMetadata ensures tags are stored in the metadata map under "tags". If tags is nil/empty and no existing tags, the key is left absent.
func SerializeFrontMatter ¶ added in v0.413.0
func SerializeFrontMatter(fm FrontMatter, body string) string
SerializeFrontMatter produces a document string with YAML front matter prepended to the body content.
func StripFrontMatter ¶ added in v0.413.0
StripFrontMatter removes any YAML front matter from raw content and returns only the body. This is a convenience wrapper around ParseFrontMatter.
Types ¶
type Document ¶
type Document struct {
ID int64
FilePath string
Content string
Metadata map[string]interface{}
Tags []string
CreatedAt time.Time
UpdatedAt time.Time
// Memory system fields (populated only when reading memory documents)
MemoryKey string
MemoryScope string
Importance float64
Hits int
Source string
Outdated bool
ExpiresAt *time.Time
}
Document represents a stored document in the knowledge base.
type DocumentRef ¶ added in v0.416.3
DocumentRef is a minimal reference to a KB document (ID + path).
type FrontMatter ¶ added in v0.413.0
type FrontMatter struct {
CreatedAt time.Time `yaml:"created_at,omitempty"`
UpdatedAt time.Time `yaml:"updated_at,omitempty"`
Tags []string `yaml:"tags,omitempty"`
// Memory fields — only serialized when non-zero/non-empty
Key string `yaml:"key,omitempty"`
Scope string `yaml:"scope,omitempty"`
Outdated bool `yaml:"outdated,omitempty"`
ExpiresAt *time.Time `yaml:"expires_at,omitempty"`
Hits int `yaml:"hits,omitempty"`
Importance float64 `yaml:"importance,omitempty"`
Source string `yaml:"source,omitempty"`
}
FrontMatter holds the YAML front matter fields for KB documents.
func MergeFrontMatter ¶ added in v0.413.0
func MergeFrontMatter(existing, incoming FrontMatter) FrontMatter
MergeFrontMatter combines existing and incoming front matter. It preserves created_at from existing, sets updated_at to now, and replaces tags with new if provided (otherwise keeps existing).
func NewFrontMatter ¶ added in v0.413.0
func NewFrontMatter(tags []string, opts *MemoryOptions) FrontMatter
NewFrontMatter creates a fresh FrontMatter with created_at/updated_at set to now. When opts is non-nil, memory fields are populated from it.
func ParseFrontMatter ¶ added in v0.413.0
func ParseFrontMatter(raw string) (FrontMatter, string, error)
ParseFrontMatter splits YAML front matter from the body of a document. If no valid front matter is found, it returns a zero FrontMatter and the original content as body.
type KBStore ¶
type KBStore struct {
// contains filtered or unexported fields
}
KBStore manages knowledge base documents with chunking, embeddings, and hybrid search. It uses SQLite for storage with FTS5 for full-text search and in-memory vector search.
func NewKBStore ¶
NewKBStore creates a new KBStore instance.
Parameters:
- db: SQLite database connection
- embedder: Embedder for generating chunk embeddings
- chunkSize: Maximum chunk size in characters (0 = use default)
- chunkOverlap: Overlap between consecutive chunks (0 = use default)
func (*KBStore) AddDocument ¶
func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
AddDocument adds a new document to the knowledge base. It chunks the content, generates embeddings, and updates the FTS index.
func (*KBStore) AddDocumentWithEmbeddings ¶ added in v0.326.0
func (s *KBStore) AddDocumentWithEmbeddings(ctx context.Context, filePath, content string, metadata map[string]interface{}, chunks []string, embeddings [][]float32) error
AddDocumentWithEmbeddings inserts a document using pre-computed chunks and embeddings. Called by the primary IPC dispatcher when a secondary forwards a KBAddDocument write. No embedding generation is performed; the provided values are stored directly.
func (*KBStore) ConfigureFilesystemMirror ¶ added in v0.160.0
ConfigureFilesystemMirror sets a base directory where KB documents are mirrored as markdown files on add/update/delete operations.
func (*KBStore) CountDocuments ¶
CountDocuments returns the total number of documents.
func (*KBStore) DeleteDocument ¶
DeleteDocument removes a document and all its chunks from the knowledge base.
func (*KBStore) DeleteDocumentFromFilesystem ¶ added in v0.160.0
DeleteDocumentFromFilesystem removes a mirrored KB document from disk. If no mirror path is configured, this is a no-op.
func (*KBStore) FilesystemMirrorPath ¶ added in v0.160.0
FilesystemMirrorPath returns the configured filesystem mirror path.
func (*KBStore) GetDocument ¶
GetDocument retrieves a document by file path.
func (*KBStore) GetExpiredDocuments ¶ added in v0.416.3
func (s *KBStore) GetExpiredDocuments(ctx context.Context) ([]DocumentRef, error)
GetExpiredDocuments returns all non-outdated documents whose expires_at is in the past.
func (*KBStore) GetMemoriesForInjection ¶ added in v0.416.3
func (s *KBStore) GetMemoriesForInjection(ctx context.Context, query string, limit int, maxChars int, pinnedScopes []string) ([]MemoryResult, error)
GetMemoriesForInjection retrieves memory documents ranked for context injection. It combines semantic search results with pinned-scope documents, deduplicates, scores, and trims to fit within the maxChars budget.
func (*KBStore) GetMemoryByKey ¶ added in v0.416.3
GetMemoryByKey retrieves a document by its memory_key. Returns nil, nil when no document with the key exists.
func (*KBStore) IncrementMemoryHits ¶ added in v0.416.3
IncrementMemoryHits increments the hit counter for a document and extends its TTL.
func (*KBStore) ListDocuments ¶
ListDocuments returns paginated documents.
func (*KBStore) MarkDocumentOutdated ¶ added in v0.416.3
MarkDocumentOutdated marks a document as outdated and updates its filesystem mirror.
func (*KBStore) RebuildFTS ¶
RebuildFTS rebuilds the FTS5 index from kb_chunks.
func (*KBStore) SearchDocuments ¶
func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)
SearchDocuments performs hybrid search combining vector similarity and FTS. Results are fused using Reciprocal Rank Fusion (RRF).
func (*KBStore) SearchDocumentsWithOptions ¶ added in v0.413.0
func (s *KBStore) SearchDocumentsWithOptions(ctx context.Context, query string, limit int, opts SearchOptions) ([]SearchResult, error)
SearchDocumentsWithOptions performs hybrid search with optional tag filtering and chronological ordering. When opts.Tags is non-empty, results are filtered to documents whose tags fuzzy-match any of the requested tags. When opts.SortByDate is true, results are sorted by updated_at descending.
func (*KBStore) SetSyncWorkers ¶ added in v0.160.0
SetSyncWorkers configures how many concurrent workers are used for KB filesystem import preprocessing (file reads and in-memory preparation). SQLite writes remain serialized through a single writer goroutine.
func (*KBStore) SetWriteProxy ¶ added in v0.326.0
SetWriteProxy configures a DB proxy for mutating operations.
func (*KBStore) SyncDirectory ¶
SyncDirectory imports or syncs all .md files from a directory. Existing documents are updated, new files are added.
func (*KBStore) SyncDirectoryWithStats ¶ added in v0.160.0
func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)
SyncDirectoryWithStats imports or syncs all markdown files from a directory. It recursively scans for .md files, upserts modified documents, and optionally deletes KB documents that no longer exist on disk for that directory source.
func (*KBStore) UpdateDocument ¶
func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
UpdateDocument updates an existing document's content and metadata. It re-chunks and re-embeds the content.
func (*KBStore) UpsertMemory ¶ added in v0.416.3
func (s *KBStore) UpsertMemory(ctx context.Context, opts MemoryUpsertOptions) (created bool, err error)
UpsertMemory stores or updates a memory document, keyed by opts.Key when provided. Returns created=true when a new document was inserted, false when an existing one was updated.
func (*KBStore) WatchDirectory ¶ added in v0.160.0
WatchDirectory monitors a KB source directory recursively and keeps KB documents in sync with .md file changes in near real-time.
func (*KBStore) WriteDocumentToFilesystem ¶ added in v0.160.0
WriteDocumentToFilesystem writes a KB document to the configured mirror path. If no mirror path is configured, this is a no-op.
type MemoryGCService ¶ added in v0.416.3
type MemoryGCService struct {
// contains filtered or unexported fields
}
MemoryGCService periodically marks expired memory documents as outdated.
func NewMemoryGCService ¶ added in v0.416.3
func NewMemoryGCService(store *KBStore, interval time.Duration) *MemoryGCService
NewMemoryGCService creates a new MemoryGCService.
func (*MemoryGCService) Start ¶ added in v0.416.3
func (g *MemoryGCService) Start(ctx context.Context)
Start launches the GC loop in a background goroutine.
func (*MemoryGCService) Stop ¶ added in v0.416.3
func (g *MemoryGCService) Stop()
Stop signals the background goroutine to exit.
type MemoryOptions ¶ added in v0.416.3
type MemoryOptions struct {
Key string
Scope string
Source string
Importance float64
TTLDays int // 0 = no expiry
}
MemoryOptions carries optional memory-layer settings for NewFrontMatter.
type MemoryResult ¶ added in v0.416.3
MemoryResult wraps a Document with its chunk content and relevance score.
type MemoryUpsertOptions ¶ added in v0.416.3
type MemoryUpsertOptions struct {
FilePath string
Key string
Scope string
Content string
Tags []string
Metadata map[string]interface{}
Importance float64
Source string
DefaultTTLDays int // 0 → 180
}
MemoryUpsertOptions configures a memory document upsert operation.
type SearchOptions ¶ added in v0.413.0
type SearchOptions struct {
Tags []string // filter results by tags (fuzzy match)
SortByDate bool // sort results by updated_at descending
ExcludeOutdated bool // exclude documents where outdated = 1
Scope string // filter by memory_scope prefix (empty = all)
}
SearchOptions provides optional filtering/sorting for search queries.
type SearchResult ¶
SearchResult represents a ranked search result from the knowledge base.