Documentation
¶
Overview ¶
Package kb provides a knowledge base system for storing and searching documents.
Documents are chunked and embedded for hybrid search combining vector similarity and full-text search using Reciprocal Rank Fusion (RRF).
Index ¶
- func ExtractTagsFromMetadata(meta map[string]interface{}) []string
- func InjectTagsIntoMetadata(meta map[string]interface{}, tags []string) map[string]interface{}
- func SerializeFrontMatter(fm FrontMatter, body string) string
- func StripFrontMatter(raw string) string
- type Document
- type DocumentConverter
- type DocumentRef
- type FrontMatter
- type KBStore
- func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
- func (s *KBStore) AddDocumentWithEmbeddings(ctx context.Context, filePath, content string, metadata map[string]interface{}, ...) error
- func (s *KBStore) ConfigureFilesystemMirror(dirPath string) error
- func (s *KBStore) CountDocuments(ctx context.Context) (int64, error)
- func (s *KBStore) DeleteDocument(ctx context.Context, filePath string) error
- func (s *KBStore) DeleteDocumentFromFilesystem(filePath string) error
- func (s *KBStore) FilesystemMirrorPath() string
- func (s *KBStore) GetDocument(ctx context.Context, filePath string) (*Document, error)
- func (s *KBStore) GetExpiredDocuments(ctx context.Context) ([]DocumentRef, error)
- func (s *KBStore) GetMemoriesForInjection(ctx context.Context, query string, limit int, maxChars int, ...) ([]MemoryResult, error)
- func (s *KBStore) GetMemoryByKey(ctx context.Context, key string) (*Document, error)
- func (s *KBStore) IncrementMemoryHits(ctx context.Context, docID int64, defaultTTLDays int) error
- func (s *KBStore) ListDocuments(ctx context.Context, limit, offset int) ([]Document, error)
- func (s *KBStore) MarkDocumentOutdated(ctx context.Context, filePath string) error
- func (s *KBStore) RebuildFTS(ctx context.Context) error
- func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)
- func (s *KBStore) SearchDocumentsWithOptions(ctx context.Context, query string, limit int, opts SearchOptions) ([]SearchResult, error)
- func (s *KBStore) SetDocumentConverter(c DocumentConverter)
- func (s *KBStore) SetSyncWorkers(workers int)
- func (s *KBStore) SetWriteProxy(proxy *dbproxy.DBProxy)
- func (s *KBStore) SyncDirectory(ctx context.Context, dirPath string) error
- func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)
- func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
- func (s *KBStore) UpsertMemory(ctx context.Context, opts MemoryUpsertOptions) (created bool, err error)
- func (s *KBStore) WatchDirectory(ctx context.Context, dirPath string) error
- func (s *KBStore) WriteDocumentToFilesystem(filePath, content string) error
- type MemoryGCService
- type MemoryOptions
- type MemoryResult
- type MemoryUpsertOptions
- type SearchOptions
- type SearchResult
- type SyncStats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ExtractTagsFromMetadata ¶ added in v0.413.0
ExtractTagsFromMetadata reads the "tags" key from a metadata map and returns them as a string slice. Returns nil if no tags are present.
func InjectTagsIntoMetadata ¶ added in v0.413.0
InjectTagsIntoMetadata ensures tags are stored in the metadata map under "tags". If tags is nil/empty and no existing tags, the key is left absent.
func SerializeFrontMatter ¶ added in v0.413.0
func SerializeFrontMatter(fm FrontMatter, body string) string
SerializeFrontMatter produces a document string with YAML front matter prepended to the body content.
func StripFrontMatter ¶ added in v0.413.0
StripFrontMatter removes any YAML front matter from raw content and returns only the body. This is a convenience wrapper around ParseFrontMatter.
Types ¶
type Document ¶
type Document struct {
ID int64
FilePath string
Content string
Metadata map[string]interface{}
Tags []string
CreatedAt time.Time
UpdatedAt time.Time
// Memory system fields (populated only when reading memory documents)
MemoryKey string
MemoryScope string
Importance float64
Hits int
Source string
Outdated bool
ExpiresAt *time.Time
}
Document represents a stored document in the knowledge base.
type DocumentConverter ¶ added in v0.605.1
type DocumentConverter interface {
// ConvertFile converts the file at path to Markdown text.
ConvertFile(path string) (string, error)
// IsConvertibleDocument reports whether path is a document format that
// should be auto-converted during ingestion (curated subset, excludes
// plain markdown/text which is indexed verbatim).
IsConvertibleDocument(path string) bool
}
DocumentConverter converts rich document formats (docx, pdf, xlsx, …) to Markdown for on-the-fly KB ingestion. It is implemented by internal/convert.Converter and injected via SetDocumentConverter to avoid a hard dependency from the kb package on the conversion library.
type DocumentRef ¶ added in v0.416.3
DocumentRef is a minimal reference to a KB document (ID + path).
type FrontMatter ¶ added in v0.413.0
type FrontMatter struct {
CreatedAt time.Time `yaml:"created_at,omitempty"`
UpdatedAt time.Time `yaml:"updated_at,omitempty"`
Tags []string `yaml:"tags,omitempty"`
// Memory fields — only serialized when non-zero/non-empty
Key string `yaml:"key,omitempty"`
Scope string `yaml:"scope,omitempty"`
Outdated bool `yaml:"outdated,omitempty"`
ExpiresAt *time.Time `yaml:"expires_at,omitempty"`
Hits int `yaml:"hits,omitempty"`
Importance float64 `yaml:"importance,omitempty"`
Source string `yaml:"source,omitempty"`
}
FrontMatter holds the YAML front matter fields for KB documents.
func MergeFrontMatter ¶ added in v0.413.0
func MergeFrontMatter(existing, incoming FrontMatter) FrontMatter
MergeFrontMatter combines existing and incoming front matter. It preserves created_at from existing, sets updated_at to now, and replaces tags with new if provided (otherwise keeps existing).
func NewFrontMatter ¶ added in v0.413.0
func NewFrontMatter(tags []string, opts *MemoryOptions) FrontMatter
NewFrontMatter creates a fresh FrontMatter with created_at/updated_at set to now. When opts is non-nil, memory fields are populated from it.
func ParseFrontMatter ¶ added in v0.413.0
func ParseFrontMatter(raw string) (FrontMatter, string, error)
ParseFrontMatter splits YAML front matter from the body of a document. If no valid front matter is found, it returns a zero FrontMatter and the original content as body.
type KBStore ¶
type KBStore struct {
// contains filtered or unexported fields
}
KBStore manages knowledge base documents with chunking, embeddings, and hybrid search. It uses SQLite for storage with FTS5 for full-text search and in-memory vector search.
func NewKBStore ¶
NewKBStore creates a new KBStore instance.
Parameters:
- db: SQLite database connection
- embedder: Embedder for generating chunk embeddings
- chunkSize: Maximum chunk size in characters (0 = use default)
- chunkOverlap: Overlap between consecutive chunks (0 = use default)
func (*KBStore) AddDocument ¶
func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
AddDocument adds a new document to the knowledge base. It chunks the content, generates embeddings, and updates the FTS index.
func (*KBStore) AddDocumentWithEmbeddings ¶ added in v0.326.0
func (s *KBStore) AddDocumentWithEmbeddings(ctx context.Context, filePath, content string, metadata map[string]interface{}, chunks []string, embeddings [][]float32) error
AddDocumentWithEmbeddings inserts a document using pre-computed chunks and embeddings. Called by the primary IPC dispatcher when a secondary forwards a KBAddDocument write. No embedding generation is performed; the provided values are stored directly.
func (*KBStore) ConfigureFilesystemMirror ¶ added in v0.160.0
ConfigureFilesystemMirror sets a base directory where KB documents are mirrored as markdown files on add/update/delete operations.
func (*KBStore) CountDocuments ¶
CountDocuments returns the total number of documents.
func (*KBStore) DeleteDocument ¶
DeleteDocument removes a document and all its chunks from the knowledge base.
func (*KBStore) DeleteDocumentFromFilesystem ¶ added in v0.160.0
DeleteDocumentFromFilesystem removes a mirrored KB document from disk. If no mirror path is configured, this is a no-op.
func (*KBStore) FilesystemMirrorPath ¶ added in v0.160.0
FilesystemMirrorPath returns the configured filesystem mirror path.
func (*KBStore) GetDocument ¶
GetDocument retrieves a document by file path.
func (*KBStore) GetExpiredDocuments ¶ added in v0.416.3
func (s *KBStore) GetExpiredDocuments(ctx context.Context) ([]DocumentRef, error)
GetExpiredDocuments returns all non-outdated documents whose expires_at is in the past.
func (*KBStore) GetMemoriesForInjection ¶ added in v0.416.3
func (s *KBStore) GetMemoriesForInjection(ctx context.Context, query string, limit int, maxChars int, pinnedScopes []string) ([]MemoryResult, error)
GetMemoriesForInjection retrieves memory documents ranked for context injection. It combines semantic search results with pinned-scope documents, deduplicates, scores, and trims to fit within the maxChars budget.
func (*KBStore) GetMemoryByKey ¶ added in v0.416.3
GetMemoryByKey retrieves a document by its memory_key. Returns nil, nil when no document with the key exists.
func (*KBStore) IncrementMemoryHits ¶ added in v0.416.3
IncrementMemoryHits increments the hit counter for a document and extends its TTL.
func (*KBStore) ListDocuments ¶
ListDocuments returns paginated documents.
func (*KBStore) MarkDocumentOutdated ¶ added in v0.416.3
MarkDocumentOutdated marks a document as outdated and updates its filesystem mirror.
func (*KBStore) RebuildFTS ¶
RebuildFTS rebuilds the FTS5 index from kb_chunks.
func (*KBStore) SearchDocuments ¶
func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)
SearchDocuments performs hybrid search combining vector similarity and FTS. Results are fused using Reciprocal Rank Fusion (RRF).
func (*KBStore) SearchDocumentsWithOptions ¶ added in v0.413.0
func (s *KBStore) SearchDocumentsWithOptions(ctx context.Context, query string, limit int, opts SearchOptions) ([]SearchResult, error)
SearchDocumentsWithOptions performs hybrid search with optional tag filtering and chronological ordering. When opts.Tags is non-empty, results are filtered to documents whose tags fuzzy-match any of the requested tags. When opts.SortByDate is true, results are sorted by updated_at descending.
func (*KBStore) SetDocumentConverter ¶ added in v0.605.1
func (s *KBStore) SetDocumentConverter(c DocumentConverter)
SetDocumentConverter installs a converter so that supported document files found in the KB source directory are converted to Markdown and indexed, referencing the original file. Passing nil disables conversion (the default), in which case only .md files are indexed.
func (*KBStore) SetSyncWorkers ¶ added in v0.160.0
SetSyncWorkers configures how many concurrent workers are used for KB filesystem import preprocessing (file reads and in-memory preparation). SQLite writes remain serialized through a single writer goroutine.
func (*KBStore) SetWriteProxy ¶ added in v0.326.0
SetWriteProxy configures a DB proxy for mutating operations.
func (*KBStore) SyncDirectory ¶
SyncDirectory imports or syncs all .md files from a directory. Existing documents are updated, new files are added.
func (*KBStore) SyncDirectoryWithStats ¶ added in v0.160.0
func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)
SyncDirectoryWithStats imports or syncs all markdown files from a directory. It recursively scans for .md files, upserts modified documents, and optionally deletes KB documents that no longer exist on disk for that directory source.
func (*KBStore) UpdateDocument ¶
func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
UpdateDocument updates an existing document's content and metadata. It re-chunks and re-embeds the content.
func (*KBStore) UpsertMemory ¶ added in v0.416.3
func (s *KBStore) UpsertMemory(ctx context.Context, opts MemoryUpsertOptions) (created bool, err error)
UpsertMemory stores or updates a memory document, keyed by opts.Key when provided. Returns created=true when a new document was inserted, false when an existing one was updated.
func (*KBStore) WatchDirectory ¶ added in v0.160.0
WatchDirectory monitors a KB source directory recursively and keeps KB documents in sync with .md file changes in near real-time.
func (*KBStore) WriteDocumentToFilesystem ¶ added in v0.160.0
WriteDocumentToFilesystem writes a KB document to the configured mirror path. If no mirror path is configured, this is a no-op.
type MemoryGCService ¶ added in v0.416.3
type MemoryGCService struct {
// contains filtered or unexported fields
}
MemoryGCService periodically marks expired memory documents as outdated.
func NewMemoryGCService ¶ added in v0.416.3
func NewMemoryGCService(store *KBStore, interval time.Duration) *MemoryGCService
NewMemoryGCService creates a new MemoryGCService.
func (*MemoryGCService) Start ¶ added in v0.416.3
func (g *MemoryGCService) Start(ctx context.Context)
Start launches the GC loop in a background goroutine.
func (*MemoryGCService) Stop ¶ added in v0.416.3
func (g *MemoryGCService) Stop()
Stop signals the background goroutine to exit.
type MemoryOptions ¶ added in v0.416.3
type MemoryOptions struct {
Key string
Scope string
Source string
Importance float64
TTLDays int // 0 = no expiry
}
MemoryOptions carries optional memory-layer settings for NewFrontMatter.
type MemoryResult ¶ added in v0.416.3
MemoryResult wraps a Document with its chunk content and relevance score.
type MemoryUpsertOptions ¶ added in v0.416.3
type MemoryUpsertOptions struct {
FilePath string
Key string
Scope string
Content string
Tags []string
Metadata map[string]interface{}
Importance float64
Source string
DefaultTTLDays int // 0 → 180
}
MemoryUpsertOptions configures a memory document upsert operation.
type SearchOptions ¶ added in v0.413.0
type SearchOptions struct {
Tags []string // filter results by tags (fuzzy match)
SortByDate bool // sort results by updated_at descending
ExcludeOutdated bool // exclude documents where outdated = 1
Scope string // filter by memory_scope prefix (empty = all)
}
SearchOptions provides optional filtering/sorting for search queries.
type SearchResult ¶
SearchResult represents a ranked search result from the knowledge base.