kb

package
v0.416.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 13, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package kb provides a knowledge base system for storing and searching documents.

Documents are chunked and embedded for hybrid search combining vector similarity and full-text search using Reciprocal Rank Fusion (RRF).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ExtractTagsFromMetadata added in v0.413.0

func ExtractTagsFromMetadata(meta map[string]interface{}) []string

ExtractTagsFromMetadata reads the "tags" key from a metadata map and returns them as a string slice. Returns nil if no tags are present.

func InjectTagsIntoMetadata added in v0.413.0

func InjectTagsIntoMetadata(meta map[string]interface{}, tags []string) map[string]interface{}

InjectTagsIntoMetadata ensures tags are stored in the metadata map under "tags". If tags is nil/empty and no existing tags, the key is left absent.

func SerializeFrontMatter added in v0.413.0

func SerializeFrontMatter(fm FrontMatter, body string) string

SerializeFrontMatter produces a document string with YAML front matter prepended to the body content.

func StripFrontMatter added in v0.413.0

func StripFrontMatter(raw string) string

StripFrontMatter removes any YAML front matter from raw content and returns only the body. This is a convenience wrapper around ParseFrontMatter.

Types

type Document

type Document struct {
	ID        int64
	FilePath  string
	Content   string
	Metadata  map[string]interface{}
	Tags      []string
	CreatedAt time.Time
	UpdatedAt time.Time
	// Memory system fields (populated only when reading memory documents)
	MemoryKey   string
	MemoryScope string
	Importance  float64
	Hits        int
	Source      string
	Outdated    bool
	ExpiresAt   *time.Time
}

Document represents a stored document in the knowledge base.

type DocumentRef added in v0.416.3

type DocumentRef struct {
	ID       int64
	FilePath string
}

DocumentRef is a minimal reference to a KB document (ID + path).

type FrontMatter added in v0.413.0

type FrontMatter struct {
	CreatedAt time.Time `yaml:"created_at,omitempty"`
	UpdatedAt time.Time `yaml:"updated_at,omitempty"`
	Tags      []string  `yaml:"tags,omitempty"`
	// Memory fields — only serialized when non-zero/non-empty
	Key        string     `yaml:"key,omitempty"`
	Scope      string     `yaml:"scope,omitempty"`
	Outdated   bool       `yaml:"outdated,omitempty"`
	ExpiresAt  *time.Time `yaml:"expires_at,omitempty"`
	Hits       int        `yaml:"hits,omitempty"`
	Importance float64    `yaml:"importance,omitempty"`
	Source     string     `yaml:"source,omitempty"`
}

FrontMatter holds the YAML front matter fields for KB documents.

func MergeFrontMatter added in v0.413.0

func MergeFrontMatter(existing, incoming FrontMatter) FrontMatter

MergeFrontMatter combines existing and incoming front matter. It preserves created_at from existing, sets updated_at to now, and replaces tags with new if provided (otherwise keeps existing).

func NewFrontMatter added in v0.413.0

func NewFrontMatter(tags []string, opts *MemoryOptions) FrontMatter

NewFrontMatter creates a fresh FrontMatter with created_at/updated_at set to now. When opts is non-nil, memory fields are populated from it.

func ParseFrontMatter added in v0.413.0

func ParseFrontMatter(raw string) (FrontMatter, string, error)

ParseFrontMatter splits YAML front matter from the body of a document. If no valid front matter is found, it returns a zero FrontMatter and the original content as body.

type KBStore

type KBStore struct {
	// contains filtered or unexported fields
}

KBStore manages knowledge base documents with chunking, embeddings, and hybrid search. It uses SQLite for storage with FTS5 for full-text search and in-memory vector search.

func NewKBStore

func NewKBStore(db *sql.DB, embedder embeddings.Embedder, chunkSize, chunkOverlap int) *KBStore

NewKBStore creates a new KBStore instance.

Parameters:

  • db: SQLite database connection
  • embedder: Embedder for generating chunk embeddings
  • chunkSize: Maximum chunk size in characters (0 = use default)
  • chunkOverlap: Overlap between consecutive chunks (0 = use default)

func (*KBStore) AddDocument

func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error

AddDocument adds a new document to the knowledge base. It chunks the content, generates embeddings, and updates the FTS index.

func (*KBStore) AddDocumentWithEmbeddings added in v0.326.0

func (s *KBStore) AddDocumentWithEmbeddings(ctx context.Context, filePath, content string, metadata map[string]interface{}, chunks []string, embeddings [][]float32) error

AddDocumentWithEmbeddings inserts a document using pre-computed chunks and embeddings. Called by the primary IPC dispatcher when a secondary forwards a KBAddDocument write. No embedding generation is performed; the provided values are stored directly.

func (*KBStore) ConfigureFilesystemMirror added in v0.160.0

func (s *KBStore) ConfigureFilesystemMirror(dirPath string) error

ConfigureFilesystemMirror sets a base directory where KB documents are mirrored as markdown files on add/update/delete operations.

func (*KBStore) CountDocuments

func (s *KBStore) CountDocuments(ctx context.Context) (int64, error)

CountDocuments returns the total number of documents.

func (*KBStore) DeleteDocument

func (s *KBStore) DeleteDocument(ctx context.Context, filePath string) error

DeleteDocument removes a document and all its chunks from the knowledge base.

func (*KBStore) DeleteDocumentFromFilesystem added in v0.160.0

func (s *KBStore) DeleteDocumentFromFilesystem(filePath string) error

DeleteDocumentFromFilesystem removes a mirrored KB document from disk. If no mirror path is configured, this is a no-op.

func (*KBStore) FilesystemMirrorPath added in v0.160.0

func (s *KBStore) FilesystemMirrorPath() string

FilesystemMirrorPath returns the configured filesystem mirror path.

func (*KBStore) GetDocument

func (s *KBStore) GetDocument(ctx context.Context, filePath string) (*Document, error)

GetDocument retrieves a document by file path.

func (*KBStore) GetExpiredDocuments added in v0.416.3

func (s *KBStore) GetExpiredDocuments(ctx context.Context) ([]DocumentRef, error)

GetExpiredDocuments returns all non-outdated documents whose expires_at is in the past.

func (*KBStore) GetMemoriesForInjection added in v0.416.3

func (s *KBStore) GetMemoriesForInjection(ctx context.Context, query string, limit int, maxChars int, pinnedScopes []string) ([]MemoryResult, error)

GetMemoriesForInjection retrieves memory documents ranked for context injection. It combines semantic search results with pinned-scope documents, deduplicates, scores, and trims to fit within the maxChars budget.

func (*KBStore) GetMemoryByKey added in v0.416.3

func (s *KBStore) GetMemoryByKey(ctx context.Context, key string) (*Document, error)

GetMemoryByKey retrieves a document by its memory_key. Returns nil, nil when no document with the key exists.

func (*KBStore) IncrementMemoryHits added in v0.416.3

func (s *KBStore) IncrementMemoryHits(ctx context.Context, docID int64, defaultTTLDays int) error

IncrementMemoryHits increments the hit counter for a document and extends its TTL.

func (*KBStore) ListDocuments

func (s *KBStore) ListDocuments(ctx context.Context, limit, offset int) ([]Document, error)

ListDocuments returns paginated documents.

func (*KBStore) MarkDocumentOutdated added in v0.416.3

func (s *KBStore) MarkDocumentOutdated(ctx context.Context, filePath string) error

MarkDocumentOutdated marks a document as outdated and updates its filesystem mirror.

func (*KBStore) RebuildFTS

func (s *KBStore) RebuildFTS(ctx context.Context) error

RebuildFTS rebuilds the FTS5 index from kb_chunks.

func (*KBStore) SearchDocuments

func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)

SearchDocuments performs hybrid search combining vector similarity and FTS. Results are fused using Reciprocal Rank Fusion (RRF).

func (*KBStore) SearchDocumentsWithOptions added in v0.413.0

func (s *KBStore) SearchDocumentsWithOptions(ctx context.Context, query string, limit int, opts SearchOptions) ([]SearchResult, error)

SearchDocumentsWithOptions performs hybrid search with optional tag filtering and chronological ordering. When opts.Tags is non-empty, results are filtered to documents whose tags fuzzy-match any of the requested tags. When opts.SortByDate is true, results are sorted by updated_at descending.

func (*KBStore) SetSyncWorkers added in v0.160.0

func (s *KBStore) SetSyncWorkers(workers int)

SetSyncWorkers configures how many concurrent workers are used for KB filesystem import preprocessing (file reads and in-memory preparation). SQLite writes remain serialized through a single writer goroutine.

func (*KBStore) SetWriteProxy added in v0.326.0

func (s *KBStore) SetWriteProxy(proxy *dbproxy.DBProxy)

SetWriteProxy configures a DB proxy for mutating operations.

func (*KBStore) SyncDirectory

func (s *KBStore) SyncDirectory(ctx context.Context, dirPath string) error

SyncDirectory imports or syncs all .md files from a directory. Existing documents are updated, new files are added.

func (*KBStore) SyncDirectoryWithStats added in v0.160.0

func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)

SyncDirectoryWithStats imports or syncs all markdown files from a directory. It recursively scans for .md files, upserts modified documents, and optionally deletes KB documents that no longer exist on disk for that directory source.

func (*KBStore) UpdateDocument

func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error

UpdateDocument updates an existing document's content and metadata. It re-chunks and re-embeds the content.

func (*KBStore) UpsertMemory added in v0.416.3

func (s *KBStore) UpsertMemory(ctx context.Context, opts MemoryUpsertOptions) (created bool, err error)

UpsertMemory stores or updates a memory document, keyed by opts.Key when provided. Returns created=true when a new document was inserted, false when an existing one was updated.

func (*KBStore) WatchDirectory added in v0.160.0

func (s *KBStore) WatchDirectory(ctx context.Context, dirPath string) error

WatchDirectory monitors a KB source directory recursively and keeps KB documents in sync with .md file changes in near real-time.

func (*KBStore) WriteDocumentToFilesystem added in v0.160.0

func (s *KBStore) WriteDocumentToFilesystem(filePath, content string) error

WriteDocumentToFilesystem writes a KB document to the configured mirror path. If no mirror path is configured, this is a no-op.

type MemoryGCService added in v0.416.3

type MemoryGCService struct {
	// contains filtered or unexported fields
}

MemoryGCService periodically marks expired memory documents as outdated.

func NewMemoryGCService added in v0.416.3

func NewMemoryGCService(store *KBStore, interval time.Duration) *MemoryGCService

NewMemoryGCService creates a new MemoryGCService.

func (*MemoryGCService) Start added in v0.416.3

func (g *MemoryGCService) Start(ctx context.Context)

Start launches the GC loop in a background goroutine.

func (*MemoryGCService) Stop added in v0.416.3

func (g *MemoryGCService) Stop()

Stop signals the background goroutine to exit.

type MemoryOptions added in v0.416.3

type MemoryOptions struct {
	Key        string
	Scope      string
	Source     string
	Importance float64
	TTLDays    int // 0 = no expiry
}

MemoryOptions carries optional memory-layer settings for NewFrontMatter.

type MemoryResult added in v0.416.3

type MemoryResult struct {
	Document     Document
	ChunkContent string
	Score        float64
}

MemoryResult wraps a Document with its chunk content and relevance score.

type MemoryUpsertOptions added in v0.416.3

type MemoryUpsertOptions struct {
	FilePath       string
	Key            string
	Scope          string
	Content        string
	Tags           []string
	Metadata       map[string]interface{}
	Importance     float64
	Source         string
	DefaultTTLDays int // 0 → 180
}

MemoryUpsertOptions configures a memory document upsert operation.

type SearchOptions added in v0.413.0

type SearchOptions struct {
	Tags            []string // filter results by tags (fuzzy match)
	SortByDate      bool     // sort results by updated_at descending
	ExcludeOutdated bool     // exclude documents where outdated = 1
	Scope           string   // filter by memory_scope prefix (empty = all)
}

SearchOptions provides optional filtering/sorting for search queries.

type SearchResult

type SearchResult struct {
	Document     Document
	ChunkContent string
	Score        float64
	Rank         int
}

SearchResult represents a ranked search result from the knowledge base.

type SyncStats added in v0.160.0

type SyncStats struct {
	Scanned   int `json:"scanned"`
	Added     int `json:"added"`
	Updated   int `json:"updated"`
	Unchanged int `json:"unchanged"`
	Deleted   int `json:"deleted"`
}

SyncStats contains counters from a filesystem-to-KB synchronization run.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL