kb

package
v0.629.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 22, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package kb provides a knowledge base system for storing and searching documents.

Documents are chunked and embedded for hybrid search combining vector similarity and full-text search using Reciprocal Rank Fusion (RRF).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DocumentSlugs added in v0.608.1 

func DocumentSlugs(filePath string, aliases []string) []string

DocumentSlugs returns every slug a document can be addressed by: its full path, its basename, and any declared aliases. Link resolution matches a target slug against this set.

func ExtractAliasesFromMetadata added in v0.608.1 

func ExtractAliasesFromMetadata(meta map[string]interface{}) []string

ExtractAliasesFromMetadata reads the "aliases" key from a metadata map. Aliases are the extra slugs a document can be reached by from a [[wiki link]].

func ExtractTagsFromMetadata added in v0.413.0 

func ExtractTagsFromMetadata(meta map[string]interface{}) []string

ExtractTagsFromMetadata reads the "tags" key from a metadata map and returns them as a string slice. Returns nil if no tags are present.

func InjectAliasesIntoMetadata added in v0.608.1 

func InjectAliasesIntoMetadata(meta map[string]interface{}, aliases []string) map[string]interface{}

InjectAliasesIntoMetadata stores aliases in the metadata map under "aliases" so link resolution can read them without re-parsing the document body.

func InjectTagsIntoMetadata added in v0.413.0 

func InjectTagsIntoMetadata(meta map[string]interface{}, tags []string) map[string]interface{}

InjectTagsIntoMetadata ensures tags are stored in the metadata map under "tags". If tags is nil/empty and no existing tags, the key is left absent.

func NormalizeSlug added in v0.608.1 

func NormalizeSlug(raw string) string

NormalizeSlug converts a link target or document path into its canonical slug: lower-cased, markdown extension stripped, spaces/underscores collapsed into single dashes, path separators preserved. 

"pando/features/Foo Bar.md" -> "pando/features/foo-bar"
"Feature_Foo"               -> "feature-foo"

func SerializeFrontMatter added in v0.413.0 

func SerializeFrontMatter(fm FrontMatter, body string) string

SerializeFrontMatter produces a document string with YAML front matter prepended to the body content.

func StripFrontMatter added in v0.413.0 

func StripFrontMatter(raw string) string

StripFrontMatter removes any YAML front matter from raw content and returns only the body. This is a convenience wrapper around ParseFrontMatter.

Types

type BackfillStats added in v0.608.1 

type BackfillStats struct {
	// Candidates is how many documents were selected for (re)scanning.
	Candidates int
	// Scanned is how many of them were actually processed (may be lower than
	// Candidates when the context is cancelled mid-pass).
	Scanned int
	// Documents is how many produced at least one link.
	Documents int
	// Links is the number of link rows written.
	Links int
}

BackfillStats reports what a link backfill pass did. 

type Backlink struct {
	// FilePath of the document that holds the link.
	FilePath string
	// Slug is the target the source wrote, which may be an alias or a bare
	// basename rather than this document's full path.
	Slug string
	// Label is the [[target|label]] text, empty when the link had none.
	Label string
}

Backlink is an incoming link: another document pointing at this one.

type Document 

type Document struct {
	ID        int64
	FilePath  string
	Content   string
	Metadata  map[string]interface{}
	Tags      []string
	CreatedAt time.Time
	UpdatedAt time.Time
	// Memory system fields (populated only when reading memory documents)
	MemoryKey   string
	MemoryScope string
	Importance  float64
	Hits        int
	Source      string
	Outdated    bool
	ExpiresAt   *time.Time
}

Document represents a stored document in the knowledge base.

type DocumentConverter added in v0.605.1 

type DocumentConverter interface {
	// ConvertFile converts the file at path to Markdown text.
	ConvertFile(path string) (string, error)
	// IsConvertibleDocument reports whether path is a document format that
	// should be auto-converted during ingestion (curated subset, excludes
	// plain markdown/text which is indexed verbatim).
	IsConvertibleDocument(path string) bool
}

DocumentConverter converts rich document formats (docx, pdf, xlsx, …) to Markdown for on-the-fly KB ingestion. It is implemented by internal/convert.Converter and injected via SetDocumentConverter to avoid a hard dependency from the kb package on the conversion library.

type DocumentRef added in v0.416.3 

type DocumentRef struct {
	ID       int64
	FilePath string
}

DocumentRef is a minimal reference to a KB document (ID + path).

type FrontMatter added in v0.413.0 

type FrontMatter struct {
	CreatedAt time.Time `yaml:"created_at,omitempty"`
	UpdatedAt time.Time `yaml:"updated_at,omitempty"`
	Tags      []string  `yaml:"tags,omitempty"`
	// Aliases are extra names a document can be addressed by from a [[wiki link]],
	// on top of its path and basename.
	Aliases []string `yaml:"aliases,omitempty"`
	// Memory fields — only serialized when non-zero/non-empty
	Key        string     `yaml:"key,omitempty"`
	Scope      string     `yaml:"scope,omitempty"`
	Outdated   bool       `yaml:"outdated,omitempty"`
	ExpiresAt  *time.Time `yaml:"expires_at,omitempty"`
	Hits       int        `yaml:"hits,omitempty"`
	Importance float64    `yaml:"importance,omitempty"`
	Source     string     `yaml:"source,omitempty"`
}

FrontMatter holds the YAML front matter fields for KB documents.

func MergeFrontMatter added in v0.413.0 

func MergeFrontMatter(existing, incoming FrontMatter) FrontMatter

MergeFrontMatter combines existing and incoming front matter. It preserves created_at from existing, sets updated_at to now, and replaces tags with new if provided (otherwise keeps existing).

func NewFrontMatter added in v0.413.0 

func NewFrontMatter(tags []string, opts *MemoryOptions) FrontMatter

NewFrontMatter creates a fresh FrontMatter with created_at/updated_at set to now. When opts is non-nil, memory fields are populated from it.

func ParseFrontMatter added in v0.413.0 

func ParseFrontMatter(raw string) (FrontMatter, string, error)

ParseFrontMatter splits YAML front matter from the body of a document. If no valid front matter is found, it returns a zero FrontMatter and the original content as body.

type KBStore 

type KBStore struct {
	// contains filtered or unexported fields
}

KBStore manages knowledge base documents with chunking, embeddings, and hybrid search. It uses SQLite for storage with FTS5 for full-text search and in-memory vector search.

func NewKBStore 

func NewKBStore(db *sql.DB, embedder embeddings.Embedder, chunkSize, chunkOverlap int) *KBStore

NewKBStore creates a new KBStore instance.

Parameters:

  • db: SQLite database connection
  • embedder: Embedder for generating chunk embeddings
  • chunkSize: Maximum chunk size in characters (0 = use default)
  • chunkOverlap: Overlap between consecutive chunks (0 = use default)

func (*KBStore) AddDocument 

func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error

AddDocument adds a new document to the knowledge base. It chunks the content, generates embeddings, and updates the FTS index.

func (*KBStore) AddDocumentWithEmbeddings added in v0.326.0 

func (s *KBStore) AddDocumentWithEmbeddings(ctx context.Context, filePath, content string, metadata map[string]interface{}, chunks []string, embeddings [][]float32) error

AddDocumentWithEmbeddings inserts a document using pre-computed chunks and embeddings. Called by the primary IPC dispatcher when a secondary forwards a KBAddDocument write. No embedding generation is performed; the provided values are stored directly. 

func (s *KBStore) BackfillLinks(ctx context.Context) (BackfillStats, error)

BackfillLinks indexes the [[wiki links]] of documents that were stored before the link graph existed. It is the compatibility path for databases upgraded from a version without kb_links: those documents keep their content but have no link rows, and the filesystem sync will not re-add them because it skips files whose mtime has not changed.

No chunking or embedding happens here: the links are extracted from the content already stored, so the pass costs local CPU and a few writes, never a call to the embedding provider.

Candidates are documents whose content contains "[[" and that have no link rows yet — the content itself acts as the marker, so no schema column is needed and the pass is idempotent: once a document is indexed it stops being a candidate. A document whose only "[[" occurrences sit inside code fences yields no links and is re-scanned by later passes, which is harmless.

The pass is a no-op on instances that write through the IPC proxy: the primary owns the writer connection and runs the backfill itself. 

func (s *KBStore) Backlinks(ctx context.Context, filePath string) ([]Backlink, error)

Backlinks returns the documents that link to this one, through its path, its basename or any of its aliases.

A link only counts when it actually resolves here: two documents may share a basename, and the shorter form belongs to whichever one the resolver awards it to, so a link to "[[foo]]" is not a backlink of every document named foo.md.

func (*KBStore) ConfigureFilesystemMirror added in v0.160.0 

func (s *KBStore) ConfigureFilesystemMirror(dirPath string) error

ConfigureFilesystemMirror sets a base directory where KB documents are mirrored as markdown files on add/update/delete operations.

func (*KBStore) CountDocuments 

func (s *KBStore) CountDocuments(ctx context.Context) (int64, error)

CountDocuments returns the total number of documents.

func (*KBStore) DeleteDocument 

func (s *KBStore) DeleteDocument(ctx context.Context, filePath string) error

DeleteDocument removes a document and all its chunks from the knowledge base.

func (*KBStore) DeleteDocumentFromFilesystem added in v0.160.0 

func (s *KBStore) DeleteDocumentFromFilesystem(filePath string) error

DeleteDocumentFromFilesystem removes a mirrored KB document from disk. If no mirror path is configured, this is a no-op.

func (*KBStore) FilesystemMirrorPath added in v0.160.0 

func (s *KBStore) FilesystemMirrorPath() string

FilesystemMirrorPath returns the configured filesystem mirror path.

func (*KBStore) GetDocument 

func (s *KBStore) GetDocument(ctx context.Context, filePath string) (*Document, error)

GetDocument retrieves a document by file path.

func (*KBStore) GetExpiredDocuments added in v0.416.3 

func (s *KBStore) GetExpiredDocuments(ctx context.Context) ([]DocumentRef, error)

GetExpiredDocuments returns all non-outdated documents whose expires_at is in the past.

func (*KBStore) GetMemoriesForInjection added in v0.416.3 

func (s *KBStore) GetMemoriesForInjection(ctx context.Context, query string, limit int, maxChars int, pinnedScopes []string) ([]MemoryResult, error)

GetMemoriesForInjection retrieves memory documents ranked for context injection. It combines semantic search results with pinned-scope documents, deduplicates, scores, and trims to fit within the maxChars budget.

func (*KBStore) GetMemoryByKey added in v0.416.3 

func (s *KBStore) GetMemoryByKey(ctx context.Context, key string) (*Document, error)

GetMemoryByKey retrieves a document by its memory_key. Returns nil, nil when no document with the key exists. 

func (s *KBStore) HasLinks(ctx context.Context) (bool, error)

HasLinks reports whether the knowledge base holds any wiki link at all.

It is the guard the tools use before doing any graph work: a knowledge base written before wiki links existed (or one that simply never uses the syntax) answers false, so the tools can skip the resolver entirely and present exactly the output they presented before the graph was added.

func (*KBStore) IncrementMemoryHits added in v0.416.3 

func (s *KBStore) IncrementMemoryHits(ctx context.Context, docID int64, defaultTTLDays int) error

IncrementMemoryHits increments the hit counter for a document and extends its TTL.

func (*KBStore) LinkCountsFor added in v0.608.1 

func (s *KBStore) LinkCountsFor(ctx context.Context, filePaths []string) (map[string]LinkCounts, error)

LinkCountsFor returns the link counts of the given documents in a single pass over the graph, for callers (search results) that want to hint at connections without fetching them. Documents with no links in either direction are absent from the map, and a link-less knowledge base returns an empty map.

func (*KBStore) LinksFrom added in v0.608.1 

func (s *KBStore) LinksFrom(ctx context.Context, filePath string) ([]WikiLink, error)

LinksFrom returns the wiki links stored for a document, ordered by position. Targets are returned unresolved; resolution against real documents is done by the graph queries.

func (*KBStore) ListDocuments 

func (s *KBStore) ListDocuments(ctx context.Context, limit, offset int) ([]Document, error)

ListDocuments returns paginated documents.

func (*KBStore) MarkDocumentOutdated added in v0.416.3 

func (s *KBStore) MarkDocumentOutdated(ctx context.Context, filePath string) error

MarkDocumentOutdated marks a document as outdated and updates its filesystem mirror. 

func (s *KBStore) OutgoingLinks(ctx context.Context, filePath string) ([]LinkTarget, error)

OutgoingLinks returns the wiki links a document declares, each resolved against the current documents. A document with no links yields an empty slice and no error, which is the normal state of a knowledge base that does not use the syntax.

func (*KBStore) RebuildFTS 

func (s *KBStore) RebuildFTS(ctx context.Context) error

RebuildFTS rebuilds the FTS5 index from kb_chunks.

func (*KBStore) RelatedDocuments added in v0.608.1 

func (s *KBStore) RelatedDocuments(ctx context.Context, filePath string, limit int) ([]RelatedDocument, error)

RelatedDocuments returns the neighbours of a document in the graph, ordered by how strongly they are connected: documents it links to, documents that link to it, and — weakly — documents sharing tags with it. limit <= 0 uses a default.

func (*KBStore) RelinkAll added in v0.608.1 

func (s *KBStore) RelinkAll(ctx context.Context) (BackfillStats, error)

RelinkAll rebuilds the whole link graph: every stored link row is dropped and re-extracted from the documents' content. Unlike BackfillLinks it does not skip documents that already have links, so it is the way to pick up a change in the extractor (a new syntax, a different slug rule) that made the stored rows stale.

func (*KBStore) SearchDocuments 

func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)

SearchDocuments performs hybrid search combining vector similarity and FTS. Results are fused using Reciprocal Rank Fusion (RRF).

func (*KBStore) SearchDocumentsWithOptions added in v0.413.0 

func (s *KBStore) SearchDocumentsWithOptions(ctx context.Context, query string, limit int, opts SearchOptions) ([]SearchResult, error)

SearchDocumentsWithOptions performs hybrid search with optional tag filtering and chronological ordering. When opts.Tags is non-empty, results are filtered to documents whose tags fuzzy-match any of the requested tags. When opts.SortByDate is true, results are sorted by updated_at descending.

func (*KBStore) SetDocumentConverter added in v0.605.1 

func (s *KBStore) SetDocumentConverter(c DocumentConverter)

SetDocumentConverter installs a converter so that supported document files found in the KB source directory are converted to Markdown and indexed, referencing the original file. Passing nil disables conversion (the default), in which case only .md files are indexed.

func (*KBStore) SetSyncWorkers added in v0.160.0 

func (s *KBStore) SetSyncWorkers(workers int)

SetSyncWorkers configures how many concurrent workers are used for KB filesystem import preprocessing (file reads and in-memory preparation). SQLite writes remain serialized through a single writer goroutine.

func (*KBStore) SetWikiLinksEnabled added in v0.609.1 

func (s *KBStore) SetWikiLinksEnabled(enabled bool)

SetWikiLinksEnabled toggles the [[wiki link]] graph. When false no link rows are written and every graph query answers empty, so the KB tools present the output they presented before the graph existed.

Turning it off does not wipe the graph: the rows already indexed stay in the database, invisible, and light up again if it is turned back on — no reindex needed. Only a document rewritten while it is off loses its rows, like any update does.

func (*KBStore) SetWriteProxy added in v0.326.0 

func (s *KBStore) SetWriteProxy(proxy *dbproxy.DBProxy)

SetWriteProxy configures a DB proxy for mutating operations.

func (*KBStore) SyncDirectory 

func (s *KBStore) SyncDirectory(ctx context.Context, dirPath string) error

SyncDirectory imports or syncs all .md files from a directory. Existing documents are updated, new files are added.

func (*KBStore) SyncDirectoryWithStats added in v0.160.0 

func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)

SyncDirectoryWithStats imports or syncs all markdown files from a directory. It recursively scans for .md files, upserts modified documents, and optionally deletes KB documents that no longer exist on disk for that directory source.

func (*KBStore) UpdateDocument 

func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error

UpdateDocument updates an existing document's content and metadata. It re-chunks and re-embeds the content.

func (*KBStore) UpsertMemory added in v0.416.3 

func (s *KBStore) UpsertMemory(ctx context.Context, opts MemoryUpsertOptions) (created bool, err error)

UpsertMemory stores or updates a memory document, keyed by opts.Key when provided. Returns created=true when a new document was inserted, false when an existing one was updated.

func (*KBStore) WantedConcepts added in v0.608.1 

func (s *KBStore) WantedConcepts(ctx context.Context, limit int) ([]WantedConcept, error)

WantedConcepts returns the link targets that no document defines, most linked first: the concepts the knowledge base refers to but never explains. limit <= 0 returns them all.

func (*KBStore) WatchDirectory added in v0.160.0 

func (s *KBStore) WatchDirectory(ctx context.Context, dirPath string) error

WatchDirectory monitors a KB source directory recursively and keeps KB documents in sync with .md file changes in near real-time.

func (*KBStore) WikiLinksEnabled added in v0.609.1 

func (s *KBStore) WikiLinksEnabled() bool

WikiLinksEnabled reports whether the wiki link graph is active.

func (*KBStore) WriteDocumentToFilesystem added in v0.160.0 

func (s *KBStore) WriteDocumentToFilesystem(filePath, content string) error

WriteDocumentToFilesystem writes a KB document to the configured mirror path. If no mirror path is configured, this is a no-op.

type LinkCounts added in v0.608.1 

type LinkCounts struct {
	Outgoing  int
	Backlinks int
}

LinkCounts is how connected a document is: the links it declares and the links it receives.

type LinkTarget added in v0.608.1 

type LinkTarget struct {
	WikiLink
	// FilePath of the document the link points at, empty when unresolved.
	FilePath string
	// Resolved reports whether the target slug matched a document.
	Resolved bool
}

LinkTarget is a wiki link together with the document it resolves to, if any. Unresolved targets are kept on purpose: they are concepts the author referred to but nobody has documented yet.

type MemoryGCService added in v0.416.3 

type MemoryGCService struct {
	// contains filtered or unexported fields
}

MemoryGCService periodically marks expired memory documents as outdated.

func NewMemoryGCService added in v0.416.3 

func NewMemoryGCService(store *KBStore, interval time.Duration) *MemoryGCService

NewMemoryGCService creates a new MemoryGCService.

func (*MemoryGCService) Start added in v0.416.3 

func (g *MemoryGCService) Start(ctx context.Context)

Start launches the GC loop in a background goroutine.

func (*MemoryGCService) Stop added in v0.416.3 

func (g *MemoryGCService) Stop()

Stop signals the background goroutine to exit.

type MemoryOptions added in v0.416.3 

type MemoryOptions struct {
	Key        string
	Scope      string
	Source     string
	Importance float64
	TTLDays    int // 0 = no expiry
}

MemoryOptions carries optional memory-layer settings for NewFrontMatter.

type MemoryResult added in v0.416.3 

type MemoryResult struct {
	Document     Document
	ChunkContent string
	Score        float64
}

MemoryResult wraps a Document with its chunk content and relevance score.

type MemoryUpsertOptions added in v0.416.3 

type MemoryUpsertOptions struct {
	FilePath       string
	Key            string
	Scope          string
	Content        string
	Tags           []string
	Metadata       map[string]interface{}
	Importance     float64
	Source         string
	DefaultTTLDays int // 0 → 180
}

MemoryUpsertOptions configures a memory document upsert operation.

type RelatedDocument added in v0.608.1 

type RelatedDocument struct {
	FilePath string
	Score    float64
	// Reasons explains the score, e.g. "links to it", "links here",
	// "shared tags: kb, wiki".
	Reasons []string
}

RelatedDocument is a neighbour in the document graph, scored by how it is connected.

type SearchOptions added in v0.413.0 

type SearchOptions struct {
	Tags            []string // filter results by tags (fuzzy match)
	SortByDate      bool     // sort results by updated_at descending
	ExcludeOutdated bool     // exclude documents where outdated = 1
	Scope           string   // filter by memory_scope prefix (empty = all)
}

SearchOptions provides optional filtering/sorting for search queries.

type SearchResult 

type SearchResult struct {
	Document     Document
	ChunkContent string
	Score        float64
	Rank         int
}

SearchResult represents a ranked search result from the knowledge base.

type SyncStats added in v0.160.0 

type SyncStats struct {
	Scanned   int `json:"scanned"`
	Added     int `json:"added"`
	Updated   int `json:"updated"`
	Unchanged int `json:"unchanged"`
	Deleted   int `json:"deleted"`
	// LinksIndexed counts the [[wiki links]] found in the documents this run
	// added or updated. Unchanged documents keep the links they already had and
	// are not counted, so this is what the run wrote, not the size of the graph.
	LinksIndexed int `json:"links_indexed"`
}

SyncStats contains counters from a filesystem-to-KB synchronization run.

type WantedConcept added in v0.608.1 

type WantedConcept struct {
	// Slug is the normalized target.
	Slug string
	// Raw is the target as an author first wrote it, which reads better than the
	// slug when suggesting a title.
	Raw string
	// Count is how many documents point at it.
	Count int
	// Sources are the documents that point at it.
	Sources []string
}

WantedConcept is a link target that no document defines yet: the wiki equivalent of a red link. Frequently wanted concepts are the best candidates for the next document to write. 

type WikiLink struct {
	Raw      string
	Slug     string
	Label    string
	Position int
}

WikiLink is a [[target]] or [[target|label]] reference found in a document body.

Slug is the normalized form used for storage and resolution; Raw preserves the target exactly as the author wrote it. A link is not required to resolve to an existing document: unresolved targets are kept on purpose, since they record concepts that are worth documenting later. 

func ExtractWikiLinks(body string) []WikiLink

ExtractWikiLinks returns the wiki links found in a document body, in order of appearance and deduplicated by slug (the first occurrence wins). Links inside fenced code blocks and inline code spans are ignored, so documentation that merely shows the syntax does not pollute the graph.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.