knowledge

type BM25Retriever ¶ added in v0.2.1

type BM25Retriever struct {
	Chunks ChunkRepo
}

BM25Retriever queries ChunkRepo with Mode=ModeBM25. It is layered: queries whose Layer is not LayerDetail short-circuit to nil.

func NewBM25Retriever(c ChunkRepo) *BM25Retriever

func (r *BM25Retriever) Name() string

func (r *BM25Retriever) Recall(ctx context.Context, q Query) ([]Candidate, error)

type CJKTokenizer ¶

type CJKTokenizer = textsearch.CJKTokenizer

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type Candidate ¶ added in v0.2.1

type Candidate struct {
	Hit    Hit
	Source string
}

Candidate is the per-item recall result returned by ChunkRepo / LayerRepo. Source identifies the producing retriever ("bm25" / "vector" / "layer") and is consumed by the Ranker for fusion.

type ChangeEvent ¶ added in v0.2.1

type ChangeEvent struct {
	DatasetID string
	DocName   string
	Kind      EventKind
}

ChangeEvent carries enough granularity for targeted rebuilds. DocName == "" denotes a dataset-level event.

type ChunkConfig ¶

type ChunkConfig struct {
	ChunkSize    int `json:"chunk_size,omitempty"`
	ChunkOverlap int `json:"chunk_overlap,omitempty"`
}

ChunkConfig controls document chunking.

func DefaultChunkConfig() ChunkConfig

type ChunkQuery ¶ added in v0.2.1

type ChunkQuery struct {
	DatasetIDs []string
	Text       string
	Vector     []float32
	Mode       Mode
	TopK       int
}

ChunkQuery is the recall input passed by Retrievers to ChunkRepo.Search.

Empty DatasetIDs means "every dataset" (cross-dataset search). When Mode is ModeVector or ModeHybrid, Vector should be supplied; backends that cannot satisfy a mode return an empty result without error.

type ChunkRepo ¶ added in v0.2.1

type ChunkRepo interface {
	Replace(ctx context.Context, datasetID, docName string, chunks []DerivedChunk) error
	DeleteByDoc(ctx context.Context, datasetID, docName string) error
	DeleteByDataset(ctx context.Context, datasetID string) error
	Search(ctx context.Context, q ChunkQuery) ([]Candidate, error)
}

ChunkRepo persists DerivedChunks and supports recall.

Replace MUST be atomic: callers rely on it to eliminate stale chunks when a SourceDocument is updated (contract guarantee #5).

type ChunkSigReader ¶ added in v0.3.16

type ChunkSigReader interface {
	GetDocSig(ctx context.Context, datasetID, docName string) (DerivedSig, bool, error)
}

ChunkSigReader is an OPTIONAL extension that ChunkRepo implementations may also satisfy. When supported, Service.Rebuild can compare the on-disk DerivedSig of an existing doc against the current (SourceVer, ChunkerSig, EmbedSig) and skip the chunk + embed work when the doc is already fresh — honouring the DerivedSig.IsStale contract documented on Rebuild.

Return (DerivedSig{}, false, nil) when the doc has no chunks yet (first ingest). Return an error only for backend faults; missing docs are NOT an error.

Backends that do not implement ChunkSigReader force Rebuild onto the unconditional re-chunk + re-embed path. Fix landed in #152.

type ChunkSpec ¶ added in v0.2.1

type ChunkSpec struct {
	Index   int
	Offset  int
	Content string
}

ChunkSpec is the chunker output: a positionally-tagged content slice without dataset/doc identity (the Service fills those in).

type Chunker ¶ added in v0.2.1

type Chunker interface {
	Split(content string) []ChunkSpec
	Sig() string
}

Chunker turns raw content into ordered ChunkSpecs.

Implementations MUST be deterministic (same input -> same output) and MUST return a stable Sig() so derived data freshness can be checked.

func NewDefaultChunker(cfg ChunkConfig) Chunker

type ContextLayer ¶

type ContextLayer = Layer

ContextLayer is the pre-v0.3.0 name for Layer. Retained as a type alias so external code that referenced knowledge.ContextLayer keeps compiling. The constant set (LayerAbstract / LayerOverview / LayerDetail) lives in types.go.

type CorpusStats ¶

type CorpusStats = textsearch.CorpusStats

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type DatasetContext ¶

type DatasetContext struct {
	Abstract string // dataset-level L0
	Overview string // dataset-level L1
}

DatasetContext groups the layered context for an entire dataset.

func GenerateDatasetContext(ctx context.Context, l llm.LLM, summaries []DocumentSummary) (DatasetContext, error)

type DerivedChunk ¶ added in v0.2.1

type DerivedChunk struct {
	DatasetID string
	DocName   string
	Index     int
	Offset    int
	Content   string
	Vector    []float32
	Sig       DerivedSig
}

DerivedChunk is one retrieval unit derived from a SourceDocument.

func ChunkText(docName, content string, cfg ChunkConfig) []DerivedChunk

type DerivedLayer ¶ added in v0.2.1

type DerivedLayer struct {
	DatasetID string
	DocName   string
	Layer     Layer
	Content   string
	Vector    []float32
	Sig       DerivedSig
}

DerivedLayer is an LLM-produced summary of a document or dataset. DocName == "" denotes a dataset-level layer.

type DerivedSig ¶ added in v0.2.1

type DerivedSig struct {
	SourceVer  uint64
	ChunkerSig string
	PromptSig  string
	EmbedSig   string
}

DerivedSig binds a derived artifact to the source revision and to the configuration that produced it. Required on every derived object.

• SourceVer is the SourceDocument.Version that produced this artifact.
• ChunkerSig is non-empty for chunk artifacts and empty for layers.
• PromptSig is non-empty for layer artifacts and empty for chunks.
• EmbedSig identifies the embedder, "" when no vector is attached.

func (sig DerivedSig) IsStale(want DerivedSig) bool

type DocLevelSearcher ¶ added in v0.3.10

type DocLevelSearcher interface {
	SearchDocs(ctx context.Context, q ChunkQuery) ([]Candidate, error)
}

DocLevelSearcher is an OPTIONAL extension that ChunkRepo implementations may also satisfy. When supported, the backend can answer searches at doc-level granularity (one Candidate per docName, scored over the whole document rather than per chunk), avoiding the chunks→docID collapse that callers (e.g. eval/beir) would otherwise have to implement themselves.

SCOPE: BM25 only. Doc-level vector / hybrid retrieval is a separate capability — see DocVectorSearcher. Service.SearchDocuments routes q.Mode == ModeBM25 here and ModeVector / ModeHybrid to DocVectorSearcher; backends that do not implement the requested capability surface a clear error rather than silently degrading. See #126 for the BM25 rationale, #145 for the vector rationale.

type DocVectorSearcher ¶ added in v0.3.16

type DocVectorSearcher interface {
	SearchDocsByVector(ctx context.Context, q ChunkQuery) ([]Candidate, error)
}

DocVectorSearcher is an OPTIONAL extension declaring doc-level vector (and hybrid) retrieval as an explicit capability (issue #145).

Pre-fix, ChunkRepo.SearchDocs accepted q.Mode and returned errdefs.NotAvailable mid-call when the requested mode was anything other than ModeBM25. That mixed two unrelated concerns — "what granularity is supported" (DocLevelSearcher) and "what scoring modes are supported" — into a runtime error that callers had to pattern-match on. This interface separates them: the capability is declared at compile time and probed via type assertion in Service.SearchDocuments. Backends that do not implement it surface a clear NotAvailable up front, not a buried mid-search error.

Hybrid handling is the implementer's responsibility (BM25 + cosine fusion, late-chunking, etc.). The interface accepts the same ChunkQuery as DocLevelSearcher and is expected to honour q.Mode == ModeVector or ModeHybrid.

No in-tree implementation yet — both [FSChunkRepo] and [RetrievalChunkRepo] hold per-chunk vectors but no per-doc representation. Mean-pool / late-chunking will land via a follow-up that adds SearchDocsByVector to one or both repos.

type DocumentContext ¶

type DocumentContext struct {
	Abstract string // L0
	Overview string // L1
}

DocumentContext groups the layered context for a single document.

func GenerateDocumentContext(ctx context.Context, l llm.LLM, content string) (DocumentContext, error)

type DocumentRepo ¶ added in v0.2.1

type DocumentRepo interface {
	Put(ctx context.Context, doc SourceDocument) error
	Get(ctx context.Context, datasetID, name string) (*SourceDocument, error)
	Delete(ctx context.Context, datasetID, name string) error
	List(ctx context.Context, datasetID string) ([]SourceDocument, error)
	ListDatasets(ctx context.Context) ([]string, error)
}

DocumentRepo persists SourceDocuments. Implementations MUST guarantee:

• Put atomically increments SourceDocument.Version.
• Get returns the most recent Put with Content losslessly preserved (contract guarantee #4).
• Delete is idempotent.

Implementations live in sdk/knowledge/backend/*.

type DocumentSummary ¶

type DocumentSummary struct {
	Name     string
	Abstract string
}

DocumentSummary pairs a document name with its L0 abstract, used as input to GenerateDatasetContext.

type Embedder ¶

type Embedder = embedding.Embedder

Embedder is an alias for the SDK embedding.Embedder interface. It supports both single-text and batch embeddings.

type EventKind ¶ added in v0.2.1

type EventKind int

EventKind classifies a ChangeEvent emitted by an EventNotifier implementation. Used by EventReloader to decide whether to perform a targeted or dataset-wide rebuild.

const (
	// EventPut signals that a single document was created or updated.
	EventPut EventKind = iota
	// EventDelete signals that a single document was removed.
	EventDelete
	// EventBulk signals a dataset-level mass change (e.g. snapshot replaced).
	EventBulk
)

type EventNotifier ¶ added in v0.2.1

type EventNotifier interface {
	Events() <-chan ChangeEvent
	Close() error
}

EventNotifier is the producer side of the reload pipeline. Events carry dataset/doc granularity so the consumer can issue targeted Rebuilds instead of a global one.

Implementations live in adapter packages so the sdk core stays dependency-free. Implementations MUST close the Events channel when Close() is called.

type EventReloader ¶ added in v0.2.1

type EventReloader struct {
	// contains filtered or unexported fields
}

EventReloader debounces ChangeEvents and triggers Rebuild on the trailing edge. Rebuilds are serialised: a new rebuild waits for the previous one to finish.

Targeted vs global rebuilds: when the debounce window contains events for a single (dataset, doc) pair, the rebuild is scoped to that pair; when it touches multiple datasets or any EventBulk event, a dataset-wide rebuild is issued instead. Mixed datasets in one window collapse to a global RebuildScope{} (every dataset).

func NewEventReloader(target Rebuilder, notifier EventNotifier, opts ReloaderOptions) *EventReloader

func (r *EventReloader) Close() error

func (r *EventReloader) Run(ctx context.Context) error

type Hit ¶ added in v0.2.1

type Hit struct {
	DatasetID  string
	DocName    string
	Layer      Layer
	Content    string
	Score      float64
	ChunkIndex int            // -1 for layer hits
	Sig        DerivedSig     // freshness traceability
	Metadata   map[string]any // backend-passthrough metadata
}

Hit is one ranked search result.

func FuseHits(perRetriever [][]Hit, k int) []Hit

type Layer ¶ added in v0.2.1

type Layer string

Layer indicates the granularity of a search result.

The pre-v0.3.0 name was ContextLayer; that name remains exported as a type alias in model.go so callers using either spelling keep compiling.

const (
	LayerAbstract Layer = "L0" // ~100 token one-sentence summary
	LayerOverview Layer = "L1" // ~1k token structured overview
	LayerDetail   Layer = "L2" // full chunk content
)

type LayerQuery ¶ added in v0.2.1

type LayerQuery struct {
	DatasetIDs []string
	Layer      Layer
	Text       string
	Vector     []float32
	Mode       Mode
	TopK       int
}

LayerQuery is the recall input for layer-tier searches.

type LayerRepo ¶ added in v0.2.1

type LayerRepo interface {
	Put(ctx context.Context, layer DerivedLayer) error
	Get(ctx context.Context, datasetID, docName string, layer Layer) (*DerivedLayer, error)
	DeleteByDoc(ctx context.Context, datasetID, docName string) error
	DeleteByDataset(ctx context.Context, datasetID string) error
	Search(ctx context.Context, q LayerQuery) ([]Candidate, error)
}

LayerRepo persists DerivedLayers and supports layer-scoped recall.

type LayerRetriever ¶ added in v0.2.1

type LayerRetriever struct {
	Layers   LayerRepo
	Embedder Embedder
}

LayerRetriever queries LayerRepo for L0/L1 hits. It activates only when Query.Layer is LayerAbstract or LayerOverview; LayerDetail queries are routed to chunk-tier retrievers instead.

Embedder is consulted only for vector lanes (ModeVector / ModeHybrid).

func NewLayerRetriever(l LayerRepo, e Embedder) *LayerRetriever

func (r *LayerRetriever) Name() string

func (r *LayerRetriever) Recall(ctx context.Context, q Query) ([]Candidate, error)

type Mode ¶ added in v0.2.1

type Mode = SearchMode

Mode is an alias for SearchMode, kept so newer call-sites can read "knowledge.Mode" without the redundant Search prefix while older "knowledge.SearchMode" call sites keep working unchanged.

func ResolveMode(m Mode) Mode

type Query ¶ added in v0.2.1

type Query struct {
	DatasetID string
	Scope     Scope
	Text      string
	Mode      Mode
	Layer     Layer
	TopK      int
	Threshold float64
	// contains filtered or unexported fields
}

Query is the canonical search input.

Validation rules (enforced by Service.Search):

• Layer must be valid; defaults to LayerDetail when zero.
• Mode must be valid; defaults to ModeBM25 when zero.
• Scope=ScopeSingleDataset requires DatasetID to be non-empty.

type RRFRanker ¶ added in v0.2.1

type RRFRanker struct {
	K int
}

RRFRanker performs reciprocal-rank fusion across candidates grouped by Candidate.Source. K defaults to DefaultRRFK when zero.

func NewRRFRanker() *RRFRanker

func (r *RRFRanker) Rank(candidates []Candidate, q Query) []Hit

type Ranker ¶ added in v0.2.1

type Ranker interface {
	Rank(candidates []Candidate, q Query) []Hit
}

Ranker fuses candidates from multiple Retrievers into ordered Hits.

type RebuildScope ¶ added in v0.2.1

type RebuildScope struct {
	DatasetID string // "" means all datasets
	DocName   string // "" means all documents in the dataset
}

RebuildScope narrows what Rebuilder.Rebuild touches. Zero value means "everything".

type Rebuilder ¶ added in v0.2.1

type Rebuilder interface {
	Rebuild(ctx context.Context, scope RebuildScope) error
}

Rebuilder is the consumer side of the change-driven reload pipeline. Service satisfies this interface; EventReloader invokes Rebuild on the trailing edge of a debounce window over an EventNotifier stream.

type ReloaderOptions ¶

type ReloaderOptions struct {
	Debounce       time.Duration
	RebuildTimeout time.Duration
}

ReloaderOptions configures EventReloader.

• Debounce controls the trailing-edge window.
• RebuildTimeout caps each rebuild call.

type Result ¶ added in v0.2.1

type Result struct {
	Hits []Hit
}

Result wraps the ordered hit list returned by Service.Search.

type Retriever ¶ added in v0.2.1

type Retriever interface {
	// Name returns a stable identifier used by Rankers (e.g. "bm25").
	Name() string
	// Recall fetches up to q.TopK candidates for q.
	Recall(ctx context.Context, q Query) ([]Candidate, error)
}

Retriever produces a candidate set for a Query. Implementations are stateless with respect to the Query; all state lives in the underlying repos.

type Scope ¶ added in v0.2.1

type Scope int

Scope expresses the dataset breadth of a query.

const (
	// ScopeSingleDataset restricts the search to Query.DatasetID.
	ScopeSingleDataset Scope = iota
	// ScopeAllDatasets searches across every known dataset.
	ScopeAllDatasets
)

type SearchEngine ¶ added in v0.2.1

type SearchEngine struct {
	Chunk []Retriever // chunk-tier retrievers (LayerDetail)
	Layer []Retriever // layer-tier retrievers (LayerAbstract / LayerOverview)
	Rank  Ranker
}

SearchEngine routes a Query through Retrievers, then a Ranker.

Behaviour:

• LayerDetail queries fan out to chunk-tier Retrievers (BM25/Vector).
• Layer{Abstract,Overview} queries fan out to layer-tier Retrievers.
• ModeHybrid runs both BM25 and Vector recall paths and lets the Ranker fuse them (RRF by default).

func NewSearchEngine(chunk, layer []Retriever, ranker Ranker) *SearchEngine

func (e *SearchEngine) Search(ctx context.Context, q Query) (*Result, error)

type SearchMode ¶

type SearchMode string

SearchMode chooses the retrieval algorithm. Legacy callers that pass the empty string are normalised to ModeBM25 by ResolveMode().

const (
	ModeBM25   SearchMode = "bm25"
	ModeVector SearchMode = "vector"
	ModeHybrid SearchMode = "hybrid"
)

type Service ¶ added in v0.2.1

type Service struct {
	// contains filtered or unexported fields
}

Service orchestrates document lifecycle, derived-data persistence and search. All public Knowledge entry points (graph node, tool adapters) route through Service so contract guarantees live in one place.

func NewService(
	docs DocumentRepo,
	chunks ChunkRepo,
	layers LayerRepo,
	engine *SearchEngine,
	opts ServiceOptions,
) *Service

func (s *Service) DatasetLayer(ctx context.Context, datasetID string, layer Layer) (string, error)

func (s *Service) DeleteDocument(ctx context.Context, datasetID, name string) error

func (s *Service) GetDocument(ctx context.Context, datasetID, name string) (*SourceDocument, error)

func (s *Service) Layer(ctx context.Context, datasetID, name string, layer Layer) (string, error)

func (s *Service) ListDatasets(ctx context.Context) ([]string, error)

func (s *Service) ListDocuments(ctx context.Context, datasetID string) ([]SourceDocument, error)

func (s *Service) PutDatasetLayer(ctx context.Context, datasetID string, layer Layer, content string) error

func (s *Service) PutDocument(ctx context.Context, datasetID, name, raw string) error

func (s *Service) PutDocumentLayer(ctx context.Context, datasetID, name string, layer Layer, content string) error

func (s *Service) Rebuild(ctx context.Context, scope RebuildScope) error

func (s *Service) Search(ctx context.Context, q Query) (*Result, error)

func (s *Service) SearchDocuments(ctx context.Context, q Query) (*Result, error)

type ServiceOptions ¶ added in v0.2.1

type ServiceOptions struct {
	// Chunker overrides the default chunker; nil means "use
	// NewDefaultChunker(DefaultChunkConfig())".
	Chunker Chunker
	// Embedder enables vector indexing and semantic search; nil
	// disables vector lanes.
	Embedder Embedder
	// EmbedSig is stamped onto every DerivedSig produced while this
	// Service runs. Embedder doesn't expose a model identifier, so
	// callers (typically the Factory) supply one. Empty string means
	// "use the embedder's Go type name", which is good enough for
	// freshness checks within a single binary but not stable across
	// processes — production wiring should set it explicitly.
	EmbedSig string
	// Now overrides the clock; nil means time.Now (unit-test hook).
	Now func() time.Time
}

ServiceOptions configures a Service. Nil-friendly: every field is optional and falls back to a sensible default.

type SimpleTokenizer ¶

type SimpleTokenizer = textsearch.SimpleTokenizer

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type SourceDocument ¶ added in v0.2.1

type SourceDocument struct {
	DatasetID string
	Name      string
	Content   string
	Metadata  map[string]string

	// Version is monotonically incremented on every successful Put.
	// Derived data uses it as a freshness key.
	Version   uint64
	UpdatedAt time.Time
}

SourceDocument is the canonical, lossless representation of user input.

It is the single source of truth for a document; every DerivedChunk and DerivedLayer carries a DerivedSig that points back to a particular SourceDocument.Version, so derived data can be detected as stale and recomputed deterministically.

type Tokenizer ¶

type Tokenizer = textsearch.Tokenizer

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type VectorRetriever ¶ added in v0.2.1

type VectorRetriever struct {
	Chunks   ChunkRepo
	Embedder Embedder
}

VectorRetriever queries ChunkRepo with Mode=ModeVector. Embedder is invoked lazily; nil disables the retriever (Recall returns nil).

func NewVectorRetriever(c ChunkRepo, e Embedder) *VectorRetriever

func (r *VectorRetriever) Name() string

func (r *VectorRetriever) Recall(ctx context.Context, q Query) ([]Candidate, error)