knowledge

type BM25Retriever ¶ added in v0.2.1

type BM25Retriever struct {
	Chunks ChunkRepo
}

BM25Retriever queries ChunkRepo with Mode=ModeBM25. It is layered: queries whose Layer is not LayerDetail short-circuit to nil.

func NewBM25Retriever(c ChunkRepo) *BM25Retriever

func (r *BM25Retriever) Name() string

func (r *BM25Retriever) Recall(ctx context.Context, q Query) ([]Candidate, error)

type CJKTokenizer ¶

type CJKTokenizer = textsearch.CJKTokenizer

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type Candidate ¶ added in v0.2.1

type Candidate struct {
	Hit    Hit
	Source string
}

Candidate is the per-item recall result returned by ChunkRepo / LayerRepo. Source identifies the producing retriever ("bm25" / "vector" / "layer") and is consumed by the Ranker for fusion.

type ChangeEvent ¶ added in v0.2.1

type ChangeEvent struct {
	DatasetID string
	DocName   string
	Kind      EventKind
}

ChangeEvent carries enough granularity for targeted rebuilds. DocName == "" denotes a dataset-level event.

NOTE (v0.2.x): The deprecated ChangeNotifier in deprecated.go still emits opaque struct{} events; sdkx/knowledge/watcher remains its only in-tree producer. The ChangeEvent shape declared here is what EventNotifier implementations will emit once watcher migrates in v0.3.0.

type ChunkConfig ¶

type ChunkConfig struct {
	ChunkSize    int `json:"chunk_size,omitempty"`
	ChunkOverlap int `json:"chunk_overlap,omitempty"`
}

ChunkConfig controls document chunking.

func DefaultChunkConfig() ChunkConfig

type ChunkQuery ¶ added in v0.2.1

type ChunkQuery struct {
	DatasetIDs []string
	Text       string
	Vector     []float32
	Mode       Mode
	TopK       int
}

ChunkQuery is the recall input passed by Retrievers to ChunkRepo.Search.

Empty DatasetIDs means "every dataset" (cross-dataset search). When Mode is ModeVector or ModeHybrid, Vector should be supplied; backends that cannot satisfy a mode return an empty result without error.

type ChunkRepo ¶ added in v0.2.1

type ChunkRepo interface {
	Replace(ctx context.Context, datasetID, docName string, chunks []DerivedChunk) error
	DeleteByDoc(ctx context.Context, datasetID, docName string) error
	DeleteByDataset(ctx context.Context, datasetID string) error
	Search(ctx context.Context, q ChunkQuery) ([]Candidate, error)
}

ChunkRepo persists DerivedChunks and supports recall.

Replace MUST be atomic: callers rely on it to eliminate stale chunks when a SourceDocument is updated (contract guarantee #5).

type ChunkSpec ¶ added in v0.2.1

type ChunkSpec struct {
	Index   int
	Offset  int
	Content string
}

ChunkSpec is the chunker output: a positionally-tagged content slice without dataset/doc identity (the Service fills those in).

type Chunker ¶ added in v0.2.1

type Chunker interface {
	Split(content string) []ChunkSpec
	Sig() string
}

Chunker turns raw content into ordered ChunkSpecs.

Implementations MUST be deterministic (same input -> same output) and MUST return a stable Sig() so derived data freshness can be checked.

func NewDefaultChunker(cfg ChunkConfig) Chunker

type ContextLayer ¶

type ContextLayer string

ContextLayer indicates the granularity of a search result.

v0.3.0 will rename ContextLayer -> Layer; the alias declared in model.go (type Layer = ContextLayer) lets new code adopt the final name today without breaking existing callers.

const (
	LayerAbstract ContextLayer = "L0" // ~100 token one-sentence summary
	LayerOverview ContextLayer = "L1" // ~1k token structured overview
	LayerDetail   ContextLayer = "L2" // full chunk content
)

type CorpusStats ¶

type CorpusStats = textsearch.CorpusStats

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type DatasetContext ¶

type DatasetContext struct {
	Abstract string // dataset-level L0
	Overview string // dataset-level L1
}

DatasetContext groups the layered context for an entire dataset.

func GenerateDatasetContext(ctx context.Context, l llm.LLM, summaries []DocumentSummary) (DatasetContext, error)

type DatasetQuery ¶

type DatasetQuery struct {
	DatasetID string `json:"dataset_id"`
	StateKey  string `json:"state_key"`
	TopK      int    `json:"top_k"`
}

DatasetQuery describes a single dataset search within a Knowledge node. Re-used by both the v0.3.0 knowledgenode.Config and the deprecated KnowledgeConfig (kept stable across versions).

type DerivedChunk ¶ added in v0.2.1

type DerivedChunk struct {
	DatasetID string
	DocName   string
	Index     int
	Offset    int
	Content   string
	Vector    []float32
	Sig       DerivedSig
}

DerivedChunk is one retrieval unit derived from a SourceDocument.

func ChunkText(docName, content string, cfg ChunkConfig) []DerivedChunk

type DerivedLayer ¶ added in v0.2.1

type DerivedLayer struct {
	DatasetID string
	DocName   string
	Layer     Layer
	Content   string
	Vector    []float32
	Sig       DerivedSig
}

DerivedLayer is an LLM-produced summary of a document or dataset. DocName == "" denotes a dataset-level layer.

type DerivedSig ¶ added in v0.2.1

type DerivedSig struct {
	SourceVer  uint64
	ChunkerSig string
	PromptSig  string
	EmbedSig   string
}

DerivedSig binds a derived artifact to the source revision and to the configuration that produced it. Required on every derived object.

• SourceVer is the SourceDocument.Version that produced this artifact.
• ChunkerSig is non-empty for chunk artifacts and empty for layers.
• PromptSig is non-empty for layer artifacts and empty for chunks.
• EmbedSig identifies the embedder, "" when no vector is attached.

func (sig DerivedSig) IsStale(want DerivedSig) bool

type DocumentContext ¶

type DocumentContext struct {
	Abstract string // L0
	Overview string // L1
}

DocumentContext groups the layered context for a single document.

func GenerateDocumentContext(ctx context.Context, l llm.LLM, content string) (DocumentContext, error)

type DocumentRepo ¶ added in v0.2.1

type DocumentRepo interface {
	Put(ctx context.Context, doc SourceDocument) error
	Get(ctx context.Context, datasetID, name string) (*SourceDocument, error)
	Delete(ctx context.Context, datasetID, name string) error
	List(ctx context.Context, datasetID string) ([]SourceDocument, error)
	ListDatasets(ctx context.Context) ([]string, error)
}

DocumentRepo persists SourceDocuments. Implementations MUST guarantee:

• Put atomically increments SourceDocument.Version.
• Get returns the most recent Put with Content losslessly preserved (contract guarantee #4).
• Delete is idempotent.

Implementations live in sdk/knowledge/backend/*.

type DocumentSummary ¶

type DocumentSummary struct {
	Name     string
	Abstract string
}

DocumentSummary pairs a document name with its L0 abstract, used as input to GenerateDatasetContext.

type Embedder ¶

type Embedder = embedding.Embedder

Embedder is an alias for the SDK embedding.Embedder interface. It supports both single-text and batch embeddings.

type EventKind ¶ added in v0.2.1

type EventKind int

EventKind classifies a ChangeEvent emitted by an EventNotifier implementation. Used by EventReloader to decide whether to perform a targeted or dataset-wide rebuild.

const (
	// EventPut signals that a single document was created or updated.
	EventPut EventKind = iota
	// EventDelete signals that a single document was removed.
	EventDelete
	// EventBulk signals a dataset-level mass change (e.g. snapshot replaced).
	EventBulk
)

type EventNotifier ¶ added in v0.2.1

type EventNotifier interface {
	Events() <-chan ChangeEvent
	Close() error
}

EventNotifier is the v0.3.0 producer side of the reload pipeline. It supersedes the deprecated ChangeNotifier (defined in deprecated.go): events carry dataset/doc granularity so the consumer can issue targeted Rebuilds instead of a global one.

Implementations live in adapter packages (e.g. sdkx/knowledge/watcher, once it migrates) so the sdk core stays dependency-free. Implementations MUST close the Events channel when Close() is called.

type EventReloader ¶ added in v0.2.1

type EventReloader struct {
	// contains filtered or unexported fields
}

EventReloader debounces ChangeEvents and triggers Rebuild on the trailing edge. Rebuilds are serialised: a new rebuild waits for the previous one to finish.

Targeted vs global rebuilds: when the debounce window contains events for a single (dataset, doc) pair, the rebuild is scoped to that pair; when it touches multiple datasets or any EventBulk event, a dataset-wide rebuild is issued instead. Mixed datasets in one window collapse to a global RebuildScope{} (every dataset).

EventReloader is the v0.3.0 successor to Reloader. The legacy Reloader (with its struct{}-channel ChangeNotifier, both in deprecated.go) remains exported during the deprecation window and will be removed in v0.3.0.

func NewEventReloader(target Rebuilder, notifier EventNotifier, opts ReloaderOptions) *EventReloader

func (r *EventReloader) Close() error

func (r *EventReloader) Run(ctx context.Context) error

type Hit ¶ added in v0.2.1

type Hit struct {
	DatasetID  string
	DocName    string
	Layer      Layer
	Content    string
	Score      float64
	ChunkIndex int            // -1 for layer hits
	Sig        DerivedSig     // freshness traceability
	Metadata   map[string]any // backend-passthrough metadata
}

Hit is one ranked search result.

func FuseHits(perRetriever [][]Hit, k int) []Hit

type Layer ¶ added in v0.2.1

type Layer = ContextLayer

Layer is the v0.3.0 name for ContextLayer. It is declared as a type alias so values flow seamlessly between old and new APIs during the deprecation window. The constant set lives in types.go.

type LayerQuery ¶ added in v0.2.1

type LayerQuery struct {
	DatasetIDs []string
	Layer      Layer
	Text       string
	Vector     []float32
	Mode       Mode
	TopK       int
}

LayerQuery is the recall input for layer-tier searches.

type LayerRepo ¶ added in v0.2.1

type LayerRepo interface {
	Put(ctx context.Context, layer DerivedLayer) error
	Get(ctx context.Context, datasetID, docName string, layer Layer) (*DerivedLayer, error)
	DeleteByDoc(ctx context.Context, datasetID, docName string) error
	DeleteByDataset(ctx context.Context, datasetID string) error
	Search(ctx context.Context, q LayerQuery) ([]Candidate, error)
}

LayerRepo persists DerivedLayers and supports layer-scoped recall.

type LayerRetriever ¶ added in v0.2.1

type LayerRetriever struct {
	Layers   LayerRepo
	Embedder Embedder
}

LayerRetriever queries LayerRepo for L0/L1 hits. It activates only when Query.Layer is LayerAbstract or LayerOverview; LayerDetail queries are routed to chunk-tier retrievers instead.

Embedder is consulted only for vector lanes (ModeVector / ModeHybrid).

func NewLayerRetriever(l LayerRepo, e Embedder) *LayerRetriever

func (r *LayerRetriever) Name() string

func (r *LayerRetriever) Recall(ctx context.Context, q Query) ([]Candidate, error)

type Mode ¶ added in v0.2.1

type Mode = SearchMode

Mode is the v0.3.0 name for SearchMode. It is declared as a type alias so values flow seamlessly between old and new APIs during the deprecation window. The constant set lives in types.go.

func ResolveMode(m Mode) Mode

type Query ¶ added in v0.2.1

type Query struct {
	DatasetID string
	Scope     Scope
	Text      string
	Mode      Mode
	Layer     Layer
	TopK      int
	Threshold float64
	// contains filtered or unexported fields
}

Query is the canonical search input.

Validation rules (enforced by Service.Search):

• Layer must be valid; defaults to LayerDetail when zero.
• Mode must be valid; defaults to ModeBM25 when zero.
• Scope=ScopeSingleDataset requires DatasetID to be non-empty.

type RRFRanker ¶ added in v0.2.1

type RRFRanker struct {
	K int
}

RRFRanker performs reciprocal-rank fusion across candidates grouped by Candidate.Source. K defaults to DefaultRRFK when zero.

func NewRRFRanker() *RRFRanker

func (r *RRFRanker) Rank(candidates []Candidate, q Query) []Hit

type Ranker ¶ added in v0.2.1

type Ranker interface {
	Rank(candidates []Candidate, q Query) []Hit
}

Ranker fuses candidates from multiple Retrievers into ordered Hits.

type RebuildScope ¶ added in v0.2.1

type RebuildScope struct {
	DatasetID string // "" means all datasets
	DocName   string // "" means all documents in the dataset
}

RebuildScope narrows what Rebuilder.Rebuild touches. Zero value means "everything".

type Rebuilder ¶ added in v0.2.1

type Rebuilder interface {
	Rebuild(ctx context.Context, scope RebuildScope) error
}

Rebuilder is the consumer side of the change-driven reload pipeline. Service satisfies this interface; EventReloader invokes Rebuild on the trailing edge of a debounce window over an EventNotifier stream.

type ReloaderOptions ¶

type ReloaderOptions struct {
	Debounce       time.Duration
	RebuildTimeout time.Duration
	Rebuild        func(context.Context) error
}

ReloaderOptions configures EventReloader (and the deprecated Reloader).

Field set is the union of both consumers:

• Debounce controls the trailing-edge window (both consumers).
• RebuildTimeout caps each rebuild call (EventReloader only; the legacy Reloader hard-codes 30s).
• Rebuild is the legacy hook used by the deprecated Reloader to swap the rebuild callback. EventReloader ignores it and always calls Rebuilder.Rebuild on the supplied target.

type Result ¶ added in v0.2.1

type Result struct {
	Hits []Hit
}

Result wraps the ordered hit list returned by Service.Search.

type Retriever ¶ added in v0.2.1

type Retriever interface {
	// Name returns a stable identifier used by Rankers (e.g. "bm25").
	Name() string
	// Recall fetches up to q.TopK candidates for q.
	Recall(ctx context.Context, q Query) ([]Candidate, error)
}

Retriever produces a candidate set for a Query. Implementations are stateless with respect to the Query; all state lives in the underlying repos.

type Scope ¶ added in v0.2.1

type Scope int

Scope expresses the dataset breadth of a query.

const (
	// ScopeSingleDataset restricts the search to Query.DatasetID.
	ScopeSingleDataset Scope = iota
	// ScopeAllDatasets searches across every known dataset.
	ScopeAllDatasets
)

type SearchEngine ¶ added in v0.2.1

type SearchEngine struct {
	Chunk []Retriever // chunk-tier retrievers (LayerDetail)
	Layer []Retriever // layer-tier retrievers (LayerAbstract / LayerOverview)
	Rank  Ranker
}

SearchEngine routes a Query through Retrievers, then a Ranker.

Behaviour:

• LayerDetail queries fan out to chunk-tier Retrievers (BM25/Vector).
• Layer{Abstract,Overview} queries fan out to layer-tier Retrievers.
• ModeHybrid runs both BM25 and Vector recall paths and lets the Ranker fuse them (RRF by default).

func NewSearchEngine(chunk, layer []Retriever, ranker Ranker) *SearchEngine

func (e *SearchEngine) Search(ctx context.Context, q Query) (*Result, error)

type SearchMode ¶

type SearchMode string

SearchMode chooses the retrieval algorithm.

v0.3.0 final values are explicit strings; legacy callers that pass the empty string are normalised to ModeBM25 by ResolveMode().

Deprecated names:

• ModeSemantic remains for backwards compatibility; new code should use ModeVector. They are recognised as equivalent at the Service boundary starting in v0.2.x and ModeSemantic is removed in v0.3.0.

const (
	ModeBM25     SearchMode = "bm25"
	ModeVector   SearchMode = "vector"
	ModeSemantic SearchMode = "semantic" // Deprecated: use ModeVector.
	ModeHybrid   SearchMode = "hybrid"
)

type Service ¶ added in v0.2.1

type Service struct {
	// contains filtered or unexported fields
}

Service orchestrates document lifecycle, derived-data persistence and search. All public Knowledge entry points (graph node, tools, deprecated stores) route through Service so contract guarantees (#1..#7 in doc.go) live in one place.

func NewService(
	docs DocumentRepo,
	chunks ChunkRepo,
	layers LayerRepo,
	engine *SearchEngine,
	opts ServiceOptions,
) *Service

func (s *Service) DatasetLayer(ctx context.Context, datasetID string, layer Layer) (string, error)

func (s *Service) DeleteDocument(ctx context.Context, datasetID, name string) error

func (s *Service) GetDocument(ctx context.Context, datasetID, name string) (*SourceDocument, error)

func (s *Service) Layer(ctx context.Context, datasetID, name string, layer Layer) (string, error)

func (s *Service) ListDatasets(ctx context.Context) ([]string, error)

func (s *Service) ListDocuments(ctx context.Context, datasetID string) ([]SourceDocument, error)

func (s *Service) PutDatasetLayer(ctx context.Context, datasetID string, layer Layer, content string) error

func (s *Service) PutDocument(ctx context.Context, datasetID, name, raw string) error

func (s *Service) PutDocumentLayer(ctx context.Context, datasetID, name string, layer Layer, content string) error

func (s *Service) Rebuild(ctx context.Context, scope RebuildScope) error

func (s *Service) Search(ctx context.Context, q Query) (*Result, error)

type ServiceOptions ¶ added in v0.2.1

type ServiceOptions struct {
	// Chunker overrides the default chunker; nil means "use
	// NewDefaultChunker(DefaultChunkConfig())".
	Chunker Chunker
	// Embedder enables vector indexing and semantic search; nil
	// disables vector lanes.
	Embedder Embedder
	// EmbedSig is stamped onto every DerivedSig produced while this
	// Service runs. Embedder doesn't expose a model identifier, so
	// callers (typically the Factory) supply one. Empty string means
	// "use the embedder's Go type name", which is good enough for
	// freshness checks within a single binary but not stable across
	// processes — production wiring should set it explicitly.
	EmbedSig string
	// Now overrides the clock; nil means time.Now (unit-test hook).
	Now func() time.Time
}

ServiceOptions configures a Service. Nil-friendly: every field is optional and falls back to a sensible default.

type SimpleTokenizer ¶

type SimpleTokenizer = textsearch.SimpleTokenizer

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type SourceDocument ¶ added in v0.2.1

type SourceDocument struct {
	DatasetID string
	Name      string
	Content   string
	Metadata  map[string]string

	// Version is monotonically incremented on every successful Put.
	// Derived data uses it as a freshness key.
	Version   uint64
	UpdatedAt time.Time
}

SourceDocument is the canonical, lossless representation of user input.

It is the single source of truth for a document; every DerivedChunk and DerivedLayer carries a DerivedSig that points back to a particular SourceDocument.Version, so derived data can be detected as stale and recomputed deterministically.

type Tokenizer ¶

type Tokenizer = textsearch.Tokenizer

Type and function aliases re-exported from sdk/textsearch for backward compatibility. Internal code and tests can use these without changing import paths.

type VectorRetriever ¶ added in v0.2.1

type VectorRetriever struct {
	Chunks   ChunkRepo
	Embedder Embedder
}

VectorRetriever queries ChunkRepo with Mode=ModeVector. Embedder is invoked lazily; nil disables the retriever (Recall returns nil).

func NewVectorRetriever(c ChunkRepo, e Embedder) *VectorRetriever

func (r *VectorRetriever) Name() string

func (r *VectorRetriever) Recall(ctx context.Context, q Query) ([]Candidate, error)