knowledge

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 8, 2026 License: Apache-2.0 Imports: 34 Imported by: 0

Documentation

Overview

Package knowledge keeps CSF's search index fed and measured: the repository at the head of main, the harness state (its JSON key names, never their values), each run's conversation (user and assistant text, never tool output) and the repository's tickets and pull requests, ingested through the knowledge service's own IngestDocument so the store stays the evidence and the index its projection. A source already ingested at its revision is skipped, a repository file deleted since the last indexed commit is forgotten, and after every pass the held-out lookups (lookups.json) are answered twice, in fused and in reranked order, giving hit@k.

The owned containers the index and its reranker run in are here too (stack.go), and the embedder that reaches the LABELER's Ollama server (embed.go).

It is a service in the house sense: no process, no flags, no environment, no listener; the harness binary composes it and runs its trigger.

Index

Constants

View Source 
const (
	EmbeddingModel     = "qwen3-embedding:0.6b"
	EmbeddingDimension = 1024
)

The embedding model: qwen3-embedding:0.6b on the LABELER's Ollama server, 1024 dimensions (measured 2026-10-05), and the instruction its queries carry, as the model's card asks of retrieval queries.

View Source 
const (
	// SearchRecordFile is the index's record under the state directory.
	SearchRecordFile = "search.json"
	// SearchImage is the pinned OpenSearch image, by digest.
	SearchImage = "opensearchproject/opensearch:3.9.0@sha256:adfa61f85025d06b4aeb562e7e74fde7e31c437039c93c3862c17e9acebd6c7c"
	// SearchIndex is the index the knowledge projection writes its chunks to.
	SearchIndex = "csf-knowledge"
)

The search index: OpenSearch, one node, on loopback only.

View Source 
const (
	// RerankerRecordFile is the reranker's record under the state directory.
	RerankerRecordFile = "reranker.json"
	// RerankerImage is the pinned llama.cpp server image (CUDA 13, which the
	// host's RTX 5070 needs), by digest.
	RerankerImage = "ghcr.io/ggml-org/llama.cpp:server-cuda13-b11277@sha256:f9f7c9f689cf6dba993075b74697253c46c639f7c5956a3421fcabf7e8aef715"
	// RerankerModel is the cross-encoder: bge-reranker-v2-m3, 568M parameters,
	// quantized to 8 bits (636 MB), pinned to a commit of its GGUF repository.
	RerankerModel = "bge-reranker-v2-m3-Q8_0"
)

The reranker: a llama.cpp server with a cross-encoder, on the GPU.

View Source 
const TriggerRefresh = "knowledge.refresh"

TriggerRefresh is the cron trigger that feeds and measures the index.

Variables

View Source 
var (
	// ErrInvalidOption reports a nil option or one the feeder cannot use.
	ErrInvalidOption = errors.New("knowledge: invalid option")
	// ErrMissingCapability reports a feeder built without the knowledge
	// service or the index.
	ErrMissingCapability = errors.New("knowledge: the knowledge service and the index are required")
)
View Source 
var ErrGPUBusy = errors.New("knowledge: another model holds the GPU; indexing yields")

ErrGPUBusy reports a document embedding refused because another model holds the GPU: indexing yields, and the projection retries it later.

Functions

func Endpoint 

func Endpoint(location docker.OwnedLocation) string

Endpoint is the URL a record's published port answers on.

func NewOllamaEmbedder 

func NewOllamaEmbedder(model IEmbeddingModel) csf.Embedder

NewOllamaEmbedder embeds through model. Queries carry the model's retrieval instruction and always run, so a search answers; documents yield the GPU while the server holds another model, which is the labeler at work. A server that cannot say what it holds does not stop indexing.

func NewOwnedReranker 

func NewOwnedReranker(state string, options ...docker.OwnedServiceOption) (*docker.OwnedService[RerankerSettings], error)

NewOwnedReranker is the reranker the state directory owns. It holds the GPU while idle, so it is labelled a model server: GPU consumers that yield to others do not yield to it.

func NewOwnedSearch 

func NewOwnedSearch(state string, options ...docker.OwnedServiceOption) (*docker.OwnedService[SearchSettings], error)

NewOwnedSearch is the OpenSearch node the state directory owns.

Types

type Evaluation 

type Evaluation struct {
	Fused    map[int]float64
	Reranked map[int]float64
	Queries  int
}

Evaluation is the index's hit@k over the held-out lookups, once in the fused order and once in the reranked order. hit@k is the share of lookups whose answer appears within the first k hits; an unanswered lookup is a miss at every k.

func Evaluate 

func Evaluate(ctx context.Context, index IIndex, lookups []Lookup) (Evaluation, error)

Evaluate answers every lookup in both orders and returns the hit@k curves. A lookup that Compare fails to answer stops the pass: the measurement would otherwise read too low for a reason no fix to the index can address.

type Feeder 

type Feeder struct {
	// contains filtered or unexported fields
}

Feeder feeds the index from its sources and measures it. One trigger runs its passes, so a pass never overlaps another.

func NewFeeder 

func NewFeeder(options ...FeederOption) (*Feeder, error)

NewFeeder builds the feeder from its options; the sources it reads are the ones granted.

func (*Feeder) Measured 

func (feeder *Feeder) Measured() *Measured

Measured is the last pass's measurements; nil before the first.

func (*Feeder) Refresh 

func (feeder *Feeder) Refresh(ctx context.Context) (*Measured, error)

Refresh runs one pass: every source read, every new revision ingested, every source gone forgotten, then the lookups answered. A failing source is logged and counted, and the others still run.

func (*Feeder) Trigger 

func (feeder *Feeder) Trigger() cronservice.Option

Trigger is the cron trigger that runs a pass every refreshInterval.

type FeederOption 

type FeederOption func(feeder *Feeder) error

FeederOption configures a Feeder.

func WithClock 

func WithClock(now func() time.Time) FeederOption

WithClock replaces the wall clock the documents' retrieval times read.

func WithGitHub 

func WithGitHub(slug string) FeederOption

WithGitHub indexes the tickets and pull requests of the repository slug.

func WithHarnessState 

func WithHarnessState(root string, files iofs.IFiles) FeederOption

WithHarnessState indexes the harness state directory at root, read through files: its JSON files' key names and each run's conversation.

func WithIndex 

func WithIndex(index IIndex) FeederOption

WithIndex grants the search projection. Required.

func WithKnowledge 

func WithKnowledge(knowledge IKnowledge) FeederOption

WithKnowledge grants the knowledge service documents are ingested through. Required.

func WithLauncher 

func WithLauncher(launcher proc.ILauncher) FeederOption

WithLauncher grants the programs the repository and GitHub sources run: git and gh.

func WithLogger 

func WithLogger(logger *slog.Logger) FeederOption

WithLogger receives one line per pass and per failed source.

func WithRepository 

func WithRepository(directory string, slug string, load func() (string, error), save func(commit string) error) FeederOption

WithRepository indexes main of the git checkout at directory, whose origin is the repository slug (owner/name), and remembers the commit it indexed through load and save, so a file deleted while the host was down is still forgotten.

type IEmbeddingModel 

type IEmbeddingModel interface {
	Model() string
	Embed(ctx context.Context, inputs []string, stay time.Duration) ([][]float32, error)
	Loaded(ctx context.Context) ([]ollama.LoadedModel, error)
}

IEmbeddingModel is the part of the Ollama brain an embedder uses.

type IIndex 

type IIndex interface {
	Forget(ctx context.Context, sourceID string) error
	Compare(ctx context.Context, request *pb.SearchRequest) (*pb.SearchResult, *pb.SearchResult, error)
}

IIndex is the part of the search projection the feeder uses: forgetting a source that is gone, and answering a lookup in both orders.

type IKnowledge 

type IKnowledge interface {
	IngestDocument(ctx context.Context, request *pb.IngestDocumentRequest) (*pb.IngestDocumentResponse, error)
	GetDocument(ctx context.Context, request *pb.GetDocumentRequest) (*pb.GetDocumentResponse, error)
}

IKnowledge is the part of the knowledge service the feeder uses: the ingest path every document takes, and the store it asks whether a revision is already in.

type Lookup 

type Lookup struct {
	Query string   `json:"query"`
	Paths []string `json:"paths"`
}

Lookup is one held-out lookup: a query and the repository paths that answer it, matched by prefix.

func Lookups 

func Lookups() []Lookup

Lookups is the held-out lookups of lookups.json.

type Measured 

type Measured struct {
	At         time.Time
	Ingested   map[string]int64
	Forgotten  int64
	Failed     map[string]int64
	Evaluation Evaluation
}

Measured is what the last pass did and the index's hit@k after it.

type RerankerSettings 

type RerankerSettings struct {
	Model string `json:"model"`
}

RerankerSettings is the reranker's own part of its record.

type SearchSettings 

type SearchSettings struct {
	Index string `json:"index"`
}

SearchSettings is the index's own part of its record.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.