domain

type Attachment ¶

type Attachment struct {
	MediaType string // e.g. "image/png", "application/pdf"
	Name      string // original filename, for display/citation; optional
	Data      []byte
}

Attachment is raw content handed to a Generator alongside retrieved chunks — an image or document the model reads directly. Unlike ingested content it is ephemeral: never chunked, embedded, or stored. How it is encoded for a provider, and whether the provider accepts it, are the Generator adapter's concern.

func NewAttachment(mediaType, name string, data []byte) (Attachment, error)

type Chunk ¶

type Chunk struct {
	ID         ChunkID
	DocumentID DocumentID
	Seq        int
	Text       string
	// HeadingPath is the document section a chunk came from, joined with " > "
	// (e.g. "Auth > Keys > Rotation"). Set by structure-aware chunkers; empty for
	// formats or strategies without headings. It is provenance for display and
	// inspection, not part of the chunk's identity.
	HeadingPath string
}

Chunk is the unit of retrieval. It belongs to exactly one Document (deleting the Document deletes its Chunks).

func NewChunk(docID DocumentID, seq int, text string) (Chunk, error)

type ChunkHit ¶

type ChunkHit struct {
	Chunk  Chunk
	Score  float64
	Source string // source URI of the chunk's document
	// Collection names the collection the chunk came from. It is empty for
	// single-collection retrieval (where it is implied by the request) and set
	// only for cross-collection results (multi-collection query/ask), so the
	// merged hits stay attributable to their origin.
	Collection  string
	RerankScore *float64
}

ChunkHit is a hydrated retrieval result: the chunk itself, its similarity score, and the URI of the source document it came from (provenance for display and citation). RerankScore is set only after a cross-encoder rerank — nil means the hit was never reranked, so it serializes additively (rerank_score omitted) and the original Score is always preserved alongside it.

type ChunkID ¶

type ChunkID string

ChunkID is deterministic, derived from (DocumentID, Seq), so re-chunking an unchanged document yields identical identities, keeping ingestion idempotent.

func DeriveChunkID(docID DocumentID, seq int) ChunkID

func (id ChunkID) Valid() bool

type ChunkResult ¶

type ChunkResult struct {
	Text string
	// EmbedText is the text to embed when it should differ from the stored Text —
	// e.g. a heading-path context prefix prepended so the embedding captures the
	// chunk's place in the document. Empty means embed Text as-is; either way the
	// stored Text (what citations and inspection show) is the original, never the
	// prefixed form.
	EmbedText string
	// HeadingPath is the document section the chunk came from (e.g.
	// "Auth > Keys > Rotation"), set by structure-aware chunkers; empty otherwise.
	HeadingPath string
}

ChunkResult is one chunk a Chunker produced: the text to store and cite, plus any section metadata. Identity (ChunkID, Seq) is assigned by the ingestor, not the chunker, since only the ingestor knows the owning Document.

func (r ChunkResult) TextToEmbed() string

type Chunker ¶

type Chunker interface {
	Chunk(doc ParsedDoc) ([]ChunkResult, error)
}

Chunker splits a parsed document into ordered chunks. Implementations are deterministic and pure: same input → identical output, no I/O, no map-iteration-order dependence. A token-aware strategy receives a token-count func at construction rather than importing a tokenizer, so stdlib-only domain stays clean.

type ChunkerSpec ¶

type ChunkerSpec struct {
	Strategy      string // "structure" | "fixed"
	Version       int    // strategy algorithm version; bump when output could change for the same input
	Size          int    // target chunk size (tokens for structure, words for fixed)
	Overlap       int    // overlap between adjacent chunks, same unit as Size
	Tokenizer     string // sizing tokenizer identity, e.g. "o200k_base" (structure) or "words" (fixed)
	ContextPrefix bool   // whether a heading-path context line is prepended to the embedded text
}

ChunkerSpec identifies the chunking configuration a Collection is pinned to: the strategy, its algorithm version, the size/overlap parameters, the tokenizer used for sizing, and whether a heading-path context prefix is embedded. Any of these changing alters chunk output (boundaries, count, or embedded text) for the same input, so a Collection records its spec at creation and refuses re-ingestion under a different one — the chunker analogue of space coherence. It is comparable with ==.

func NewChunkerSpec(strategy string, version, size, overlap int, tokenizer string, contextPrefix bool) (ChunkerSpec, error)

func (s ChunkerSpec) IsZero() bool

func (s ChunkerSpec) String() string

func (s ChunkerSpec) Validate() error

type Citation ¶

type Citation struct {
	ChunkID ChunkID
	Source  string // source URI of the chunk's document
	Seq     int    // the chunk's position within that document
	// Collection names the chunk's collection, set only for cross-collection
	// answers (multi-collection ask) so a citation stays attributable to its
	// origin; empty for single-collection answers.
	Collection string
}

Citation references a chunk an answer was grounded in, carrying the provenance needed to display it: the source document's URI and the chunk's position within it.

type Collection ¶

type Collection struct {
	Name      string
	Space     EmbeddingSpace
	Chunker   ChunkerSpec
	CreatedAt time.Time
	// Sources are the paths ingested into this collection, recorded by add and
	// sync so `lore sync` can replay them without a path argument.
	Sources []string
}

Collection is the aggregate root: a named corpus bound to exactly one EmbeddingSpace, and pinned to exactly one ChunkerSpec, for its entire lifetime.

func NewCollection(name string, space EmbeddingSpace, chunker ChunkerSpec, now time.Time) (*Collection, error)

func (c *Collection) AcceptsChunker(spec ChunkerSpec) error

func (c *Collection) AcceptsSpace(s EmbeddingSpace) error

type ContentHash ¶

type ContentHash string

ContentHash is the SHA-256 of document content, hex-encoded. Hash equality is what makes ingestion idempotent.

func HashContent(content []byte) ContentHash

type Document ¶

type Document struct {
	ID         DocumentID
	Collection string
	SourceURI  string
	Hash       ContentHash
	IngestedAt time.Time
	// Fingerprint is a cheap source-side signature (e.g. size + sampled-content
	// hash) used to skip re-reading unchanged files before extraction. Unlike
	// Hash it is a heuristic, not an identity; empty means "unknown".
	Fingerprint string
}

Document is source content with identity. Whether its content *changed* is decided by Hash (hash equality => ingestion is a no-op).

func NewDocument(collection, sourceURI string, hash ContentHash, now time.Time) (*Document, error)

func (d *Document) Unchanged(hash ContentHash) bool

type DocumentID ¶

type DocumentID string

DocumentID is deterministic, derived from (collection, source URI): re-ingesting the same source addresses the same identity, which is what makes `lore add` an upsert rather than an append.

func DeriveDocumentID(collection, sourceURI string) DocumentID

type EmbeddingSpace ¶

type EmbeddingSpace struct {
	Model      string
	Dimensions int
}

EmbeddingSpace identifies the vector space a Collection is bound to: the embedding model plus its output dimensionality. Two vectors are comparable only if they come from the same space.

func NewEmbeddingSpace(model string, dimensions int) (EmbeddingSpace, error)

func (s EmbeddingSpace) Equal(other EmbeddingSpace) bool

func (s EmbeddingSpace) IsZero() bool

func (s EmbeddingSpace) String() string

type FixedChunker ¶

type FixedChunker struct {
	// contains filtered or unexported fields
}

FixedChunker emits fixed-size, overlapping windows measured in whitespace words — the legacy chunking behavior, preserved verbatim as the `fixed` strategy and the default fallback. Word sizing is dependency-free and deterministic; the structure-aware strategies use real token counts instead. Chunk text is whitespace-normalized as a side effect.

Construct with NewFixedChunker; the zero value is not usable.

func NewFixedChunker(size, overlap int) (FixedChunker, error)

func (c FixedChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

func (c FixedChunker) Split(text string) []string

type MarkdownChunker ¶

type MarkdownChunker struct {
	// contains filtered or unexported fields
}

MarkdownChunker splits markdown into token-sized chunks that respect the document's heading hierarchy: each chunk is a section (the content under a heading), carrying the full heading path as metadata. It never breaks inside a fenced code block; oversized sections split at paragraph (then sentence, then word) boundaries, and undersized adjacent sections merge up to the target size. Deterministic and pure.

Construct with NewMarkdownChunker; the zero value is not usable.

func NewMarkdownChunker(size, overlap int, contextPrefix bool, countTokens func(string) int) (MarkdownChunker, error)

func (c MarkdownChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

type ParsedDoc ¶

type ParsedDoc struct {
	Text        string
	ContentType string
	SourceURI   string
}

ParsedDoc is the input to a Chunker: the extracted text of one document plus the context a strategy may need. ContentType lets the Registry dispatch to a per-format strategy; SourceURI is carried for diagnostics.

type Registry ¶

type Registry struct {
	// contains filtered or unexported fields
}

Registry selects a Chunker by content type, falling back to a default. It is how per-format strategies (markdown, plain text, ...) plug in without the ingestor knowing which is which; a future code-aware strategy is just another registered entry (though a tree-sitter strategy could not be pure-domain). It also carries the ChunkerSpec the active configuration corresponds to, so the use cases can pin it on new collections and refuse re-ingest under a different one. Construct with NewRegistry; the zero value is not usable.

func NewRegistry(spec ChunkerSpec, def Chunker, byType map[string]Chunker) (Registry, error)

func (r Registry) Chunk(doc ParsedDoc) ([]ChunkResult, error)

func (r Registry) Spec() ChunkerSpec

type TextChunker ¶

type TextChunker struct {
	// contains filtered or unexported fields
}

TextChunker splits plain text into token-sized chunks at paragraph (then sentence, then word) boundaries, never mid-sentence where avoidable. It has no heading awareness: it is the structure strategy's chunker for plain text and the default fallback for formats without their own structure-aware chunker (docx paragraphs; best-effort pdf/xlsx text). Deterministic and pure.

Construct with NewTextChunker; the zero value is not usable.

func NewTextChunker(size, overlap int, countTokens func(string) int) (TextChunker, error)

func (c TextChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

type VectorMatch ¶

type VectorMatch struct {
	ChunkID ChunkID
	Score   float64
}

VectorMatch is what a VectorIndex returns: an identity and a similarity score (higher is more similar), not yet hydrated with chunk content.