domain

type Attachment ¶

type Attachment struct {
	MediaType string // e.g. "image/png", "application/pdf"
	Name      string // original filename, for display/citation; optional
	Data      []byte
}

Attachment is raw content handed to a Generator alongside retrieved chunks — an image or document the model reads directly. Unlike ingested content it is ephemeral: never chunked, embedded, or stored. How it is encoded for a provider, and whether the provider accepts it, are the Generator adapter's concern.

func NewAttachment(mediaType, name string, data []byte) (Attachment, error)

type Chunk ¶

type Chunk struct {
	ID         ChunkID
	DocumentID DocumentID
	Seq        int
	Text       string
	// HeadingPath is the document section a chunk came from, joined with " > "
	// (e.g. "Auth > Keys > Rotation"). Set by structure-aware chunkers; empty for
	// formats or strategies without headings. It is provenance for display and
	// inspection, not part of the chunk's identity.
	HeadingPath string
}

Chunk is the unit of retrieval. It belongs to exactly one Document (deleting the Document deletes its Chunks).

func NewChunk(docID DocumentID, seq int, text string) (Chunk, error)

type ChunkHit ¶

type ChunkHit struct {
	Chunk  Chunk
	Score  float64
	Source string // source URI of the chunk's document
	// Collection names the collection the chunk came from. It is empty for
	// single-collection retrieval (where it is implied by the request) and set
	// only for cross-collection results (multi-collection query/ask), so the
	// merged hits stay attributable to their origin.
	Collection  string
	RerankScore *float64
	// Metadata is the chunk's document-level attributes, attached at hydration for
	// display (--json, human output). Empty unless the document carried metadata.
	Metadata Metadata
	// IngestedAt is when the chunk's document was ingested, attached at hydration.
	// It is the recency fallback (see HitTime) when the document carries no date
	// metadata; the zero value means unknown.
	IngestedAt time.Time
}

ChunkHit is a hydrated retrieval result: the chunk itself, its similarity score, and the URI of the source document it came from (provenance for display and citation). RerankScore is set only after a cross-encoder rerank — nil means the hit was never reranked, so it serializes additively (rerank_score omitted) and the original Score is always preserved alongside it.

func CapPerSource(hits []ChunkHit, max int) []ChunkHit

func DecayByRecency(hits []ChunkHit, halfLife time.Duration, now time.Time, k int) []ChunkHit

func SelectMMR(candidates []MMRCandidate, lambda float64, k int) []ChunkHit

type ChunkID ¶

type ChunkID string

ChunkID is deterministic, derived from (DocumentID, Seq), so re-chunking an unchanged document yields identical identities, keeping ingestion idempotent.

func DeriveChunkID(docID DocumentID, seq int) ChunkID

func (id ChunkID) Valid() bool

type ChunkResult ¶

type ChunkResult struct {
	Text string
	// EmbedText is the text to embed when it should differ from the stored Text —
	// e.g. a heading-path context prefix prepended so the embedding captures the
	// chunk's place in the document. Empty means embed Text as-is; either way the
	// stored Text (what citations and inspection show) is the original, never the
	// prefixed form.
	EmbedText string
	// HeadingPath is the document section the chunk came from (e.g.
	// "Auth > Keys > Rotation"), set by structure-aware chunkers; empty otherwise.
	HeadingPath string
}

ChunkResult is one chunk a Chunker produced: the text to store and cite, plus any section metadata. Identity (ChunkID, Seq) is assigned by the ingestor, not the chunker, since only the ingestor knows the owning Document.

func (r ChunkResult) TextToEmbed() string

type Chunker ¶

type Chunker interface {
	Chunk(doc ParsedDoc) ([]ChunkResult, error)
}

Chunker splits a parsed document into ordered chunks. Implementations are deterministic and pure: same input → identical output, no I/O, no map-iteration-order dependence. A token-aware strategy receives a token-count func at construction rather than importing a tokenizer, so stdlib-only domain stays clean.

type ChunkerSpec ¶

type ChunkerSpec struct {
	Strategy      string // "structure" | "fixed"
	Version       int    // strategy algorithm version; bump when output could change for the same input
	Size          int    // target chunk size (tokens for structure, words for fixed)
	Overlap       int    // overlap between adjacent chunks, same unit as Size
	Tokenizer     string // sizing tokenizer identity, e.g. "o200k_base" (structure) or "words" (fixed)
	ContextPrefix bool   // whether a heading-path context line is prepended to the embedded text
}

ChunkerSpec identifies the chunking configuration a Collection is pinned to: the strategy, its algorithm version, the size/overlap parameters, the tokenizer used for sizing, and whether a heading-path context prefix is embedded. Any of these changing alters chunk output (boundaries, count, or embedded text) for the same input, so a Collection records its spec at creation and refuses re-ingestion under a different one — the chunker analogue of space coherence. It is comparable with ==.

func NewChunkerSpec(strategy string, version, size, overlap int, tokenizer string, contextPrefix bool) (ChunkerSpec, error)

func (s ChunkerSpec) IsZero() bool

func (s ChunkerSpec) String() string

func (s ChunkerSpec) Validate() error

type Citation ¶

type Citation struct {
	ChunkID ChunkID
	Source  string // source URI of the chunk's document
	Seq     int    // the chunk's position within that document
	// Collection names the chunk's collection, set only for cross-collection
	// answers (multi-collection ask) so a citation stays attributable to its
	// origin; empty for single-collection answers.
	Collection string
}

Citation references a chunk an answer was grounded in, carrying the provenance needed to display it: the source document's URI and the chunk's position within it.

type Claim ¶ added in v1.0.0

type Claim struct {
	Text        string
	CitedChunks []ChunkID
	Verdict     ClaimVerdict
	Rationale   string
}

Claim is one sentence of an answer with the chunks it cites and its verdict. A cited claim starts with an empty Verdict (the Verifier fills it); an uncited claim is VerdictUncited from segmentation.

func SegmentClaims(answer string, citations []Citation) []Claim

type ClaimVerdict ¶ added in v1.0.0

type ClaimVerdict string

ClaimVerdict is the faithfulness verdict for one claim (sentence) of an answer.

const (
	// VerdictSupported: the claim is entailed by the chunk(s) it cites.
	VerdictSupported ClaimVerdict = "supported"
	// VerdictUnsupported: the claim cites chunks but is not entailed by them.
	VerdictUnsupported ClaimVerdict = "unsupported"
	// VerdictUncited: the claim cites no chunk in the grounding set, so it cannot
	// be verified — flagged rather than trusted.
	VerdictUncited ClaimVerdict = "uncited"
)

type Collection ¶

type Collection struct {
	Name      string
	Space     EmbeddingSpace
	Chunker   ChunkerSpec
	CreatedAt time.Time
	// Sources are the paths ingested into this collection, recorded by add and
	// sync so `lore sync` can replay them without a path argument.
	Sources []string
}

Collection is the aggregate root: a named corpus bound to exactly one EmbeddingSpace, and pinned to exactly one ChunkerSpec, for its entire lifetime.

func NewCollection(name string, space EmbeddingSpace, chunker ChunkerSpec, now time.Time) (*Collection, error)

func (c *Collection) AcceptsChunker(spec ChunkerSpec) error

func (c *Collection) AcceptsSpace(s EmbeddingSpace) error

type Condition ¶ added in v1.0.0

type Condition struct {
	Key   string
	Op    Operator
	Value string
}

Condition is one metadata comparison: <key> <op> <value>.

type ContentHash ¶

type ContentHash string

ContentHash is the SHA-256 of document content, hex-encoded. Hash equality is what makes ingestion idempotent.

func HashContent(content []byte) ContentHash

type CorpusSnapshot ¶ added in v1.0.0

type CorpusSnapshot struct {
	// Digest is the hex sha256 over the corpus's (SourceURI, Hash) pairs.
	Digest ContentHash
	// LastIngest is the maximum Document.IngestedAt, or the zero time when the
	// corpus is empty.
	LastIngest time.Time
}

CorpusSnapshot is a content-derived identity for a collection's document set. Its Digest changes if and only if the set of (source, content-hash) pairs changes — any addition, removal, or edit — and is therefore the field a provenance trail should stamp, unlike Collection.CreatedAt, which is a birth/ rebuild marker that a sync leaves untouched. LastIngest names when the most recently (re)ingested document entered the corpus.

The snapshot is space- and chunker-independent by design: it identifies what was ingested, not how it was embedded, mirroring CollectionDiff.

func SnapshotOf(docs []*Document) CorpusSnapshot

type Document ¶

type Document struct {
	ID         DocumentID
	Collection string
	SourceURI  string
	Hash       ContentHash
	IngestedAt time.Time
	// Fingerprint is a cheap source-side signature (e.g. size + sampled-content
	// hash) used to skip re-reading unchanged files before extraction. Unlike
	// Hash it is a heuristic, not an identity; empty means "unknown".
	Fingerprint string
	// Metadata holds the document's structured attributes (user-supplied --meta
	// pairs, extracted front-matter, path/mtime-derived fields). It is provenance
	// for display and the substrate for --where filtering; it is not part of the
	// document's identity (Hash) and may change without re-chunking.
	Metadata Metadata
}

Document is source content with identity. Whether its content *changed* is decided by Hash (hash equality => ingestion is a no-op).

func NewDocument(collection, sourceURI string, hash ContentHash, now time.Time) (*Document, error)

func (d *Document) Unchanged(hash ContentHash) bool

type DocumentID ¶

type DocumentID string

DocumentID is deterministic, derived from (collection, source URI): re-ingesting the same source addresses the same identity, which is what makes `lore add` an upsert rather than an append.

func DeriveDocumentID(collection, sourceURI string) DocumentID

type EmbeddingSpace ¶

type EmbeddingSpace struct {
	Model      string
	Dimensions int
}

EmbeddingSpace identifies the vector space a Collection is bound to: the embedding model plus its output dimensionality. Two vectors are comparable only if they come from the same space.

func NewEmbeddingSpace(model string, dimensions int) (EmbeddingSpace, error)

func (s EmbeddingSpace) Equal(other EmbeddingSpace) bool

func (s EmbeddingSpace) IsZero() bool

func (s EmbeddingSpace) String() string

type FixedChunker ¶

type FixedChunker struct {
	// contains filtered or unexported fields
}

FixedChunker emits fixed-size, overlapping windows measured in whitespace words — the legacy chunking behavior, preserved verbatim as the `fixed` strategy and the default fallback. Word sizing is dependency-free and deterministic; the structure-aware strategies use real token counts instead. Chunk text is whitespace-normalized as a side effect.

Construct with NewFixedChunker; the zero value is not usable.

func NewFixedChunker(size, overlap int) (FixedChunker, error)

func (c FixedChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

func (c FixedChunker) Split(text string) []string

type MMRCandidate ¶ added in v1.0.0

type MMRCandidate struct {
	Hit    ChunkHit
	Vector []float32
}

MMRCandidate is a hit paired with its embedding vector, the input to Maximal Marginal Relevance selection. The hit's Score is used as its relevance to the query; the vector is used for the redundancy penalty against already-selected hits.

type MarkdownChunker ¶

type MarkdownChunker struct {
	// contains filtered or unexported fields
}

MarkdownChunker splits markdown into token-sized chunks that respect the document's heading hierarchy: each chunk is a section (the content under a heading), carrying the full heading path as metadata. It never breaks inside a fenced code block; oversized sections split at paragraph (then sentence, then word) boundaries, and undersized adjacent sections merge up to the target size. Deterministic and pure.

Construct with NewMarkdownChunker; the zero value is not usable.

func NewMarkdownChunker(size, overlap int, contextPrefix bool, countTokens func(string) int) (MarkdownChunker, error)

func (c MarkdownChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

type Metadata ¶ added in v1.0.0

type Metadata map[string]string

Metadata is a small, flat set of structured attributes attached to a Document and carried on its chunks' index entries for retrieval filtering. Values are stored canonically as strings; the typed comparison semantics (numeric, date, glob, tag membership) live in Predicate, not in the storage. Keys are caller-defined: user-supplied --meta pairs, extracted front-matter, or path/mtime-derived fields.

func ParseFrontMatter(text string) (Metadata, string)

func (m Metadata) Clone() Metadata

type Operator ¶ added in v1.0.0

type Operator string

Operator is the comparison in a --where Condition.

const (
	OpEqual        Operator = "="
	OpNotEqual     Operator = "!="
	OpLess         Operator = "<"
	OpLessEqual    Operator = "<="
	OpGreater      Operator = ">"
	OpGreaterEqual Operator = ">="
	OpMatch        Operator = "~"
)

type ParsedDoc ¶

type ParsedDoc struct {
	Text        string
	ContentType string
	SourceURI   string
}

ParsedDoc is the input to a Chunker: the extracted text of one document plus the context a strategy may need. ContentType lets the Registry dispatch to a per-format strategy; SourceURI is carried for diagnostics.

type Predicate ¶ added in v1.0.0

type Predicate struct {
	Conditions []Condition
}

Predicate is a conjunction (AND) of metadata Conditions — the parsed form of the repeatable --where flag. The zero Predicate has no conditions and matches every document, so an absent --where filters nothing.

func ParseWhere(clauses []string) (Predicate, error)

func (p Predicate) IsZero() bool

func (p Predicate) Match(md Metadata) bool

type RankedID ¶ added in v1.0.0

type RankedID struct {
	ID    ChunkID
	Score float64
}

RankedID is a chunk identity with its fusion score, best (highest) first.

func FuseRRF(k int, lists ...[]ChunkID) []RankedID

type Registry ¶

type Registry struct {
	// contains filtered or unexported fields
}

Registry selects a Chunker by content type, falling back to a default. It is how per-format strategies (markdown, plain text, ...) plug in without the ingestor knowing which is which; a future code-aware strategy is just another registered entry (though a tree-sitter strategy could not be pure-domain). It also carries the ChunkerSpec the active configuration corresponds to, so the use cases can pin it on new collections and refuse re-ingest under a different one. Construct with NewRegistry; the zero value is not usable.

func NewRegistry(spec ChunkerSpec, def Chunker, byType map[string]Chunker) (Registry, error)

func (r Registry) Chunk(doc ParsedDoc) ([]ChunkResult, error)

func (r Registry) Spec() ChunkerSpec

type SheetChunker ¶ added in v1.1.0

type SheetChunker struct {
	// contains filtered or unexported fields
}

SheetChunker splits tabular text — one record per line — into token-sized chunks that stay readable on their own. A table's header row is the only thing naming its columns, and a sheet's tab name the only thing naming the table, so both are repeated at the top of every chunk cut from that table. Without this a chunk from the middle of a workbook retrieves as bare cells whose columns the reader (and the model) cannot recover.

Unlike MarkdownChunker's context prefix, the repetition goes into the stored Text, not just EmbedText: the header is genuinely absent from the interior of a table, so a reader of the cited chunk needs it as much as the embedding does.

Rows are never split across chunks and never duplicated between them; the header is the shared context instead of an overlap window. The first line of each table is taken to be its header, which is wrong for a workbook with banner rows above the real header — a limitation, not a heuristic to grow.

Deterministic and pure. Construct with NewSheetChunker; the zero value is not usable.

func NewSheetChunker(size, overlap int, countTokens func(string) int) (SheetChunker, error)

func (c SheetChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

type TextChunker ¶

type TextChunker struct {
	// contains filtered or unexported fields
}

TextChunker splits plain text into token-sized chunks at paragraph (then sentence, then word) boundaries, never mid-sentence where avoidable. It has no heading awareness: it is the structure strategy's chunker for plain text and the default fallback for formats without their own structure-aware chunker (docx paragraphs; best-effort pdf/xlsx text). Deterministic and pure.

Construct with NewTextChunker; the zero value is not usable.

func NewTextChunker(size, overlap int, countTokens func(string) int) (TextChunker, error)

func (c TextChunker) Chunk(doc ParsedDoc) ([]ChunkResult, error)

type VectorMatch ¶

type VectorMatch struct {
	ChunkID ChunkID
	Score   float64
}

VectorMatch is what a VectorIndex returns: an identity and a similarity score (higher is more similar), not yet hydrated with chunk content.