chunk

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2026 License: MIT Imports: 2 Imported by: 0

Documentation

Overview

Package chunk defines lore's chunk data model and the structure-aware chunker, decided in https://github.com/cheetahbyte/lore/issues/7 and https://github.com/cheetahbyte/lore/issues/8.

Index

Constants

View Source
const DefaultTargetSize = 1500

DefaultTargetSize is the merge target used by Split when the caller doesn't override it, per the char-based sizing decided in issue #7.

Variables

This section is empty.

Functions

This section is empty.

Types

type Chunk

type Chunk struct {
	ID          string
	TenantID    string // always "local" outside a hosted deployment; see issue #10
	LibraryID   string // typed-prefix identity, e.g. "github:owner/repo"
	Version     string // "" for sources with no version concept
	DocURL      string
	SectionPath string // heading trail, e.g. "Installation > Configuration"
	Ordinal     int    // position within the document, for reassembly
	Content     string
	Embedding   []float32 // nil unless vector search is enabled, per issue #7
	ContentHash string
	FetchedAt   time.Time
}

Chunk is one row of indexed, searchable content.

type Page

type Page struct {
	URL         string
	Path        string // e.g. file path within a repo; "" where not applicable
	Content     string
	ContentType string // "markdown" | "html" | "text"
}

Page is one normalized, fetched document ready to be chunked.

type Piece

type Piece struct {
	SectionPath string
	Content     string
}

Piece is one structure-aware chunk of markdown, before the caller fills in the rest of the Chunk fields (library id, version, tenant, etc).

func Split

func Split(markdown string, targetSize int) []Piece

Split performs the chunking strategy decided in issue #7: split first at natural boundaries (headings, code fences, paragraph breaks), then greedily merge adjacent pieces up to targetSize — never splitting inside a fenced code block, even if the fence itself exceeds targetSize.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL