chunk

type KindRouterPolicy ¶

type KindRouterPolicy struct {
	Default Policy
	ByKind  map[string]Policy
}

KindRouterPolicy dispatches chunking to a per-Kind Policy, falling back to Default when no specific policy is registered for the record's Kind. Use it when one corpus mixes record kinds with different optimal chunking shapes (e.g., reference docs with LateChunkPolicy, raw notes with MarkdownPolicy).

A nil Default with a Kind miss returns an error rather than silently producing zero sections; the caller almost always wants to know.

func (k KindRouterPolicy) Chunk(ctx context.Context, record corpus.Record) ([]SectionWithLineage, error)

type LateChunkPolicy ¶

type LateChunkPolicy struct {
	ParentMaxTokens    int
	ChildMaxTokens     int
	ChildOverlapTokens int
	MaxSections        int
}

LateChunkPolicy is a reference implementation of the late-chunking pattern from the research signal in #16 (Late Chunking, hierarchical text segmentation). It produces a parent span per heading-aware section, then token-budget-split children that point at the parent.

Concretely, for each heading-aware section the policy emits:

• one parent SectionWithLineage holding the full section body, with ParentIndex == NoParent. Parents are not embedded by the index layer (Q decision in the design spec); they live as storage-only context that ExpandContext can surface on IncludeParent.
• N child SectionWithLineage rows, produced by SplitSection on the parent body with (ChildMaxTokens, ChildOverlapTokens), each carrying ParentIndex pointing at the parent's slice index. Children are embedded and participate in retrieval as usual.

ParentMaxTokens caps the parent span itself: if a heading-aware section exceeds it, that section is split into multiple parents (each with its own children). Zero means "one parent per heading-aware section, regardless of size."

MaxSections caps the total chunk count (parents + leaves combined) per record, mirroring chunk.Options.MaxSections semantics. Zero means no cap.

func (p LateChunkPolicy) Chunk(_ context.Context, record corpus.Record) ([]SectionWithLineage, error)

type MarkdownPolicy ¶

type MarkdownPolicy struct {
	Options Options
}

MarkdownPolicy is the default Policy. It reproduces the pre-1.0 chunking pipeline exactly: heading-aware Markdown sectioning followed by optional token-budget splitting. Every returned section has ParentIndex == NoParent — the policy emits flat chunks only — so a snapshot built with a MarkdownPolicy is byte-equivalent to today's output, with parent_chunk_id NULL on every row.

Plaintext records (BodyFormat corpus.FormatPlaintext) bypass Markdown parsing and become a single section keyed on the record title; MaxTokens still drives token-budget splitting if positive.

func (p MarkdownPolicy) Chunk(_ context.Context, record corpus.Record) ([]SectionWithLineage, error)

type Options ¶

type Options struct {
	// MaxTokens is the approximate maximum number of tokens (words) per
	// section. Sections exceeding this limit are split into sub-sections
	// unless OverlapTokens is an invalid value that disables splitting.
	// Zero disables token-budget splitting.
	MaxTokens int

	// OverlapTokens is the approximate number of tokens to overlap between
	// adjacent sub-sections when a section is split. Zero disables overlap.
	// Values greater than or equal to MaxTokens are treated as invalid and
	// leave oversized sections unsplit.
	OverlapTokens int

	// MaxSections caps the number of heading-aware sections
	// MarkdownWithOptions will emit for a single body. Zero means no
	// cap (backward-compatible default for direct callers); any positive
	// value causes MarkdownWithOptions to return ErrTooManySections when
	// the body would produce more sections than the cap. Stroma's index
	// layer applies a conservative default (see index.DefaultMaxChunkSections)
	// so a pathological body can't DoS the embedder or balloon the snapshot.
	MaxSections int
}

Options controls how Markdown sections are split.

type Policy ¶

type Policy interface {
	Chunk(ctx context.Context, record corpus.Record) ([]SectionWithLineage, error)
}

Policy decides how a record's body becomes chunks. The default MarkdownPolicy reproduces the pre-1.0 chunking pipeline exactly (heading-aware Markdown sectioning followed by optional token-budget splitting), so callers that do not opt in see no behavior change.

A Policy may emit hierarchical chunks (parent + leaves with ParentIndex set) so consumers like Hippocampus and Pituitary can retrieve a small leaf and walk back to a broader parent span via Snapshot.ExpandContext (#16). Substrate-neutrality is preserved by keeping the contract narrow: in goes a corpus.Record, out comes a flat slice of (Section, ParentIndex) pairs. Policies have no awareness of indexing, embeddings, or storage — those concerns remain in the index package.

Implementations must be safe for concurrent use: the index session invokes Chunk on one record at a time today, but library callers that drive chunking themselves (tests, offline pipelines) should be able to fan out across goroutines without hidden shared state. The shipped Policy types (MarkdownPolicy, KindRouterPolicy, LateChunkPolicy) are immutable post-construction and derive all mutable state from the per-call record.

type Section ¶

type Section struct {
	Heading string
	Body    string
}

Section is one heading-aware Markdown chunk.

func Markdown(title, body string) []Section

func MarkdownWithOptions(title, body string, opts Options) ([]Section, error)

func SplitSection(s Section, maxTokens, overlapTokens int) []Section

type SectionWithLineage ¶

type SectionWithLineage struct {
	Section
	ParentIndex int
}

SectionWithLineage decorates a Section with optional parent linkage inside a Policy's returned slice. ParentIndex is either NoParent (root) or the slice index of another SectionWithLineage. Forward references — a chunk pointing at an index later in the slice — are rejected when the index session validates topology, so the resulting FK chain is always acyclic and parents always precede their leaves at insert time.