Documentation
¶
Overview ¶
Package textsplitter chunks markdown documents into metadata-bearing units suitable for embedding. It is vendored from github.com/tmc/langchaingo/textsplitter (MIT, see LICENSE-langchaingo) and adapted to the SDD chunker contract — see NOTICE.md for the full list of divergences from upstream.
Two splitters are exposed:
MarkdownTextSplitter: parses CommonMark, splits on h1–h6 headings, recursively re-splits oversized sections via a second splitter, and emits []Chunk with breadcrumb (heading-chain array), depth, and IsSummary / IsAttachment flags. The TextSplitter string interface is not satisfied; use SplitChunks for the chunk-bearing output.
RecursiveCharacter: splits on a separator hierarchy with overlap control. Used internally as the second splitter for oversized sections; satisfies the original TextSplitter string interface.
The high-level entry point is Splitter.Split, which strips YAML frontmatter, prepends an "Entry: <summary>" + "Breadcrumb: <chain>" preamble to each chunk's text per d-tac-jvd, and tags chunks with attachment provenance.
Index ¶
- Constants
- type Chunk
- type MarkdownTextSplitter
- type Option
- func WithChunkOverlap(chunkOverlap int) Option
- func WithChunkSize(chunkSize int) Option
- func WithCodeBlocks(renderCode bool) Option
- func WithHeadingHierarchy(trackHeadingHierarchy bool) Option
- func WithJoinTableRows(join bool) Option
- func WithKeepSeparator(keepSeparator bool) Option
- func WithLenFunc(lenFunc func(string) int) Option
- func WithReferenceLinks(referenceLinks bool) Option
- func WithSecondSplitter(secondSplitter TextSplitter) Option
- func WithSeparators(separators []string) Option
- type Options
- type RecursiveCharacter
- type SplitInput
- type SplitOutput
- type Splitter
- type TextSplitter
Constants ¶
const ( DefaultChunkSize = 3200 DefaultChunkOverlap = 320 )
Default rune-count chunk size and overlap for markdown body splitting. Calibrated to roughly the d-tac-lqr "~800 tokens hard cap, ~10% overlap" targets (1 token ≈ 4 runes English prose).
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chunk ¶
type Chunk struct {
Text string
Breadcrumb []string
Depth int
IsSummary bool
IsAttachment bool
SourceAttachmentPath string
// Body is the section text without the Entry/Breadcrumb preamble. Useful
// for citation snippets (the user-facing surface) where the preamble
// would be noise.
Body string
}
Chunk is one unit of text destined for embedding, carrying the metadata needed to filter, score, and cite it. The Text field already includes the `Entry: <summary>\nBreadcrumb: <chain>\n\n<body>` preamble (per d-tac-jvd) — embedders consume Text directly.
Breadcrumb is the heading-chain in array form (e.g. ["Approach", "Storage"]) without `#` markers. Depth is the deepest heading level the chunk sits under (0 means pre-heading body).
IsSummary, IsAttachment, and SourceAttachmentPath are provenance flags the indexer uses for filtering and citation rendering.
type MarkdownTextSplitter ¶
type MarkdownTextSplitter struct {
ChunkSize int
ChunkOverlap int
SecondSplitter TextSplitter
CodeBlocks bool
ReferenceLinks bool
JoinTableRows bool
LenFunc func(string) int
}
MarkdownTextSplitter parses CommonMark markdown via the gitlab.com/golang-commonmark/markdown tokenizer and splits the input on h1–h6 heading boundaries, recursively re-splitting oversized sections via SecondSplitter. Output chunks carry breadcrumb metadata (the heading-chain as []string) and depth (the deepest heading level encountered). The caller is responsible for composing any Entry/Breadcrumb preamble — see splitter.go.
func NewMarkdownTextSplitter ¶
func NewMarkdownTextSplitter(opts ...Option) *MarkdownTextSplitter
NewMarkdownTextSplitter creates a new Markdown text splitter.
func (MarkdownTextSplitter) SplitChunks ¶
func (sp MarkdownTextSplitter) SplitChunks(text string) ([]rawChunk, error)
SplitChunks tokenizes the input markdown and emits rawChunks split on h1–h6 heading boundaries.
type Option ¶
type Option func(*Options)
Option is a function that can be used to set options for a text splitter.
func WithChunkOverlap ¶
WithChunkOverlap sets the chunk overlap for a text splitter.
func WithChunkSize ¶
WithChunkSize sets the chunk size for a text splitter.
func WithCodeBlocks ¶
WithCodeBlocks sets whether indented and fenced codeblocks should be included in the output. SDD default is true.
func WithHeadingHierarchy ¶
WithHeadingHierarchy is retained for upstream API parity but unused by the SDD entry point: breadcrumb prepending is handled by the high-level splitter rather than by the markdown splitter inlining heading-stack text into chunks.
func WithJoinTableRows ¶
WithJoinTableRows sets whether tables should be split by row or not.
func WithKeepSeparator ¶
WithKeepSeparator sets whether the separators should be kept in the resulting split text or not.
func WithLenFunc ¶
WithLenFunc sets the lenfunc for a text splitter.
func WithReferenceLinks ¶
WithReferenceLinks sets whether reference links (i.e. `[text][label]`) should be patched with the url and title from their definition.
func WithSecondSplitter ¶
func WithSecondSplitter(secondSplitter TextSplitter) Option
WithSecondSplitter sets the second splitter for a text splitter.
func WithSeparators ¶
WithSeparators sets the separators for a text splitter.
type Options ¶
type Options struct {
ChunkSize int
ChunkOverlap int
Separators []string
KeepSeparator bool
LenFunc func(string) int
SecondSplitter TextSplitter
CodeBlocks bool
ReferenceLinks bool
KeepHeadingHierarchy bool // retained for upstream API parity; unused by the SDD entry point
JoinTableRows bool
}
Options is a struct that contains options for a text splitter.
func DefaultOptions ¶
func DefaultOptions() Options
DefaultOptions returns the default options for all text splitters.
type RecursiveCharacter ¶
type RecursiveCharacter struct {
Separators []string
ChunkSize int
ChunkOverlap int
LenFunc func(string) int
KeepSeparator bool
}
RecursiveCharacter is a text splitter that will split texts recursively by different characters.
func NewRecursiveCharacter ¶
func NewRecursiveCharacter(opts ...Option) RecursiveCharacter
NewRecursiveCharacter creates a new recursive character splitter with default values. By default the separators are "\n\n", "\n", " " and "".
type SplitInput ¶
type SplitInput struct {
// Markdown is the document text. May start with a YAML frontmatter block
// delimited by `---` lines; if so it is stripped before tokenization
// and surfaced as Frontmatter on the result.
Markdown string
// EntrySummary is prepended to each chunk's Text as `Entry: <first
// sentence>` so embeddings carry entry-level identity (per d-tac-jvd).
// Only the first sentence is used; pass the entry's stored summary
// verbatim and the splitter handles trimming.
EntrySummary string
// IsAttachment marks this input as an attachment. Each chunk's
// IsAttachment flag is set accordingly and SourceAttachmentPath is
// recorded on each chunk.
IsAttachment bool
// SourceAttachmentPath is the relative path of the source attachment
// (e.g. `2026/05/04-235258-d-tac-lqr/design.md`). Recorded on each
// chunk when IsAttachment is true.
SourceAttachmentPath string
}
SplitInput captures everything the splitter needs to produce per-chunk output for a single source document.
type SplitOutput ¶
SplitOutput carries the per-input results: chunks ready to embed plus the stripped frontmatter as a generic map for the caller's filterable metadata layer (frontmatter values are not embedded as text per d-tac-lqr).
type Splitter ¶
type Splitter struct {
// contains filtered or unexported fields
}
Splitter is the high-level entry point for chunking SDD entry bodies and attachments. It strips YAML frontmatter at the package boundary, runs the markdown splitter, and composes the `Entry: <summary>\nBreadcrumb: <chain>` preamble onto each chunk's Text. Construct via NewSplitter.
func NewSplitter ¶
NewSplitter constructs a Splitter with the given markdown splitter options. Defaults match DefaultOptions() (CodeBlocks=true, ChunkSize=DefaultChunkSize, ChunkOverlap=DefaultChunkOverlap).
func (*Splitter) Split ¶
func (s *Splitter) Split(input SplitInput) (SplitOutput, error)
Split chunks input.Markdown. Heading-only sections produce no chunk; entries without `##` headings produce a single body chunk after summary is added separately by the caller. Frontmatter is parsed from the document head (delimited by `---` lines) and returned via SplitOutput.Frontmatter.
func (*Splitter) SummaryChunk ¶
SummaryChunk produces the dedicated entry-summary chunk. The text is the raw summary (no preamble — the summary already self-identifies). IsSummary is set so the indexer can apply the depth-aware boost at retrieval time. Returns the zero-value Chunk and false when summary is empty so callers can skip indexing entries without summaries.
type TextSplitter ¶
TextSplitter is the standard interface for splitting texts into string fragments. The MarkdownTextSplitter does not satisfy this interface — see SplitChunks for the chunk-bearing output. RecursiveCharacter still implements TextSplitter and is used internally as the secondary splitter for oversized markdown sections.
Source Files