Documentation
¶
Overview ¶
Package markdown extracts and transforms Markdown documents.
Reader can emit one whole document or heading-scoped documents with source metadata. Splitter performs structure-aware, token-bounded chunking: headings are repeated as retrieval context, tables split only between rows, lists only between items, and fenced code blocks only between lines while retaining their fences.
Index ¶
Constants ¶
const ( MetadataHeading = "markdown.heading" MetadataHeadingLevel = "markdown.heading.level" MetadataHeadingPath = "markdown.heading.path" MetadataSourceName = "markdown.source" )
Markdown metadata keys keep structural lineage explicit without changing the core document contract.
Variables ¶
var ErrSemanticUnitTooLarge = errors.New("markdown splitter: semantic unit exceeds token limit")
ErrSemanticUnitTooLarge reports a Markdown unit that cannot be split without losing its semantic boundary.
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader extracts documents from Markdown.
type ReaderConfig ¶
type ReaderConfig struct {
HeadingSplitLevel int
SourceName string
Metadata coremetadata.Map
SourceBudget etl.SourceBudget
}
ReaderConfig controls Markdown extraction. HeadingSplitLevel emits one document per section split on headings at or above that level (1 = H1, 2 = H1+H2). Zero disables splitting; non-zero values must be in [1, 6]. Metadata is cloned by NewReader, and reader-derived markdown.* keys take precedence. A zero SourceBudget uses etl.DefaultMaxSourceBytes.
type Splitter ¶
type Splitter struct {
// contains filtered or unexported fields
}
Splitter produces token-bounded Markdown chunks without severing tables, list items, or code lines. Active heading ancestry is repeated in each chunk so independently retrieved chunks retain their section context.
func NewSplitter ¶
func NewSplitter(config SplitterConfig) (*Splitter, error)
NewSplitter validates semantic and output bounds before retaining policy.
type SplitterConfig ¶
type SplitterConfig struct {
Tokenizer tokenizer.Tokenizer
MaxTokensPerChunk int
MaxChunks int
// MaxSearchWork bounds each paragraph prefix search in bytes rendered or
// encoded plus tokens decoded; zero uses etl.DefaultMaxSearchWork.
MaxSearchWork int
IDGenerator etl.IDGenerator
}
SplitterConfig configures structure-aware Markdown chunking. Zero limits use documented defaults; negative limits are rejected.
Source Files