Documentation
¶
Overview ¶
Package markdown extracts and transforms Markdown documents.
Reader can emit one whole document or heading-scoped documents with source metadata. TextSplitter is the etl.TextSplitter for structure-aware, token-bounded chunking: headings are repeated as retrieval context, tables split only between rows, lists only between items, and fenced code blocks only between lines while retaining their fences.
Index ¶
Constants ¶
const ( MetadataHeading = "markdown.heading" MetadataHeadingLevel = "markdown.heading.level" MetadataHeadingPath = "markdown.heading.path" )
const DefaultMaxExpandedBytes = 32 << 20
Variables ¶
var ErrReferenceExpansionTooLarge = errors.New("markdown: reference expansion exceeds byte limit")
ErrReferenceExpansionTooLarge means no partially expanded source is exposed.
var ErrSemanticUnitTooLarge = errors.New("markdown splitter: semantic unit exceeds token limit")
ErrSemanticUnitTooLarge reports a Markdown unit that cannot be split without losing its semantic boundary.
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Read admits one attempt on the sequential source. Reopen the input to retry after a failure; concurrent or later attempts return etl.ErrSourceConsumed.
type ReaderConfig ¶
type ReaderConfig struct {
HeadingSplitLevel int
SourceName string
Metadata coremetadata.Map
SourceBudget etl.SourceBudget
// MaxExpandedBytes bounds source bytes before parsing and after reference expansion.
// Zero uses DefaultMaxExpandedBytes; whole-document reads do not expand links.
MaxExpandedBytes int
}
ReaderConfig controls Markdown extraction. HeadingSplitLevel emits one document per section split on headings at or above that level (1 = H1, 2 = H1+H2). Zero disables splitting; non-zero values must be in [1, 6]. Metadata is cloned by NewReader and cannot contain the reader's heading keys. A zero SourceBudget uses etl.DefaultMaxSourceBytes.
type TextSplitter ¶ added in v0.60.0
type TextSplitter struct {
// contains filtered or unexported fields
}
TextSplitter is the etl.TextSplitter that produces token-bounded Markdown chunks without severing tables, list items, or code lines. Active heading ancestry is repeated in each chunk so independently retrieved chunks retain their section context.
func NewTextSplitter ¶ added in v0.60.0
func NewTextSplitter(config TextSplitterConfig) (*TextSplitter, error)
type TextSplitterConfig ¶ added in v0.60.0
type TextSplitterConfig struct {
Tokenizer tokenizer.Tokenizer
MaxTokensPerChunk int
MaxChunks int
// MaxSearchWork bounds each paragraph prefix search in bytes rendered or
// encoded plus tokens decoded; zero uses etl.DefaultMaxSearchWork.
MaxSearchWork int
// MaxExpandedBytes bounds source bytes before parsing and after reference expansion.
// Zero uses DefaultMaxExpandedBytes.
MaxExpandedBytes int
}
TextSplitterConfig configures structure-aware Markdown chunking. Zero limits use documented defaults; negative limits are rejected.
Source Files