Documentation
¶
Overview ¶
Package markdown provides a smart markdown chunker using scored break points. Chunks are delimited by headings, code fences, horizontal rules, and blank lines. Each candidate break point is scored by its semantic weight and by how close it falls to an ideal chunk boundary (a multiple of targetSize lines), so that chunks stay roughly uniform in length.
Index ¶
Constants ¶
const ChunkTypeSection chunking.ChunkType = "section"
ChunkTypeSection represents a heading-delimited markdown section.
const LanguageMarkdown chunking.Language = "markdown"
LanguageMarkdown is the Language constant for Markdown files.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chunker ¶
type Chunker struct {
// contains filtered or unexported fields
}
Chunker implements chunking.Chunker for Markdown files.
func NewChunker ¶
func NewChunker(options chunking.ChunkOptions) *Chunker
NewChunker creates a new Markdown chunker with the given options.
func (*Chunker) ChunkContent ¶
func (c *Chunker) ChunkContent(ctx context.Context, filePath, content string) ([]chunking.Chunk, error)
ChunkContent chunks markdown content supplied as a string. This is useful for content-addressable storage where the raw bytes are already available without a round-trip through the filesystem.
func (*Chunker) SupportedExtensions ¶
SupportedExtensions returns file extensions handled by this chunker.
Source Files