Documentation
¶
Index ¶
- func ExtractMarkdownTree(markdownText string) map[string]string
- func ExtractTablesAndRemainder(markdownText string) (string, []string, error)
- func ExtractTablesWithContext(markdownText string, contextKey string) (string, []string, error)
- type BasicSplitProvider
- type CodeSplitProvider
- type DefaultSplitProvider
- type MarkdownSplitProvider
- type QaSplitProvider
- type RecursiveSplitProvider
- type SplitProvider
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ExtractMarkdownTree ¶
Types ¶
type BasicSplitProvider ¶
type BasicSplitProvider struct{}
func NewBasicSplitProvider ¶
func NewBasicSplitProvider() (*BasicSplitProvider, error)
type CodeSplitProvider ¶ added in v1.796.1
type CodeSplitProvider struct {
// contains filtered or unexported fields
}
CodeSplitProvider chunks SOURCE CODE along structural boundaries — declarations (functions, types, classes, impls) — instead of blind line windows, so each chunk is a semantically-complete unit: a whole function keeps its signature with its body, which is what makes code retrieval actually useful. This is the standard language-aware RAG chunking (the RecursiveCharacter/from_language approach): try the most-structural separator first and only fall to finer ones when a unit still exceeds the token budget. Orthogonal to the Markdown/QA/Basic/Default providers — same SplitText contract, selected per file extension by splitTypeForFile.
func NewCodeSplitProvider ¶ added in v1.796.1
func NewCodeSplitProvider(lang string) (*CodeSplitProvider, error)
NewCodeSplitProvider builds a splitter for a language key (extension stem, e.g. "go"). Unknown languages fall back to the generic block separators — always structural.
func (*CodeSplitProvider) SplitText ¶ added in v1.796.1
func (p *CodeSplitProvider) SplitText(text string) ([]string, error)
SplitText recursively splits code on the most-structural separator that keeps each piece within the token budget, then coalesces tiny adjacent pieces so a lone signature or one-line type isn't stranded as its own chunk.
type DefaultSplitProvider ¶
type DefaultSplitProvider struct {
TextType string
}
func NewDefaultSplitProvider ¶
func NewDefaultSplitProvider(textType string) (*DefaultSplitProvider, error)
type MarkdownSplitProvider ¶
type MarkdownSplitProvider struct{}
func NewMarkdownSplitProvider ¶
func NewMarkdownSplitProvider() (*MarkdownSplitProvider, error)
type QaSplitProvider ¶
type QaSplitProvider struct{}
QaSplitProvider structure
func NewQaSplitProvider ¶
func NewQaSplitProvider() (*QaSplitProvider, error)
NewQaSplitProvider creates a new instance of QaSplitProvider
type RecursiveSplitProvider ¶ added in v1.790.2
RecursiveSplitProvider is a character-based recursive text splitter with the same contract as LangChain's RecursiveCharacterTextSplitter: it splits on a cascade of separators (paragraph → line → word → char), keeping each chunk at or below ChunkSize characters and overlapping consecutive chunks by ChunkOverlap characters.
It exists so the Hanzo RAG surface (/v1/rag/*, controllers/rag.go → object.Rag*) chunks documents identically to the retired standalone rag-api (danny-avila fork), which configured RecursiveCharacterTextSplitter with CHUNK_SIZE=1500 / CHUNK_OVERLAP=100. Chunking is a splitting concern, so it lives here in the ONE splitting home rather than being reinvented in the RAG module — GetSplitProvider("Recursive") returns it.
func NewRecursiveSplitProvider ¶ added in v1.790.2
func NewRecursiveSplitProvider(chunkSize, chunkOverlap int) (*RecursiveSplitProvider, error)
NewRecursiveSplitProvider builds a recursive splitter. Non-positive chunkSize falls back to 1500 and a negative overlap to 100 — the retired rag-api defaults — so a zero-value construction is still the parity configuration. Overlap is clamped strictly below chunkSize (an overlap ≥ size cannot make progress and would loop).
type SplitProvider ¶
func GetSplitProvider ¶
func GetSplitProvider(typ string) (SplitProvider, error)
Source Files