Documentation
¶
Overview ¶
Package chunking provides AST-aware code chunking for semantic code search. Chunks code files into logical units (functions, classes, methods) that preserve semantic boundaries for better vector embedding and retrieval.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Chunk ¶
type Chunk struct {
Metadata map[string]interface{}
FilePath string
Language Language
Type ChunkType
Name string
ParentName string
Content string
Signature string
DocComment string
StartLine int
EndLine int
}
Chunk represents a semantic code chunk with AST-derived boundaries.
func (*Chunk) Identifier ¶
Identifier returns a human-readable identifier for this chunk. Format: "ParentName.Name" for methods, "Name" for top-level.
func (*Chunk) SearchableContent ¶
SearchableContent returns content optimized for semantic search. Combines signature, doc comment, and content in a structured format.
type ChunkOptions ¶
type ChunkOptions struct {
// MaxChunkSize is the maximum size of a chunk in bytes.
// Chunks larger than this will be split (respecting boundaries where possible).
// 0 means no limit.
MaxChunkSize int
// IncludeDocComments controls whether to include documentation comments.
IncludeDocComments bool
// IncludePrivate controls whether to include private/unexported symbols.
IncludePrivate bool
// MinLines is the minimum number of lines for a chunk to be included.
// Chunks smaller than this will be skipped.
// 0 means no minimum.
MinLines int
}
ChunkOptions provides options for chunking behavior.
func DefaultChunkOptions ¶
func DefaultChunkOptions() ChunkOptions
DefaultChunkOptions returns sensible default options.
type ChunkType ¶
type ChunkType string
ChunkType represents the type of code chunk.
const ( // ChunkTypeFunction represents a standalone function. ChunkTypeFunction ChunkType = "function" // ChunkTypeMethod represents a method on a class/struct/type. ChunkTypeMethod ChunkType = "method" // ChunkTypeClass represents a class or struct definition. ChunkTypeClass ChunkType = "class" // ChunkTypeInterface represents an interface definition. ChunkTypeInterface ChunkType = "interface" // ChunkTypeType represents a type alias or type definition. ChunkTypeType ChunkType = "type" // ChunkTypeConst represents constant declarations. ChunkTypeConst ChunkType = "const" // ChunkTypeVar represents variable declarations. ChunkTypeVar ChunkType = "var" )
type Chunker ¶
type Chunker interface {
// Chunk parses a source file and returns semantic code chunks.
// Returns an error if the file cannot be parsed or read.
Chunk(ctx context.Context, filePath string) ([]Chunk, error)
// Language returns the language this chunker supports.
Language() Language
// SupportedExtensions returns file extensions this chunker handles.
// Example: []string{".go"} for Go chunker
SupportedExtensions() []string
}
Chunker is the interface for language-specific code chunkers.
type Language ¶
type Language string
Language represents a programming language.
const ( // LanguageGo represents the Go programming language. LanguageGo Language = "go" )
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager dispatches files to appropriate language-specific chunkers.
func NewManager ¶
func NewManager(chunkers []Chunker, options ChunkOptions) *Manager
NewManager creates a new chunking manager with the given chunkers.
func (*Manager) ChunkFile ¶
ChunkFile chunks a single file using the appropriate language chunker. Returns an error if no chunker is found for the file extension.
func (*Manager) ChunkFiles ¶
ChunkFiles chunks multiple files in parallel. Returns a map of file path to chunks, and any errors encountered. Errors for individual files do not stop processing of other files.
func (*Manager) SupportedExtensions ¶
SupportedExtensions returns all file extensions supported by registered chunkers.
func (*Manager) SupportsFile ¶
SupportsFile checks if the manager can chunk the given file based on extension.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package golang provides AST-aware chunking for Go source files.
|
Package golang provides AST-aware chunking for Go source files. |
|
Package markdown provides a smart markdown chunker using scored break points.
|
Package markdown provides a smart markdown chunker using scored break points. |