Documentation
¶
Overview ¶
Package chunker splits source files into semantically coherent chunks using tree-sitter ASTs.
The cAST algorithm recursively traverses a file's AST. Nodes whose byte span exceeds Config.MaxChunkSize are split; adjacent small nodes are merged until Config.MinChunkSize is reached. The result is a []Chunk where each element is a coherent source unit ready for embedding.
Chunker is constructed once via NewChunker and is safe to reuse across files. Language configurations are injected at construction time; see the langs package for the full set of built-in languages.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrUnsupportedLanguage is returned by ChunkFile when no LanguageConfig // matches the file's extension. ErrUnsupportedLanguage = errors.New("unsupported language") // ErrParseFailed is returned by ChunkFile when tree-sitter cannot produce // a valid syntax tree for the given source. ErrParseFailed = errors.New("tree-sitter parse failed") )
Functions ¶
This section is empty.
Types ¶
type Analysis ¶ added in v0.0.2
Analysis is the combined result of chunking and edge extraction for one file.
type Chunk ¶
type Chunk struct {
Content string // raw source text of this chunk
FilePath string // absolute path to the source file
Language string // language name as reported by LanguageConfig.Name
NodeKind string // tree-sitter node kind, e.g. "function_declaration", "class_definition"
Name string // symbol name from the NameField, e.g. "MyFunc"; empty if not extractable
Parent string // containing symbol for nested nodes (e.g. class name for a method); empty for top-level
Start Position // start of the chunk in the source file (inclusive)
End Position // end of the chunk in the source file (inclusive)
}
Chunk is one semantically coherent unit of source code, ready for embedding.
type Chunker ¶
type Chunker struct {
// contains filtered or unexported fields
}
Chunker parses and chunks source files. Construct once via NewChunker, reuse across files.
func NewChunker ¶
func NewChunker(langs []LanguageConfig, cfg Config) (*Chunker, error)
NewChunker constructs a Chunker from the provided language configs and size thresholds. Returns an error if langs is empty, any Grammar pointer is nil, or two configs claim the same file extension. The returned Chunker is safe to reuse across files and goroutines.
Example ¶
package main
import (
"fmt"
"github.com/ieshan/go-code-chunker/chunker"
"github.com/ieshan/go-code-chunker/langs"
)
func main() {
c, err := chunker.NewChunker(langs.AllLanguages(), chunker.DefaultConfig())
if err != nil {
panic(err)
}
_ = c
fmt.Println("chunker ready")
}
Output: chunker ready
func (*Chunker) Analyze ¶ added in v0.0.2
Analyze parses src once and returns both its chunks and the graph edges (calls, imports, supertypes, type references) found in it. Edges are nil when the language declares no EdgeRules.
Analyze reuses the single parse that chunking already performs, so it costs only the extra AST walk. Error behaviour matches [ChunkFile]; empty src returns a zero Analysis.
Edge targets are unresolved by design — see Edge.
Example ¶
package main
import (
"fmt"
"github.com/ieshan/go-code-chunker/chunker"
"github.com/ieshan/go-code-chunker/langs"
)
func main() {
c, err := chunker.NewChunker(
[]chunker.LanguageConfig{langs.GoLanguage()},
chunker.DefaultConfig(),
)
if err != nil {
panic(err)
}
src := `package main
import "fmt"
func Greet() {
fmt.Println(name())
}
func name() string { return "world" }
`
res, err := c.Analyze("main.go", []byte(src))
if err != nil {
panic(err)
}
for _, e := range res.Edges {
if e.Kind == chunker.EdgeReference {
continue // omit type references to keep the output short
}
target := e.Target
if e.TargetQualifier != "" {
target = e.TargetQualifier + "." + e.Target
}
fmt.Printf("%s: %q -> %s\n", e.Kind, e.Source, target)
}
}
Output: import: "" -> fmt call: "Greet" -> fmt.Println call: "Greet" -> name
func (*Chunker) ChunkFile ¶
ChunkFile parses src as the language inferred from filePath's extension and returns semantically coherent chunks ready for embedding. Returns ErrUnsupportedLanguage when no LanguageConfig matches the extension, and ErrParseFailed when tree-sitter cannot parse the source. Empty src returns nil, nil.
Example ¶
package main
import (
"fmt"
"github.com/ieshan/go-code-chunker/chunker"
"github.com/ieshan/go-code-chunker/langs"
)
func main() {
c, err := chunker.NewChunker(
[]chunker.LanguageConfig{langs.GoLanguage()},
chunker.DefaultConfig(),
)
if err != nil {
panic(err)
}
src := `package main
func Hello() string {
return "hello"
}
`
chunks, err := c.ChunkFile("main.go", []byte(src))
if err != nil {
panic(err)
}
for _, ch := range chunks {
fmt.Printf("kind=%s name=%s\n", ch.NodeKind, ch.Name)
}
}
Output: kind=function_declaration name=Hello
type Config ¶
type Config struct {
MaxChunkSize int // split when a node exceeds this; default 1500
MinChunkSize int // merge siblings until this is reached; default 50
}
Config controls the cAST algorithm size thresholds (in bytes).
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns production-tuned defaults (~500 tokens at ~3 chars/token).
type Edge ¶ added in v0.0.2
type Edge struct {
Kind EdgeKind // relationship classification
FilePath string // absolute path to the source file
Language string // language name as reported by LanguageConfig.Name
Source string // name of the enclosing definition; empty at file level (e.g. imports)
SourceParent string // containing symbol of the enclosing definition (e.g. class name for a method); empty for top-level
SourceLine int // 1-based start line of the enclosing definition; 0 when there is none
Target string // referenced identifier, or module path for EdgeImport
TargetQualifier string // receiver/package part when qualified, or local alias for EdgeImport; empty otherwise
Line int // 1-based line of the reference itself
}
Edge is one relationship extracted from a source file, pointing from the enclosing definition to a named target.
The target is deliberately left unresolved: [Target] holds the identifier as written and [TargetQualifier] the receiver or package part when present (e.g. "fmt" for `fmt.Println`). Mapping a target to a definition requires whole-project knowledge, so resolution is the caller's responsibility.
For EdgeImport edges the module path is in [Target] and [TargetQualifier] holds the local alias when the import declares one.
type EdgeKind ¶ added in v0.0.2
type EdgeKind string
EdgeKind classifies a graph edge extracted from source.
const ( // EdgeCall is a function, method, or command invocation. EdgeCall EdgeKind = "call" // EdgeImport is an import, include, or require of another module. EdgeImport EdgeKind = "import" // EdgeInherit is a supertype relationship: extends, implements, embeds, // subclasses, or includes a mixin. EdgeInherit EdgeKind = "inherit" // EdgeReference is a type reference or instantiation, e.g. a type // annotation, a struct field type, or `new Thing()`. EdgeReference EdgeKind = "reference" )
type EdgeRule ¶ added in v0.0.2
type EdgeRule struct {
Kind string // AST node kind that triggers this rule, e.g. "call_expression"
Edge EdgeKind // edge kind to emit
TargetField string // field holding the target expression; empty means the node's first named child
QualifierField string // field holding the receiver, for grammars that qualify on the node itself (e.g. Ruby's call)
Descend bool // emit one edge per descendant whose kind is in EdgeRules.NameKinds
}
EdgeRule maps one AST node kind to the edge it produces.
The node named by TargetField is resolved to a target name: qualified-name nodes are decomposed via EdgeRules.Qualified, and any other node contributes its own source text. When Descend is set the rule instead emits one edge per matching descendant, which is how supertype lists are handled.
type EdgeRules ¶ added in v0.0.2
type EdgeRules struct {
// Rules lists the node kinds that produce edges.
Rules []EdgeRule
// Qualified maps a qualified-name node kind to the fields that split it
// into qualifier and name.
Qualified map[string]QualifiedFields
// NameKinds lists node kinds that count as a name when a rule descends,
// e.g. "identifier", "type_identifier", "constant".
NameKinds []string
// ImportModuleKinds lists node kinds that carry an import's module text.
// The first matching descendant of an import node wins. Surrounding
// quotes and angle brackets are stripped from the result.
ImportModuleKinds []string
// ImportAliasKinds lists node kinds that carry an import's local alias,
// searched among the import node's direct named children.
ImportAliasKinds []string
// CallNameEdges reclassifies calls to specific well-known names, for
// languages that spell structural relationships as ordinary function calls:
// Ruby's require and include, Bash's source, JavaScript's require.
//
// A call whose unqualified target matches a key emits the mapped edge kind
// instead of [EdgeCall]. [EdgeImport] takes its target from the first
// ImportModuleKinds descendant; any other kind takes one edge per NameKinds
// descendant.
CallNameEdges map[string]EdgeKind
}
EdgeRules declares how to extract graph edges for one language. The zero value disables edge extraction, which is the correct behaviour for data and markup languages such as JSON, YAML, and CSS.
type InjectionRule ¶
type InjectionRule struct {
ContainerKind string // node kind wrapping embedded content, e.g. "script_element"
ContentKind string // node kind of the raw content child, e.g. "raw_text"
LangAttr string // start_tag attribute that selects the grammar, e.g. "lang"
DefaultLang string // grammar key when LangAttr is absent, e.g. "javascript"
Grammars map[string]LanguageConfig // lang attribute value -> LanguageConfig
}
InjectionRule instructs the Chunker to re-parse a container node's embedded content using a secondary grammar, replacing the section-level atom with semantically richer atoms from the injected parse.
type LanguageConfig ¶
type LanguageConfig struct {
Name string // canonical language name, e.g. "go", "python"
Extensions []string // file extensions including dot, e.g. [".go"], [".py", ".pyw"]
Grammar *sitter.Language // tree-sitter grammar; obtain via sitter.NewLanguage(binding.Language())
NodeKinds NodeKindSet
Injections []InjectionRule // optional: re-parse embedded content with secondary grammars; nil means no injection
Edges EdgeRules // optional: graph edge extraction for Analyze; zero value disables it
}
LanguageConfig fully describes one language the Chunker can handle. Construct these in the langs/ package and pass them to NewChunker.
type NodeKindSet ¶
type NodeKindSet struct {
TopLevel []string // node types that become top-level chunk boundaries
Nested []string // node types nested inside top-level nodes (e.g. methods in a class)
NameField string // tree-sitter field name for the symbol identifier, e.g. "name"; empty means skip name extraction
}
NodeKindSet describes which AST node types are semantic boundaries and how to extract symbol names from them.
type QualifiedFields ¶ added in v0.0.2
type QualifiedFields struct {
Qualifier string // field name holding the receiver/package part
Name string // field name holding the selected identifier
}
QualifiedFields decomposes a qualified-name node into its two halves, e.g. Go's selector_expression ("fmt.Println" -> operand "fmt", field "Println") or Python's attribute ("os.path.join" -> object "os.path", attribute "join").