chunker

package
v0.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 29, 2026 License: MPL-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package chunker splits source files into semantically coherent chunks using tree-sitter ASTs.

The cAST algorithm recursively traverses a file's AST. Nodes whose byte span exceeds Config.MaxChunkSize are split; adjacent small nodes are merged until Config.MinChunkSize is reached. The result is a []Chunk where each element is a coherent source unit ready for embedding.

Chunker is constructed once via NewChunker and is safe to reuse across files. Language configurations are injected at construction time; see the langs package for the full set of built-in languages.

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	// ErrUnsupportedLanguage is returned by ChunkFile when no LanguageConfig
	// matches the file's extension.
	ErrUnsupportedLanguage = errors.New("unsupported language")

	// ErrParseFailed is returned by ChunkFile when tree-sitter cannot produce
	// a valid syntax tree for the given source.
	ErrParseFailed = errors.New("tree-sitter parse failed")
)

Functions

This section is empty.

Types

type Analysis added in v0.0.2

type Analysis struct {
	Chunks []Chunk
	Edges  []Edge // nil when the language declares no EdgeRules
}

Analysis is the combined result of chunking and edge extraction for one file.

type Chunk

type Chunk struct {
	Content  string   // raw source text of this chunk
	FilePath string   // absolute path to the source file
	Language string   // language name as reported by LanguageConfig.Name
	NodeKind string   // tree-sitter node kind, e.g. "function_declaration", "class_definition"
	Name     string   // symbol name from the NameField, e.g. "MyFunc"; empty if not extractable
	Parent   string   // containing symbol for nested nodes (e.g. class name for a method); empty for top-level
	Start    Position // start of the chunk in the source file (inclusive)
	End      Position // end of the chunk in the source file (inclusive)
}

Chunk is one semantically coherent unit of source code, ready for embedding.

type Chunker

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker parses and chunks source files. Construct once via NewChunker, reuse across files.

func NewChunker

func NewChunker(langs []LanguageConfig, cfg Config) (*Chunker, error)

NewChunker constructs a Chunker from the provided language configs and size thresholds. Returns an error if langs is empty, any Grammar pointer is nil, or two configs claim the same file extension. The returned Chunker is safe to reuse across files and goroutines.

Example
package main

import (
	"fmt"

	"github.com/ieshan/go-code-chunker/chunker"
	"github.com/ieshan/go-code-chunker/langs"
)

func main() {
	c, err := chunker.NewChunker(langs.AllLanguages(), chunker.DefaultConfig())
	if err != nil {
		panic(err)
	}
	_ = c
	fmt.Println("chunker ready")
}
Output:
chunker ready

func (*Chunker) Analyze added in v0.0.2

func (c *Chunker) Analyze(filePath string, src []byte) (Analysis, error)

Analyze parses src once and returns both its chunks and the graph edges (calls, imports, supertypes, type references) found in it. Edges are nil when the language declares no EdgeRules.

Analyze reuses the single parse that chunking already performs, so it costs only the extra AST walk. Error behaviour matches [ChunkFile]; empty src returns a zero Analysis.

Edge targets are unresolved by design — see Edge.

Example
package main

import (
	"fmt"

	"github.com/ieshan/go-code-chunker/chunker"
	"github.com/ieshan/go-code-chunker/langs"
)

func main() {
	c, err := chunker.NewChunker(
		[]chunker.LanguageConfig{langs.GoLanguage()},
		chunker.DefaultConfig(),
	)
	if err != nil {
		panic(err)
	}

	src := `package main

import "fmt"

func Greet() {
	fmt.Println(name())
}

func name() string { return "world" }
`
	res, err := c.Analyze("main.go", []byte(src))
	if err != nil {
		panic(err)
	}
	for _, e := range res.Edges {
		if e.Kind == chunker.EdgeReference {
			continue // omit type references to keep the output short
		}
		target := e.Target
		if e.TargetQualifier != "" {
			target = e.TargetQualifier + "." + e.Target
		}
		fmt.Printf("%s: %q -> %s\n", e.Kind, e.Source, target)
	}
}
Output:
import: "" -> fmt
call: "Greet" -> fmt.Println
call: "Greet" -> name

func (*Chunker) ChunkFile

func (c *Chunker) ChunkFile(filePath string, src []byte) ([]Chunk, error)

ChunkFile parses src as the language inferred from filePath's extension and returns semantically coherent chunks ready for embedding. Returns ErrUnsupportedLanguage when no LanguageConfig matches the extension, and ErrParseFailed when tree-sitter cannot parse the source. Empty src returns nil, nil.

Example
package main

import (
	"fmt"

	"github.com/ieshan/go-code-chunker/chunker"
	"github.com/ieshan/go-code-chunker/langs"
)

func main() {
	c, err := chunker.NewChunker(
		[]chunker.LanguageConfig{langs.GoLanguage()},
		chunker.DefaultConfig(),
	)
	if err != nil {
		panic(err)
	}

	src := `package main

func Hello() string {
	return "hello"
}
`
	chunks, err := c.ChunkFile("main.go", []byte(src))
	if err != nil {
		panic(err)
	}
	for _, ch := range chunks {
		fmt.Printf("kind=%s name=%s\n", ch.NodeKind, ch.Name)
	}
}
Output:
kind=function_declaration name=Hello

type Config

type Config struct {
	MaxChunkSize int // split when a node exceeds this; default 1500
	MinChunkSize int // merge siblings until this is reached; default 50
}

Config controls the cAST algorithm size thresholds (in bytes).

func DefaultConfig

func DefaultConfig() Config

DefaultConfig returns production-tuned defaults (~500 tokens at ~3 chars/token).

type Edge added in v0.0.2

type Edge struct {
	Kind            EdgeKind // relationship classification
	FilePath        string   // absolute path to the source file
	Language        string   // language name as reported by LanguageConfig.Name
	Source          string   // name of the enclosing definition; empty at file level (e.g. imports)
	SourceParent    string   // containing symbol of the enclosing definition (e.g. class name for a method); empty for top-level
	SourceLine      int      // 1-based start line of the enclosing definition; 0 when there is none
	Target          string   // referenced identifier, or module path for EdgeImport
	TargetQualifier string   // receiver/package part when qualified, or local alias for EdgeImport; empty otherwise
	Line            int      // 1-based line of the reference itself
}

Edge is one relationship extracted from a source file, pointing from the enclosing definition to a named target.

The target is deliberately left unresolved: [Target] holds the identifier as written and [TargetQualifier] the receiver or package part when present (e.g. "fmt" for `fmt.Println`). Mapping a target to a definition requires whole-project knowledge, so resolution is the caller's responsibility.

For EdgeImport edges the module path is in [Target] and [TargetQualifier] holds the local alias when the import declares one.

type EdgeKind added in v0.0.2

type EdgeKind string

EdgeKind classifies a graph edge extracted from source.

const (
	// EdgeCall is a function, method, or command invocation.
	EdgeCall EdgeKind = "call"
	// EdgeImport is an import, include, or require of another module.
	EdgeImport EdgeKind = "import"
	// EdgeInherit is a supertype relationship: extends, implements, embeds,
	// subclasses, or includes a mixin.
	EdgeInherit EdgeKind = "inherit"
	// EdgeReference is a type reference or instantiation, e.g. a type
	// annotation, a struct field type, or `new Thing()`.
	EdgeReference EdgeKind = "reference"
)

type EdgeRule added in v0.0.2

type EdgeRule struct {
	Kind           string   // AST node kind that triggers this rule, e.g. "call_expression"
	Edge           EdgeKind // edge kind to emit
	TargetField    string   // field holding the target expression; empty means the node's first named child
	QualifierField string   // field holding the receiver, for grammars that qualify on the node itself (e.g. Ruby's call)
	Descend        bool     // emit one edge per descendant whose kind is in EdgeRules.NameKinds
}

EdgeRule maps one AST node kind to the edge it produces.

The node named by TargetField is resolved to a target name: qualified-name nodes are decomposed via EdgeRules.Qualified, and any other node contributes its own source text. When Descend is set the rule instead emits one edge per matching descendant, which is how supertype lists are handled.

type EdgeRules added in v0.0.2

type EdgeRules struct {
	// Rules lists the node kinds that produce edges.
	Rules []EdgeRule

	// Qualified maps a qualified-name node kind to the fields that split it
	// into qualifier and name.
	Qualified map[string]QualifiedFields

	// NameKinds lists node kinds that count as a name when a rule descends,
	// e.g. "identifier", "type_identifier", "constant".
	NameKinds []string

	// ImportModuleKinds lists node kinds that carry an import's module text.
	// The first matching descendant of an import node wins. Surrounding
	// quotes and angle brackets are stripped from the result.
	ImportModuleKinds []string

	// ImportAliasKinds lists node kinds that carry an import's local alias,
	// searched among the import node's direct named children.
	ImportAliasKinds []string

	// CallNameEdges reclassifies calls to specific well-known names, for
	// languages that spell structural relationships as ordinary function calls:
	// Ruby's require and include, Bash's source, JavaScript's require.
	//
	// A call whose unqualified target matches a key emits the mapped edge kind
	// instead of [EdgeCall]. [EdgeImport] takes its target from the first
	// ImportModuleKinds descendant; any other kind takes one edge per NameKinds
	// descendant.
	CallNameEdges map[string]EdgeKind
}

EdgeRules declares how to extract graph edges for one language. The zero value disables edge extraction, which is the correct behaviour for data and markup languages such as JSON, YAML, and CSS.

type InjectionRule

type InjectionRule struct {
	ContainerKind string                    // node kind wrapping embedded content, e.g. "script_element"
	ContentKind   string                    // node kind of the raw content child, e.g. "raw_text"
	LangAttr      string                    // start_tag attribute that selects the grammar, e.g. "lang"
	DefaultLang   string                    // grammar key when LangAttr is absent, e.g. "javascript"
	Grammars      map[string]LanguageConfig // lang attribute value -> LanguageConfig
}

InjectionRule instructs the Chunker to re-parse a container node's embedded content using a secondary grammar, replacing the section-level atom with semantically richer atoms from the injected parse.

type LanguageConfig

type LanguageConfig struct {
	Name       string           // canonical language name, e.g. "go", "python"
	Extensions []string         // file extensions including dot, e.g. [".go"], [".py", ".pyw"]
	Grammar    *sitter.Language // tree-sitter grammar; obtain via sitter.NewLanguage(binding.Language())
	NodeKinds  NodeKindSet
	Injections []InjectionRule // optional: re-parse embedded content with secondary grammars; nil means no injection
	Edges      EdgeRules       // optional: graph edge extraction for Analyze; zero value disables it
}

LanguageConfig fully describes one language the Chunker can handle. Construct these in the langs/ package and pass them to NewChunker.

type NodeKindSet

type NodeKindSet struct {
	TopLevel  []string // node types that become top-level chunk boundaries
	Nested    []string // node types nested inside top-level nodes (e.g. methods in a class)
	NameField string   // tree-sitter field name for the symbol identifier, e.g. "name"; empty means skip name extraction
}

NodeKindSet describes which AST node types are semantic boundaries and how to extract symbol names from them.

type Position

type Position struct {
	Line       int // 1-based
	Column     int // 0-based
	ByteOffset int
}

Position is a point in source text.

type QualifiedFields added in v0.0.2

type QualifiedFields struct {
	Qualifier string // field name holding the receiver/package part
	Name      string // field name holding the selected identifier
}

QualifiedFields decomposes a qualified-name node into its two halves, e.g. Go's selector_expression ("fmt.Println" -> operand "fmt", field "Println") or Python's attribute ("os.path.join" -> object "os.path", attribute "join").

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL