parser

package
v1.31.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package parser provides Python code parsing capabilities using tree-sitter.

This package wraps the tree-sitter Go bindings to parse Python source code into Abstract Syntax Trees (AST). It provides utilities for traversing, querying, and analyzing Python code structure.

Key features:

  • Fast and accurate Python parsing using tree-sitter
  • Support for Python 3.8+ syntax
  • Error-tolerant parsing with syntax error detection
  • Tree traversal and node searching utilities
  • Cross-platform compatibility

Basic usage: 

p := parser.New()
result, err := p.Parse(ctx, []byte("def hello(): pass"))
if err != nil {
    // Handle parsing error
}
// Use result.RootNode to traverse the AST

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ASTBuilder 

type ASTBuilder struct {
	// contains filtered or unexported fields
}

ASTBuilder converts tree-sitter parse trees to internal AST representation

func NewASTBuilder 

func NewASTBuilder(source []byte) *ASTBuilder

NewASTBuilder creates a new AST builder

func (*ASTBuilder) Build 

func (b *ASTBuilder) Build(tree *sitter.Tree) (*Node, error)

Build converts a tree-sitter tree to internal AST

type BodyChildFilter added in v1.20.0 

type BodyChildFilter func(parent, bodyNode *Node, bodyIndex int) bool

BodyChildFilter decides whether a body child should be skipped while walking a node's canonical children.

type CollectorVisitor 

type CollectorVisitor struct {
	// contains filtered or unexported fields
}

CollectorVisitor collects nodes matching a predicate

func NewCollectorVisitor 

func NewCollectorVisitor(predicate func(*Node) bool) *CollectorVisitor

NewCollectorVisitor creates a visitor that collects matching nodes

func (*CollectorVisitor) GetNodes 

func (v *CollectorVisitor) GetNodes() []*Node

GetNodes returns the collected nodes

func (*CollectorVisitor) Visit 

func (v *CollectorVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type DepthFirstVisitor 

type DepthFirstVisitor struct {
	// contains filtered or unexported fields
}

DepthFirstVisitor performs depth-first traversal

func NewDepthFirstVisitor 

func NewDepthFirstVisitor(preOrder func(*Node) bool, postOrder func(*Node)) *DepthFirstVisitor

NewDepthFirstVisitor creates a depth-first visitor

func (*DepthFirstVisitor) Visit 

func (v *DepthFirstVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type FuncVisitor 

type FuncVisitor struct {
	// contains filtered or unexported fields
}

FuncVisitor is a visitor that uses a function

func NewFuncVisitor 

func NewFuncVisitor(fn func(*Node) bool) *FuncVisitor

NewFuncVisitor creates a visitor from a function

func (*FuncVisitor) Visit 

func (v *FuncVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type Location 

type Location struct {
	File      string
	StartLine int
	StartCol  int
	EndLine   int
	EndCol    int
}

Location represents the position of a node in the source code

type Node 

type Node struct {
	Type     NodeType
	Value    interface{} // Can hold various values depending on node type
	Children []*Node
	Location Location
	Parent   *Node

	// Additional fields for specific node types
	Name      string   // For function/class definitions, variables
	Target    *Node    // For with-item `as` alias (may be a name, tuple, list, or starred pattern)
	Targets   []*Node  // For assignments
	Body      []*Node  // For compound statements
	Orelse    []*Node  // For if/for/while/try statements
	Finalbody []*Node  // For try statements
	Handlers  []*Node  // For try statements
	Test      *Node    // For if/while statements
	Iter      *Node    // For for loops
	Args      []*Node  // For function calls
	Keywords  []*Node  // For function calls
	Decorator []*Node  // For decorated functions/classes
	Bases     []*Node  // For class definitions
	Left      *Node    // For binary operations
	Right     *Node    // For binary operations
	Op        string   // For operations
	Module    string   // For imports
	Names     []string // For imports
	Level     int      // For relative imports
}

Node represents an AST node

func CloneNode added in v1.30.0 

func CloneNode(root *Node) *Node

CloneNode returns an independent copy of an AST while preserving shared references and parent links within the copied graph.

func NewNode 

func NewNode(nodeType NodeType) *Node

NewNode creates a new AST node

func OrderedChildren added in v1.20.0 

func OrderedChildren(node *Node, skipBodyNode BodyChildFilter) []*Node

OrderedChildren returns every structural child exactly once in a stable order. Value is included when it holds a *Node because many builders store core expression children there.

func (*Node) Accept 

func (n *Node) Accept(visitor Visitor)

Accept implements the visitor pattern for AST nodes

func (*Node) AddChild 

func (n *Node) AddChild(child *Node)

AddChild adds a child node

func (*Node) AddToBody 

func (n *Node) AddToBody(node *Node)

AddToBody adds a node to the body

func (*Node) Copy 

func (n *Node) Copy() *Node

Copy creates a deep copy of the node

func (*Node) Find 

func (n *Node) Find(predicate func(*Node) bool) []*Node

Find finds all nodes matching a predicate

func (*Node) FindByType 

func (n *Node) FindByType(nodeType NodeType) []*Node

FindByType finds all nodes of a specific type

func (*Node) GetChildren 

func (n *Node) GetChildren() []*Node

GetChildren returns all child nodes in the parser's canonical order.

func (*Node) GetParentOfType 

func (n *Node) GetParentOfType(nodeType NodeType) *Node

GetParentOfType finds the nearest parent of a specific type

func (*Node) IsControlFlow 

func (n *Node) IsControlFlow() bool

IsControlFlow returns true if the node represents control flow

func (*Node) IsExpression 

func (n *Node) IsExpression() bool

IsExpression returns true if the node is an expression

func (*Node) IsStatement 

func (n *Node) IsStatement() bool

IsStatement returns true if the node is a statement 

func (n *Node) RefreshParentLinks()

RefreshParentLinks rebuilds Parent pointers from the canonical child graph.

func (*Node) String 

func (n *Node) String() string

String returns a string representation of the node

func (*Node) Walk 

func (n *Node) Walk(visitor func(*Node) bool)

Walk traverses the AST using depth-first search

func (*Node) WalkDeep added in v1.16.0 

func (n *Node) WalkDeep(visitor func(*Node) bool)

WalkDeep is kept for callers that used the old explicit deep walk. Walk now uses the same canonical child contract.

type NodeType 

type NodeType string

NodeType represents the type of AST node 

const (
	// Module and structure
	NodeModule      NodeType = "Module"
	NodeInteractive NodeType = "Interactive"
	NodeExpression  NodeType = "Expression"
	NodeSuite       NodeType = "Suite"

	// Statements
	NodeFunctionDef      NodeType = "FunctionDef"
	NodeAsyncFunctionDef NodeType = "AsyncFunctionDef"
	NodeClassDef         NodeType = "ClassDef"
	NodeReturn           NodeType = "Return"
	NodeDelete           NodeType = "Delete"
	NodeAssign           NodeType = "Assign"
	NodeAugAssign        NodeType = "AugAssign"
	NodeAnnAssign        NodeType = "AnnAssign"
	NodeFor              NodeType = "For"
	NodeAsyncFor         NodeType = "AsyncFor"
	NodeWhile            NodeType = "While"
	NodeIf               NodeType = "If"
	NodeWith             NodeType = "With"
	NodeAsyncWith        NodeType = "AsyncWith"
	NodeMatch            NodeType = "Match"
	NodeRaise            NodeType = "Raise"
	NodeTry              NodeType = "Try"
	NodeAssert           NodeType = "Assert"
	NodeImport           NodeType = "Import"
	NodeImportFrom       NodeType = "ImportFrom"
	NodeGlobal           NodeType = "Global"
	NodeNonlocal         NodeType = "Nonlocal"
	NodeExpr             NodeType = "Expr"
	NodePass             NodeType = "Pass"
	NodeBreak            NodeType = "Break"
	NodeContinue         NodeType = "Continue"

	// Expressions
	NodeBoolOp         NodeType = "BoolOp"
	NodeNamedExpr      NodeType = "NamedExpr"
	NodeBinOp          NodeType = "BinOp"
	NodeUnaryOp        NodeType = "UnaryOp"
	NodeLambda         NodeType = "Lambda"
	NodeIfExp          NodeType = "IfExp"
	NodeDict           NodeType = "Dict"
	NodeSet            NodeType = "Set"
	NodeListComp       NodeType = "ListComp"
	NodeSetComp        NodeType = "SetComp"
	NodeDictComp       NodeType = "DictComp"
	NodeGeneratorExp   NodeType = "GeneratorExp"
	NodeAwait          NodeType = "Await"
	NodeYield          NodeType = "Yield"
	NodeYieldFrom      NodeType = "YieldFrom"
	NodeCompare        NodeType = "Compare"
	NodeCall           NodeType = "Call"
	NodeFormattedValue NodeType = "FormattedValue"
	NodeJoinedStr      NodeType = "JoinedStr"
	NodeConstant       NodeType = "Constant"
	NodeAttribute      NodeType = "Attribute"
	NodeSubscript      NodeType = "Subscript"
	NodeStarred        NodeType = "Starred"
	NodeName           NodeType = "Name"
	NodeList           NodeType = "List"
	NodeTuple          NodeType = "Tuple"
	NodeSlice          NodeType = "Slice"

	// Patterns (for match statements)
	NodeMatchValue     NodeType = "MatchValue"
	NodeMatchSingleton NodeType = "MatchSingleton"
	NodeMatchSequence  NodeType = "MatchSequence"
	NodeMatchMapping   NodeType = "MatchMapping"
	NodeMatchClass     NodeType = "MatchClass"
	NodeMatchStar      NodeType = "MatchStar"
	NodeMatchAs        NodeType = "MatchAs"
	NodeMatchOr        NodeType = "MatchOr"

	// Other
	NodeAlias           NodeType = "Alias"
	NodeExceptHandler   NodeType = "ExceptHandler"
	NodeArguments       NodeType = "Arguments"
	NodeArg             NodeType = "Arg"
	NodeKeyword         NodeType = "Keyword"
	NodeKeywordArgument NodeType = "keyword_argument"
	NodeComprehension   NodeType = "Comprehension"
	NodeDecorator       NodeType = "Decorator"
	NodeWithItem        NodeType = "WithItem"
	NodeMatchCase       NodeType = "MatchCase"
	NodeElseClause      NodeType = "else_clause" // Structural marker from parser
	NodeElifClause      NodeType = "elif_clause" // Structural marker from parser
	NodeBlock           NodeType = "block"       // Block of statements from parser

	// Tree-sitter specific nodes
	NodeGenericType   NodeType = "generic_type"
	NodeTypeParameter NodeType = "type_parameter"
	NodeTypeNode      NodeType = "type"
)

Python AST node types

type ParseResult 

type ParseResult struct {
	Tree       *sitter.Tree
	RootNode   *sitter.Node
	SourceCode []byte
	AST        *Node // Internal AST representation
}

ParseResult represents the result of parsing Python code

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser provides Python code parsing capabilities using tree-sitter

func New 

func New() *Parser

New creates a new Parser instance with Python grammar

func (*Parser) CheckSyntax added in v1.29.1 

func (p *Parser) CheckSyntax(node *sitter.Node, source []byte) error

CheckSyntax returns an error describing the first syntax problem in the parsed tree, or nil when the source is valid Python 3. Beyond the ERROR and MISSING nodes tree-sitter reports, it also rejects legacy constructs that the grammar still accepts but no Python 3 release compiles (see python3_syntax.go).

func (*Parser) FindNodes 

func (p *Parser) FindNodes(node *sitter.Node, nodeType string) []*sitter.Node

FindNodes finds all nodes of a specific type in the tree

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/ludo-technologies/pyscn/internal/parser"
)

func main() {
	p := parser.New()
	ctx := context.Background()

	source := []byte(`
class Calculator:
    def add(self, a, b):
        return a + b
    
    def subtract(self, a, b):
        return a - b
    
    def multiply(self, a, b):
        return a * b
`)

	result, err := p.Parse(ctx, source)
	if err != nil {
		log.Fatal(err)
	}

	// Find all class definitions
	classes := p.FindNodes(result.RootNode, "class_definition")
	fmt.Printf("Classes: %d\n", len(classes))

	// Find all function definitions
	functions := p.FindNodes(result.RootNode, "function_definition")
	fmt.Printf("Methods: %d\n", len(functions))

}

Output:
Classes: 1
Methods: 3

func (*Parser) GetNodeText 

func (p *Parser) GetNodeText(node *sitter.Node, source []byte) string

GetNodeText returns the text content of a node

func (*Parser) Parse 

func (p *Parser) Parse(ctx context.Context, source []byte) (*ParseResult, error)

Parse parses Python source code and returns the AST

Example 
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/ludo-technologies/pyscn/internal/parser"
)

func main() {
	p := parser.New()
	ctx := context.Background()

	source := []byte(`def greet(name):
    return f"Hello, {name}!"

print(greet("World"))`)

	result, err := p.Parse(ctx, source)
	if err != nil {
		log.Fatal(err)
	}

	// Find all function definitions
	functions := p.FindNodes(result.RootNode, "function_definition")
	fmt.Printf("Found %d function(s)\n", len(functions))

}

Output:
Found 1 function(s)

func (*Parser) ParseFile 

func (p *Parser) ParseFile(ctx context.Context, reader io.Reader) (*ParseResult, error)

ParseFile parses a Python file from a reader

func (*Parser) WalkTree 

func (p *Parser) WalkTree(node *sitter.Node, visitor func(*sitter.Node) error) error

WalkTree traverses the AST and calls the visitor function for each node

type PathVisitor 

type PathVisitor struct {
	// contains filtered or unexported fields
}

PathVisitor tracks the path to each visited node

func NewPathVisitor 

func NewPathVisitor(visitor func(node *Node, path []*Node) bool) *PathVisitor

NewPathVisitor creates a visitor that tracks the path to each node

func (*PathVisitor) Visit 

func (v *PathVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type PrinterVisitor 

type PrinterVisitor struct {
	// contains filtered or unexported fields
}

PrinterVisitor prints the AST structure

func NewPrinterVisitor 

func NewPrinterVisitor(w io.Writer) *PrinterVisitor

NewPrinterVisitor creates a visitor that prints the AST

func (*PrinterVisitor) Visit 

func (v *PrinterVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type SimplifierVisitor 

type SimplifierVisitor struct {
	// contains filtered or unexported fields
}

SimplifierVisitor simplifies the AST by removing unnecessary nodes

func NewSimplifierVisitor 

func NewSimplifierVisitor() *SimplifierVisitor

NewSimplifierVisitor creates a visitor that simplifies the AST

func (*SimplifierVisitor) Visit 

func (v *SimplifierVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

func (*SimplifierVisitor) WasSimplified 

func (v *SimplifierVisitor) WasSimplified() bool

WasSimplified returns true if any simplification was performed

type StatisticsVisitor 

type StatisticsVisitor struct {
	NodeCounts map[NodeType]int
	TotalNodes int
	MaxDepth   int
	// contains filtered or unexported fields
}

StatisticsVisitor collects statistics about the AST

func NewStatisticsVisitor 

func NewStatisticsVisitor() *StatisticsVisitor

NewStatisticsVisitor creates a visitor that collects statistics

func (*StatisticsVisitor) Visit 

func (v *StatisticsVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type TransformVisitor 

type TransformVisitor struct {
	// contains filtered or unexported fields
}

TransformVisitor transforms nodes in the AST

func NewTransformVisitor 

func NewTransformVisitor(transformer func(*Node) *Node) *TransformVisitor

NewTransformVisitor creates a visitor that transforms nodes

func (*TransformVisitor) Visit 

func (v *TransformVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type ValidatorVisitor 

type ValidatorVisitor struct {
	// contains filtered or unexported fields
}

ValidatorVisitor validates the AST structure

func NewValidatorVisitor 

func NewValidatorVisitor() *ValidatorVisitor

NewValidatorVisitor creates a visitor that validates the AST

func (*ValidatorVisitor) GetErrors 

func (v *ValidatorVisitor) GetErrors() []string

GetErrors returns validation errors

func (*ValidatorVisitor) IsValid 

func (v *ValidatorVisitor) IsValid() bool

IsValid returns true if no errors were found

func (*ValidatorVisitor) Visit 

func (v *ValidatorVisitor) Visit(node *Node) bool

Visit implements the Visitor interface

type Visitor 

type Visitor interface {
	// Visit is called for each node in the AST
	// Return false to stop traversal
	Visit(node *Node) bool
}

Visitor defines the interface for visiting AST nodes

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.