gotreesitter

package module
v0.48.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 4, 2026 License: MIT Imports: 29 Imported by: 26

README

gotreesitter

Pure-Go tree-sitter runtime. No CGo, no C toolchain. It cross-compiles to any GOOS/GOARCH target Go supports, including wasip1.

go get github.com/odvcencio/gotreesitter

gotreesitter loads the same parse-table format that tree-sitter's C runtime uses. ts2go extracts grammar tables from upstream parser.c files, compresses them into binary blobs, and deserializes them on first use. 206 grammars ship in the registry.

Agent Skill

Agents working with gotreesitter should use the using-gotreesitter skill.

Motivation

Every Go tree-sitter binding in the ecosystem depends on CGo:

  • Cross-compilation needs a C cross-toolchain per target. A build with GOOS=wasip1, GOARCH=arm64 from a Linux host, or any Windows target without MSYS2/MinGW will not link.
  • CI images must carry gcc and the grammar's C sources. go install fails for downstream users without a C compiler.
  • The Go race detector, coverage instrumentation, and fuzzer cannot see across the CGo boundary. Bugs in the C runtime or in FFI marshaling stay invisible to go test -race.

gotreesitter eliminates the C dependency entirely. The parser, lexer, query engine, incremental reparsing, arena allocator, external scanners, and tree cursor are all implemented in Go. The grammar blob is the only input.

Quick start

import (
    "fmt"

    "github.com/odvcencio/gotreesitter"
    "github.com/odvcencio/gotreesitter/grammars"
)

func main() {
    src := []byte(`package main

func main() {}
`)

    lang := grammars.GoLanguage()
    parser := gotreesitter.NewParser(lang)

    tree, _ := parser.Parse(src)
    fmt.Println(tree.RootNode())
}

grammars.DetectLanguage("main.go") resolves a filename to the matching LangEntry.

Browser and WebAssembly

The repository ships two GOOS=js GOARCH=wasm targets: a blob-loading runtime, and a grammargen build that imports tree-sitter grammar JSON and generates tables in the browser. The runtime exposes parsing, queries, and highlighting; structured parse and query results include both UTF-8 byte offsets and JavaScript UTF-16 code-unit offsets. It can also retain an incrementally updated document tree and reuse it for highlights, tags, and queries. cmd/wasmassets emits a reproducible, single-language browser bundle for either the Go or TinyGo WebAssembly compiler.

See the WebAssembly guide for build commands, the complete JavaScript APIs, node and match limits, and a browser example.

Strict parsing and partial trees

The default parse methods preserve tree-sitter's partial-tree behavior. If a timeout, cancellation flag, token-source EOF, or parser safety limit stops the parse early, the returned tree records the stop reason and the error stays nil. This is useful for editors and diagnostics, where a partial tree still has value.

Use strict methods when partial output should fail a request:

parser.SetTimeoutMicros(50_000)

tree, err := parser.ParseStrict(src)
if errors.Is(err, gotreesitter.ErrParseStoppedEarly) {
    fmt.Println(tree.ParseStopReason())
}

Strict variants are available for full parse, incremental parse, token-source parse, factory parse, ParseWith, and ParserPool.

High-level analysis can use WithHighlighterTimeoutMicros or WithTaggerTimeoutMicros to bound parser work. The HighlightIncrementalStrict and TagIncrementalStrict methods return the partial tree together with ErrParseStoppedEarly and skip running queries after an early stop.

Tree lookup helpers

Use NodeAtByte or NamedNodeAtByte to turn an editor byte offset into a syntax node:

node := tree.NamedNodeAtByte(offset)

The helpers return the smallest matching descendant and handle exact end-byte boundaries, so callers do not need to hand-roll a tree walk.

Code-understanding helpers

For hot indexing paths that need common symbols but not arbitrary tags-query semantics, use the one-pass extractors instead of running a fanout of queries:

defs := gotreesitter.ExtractDefinitionSpans(tree)
calls := gotreesitter.ExtractCalls(tree)
bases := gotreesitter.ExtractHeritage(tree)

enclosing, ok := tree.EnclosingDefinition(offset)

These helpers cover common Go, JavaScript, TypeScript/TSX, Python, Starlark, and Java definitions and calls, plus JavaScript/TypeScript, Python, and Java heritage edges. They skip unsupported languages or ambiguous shapes conservatively, rather than guess.

Taproot DSL harness

The taproot package is a small front-end harness for grammargen-backed DSLs. It caches generated or blob-loaded languages, parses source, returns a Walker with common CST helpers, and reports syntax errors while still returning the partial root for diagnostics:

root, walker, err := taproot.ParseFromBlob("dsl", blob, buildGrammar, src)
if err != nil {
    fmt.Println(err)
}
fmt.Println(walker.Type(root))

Use taproot.LanguageFromBlob when a DSL embeds a generated grammar blob but still wants a source-grammar fallback during development.

Queries
q, _ := gotreesitter.NewQuery(`(function_declaration name: (identifier) @fn)`, lang)
cursor := q.Exec(tree.RootNode(), lang, src)

for {
    match, ok := cursor.NextMatch()
    if !ok {
        break
    }
    for _, cap := range match.Captures {
        fmt.Println(cap.Node.Text(src))
    }
}

The query engine supports the full S-expression pattern language: structural quantifiers (?, *, +), alternation ([...]), field constraints, negated fields, anchor (!), and all standard predicates. See Query API.

Typed query codegen

Generate type-safe Go wrappers from .scm query files:

go run ./cmd/tsquery -input queries/go_functions.scm -lang go -output go_functions_query.go -package queries

Given a query like (function_declaration name: (identifier) @name body: (block) @body), tsquery generates:

type FunctionDeclarationMatch struct {
    Name *gotreesitter.Node
    Body *gotreesitter.Node
}

q, _ := queries.NewGoFunctionsQuery(lang)
cursor := q.Exec(tree.RootNode(), lang, src)
for {
    match, ok := cursor.Next()
    if !ok { break }
    fmt.Println(match.Name.Text(src))
}

Multi-pattern queries generate one struct per pattern with MatchPatternN conversion helpers.

Multi-language documents (injection parsing)

Parse documents with embedded languages (HTML+JS+CSS, Markdown+code fences, Vue/Svelte templates):

ip := gotreesitter.NewInjectionParser()
ip.RegisterLanguage("html", htmlLang)
ip.RegisterLanguage("javascript", jsLang)
ip.RegisterLanguage("css", cssLang)
ip.RegisterInjectionQuery("html", injectionQuery)

result, _ := ip.Parse(source, "html")

for _, inj := range result.Injections {
    fmt.Printf("%s: %d ranges\n", inj.Language, len(inj.Ranges))
    // inj.Tree is the child language's parse tree
}

It supports static (#set! injection.language "javascript") and dynamic (@injection.language capture) language detection, recursive nested injections, and incremental reparse with child tree reuse.

Source rewriting

Collect source-level edits and apply them atomically, producing InputEdit records for incremental reparse:

rw := gotreesitter.NewRewriter(src)
rw.Replace(funcNameNode, []byte("newName"))
rw.InsertBefore(bodyNode, []byte("// added\n"))
rw.Delete(unusedNode)

newSrc, _ := rw.ApplyToTree(tree)
newTree, _ := parser.ParseIncremental(newSrc, tree)

Apply() returns both the new source bytes and the []InputEdit records. ApplyToTree() is a convenience method that calls tree.Edit() for each edit and returns source ready for ParseIncremental.

Incremental reparsing
tree, _ := parser.Parse(src)

// User types "x" at byte offset 42
src = append(src[:42], append([]byte("x"), src[42:]...)...)

tree.Edit(gotreesitter.InputEdit{
    StartByte:   42,
    OldEndByte:  42,
    NewEndByte:  43,
    StartPoint:  gotreesitter.Point{Row: 3, Column: 10},
    OldEndPoint: gotreesitter.Point{Row: 3, Column: 10},
    NewEndPoint: gotreesitter.Point{Row: 3, Column: 11},
})

tree2, _ := parser.ParseIncremental(src, tree)

ParseIncremental walks the old tree's spine, identifies the edit region, and reuses unchanged content by reference. For an admitted clean edit, re-lex/reparse work can stay close to the invalidated region: reuse covers leaf nodes, the unchanged suffix, the untouched root, and unchanged top-level siblings after the edit. Admission for top-level sibling reuse is per node: a fragility bit plus byte-range equality, not a language allowlist.

That parse reuse is not an absolute O(edit) guarantee. For a length- or point-changing edit, Tree.Edit must also update byte and point coordinates through affected trailing sibling subtrees. The measured cost model is therefore cheap edited-region parse work plus sibling-offset maintenance that grows linearly with the number of trailing nodes or compact entries whose coordinates move. Same-length edits whose points do not move avoid that tail shift. Interior, non-top-level subtree reuse does not ship yet; it stays on the roadmap. A production tree first gets one narrow, independently reauthenticated same-length token-invariant leaf fast path. If that path declines, a language with an external scanner enters general old-tree reuse only when the scanner is certified; otherwise it uses the production full-parse fallback. See the per-language incremental scanner matrix.

When no edit has occurred, ParseIncremental detects the nil-edit on a pointer check and returns in single-digit nanoseconds with zero allocations.

UTF-16 input and editor coordinates

UTF-16 callers can parse Go-native code units or endian-specific byte buffers without converting offsets by hand. The parser core keeps its canonical UTF-8 view internally, while the returned tree retains the original UTF-16 source and maps nodes, edits, included ranges, query filters, highlights, tags, and injections back to UTF-16 code-unit coordinates.

src := utf16.Encode([]rune("1+2"))

parser := gotreesitter.NewParser(lang)
tree, _ := parser.ParseUTF16(src)

rng, _ := tree.UTF16RangeForNode(tree.RootNode())
fmt.Println(rng.StartCodeUnit, rng.EndCodeUnit)

node := tree.DescendantForUTF16Range(0, uint32(len(src)))
_ = node

// Incremental edits can be described in UTF-16 code units.
next := utf16.Encode([]rune("1+3"))
tree.EditUTF16(gotreesitter.UTF16Edit{
    StartCodeUnit:  2,
    OldEndCodeUnit: 3,
    NewEndCodeUnit: 3,
}, next)
tree2, _ := parser.ParseIncrementalUTF16(next, tree)
_ = tree2

UTF-16 byte input states its byte order explicitly:

tree, _ := parser.ParseUTF16Bytes(buf, gotreesitter.UTF16LittleEndian)

Editor-facing APIs have UTF-16 variants:

q, _ := gotreesitter.NewQuery(`(NUMBER) @number`, lang)
cursor := q.Exec(tree.RootNode(), lang, tree.Source())
cursor.SetUTF16Range(tree, 2, 3)

hl, _ := gotreesitter.NewHighlighter(lang, `(NUMBER) @number`)
highlightRanges := hl.HighlightUTF16(src)

tagger, _ := gotreesitter.NewTagger(lang, `(NUMBER) @name @definition.number`)
tags := tagger.TagUTF16(src)

Node byte APIs such as DescendantForByteRange still use the tree's canonical UTF-8 byte offsets. Use DescendantForUTF16Range, or convert with UTF8ByteForUTF16Offset when starting from editor UTF-16 offsets.

Tree cursor

TreeCursor maintains an explicit (node, childIndex) frame stack. Parent, child, and sibling movement run in O(1) with zero allocations — sibling traversal indexes directly into the parent's children[] slice.

c := gotreesitter.NewTreeCursorFromTree(tree)

c.GotoFirstChild()
c.GotoChildByFieldName("body")

for ok := c.GotoFirstNamedChild(); ok; ok = c.GotoNextNamedSibling() {
    fmt.Printf("%s at %d\n", c.CurrentNodeType(), c.CurrentNode().StartByte())
}

idx := c.GotoFirstChildForByte(128)

Movement methods: GotoFirstChild, GotoLastChild, GotoNextSibling, GotoPrevSibling, GotoParent, named-only variants (for example, GotoFirstNamedChild), field-based (GotoChildByFieldName, GotoChildByFieldID), and position-based (GotoFirstChildForByte, GotoFirstChildForPoint).

Cursors hold direct pointers into tree nodes. Recreate the cursor after Tree.Release(), Tree.Edit(...), or an incremental reparse.

Highlighting
hl, _ := gotreesitter.NewHighlighter(lang, highlightQuery)
ranges := hl.Highlight(src)

for _, r := range ranges {
    fmt.Printf("%s: %q\n", r.Capture, src[r.StartByte:r.EndByte])
}
Tagging
entry := grammars.DetectLanguage("main.go")
lang := entry.Language()

tagger, _ := gotreesitter.NewTagger(lang, entry.TagsQuery)
tags := tagger.Tag(src)

for _, tag := range tags {
    fmt.Printf("%s %s at %d:%d\n", tag.Kind, tag.Name,
        tag.NameRange.StartPoint.Row, tag.NameRange.StartPoint.Column)
}

Benchmarks

Canonical, linkable performance claims live in BENCH.md — the real-code full-parse matrix, historical incremental controls, the Go-vs-C fleet scoreboard, memory receipts, and the methodology that keeps them honest.

BenchmarkGoParseFullDFA calls Parser.Parse, requires a complete root, and releases the materialized tree. Its generated 500-function source is now a historical straight-LR control, not the representative full-parse headline. BenchmarkGoParseCoreDFA remains a parser-loop diagnostic that suppresses ordinary tree materialization.

A 2026-07-11 audit found that older versions of BenchmarkGoParseFullDFA called ParseNoResultCompatibilityBenchmarkOnly, which also enabled the no-tree path. The previously published 1.54 ms, 728 B/op, and 7 allocs/op figures — and v0.24.0's later 978 B/op and 5 allocs/op figures — therefore described parser-core diagnostics, not a full materialized parse. The project withdrew those headline comparisons. The corrected historical control later measured 10.907 ms on a pinned quiet host. A second audit found that this source never forks and that its C comparison used a different Go grammar from gotreesitter. The project withdrew the former 1.895x ratio and 29% materialization decomposition rather than promote them as representative claims.

The replacement canonical matrix freezes four clean, human-authored Go files spanning 5-236 KiB. They reach 12-18 live stacks and constructed-to-selected node ratios of 3.65-4.47. Admission requires exact deep parity against one oracle: upstream tree-sitter v0.25.1 commit f5afe475…, tree-sitter-go commit 2346a3ab…, compiled with -O2 into a fingerprinted static C artifact. The benchmark and parity lanes share those oracle sources and identity; see BENCH.md for the full contract and fixture hashes.

The first strict materialized real-Go publication receipt, shipped with v0.37.0, measured 5.481673x C by equal-fixture geomean and 6.313799x C for the fixed-suite sum of medians. Those corrected real-code results, rather than the withdrawn 1.895x straight-LR comparison, established the full-parse baseline; BENCH.md records every per-fixture median, RSS value, and receipt hash.

A strict v0.40.0 production receipt at 1935a42c measures public Parser.Parse at 4.851050x C by equal-fixture geomean and 5.472406x C by fixed-suite sum of medians, with a 5.608320x C worst fixture. The latest clean publication of the separately build-tagged, fail-closed selected-store candidate, at ba1ed1bf, measures 2.685181x C and 2.676794x C, respectively, with a 2.791974x C worst fixture and zero timed fallbacks. The measured lifecycle seals the accepted compact payloads into the indexed selected store, walks them through SelectedCursor, and releases the store. That candidate result is diagnostic: it authenticates visible gts-deep-tree-v1 structure for the four clean fresh-full fixtures, not parser-state metadata, recovery, incremental reuse, included ranges, or public Parser.Parse. See BENCH.md for the paired tables, exact identities, hashes, and support boundary.

The historical incremental measurements on the same generated 500-function Go workload were 649 ns for a one-byte edit and 2.43 ns for a no-edit reparse. They remain narrow workload-specific controls. The locked incremental matrix validates correctness and classifies work, but it does not establish a general comparative Go/C speed headline.

# Authenticated real-code Go full-parse lanes:
GOMAXPROCS=1 go test . -run '^$' \
  -bench '^BenchmarkGoParseWarmRealDFA$' \
  -benchmem -count=10 -benchtime=750ms

# Complete locked static-C publication receipt (clean, quiet, Docker host):
bash cgo_harness/pure_c/run_canonical_go_full_parse.sh --core <idle-cpu>

# Build-tagged compact fresh-full candidate (diagnostic, fail-closed):
bash cgo_harness/pure_c/run_canonical_go_full_parse.sh \
  --go-backend candidate --core <idle-cpu>

# Historical straight-LR full parse plus incremental API controls:
GOMAXPROCS=1 go test . -run '^$' \
  -bench 'BenchmarkGoParseFullDFA|BenchmarkGoParseIncrementalSingleByteEditDFA|BenchmarkGoParseIncrementalNoEditDFA' \
  -benchmem -count=10 -benchtime=750ms

# Parser-core attribution only:
GOMAXPROCS=1 go test . -run '^$' \
  -bench '^BenchmarkGoParseCoreDFA$' \
  -benchmem -count=10 -benchtime=750ms

Correctness and performance are gated separately. For Go-vs-C real-corpus timing, see cgo_harness/perf_scan; its ratchet records caveats, timeouts, and resource gaps instead of extrapolating from this generated-Go microbenchmark.

Benchmark matrix

For repeatable multi-workload tracking:

go run ./cmd/benchmatrix --count 10

This emits bench_out/matrix.json (machine-readable), bench_out/matrix.md (summary), and raw logs under bench_out/raw/. The default matrix includes a bounded, warmed language-family full-parse group, reported with MB/s so you can compare parser throughput across generated source sizes. Use --only-family to isolate that group, --family-unit-count to scale it, or --no-family for the narrower Go/editor matrix.

Supported languages

206 grammars ship in the registry. All 206 produce error-free parse trees on smoke samples. Run go run ./cmd/parity_report for current status.

  • 119 external scanners (hand-written Go implementations of upstream C scanners)
  • 7 hand-written Go token sources (authzed, c, cpp, go, java, json, lua)
  • Remaining languages use the DFA lexer generated from grammar tables
Parse quality

Each LangEntry carries a Quality field:

Quality Meaning
full All scanner and lexer components present. Parser has full access to the grammar.
partial Missing external scanner. DFA lexer handles what it can; external tokens are skipped.
none Cannot parse.

full means the parser has every component the grammar requires. It does not guarantee error-free trees on all inputs — grammars with high GLR ambiguity may produce syntax errors on very large or deeply nested constructs, because of parser safety limits (iteration cap, stack depth cap, node count cap). These limits scale with input size. Check tree.RootNode().HasError() at runtime.

Full language list (206)

ada, agda, angular, apex, arduino, asm, astro, authzed, awk, bash, bass, beancount, bibtex, bicep, bitbake, blade, brightscript, c, c_sharp, caddy, cairo, capnp, chatito, circom, clojure, cmake, cobol, comment, commonlisp, cooklang, corn, cpon, cpp, crystal, css, csv, cuda, cue, cylc, d, dart, desktop, devicetree, dhall, diff, disassembly, djot, dockerfile, dot, doxygen, dtd, earthfile, ebnf, editorconfig, eds, eex, elisp, elixir, elm, elsa, embedded_template, enforce, erlang, facility, faust, fennel, fidl, firrtl, fish, foam, forth, fortran, fsharp, gdscript, git_config, git_rebase, gitattributes, gitcommit, gitignore, gleam, glsl, gn, go, godot_resource, gomod, graphql, groovy, hack, hare, haskell, haxe, hcl, heex, hlsl, html, http, hurl, hyprlang, ini, janet, java, javascript, jinja2, jq, jsdoc, json, json5, jsonnet, julia, just, kconfig, kdl, kotlin, ledger, less, linkerscript, liquid, llvm, lua, luau, make, markdown, markdown_inline, matlab, mermaid, meson, mojo, move, nginx, nickel, nim, ninja, nix, norg, nushell, objc, ocaml, odin, org, pascal, pem, perl, php, pkl, powershell, prisma, prolog, promql, properties, proto, pug, puppet, purescript, python, ql, r, racket, regex, rego, requirements, rescript, robot, ron, rst, ruby, rust, scala, scheme, scss, smithy, solidity, sparql, sql, squirrel, ssh_config, starlark, svelte, swift, tablegen, tcl, teal, templ, textproto, thrift, tlaplus, tmux, todotxt, toml, tsx, turtle, twig, typescript, typst, uxntal, v, verilog, vhdl, vimdoc, vue, wat, wgsl, wolfram, xml, yaml, yuck, zig

Query API

Feature Status
Compile + execute (NewQuery, Execute, ExecuteNode) supported
Cursor streaming (Exec, NextMatch, NextCapture) supported
Structural quantifiers (?, *, +) supported
Alternation ([...]) supported
Field matching (name: (identifier)) supported
#eq? / #not-eq? supported
#match? / #not-match? supported
#any-of? / #not-any-of? supported
#lua-match? supported
#has-ancestor? / #not-has-ancestor? supported
#has-parent? / #not-has-parent? supported
#is? / #is-not? supported
#any-eq? / #any-not-eq? supported
#any-match? / #any-not-match? supported
#select-adjacent! supported
#strip! supported
#set! / #offset! directives parsed and accepted
SetValues (read #set! metadata from matches) supported

All shipped highlight and tags queries compile (156/156 highlight, 69/69 tags).

Repository layout note: the compat tier

The root package carries roughly 180 parser_result_<language>*.go files — the C-faithful result-normalization tier that reshapes raw GLR output into the exact tree the C runtime selects. It is internal plumbing, oracle-gated per witness, and it shrinks as certified engine mechanisms subsume shims. See docs/compat-tier.md.

For a maintainer-oriented map of the full root package, ownership seams, review-sensitive hotspots, and local artifact policy, see docs/repository-map.md. The ordered, receipt-driven cleanup program is documented in docs/root-normalization-retirement.md.

Known limitations

  • Full-parse throughput: the strict materialized real-Go production receipt at the v0.40.0 tag target 1935a42c measures public Parser.Parse at 4.851050x C by equal-fixture geomean and 5.472406x C by fixed-suite sum of medians against the exact static -O2 oracle (see BENCH.md); its worst fixture is 5.608320x C. The compact candidate's lower diagnostic ratio is not a public-parser claim. The former ~2.1x row used a straight-LR synthetic and a different Go grammar, so it stays historical only. The locked incremental matrix validates correctness and classifies work, but general incremental Go/C performance has no current publication-grade headline. Full-parse throughput varies by grammar and corpus shape; GLR-heavy code, highly ambiguous languages, and very large generated files remain the main performance frontier.
  • GLR safety caps: the parser enforces iteration, stack depth, and node count limits proportional to input size. These prevent pathological blowup on grammars with high ambiguity, but they impose a ceiling on the maximum input complexity that parses without error. The caps are tunable but not removable without risking unbounded resource consumption.

Adding a language

Adding a language to your own project does not require forking this repo or following the in-tree steps below. See docs/authoring-languages.md for the out-of-tree pipeline (grammar.json → blob → LoadLanguage / RegisterExtension) and docs/external-scanners.md for external scanners in Go. The steps below are for grammars embedded in this repo.

  1. Add the grammar repo to grammars/languages.manifest
  2. Refresh pinned refs in grammars/languages.lock: go run ./cmd/grammar_updater -lock grammars/languages.lock -write -report grammars/grammar_updates.json
  3. Generate tables: go run ./cmd/ts2go -manifest grammars/languages.manifest -outdir ./grammars -package grammars -compact=true
  4. Add smoke samples to cmd/parity_report/main.go and grammars/parse_support_test.go
  5. Verify: go run ./cmd/parity_report && go test ./grammars/...

Grammar lock updates

  • grammars/languages.lock stores pinned refs for grammar update + parity automation.
  • cmd/grammar_updater refreshes refs and emits a machine-readable report.
  • .github/workflows/grammar-lock-update.yml opens scheduled/dispatch update PRs.
  • Hand-written scanner ports can also declare ExternalScannerSpec metadata with upstream source hashes and external-token names. When a grammar update changes src/scanner.c or the external-token list, treat it as scanner work: update the Go scanner binding/port before replacing generated blobs. A grammar JSON-only change with unchanged externals can usually follow the normal grammar.json -> grammargen Go DSL -> blob -> parity path.

Manual refresh:

go run ./cmd/grammar_updater \
  -lock grammars/languages.lock \
  -allow-list grammars/update_tier1_core100.txt \
  -max-updates 10 \
  -write \
  -report grammars/grammar_updates.json

Architecture

gotreesitter is a ground-up reimplementation of the tree-sitter runtime in Go. No code is shared with, or translated from, the C implementation.

Parser — Table-driven LR(1) with GLR fallback. When a (state, symbol) pair maps to multiple actions in the parse table, the parser forks the stack and explores all alternatives in parallel. Stack merging collapses equivalent paths. Safety limits (iteration count, stack depth, node count) scale with input size and prevent runaway exploration on ambiguous grammars.

Incremental engine — Walks the edit region of the previous tree and reuses unchanged content by reference. Reuse covers leaf nodes, the unchanged suffix, the untouched root, and — on a clean old tree — unchanged top-level siblings after the edited node. Admission for top-level sibling reuse is per node: a fragility bit plus byte-range equality, not a language allowlist. On admitted clean edits, parse work can stay close to the invalidated region, but a length- or point-changing Tree.Edit still performs sibling-offset maintenance proportional to the trailing nodes or compact entries whose coordinates move. This measured model is not an absolute O(edit) guarantee. Interior, non-top-level subtree reuse does not ship yet; it stays on the roadmap. A production tree may first take the narrow independently reauthenticated same-length token-invariant leaf fast path. Once that path declines, external-scanner languages use general old-tree reuse only when their scanner explicitly certifies it, with boundary checkpoints where configured; uncertified scanners use the production full-parse fallback documented in the scanner matrix.

Lexer — Two paths. ts2go generates a DFA lexer from the grammar's lex tables, and it handles most languages. For grammars where the DFA is not enough (for example, Go's automatic semicolons, or YAML's indentation-sensitive structure), hand-written Go token sources implement the TokenSource interface directly.

External scanners — 119 registered grammars require external scanners for context-sensitive tokens (Python indentation, HTML implicit close tags, Rust raw string delimiters, Swift operator disambiguation, and similar cases). Each scanner is a hand-written Go implementation of the grammar's ExternalScanner interface: Create, Serialize, Deserialize, Scan. Certified checkpoint-enabled scanners snapshot state at external-token boundaries so incremental reuse can restore compatible state. After the narrow independently reauthenticated same-length token-invariant leaf path declines, an uncertified scanner fails closed to a fresh production parse; certification and fallback behavior are explicit in the per-language matrix.

Arena allocator — Nodes are allocated from slab-based arenas to reduce GC pressure. Arenas are released in bulk when a tree is freed.

Query engine — S-expression pattern compiler with predicate evaluation and streaming cursor iteration. It supports all standard tree-sitter predicates (#eq?, #match?, #any-of?, #has-ancestor?, and similar predicates) and directive annotations (#set!, #offset!, #select-adjacent!, #strip!).

Injection parser — Orchestrates multi-language parsing. It runs injection queries against a parent tree to find embedded regions, spawns child parsers with SetIncludedRanges(), and recurses for nested injections. Incremental reparse reuses unchanged child trees.

Rewriter — Collects source-level edits (replace, insert, delete) targeting byte ranges, applies them atomically, and produces InputEdit records for incremental reparse. It validates edits for non-overlap and applies them in a single pass.

Grammar loading — ts2go extracts parse tables, lex tables, field maps, symbol metadata, and external token lists from upstream parser.c files. These are serialized to compressed binary blobs under grammars/grammar_blobs/ and lazy-loaded through loadEmbeddedLanguage() with an LRU cache. String and transition interning reduce memory footprint across loaded grammars. Grammargen-backed blobs use the same CLI surface; for example, you can regenerate the Go blob with go run ./cmd/grammargen -lr-split -bin grammars/grammar_blobs/go.bin go. When loading a raw blob yourself, prefer grammars.LoadLanguage(name, blob) over gotreesitter.LoadLanguage(blob), so the runtime attaches the registered external scanner and external lex-state support for that language automatically.

Build tags and environment

External grammar blobs (avoid embedding in the binary):

go build -tags grammar_blobs_external
GOTREESITTER_GRAMMAR_BLOB_DIR=/path/to/blobs  # required
GOTREESITTER_GRAMMAR_BLOB_MMAP=false           # disable mmap (Unix only)

Curated language set (smaller binary):

go build -tags grammar_set_core  # curated Core100 embedded grammar set
GOTREESITTER_GRAMMAR_SET=go,json,python  # runtime restriction

Selective embedded grammars (smallest self-contained binary — pick exactly the languages you ship):

# Embeds ONLY go.bin + java.bin into the binary (everything else is dropped at
# link time). No GOTREESITTER_GRAMMAR_BLOB_DIR needed — still a single static binary.
go build -tags 'grammar_subset grammar_subset_go grammar_subset_java'

Add one grammar_subset_<lang> tag per grammar you need (names match the blob file: grammar_subset_c_sharp, grammar_subset_python, and so on). A single-language build drops from ~24MB to a few MB. This is finer-grained than grammar_set_core (a fixed set) and, unlike grammar_blobs_external, it keeps the blobs embedded. Pairing grammar_subset with grammar_blobs_external instead loads the selected blobs from GOTREESITTER_GRAMMAR_BLOB_DIR at runtime (no embedded blobs at all).

The four embedding modes are mutually exclusive at the build-tag level: default (all embedded) · grammar_set_core (Core100 embedded) · grammar_subset + grammar_subset_<lang> (selected embedded) · grammar_blobs_external (none embedded). Regenerate the per-language embed files after adding a grammar with go run ./cmd/gen_subset_blob_embeds.

Grammar cache tuning (long-lived processes):

grammars.SetEmbeddedLanguageCacheLimit(8)    // LRU cap
grammars.UnloadEmbeddedLanguage("rust.bin")  // drop one
grammars.PurgeEmbeddedLanguageCache()        // drop all

fmt.Println(lang.Size())                      // approximate decoded table bytes

GOTREESITTER_GRAMMAR_CACHE_LIMIT=8       # LRU cap via env
GOTREESITTER_GRAMMAR_IDLE_TTL=5m         # evict after idle
GOTREESITTER_GRAMMAR_IDLE_SWEEP=30s      # sweep interval
GOTREESITTER_GRAMMAR_COMPACT=true        # loader compaction (default)
GOTREESITTER_GRAMMAR_STRING_INTERN_LIMIT=200000
GOTREESITTER_GRAMMAR_TRANSITION_INTERN_LIMIT=20000

GLR stack cap override:

GOT_GLR_MAX_STACKS=8  # overrides default GLR stack cap (default: 8)

The default is tuned for correctness. Increase it only if a grammar or workload needs more GLR alternatives to preserve parity.

Legacy benchmark compatibility only:

GOT_PARSE_NODE_LIMIT_SCALE=3

GOT_PARSE_NODE_LIMIT_SCALE is needed only for comparisons against older truncation-prone benchmark baselines. Keep it unset on current branches.

Testing

bash cgo_harness/docker/run_single_grammar_parity.sh typescript

For local correctness/parity work, prefer isolated one-language Docker runs:

# Real-corpus parity for one grammar
bash cgo_harness/docker/run_single_grammar_parity.sh typescript

# Focused grammargen real-corpus lane for one language
bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode real-corpus --langs typescript

# Focused grammargen-vs-C lane for one language
bash cgo_harness/docker/run_grammargen_focus_targets.sh --mode cgo --langs typescript

run_grammargen_focus_targets.sh is the safest local lane for high-value grammars: it runs one grammar per container and defaults to a single-worker profile (--cpus 1, --pids 512, GOMAXPROCS=1, GOFLAGS=-p=1).

For Fortran, both real-corpus runners also default to a tighter bounded local preset unless you explicitly override it or pass --unsafe-fortran-defaults: --memory 3g, --cpus 1, --pids 512, GOMAXPROCS=1, GOFLAGS=-p=1, GOT_LALR_LR0_CORE_BUDGET=160000000, and GTS_GRAMMARGEN_REAL_CORPUS_GENERATE_TIMEOUT=15m.

If you only need a fast package-local regression check, keep it in Docker and narrow the -run regex:

bash cgo_harness/docker/run_parity_in_docker.sh \
  -- "cd /workspace && go test ./grammargen -run '^TestTypeScriptConditionalTypeParity$' -count=1"

Avoid go test ./... and host-side multi-language or race sweeps on developer machines while chasing OOMs. Use CI or a dedicated container when you need broader race coverage.

Other focused correctness/parity commands:

# Top-50 smoke correctness for the grammars package only
bash cgo_harness/docker/run_parity_in_docker.sh \
  -- "cd /workspace && go test ./grammars -run '^TestTop50(ParseSmokeNoErrors|CorrectnessListMatchesLockFile)$' -count=1 -v"

# Top-50 grammargen import/parity registry coverage
bash cgo_harness/docker/run_parity_in_docker.sh \
  -- "cd /workspace && go test ./grammargen -run '^TestTop50GrammarImportParityCoverage$' -count=1 -v"

# C-oracle parity suites inside the cgo harness
bash cgo_harness/docker/run_parity_in_docker.sh \
  --run '^TestParityFreshParse$|^TestParityHasNoErrors$|^TestParityIssue3Repros$|^TestParityGLRCanaryGo$'
bash cgo_harness/docker/run_parity_in_docker.sh \
  --run '^TestParityCorpusFreshParse$'

CI may still run broader race coverage on hosted runners. Do not copy those commands onto a developer host during OOM diagnosis.

Test suite covers: smoke tests (206 grammars), golden S-expression snapshots, highlight query validation, query pattern matching, incremental reparse correctness, error recovery, GLR fork/merge, injection parsing, source rewriting, and fuzz targets.

Roadmap

The current release is v0.48.1. It restores reference-compatible field names for anonymous Python separator nodes. It also keeps the v0.48.0 Swift and parser performance improvements.

Detailed shipped evidence lives in CHANGELOG.md. Planned minor releases are batched on Thursdays; the immutable-tag process and exceptions are documented in docs/releasing.md.

Now — cleanup, ownership, and explainability
  • Correctness, portability, and supported parser depth are banked. Preserve their receipts and keep correctness gates distinct from the currently advisory performance gates.
  • Prioritize repository maintenance: remove obsolete experiments and telemetry, reduce duplication, clarify subsystem ownership, improve documentation, and keep ownership receipts current.
  • Retire result-normalization shims only after the authoritative parser, scanner, materializer, or incremental mechanism owns the behavior and the required route receipts prove the shim inert. Follow the compat-tier retirement guide for the mechanical retirement contract.
  • Keep the authenticated public benchmarks as regression signals and preserve their historical claims. Parser-core, no-tree, compact-candidate, and synthetic lanes remain diagnostic rather than public performance claims.
  • After cleanup, the next major performance milestone is public Parser.Parse at no more than 1.5x C on the locked canonical real-code benchmark. It is a future target, not a current gate or achieved result.
Measured memory boundary
  • A production frozen-tree store remains closed by the current measurements: whole-tree conversion does not recover enough full-parse time and regresses incremental reuse. Do not treat pointer-light migration as an authorized next step without new evidence that clears those gates.
  • Bounded semispace retention, earlier reclamation of unselected construction debris, and tighter parser-budget-to-process-RSS behavior remain viable experiments. Each must preserve query, cursor, and incremental semantics.
Deferred — Go-native experience and broader architecture
  • Context-aware Engine/ParseRequest, typed limits and stop reasons, immutable registries/profiles, and internal pooling/diagnostic modes.
  • Document/Snapshot ownership for edits, changed ranges, cached queries, highlights, tags, injections, and release lifetimes.
  • Changed-range analysis plans, a provenance-rich sectioned grammar bundle, stronger scanner conformance tooling, and measured optional AOT tiers.

License

MIT

Documentation

Overview

Package gotreesitter implements a pure Go tree-sitter runtime.

This file defines the core data structures that mirror tree-sitter's TSLanguage C struct and related types. They form the foundation on which the lexer, parser, query engine, and syntax tree are built.

Index

Constants

View Source 
const (
	// RuntimeLanguageVersion is the maximum tree-sitter language version this
	// runtime is known to support.
	RuntimeLanguageVersion uint32 = 15
	// MinCompatibleLanguageVersion is the minimum accepted language version.
	MinCompatibleLanguageVersion uint32 = 13
)
View Source 
const (
	ConflictPolicyAnyState     StateID = ^StateID(0)
	ConflictPolicyAnyLookahead Symbol  = ^Symbol(0)
)

ConflictPolicyAnyState and ConflictPolicyAnyLookahead are sentinel State/ Lookahead values matching every state or every lookahead symbol instead of one exact table row. They support policies scoped by state and/or reduce symbol identity when enumerating every reachable row would add lookup cost without strengthening the action-shape check. Built-in wildcard policies are hand-certified and blob-SHA-pinned; callers supplying ConflictPolicies directly are responsible for scoping their own wildcard policies safely.

Variables

View Source 
var (
	// ErrInvalidUTF16ByteLength is returned when a UTF-16 byte source has a
	// dangling trailing byte.
	ErrInvalidUTF16ByteLength = errors.New("utf16: byte source length must be even")

	// ErrInvalidUTF16ByteOrder is returned for an unknown UTF-16ByteOrder.
	ErrInvalidUTF16ByteOrder = errors.New("utf16: invalid byte order")

	// ErrInvalidUTF16Range is returned when a UTF-16 range does not align to
	// valid code-point boundaries or has an inverted span.
	ErrInvalidUTF16Range = errors.New("utf16: invalid range")
)
View Source 
var DebugDFA atomic.Bool

DebugDFA enables trace logging for DFA token production.

Use `DebugDFA.Store(true/false)` to toggle at runtime.

View Source 
var ErrNoLanguage = errors.New("parser has no language configured")

ErrNoLanguage is returned when a Parser has no language configured.

View Source 
var ErrNoTokenSource = errors.New("parser has no token source")

ErrNoTokenSource is returned when a token-source parse is called without a token source.

View Source 
var ErrNoTokenSourceFactory = errors.New("parser has no token source factory")

ErrNoTokenSourceFactory is returned when a factory-based parse is called without a token source factory.

View Source 
var ErrParseStoppedEarly = errors.New("parse stopped before accepting input")

ErrParseStoppedEarly is matched by ParseStoppedEarlyError when a strict parse returns a partial tree.

Functions

func AdmissionCandidateCounters added in v0.46.0 

func AdmissionCandidateCounters() (routed, fallbacks uint64)

AdmissionCandidateCounters returns the number of full parses the compact candidate route served, and the number of eligible full parses that fell back to production.

func AdmissionCandidateLastFallbackReason added in v0.46.0 

func AdmissionCandidateLastFallbackReason() string

AdmissionCandidateLastFallbackReason returns the most recent candidate-route decline detail, or the empty string if none has been recorded.

func AdmissionCandidateRouteDefault added in v0.46.0 

func AdmissionCandidateRouteDefault() bool

AdmissionCandidateRouteDefault reports the current process-wide default.

func DecodeLargeStateGotosTrailer added in v0.23.1 

func DecodeLargeStateGotosTrailer(r *bytes.Reader) (map[uint64]StateID, error)

DecodeLargeStateGotosTrailer reads a trailer written by EncodeLargeStateGotosTrailer from r, which must be positioned immediately after a Language's gob message within the same decompressed blob stream (callers must gob-decode from a *bytes.Reader, not directly from a gzip.Reader: gob's Decoder can read ahead past its own message boundary on a streaming reader, silently discarding trailer bytes it never needed -- bytes.Reader has no such read-ahead and leaves r positioned exactly at the end of the decoded message).

It returns (nil, nil) when r has no remaining bytes -- the common case for every blob whose Language never populated LargeStateGotos, including every blob encoded before this trailer mechanism existed.

func DecodeUTF16Bytes added in v0.16.0 

func DecodeUTF16Bytes(source []byte, order UTF16ByteOrder) ([]uint16, error)

DecodeUTF16Bytes decodes an endian-specific UTF-16 byte source into Go UTF-16 code units.

func DrainArenaPools added in v0.14.0 

func DrainArenaPools()

DrainArenaPools releases all cached arenas from both incremental and full-parse pools. Arenas held in the pool are strong Go references and are not collected by the GC until explicitly drained or the process exits.

Call this after a large batch scan (e.g. after WalkAndParse returns) to allow the GC to reclaim the arena memory. The next parse will allocate a fresh arena.

func EnableArenaBreakdown added in v0.18.0 

func EnableArenaBreakdown(enabled bool)

EnableArenaBreakdown toggles detailed arena accounting for subsequently acquired arenas. It is intended for diagnostics and benchmark attribution; normal parser paths leave it disabled to avoid perturbing hot allocation paths.

func EnableArenaProfile added in v0.6.0 

func EnableArenaProfile(enabled bool)

EnableArenaProfile toggles arena pool counters. This debug hook is not concurrency-safe and is intended for single-threaded benchmark/profiling runs.

func EnableGLREquivAudit added in v0.19.0 

func EnableGLREquivAudit(enabled bool)

EnableGLREquivAudit toggles lightweight GLR equivalence attribution. This is intended for parser gap diagnostics and avoids the heavier survivor maps used by EnableRuntimeAudit.

func EnableRuntimeAudit added in v0.7.0 

func EnableRuntimeAudit(enabled bool)

EnableRuntimeAudit toggles per-parse survivor instrumentation. This debug hook is intended for single-threaded benchmark/profiling runs.

func EncodeLanguageBlob added in v0.23.1 

func EncodeLanguageBlob(lang *Language) ([]byte, error)

EncodeLanguageBlob serializes lang in the runtime's stable grammar-blob format. Languages without LargeStateGotos retain the legacy gzip+gob wire representation byte-for-byte. When LargeStateGotos is populated, the map is removed from a shallow exported-field copy, encoded as a sorted trailer, and wrapped in the versioned fail-closed envelope understood by LoadLanguage.

Keeping this encoder in the root package makes the format invariant shared by every producer, including grammargen and ts2go. The input Language is never mutated and can remain in concurrent use while it is encoded.

func EncodeLargeStateGotosTrailer added in v0.23.1 

func EncodeLargeStateGotosTrailer(m map[uint64]StateID) ([]byte, error)

EncodeLargeStateGotosTrailer serializes m as a self-contained gob stream of (key, target) pairs sorted by Key ascending, suitable for deterministic appending after a Language's gob-encoded blob payload (see DecodeLargeStateGotosTrailer). It returns (nil, nil) for an empty map so callers can skip writing a trailer entirely, leaving the blob byte-for-byte identical to one for a Language that never populated LargeStateGotos.

func InferGeneratedRepeatAuxMetadata added in v0.21.0 

func InferGeneratedRepeatAuxMetadata(lang *Language)

InferGeneratedRepeatAuxMetadata fills GeneratedRepeatAux for older language blobs that predate the explicit metadata bit.

func ReadAllGzipWithSizeHint added in v0.43.1 

func ReadAllGzipWithSizeHint(r io.Reader, compressed []byte) ([]byte, error)

ReadAllGzipWithSizeHint reads all of r — an open gzip.Reader positioned at the start of the member whose raw (still-compressed) bytes are compressed — into memory, pre-sizing the destination buffer from the gzip ISIZE trailer (the last 4 bytes of compressed) instead of letting io.ReadAll grow the buffer by repeated doubling. ISIZE is the uncompressed size mod 2^32, which is exact for every blob under 4 GB; grammar blobs are always far smaller. Falls back to io.ReadAll when the hint is missing, zero, or exceeds gzipSizeHintCap.

This matters at grammar-load time: io.ReadAll's doubling growth roughly doubles peak transient allocation versus the final size, and for the largest shipped grammar blobs that transient churn is measured in hundreds of MB to low GB (observed via alloc-space pprof on Language() calls), which is what actually trips container memory limits even though the final retained Language is smaller.

func RegisterHighlighterInjection added in v0.7.0 

func RegisterHighlighterInjection(parentLanguage string, spec HighlighterInjectionSpec)

RegisterHighlighterInjection registers nested-highlighting configuration for a parent language name (for example "markdown").

func RepairNoLookaheadLexModes added in v0.9.0 

func RepairNoLookaheadLexModes(lang *Language)

RepairNoLookaheadLexModes marks parser states as no-lookahead when they only need EOF-triggered reductions plus external/trivia handling. Tree-sitter's C runtime uses these states to reduce before lexing the next real token.

func ResetArenaProfile added in v0.6.0 

func ResetArenaProfile()

ResetArenaProfile resets arena pool counters. This debug hook is not concurrency-safe and is intended for single-threaded benchmark/profiling runs.

func ResetParseEnvConfigCacheForTests added in v0.7.0 

func ResetParseEnvConfigCacheForTests()

ResetParseEnvConfigCacheForTests clears memoized parser env config.

Tests in this repo mutate env vars between cases; this helper ensures subsequent parses observe the new values in the same process.

glrFaithfulCapOneMerge is intentionally excluded: unlike the values below, it is a process-start mode rather than a memoized config value. Tests that override it must restore the variable directly. Mixing that mode into this cache reset made cleanup order observable when t.Setenv restored the environment after ResetParseEnvConfigCacheForTests ran.

func ResetPerfCounters added in v0.6.0 

func ResetPerfCounters()

func RunExternalScanner 

func RunExternalScanner(lang *Language, payload any, lexer *ExternalLexer, validSymbols []bool) bool

RunExternalScanner invokes the language's external scanner if present. Returns true if the scanner produced a token, false otherwise.

func SetAdmissionCandidateRouteDefault added in v0.46.0 

func SetAdmissionCandidateRouteDefault(enabled bool)

SetAdmissionCandidateRouteDefault sets the process-wide default the Phase-3 admission switch applies to Parsers with no explicit override. A per-Parser override still wins.

The candidate route does not enforce the automatic large-input memory budget at scheduler granularity, so the switch declines every input at or above the source-length floor where the production route arms that budget (parseRuntimeMemoryMinSourceBytes). Such inputs stay on production and honor ParseStopMemoryBudget. Explicit timeouts, cancellation flags, included ranges, and observability hooks also keep a parse on production.

func SetGLRForestEnabled added in v0.20.0 

func SetGLRForestEnabled(on bool)

SetGLRForestEnabled toggles the GSS-forest path at runtime (tests/benchmarks).

func SetGLRForestRecover added in v0.21.0 

func SetGLRForestRecover(on bool)

SetGLRForestRecover toggles experimental forest error recovery (tests).

func SetInternLeavesObserveEnabled added in v0.20.0 

func SetInternLeavesObserveEnabled(on bool)

SetInternLeavesObserveEnabled toggles leaf-interning observation at runtime. Tests and benches that want to A/B observation without re-running the test binary set this directly. Not safe to flip while a parse is in flight on another goroutine. Phase 2 scaffolding; the API may change before becoming public.

func SetInternLeavesSubstituteEnabled added in v0.20.0 

func SetInternLeavesSubstituteEnabled(on bool)

SetInternLeavesSubstituteEnabled toggles canonical substitution at runtime. See internLeavesSubstituteEnabled.

func SetRawShapeElisionDisabledForDiagnostics added in v0.27.0 

func SetRawShapeElisionDisabledForDiagnostics(disabled bool)

SetRawShapeElisionDisabledForDiagnostics toggles rawShapeElisionDisabledForDiagnostics. See its doc comment: this is a diagnostics-only hook, not part of the parser's production behavior contract.

func UnwrapLanguageBlobEnvelope added in v0.23.1 

func UnwrapLanguageBlobEnvelope(data []byte) (compressed []byte, expectsTrailer bool, err error)

UnwrapLanguageBlobEnvelope returns the gzip payload and whether its envelope requires a non-empty LargeStateGotos trailer. Legacy gzip blobs are returned unchanged with expectsTrailer=false, preserving the original wire format.

func Walk 

func Walk(node *Node, fn func(node *Node, depth int) WalkAction)

Walk performs a depth-first traversal of the syntax tree rooted at node. The callback receives each node and its depth (0 for the starting node). Return WalkSkipChildren to skip a node's children, or WalkStop to end early.

func WrapLanguageBlobEnvelope added in v0.23.1 

func WrapLanguageBlobEnvelope(compressed []byte) ([]byte, error)

WrapLanguageBlobEnvelope wraps a gzip-compressed Language blob whose decompressed stream contains a LargeStateGotos trailer. The outer magic is deliberately not a gzip header: runtimes predating trailer support therefore reject the blob at gzip.NewReader instead of successfully decoding the gob payload with LargeStateGotos missing.

The envelope is versioned and length-delimited so current runtimes also fail closed on unknown versions, flags, truncation, and trailing bytes. Callers must only wrap blobs that actually contain a non-empty trailer.

Types

type AmbiguityKey added in v0.17.0 

type AmbiguityKey struct {
	State                          StateID
	Lookahead                      Symbol
	ActionCount                    uint8
	ShiftCount                     uint8
	ReduceCount                    uint8
	ReduceSymbol                   Symbol
	ChildCount                     uint8
	ProductionID                   uint16
	ReduceChainTerminalState       StateID
	ReduceChainTerminalActionClass uint8
}

AmbiguityKey identifies one parse-table ambiguity bucket.

type AmbiguityProfile added in v0.17.0 

type AmbiguityProfile struct {
	// contains filtered or unexported fields
}

AmbiguityProfile aggregates parser states/lookaheads that contribute to GLR fanout. It is intended for diagnostics and benchmark runs, not normal API use.

func NewAmbiguityProfile added in v0.17.0 

func NewAmbiguityProfile() *AmbiguityProfile

NewAmbiguityProfile creates an empty GLR ambiguity profile.

func (*AmbiguityProfile) Reset added in v0.17.0 

func (p *AmbiguityProfile) Reset()

Reset clears all accumulated ambiguity counters.

func (*AmbiguityProfile) SnapshotReduceChainTotals added in v0.19.0 

func (p *AmbiguityProfile) SnapshotReduceChainTotals() AmbiguityStat

SnapshotReduceChainTotals returns aggregate deterministic reduce-chain run counters across all profiled start states/lookaheads.

func (*AmbiguityProfile) SnapshotTop added in v0.17.0 

func (p *AmbiguityProfile) SnapshotTop(limit int) []AmbiguityStat

SnapshotTop returns the highest-impact ambiguity buckets ordered by stack pressure, then hit count.

func (*AmbiguityProfile) SnapshotTopMergeStates added in v0.19.0 

func (p *AmbiguityProfile) SnapshotTopMergeStates(limit int) []AmbiguityStat

SnapshotTopMergeStates returns parser states that most often participate in multi-stack merge passes. These rows are keyed by state only, because merge happens before the next lookahead dispatch.

func (*AmbiguityProfile) SnapshotTopReduceChainRuns added in v0.19.0 

func (p *AmbiguityProfile) SnapshotTopReduceChainRuns(limit int) []AmbiguityStat

SnapshotTopReduceChainRuns returns the starting states/lookaheads that begin the most expensive deterministic reduce chains.

func (*AmbiguityProfile) SnapshotTopReduceChains added in v0.19.0 

func (p *AmbiguityProfile) SnapshotTopReduceChains(limit int) []AmbiguityStat

SnapshotTopReduceChains returns the parser states/lookaheads that spent the most time in deterministic reduce-chain fusion.

type AmbiguityStat added in v0.17.0 

type AmbiguityStat struct {
	State                          StateID
	Lookahead                      Symbol
	ActionCount                    uint8
	ShiftCount                     uint8
	ReduceCount                    uint8
	ReduceSymbol                   Symbol
	ChildCount                     uint8
	ProductionID                   uint16
	Actions                        []ParseAction
	Hits                           uint64
	Forks                          uint64
	MultiStackHits                 uint64
	StackInTotal                   uint64
	StackInMax                     int
	ReduceChainHits                uint64
	ReduceChainSteps               uint64
	ReduceChainMaxLen              int
	ReduceChainNanos               int64
	ReduceChainRuns                uint64
	ReduceChainClassHits           uint64
	ReduceChainStopNoAction        uint64
	ReduceChainStopMulti           uint64
	ReduceChainStopShift           uint64
	ReduceChainStopAccept          uint64
	ReduceChainStopDead            uint64
	ReduceChainStopCycle           uint64
	ReduceChainStopLimit           uint64
	ReduceChainTerminalState       StateID
	ReduceChainTerminalActionClass uint8
	ActionNanos                    int64
	ExtraShiftNanos                int64
	NoActionNanos                  int64
	ConflictChoiceNanos            int64
	ConflictForkNanos              int64
	SingleShiftNanos               int64
	SingleReduceNanos              int64
	SingleAcceptNanos              int64
	SingleRecoverNanos             int64
	SingleOtherNanos               int64
	MergeCalls                     uint64
	MergeStacksIn                  uint64
	MergeStacksOut                 uint64
	MergeStacksInMax               int
	MergeStacksOutMax              int
}

AmbiguityStat is a snapshot row from AmbiguityProfile.

type ArenaBreakdown added in v0.18.0 

type ArenaBreakdown struct {
	NodeStructBytesAllocated            int64
	NodeFieldMetadataBytesAllocated     int64
	NoTreeNodeBytesAllocated            int64
	CompactFullLeafBytesAllocated       int64
	PendingParentBytesAllocated         int64
	PendingChildEntryBytesAllocated     int64
	RawShapeBytesAllocated              int64
	RawShapeChildBytesAllocated         int64
	FinalChildSidecarBytesAllocated     int64
	CompactCheckpointLeafBytesAllocated int64
	PendingChildEntriesAllocated        uint64
	PendingChildEntryCapacity           uint64
	PendingChildEntryWaste              uint64
	ChildSliceBytesAllocated            int64
	FieldIDBytesAllocated               int64
	FieldSourceBytesAllocated           int64
	MergeScratchBytesAllocated          int64

	ArenaNodesConstructed uint64
	// NodeLiveCount is arena allocation-slot usage, not root-reachable tree
	// liveness. It includes parser alternatives and recovery nodes allocated
	// during the parse.
	NodeLiveCount                     uint64
	NodeCapacityCount                 uint64
	NodeCapacityWaste                 uint64
	PrimaryNodeCapacity               uint64
	PrimaryNodeUsed                   uint64
	OverflowNodeCapacity              uint64
	OverflowNodeUsed                  uint64
	OverflowNodeSlabs                 uint64
	LargestNodeSlabUsedFraction       float64
	LeafNodesConstructed              uint64
	ParentNodesConstructed            uint64
	FieldedParentNodesConstructed     uint64
	UnfieldedParentNodesConstructed   uint64
	ParentConstructedChildLen0        uint64
	ParentConstructedChildLen1        uint64
	ParentConstructedChildLen2        uint64
	ParentConstructedChildLen3        uint64
	ParentConstructedChildLen4Plus    uint64
	ParentConstructedNoLinks          uint64
	ParentConstructedWithLinks        uint64
	ParentConstructedTrackErrors      uint64
	ParentConstructedFieldSources     uint64
	ParentReductionVisible            uint64
	ParentReductionInvisible          uint64
	ParentReductionVisibleFielded     uint64
	ParentReductionVisibleUnfielded   uint64
	ParentReductionInvisibleFielded   uint64
	ParentReductionInvisibleUnfielded uint64
	ParentReductionVisibleChildPtrs   uint64
	ParentReductionInvisibleChildPtrs uint64
	ParentReductionVisibleLen0        uint64
	ParentReductionVisibleLen1        uint64
	ParentReductionVisibleLen2        uint64
	ParentReductionVisibleLen3        uint64
	ParentReductionVisibleLen4Plus    uint64
	ParentReductionInvisibleLen0      uint64
	ParentReductionInvisibleLen1      uint64
	ParentReductionInvisibleLen2      uint64
	ParentReductionInvisibleLen3      uint64
	ParentReductionInvisibleLen4Plus  uint64
	ReduceChildSlicesFastGSS          uint64
	ReduceChildPointersFastGSS        uint64
	ReduceChildSlicesAllVisible       uint64
	ReduceChildPointersAllVisible     uint64
	ReduceChildSlicesScratchGeneral   uint64
	ReduceChildPointersScratchGeneral uint64
	ReduceChildSlicesScratchNoAlias   uint64
	ReduceChildPointersScratchNoAlias uint64
	CollapseRawUnaryAttempts          uint64
	CollapseRawUnarySuccesses         uint64
	CollapseRawUnaryMissShape         uint64
	CollapseRawUnaryMissGrammar       uint64
	CollapseRawUnaryMissChild         uint64
	CollapseRawUnaryMissRule          uint64
	CollapseUnaryAttempts             uint64
	CollapseUnarySuccesses            uint64
	CollapseUnaryMissShape            uint64
	CollapseUnaryMissGrammar          uint64
	CollapseUnaryMissFielded          uint64
	CollapseUnaryMissChild            uint64
	CollapseUnaryMissRule             uint64
	CollapseRuleSameSymbol            uint64
	CollapseRuleInvisibleWrapper      uint64
	CollapseRuleNamedLeafAlias        uint64
	NoTreeReduceNodesConstructed      uint64
	NoTreeLeafNodesConstructed        uint64
	NoTreePlaceholderNodesConstructed uint64
	OtherNodesConstructed             uint64
	ExtraNodesConstructed             uint64
	ErrorSymbolNodesConstructed       uint64
	HasErrorNodesConstructed          uint64
	ChildSlicesConstructed            uint64
	ChildPointersConstructed          uint64
	ChildSlicesLen1                   uint64
	ChildSlicesLen2                   uint64
	ChildSlicesLen3                   uint64
	ChildSlicesLen4Plus               uint64
	ParentChildPointersConstructed    uint64
	ParentChildrenLen0                uint64
	ParentChildrenLen1                uint64
	ParentChildrenLen2                uint64
	ParentChildrenLen3                uint64
	ParentChildrenLen4Plus            uint64
	FieldIDElementsConstructed        uint64
	FieldSourceElementsConstructed    uint64
}

ArenaBreakdown captures optional arena/materialization attribution. It is populated only when EnableArenaBreakdown(true) is set before parsing.

type ArenaProfile added in v0.6.0 

type ArenaProfile struct {
	IncrementalAcquire uint64
	IncrementalNew     uint64
	FullAcquire        uint64
	FullNew            uint64
}

ArenaProfile captures node arena allocation statistics. Enable with SetArenaProfileEnabled(true) and retrieve with GetArenaProfile().

func ArenaProfileSnapshot added in v0.6.0 

func ArenaProfileSnapshot() ArenaProfile

ArenaProfileSnapshot returns current arena pool counters. This debug hook is not concurrency-safe and is intended for single-threaded benchmark/profiling runs.

type BoundTree 

type BoundTree struct {
	// contains filtered or unexported fields
}

BoundTree pairs a Tree with its Language and source, eliminating the need to pass *Language and []byte to every node method call.

func Bind 

func Bind(tree *Tree) *BoundTree

Bind creates a BoundTree from a Tree. The Tree must have been created with a Language (via NewTree or a Parser). Returns a BoundTree that delegates to the underlying Tree's Language and Source.

func (*BoundTree) ChildByField 

func (bt *BoundTree) ChildByField(n *Node, fieldName string) *Node

ChildByField returns the first child assigned to the given field name.

func (*BoundTree) Language 

func (bt *BoundTree) Language() *Language

Language returns the tree's language.

func (*BoundTree) NodeText 

func (bt *BoundTree) NodeText(n *Node) string

NodeText returns the source text covered by the node.

func (*BoundTree) NodeType 

func (bt *BoundTree) NodeType(n *Node) string

NodeType returns the node's type name, resolved via the bound language.

func (*BoundTree) Release 

func (bt *BoundTree) Release()

Release releases the underlying tree's arena memory.

func (*BoundTree) RootNode 

func (bt *BoundTree) RootNode() *Node

RootNode returns the tree's root node.

func (*BoundTree) Source 

func (bt *BoundTree) Source() []byte

Source returns the tree's source bytes.

func (*BoundTree) TreeCursor added in v0.6.0 

func (bt *BoundTree) TreeCursor() *TreeCursor

TreeCursor returns a new TreeCursor starting at the tree's root node.

type ByteSkippableTokenSource 

type ByteSkippableTokenSource interface {
	TokenSource
	SkipToByte(offset uint32) Token
}

ByteSkippableTokenSource can jump to a byte offset and return the first token at or after that position.

type CRecoveryGateDiagnostics added in v0.21.0 

type CRecoveryGateDiagnostics struct {
	Supported bool
	Reason    string

	StateCount       int
	SymbolCount      int
	TokenCount       int
	LexModeCount     int
	LexStateCount    int
	ParseActionCount int

	HasExternalScanner     bool
	ExternalSymbolCount    int
	ExternalTokenCount     int
	ExternalLexStateRows   int
	ExternalLexStateMinLen int
}

CRecoveryGateDiagnostics describes the runtime validation result for the C recovery-cost competition gate. Reason is empty when Supported is true; otherwise it names the first failed validation check.

func CertifyCRecoveryCostCompetition added in v0.21.0 

func CertifyCRecoveryCostCompetition(lang *Language) CRecoveryGateDiagnostics

CertifyCRecoveryCostCompetition validates a language's runtime recovery surface and updates the C recovery metadata.

Capability means the tables satisfy the runtime gate. Default enablement is narrower: grammars with external recovery metadata are enabled only after an actual external scanner is attached and precise ExternalLexStates are available.

func CertifyGeneratedCRecoveryCostCompetition added in v0.21.0 

func CertifyGeneratedCRecoveryCostCompetition(lang *Language) CRecoveryGateDiagnostics

CertifyGeneratedCRecoveryCostCompetition is kept for existing generated grammar call sites. New callers should use CertifyCRecoveryCostCompetition.

func DiagnoseCRecoveryGate added in v0.21.0 

func DiagnoseCRecoveryGate(lang *Language) CRecoveryGateDiagnostics

DiagnoseCRecoveryGate validates the runtime table surface required by the faithful C recovery-cost competition path and returns the first failure.

The full validation scans every parse-table row, so the result is memoized per Language behind an input fingerprint (cRecoveryGateCacheKey): callers on parse-adjacent paths (errorCostCompetitionLanguage via gap-token replay and token-source construction) would otherwise redo an O(tables) scan per call, which measurably dominated error-heavy parses.

type CallRef added in v0.20.6 

type CallRef struct {
	Lang          string
	Kind          string
	Name          string
	Receiver      string
	NodeType      string
	StartByte     uint32
	EndByte       uint32
	NameStartByte uint32
	NameEndByte   uint32
}

CallRef is a compact language-neutral call-site reference extracted from a syntax tree.

func ExtractCalls added in v0.20.6 

func ExtractCalls(tree *Tree) []CallRef

ExtractCalls returns language-neutral call-site references for common code-understanding workflows.

type CheckpointedExternalScanner added in v0.47.0 

type CheckpointedExternalScanner interface {
	ExternalScanner
	UsesExternalScannerCheckpoints() bool
}

CheckpointedExternalScanner is implemented by stateful external scanners whose non-empty serialized payload is a complete checkpoint of every value that can affect a later Scan call. The runtime records that checkpoint at token boundaries and restores it when fast-forwarding across a reused subtree. A scanner may return a zero-length serialization for a reachable state that cannot be represented exactly; that boundary is recorded as checkpoint-absent and fails closed for incremental reuse.

Implementations must encode the empty payload as a non-empty byte sequence: a zero-length serialization is reserved by the checkpoint store to mean "checkpoint absent". A non-empty serialization must never truncate, alias, or otherwise collide with a different scanner state. Returning true is a soundness claim; an incomplete non-empty checkpoint can cause silent incremental corruption.

type CheckpointlessExternalScannerReuse added in v0.47.0 

type CheckpointlessExternalScannerReuse interface {
	ExternalScanner
	AllowsIncrementalReuseWithoutCheckpoint() bool
}

CheckpointlessExternalScannerReuse is implemented by checkpointed scanners that can additionally prove subtree reuse safe when a candidate node has no scanner checkpoint. Most stateful scanners must not implement this: absence of a checkpoint then fails closed.

type ConflictPolicy added in v0.21.0 

type ConflictPolicy struct {
	State     StateID
	Lookahead Symbol
	Kind      ConflictPolicyKind

	// ReduceSymbols, when non-empty, requires every reduce in the conflict to
	// reduce one of these symbols. The generic action-shape validator still
	// requires at least one reduce.
	ReduceSymbols []Symbol
}

ConflictPolicy describes one table row/lookahead conflict that can be collapsed deterministically after validating the action shape.

type ConflictPolicyKind added in v0.21.0 

type ConflictPolicyKind uint8

ConflictPolicyKind identifies a deterministic conflict policy class. 

const (
	ConflictPolicyNone ConflictPolicyKind = iota
	ConflictPolicyRepetitionShift
	ConflictPolicyShift
	// ConflictPolicyRecoveredRepetitionReduce is an exact-row certification
	// for a recovered lineage. It chooses the single reduce from a
	// {1 reduce, 1 repetition shift} row only after the lineage has previously
	// entered recovery and only while no recovery action is currently active.
	// Like other conflict policies, it is disabled during incremental reuse.
	//
	// Keep new kinds append-only: Language blobs encode these numeric values.
	ConflictPolicyRecoveredRepetitionReduce
	// ConflictPolicyRepetitionReduce is an exact-row certification for the
	// ordinary C repetition fold: one reduce wins over one repetition shift.
	ConflictPolicyRepetitionReduce
)

type DefinitionSpan added in v0.20.6 

type DefinitionSpan struct {
	Lang          string
	Kind          string
	Name          string
	NodeType      string
	StartByte     uint32
	EndByte       uint32
	NameStartByte uint32
	NameEndByte   uint32
}

DefinitionSpan is a compact language-neutral declaration span extracted from a syntax tree.

func EnclosingDefinition added in v0.20.6 

func EnclosingDefinition(tree *Tree, byteOffset uint32) (DefinitionSpan, bool)

EnclosingDefinition returns the nearest definition node that contains byteOffset. It uses NodeAtByte and parent links instead of scanning every definition span.

func ExtractDefinitionSpans added in v0.20.6 

func ExtractDefinitionSpans(tree *Tree) []DefinitionSpan

ExtractDefinitionSpans returns language-neutral declaration spans for common code-understanding workflows. The extractor is intentionally conservative: unsupported languages or declaration shapes are skipped rather than guessed.

type DiagnosticParserCoreBoundaryKind added in v0.46.0 

type DiagnosticParserCoreBoundaryKind string
const (
	DiagnosticParserCoreExtra         DiagnosticParserCoreBoundaryKind = "extra"
	DiagnosticParserCoreExtraChain    DiagnosticParserCoreBoundaryKind = "extra_chain"
	DiagnosticParserCoreNoAction      DiagnosticParserCoreBoundaryKind = "no_action"
	DiagnosticParserCoreRecovery      DiagnosticParserCoreBoundaryKind = "recovery"
	DiagnosticParserCoreAccept        DiagnosticParserCoreBoundaryKind = "accept_without_materialization"
	DiagnosticParserCoreCap           DiagnosticParserCoreBoundaryKind = "cap"
	DiagnosticParserCoreIdentity      DiagnosticParserCoreBoundaryKind = "identity"
	DiagnosticParserCoreRoute         DiagnosticParserCoreBoundaryKind = "unsupported_route"
	DiagnosticParserCoreGenericClosed DiagnosticParserCoreBoundaryKind = "generic_scheduler_closed"
)

type DiagnosticParserCoreDispatchRound added in v0.46.0 

type DiagnosticParserCoreDispatchRound struct {
	Index   int
	Before  []DiagnosticParserCoreHeaderReceipt
	Actions []DiagnosticParserCoreRoundAction
	After   []DiagnosticParserCoreHeaderReceipt
}

type DiagnosticParserCoreElection added in v0.46.0 

type DiagnosticParserCoreElection struct {
	States                 []StateID
	Token                  Token
	ScannerBefore          DiagnosticParserCoreScannerCheckpoint
	ScannerAfter           DiagnosticParserCoreScannerCheckpoint
	CurrentCheckpointValid bool
	CurrentCheckpointStart DiagnosticParserCoreScannerCheckpoint
	CurrentCheckpointEnd   DiagnosticParserCoreScannerCheckpoint
	CurrentCheckpointBytes [2]uint32
}

type DiagnosticParserCoreGenericAcceptance added in v0.46.0 

type DiagnosticParserCoreGenericAcceptance struct {
	ElectionIndex   int
	Token           Token
	Header          DiagnosticParserCoreHeaderPathReceipt
	Payloads        []uint32
	Score           int64
	BranchOrder     uint64
	HasBranchOrder  bool
	CoreWork        core.Work
	Accepts         uint64
	SelectedNodes   uint64
	SelectedParents uint64
	SelectedLeaves  uint64
	Stats           core.Stats
	Work            DiagnosticParserCoreGenericWork
}

DiagnosticParserCoreGenericAcceptance records an authenticated EOF accept after the compact frontier has converged to one exact derivation. Payloads are the selected bottom-to-top compact stack; materialization does not mutate that graph.

type DiagnosticParserCoreGenericCompletion added in v0.46.0 

type DiagnosticParserCoreGenericCompletion struct {
	TargetByte    uint32
	ElectionIndex int
	LastToken     Token
	State         StateID
	Headers       []DiagnosticParserCoreHeaderPathReceipt
	Stats         core.Stats
	Work          DiagnosticParserCoreGenericWork
}

DiagnosticParserCoreGenericCompletion is a caller-selected, successfully closed scheduler frontier. LastToken is consumed; no pending lookahead has been read.

type DiagnosticParserCoreGenericConflict added in v0.46.0 

type DiagnosticParserCoreGenericConflict struct {
	ElectionIndex            int
	Token                    Token
	HeaderIndex              int
	BranchOrderBefore        uint64
	BranchOrderAfter         uint64
	NextCreationSeqBefore    uint64
	NextCreationSeqAfter     uint64
	Round                    DiagnosticParserCoreDispatchRound
	Prefix                   []DiagnosticParserCoreHeaderReceipt
	PrimaryOutput            DiagnosticParserCoreHeaderReceipt
	PrimaryPaused            bool
	PrimaryAdopted           bool
	OriginalSuffix           []DiagnosticParserCoreHeaderReceipt
	SecondaryArms            []DiagnosticParserCoreGenericConflictArm
	AdditionalPrimaryOutputs []DiagnosticParserCoreHeaderReceipt
	After                    []DiagnosticParserCoreHeaderReceipt
}

type DiagnosticParserCoreGenericConflictArm added in v0.46.0 

type DiagnosticParserCoreGenericConflictArm struct {
	Ordinal     int
	BranchOrder uint64
	Outputs     []DiagnosticParserCoreHeaderReceipt
	Paused      bool
	Adopted     bool
}

DiagnosticParserCoreGenericConflict records one table-driven conflict cell. Actions preserve execution order: secondary ordinals first, then primary.

type DiagnosticParserCoreGenericExternalShift added in v0.46.0 

type DiagnosticParserCoreGenericExternalShift struct {
	ElectionIndex int
	Token         Token
	ScannerBefore DiagnosticParserCoreScannerCheckpoint
	ScannerAfter  DiagnosticParserCoreScannerCheckpoint
	RoundIndex    int
	Payloads      []DiagnosticParserCoreTerminalPayloadView
}

DiagnosticParserCoreGenericExternalShift ties every compact external terminal payload created by one generic scheduler round to its scanner-authenticated election without embedding scanner state in the compact graph. The round may be an ordinary or extra shift cohort, or a conflict with one or more shift arms.

type DiagnosticParserCoreGenericNoActionDrop added in v0.46.0 

type DiagnosticParserCoreGenericNoActionDrop struct {
	ElectionIndex int
	Token         Token
	Header        DiagnosticParserCoreHeaderPathReceipt
}

DiagnosticParserCoreGenericNoActionDrop records a paused scheduler head removed only after a sibling made real progress in the same token epoch.

type DiagnosticParserCoreGenericScheduler added in v0.46.0 

type DiagnosticParserCoreGenericScheduler struct {
	ReceiptMode     DiagnosticParserCoreReceiptMode
	StartCheckpoint DiagnosticParserCoreScannerCheckpoint
	StartHeaders    []DiagnosticParserCoreHeaderPathReceipt
	Rounds          []DiagnosticParserCoreDispatchRound
	Conflicts       []DiagnosticParserCoreGenericConflict
	ExternalShifts  []DiagnosticParserCoreGenericExternalShift
	Elections       []DiagnosticParserCoreElection
	NoActionDrops   []DiagnosticParserCoreGenericNoActionDrop
	Completion      *DiagnosticParserCoreGenericCompletion
	Acceptance      *DiagnosticParserCoreGenericAcceptance

	Stop              DiagnosticParserCoreGenericStop
	Tokens            uint64
	Dispatches        uint64
	GlobalBranchOrder uint64
	NextCreationSeq   uint64
	// contains filtered or unexported fields
}

DiagnosticParserCoreGenericScheduler records one committed compact scheduler run from the sole authenticated seed lifecycle before its first election.

type DiagnosticParserCoreGenericStop added in v0.46.0 

type DiagnosticParserCoreGenericStop struct {
	Boundary      DiagnosticParserCoreBoundaryKind
	Detail        string
	ElectionIndex int
	HeaderIndex   int
	State         StateID
	ByteOffset    uint32
	Token         Token
	Headers       []DiagnosticParserCoreHeaderPathReceipt
	Stats         core.Stats
	Work          DiagnosticParserCoreGenericWork
}

DiagnosticParserCoreGenericStop is the first semantic the table-driven clean scheduler deliberately does not implement.

type DiagnosticParserCoreGenericWork added in v0.46.0 

type DiagnosticParserCoreGenericWork struct {
	Passes                     uint64
	ActionLookups              uint64
	Dispatches                 uint64
	Conflicts                  uint64
	ConflictActions            uint64
	Forks                      uint64
	ConflictActionArmsAdmitted uint64
	CausalConflictForks        uint64
	ConflictHeads              uint64
	// ConvergedReductionSplitDrops counts no-action drops descended from a
	// reduction that split multiple compact predecessor paths into live heads.
	ConvergedReductionSplitDrops uint64
	// SelectedLineageDrops counts converged split drops whose unselected rank
	// matched one surviving selected header from the same reduction.
	SelectedLineageDrops uint64
	RepetitionFolds      uint64
	Reductions           uint64
	OrdinaryShifts       uint64
	OrdinaryCohorts      uint64
	ExtraShifts          uint64
	ExtraCohorts         uint64
	Accepts              uint64
	ReductionPauses      uint64
	NoActionDrops        uint64
	Elections            uint64
	Canonicalizations    uint64
	PeakHeaders          uint64
	Overflow             bool
}

DiagnosticParserCoreGenericWork records semantic scheduler work separately from the compact core's physical arena storage.

type DiagnosticParserCoreHeaderPathReceipt added in v0.46.0 

type DiagnosticParserCoreHeaderPathReceipt struct {
	Header               DiagnosticParserCoreHeaderReceipt
	Derivations          []DiagnosticParserCorePackedDerivation
	DerivationsTruncated bool
}

type DiagnosticParserCoreHeaderReceipt added in v0.46.0 

type DiagnosticParserCoreHeaderReceipt struct {
	CreationSeq uint64
	State       StateID
	ByteOffset  uint32
	Shifted     bool
	Accepted    bool
	Paused      bool
	ExactPaths  uint64
	Checkpoint  [32]byte
}

type DiagnosticParserCorePackedDerivation added in v0.46.0 

type DiagnosticParserCorePackedDerivation struct {
	Score          int64
	BranchOrder    uint64
	HasBranchOrder bool
}

type DiagnosticParserCorePrefixOptions added in v0.46.0 

type DiagnosticParserCorePrefixOptions struct {
	Recovery       bool
	Retry          bool
	Incremental    bool
	IncludedRanges bool
	// GenericStopAtClosedByte publishes a successful closed-frontier receipt
	// when every authenticated scheduler head closes at this byte. Nil is
	// unbounded. The boundary is checked before another scanner election.
	GenericStopAtClosedByte *uint32
	ReceiptMode             DiagnosticParserCoreReceiptMode
	MaxDispatches           uint64
	MaxTokens               uint64
	Limits                  core.Limits
	// contains filtered or unexported fields
}

type DiagnosticParserCorePrefixResult added in v0.46.0 

type DiagnosticParserCorePrefixResult struct {
	Boundary          DiagnosticParserCoreBoundaryKind
	Detail            string
	Dispatches        uint64
	Tokens            uint64
	State             StateID
	Lookahead         Token
	LastBranchOrder   uint64
	GenericScheduler  *DiagnosticParserCoreGenericScheduler
	Completed         bool
	Elections         []DiagnosticParserCoreElection
	SourceSHA256      [32]byte
	GrammarBlobSHA256 [32]byte
	Grammar           string
	ExactRootDFA      bool
	Materialized      bool
	// MaterializedTree is a structural diagnostic owned by the caller and must
	// be released. It is set only after authenticated EOF acceptance and
	// one-shot compact-tree materialization succeed. The diagnostic runner does
	// not force parser-state replay, so its default tree retains the hard
	// incremental-reuse bar. The production admission runner separately forces
	// replay and may clear that bar only when materialization proves the required
	// states and scanner quiescence per tree.
	MaterializedTree *Tree
}

func DiagnosticParseParserCorePrefix added in v0.46.0 

func DiagnosticParseParserCorePrefix(scanner ExternalScanner, source []byte, options DiagnosticParserCorePrefixOptions) (DiagnosticParserCorePrefixResult, error)

DiagnosticParseParserCorePrefix independently schedules one compact seed against the complete production DFA/scanner election stream. Unsupported boundaries remain fail-closed. It never calls the production parser.

type DiagnosticParserCoreReceiptMode added in v0.46.0 

type DiagnosticParserCoreReceiptMode uint8

DiagnosticParserCoreReceiptMode controls diagnostic observation only. It never changes parser-core scheduling or selection semantics. The zero value preserves the complete historical receipt; summary mode retains only the authenticated result and aggregate work needed for larger-fixture study. 

const (
	DiagnosticParserCoreReceiptFull DiagnosticParserCoreReceiptMode = iota
	DiagnosticParserCoreReceiptSummary
)

type DiagnosticParserCoreRoundAction added in v0.46.0 

type DiagnosticParserCoreRoundAction struct {
	HeaderIndex int
	State       StateID
	ByteOffset  uint32
	Ordinal     int
	Action      ParseAction
	BranchOrder uint64
}

type DiagnosticParserCoreScannerCheckpoint added in v0.46.0 

type DiagnosticParserCoreScannerCheckpoint struct {
	Length int
	SHA256 [32]byte
}

type DiagnosticParserCoreTerminalPayloadView added in v0.46.0 

type DiagnosticParserCoreTerminalPayloadView struct {
	ID                uint32
	Symbol            Symbol
	ProductionID      uint16
	DynamicPrecedence int16
	StartByte         uint32
	EndByte           uint32
	Children          []uint32
	Fields            []FieldMapEntry
	Aliases           []Symbol
	Extra             bool
	External          bool
	Terminal          bool
}

type ErrorTreeIncrementalReuseExternalScanner added in v0.47.0 

type ErrorTreeIncrementalReuseExternalScanner interface {
	ExternalScanner
	SupportsIncrementalReuseFromErrorTree() bool
}

ErrorTreeIncrementalReuseExternalScanner is an optional refinement for a scanner whose checkpoints are certified on clean old trees but whose parser recovery ownership has not yet been certified. Returning false makes a changed edit over an error-bearing old tree take the fresh-parse fallback. Independently reauthenticated token-invariant leaf edits may still return before this gate.

type ExternalLexer 

type ExternalLexer struct {
	// contains filtered or unexported fields
}

ExternalLexer is the scanner-facing lexer API used by external scanners. It mirrors the essential tree-sitter scanner API: lookahead, advance, mark_end, and result_symbol.

func (*ExternalLexer) Advance 

func (l *ExternalLexer) Advance(skip bool)

Advance consumes one rune. When skip is true, consumed bytes are excluded from the token span (scanner whitespace skipping behavior).

func (*ExternalLexer) AdvanceSpaces added in v0.20.6 

func (l *ExternalLexer) AdvanceSpaces(skip bool) int

AdvanceSpaces consumes consecutive ASCII spaces and returns the number of bytes consumed. It is equivalent to repeated Advance(skip) while Lookahead is a space, but avoids per-byte rune decoding in external scanners that skip indentation runs.

func (*ExternalLexer) AdvanceUntilNewline added in v0.20.6 

func (l *ExternalLexer) AdvanceUntilNewline(skip bool) int

AdvanceUntilNewline consumes bytes up to, but not including, '\n' or EOF and returns the number of bytes consumed. For non-newline bytes, Advance updates Column by the UTF-8 width, which is equal to the byte count for the whole consumed span.

func (*ExternalLexer) Column added in v0.6.0 

func (l *ExternalLexer) Column() uint32

Column returns the current column (0-based) at the scanner cursor.

func (*ExternalLexer) GetColumn deprecated 

func (l *ExternalLexer) GetColumn() uint32

GetColumn returns the current column (0-based) at the scanner cursor.

Deprecated: use Column.

func (*ExternalLexer) HasPreviousBytes added in v0.21.0 

func (l *ExternalLexer) HasPreviousBytes(text string) bool

HasPreviousBytes reports whether the bytes immediately before the scanner cursor match text. External scanners use this to guard context-sensitive content tokens when merged parser states expose them too broadly.

func (*ExternalLexer) Lookahead 

func (l *ExternalLexer) Lookahead() rune

Lookahead returns the current rune or 0 at EOF.

func (*ExternalLexer) MarkEnd 

func (l *ExternalLexer) MarkEnd()

MarkEnd marks the current scanner position as the token end.

func (*ExternalLexer) Previous added in v0.48.0 

func (l *ExternalLexer) Previous() rune

Previous returns the rune immediately before the current lexer position. It returns 0 only at the start of the source (pos <= 0) or when pos is out of range. For invalid UTF-8 immediately before pos, it returns utf8.RuneError (U+FFFD), matching utf8.DecodeLastRune -- never 0.

Warning: a scanner that reads Previous() looks behind the current scan position, which is a byte, not a grammar token. A caller cannot assume the preceding byte belongs to the token that logically precedes this position: extras (comments, whitespace) the core lexer matches between two scanner invocations are invisible to the scanner, so the byte immediately before pos can be the tail of a comment rather than the true previous token. A scanner that relies on Previous() to decide something as history-sensitive as this must track token-boundary state itself across calls (see grammars/swift_scanner.go) rather than trust a single raw byte read. This lookbehind also breaks external-scanner quiescence obligation 2 (see external_scanner_quiescence.go): Scan must depend only on bytes at or after the current position plus the valid-symbol set, not on bytes before it. A scanner that calls Previous() must not claim StatelessExternalScanner.

func (*ExternalLexer) SetResultSymbol 

func (l *ExternalLexer) SetResultSymbol(sym Symbol)

SetResultSymbol sets the token symbol to emit when Scan returns true.

type ExternalScanner 

type ExternalScanner interface {
	Create() any
	Destroy(payload any)
	Serialize(payload any, buf []byte) int
	Deserialize(payload any, buf []byte)
	Scan(payload any, lexer *ExternalLexer, validSymbols []bool) bool
}

ExternalScanner is the interface for language-specific external scanners. Languages like Python and JavaScript need these for indent tracking, template literals, regex vs division, etc.

The value returned by Create must be accepted by Destroy/Serialize/ Deserialize/Scan for that scanner implementation. Most scanners use a concrete payload pointer type and will panic on mismatched payload types.

func AdaptExternalScannerByExternalOrder added in v0.9.0 

func AdaptExternalScannerByExternalOrder(sourceLang, targetLang *Language) (ExternalScanner, bool)

AdaptExternalScannerByExternalOrder builds an ExternalScanner adapter that reuses sourceLang's scanner for targetLang by remapping external symbols.

Mapping strategy:

  1. If either side has duplicate external names, use index mapping (capped to the shorter list length).
  2. Otherwise, prefer exact external-symbol-name matches.
  3. Fill remaining slots by index order (within the shorter dimension).

When source and target have different external symbol counts, name-based matching pairs tokens that exist in both grammars. Target externals with no source match get -1 (the scanner will never produce them). Source externals with no target match are silently ignored.

Returns (nil, false) when adaptation is not possible.

type ExternalScannerFullParseRetryPolicy added in v0.24.1 

type ExternalScannerFullParseRetryPolicy uint8

ExternalScannerFullParseRetryPolicy controls whether a full parse with an external scanner may schedule the generic second retry ladder after the normal retry ladder has already selected its best tree.

Keep new values append-only: Language blobs encode these numeric values. 

const (
	// ExternalScannerFullParseRetryDefault preserves the generic behavior: an
	// accepted error-bearing tree may schedule one more full retry ladder.
	ExternalScannerFullParseRetryDefault ExternalScannerFullParseRetryPolicy = iota

	// ExternalScannerFullParseRetrySkipRepeat certifies that the tree selected
	// by the first retry ladder is authoritative. The parser retains that exact
	// selected tree and does not schedule the extra external-scanner retry.
	ExternalScannerFullParseRetrySkipRepeat
)

type ExternalScannerState 

type ExternalScannerState struct {
	Data []byte
}

ExternalScannerState holds serialized state for an external scanner between incremental parse runs.

type ExternalSymbolResolver added in v0.9.0 

type ExternalSymbolResolver struct {
	// contains filtered or unexported fields
}

ExternalSymbolResolver maps external token names to their concrete Symbol IDs in a specific Language. This allows external scanners to resolve symbol IDs at runtime rather than using hardcoded constants, making them compatible with any Language that defines the same external tokens (whether from ts2go extraction or grammargen).

func NewExternalSymbolResolver added in v0.9.0 

func NewExternalSymbolResolver(lang *Language) *ExternalSymbolResolver

NewExternalSymbolResolver builds a resolver from a Language's external symbol definitions. Returns nil if the Language has no external symbols.

func (*ExternalSymbolResolver) ByIndex added in v0.9.0 

func (r *ExternalSymbolResolver) ByIndex(idx int) (Symbol, bool)

ByIndex returns the Symbol ID for the given external token index (position in the grammar's externals array). Returns 0, false if the index is out of range.

func (*ExternalSymbolResolver) ByName added in v0.9.0 

func (r *ExternalSymbolResolver) ByName(name string) (Symbol, bool)

ByName returns the Symbol ID for the given external token name. Returns 0, false if the name is not found.

func (*ExternalSymbolResolver) Count added in v0.9.0 

func (r *ExternalSymbolResolver) Count() int

Count returns the number of external tokens.

type ExternalVMInstr 

type ExternalVMInstr struct {
	Op  ExternalVMOp
	A   int32
	B   int32
	Alt int32
}

ExternalVMInstr is one instruction in an external scanner VM program.

Operands:

  • A: primary operand (opcode-specific)
  • B: secondary operand (used by range checks)
  • Alt: alternate program counter when a condition fails

func VMAdvance 

func VMAdvance(skip bool) ExternalVMInstr

VMAdvance constructs an advance instruction. When skip is true, the advanced rune is skipped from the token text.

func VMEmit 

func VMEmit(sym Symbol) ExternalVMInstr

VMEmit constructs an emit instruction for the given symbol.

func VMFail 

func VMFail() ExternalVMInstr

VMFail constructs a fail instruction that terminates scan with no token.

func VMIfRuneClass 

func VMIfRuneClass(class ExternalVMRuneClass, alt int) ExternalVMInstr

VMIfRuneClass constructs a rune-class branch with alternate target on miss.

func VMIfRuneEq 

func VMIfRuneEq(r rune, alt int) ExternalVMInstr

VMIfRuneEq constructs a rune-equality branch with alternate target on miss.

func VMIfRuneInRange 

func VMIfRuneInRange(start, end rune, alt int) ExternalVMInstr

VMIfRuneInRange constructs a rune-range branch with alternate target on miss.

func VMJump 

func VMJump(target int) ExternalVMInstr

VMJump constructs an unconditional branch to the target instruction index.

func VMMarkEnd 

func VMMarkEnd() ExternalVMInstr

VMMarkEnd constructs a mark-end instruction for the current token extent.

func VMRequireStateEq 

func VMRequireStateEq(state uint32, alt int) ExternalVMInstr

VMRequireStateEq constructs a payload-state guard with alternate branch on miss.

func VMRequireValid 

func VMRequireValid(validSymbolIndex, alt int) ExternalVMInstr

VMRequireValid constructs a valid-symbol guard with alternate branch on miss.

func VMSetState 

func VMSetState(state uint32) ExternalVMInstr

VMSetState constructs a payload-state assignment instruction.

type ExternalVMOp 

type ExternalVMOp uint8

ExternalVMOp is an opcode for the native-Go external scanner VM. 

const (
	ExternalVMOpFail ExternalVMOp = iota
	ExternalVMOpJump
	ExternalVMOpRequireValid
	ExternalVMOpRequireStateEq
	ExternalVMOpSetState
	ExternalVMOpIfRuneEq
	ExternalVMOpIfRuneInRange
	ExternalVMOpIfRuneClass
	ExternalVMOpAdvance
	ExternalVMOpMarkEnd
	ExternalVMOpEmit
)

type ExternalVMProgram 

type ExternalVMProgram struct {
	Code     []ExternalVMInstr
	MaxSteps int // <=0 uses a safe default based on program size
}

ExternalVMProgram is a small bytecode program interpreted by ExternalVMScanner.

type ExternalVMRuneClass 

type ExternalVMRuneClass uint8

ExternalVMRuneClass is a character class used by ExternalVMOpIfRuneClass. 

const (
	ExternalVMRuneClassWhitespace ExternalVMRuneClass = iota
	ExternalVMRuneClassDigit
	ExternalVMRuneClassLetter
	ExternalVMRuneClassWord
	ExternalVMRuneClassNewline
)

type ExternalVMScanner 

type ExternalVMScanner struct {
	// contains filtered or unexported fields
}

ExternalVMScanner executes an ExternalVMProgram and implements ExternalScanner.

func MustNewExternalVMScanner 

func MustNewExternalVMScanner(program ExternalVMProgram) *ExternalVMScanner

MustNewExternalVMScanner is like NewExternalVMScanner but panics on error. It is intended for package-level initialization where invalid programs are programmer errors.

func NewExternalVMScanner 

func NewExternalVMScanner(program ExternalVMProgram) (*ExternalVMScanner, error)

NewExternalVMScanner validates and constructs an ExternalVMScanner.

func (*ExternalVMScanner) Create 

func (s *ExternalVMScanner) Create() any

Create allocates scanner payload (currently a single uint32 state slot).

func (*ExternalVMScanner) Deserialize 

func (s *ExternalVMScanner) Deserialize(payload any, buf []byte)

Deserialize restores payload state from buf.

func (*ExternalVMScanner) Destroy 

func (s *ExternalVMScanner) Destroy(payload any)

Destroy releases scanner payload resources.

func (*ExternalVMScanner) Scan 

func (s *ExternalVMScanner) Scan(payload any, lexer *ExternalLexer, validSymbols []bool) bool

Scan executes the scanner program against the current lexer position.

func (*ExternalVMScanner) Serialize 

func (s *ExternalVMScanner) Serialize(payload any, buf []byte) int

Serialize writes payload state into buf.

type FailurePreservingExternalScanner added in v0.20.6 

type FailurePreservingExternalScanner interface {
	ExternalScanner
	PreservesStateOnScanFailure() bool
}

FailurePreservingExternalScanner is implemented by external scanners whose Scan method does not mutate serialized scanner payload state before returning false. The token source can defer snapshotting until retry is actually needed.

type FieldID 

type FieldID uint16

FieldID is a named field index.

type FieldMapEntry 

type FieldMapEntry struct {
	FieldID    FieldID
	ChildIndex uint8
	Inherited  bool
}

FieldMapEntry maps a child index to a field name.

type FullParseAcceptedErrorRetryProfile added in v0.25.0 

type FullParseAcceptedErrorRetryProfile struct {
	MinSourceBytes                 uint32
	InitialStackCeiling            uint16
	SkipCompleteAcceptedErrorRetry bool
	FreshErrorNoStacksMaxPasses    uint8
	// SkipCompleteMaxEntryScratchPeak limits the complete-tree skip to a
	// certified peak number of live GLR scratch entries. Zero is unbounded.
	SkipCompleteMaxEntryScratchPeak uint32
	// FreshErrorNoStacksRetryMaxStacks replaces the generic widened-stack
	// target for a fresh error-bearing no-stacks parse. Zero keeps the generic
	// target. Incremental fallbacks and explicit environment overrides ignore it.
	FreshErrorNoStacksRetryMaxStacks uint16
	// SkipInitialCompleteAcceptedErrorMergeRetry skips only the first
	// same-stack merge retry for a fresh, complete accepted-error parse. Later
	// widened-stack and merge retries remain available. Incremental fallbacks
	// and explicit stack/merge environment overrides ignore it.
	SkipInitialCompleteAcceptedErrorMergeRetry bool
	// SkipCompleteMinSourceBytes limits SkipCompleteAcceptedErrorRetry to
	// sources at least this large. Zero preserves the legacy unbounded skip.
	// It has no effect on the profile's other retry policies.
	SkipCompleteMinSourceBytes uint32
	// ReuseCleanWideForWideRetry certifies that the complete accepted-error tree
	// from the non-recovery widened-stack pass is identical to the following
	// recovery-enabled widened-stack pass. The parser may retain that tree and
	// substitute it at the recovery-wide slot while preserving retry accounting.
	ReuseCleanWideForWideRetry bool
	// ReuseCleanWideMinSourceBytes limits clean-wide reuse to the certified
	// large-source class. Zero disables the policy even when the boolean is set.
	ReuseCleanWideMinSourceBytes uint32
	// GSSConvergenceAcceptedErrorMergePerKey sets the exact merge width for an
	// accepted-error retry after a certified cap-one full parse. Zero keeps the
	// other retry policies. Explicit environment settings disable this policy.
	GSSConvergenceAcceptedErrorMergePerKey uint16
}

FullParseAcceptedErrorRetryProfile certifies narrow full-parse retry policies. When a fresh full parse accepts an error-bearing tree that covers EOF, the parser may keep the initial GLR stack ceiling after the ordinary same-stack merge retry or skip that ladder entirely. A grammar may also cap retries or select a narrower widening target for a fresh, error-bearing no-stacks result after proving the generic ladder does not improve the selected tree. The zero value preserves the conservative generic ladder.

Keep fields append-only: Language blobs encode this structure.

type HeritageRef added in v0.20.6 

type HeritageRef struct {
	Lang            string
	Kind            string
	Name            string
	Parent          string
	NodeType        string
	StartByte       uint32
	EndByte         uint32
	ParentStartByte uint32
	ParentEndByte   uint32
}

HeritageRef is a compact language-neutral inheritance/base-class reference.

func ExtractHeritage added in v0.20.6 

func ExtractHeritage(tree *Tree) []HeritageRef

ExtractHeritage returns language-neutral inheritance/base-class references for common code-understanding workflows.

type HighlightRange 

type HighlightRange struct {
	StartByte    uint32
	EndByte      uint32
	Capture      string // "keyword", "string", "function", etc.
	PatternIndex int    // query pattern index; later patterns override earlier for identical ranges
}

HighlightRange represents a styled range of source code, mapping a byte span to a capture name from a highlight query. The editor maps capture names (e.g., "keyword", "string", "function") to FSS style classes.

type Highlighter 

type Highlighter struct {
	// contains filtered or unexported fields
}

Highlighter is a high-level API that takes source code and returns styled ranges. It combines a Parser, a compiled Query, and a Language to provide a single Highlight() call for the editor.

func NewHighlighter 

func NewHighlighter(lang *Language, highlightQuery string, opts ...HighlighterOption) (*Highlighter, error)

NewHighlighter creates a Highlighter for the given language and highlight query (in tree-sitter .scm format). Returns an error if the query fails to compile.

func (*Highlighter) Highlight 

func (h *Highlighter) Highlight(source []byte) []HighlightRange

Highlight parses the source code and executes the highlight query, returning a slice of HighlightRange sorted by StartByte. When ranges overlap, inner (more specific) captures take priority over outer ones.

func (*Highlighter) HighlightIncremental 

func (h *Highlighter) HighlightIncremental(source []byte, oldTree *Tree) ([]HighlightRange, *Tree)

HighlightIncremental re-highlights source after edits were applied to oldTree. Returns the new highlight ranges and the new parse tree (for use in subsequent incremental calls). Call oldTree.Edit() before calling this.

func (*Highlighter) HighlightIncrementalStrict added in v0.37.0 

func (h *Highlighter) HighlightIncrementalStrict(source []byte, oldTree *Tree) ([]HighlightRange, *Tree, error)

HighlightIncrementalStrict is like HighlightIncremental, but reports ErrParseStoppedEarly and skips query execution when parsing is stopped by a timeout, cancellation, token-source EOF, or parser safety limit. The partial tree is returned so callers can release it or use it for diagnostics.

func (*Highlighter) HighlightIncrementalUTF16 added in v0.16.0 

func (h *Highlighter) HighlightIncrementalUTF16(source []uint16, oldTree *Tree) ([]UTF16HighlightRange, *Tree)

HighlightIncrementalUTF16 re-highlights UTF-16 source after edits were applied to oldTree with Tree.EditUTF16.

func (*Highlighter) HighlightIncrementalUTF16Bytes added in v0.16.0 

func (h *Highlighter) HighlightIncrementalUTF16Bytes(source []byte, oldTree *Tree, order UTF16ByteOrder) ([]UTF16HighlightRange, *Tree, error)

HighlightIncrementalUTF16Bytes is like HighlightIncrementalUTF16 for endian-specific UTF-16 bytes.

func (*Highlighter) HighlightTreeUTF16 added in v0.35.0 

func (h *Highlighter) HighlightTreeUTF16(tree *Tree) []UTF16HighlightRange

HighlightTreeUTF16 executes the highlighter query against an already parsed UTF-16 tree. It lets editor runtimes share one persistent parse tree across syntax highlighting, tags, and ad-hoc queries.

func (*Highlighter) HighlightUTF16 added in v0.16.0 

func (h *Highlighter) HighlightUTF16(source []uint16) []UTF16HighlightRange

HighlightUTF16 parses UTF-16 source and returns highlight ranges in UTF-16 code-unit coordinates.

func (*Highlighter) HighlightUTF16Bytes added in v0.16.0 

func (h *Highlighter) HighlightUTF16Bytes(source []byte, order UTF16ByteOrder) ([]UTF16HighlightRange, error)

HighlightUTF16Bytes is like HighlightUTF16 for endian-specific UTF-16 bytes.

type HighlighterInjectionResolver added in v0.7.0 

type HighlighterInjectionResolver func(languageHint string) (lang *Language, highlightQuery string, tokenSourceFactory func(source []byte) TokenSource, ok bool)

HighlighterInjectionResolver maps a language hint (for example "go" from a markdown code fence) to a child language and highlight query.

type HighlighterInjectionSpec added in v0.7.0 

type HighlighterInjectionSpec struct {
	Query           string
	ResolveLanguage HighlighterInjectionResolver
}

HighlighterInjectionSpec configures nested highlighting for a parent language. Query must emit @injection.content and either @injection.language or #set! injection.language metadata.

type HighlighterOption 

type HighlighterOption func(*Highlighter)

HighlighterOption configures a Highlighter.

func WithHighlighterTimeoutMicros added in v0.37.0 

func WithHighlighterTimeoutMicros(timeoutMicros uint64) HighlighterOption

WithHighlighterTimeoutMicros bounds every full and incremental parse performed by the highlighter. A value of zero disables timeout checks.

func WithTokenSourceFactory 

func WithTokenSourceFactory(factory func(source []byte) TokenSource) HighlighterOption

WithTokenSourceFactory sets a factory function that creates a TokenSource for each Highlight call. This is needed for languages that use a custom lexer bridge (like Go, which uses go/scanner instead of a DFA lexer).

When set, Highlight() calls ParseWithTokenSource instead of Parse.

type ImportExtractResult added in v0.18.0 

type ImportExtractResult struct {
	Imports             []ImportRef
	Status              ImportExtractStatus
	Reason              string
	FallbackRecommended bool
}

ImportExtractResult is returned by source-only dependency extraction. When FallbackRecommended is true, callers that need exact tree-sitter behavior should parse the file and use ExtractImports.

func ExtractImportsFromSourceWithReport added in v0.18.0 

func ExtractImportsFromSourceWithReport(lang *Language, source []byte) ImportExtractResult

ExtractImportsFromSourceWithReport returns source-only dependency declarations and a confidence report for fallback policy.

type ImportExtractStatus added in v0.18.0 

type ImportExtractStatus string

ImportExtractStatus describes the confidence of source-only import extraction. 

const (
	ImportExtractOK                   ImportExtractStatus = "ok"
	ImportExtractUnsupportedConstruct ImportExtractStatus = "unsupported_construct"
	ImportExtractScannerError         ImportExtractStatus = "scanner_error"
	ImportExtractAmbiguous            ImportExtractStatus = "ambiguous"
	ImportExtractFallbackToTree       ImportExtractStatus = "fallback_to_tree"
)

type ImportRef added in v0.18.0 

type ImportRef struct {
	Lang      string
	Kind      string
	Path      string
	From      string
	Name      string
	Alias     string
	Static    bool
	Wildcard  bool
	Relative  int
	StartByte uint32
	EndByte   uint32
}

ImportRef is a compact language-neutral dependency declaration extracted from a syntax tree.

func ExtractImports added in v0.18.0 

func ExtractImports(tree *Tree) []ImportRef

ExtractImports returns package/import declarations for the languages used by Gazelle-style dependency extraction. It is intentionally independent from the generic query engine so it can later be backed by compact parser refs.

func ExtractImportsFromSource added in v0.18.0 

func ExtractImportsFromSource(lang *Language, source []byte) []ImportRef

ExtractImportsFromSource returns language-neutral dependency declarations directly from source text. It is intended for cold dependency-extraction workflows that do not need a public syntax tree.

type IncrementalParseProfile added in v0.6.0 

type IncrementalParseProfile struct {
	ReuseCursorNanos                   int64
	ReparseNanos                       int64
	ReusedSubtrees                     uint64
	ReusedBytes                        uint64
	NewNodesAllocated                  uint64
	ReuseUnsupported                   bool
	ReuseUnsupportedReason             string
	AcceptedErrorRetryAttempts         uint8
	AcceptedErrorRetryAdopted          bool
	AcceptedErrorRetryMergePerKey      int
	AcceptedErrorRetryCause            IncrementalRetryCause
	OldTreeReuseRoute                  bool
	ReuseRejectDirty                   uint64
	ReuseRejectAncestorDirtyBeforeEdit uint64
	ReuseRejectHasError                uint64
	ReuseRejectInvalidSpan             uint64
	ReuseRejectOutOfBounds             uint64
	ReuseRejectRootNonLeafChanged      uint64
	// ReuseObservedPreGotoStateMismatch counts top-level block-splice candidates
	// observed at a live parser state different from the node's recorded
	// PreGotoState. This is diagnostic only: the established admission contract
	// remains goto-target compatibility plus fragility until #432 replaces it
	// with a complete ownership proof.
	ReuseObservedPreGotoStateMismatch uint64
	ReuseRejectLargeNonLeaf           uint64
	ReuseRejectStaleNonLeafBoundary   uint64
	// ReuseRejectFragileNonLeaf counts interior (non-leaf) reuse candidates
	// rejected because Node.isFragile() reported the candidate was built
	// under an ambiguous parse decision (LR-table conflict, GSS multi-pop, or
	// concurrent GLR stack versions) or is itself an ERROR/MISSING node -- see
	// markReduceFragility (parser_reduce.go) and reuseNonLeafTargetStateOnStack
	// (incremental.go). A nonzero count on a conflict-heavy grammar (e.g. js)
	// is expected and correct: it is exactly the unsound reuse this gate is
	// designed to prevent.
	ReuseRejectFragileNonLeaf uint64
	// BlockSpliceSteps is the number of top-level sibling reuses taken inside
	// the W1 block-splice composition loop (spec.campaign.oedit): one per
	// sibling spliced without a full main-loop round trip. It is O(edit) and
	// deterministic for a fixed (source, edit, language).
	BlockSpliceSteps uint64
	// ReuseRejectScannerUnquiescent counts reuse candidates the external
	// scanner checkpoint/quiescence gate rejected -- a checkpoint state
	// mismatch on a checkpoint language, or a refuted quiescence proof on an
	// opt-out scanner (campaign O(edit) workstream W4, spec.campaign.oedit).
	// It is 0 for stateless-scanner languages such as Go, whose ASI scanner
	// carries no cross-token state and is proven quiescent at every boundary
	// (external_scanner_quiescence.go). A nonzero count marks boundaries where
	// scanner state, not fragility or byte drift, is the binding constraint.
	ReuseRejectScannerUnquiescent       uint64
	RecoverSearches                     uint64
	RecoverStateChecks                  uint64
	RecoverStateSkips                   uint64
	RecoverSymbolSkips                  uint64
	RecoverLookups                      uint64
	RecoverHits                         uint64
	MaxStacksSeen                       int
	EntryScratchPeak                    uint64
	StopReason                          ParseStopReason
	TokensConsumed                      uint64
	LastTokenEndByte                    uint32
	ExpectedEOFByte                     uint32
	ArenaBytesAllocated                 int64
	ScratchBytesAllocated               int64
	EntryScratchBytesAllocated          int64
	GSSBytesAllocated                   int64
	SingleStackIterations               int
	MultiStackIterations                int
	SingleStackTokens                   uint64
	MultiStackTokens                    uint64
	SingleStackGSSNodes                 uint64
	MultiStackGSSNodes                  uint64
	GSSNodesAllocated                   uint64
	GSSNodesRetained                    uint64
	GSSNodesDroppedSameToken            uint64
	ParentNodesAllocated                uint64
	ParentNodesRetained                 uint64
	ParentNodesDroppedSameToken         uint64
	LeafNodesAllocated                  uint64
	LeafNodesRetained                   uint64
	LeafNodesDroppedSameToken           uint64
	MergeStacksIn                       uint64
	MergeStacksOut                      uint64
	MergeSlotsUsed                      uint64
	GlobalCullStacksIn                  uint64
	GlobalCullStacksOut                 uint64
	ParserLoopNanos                     int64
	TokenNextNanos                      int64
	ActionDispatchNanos                 int64
	ActionLookupNanos                   int64
	GLRMergeNanos                       int64
	GLRCullNanos                        int64
	ResultSelectionNanos                int64
	TransientParentMaterializationNanos int64
	ResultTreeBuildNanos                int64
	TransientChildMaterializationNanos  int64
	ResultPythonKeywordRepairNanos      int64
	ResultPythonRootRepairNanos         int64
	ResultFinalizeRootNanos             int64
	ResultExtendTrailingNanos           int64
	ResultNormalizeRootStartNanos       int64
	ResultCompatibilityNanos            int64
	ResultParentLinkNanos               int64
	ReduceRangeNanos                    int64
	ReducePendingParentNanos            int64
	ReduceChildBuildNanos               int64
	ReduceParentBuildNanos              int64
	ReduceSpanNanos                     int64
	ReduceStackPushNanos                int64
	ReduceNoTreeBuildNanos              int64
	ActionExtraShiftNanos               int64
	ActionNoActionNanos                 int64
	ActionNoActionRelexNanos            int64
	ActionNoActionMissingNanos          int64
	ActionNoActionRecoverNanos          int64
	ActionNoActionErrorNanos            int64
	ActionConflictChoiceNanos           int64
	ActionConflictForkNanos             int64
	ActionSingleShiftNanos              int64
	ActionSingleReduceNanos             int64
	ActionSingleAcceptNanos             int64
	ActionSingleRecoverNanos            int64
	ActionSingleOtherNanos              int64
	NormalizationNanos                  int64
}

IncrementalParseProfile attributes incremental parse time into coarse buckets.

ReuseCursorNanos includes reuse-cursor setup and subtree-candidate checks. ReparseNanos includes the remainder of incremental parsing/rebuild work.

type IncrementalRetryCause added in v0.39.0 

type IncrementalRetryCause uint8

IncrementalRetryCause identifies why an incremental parse ran an additional bounded attempt. 

const (
	IncrementalRetryCauseNone IncrementalRetryCause = iota
	IncrementalRetryCauseAcceptedErrorBaseMerge
)

type IncrementalReuseExternalScanner added in v0.7.0 

type IncrementalReuseExternalScanner interface {
	ExternalScanner
	SupportsIncrementalReuse() bool
}

IncrementalReuseExternalScanner is implemented by external scanners that can safely participate in DFA subtree reuse during incremental parses. Scanners with serialized mutable state, such as Python's indentation stack, should leave this unimplemented so edited incremental parses fall back to the conservative full-reparse path.

type IncrementalReuseTokenSource added in v0.7.0 

type IncrementalReuseTokenSource interface {
	TokenSource
	SupportsIncrementalReuse() bool
}

IncrementalReuseTokenSource is an opt-in marker for custom token sources that are safe for incremental subtree reuse. Implementations must provide stable token boundaries across edits and support deterministic SkipToByte* behavior so reused-tree fast-forwarding remains correct.

type Injection added in v0.6.0 

type Injection struct {
	// Language is the detected language name (e.g., "javascript").
	Language string
	// Tree is the parse tree for this region, or nil if the language
	// was not registered.
	Tree *Tree
	// Ranges are the source ranges this tree covers.
	Ranges []Range
	// Node is the parent tree node that triggered the injection.
	Node *Node
}

Injection is a single embedded language region.

type InjectionParser added in v0.6.0 

type InjectionParser struct {
	// contains filtered or unexported fields
}

InjectionParser parses documents with embedded languages.

InjectionParser is not safe for concurrent use. It caches child parsers and mutates shared maps during parse operations.

func NewInjectionParser added in v0.6.0 

func NewInjectionParser() *InjectionParser

NewInjectionParser creates an InjectionParser.

func (*InjectionParser) Parse added in v0.6.0 

func (ip *InjectionParser) Parse(source []byte, parentLang string) (*InjectionResult, error)

Parse parses source as parentLang, then recursively parses injected regions.

func (*InjectionParser) ParseIncremental added in v0.6.0 

func (ip *InjectionParser) ParseIncremental(source []byte, parentLang string,
	oldResult *InjectionResult) (*InjectionResult, error)

ParseIncremental re-parses after edits, reusing unchanged child trees.

func (*InjectionParser) ParseIncrementalUTF16 added in v0.16.0 

func (ip *InjectionParser) ParseIncrementalUTF16(source []uint16, parentLang string,
	oldResult *UTF16InjectionResult) (*UTF16InjectionResult, error)

ParseIncrementalUTF16 re-parses UTF-16 source after edits, reusing unchanged child trees. Call oldResult.Tree.EditUTF16 before calling this.

func (*InjectionParser) ParseIncrementalUTF16Bytes added in v0.16.0 

func (ip *InjectionParser) ParseIncrementalUTF16Bytes(source []byte, parentLang string,
	oldResult *UTF16InjectionResult, order UTF16ByteOrder) (*UTF16InjectionResult, error)

ParseIncrementalUTF16Bytes is like ParseIncrementalUTF16 for endian-specific UTF-16 bytes.

func (*InjectionParser) ParseUTF16 added in v0.16.0 

func (ip *InjectionParser) ParseUTF16(source []uint16, parentLang string) (*UTF16InjectionResult, error)

ParseUTF16 parses UTF-16 source as parentLang, then recursively parses injected regions. The returned injection ranges are in UTF-16 code units.

func (*InjectionParser) ParseUTF16Bytes added in v0.16.0 

func (ip *InjectionParser) ParseUTF16Bytes(source []byte, parentLang string, order UTF16ByteOrder) (*UTF16InjectionResult, error)

ParseUTF16Bytes is like ParseUTF16 for endian-specific UTF-16 bytes.

func (*InjectionParser) RegisterInjectionQuery added in v0.6.0 

func (ip *InjectionParser) RegisterInjectionQuery(parentLang string, query string) error

RegisterInjectionQuery sets the injection query for a parent language. The query should use @injection.content and #set! injection.language conventions. It is compiled against the registered parent language.

func (*InjectionParser) RegisterLanguage added in v0.6.0 

func (ip *InjectionParser) RegisterLanguage(name string, lang *Language)

RegisterLanguage adds a language that can be used as parent or child.

func (*InjectionParser) SetMaxDepth added in v0.6.0 

func (ip *InjectionParser) SetMaxDepth(depth int)

SetMaxDepth overrides the nested injection recursion limit. Depth values <= 0 restore the default limit.

type InjectionResult added in v0.6.0 

type InjectionResult struct {
	// Tree is the parent language's parse tree.
	Tree *Tree
	// Injections contains child language parse results, ordered by position.
	Injections []Injection
}

InjectionResult holds parse results for a multi-language document.

type InputEdit 

type InputEdit struct {
	StartByte   uint32
	OldEndByte  uint32
	NewEndByte  uint32
	StartPoint  Point
	OldEndPoint Point
	NewEndPoint Point
}

InputEdit describes a single edit to the source text. It tells the parser what byte range was replaced and what the new range looks like, so the incremental parser can skip unchanged subtrees.

type InputEncoding added in v0.16.0 

type InputEncoding uint8

InputEncoding identifies the source encoding used to build a Tree. 

const (
	InputEncodingUTF8 InputEncoding = iota
	InputEncodingUTF16
)

func (InputEncoding) String added in v0.16.0 

func (e InputEncoding) String() string

type InternObservationStats added in v0.20.0 

type InternObservationStats struct {
	// Phase 2 counters (parseState-blind observation across ALL leaves).
	LeafLookups uint64
	LeafHits    uint64
	LeafMisses  uint64
	LeafStores  uint64
	LeafGrowths uint64
	// Phase 3 attribution. Shift-path leaves get parseState set per-fork
	// so they can't be canonically substituted via the parseState-blind
	// measurement; non-shift leaves can. "Safe to substitute" via blind
	// measurement = (LeafMisses+LeafHits) - ShiftLeafObserved.
	ShiftLeafObserved uint64
	// Phase 3 parseState-aware measurement. Same hook as LeafLookups
	// but with parseState/preGotoState included in the key. A hit here
	// means a truly dedup-safe duplicate; the difference between this
	// and the blind hit rate quantifies how much of the blind
	// observation was an artifact of ignoring state.
	FullLookups uint64
	FullHits    uint64
	FullMisses  uint64
}

InternObservationStats is the externally-visible snapshot of leaf-interning observation counters for a single parse. Returned from InternStatsFor.

func InternStatsFor added in v0.20.0 

func InternStatsFor(root *Node) InternObservationStats

InternStatsFor returns a snapshot of the leaf-interning observation counters for the arena that owns the given root node. Returns the zero value if observation is disabled or the root is not arena-backed. Exposed so external benches can read hit rates without grepping internal logs.

type Language 

type Language struct {
	Name string
	// GeneratedByGrammargen is true for languages assembled by grammargen at
	// runtime rather than decoded from a checked-in ts2go blob.
	GeneratedByGrammargen bool

	// CRecoveryCostCompetitionCapable records parser.c/table evidence that the
	// grammar exposes the C recovery surface: RECOVER actions plus an
	// ERROR_STATE lex mode. It is capability metadata only, not a default-on
	// parity certification.
	CRecoveryCostCompetitionCapable bool

	// CRecoveryCostCompetitionEnabledByDefault explicitly certifies that the
	// faithful C recovery-cost competition gate is parity-safe as default
	// behavior for this language. Runtime gating also requires capability
	// metadata and conservative table validation.
	CRecoveryCostCompetitionEnabledByDefault bool

	// WantsForest opts this language into the GSS-forest GLR fast path.
	// Consumers generating a Language via grammargen set this (directly or via
	// grammargen.Grammar.WantsForest) to enable forest for their own grammar. This
	// bypasses the byte-range parity certification built-ins undergo — the
	// decline->production fallback still prevents hard failures on declined inputs,
	// but a clean-but-different tree is the consumer's responsibility.
	WantsForest bool

	// LanguageVersion is the tree-sitter language ABI version.
	// A value of 0 means "unknown/unspecified" and is treated as compatible.
	LanguageVersion uint32

	// Counts
	SymbolCount        uint32
	TokenCount         uint32
	ExternalTokenCount uint32
	StateCount         uint32
	LargeStateCount    uint32
	FieldCount         uint32
	ProductionIDCount  uint32

	// Symbol metadata
	SymbolNames    []string
	SymbolMetadata []SymbolMetadata
	FieldNames     []string // index 0 is ""

	// Parse tables
	ParseTable         [][]uint16 // dense: [state][symbol] -> action index
	SmallParseTable    []uint16   // compressed sparse table
	SmallParseTableMap []uint32   // state -> offset into SmallParseTable
	ParseActions       []ParseActionEntry
	// LargeStateGotos stores nonterminal GOTO targets that do not fit in the
	// uint16 parse-table cells used by tree-sitter C tables. Keys are
	// uint64(state)<<32 | uint64(symbol). Terminal actions must never live here.
	//
	// This is the only exported map field on Language, which matters for blob
	// serialization: gob's map codec iterates via reflect's randomized
	// MapRange, so blob encoders never gob-encode this field directly when
	// it's non-empty (today, only c_sharp populates it). See
	// large_state_gotos_trailer.go for the deterministic encode/decode path.
	LargeStateGotos map[uint64]StateID

	// ReduceChainHints are optional generated hot-path hints for deterministic
	// reduce runs. They are only consumed when reduce-chain hints are enabled.
	ReduceChainHints []ReduceChainHint

	// ConflictPolicies are optional deterministic conflict policies derived
	// from grammar tables.
	ConflictPolicies []ConflictPolicy

	// Lex tables
	LexModes            []LexMode
	LexStates           []LexState // main lexer DFA
	KeywordLexStates    []LexState // keyword lexer DFA (optional)
	KeywordCaptureToken Symbol
	// LayoutFallbackLexState is an optional broad DFA start state used only in
	// layout-entry parser states. It lets the runtime avoid skipping over
	// zero-width external layout markers before the layout scanner fires.
	LayoutFallbackLexState    uint16
	HasLayoutFallbackLexState bool

	// Field mapping
	FieldMapSlices  [][2]uint16 // [production_id] -> (index, length)
	FieldMapEntries []FieldMapEntry

	// Alias sequences
	AliasSequences [][]Symbol // [production_id][child_index] -> alias symbol

	// ProductionSignatures records LHS/RHS shape for grammargen productions.
	// It is separate from ProductionID because ProductionID is deliberately
	// compacted by field/alias pattern and is not a unique RHS identity.
	ProductionSignatures []ProductionSignature

	// Primary state IDs (for table dedup)
	PrimaryStateIDs []StateID

	// ABI 15: Reserved words — flat array indexed by
	// (reserved_word_set_id * MaxReservedWordSetSize + i), terminated by 0.
	ReservedWords          []Symbol
	MaxReservedWordSetSize uint16

	// ABI 15: Supertype hierarchy
	SupertypeSymbols    []Symbol
	SupertypeMapSlices  [][2]uint16 // [supertype_symbol] -> (index, length)
	SupertypeMapEntries []Symbol

	// HiddenChoicePassthroughSymbols marks generated hidden nonterminals whose
	// productions are only neutral single-symbol pass-throughs. These wrappers
	// are structural parser routes, not tree nodes; checked-in blobs leave this
	// nil to preserve legacy behavior.
	HiddenChoicePassthroughSymbols []bool

	// ABI 15: Grammar semantic version
	Metadata LanguageMetadata

	// External scanner (nil if not needed)
	ExternalScanner ExternalScanner
	ExternalSymbols []Symbol // external token index -> symbol
	// ImmediateTokens is a bitmask of symbol IDs that are token.immediate() tokens.
	// When the lexer matches one of these after consuming whitespace, the match
	// should be rejected — immediate tokens must match at the original position.
	// nil means no immediate tokens (common for ts2go grammars).
	ImmediateTokens []bool
	// ZeroWidthTokens is a bitmask of symbol IDs whose DFA terminal pattern can
	// intentionally match empty input. nil means this information is unavailable,
	// which preserves historical lexer behavior for ts2go blobs.
	ZeroWidthTokens []bool

	// ExternalLexStates maps external lex state IDs (from LexMode.ExternalLexState)
	// to a boolean slice indicating which external tokens are valid. Row 0 is
	// always all-false (no external tokens valid). When non-nil, this table is
	// used instead of parse-action-table probing to compute validSymbols for the
	// external scanner, matching C tree-sitter's ts_external_scanner_states.
	ExternalLexStates [][]bool

	// InitialState is the parser's start state. In tree-sitter grammars
	// this is always 1 (state 0 is reserved for error recovery). For
	// hand-built grammars it defaults to 0.
	InitialState StateID

	// NonTerminalAliasMap mirrors tree-sitter C's ts_non_terminal_alias_map.
	// Rows are indexed by nonterminal symbol and contain aliases that require
	// preserving the wrapper during alias-bearing reductions. This is cold
	// metadata and intentionally lives in the struct's cold tail so parser hot
	// field offsets stay stable.
	NonTerminalAliasMap [][]Symbol

	// ExternalScannerFullParseRetryPolicy is a certified language-level policy
	// for scheduling the extra external-scanner full-parse retry. Zero preserves
	// the generic behavior for legacy blobs and caller-constructed languages.
	ExternalScannerFullParseRetryPolicy ExternalScannerFullParseRetryPolicy

	// FullParseAcceptedErrorRetryProfile is certified against an exact language
	// blob. Zero preserves widened-stack and no-stacks retries for legacy blobs,
	// caller-constructed languages, and language overrides.
	FullParseAcceptedErrorRetryProfile FullParseAcceptedErrorRetryProfile

	// AutomaticForestMemoryAllowanceBytes bounds only the speculative forest
	// phase used by automatic dispatch. Zero preserves the full parse budget for
	// legacy blobs, caller-constructed languages, language overrides, and
	// explicit ParseForestExperimental calls. Built-in values are certified and
	// attached only after exact blob-identity verification.
	AutomaticForestMemoryAllowanceBytes int64

	// AutomaticForestEnabledByDefault certifies the speculative forest route for
	// this exact language artifact. Checked-in built-ins receive this bit only
	// from an exact-blob runtime profile; legacy blobs, same-name custom grammars,
	// adapted grammars, and overrides retain the conservative false default.
	AutomaticForestEnabledByDefault bool

	// FullParseArenaDensityCapEnabled explicitly opts this language into the ASCII
	// structural-density arena cap. Checked-in built-ins are enabled automatically
	// only through an exact-blob runtime profile. Callers may explicitly opt in
	// custom or adapted languages; false preserves the baseline arena policy.
	FullParseArenaDensityCapEnabled bool

	// FullParseGSSConvergenceEnabled certifies faithful convergence for this
	// language artifact. The parser keeps clean alternatives in the graph when
	// one stack survives each merge group. Checked-in languages receive this
	// setting only through an exact blob profile. Callers can enable it for
	// custom languages after equivalent tree tests. An explicit merge limit of
	// one also enables this behavior for a fresh full parse.
	FullParseGSSConvergenceEnabled bool

	// NativeResultCompatibility identifies result-tree shapes produced natively
	// by this exact language artifact. Zero keeps conservative post-parse
	// compatibility fallbacks for legacy blobs, generated grammars, caller-built
	// languages, and overrides whose native behavior has not been certified.
	NativeResultCompatibility ResultCompatibilityCapability

	// NativeUnaryWrapperFlattening identifies exact same-span unary wrappers
	// that C omits below a public parent in one parser state. Exact runtime
	// profiles populate the symbol and state identities. Custom and stale
	// artifacts keep the wrappers.
	NativeUnaryWrapperFlattening []UnaryWrapperFlatteningRule

	// CompactConvergedReductionSplitDropsCertified permits the compact
	// fresh-full route to accept after it drops a no-action head descended from
	// a converged-path reduction split. Exact built-in artifact profiles set
	// this only after C-oracle parity proves that production selects the same
	// surviving path. Custom and adapted languages fail closed by default.
	CompactConvergedReductionSplitDropsCertified bool

	// CompactEOFAcceptNoActionSiblingsCertified permits one authenticated EOF
	// accept head to discard siblings that have no action for that same EOF.
	// Exact built-in profiles set this only after C-oracle parity proves the
	// accepted head matches production. Custom and adapted languages fail closed.
	CompactEOFAcceptNoActionSiblingsCertified bool

	// CompactPrimaryAcceptanceDerivationCertified permits the compact fresh-full
	// route to select one primary derivation over secondary conflict derivations.
	// The primary score must be at least every secondary score. Exact built-in
	// profiles set this only after C-oracle parity proves production selects it.
	CompactPrimaryAcceptanceDerivationCertified bool

	// ExactStackNodeEquivalenceCertified preserves deep stack-node alternatives
	// until generic result selection. Exact built-in profiles set this only when
	// bounded equivalence can merge parity-relevant shapes. Custom, adapted, and
	// stale artifacts retain bounded equivalence unless callers opt in.
	ExactStackNodeEquivalenceCertified bool
	// contains filtered or unexported fields
}

Language holds all data needed to parse a specific language. It mirrors tree-sitter's TSLanguage C struct, translated into idiomatic Go types with slice-based tables instead of raw pointers.

func LoadLanguage added in v0.9.0 

func LoadLanguage(data []byte) (*Language, error)

LoadLanguage deserializes a compressed grammar blob into a Language. Blobs are produced by EncodeLanguageBlob, grammargen.Generate, or the grammar build toolchain. This is the only function needed at runtime to load pre-compiled grammars — no grammargen import required. It accepts legacy gzip blobs and version-enveloped trailer-bearing blobs.

func (*Language) CompatibleWithRuntime 

func (l *Language) CompatibleWithRuntime() bool

CompatibleWithRuntime reports whether this language can be parsed by the current runtime version. Unspecified versions (0) are treated as compatible.

func (*Language) FieldByName 

func (l *Language) FieldByName(name string) (FieldID, bool)

FieldByName returns the field ID for a given name, or (0, false) if not found. Builds an internal map on first call for O(1) subsequent lookups.

func (*Language) IsSupertype added in v0.6.0 

func (l *Language) IsSupertype(sym Symbol) bool

IsSupertype reports whether sym is a supertype symbol.

func (*Language) KeywordLexAsciiTable added in v0.10.2 

func (l *Language) KeywordLexAsciiTable() [][128]int32

KeywordLexAsciiTable returns the ASCII fast-path table for the keyword lexer DFA.

func (*Language) LexAsciiTable added in v0.10.2 

func (l *Language) LexAsciiTable() [][128]int32

LexAsciiTable returns the pre-built ASCII fast-path transition table for the main lexer DFA. The table is built once per Language. Entry format: 

bit 31 set  → skip transition (consume and reset token start)
bits 0-30   → next state ID (lexAsciiNoMatch if no transition)

func (*Language) LexModeStarts added in v0.19.0 

func (l *Language) LexModeStarts() []lexModeStart

func (*Language) PublicSymbol added in v0.7.0 

func (l *Language) PublicSymbol(sym Symbol) Symbol

PublicSymbol maps an internal symbol to its canonical public form. Multiple internal symbols may share the same visible name (e.g. HTML's _start_tag_name and _end_tag_name both display as "tag_name"). PublicSymbol returns the first symbol with that name, matching what SymbolByName returns. This ensures query patterns compiled with SymbolByName match nodes regardless of which alias produced them.

func (*Language) PublicSymbolForNamedness added in v0.19.0 

func (l *Language) PublicSymbolForNamedness(sym Symbol, named bool) Symbol

PublicSymbolForNamedness maps an internal symbol to the canonical public symbol with the same display name and requested namedness. This lets query matching distinguish named nodes from anonymous tokens that share text.

func (*Language) Size added in v0.20.6 

func (l *Language) Size() int64

Size returns an approximate number of bytes retained by the decoded language tables and lazily-built lookup caches. It is intended for diagnostics and cache policy decisions, not as an exact Go heap accounting API.

func (*Language) SupertypeChildren added in v0.6.0 

func (l *Language) SupertypeChildren(sym Symbol) []Symbol

SupertypeChildren returns the subtype symbols for a given supertype. Returns nil if sym is not a supertype or has no entries.

func (*Language) SymbolByName 

func (l *Language) SymbolByName(name string) (Symbol, bool)

SymbolByName returns the symbol ID for a given name, or (0, false) if not found. The "_" wildcard returns (0, true) as a special case. Builds an internal map on first call for O(1) subsequent lookups.

func (*Language) TokenSymbolsByName 

func (l *Language) TokenSymbolsByName(name string) []Symbol

TokenSymbolsByName returns all terminal token symbols whose display name matches name. The returned symbols are in grammar order.

func (*Language) Version 

func (l *Language) Version() uint32

Version returns the tree-sitter language ABI version.

type LanguageMetadata added in v0.6.0 

type LanguageMetadata struct {
	MajorVersion uint8
	MinorVersion uint8
	PatchVersion uint8
}

LanguageMetadata holds the grammar's semantic version (ABI 15+).

type LexMode 

type LexMode struct {
	LexState                  uint16
	ExternalLexState          uint16
	ReservedWordSetID         uint16
	AfterWhitespaceLexState   uint16 // DFA start state to use after whitespace (0 = same as LexState)
	LexStateID                uint32 // widened DFA start state for grammargen tables with >64K lexer states
	AfterWhitespaceLexStateID uint32
}

LexMode maps a parser state to its lexer configuration.

func (LexMode) AfterWhitespaceLexStateIndex added in v0.16.0 

func (m LexMode) AfterWhitespaceLexStateIndex() uint32

AfterWhitespaceLexStateIndex returns the alternate DFA start state used after whitespace, or zero when the primary lex state should be used.

func (LexMode) LexStateIndex added in v0.16.0 

func (m LexMode) LexStateIndex() uint32

LexStateIndex returns the DFA start state for this lex mode. Older grammar blobs only populate the uint16 LexState field; grammargen-generated tables can populate LexStateID when the DFA table exceeds 64K states.

func (*LexMode) SetAfterWhitespaceLexStateIndex added in v0.16.0 

func (m *LexMode) SetAfterWhitespaceLexStateIndex(idx uint32)

func (*LexMode) SetLexStateIndex added in v0.16.0 

func (m *LexMode) SetLexStateIndex(idx uint32)

type LexState 

type LexState struct {
	AcceptToken    Symbol // 0 if this state doesn't accept
	AcceptPriority int16  // lower = higher priority (0 for ts2go blobs = longest-match)
	Skip           bool   // true if accepted chars are whitespace
	Default        int    // default next state (-1 if none)
	EOF            int    // state on EOF (-1 if none)
	Transitions    []LexTransition
}

LexState is one state in the table-driven lexer DFA.

type LexTransition 

type LexTransition struct {
	Lo, Hi    rune // inclusive character range
	NextState int
	// Skip mirrors tree-sitter's SKIP(state): consume the matched rune
	// and continue lexing while resetting token start.
	Skip bool
}

LexTransition maps a character range to a next state.

type Lexer 

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer tokenizes source text using a table-driven DFA.

func NewLexer 

func NewLexer(states []LexState, source []byte) *Lexer

NewLexer creates a new Lexer that will tokenize source using the given DFA state table.

func (*Lexer) Next 

func (l *Lexer) Next(startState uint32) Token

Next lexes the next token starting from the given lex state index. It automatically skips tokens from states where Skip=true (whitespace). Returns a zero-Symbol token with StartByte==EndByte at EOF.

func (*Lexer) NextWithErrorRuns added in v0.21.0 

func (l *Lexer) NextWithErrorRuns(startState uint32) Token

NextWithErrorRuns behaves like Next, except that bytes for which no accepting DFA state exists are not silently dropped: the whole unlexable run is consumed and returned as an errorSymbol token. This mirrors C ts_parser__lex, which surfaces skipped characters as an error subtree — the run starts after any whitespace the DFA legitimately skipped and ends at the first position where a token can be lexed (or EOF).

type LookaheadIterator added in v0.6.0 

type LookaheadIterator struct {
	// contains filtered or unexported fields
}

LookaheadIterator iterates over valid symbols for a given parse state. It precomputes the full set of symbols that have valid parse actions in the specified state, enabling autocomplete and error diagnostic use cases.

func NewLookaheadIterator added in v0.6.0 

func NewLookaheadIterator(lang *Language, state StateID) (*LookaheadIterator, error)

NewLookaheadIterator creates an iterator over all symbols that have valid parse actions in the given state. Returns an error if the state is out of range for the language's parse tables.

func (*LookaheadIterator) CurrentSymbol added in v0.6.0 

func (it *LookaheadIterator) CurrentSymbol() Symbol

CurrentSymbol returns the symbol at the current iterator position. Must be called after a successful Next().

func (*LookaheadIterator) CurrentSymbolName added in v0.6.0 

func (it *LookaheadIterator) CurrentSymbolName() string

CurrentSymbolName returns the name of the symbol at the current iterator position. Returns "" if the position is invalid or the symbol has no name.

func (*LookaheadIterator) Language added in v0.6.0 

func (it *LookaheadIterator) Language() *Language

Language returns the language associated with this iterator.

func (*LookaheadIterator) Next added in v0.6.0 

func (it *LookaheadIterator) Next() bool

Next advances the iterator to the next valid symbol. Returns false when there are no more symbols.

func (*LookaheadIterator) ResetState added in v0.6.0 

func (it *LookaheadIterator) ResetState(state StateID) error

ResetState resets the iterator to enumerate valid symbols for a different parse state within the same language. Returns an error if the state is out of range.

type Node 

type Node struct {
	// contains filtered or unexported fields
}

Node is a syntax tree node.

func NewLeafNode 

func NewLeafNode(sym Symbol, named bool, startByte, endByte uint32, startPoint, endPoint Point) *Node

NewLeafNode creates a terminal/leaf node.

func NewParentNode 

func NewParentNode(sym Symbol, named bool, children []*Node, fieldIDs []FieldID, productionID uint16) *Node

NewParentNode creates a non-terminal node with children. It sets parent pointers on all children and computes byte/point spans from the first and last children. If any child has an error, the parent is marked as having an error too.

func (*Node) Child 

func (n *Node) Child(i int) *Node

Child returns the i-th child, or nil if i is out of range.

func (*Node) ChildByFieldName 

func (n *Node) ChildByFieldName(name string, lang *Language) *Node

ChildByFieldName returns the first child assigned to the given field name, or nil if no child has that field. The Language is needed to resolve field names to IDs. Uses Language.FieldByName for O(1) lookup.

func (*Node) ChildCount 

func (n *Node) ChildCount() int

ChildCount returns the number of children (both named and anonymous).

func (*Node) Children 

func (n *Node) Children() []*Node

Children returns a slice of all children.

func (*Node) DescendantForByteRange added in v0.6.0 

func (n *Node) DescendantForByteRange(startByte, endByte uint32) *Node

DescendantForByteRange returns the smallest descendant selected by upstream tree-sitter's byte-range walk. For a valid unsatisfiable or out-of-bounds range it returns the receiver; a nil receiver or reversed range returns nil.

func (*Node) DescendantForPointRange added in v0.6.0 

func (n *Node) DescendantForPointRange(startPoint, endPoint Point) *Node

DescendantForPointRange returns the smallest descendant selected by upstream tree-sitter's point-range walk. For a valid unsatisfiable or out-of-bounds range it returns the receiver; a nil receiver or reversed range returns nil.

func (*Node) Edit added in v0.7.0 

func (n *Node) Edit(edit InputEdit)

Edit adjusts this node's byte/point span for a source edit.

If the node belongs to a larger tree, the edit is applied from the containing root so sibling and ancestor spans remain consistent. Unlike Tree.Edit, this method does not record edit history on a Tree.

func (*Node) EndByte 

func (n *Node) EndByte() uint32

EndByte returns the byte offset where this node ends (exclusive).

func (*Node) EndPoint 

func (n *Node) EndPoint() Point

EndPoint returns the row/column position where this node ends.

func (*Node) FieldNameForChild added in v0.6.0 

func (n *Node) FieldNameForChild(i int, lang *Language) string

FieldNameForChild returns the field name assigned to the i-th child, or an empty string when no field is assigned.

func (*Node) HasChanges added in v0.6.0 

func (n *Node) HasChanges() bool

HasChanges reports whether this node was marked dirty by Tree.Edit.

func (*Node) HasError 

func (n *Node) HasError() bool

HasError reports whether this node or any descendant contains a parse error.

func (*Node) IsError added in v0.6.0 

func (n *Node) IsError() bool

IsError reports whether this node is an explicit error node.

func (*Node) IsExtra added in v0.6.0 

func (n *Node) IsExtra() bool

IsExtra reports whether this node was marked as extra syntax (e.g. whitespace/comments outside the core parse structure).

func (*Node) IsMissing 

func (n *Node) IsMissing() bool

IsMissing reports whether this node was inserted by error recovery.

func (*Node) IsNamed 

func (n *Node) IsNamed() bool

IsNamed reports whether this is a named node (as opposed to anonymous syntax like punctuation).

func (*Node) NamedChild 

func (n *Node) NamedChild(i int) *Node

NamedChild returns the i-th named child (skipping anonymous children), or nil if i is out of range.

func (*Node) NamedChildCount 

func (n *Node) NamedChildCount() int

NamedChildCount returns the number of named children.

func (*Node) NamedDescendantForByteRange added in v0.6.0 

func (n *Node) NamedDescendantForByteRange(startByte, endByte uint32) *Node

NamedDescendantForByteRange returns the smallest named descendant selected by upstream tree-sitter's byte-range walk. For a valid unsatisfiable or out-of-bounds range it returns the receiver; a nil receiver or reversed range returns nil.

func (*Node) NamedDescendantForPointRange added in v0.6.0 

func (n *Node) NamedDescendantForPointRange(startPoint, endPoint Point) *Node

NamedDescendantForPointRange returns the smallest named descendant selected by upstream tree-sitter's point-range walk. For a valid unsatisfiable or out-of-bounds range it returns the receiver; a nil receiver or reversed range returns nil.

func (*Node) NamedNodeAtByte added in v0.20.6 

func (n *Node) NamedNodeAtByte(byteOffset uint32) *Node

NamedNodeAtByte returns the smallest named descendant that contains byteOffset. It follows the same boundary behavior as NodeAtByte.

func (*Node) NextSibling 

func (n *Node) NextSibling() *Node

NextSibling returns the next sibling node, or nil when this is the last child or has no parent.

func (*Node) NodeAtByte added in v0.20.6 

func (n *Node) NodeAtByte(byteOffset uint32) *Node

NodeAtByte returns the smallest descendant that contains byteOffset. If the offset is exactly at this node's end byte, it performs a zero-width lookup at that boundary. Returns nil when the offset is outside this node.

func (*Node) Parent 

func (n *Node) Parent() *Node

Parent returns this node's parent, or nil if it is the root.

func (*Node) ParseState 

func (n *Node) ParseState() StateID

ParseState returns the parser state associated with this node.

func (*Node) PreGotoState added in v0.6.0 

func (n *Node) PreGotoState() StateID

PreGotoState returns the parser state that was on top of the stack before this node was pushed (i.e., the state exposed after popping children during reduce). For non-leaf nodes: lookupGoto(PreGotoState, Symbol) == ParseState.

func (*Node) PrevSibling 

func (n *Node) PrevSibling() *Node

PrevSibling returns the previous sibling node, or nil when this is the first child or has no parent.

func (*Node) Range 

func (n *Node) Range() Range

Range returns the full span of this node as a Range.

func (*Node) SExpr added in v0.6.0 

func (n *Node) SExpr(lang *Language) string

SExpr returns a tree-sitter-style S-expression for this node. It includes only named nodes for stable debug snapshots.

func (*Node) StartByte 

func (n *Node) StartByte() uint32

StartByte returns the byte offset where this node begins.

func (*Node) StartPoint 

func (n *Node) StartPoint() Point

StartPoint returns the row/column position where this node begins.

func (*Node) Symbol 

func (n *Node) Symbol() Symbol

Symbol returns the node's grammar symbol.

func (*Node) Text 

func (n *Node) Text(source []byte) string

Text returns the source text covered by this node. Returns an empty string for nil nodes or invalid byte ranges.

func (*Node) Type 

func (n *Node) Type(lang *Language) string

Type returns the node's type name from the language.

type NormalizationPassRuntime added in v0.20.0 

type NormalizationPassRuntime struct {
	Name           string
	Checked        uint64
	Run            uint64
	NodesVisited   uint64
	NodesRewritten uint64
	Nanos          int64
}

type ParseAction 

type ParseAction struct {
	Type              ParseActionType
	State             StateID // target state (shift/recover)
	Symbol            Symbol  // reduced symbol (reduce)
	ChildCount        uint8   // children consumed (reduce)
	DynamicPrecedence int16   // precedence (reduce)
	ProductionID      uint16  // which production (reduce)
	Extra             bool    // is this an extra token (shift)
	ExtraChain        bool    // does this shift enter a nonterminal extra chain
	Repetition        bool    // is this a repetition (shift)
}

ParseAction is a single parser action from the parse table.

type ParseActionEntry 

type ParseActionEntry struct {
	Reusable bool
	Actions  []ParseAction
}

ParseActionEntry is a group of actions for a (state, symbol) pair.

type ParseActionTiming added in v0.19.0 

type ParseActionTiming struct {
	ExtraShiftNanos      int64
	NoActionNanos        int64
	NoActionRelexNanos   int64
	NoActionMissingNanos int64
	NoActionRecoverNanos int64
	NoActionErrorNanos   int64
	ConflictChoiceNanos  int64
	ConflictForkNanos    int64
	SingleShiftNanos     int64
	SingleReduceNanos    int64
	SingleAcceptNanos    int64
	SingleRecoverNanos   int64
	SingleOtherNanos     int64
}

type ParseActionType 

type ParseActionType uint8

ParseActionType identifies the kind of parse action. 

const (
	ParseActionShift ParseActionType = iota
	ParseActionReduce
	ParseActionAccept
	ParseActionRecover
)

type ParseEquivStateRuntime added in v0.19.0 

type ParseEquivStateRuntime struct {
	State                                 StateID
	StackEquivCalls                       uint64
	StackEquivTrue                        uint64
	StackEquivDepthMismatch               uint64
	StackEquivHashMismatch                uint64
	StackEquivStateMismatch               uint64
	StackEquivPayloadMismatch             uint64
	StackEquivEntryCompares               uint64
	StackEquivStateMismatchDepthSum       uint64
	StackEquivStateMismatchMaxDepth       uint32
	StackEquivStateMismatchDepthBuckets   [stackEquivMismatchDepthBucketCount]uint64
	StackEquivPayloadMismatchDepthSum     uint64
	StackEquivPayloadMismatchMaxDepth     uint32
	StackEquivPayloadMismatchDepthBuckets [stackEquivMismatchDepthBucketCount]uint64
	StackEquivPayloadHeaderSigDiff        uint64
	StackEquivPayloadHeaderSigSame        uint64
	StackEquivPayloadShallowSigDiff       uint64
	StackEquivPayloadShallowSigSame       uint64
	StackEquivPairKeyed                   uint64
	StackEquivPairUnkeyed                 uint64
	StackEquivPairRepeats                 uint64
	StackEquivPairRepeatTrue              uint64
	StackEquivPairRepeatFalse             uint64
	StackEquivPairRepeatMismatch          uint64
	StackEquivPairStores                  uint64
	MergeHeaderEqTotal                    uint64
	MergeDeepTrue                         uint64
	MergeDeepFalse                        uint64
	MergeHeaderDeepDivergent              uint64
	EquivCacheLookups                     uint64
	EquivCacheHits                        uint64
	EquivCacheStores                      uint64
	EquivCacheMisses                      uint64
	EquivCacheTrueHits                    uint64
	EquivCacheFalseHits                   uint64
	EquivCacheEpochMisses                 uint64
	EquivCacheKeyMisses                   uint64
	EquivCacheVersionMisses               uint64
	EquivSkipError                        uint64
	EquivSkipLeaf                         uint64
	EquivSkipFieldMismatch                uint64
	EquivExactCalls                       uint64
	EquivExactTrue                        uint64
	EquivExactPointerTrue                 uint64
	EquivExactNilMismatch                 uint64
	EquivExactHeaderMismatch              uint64
	EquivExactChildMismatch               uint64
	EquivExactTerminalCalls               uint64
	EquivExactTerminalTrue                uint64
	EquivExactTerminalFalse               uint64
	EquivFrontierCalls                    uint64
	EquivFrontierTrue                     uint64
	EquivExactChildCompares               uint64
	EquivFrontierChildScans               uint64
	EquivFrontierCandidateCompares        uint64
}

type ParseOption added in v0.6.0 

type ParseOption func(*parseConfig)

ParseOption configures ParseWith behavior.

func WithOldTree added in v0.6.0 

func WithOldTree(oldTree *Tree) ParseOption

WithOldTree enables incremental parsing against an edited prior tree.

func WithProfiling added in v0.6.0 

func WithProfiling() ParseOption

WithProfiling enables incremental parse attribution in ParseResult.Profile.

func WithTokenSource added in v0.6.0 

func WithTokenSource(ts TokenSource) ParseOption

WithTokenSource provides a custom token source for parsing.

type ParseReduceTiming added in v0.19.0 

type ParseReduceTiming struct {
	RangeNanos         int64
	PendingParentNanos int64
	ChildBuildNanos    int64
	ParentBuildNanos   int64
	SpanNanos          int64
	StackPushNanos     int64
	NoTreeBuildNanos   int64
}

type ParseResult added in v0.6.0 

type ParseResult struct {
	Tree *Tree
	// Profile is populated only when ParseWith uses WithProfiling for
	// incremental parsing.
	Profile IncrementalParseProfile
	// ProfileAvailable reports whether Profile contains attribution data.
	ProfileAvailable bool
}

ParseResult is returned by ParseWith.

type ParseRuntime added in v0.6.0 

type ParseRuntime struct {
	StopReason     ParseStopReason
	ForestFastPath bool
	// IncrementalAcceptedErrorRetryAttempts records the bounded second
	// incremental pass used when an accepted, full-span ERROR tree was produced
	// under the wider incremental merge policy. The retry keeps the edited old
	// tree and reruns with the ordinary full-parse merge cap; it is not a fresh
	// parse fallback.
	IncrementalAcceptedErrorRetryAttempts uint8
	// IncrementalAcceptedErrorRetryAdopted is true only when that retry produced
	// a strictly better tree and replaced the first incremental result.
	IncrementalAcceptedErrorRetryAdopted bool
	// IncrementalAcceptedErrorRetryMergePerKey is the exact merge-per-key cap
	// used by the bounded retry.
	IncrementalAcceptedErrorRetryMergePerKey int
	// IncrementalAcceptedErrorRetryCause is a stable diagnostic reason for the
	// retry so corpus and benchmark matrices can distinguish this path.
	IncrementalAcceptedErrorRetryCause IncrementalRetryCause
	// IncrementalOldTreeReuseRoute reports whether this result was produced by
	// an actual old-tree reuse parse rather than an internal fresh fallback.
	IncrementalOldTreeReuseRoute bool
	// CRecoveryEnteredErrorState is true when the faithful C error-recovery
	// port (parser_recover_c.go) actually ran ts_parser__handle_error at
	// least once while producing this specific tree — i.e. some no-action
	// point was hit for the current lookahead. This is NOT proof the input is
	// malformed: LALR table limitations routinely drive well-formed,
	// compiling input into a momentary no-action point that C-recovery
	// resolves losslessly (ordinary GLR disambiguation). It is only a cheap
	// pre-filter for Parse()'s post-parse swallowed-error safety net
	// (resolveCRecoverySwallowedError) — see CRecoveryDroppedErrorForClean
	// for the actual, precise suspicion signal. It is captured per finalized
	// tree (not read from a raw Parser field) so a discarded retry attempt
	// can never leave a stale value on the tree that is actually returned.
	CRecoveryEnteredErrorState bool
	// CRecoveryDroppedErrorForClean is true when the stack SELECTED as this
	// tree's parse result (buildResultFromGLR, parser_result.go) carried an
	// unvalidated C-recovery marker (see glrStack.cRecoveryUnvalidatedMarker)
	// with no unflagged sibling reaching the same final position — i.e. the
	// selected lineage itself created a real ERROR node via cRecoverToState
	// (for a single-stack dead end with a small recovered span) and was
	// never re-validated by another cost competition. This is the precise
	// signature of the swallowed-error defect class (see
	// resolveCRecoverySwallowedError): unlike an ordinary clean recovery (no
	// unvalidated marker, or a corroborating clean sibling, at the same
	// position), this means the specific result being returned lost real
	// ERROR content somewhere along its own lineage. Deliberately scoped to
	// the selected result only — NOT set for drops/forks on discarded
	// lineages elsewhere in the parse, nor for large/multi-stack recoveries;
	// both were tried and found to fire on ordinary GLR disambiguation for a
	// measurable fraction of valid, compiling source in a real repo-file
	// walk (the discarded-lineage version: thousands of times per large Go
	// file) and were the cause of a prior, over-broad version of this
	// signal's false-fire rate.
	CRecoveryDroppedErrorForClean bool
	// CRecoverySwallowedErrorFallbackAttempted is true when
	// resolveCRecoverySwallowedError actually re-parsed the source with the
	// C-recovery gate disabled to double-check a suspicious clean result (see
	// CRecoveryDroppedErrorForClean). Diagnostic only — lets callers measure
	// the fallback's false-fire rate (extra latency from a re-parse that is
	// usually discarded) against their own corpora, e.g. by walking a tree of
	// known-valid source files and counting how often this is true.
	CRecoverySwallowedErrorFallbackAttempted      bool
	SourceLen                                     uint32
	ExpectedEOFByte                               uint32
	RootEndByte                                   uint32
	Truncated                                     bool
	TokenSourceEOFEarly                           bool
	TokensConsumed                                uint64
	LastTokenEndByte                              uint32
	LastTokenSymbol                               Symbol
	LastTokenWasEOF                               bool
	StopDiagnosticCaptured                        bool
	StopDiagnosticCRecoveryEnabled                bool
	StopDiagnosticCRecoveryGateReason             string
	StopDiagnosticRecoverActionAvailable          bool
	StopDiagnosticLastStackState                  StateID
	StopDiagnosticLastStackByte                   uint32
	StopDiagnosticLastStackDepth                  int
	StopDiagnosticTokenSymbol                     Symbol
	StopDiagnosticTokenStartByte                  uint32
	StopDiagnosticTokenEndByte                    uint32
	StopDiagnosticTokenNoLookahead                bool
	StopDiagnosticRootType                        string
	StopDiagnosticRootStartByte                   uint32
	StopDiagnosticRootEndByte                     uint32
	StopDiagnosticRootHasError                    bool
	StopDiagnosticFirstErrorFound                 bool
	StopDiagnosticFirstErrorStartByte             uint32
	StopDiagnosticFirstErrorEndByte               uint32
	StopDiagnosticFrontierStacks                  string
	StopDiagnosticFrontierActions                 string
	StopDiagnosticSameHeaderGroups                string
	StopDiagnosticCondenseGating                  string
	StopDiagnosticActionCaptured                  bool
	StopDiagnosticActionPhase                     string
	StopDiagnosticActionStackState                StateID
	StopDiagnosticActionStackByte                 uint32
	StopDiagnosticActionStackDepth                int
	StopDiagnosticActionTokenSymbol               Symbol
	StopDiagnosticActionTokenStartByte            uint32
	StopDiagnosticActionTokenEndByte              uint32
	StopDiagnosticActionTokenNoLookahead          bool
	StopDiagnosticActionType                      ParseActionType
	StopDiagnosticActionState                     StateID
	StopDiagnosticActionSymbol                    Symbol
	StopDiagnosticActionChildCount                uint8
	StopDiagnosticActionProductionID              uint16
	StopDiagnosticActionDynamicPrecedence         int16
	StopDiagnosticActionCount                     int
	StopDiagnosticActionResultState               StateID
	StopDiagnosticActionInReduceChain             bool
	StopDiagnosticActionReduceChainStep           int
	StopDiagnosticActionRepeatedSignatureCount    int
	StopDiagnosticActionReduceChainCycle          bool
	StopDiagnosticActionForceAdvanceAfterReduce   bool
	StopDiagnosticActionPostDispatchDefaultReduce bool
	StopDiagnosticActionAnyReduced                bool
	StopDiagnosticActionConsumedToken             bool
	StopDiagnosticActionDispatchShiftActions      int
	StopDiagnosticActionDispatchReduceActions     int
	StopDiagnosticActionDispatchAcceptActions     int
	StopDiagnosticActionDispatchRecoverActions    int
	StopDiagnosticActionDispatchOtherActions      int
	StopDiagnosticLastReduceCaptured              bool
	StopDiagnosticLastReducePhase                 string
	StopDiagnosticLastReduceStackState            StateID
	StopDiagnosticLastReduceStackByte             uint32
	StopDiagnosticLastReduceStackDepth            int
	StopDiagnosticLastReduceSymbol                Symbol
	StopDiagnosticLastReduceChildCount            uint8
	StopDiagnosticLastReduceProductionID          uint16
	StopDiagnosticLastReduceDynamicPrecedence     int16
	StopDiagnosticLastReduceResultState           StateID
	StopDiagnosticLastReduceInChain               bool
	StopDiagnosticLastReduceChainStep             int
	StopDiagnosticLastReduceRepeatedSigCount      int
	StopDiagnosticLastReduceChainCycle            bool
	IterationLimit                                int
	StackDepthLimit                               int
	NodeLimit                                     int
	MemoryBudgetBytes                             int64
	MemoryBudgetStopSource                        string // First budget guard: arena, scratch, runtime_heap, runtime_sys, or hard_ceiling.
	RuntimeHeapGrowthBytes                        uint64
	RuntimeSysGrowthBytes                         uint64
	Iterations                                    int
	NodesAllocated                                int
	ArenaBytesAllocated                           int64
	ArenaBaselineBytes                            int64
	ScratchBytesAllocated                         int64
	ScratchBaselineBytes                          int64
	EntryScratchBytesAllocated                    int64
	EntryScratchPeak                              uint64
	GSSBytesAllocated                             int64
	GSSBaselineBytes                              int64
	GSSSlabCount                                  int
	GSSNodesUsed                                  int
	GSSNodesCapacity                              int
	GSSDemotions                                  uint64
	GSSNodesDemoted                               uint64
	TransientScratchCheckpoints                   uint64
	PeakStackDepth                                int
	MaxStacksSeen                                 int
	SingleStackIterations                         int
	MultiStackIterations                          int
	SingleStackTokens                             uint64
	MultiStackTokens                              uint64
	SingleStackGSSNodes                           uint64
	MultiStackGSSNodes                            uint64
	GSSNodesAllocated                             uint64
	GSSNodesRetained                              uint64
	GSSNodesDroppedSameToken                      uint64
	ParentNodesAllocated                          uint64
	ParentNodesRetained                           uint64
	ParentNodesDroppedSameToken                   uint64
	LeafNodesAllocated                            uint64
	LeafNodesRetained                             uint64
	LeafNodesDroppedSameToken                     uint64
	ChildSlicesAllocated                          uint64
	ChildSlicesRetained                           uint64
	ChildSlicesDroppedSameToken                   uint64
	ChildPointersAllocated                        uint64
	ChildPointersRetained                         uint64
	ChildPointersDroppedSameToken                 uint64
	ReduceChildFastGSS                            ReduceChildPathRuntime
	ReduceChildAllVisible                         ReduceChildPathRuntime
	ReduceChildScratchGeneral                     ReduceChildPathRuntime
	ReduceChildScratchNoAlias                     ReduceChildPathRuntime
	TransientChildSlicesAllocated                 uint64
	TransientChildPointersAllocated               uint64
	TransientChildSlicesMaterialized              uint64
	TransientChildPointersMaterialized            uint64
	TransientParentNodesAllocated                 uint64
	TransientParentNodesMaterialized              uint64
	FinalNodes                                    uint64
	FinalParentNodes                              uint64
	FinalLeafNodes                                uint64
	FinalFieldedParentNodes                       uint64
	FinalUnfieldedParentNodes                     uint64
	FinalVisibleParentNodes                       uint64
	FinalHiddenParentNodes                        uint64
	FinalCheckpointLeafNodes                      uint64
	FinalChildSlices                              uint64
	FinalChildPointers                            uint64
	FinalFieldIDElements                          uint64
	FinalFieldSourceElements                      uint64
	FinalChildRefParents                          uint64
	FinalChildRefs                                uint64
	FinalChildRefMaterializedParents              uint64
	FinalChildRefMaterializedChildren             uint64
	FinalChildRefSingleChildAccesses              uint64
	FinalChildRefSingleChildMaterializedChildren  uint64
	MergeStacksIn                                 uint64
	MergeStacksOut                                uint64
	MergeSlotsUsed                                uint64
	GlobalCullStacksIn                            uint64
	GlobalCullStacksOut                           uint64
	StackEquivCalls                               uint64
	StackEquivTrue                                uint64
	StackEquivDepthMismatch                       uint64
	StackEquivHashMismatch                        uint64
	StackEquivStateMismatch                       uint64
	StackEquivPayloadMismatch                     uint64
	StackEquivEntryCompares                       uint64
	StackEquivStateMismatchDepthSum               uint64
	StackEquivStateMismatchMaxDepth               uint32
	StackEquivStateMismatchDepthBuckets           [stackEquivMismatchDepthBucketCount]uint64
	StackEquivPayloadMismatchDepthSum             uint64
	StackEquivPayloadMismatchMaxDepth             uint32
	StackEquivPayloadMismatchDepthBuckets         [stackEquivMismatchDepthBucketCount]uint64
	StackEquivPayloadHeaderSigDiff                uint64
	StackEquivPayloadHeaderSigSame                uint64
	StackEquivPayloadShallowSigDiff               uint64
	StackEquivPayloadShallowSigSame               uint64
	StackEquivPairKeyed                           uint64
	StackEquivPairUnkeyed                         uint64
	StackEquivPairRepeats                         uint64
	StackEquivPairRepeatTrue                      uint64
	StackEquivPairRepeatFalse                     uint64
	StackEquivPairRepeatMismatch                  uint64
	StackEquivPairStores                          uint64
	MergeHeaderEqTotal                            uint64
	MergeDeepTrue                                 uint64
	MergeDeepFalse                                uint64
	MergeHeaderDeepDivergent                      uint64
	EquivCacheLookups                             uint64
	EquivCacheHits                                uint64
	EquivCacheStores                              uint64
	EquivCacheMisses                              uint64
	EquivCacheTrueHits                            uint64
	EquivCacheFalseHits                           uint64
	EquivCacheEpochMisses                         uint64
	EquivCacheKeyMisses                           uint64
	EquivCacheVersionMisses                       uint64
	EquivSkipError                                uint64
	EquivSkipLeaf                                 uint64
	EquivSkipFieldMismatch                        uint64
	EquivExactCalls                               uint64
	EquivExactTrue                                uint64
	EquivExactPointerTrue                         uint64
	EquivExactNilMismatch                         uint64
	EquivExactHeaderMismatch                      uint64
	EquivExactChildMismatch                       uint64
	EquivExactTerminalCalls                       uint64
	EquivExactTerminalTrue                        uint64
	EquivExactTerminalFalse                       uint64
	EquivFrontierCalls                            uint64
	EquivFrontierTrue                             uint64
	EquivExactChildCompares                       uint64
	EquivFrontierChildScans                       uint64
	EquivFrontierCandidateCompares                uint64
	EquivStateStats                               []ParseEquivStateRuntime
	ParseWallNanos                                int64
	ParserLoopNanos                               int64
	TokenNextNanos                                int64
	ActionDispatchNanos                           int64
	ActionLookupNanos                             int64
	GLRMergeNanos                                 int64
	GLRCullNanos                                  int64
	ReduceTiming                                  *ParseReduceTiming
	ActionTiming                                  *ParseActionTiming

	ExternalScannerCheckpointRecords                 uint64
	ExternalScannerCheckpointSlotsAllocated          uint64
	ExternalScannerCheckpointBytesAllocated          int64
	ExternalScannerSnapshotBytesAllocated            uint64
	ExternalScannerCheckpointLeafNodes               uint64
	CompactFullLeafCreated                           uint64
	CompactFullLeafMaterialized                      uint64
	CompactFullLeafMaterializedForParentReduce       uint64
	CompactFullLeafMaterializedForParentReject       PendingParentRejectStats
	CompactFullLeafMaterializedForFinalTree          uint64
	CompactFullLeafMaterializedForNormalization      uint64
	CompactFullLeafMaterializedForRecovery           uint64
	CompactFullLeafMaterializedForQuery              uint64
	CompactFullLeafMaterializedForCursor             uint64
	CompactFullLeafMaterializedForParentAPI          uint64
	CompactFullLeafMaterializedForEdit               uint64
	CompactFullLeafMaterializedForCheckpointRebuild  uint64
	CompactFullLeafDropped                           uint64
	CompactFullLeafMaterializedForFieldRejectPayload PendingParentFieldRejectPayloadStats
	PendingParentCreated                             uint64
	PendingParentMaterialized                        uint64
	PendingParentMaterializedForParentReduce         uint64
	PendingParentMaterializedForParentReject         PendingParentRejectStats
	PendingParentMaterializedForFieldReject          PendingParentFieldRejectStats
	PendingParentMaterializedForFieldRejectPayload   PendingParentFieldRejectPayloadStats
	PendingParentMaterializedForFinalTree            uint64
	PendingParentMaterializedForNormalization        uint64
	PendingParentMaterializedForRecovery             uint64
	PendingParentMaterializedForQuery                uint64
	PendingParentMaterializedForCursor               uint64
	PendingParentMaterializedForParentAPI            uint64
	PendingParentMaterializedForEdit                 uint64
	PendingParentMaterializedForCheckpointRebuild    uint64
	PendingParentDropped                             uint64
	PendingParentsFlattened                          uint64
	PendingChildRefsFlattened                        uint64
	PendingChildEntriesAllocated                     uint64
	PendingChildEntryCapacity                        uint64
	PendingChildEntryWaste                           uint64
	PendingParentCandidates                          uint64
	PendingParentRejectedEmpty                       uint64
	PendingParentRejectedChildLimit                  uint64
	PendingParentRejectedAlias                       uint64
	PendingParentRejectedRawSpan                     uint64
	PendingParentRejectedFields                      uint64
	PendingParentRejectedFieldsParentHidden          uint64
	PendingParentRejectedFieldsNoIDs                 uint64
	PendingParentRejectedFieldsInherited             uint64
	PendingParentRejectedFieldsHiddenChild           uint64
	PendingParentRejectedFieldsHiddenChildPlain      uint64
	PendingParentRejectedFieldsHiddenChildPlainEmpty uint64
	PendingParentRejectedFieldsHiddenChildPlainOne   uint64
	PendingParentRejectedFieldsHiddenChildPlainMany  uint64
	PendingParentRejectedFieldsHiddenChildWithFields uint64
	PendingParentRejectedFieldsChild                 uint64
	PendingParentRejectedFieldsAllVisibleDirect      uint64
	PendingParentRejectedChild                       uint64
	PendingParentRejectedSpan                        uint64
	PendingParentRejectedFill                        uint64
	PreMaterializationFieldRejectCandidates          uint64
	PreMaterializationFieldRejectSameKeyCandidates   uint64
	PreMaterializationFieldRejectOverflowCandidates  uint64

	CheckpointLeafFullNodesAvoided      uint64
	LeafNodesConstructed                uint64
	ParentNodesConstructed              uint64
	NoTreeReduceNodesConstructed        uint64
	NoTreeLeafNodesConstructed          uint64
	ResultSelectionNanos                int64
	TransientParentMaterializationNanos int64
	ResultTreeBuildNanos                int64
	TransientChildMaterializationNanos  int64
	ResultPythonKeywordRepairNanos      int64
	ResultPythonRootRepairNanos         int64
	ResultFinalizeRootNanos             int64
	ResultExtendTrailingNanos           int64
	ResultNormalizeRootStartNanos       int64
	ResultCompatibilityNanos            int64
	ResultParentLinkNanos               int64
	NormalizationPassesChecked          uint64
	NormalizationPassesRun              uint64
	NormalizationNodesVisited           uint64
	NormalizationNodesRewritten         uint64
	NormalizationNanos                  int64
	NormalizationPasses                 *[]NormalizationPassRuntime
	// The parser sets NativeRecoveredStructureAuthoritative when an exact
	// grammar profile certifies the recovered tree before compatibility.
	NativeRecoveredStructureAuthoritative bool
}

ParseRuntime captures parser-loop diagnostics for a completed tree.

func (ParseRuntime) Summary added in v0.6.0 

func (rt ParseRuntime) Summary() string

Summary returns a stable one-line diagnostic string for parse-runtime stats.

type ParseStopReason added in v0.6.0 

type ParseStopReason string

ParseStopReason reports why parseInternal terminated. 

const (
	ParseStopNone               ParseStopReason = "none"
	ParseStopAccepted           ParseStopReason = "accepted"
	ParseStopNoStacksAlive      ParseStopReason = "no_stacks_alive"
	ParseStopTokenSourceEOF     ParseStopReason = "token_source_eof"
	ParseStopTimeout            ParseStopReason = "timeout"
	ParseStopCancelled          ParseStopReason = "cancelled"
	ParseStopIterationLimit     ParseStopReason = "iteration_limit"
	ParseStopStackDepthLimit    ParseStopReason = "stack_depth_limit"
	ParseStopNodeLimit          ParseStopReason = "node_limit"
	ParseStopMemoryBudget       ParseStopReason = "memory_budget"
	ParseStopInvariantViolation ParseStopReason = "invariant_violation"
)

type ParseStoppedEarlyError added in v0.20.6 

type ParseStoppedEarlyError struct {
	Reason  ParseStopReason
	Runtime ParseRuntime
}

ParseStoppedEarlyError reports a parse that returned a tree but stopped before accepting the input. The returned tree is still available to callers that want diagnostics or partial output.

func (*ParseStoppedEarlyError) Error added in v0.20.6 

func (e *ParseStoppedEarlyError) Error() string

func (*ParseStoppedEarlyError) Is added in v0.20.6 

func (e *ParseStoppedEarlyError) Is(target error) bool

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser reads parse tables from a Language and produces a syntax tree. It supports GLR parsing: when a (state, symbol) pair maps to multiple actions, the parser forks the stack and explores all alternatives in parallel while preserving distinct parse paths. Duplicate stack versions are collapsed and ambiguities are resolved at selection time.

Parser is not safe for concurrent use. Use one parser per goroutine, a ParserPool, or guard shared parser instances with external synchronization.

func NewParser 

func NewParser(lang *Language) *Parser

NewParser creates a new Parser for the given language.

func (*Parser) CancellationFlag added in v0.7.0 

func (p *Parser) CancellationFlag() *uint32

CancellationFlag returns the parser's current cancellation flag pointer.

func (*Parser) ClearAdmissionCandidateRoute added in v0.46.0 

func (p *Parser) ClearAdmissionCandidateRoute()

ClearAdmissionCandidateRoute drops the per-Parser override so the Parser follows the process-wide default again.

func (*Parser) DebugCNodeMemoCacheStats added in v0.44.0 

func (p *Parser) DebugCNodeMemoCacheStats() (cacheLen int, thrash uint32)

DebugCNodeMemoCacheStats reports the current size of the parser's C-recovery per-subtree memo cache (0 while unprovisioned/gate off, cNodeMemoCacheInitialSize while small, cNodeMemoCacheSize once grown by either growCNodeMemoCache trigger) and the live cNodeMemoThrash contention counter. It exists so callers (tests, latency tooling) can observe the adaptive-growth decision directly -- deterministically, from a collision count -- instead of inferring it from wall-clock timing. Safe on a nil receiver.

func (*Parser) ForestDeclineInfo added in v0.20.6 

func (p *Parser) ForestDeclineInfo() (offset uint32, sym Symbol, reason string, states []StateID)

ForestDeclineInfo returns where/why the forest fast path last declined: the byte offset and lookahead symbol at the decline, a short reason code, and (for reason "dead_end") the surviving GLR states. The normal Parse path may then fall back to production; ParseForestExperimental does not. This drives language-burndown triage without re-instrumenting. Valid after a ParseForestExperimental that returned ok=false.

func (*Parser) IncludedRanges added in v0.6.0 

func (p *Parser) IncludedRanges() []Range

IncludedRanges returns a copy of the configured include ranges.

func (*Parser) InferredRootSymbol added in v0.9.0 

func (p *Parser) InferredRootSymbol() (Symbol, bool)

InferredRootSymbol returns the root symbol inferred during parser construction, and whether inference succeeded.

func (*Parser) Language added in v0.7.0 

func (p *Parser) Language() *Language

Language returns the parser's configured language.

func (*Parser) Logger added in v0.7.0 

func (p *Parser) Logger() ParserLogger

Logger returns the currently configured parser debug logger.

func (*Parser) Parse 

func (p *Parser) Parse(source []byte) (*Tree, error)

Parse tokenizes and parses source using the built-in DFA lexer, returning a syntax tree. This works for hand-built grammars that provide LexStates. For real grammars that need a custom lexer, use ParseWithTokenSource. If the input is empty, it returns a tree with a nil root and no error.

func (*Parser) ParseForestExperimental added in v0.20.0 

func (p *Parser) ParseForestExperimental(source []byte) (*Tree, bool)

ParseForestExperimental parses source with the experimental GSS-forest GLR path and returns a releasable forest-produced tree. It returns nil,false when the forest declines for any reason; unlike Parse, this diagnostic entry point never hides a decline by running the production parser. Exported so out-of-tree benchmarks and validation in packages that attach external scanners (e.g. grammars) can drive it; not part of the stable API.

func (*Parser) ParseIncremental 

func (p *Parser) ParseIncremental(source []byte, oldTree *Tree) (*Tree, error)

ParseIncremental re-parses source after edits were applied to oldTree. It reuses unchanged subtrees from the old tree for better performance. Call oldTree.Edit() for each edit before calling this method.

func (*Parser) ParseIncrementalProfiled added in v0.6.0 

func (p *Parser) ParseIncrementalProfiled(source []byte, oldTree *Tree) (*Tree, IncrementalParseProfile, error)

ParseIncrementalProfiled is like ParseIncremental and also returns runtime attribution for incremental reuse work vs parse/rebuild work.

func (*Parser) ParseIncrementalStrict added in v0.20.6 

func (p *Parser) ParseIncrementalStrict(source []byte, oldTree *Tree) (*Tree, error)

ParseIncrementalStrict is like ParseIncremental, but returns ErrParseStoppedEarly when parsing returns a partial tree.

func (*Parser) ParseIncrementalUTF16 added in v0.16.0 

func (p *Parser) ParseIncrementalUTF16(source []uint16, oldTree *Tree) (*Tree, error)

ParseIncrementalUTF16 re-parses UTF-16 source after edits were applied to oldTree. oldTree should have been produced by ParseUTF16, and UTF-16 edits can be recorded with Tree.EditUTF16.

func (*Parser) ParseIncrementalUTF16Bytes added in v0.16.0 

func (p *Parser) ParseIncrementalUTF16Bytes(source []byte, oldTree *Tree, order UTF16ByteOrder) (*Tree, error)

ParseIncrementalUTF16Bytes re-parses UTF-16 bytes after edits were applied to oldTree.

func (*Parser) ParseIncrementalUTF16BytesWithTokenSourceFactory added in v0.16.0 

func (p *Parser) ParseIncrementalUTF16BytesWithTokenSourceFactory(source []byte, oldTree *Tree, order UTF16ByteOrder, factory TokenSourceFactory) (*Tree, error)

ParseIncrementalUTF16BytesWithTokenSourceFactory re-parses UTF-16 bytes using a token source built from the parser's canonical UTF-8 source view.

func (*Parser) ParseIncrementalUTF16WithTokenSourceFactory added in v0.16.0 

func (p *Parser) ParseIncrementalUTF16WithTokenSourceFactory(source []uint16, oldTree *Tree, factory TokenSourceFactory) (*Tree, error)

ParseIncrementalUTF16WithTokenSourceFactory re-parses UTF-16 source using a token source built from the parser's canonical UTF-8 source view.

func (*Parser) ParseIncrementalWithTokenSource 

func (p *Parser) ParseIncrementalWithTokenSource(source []byte, oldTree *Tree, ts TokenSource) (*Tree, error)

ParseIncrementalWithTokenSource is like ParseIncremental but uses a custom token source.

func (*Parser) ParseIncrementalWithTokenSourceFactory added in v0.16.0 

func (p *Parser) ParseIncrementalWithTokenSourceFactory(source []byte, oldTree *Tree, factory TokenSourceFactory) (*Tree, error)

ParseIncrementalWithTokenSourceFactory is like ParseWithTokenSourceFactory for an edited old tree.

func (*Parser) ParseIncrementalWithTokenSourceFactoryStrict added in v0.20.6 

func (p *Parser) ParseIncrementalWithTokenSourceFactoryStrict(source []byte, oldTree *Tree, factory TokenSourceFactory) (*Tree, error)

ParseIncrementalWithTokenSourceFactoryStrict is like ParseIncrementalWithTokenSourceFactory, but returns ErrParseStoppedEarly when parsing returns a partial tree.

func (*Parser) ParseIncrementalWithTokenSourceProfiled added in v0.6.0 

func (p *Parser) ParseIncrementalWithTokenSourceProfiled(source []byte, oldTree *Tree, ts TokenSource) (*Tree, IncrementalParseProfile, error)

ParseIncrementalWithTokenSourceProfiled is like ParseIncrementalWithTokenSource and also returns runtime attribution for incremental reuse work vs parse/rebuild work.

func (*Parser) ParseIncrementalWithTokenSourceStrict added in v0.20.6 

func (p *Parser) ParseIncrementalWithTokenSourceStrict(source []byte, oldTree *Tree, ts TokenSource) (*Tree, error)

ParseIncrementalWithTokenSourceStrict is like ParseIncrementalWithTokenSource, but returns ErrParseStoppedEarly when parsing returns a partial tree.

func (*Parser) ParseNoResultCompatibilityBenchmarkOnly added in v0.18.0 

func (p *Parser) ParseNoResultCompatibilityBenchmarkOnly(source []byte) (*Tree, error)

ParseNoResultCompatibilityBenchmarkOnly parses source while suppressing language-specific result compatibility rewrites while preserving result-tree materialization. Other diagnostic materialization strategies may still key off this mode, so it is not a pure compatibility-only A/B. It is intended only for performance attribution; the returned tree is not API-compatible.

func (*Parser) ParseNoTreeBenchmarkOnly added in v0.17.0 

func (p *Parser) ParseNoTreeBenchmarkOnly(source []byte) (*Tree, error)

ParseNoTreeBenchmarkOnly parses source while suppressing parent/child tree materialization in reduce actions. It is intended only for parser-loop performance experiments; the returned tree is not API-compatible.

func (*Parser) ParseNoTreeWithExternalCheckpointsBenchmarkOnly added in v0.18.0 

func (p *Parser) ParseNoTreeWithExternalCheckpointsBenchmarkOnly(source []byte) (*Tree, error)

ParseNoTreeWithExternalCheckpointsBenchmarkOnly parses source while suppressing parent/child tree materialization in reduce actions but keeping external-scanner checkpoint capture enabled. It is intended only for parser performance attribution; the returned tree is not API-compatible.

func (*Parser) ParseStrict added in v0.20.6 

func (p *Parser) ParseStrict(source []byte) (*Tree, error)

ParseStrict is like Parse, but returns ErrParseStoppedEarly when parsing returns a partial tree due to timeout, cancellation, token-source EOF, or a parser safety limit. The partial tree is returned alongside the error.

func (*Parser) ParseUTF16 added in v0.16.0 

func (p *Parser) ParseUTF16(source []uint16) (*Tree, error)

ParseUTF16 parses UTF-16 source represented as Go UTF-16 code units.

The parser core uses a canonical UTF-8 view internally so existing byte-based APIs remain unchanged. The returned tree retains the original UTF-16 source and can convert node ranges back to UTF-16 code-unit coordinates.

func (*Parser) ParseUTF16Bytes added in v0.16.0 

func (p *Parser) ParseUTF16Bytes(source []byte, order UTF16ByteOrder) (*Tree, error)

ParseUTF16Bytes parses UTF-16 source encoded as bytes with an explicit byte order.

func (*Parser) ParseUTF16BytesWithTokenSourceFactory added in v0.16.0 

func (p *Parser) ParseUTF16BytesWithTokenSourceFactory(source []byte, order UTF16ByteOrder, factory TokenSourceFactory) (*Tree, error)

ParseUTF16BytesWithTokenSourceFactory parses UTF-16 bytes using a token source built from the parser's canonical UTF-8 source view.

func (*Parser) ParseUTF16WithTokenSourceFactory added in v0.16.0 

func (p *Parser) ParseUTF16WithTokenSourceFactory(source []uint16, factory TokenSourceFactory) (*Tree, error)

ParseUTF16WithTokenSourceFactory parses UTF-16 source using a token source built from the parser's canonical UTF-8 source view.

func (*Parser) ParseWith added in v0.6.0 

func (p *Parser) ParseWith(source []byte, opts ...ParseOption) (ParseResult, error)

ParseWith parses source using option-based configuration.

func (*Parser) ParseWithStrict added in v0.20.6 

func (p *Parser) ParseWithStrict(source []byte, opts ...ParseOption) (ParseResult, error)

ParseWithStrict is like ParseWith, but returns ErrParseStoppedEarly when parsing returns a partial tree. The ParseResult still carries that tree.

func (*Parser) ParseWithTokenSource 

func (p *Parser) ParseWithTokenSource(source []byte, ts TokenSource) (*Tree, error)

ParseWithTokenSource parses source using a custom token source. This is used for real grammars where the lexer DFA isn't available as data tables (e.g., Go grammar using go/scanner as a bridge).

func (*Parser) ParseWithTokenSourceFactory added in v0.16.0 

func (p *Parser) ParseWithTokenSourceFactory(source []byte, factory TokenSourceFactory) (*Tree, error)

ParseWithTokenSourceFactory parses source using a freshly built custom token source. The factory is also retained for recovery reparses.

func (*Parser) ParseWithTokenSourceFactoryStrict added in v0.20.6 

func (p *Parser) ParseWithTokenSourceFactoryStrict(source []byte, factory TokenSourceFactory) (*Tree, error)

ParseWithTokenSourceFactoryStrict is like ParseWithTokenSourceFactory, but returns ErrParseStoppedEarly when parsing returns a partial tree.

func (*Parser) ParseWithTokenSourceStrict added in v0.20.6 

func (p *Parser) ParseWithTokenSourceStrict(source []byte, ts TokenSource) (*Tree, error)

ParseWithTokenSourceStrict is like ParseWithTokenSource, but returns ErrParseStoppedEarly when parsing returns a partial tree.

func (*Parser) SetAdmissionCandidateRoute added in v0.46.0 

func (p *Parser) SetAdmissionCandidateRoute(enabled bool)

SetAdmissionCandidateRoute sets a per-Parser override that takes precedence over the process-wide default. enabled=true forces the candidate route on for eligible full parses; enabled=false forces the production route.

enabled=true still respects every per-input eligibility decline: the switch declines inputs at or above the source-length floor where the production route arms the automatic memory budget (parseRuntimeMemoryMinSourceBytes), so large inputs stay on production and honor ParseStopMemoryBudget. Explicit timeouts, cancellation flags, included ranges, and observability hooks also keep a parse on production.

func (*Parser) SetAmbiguityProfile added in v0.17.0 

func (p *Parser) SetAmbiguityProfile(profile *AmbiguityProfile)

SetAmbiguityProfile installs an optional diagnostic ambiguity profile. The profile receives parser state/lookahead/action counters for GLR-heavy benchmark runs. Pass nil to disable profiling.

func (*Parser) SetCancellationFlag added in v0.7.0 

func (p *Parser) SetCancellationFlag(flag *uint32)

SetCancellationFlag configures a caller-owned cancellation flag. Parsing stops when the pointed value becomes non-zero.

func (*Parser) SetGLRTrace added in v0.7.0 

func (p *Parser) SetGLRTrace(enabled bool)

SetGLRTrace enables verbose GLR stack tracing to stdout (debug only).

func (*Parser) SetIncludedRanges added in v0.6.0 

func (p *Parser) SetIncludedRanges(ranges []Range)

SetIncludedRanges configures parser include ranges. Tokens outside these ranges are skipped.

func (*Parser) SetIncludedUTF16ByteRanges added in v0.16.0 

func (p *Parser) SetIncludedUTF16ByteRanges(source []byte, order UTF16ByteOrder, ranges []UTF16Range) error

SetIncludedUTF16ByteRanges configures parser include ranges from endian-specific UTF-16 bytes.

func (*Parser) SetIncludedUTF16Ranges added in v0.16.0 

func (p *Parser) SetIncludedUTF16Ranges(source []uint16, ranges []UTF16Range) bool

SetIncludedUTF16Ranges configures parser include ranges from UTF-16 code-unit ranges. Internal parser points are derived from source as UTF-8 columns.

func (*Parser) SetLogger added in v0.7.0 

func (p *Parser) SetLogger(logger ParserLogger)

SetLogger installs a parser debug logger. Pass nil to disable logging.

func (*Parser) SetTimeoutMicros added in v0.7.0 

func (p *Parser) SetTimeoutMicros(timeoutMicros uint64)

SetTimeoutMicros configures a per-parse timeout in microseconds. A value of 0 disables timeout checks. Parse methods preserve tree-sitter's partial-tree behavior on timeout: they return a tree and nil error, with tree.ParseStopReason() == ParseStopTimeout and tree.ParseStoppedEarly() true. Use ParseStrict or another strict parse method to treat early stops as errors.

func (*Parser) TimeoutMicros added in v0.7.0 

func (p *Parser) TimeoutMicros() uint64

TimeoutMicros returns the parser timeout in microseconds.

type ParserLogType added in v0.7.0 

type ParserLogType uint8

ParserLogType categorizes parser log messages. 

const (
	// ParserLogParse emits parser-loop lifecycle and control-flow logs.
	ParserLogParse ParserLogType = iota
	// ParserLogLex emits token-source and token-consumption logs.
	ParserLogLex
)

type ParserLogger added in v0.7.0 

type ParserLogger func(kind ParserLogType, message string)

ParserLogger receives parser debug logs when configured via SetLogger.

type ParserPool added in v0.7.0 

type ParserPool struct {
	// contains filtered or unexported fields
}

ParserPool provides concurrency-safe parsing by reusing Parser instances.

ParserPool is safe for concurrent use. Each call checks out one parser from an internal sync.Pool, applies configured defaults, runs the parse, and returns the parser to the pool.

Mutable parser state (logger, timeout, cancellation flag, included ranges, GLR trace) is reset on checkout so request-local state cannot bleed across callers.

func NewParserPool added in v0.7.0 

func NewParserPool(lang *Language, opts ...ParserPoolOption) *ParserPool

NewParserPool creates a concurrency-safe parser pool for lang.

func (*ParserPool) Language added in v0.7.0 

func (pp *ParserPool) Language() *Language

Language returns the pool's configured language.

func (*ParserPool) Parse added in v0.7.0 

func (pp *ParserPool) Parse(source []byte) (*Tree, error)

Parse delegates to a pooled Parser.Parse call.

func (*ParserPool) ParseIncrementalUTF16 added in v0.16.0 

func (pp *ParserPool) ParseIncrementalUTF16(source []uint16, oldTree *Tree) (*Tree, error)

ParseIncrementalUTF16 delegates to a pooled Parser.ParseIncrementalUTF16 call.

func (*ParserPool) ParseIncrementalUTF16Bytes added in v0.16.0 

func (pp *ParserPool) ParseIncrementalUTF16Bytes(source []byte, oldTree *Tree, order UTF16ByteOrder) (*Tree, error)

ParseIncrementalUTF16Bytes delegates to a pooled Parser.ParseIncrementalUTF16Bytes call.

func (*ParserPool) ParseIncrementalUTF16BytesWithTokenSourceFactory added in v0.16.0 

func (pp *ParserPool) ParseIncrementalUTF16BytesWithTokenSourceFactory(source []byte, oldTree *Tree, order UTF16ByteOrder, factory TokenSourceFactory) (*Tree, error)

ParseIncrementalUTF16BytesWithTokenSourceFactory delegates to a pooled Parser.ParseIncrementalUTF16BytesWithTokenSourceFactory call.

func (*ParserPool) ParseIncrementalUTF16WithTokenSourceFactory added in v0.16.0 

func (pp *ParserPool) ParseIncrementalUTF16WithTokenSourceFactory(source []uint16, oldTree *Tree, factory TokenSourceFactory) (*Tree, error)

ParseIncrementalUTF16WithTokenSourceFactory delegates to a pooled Parser.ParseIncrementalUTF16WithTokenSourceFactory call.

func (*ParserPool) ParseNoResultCompatibilityBenchmarkOnly added in v0.18.0 

func (pp *ParserPool) ParseNoResultCompatibilityBenchmarkOnly(source []byte) (*Tree, error)

ParseNoResultCompatibilityBenchmarkOnly delegates to Parser.ParseNoResultCompatibilityBenchmarkOnly. It is intended only for performance attribution. The result tree is materialized, but other diagnostic materialization strategies may still key off this mode, and the returned tree is not API-compatible.

func (*ParserPool) ParseNoTreeBenchmarkOnly added in v0.17.0 

func (pp *ParserPool) ParseNoTreeBenchmarkOnly(source []byte) (*Tree, error)

ParseNoTreeBenchmarkOnly delegates to Parser.ParseNoTreeBenchmarkOnly. It is intended only for parser-loop performance experiments; the returned tree is not API-compatible.

func (*ParserPool) ParseNoTreeWithExternalCheckpointsBenchmarkOnly added in v0.18.0 

func (pp *ParserPool) ParseNoTreeWithExternalCheckpointsBenchmarkOnly(source []byte) (*Tree, error)

ParseNoTreeWithExternalCheckpointsBenchmarkOnly delegates to Parser.ParseNoTreeWithExternalCheckpointsBenchmarkOnly. It is intended only for parser performance attribution; the returned tree is not API-compatible.

func (*ParserPool) ParseStrict added in v0.20.6 

func (pp *ParserPool) ParseStrict(source []byte) (*Tree, error)

ParseStrict delegates to a pooled Parser.ParseStrict call.

func (*ParserPool) ParseUTF16 added in v0.16.0 

func (pp *ParserPool) ParseUTF16(source []uint16) (*Tree, error)

ParseUTF16 delegates to a pooled Parser.ParseUTF16 call.

func (*ParserPool) ParseUTF16Bytes added in v0.16.0 

func (pp *ParserPool) ParseUTF16Bytes(source []byte, order UTF16ByteOrder) (*Tree, error)

ParseUTF16Bytes delegates to a pooled Parser.ParseUTF16Bytes call.

func (*ParserPool) ParseUTF16BytesWithTokenSourceFactory added in v0.16.0 

func (pp *ParserPool) ParseUTF16BytesWithTokenSourceFactory(source []byte, order UTF16ByteOrder, factory TokenSourceFactory) (*Tree, error)

ParseUTF16BytesWithTokenSourceFactory delegates to a pooled Parser.ParseUTF16BytesWithTokenSourceFactory call.

func (*ParserPool) ParseUTF16WithTokenSourceFactory added in v0.16.0 

func (pp *ParserPool) ParseUTF16WithTokenSourceFactory(source []uint16, factory TokenSourceFactory) (*Tree, error)

ParseUTF16WithTokenSourceFactory delegates to a pooled Parser.ParseUTF16WithTokenSourceFactory call.

func (*ParserPool) ParseWith added in v0.7.0 

func (pp *ParserPool) ParseWith(source []byte, opts ...ParseOption) (ParseResult, error)

ParseWith delegates to a pooled Parser.ParseWith call.

func (*ParserPool) ParseWithStrict added in v0.20.6 

func (pp *ParserPool) ParseWithStrict(source []byte, opts ...ParseOption) (ParseResult, error)

ParseWithStrict delegates to a pooled Parser.ParseWithStrict call.

func (*ParserPool) ParseWithTokenSource added in v0.7.0 

func (pp *ParserPool) ParseWithTokenSource(source []byte, ts TokenSource) (*Tree, error)

ParseWithTokenSource delegates to a pooled Parser.ParseWithTokenSource call.

func (*ParserPool) ParseWithTokenSourceFactory added in v0.16.0 

func (pp *ParserPool) ParseWithTokenSourceFactory(source []byte, factory TokenSourceFactory) (*Tree, error)

ParseWithTokenSourceFactory delegates to a pooled Parser.ParseWithTokenSourceFactory call.

func (*ParserPool) ParseWithTokenSourceFactoryStrict added in v0.20.6 

func (pp *ParserPool) ParseWithTokenSourceFactoryStrict(source []byte, factory TokenSourceFactory) (*Tree, error)

ParseWithTokenSourceFactoryStrict delegates to a pooled Parser.ParseWithTokenSourceFactoryStrict call.

func (*ParserPool) ParseWithTokenSourceStrict added in v0.20.6 

func (pp *ParserPool) ParseWithTokenSourceStrict(source []byte, ts TokenSource) (*Tree, error)

ParseWithTokenSourceStrict delegates to a pooled Parser.ParseWithTokenSourceStrict call.

type ParserPoolOption added in v0.7.0 

type ParserPoolOption func(*parserPoolConfig)

ParserPoolOption configures a ParserPool.

func WithParserPoolAmbiguityProfile added in v0.17.0 

func WithParserPoolAmbiguityProfile(profile *AmbiguityProfile) ParserPoolOption

WithParserPoolAmbiguityProfile installs an optional diagnostic ambiguity profile on checked-out parsers.

func WithParserPoolGLRTrace added in v0.7.0 

func WithParserPoolGLRTrace(enabled bool) ParserPoolOption

WithParserPoolGLRTrace toggles GLR trace logs on pooled parser instances.

func WithParserPoolIncludedRanges added in v0.7.0 

func WithParserPoolIncludedRanges(ranges []Range) ParserPoolOption

WithParserPoolIncludedRanges sets default include ranges for pooled parsers.

func WithParserPoolLogger added in v0.7.0 

func WithParserPoolLogger(logger ParserLogger) ParserPoolOption

WithParserPoolLogger sets the logger applied to pooled parser instances.

func WithParserPoolTimeoutMicros added in v0.7.0 

func WithParserPoolTimeoutMicros(timeoutMicros uint64) ParserPoolOption

WithParserPoolTimeoutMicros sets the parse timeout for pooled parsers.

type Pattern 

type Pattern struct {
	// contains filtered or unexported fields
}

Pattern is a single top-level S-expression pattern in a query.

type PendingParentFieldRejectPayloadStats added in v0.19.0 

type PendingParentFieldRejectPayloadStats struct {
	Unknown              uint64
	Visible              uint64
	VisibleFinalLike     uint64
	VisibleNestedPayload uint64
	VisibleCompactLeaf   uint64
	VisibleFieldedDesc   uint64
	HiddenEmpty          uint64
	HiddenOne            uint64
	HiddenMany           uint64
	HiddenWithFields     uint64
}

type PendingParentFieldRejectStats added in v0.19.0 

type PendingParentFieldRejectStats struct {
	Unknown               uint64
	ParentHidden          uint64
	NoIDs                 uint64
	Inherited             uint64
	HiddenChild           uint64
	HiddenChildPlain      uint64
	HiddenChildPlainEmpty uint64
	HiddenChildPlainOne   uint64
	HiddenChildPlainMany  uint64
	HiddenChildWithFields uint64
	Child                 uint64
	AllVisibleDirect      uint64
}

type PendingParentRejectStats added in v0.19.0 

type PendingParentRejectStats struct {
	Unknown    uint64
	Empty      uint64
	ChildLimit uint64
	Alias      uint64
	RawSpan    uint64
	Fields     uint64
	Child      uint64
	Span       uint64
	Fill       uint64
}

type PerfCounters added in v0.6.0 

type PerfCounters struct {
	MergeCalls                           uint64
	MergeDeadPruned                      uint64
	MergePerKeyOverflow                  uint64
	MergeReplacements                    uint64
	StackEquivalentCalls                 uint64
	StackEquivalentTrue                  uint64
	StackEqHashMissSkips                 uint64
	StackCompareCalls                    uint64
	ConflictRR                           uint64
	ConflictRS                           uint64
	ConflictOther                        uint64
	ForkCount                            uint64
	FirstConflictToken                   uint64
	MaxConcurrentStacks                  uint64
	LexBytes                             uint64
	LexTokens                            uint64
	ReuseNodesVisited                    uint64
	ReuseNodesPushed                     uint64
	ReuseNodesPopped                     uint64
	ReuseCandidatesChecked               uint64
	ReuseSuccesses                       uint64
	ReuseLeafSuccesses                   uint64
	ReuseNonLeafChecks                   uint64
	ReuseNonLeafSuccesses                uint64
	ReuseNonLeafBytes                    uint64
	ReuseNonLeafNoGoto                   uint64
	ReuseNonLeafNoGotoTerm               uint64
	ReuseNonLeafNoGotoNt                 uint64
	ReuseNonLeafStateMiss                uint64
	ReuseNonLeafStateZero                uint64
	MergeHashZero                        uint64
	GlobalCapCulls                       uint64
	GlobalCapCullDropped                 uint64
	ReduceChainSteps                     uint64
	ReduceChainMaxLen                    uint64
	ReduceChainBreakMulti                uint64
	ReduceChainBreakShift                uint64
	ReduceChainBreakAccept               uint64
	ReduceChainHintCandidates            uint64
	ReduceChainHintTaken                 uint64
	ReduceChainHintSteps                 uint64
	ReduceChainHintTerminalOK            uint64
	ReduceChainHintTerminalMismatch      uint64
	ReduceChainHintLimit                 uint64
	ReduceChainHintDead                  uint64
	ReduceChainHintUnexpected            uint64
	ParentChildPointers                  uint64
	ReduceChildrenFastGSS                uint64
	ReduceChildrenAllVis                 uint64
	ReduceChildrenScratch                uint64
	ReduceScratchNoAlias                 uint64
	ReduceScratchGeneral                 uint64
	ReduceForkCalls                      uint64
	ReduceForkWindows                    uint64
	ReduceForkMaxWindows                 uint64
	PostReduceMergeAttempts              uint64
	PostReduceMergePrimarySuccesses      uint64
	PostReduceMergePendingSuccesses      uint64
	PostReduceMergeDisabledSkips         uint64
	PostReduceMergeFinalizationRiskSkips uint64
	PendingForkStackAppends              uint64
	PendingForkStacksMaxLen              uint64
	ForestReduceCalls                    uint64
	ForestReduceZero                     uint64
	ForestReduceLinearNoExtras           uint64
	ForestReduceDFS                      uint64
	ForestReduceDFSLinks                 uint64
	ForestReduceDFSMultiLinkSteps        uint64
	ForestReduceDFSExtraLinks            uint64
	ForestReduceDFSVisits                uint64
	ForestReduceDFSPathEntries           uint64
	ForestReduceGotoHits                 uint64
	ForestReduceGotoMisses               uint64
	ForestReduceMaxPathLen               uint64
	ForestReduceMaxChildCount            uint64
	ForestCoalesceCalls                  uint64
	ForestCoalesceNewNodes               uint64
	ForestCoalesceLinkAppends            uint64
	ForestCoalesceDedupHits              uint64
	ForestCoalesceDedupReplacements      uint64
	ForestCoalescePreCapDrops            uint64
	ForestCoalesceCapDrops               uint64
	ForestCoalesceCapReplacements        uint64
	ExtraNodes                           uint64
	ErrorNodes                           uint64
	MergeStacksInHist                    [maxGLRStacks + 2]uint64
	MergeAliveHist                       [maxGLRStacks + 2]uint64
	MergeOutHist                         [maxGLRStacks + 2]uint64
	ForkActionsHist                      [8]uint64
	CloneTreeCalls                       uint64
	CloneTreePublicNodes                 uint64
	CloneTreeFinalRefs                   uint64
	CloneTreeCompactCopies               uint64
	CloneTreeChildRefs                   uint64
	CloneOffsetCalls                     uint64
	CloneOffsetPublicNodes               uint64
	CloneOffsetCopies                    uint64
	CloneOffsetShifted                   uint64
	NodeEditCalls                        uint64
	NodeEditNoopCalls                    uint64
	NodeEditCompactRefs                  uint64
	NodeEditShifted                      uint64
	NodeEditMarked                       uint64
	DenseMutationCalls                   uint64
	DenseMutationDrains                  uint64
	MutationChildRefCOW                  uint64
}

func PerfCountersSnapshot added in v0.6.0 

func PerfCountersSnapshot() PerfCounters

type Point 

type Point struct {
	Row    uint32
	Column uint32
}

Point is a row/column position in source text.

type PointSkippableTokenSource 

type PointSkippableTokenSource interface {
	ByteSkippableTokenSource
	SkipToByteWithPoint(offset uint32, pt Point) Token
}

PointSkippableTokenSource extends ByteSkippableTokenSource with a hint-based skip that avoids recomputing row/column from byte offset. During incremental parsing the reused node already carries its endpoint, so passing it directly eliminates the O(n) offset-to-point scan.

type ProductionSignature added in v0.21.0 

type ProductionSignature struct {
	LHS          Symbol
	ProductionID uint16
	RHS          []Symbol
}

ProductionSignature records the raw RHS symbols for a grammar production. It is generated by grammargen, where the normalized grammar still has RHS structure. Legacy ts2go blobs leave this empty.

type Query 

type Query struct {
	// contains filtered or unexported fields
}

Query holds compiled patterns parsed from a tree-sitter .scm query file. It can be executed against a syntax tree to find matching nodes and return captured names.

Query is safe for concurrent calls to Execute, ExecuteInto, ExecuteNode, and Exec after construction. Each goroutine must use its own QueryCursor and any ExecuteInto destination slice must remain caller-owned. The mutating methods DisableCapture and DisablePattern are NOT safe to call concurrently with execution (or with each other); call them before sharing the Query.

func NewQuery 

func NewQuery(source string, lang *Language) (*Query, error)

NewQuery compiles query source (tree-sitter .scm format) against a language. It returns an error if the query syntax is invalid or references unknown node types or field names.

func (*Query) CaptureCount added in v0.7.0 

func (q *Query) CaptureCount() uint32

CaptureCount returns the number of unique capture names in this query.

func (*Query) CaptureNameForID added in v0.7.0 

func (q *Query) CaptureNameForID(id uint32) (string, bool)

CaptureNameForID returns the capture name for the given capture id.

func (*Query) CaptureNames 

func (q *Query) CaptureNames() []string

CaptureNames returns the list of unique capture names used in the query.

func (*Query) DisableCapture added in v0.7.0 

func (q *Query) DisableCapture(name string)

DisableCapture removes captures with the given name from future query results. Matching behavior is unchanged; only returned captures are filtered.

func (*Query) DisablePattern added in v0.7.0 

func (q *Query) DisablePattern(patternIndex uint32)

DisablePattern disables a pattern by index.

func (*Query) EndByteForPattern added in v0.7.0 

func (q *Query) EndByteForPattern(patternIndex uint32) (uint32, bool)

EndByteForPattern returns the query-source end byte for patternIndex.

func (*Query) Exec 

func (q *Query) Exec(node *Node, lang *Language, source []byte) *QueryCursor

Exec creates a streaming cursor over matches rooted at node.

func (*Query) Execute 

func (q *Query) Execute(tree *Tree) []QueryMatch

Execute runs the query against a syntax tree and returns all matches.

func (*Query) ExecuteInto added in v0.10.2 

func (q *Query) ExecuteInto(tree *Tree, dst []QueryMatch) []QueryMatch

ExecuteInto runs the query against a syntax tree, appending matches into dst and returning the updated slice. Callers can pre-allocate or reuse dst across calls to eliminate the per-call slice allocation from Execute.

Example: 

var buf []QueryMatch
for _, tree := range trees {
    buf = q.ExecuteInto(tree, buf[:0])
    process(buf)
}

func (*Query) ExecuteNode 

func (q *Query) ExecuteNode(node *Node, lang *Language, source []byte) []QueryMatch

ExecuteNode runs the query starting from a specific node.

source is required for text predicates (like #eq? / #match?); pass the originating source bytes for correct predicate evaluation.

func (*Query) IsPatternGuaranteedAtStep added in v0.7.0 

func (q *Query) IsPatternGuaranteedAtStep(patternIndex uint32, stepIndex uint32) bool

IsPatternGuaranteedAtStep reports whether all steps through stepIndex are definite and non-quantified.

func (*Query) IsPatternNonLocal added in v0.7.0 

func (q *Query) IsPatternNonLocal(patternIndex uint32) bool

IsPatternNonLocal reports whether the pattern can begin at multiple roots.

func (*Query) IsPatternRooted added in v0.7.0 

func (q *Query) IsPatternRooted(patternIndex uint32) bool

IsPatternRooted reports whether the pattern has exactly one root step at depth 0. Rooted patterns start matching from a single concrete root.

func (*Query) PatternCount 

func (q *Query) PatternCount() int

PatternCount returns the number of patterns in the query.

func (*Query) PredicatesForPattern added in v0.7.0 

func (q *Query) PredicatesForPattern(patternIndex uint32) ([]QueryPredicate, bool)

PredicatesForPattern returns a copy of predicates attached to patternIndex.

func (*Query) PropertyPredicatesForPattern added in v0.39.0 

func (q *Query) PropertyPredicatesForPattern(patternIndex uint32) ([]QueryPropertyPredicate, bool)

PropertyPredicatesForPattern returns the #is? and #is-not? metadata attached to patternIndex in source order. A valid pattern with no property predicates returns an empty slice and ok=true.

func (*Query) StartByteForPattern added in v0.7.0 

func (q *Query) StartByteForPattern(patternIndex uint32) (uint32, bool)

StartByteForPattern returns the query-source start byte for patternIndex.

func (*Query) StepIsDefinite added in v0.7.0 

func (q *Query) StepIsDefinite(patternIndex uint32, stepIndex uint32) bool

StepIsDefinite reports whether a pattern step matches a definite symbol (i.e. not wildcard).

func (*Query) StringCount added in v0.7.0 

func (q *Query) StringCount() uint32

StringCount returns the number of unique string literals in this query.

func (*Query) StringValueForID added in v0.7.0 

func (q *Query) StringValueForID(id uint32) (string, bool)

StringValueForID returns the string literal for the given string id.

type QueryCapture 

type QueryCapture struct {
	Name string
	Node *Node
	// TextOverride, when non-empty, replaces the node's source text for
	// downstream consumers. It is set by the #strip! directive.
	TextOverride string
}

QueryCapture is a single captured node within a match.

func (QueryCapture) Text added in v0.6.0 

func (c QueryCapture) Text(source []byte) string

Text returns the effective text for this capture. If TextOverride is set (e.g. by the #strip! directive), it is returned. Otherwise the node's source text is returned.

func (QueryCapture) UTF16Range added in v0.16.0 

func (c QueryCapture) UTF16Range(tree *Tree) (UTF16Range, bool)

UTF16Range returns this capture's node range in UTF-16 code-unit coordinates for trees produced by UTF-16 parse APIs.

type QueryCursor 

type QueryCursor struct {
	// contains filtered or unexported fields
}

QueryCursor incrementally walks a node subtree and yields matches one by one. It is the streaming counterpart to Query.Execute and avoids materializing all matches up front. QueryCursor is not safe for concurrent use.

func (*QueryCursor) DidExceedMatchLimit added in v0.7.0 

func (c *QueryCursor) DidExceedMatchLimit() bool

DidExceedMatchLimit reports whether query execution had additional matches beyond the configured match limit.

func (*QueryCursor) NextCapture 

func (c *QueryCursor) NextCapture() (QueryCapture, bool)

NextCapture yields captures in match order by draining NextMatch results. This is a practical first-pass ordering: captures are returned in each match's capture order, then by subsequent matches in DFS match order.

func (*QueryCursor) NextMatch 

func (c *QueryCursor) NextMatch() (QueryMatch, bool)

NextMatch yields the next query match from the cursor.

func (*QueryCursor) SetByteRange added in v0.6.0 

func (c *QueryCursor) SetByteRange(startByte, endByte uint32)

SetByteRange restricts matches to nodes that intersect [startByte, endByte). As in tree-sitter C, endByte == 0 is the unbounded-end sentinel.

func (*QueryCursor) SetMatchLimit added in v0.7.0 

func (c *QueryCursor) SetMatchLimit(limit uint32)

SetMatchLimit sets the maximum number of matches this cursor can return. A limit of 0 means unlimited.

func (*QueryCursor) SetMatchWorkBudget added in v0.40.0 

func (c *QueryCursor) SetMatchWorkBudget(limit int)

SetMatchWorkBudget bounds the number of enumeration steps the matcher may take per (pattern,node) attempt, guarding against pathological O(2^n) queries. A limit of 0 means unlimited. The default is defaultQueryMatchWorkBudget. On exhaustion the cursor returns bounded partial results and DidExceedMatchLimit reports true, mirroring C tree-sitter's over-limit behavior.

func (*QueryCursor) SetMaxStartDepth added in v0.7.0 

func (c *QueryCursor) SetMaxStartDepth(depth uint32)

SetMaxStartDepth limits the depth at which new matches can begin. Depth 0 means only the starting node passed to Exec.

func (*QueryCursor) SetPointRange added in v0.6.0 

func (c *QueryCursor) SetPointRange(startPoint, endPoint Point)

SetPointRange restricts matches to nodes that intersect [startPoint, endPoint). As in tree-sitter C, an all-zero end point is the unbounded-end sentinel.

func (*QueryCursor) SetUTF16Range added in v0.16.0 

func (c *QueryCursor) SetUTF16Range(tree *Tree, startCodeUnit, endCodeUnit uint32) bool

SetUTF16Range restricts matches to nodes that intersect the given UTF-16 code-unit range. tree must have been produced by a UTF-16 parse API.

type QueryMatch 

type QueryMatch struct {
	PatternIndex int
	Captures     []QueryCapture
}

QueryMatch represents a successful pattern match with its captures.

func (QueryMatch) SetValues added in v0.6.0 

func (m QueryMatch) SetValues(q *Query, key string) []string

SetValues returns the values of a #set! directive with the given key for a match's pattern, or nil if not present. This is used by InjectionParser to read injection.language metadata.

type QueryPredicate 

type QueryPredicate struct {
	// contains filtered or unexported fields
}

QueryPredicate is a post-match constraint attached to a pattern. Supported forms:

  • (#eq? @a @b)
  • (#eq? @a "literal")
  • (#not-eq? @a @b)
  • (#not-eq? @a "literal")
  • (#match? @a "regex")
  • (#not-match? @a "regex")
  • (#lua-match? @a "lua-pattern")
  • (#any-of? @a "v1" "v2" ...)
  • (#not-any-of? @a "v1" "v2" ...)
  • (#any-eq? @a "literal"), (#any-eq? @a @b)
  • (#any-not-eq? @a "literal"), (#any-not-eq? @a @b)
  • (#any-match? @a "regex")
  • (#any-not-match? @a "regex")
  • (#has-ancestor? @a type ...)
  • (#not-has-ancestor? @a type ...)
  • (#has-parent? @a type ...)
  • (#not-has-parent? @a type ...)
  • (#is? ...), (#is-not? ...)
  • (#set! key value), (#offset! @cap ...)
  • (#count? @a op value) -- op: >, <, >=, <=, ==, !=
  • (#is-exported? @a)

func (QueryPredicate) PropertyPredicate added in v0.39.0 

func (p QueryPredicate) PropertyPredicate() (property QueryPropertyPredicate, ok bool)

PropertyPredicate exposes p when it represents #is? or #is-not? metadata. Other predicate kinds return ok=false.

type QueryPropertyPredicate added in v0.39.0 

type QueryPropertyPredicate struct {
	Property string
	Capture  string
	Positive bool
}

QueryPropertyPredicate is host-consumable metadata for an inert #is? or #is-not? predicate. Property is the queried property name, Capture is the optional capture name without its leading '@', and Positive distinguishes #is? from #is-not?. Property predicates do not filter query matches.

type QueryStep 

type QueryStep struct {
	// contains filtered or unexported fields
}

QueryStep is one matching instruction within a pattern.

type Range 

type Range struct {
	StartByte  uint32
	EndByte    uint32
	StartPoint Point
	EndPoint   Point
}

Range is a span of source text.

func DiffChangedRanges added in v0.6.0 

func DiffChangedRanges(oldTree, newTree *Tree) []Range

DiffChangedRanges compares two syntax trees and returns the minimal ranges where syntactic structure differs. The old tree should have been edited (via Tree.Edit) to match the new tree's source positions before reparsing.

This is equivalent to C tree-sitter's ts_tree_get_changed_ranges().

func IncludedRangesForUTF16 added in v0.16.0 

func IncludedRangesForUTF16(source []uint16, ranges []UTF16Range) ([]Range, bool)

IncludedRangesForUTF16 converts UTF-16 included ranges into the parser's internal UTF-8 byte ranges. The returned Range points use UTF-8 columns.

func IncludedRangesForUTF16Bytes added in v0.16.0 

func IncludedRangesForUTF16Bytes(source []byte, order UTF16ByteOrder, ranges []UTF16Range) ([]Range, error)

IncludedRangesForUTF16Bytes converts endian-specific UTF-16 byte ranges into the parser's internal UTF-8 byte ranges. The returned Range points use UTF-8 columns.

type ReduceChainHint added in v0.19.0 

type ReduceChainHint struct {
	StartState     StateID
	Lookahead      Symbol
	TerminalStates []StateID
	TerminalAction ReduceChainTerminalAction
	MaxSteps       uint16
}

ReduceChainHint describes a terminal-verified parser hot path for a deterministic reduce chain. The runtime still applies normal reduce semantics and stops before the terminal action; this metadata only lets it avoid repeated generic action dispatch for approved state/lookahead pairs.

type ReduceChainTerminalAction added in v0.19.0 

type ReduceChainTerminalAction uint8

ReduceChainTerminalAction describes the action class expected after a generated reduce-chain hint finishes applying deterministic reductions. 

const (
	ReduceChainTerminalNoAction ReduceChainTerminalAction = iota
	ReduceChainTerminalSingleReduce
	ReduceChainTerminalSingleShift
	ReduceChainTerminalSingleAccept
	ReduceChainTerminalSingleOther
	ReduceChainTerminalMulti
)

type ReduceChildPathRuntime added in v0.18.0 

type ReduceChildPathRuntime struct {
	SlicesAllocated   uint64
	SlicesRetained    uint64
	SlicesDropped     uint64
	PointersAllocated uint64
	PointersRetained  uint64
	PointersDropped   uint64
}

type ResultCompatibilityCapability added in v0.33.0 

type ResultCompatibilityCapability uint64

ResultCompatibilityCapability records result-tree shapes that a language produces natively and therefore does not need the runtime to repair after parsing. Keep capability values append-only: Language blobs encode these numeric bits, and a zero value deliberately preserves legacy behavior. 

const (
	ResultCompatibilityCSharpNativeNotNull                ResultCompatibilityCapability = 1 << 0
	ResultCompatibilityCSharpNativeUnicodeIdentifiers     ResultCompatibilityCapability = 1 << 1
	ResultCompatibilityCSharpNativeScopedLambdaStatements ResultCompatibilityCapability = 1 << 2
	ResultCompatibilityCSharpNativeScopedLambdaBlocks     ResultCompatibilityCapability = 1 << 3
	ResultCompatibilityCSharpNativeQueryExpressions       ResultCompatibilityCapability = 1 << 4
	// ResultCompatibilityNativeCollapsedChildren records the v0.46 exact-profile
	// certification receipt for collapsed-child rows. It admits exact built-ins
	// and true adapted clones; native retention then keys off exact registered
	// parent/raw-child metadata identities. A matching display name or pair-level
	// metadata alone is insufficient. Keep the bit append-only because Language
	// blobs encode capability values.
	ResultCompatibilityNativeCollapsedChildren ResultCompatibilityCapability = 1 << 5
	// ResultCompatibilityNativeRecoveredStructure permits receipt checks for
	// error-bearing native roots. Consumers must also verify the complete source
	// span, raw top-level spans, and the isolated-error shape.
	ResultCompatibilityNativeRecoveredStructure ResultCompatibilityCapability = 1 << 6
)

type Rewriter added in v0.6.0 

type Rewriter struct {
	// contains filtered or unexported fields
}

Rewriter collects source-text edits and applies them atomically. Edits target byte ranges (usually from Node.StartByte/EndByte). Apply returns new source bytes and InputEdit records for incremental reparsing. Rewriter is not safe for concurrent use.

func NewRewriter added in v0.6.0 

func NewRewriter(source []byte) *Rewriter

NewRewriter creates a Rewriter for the given source text.

func (*Rewriter) Apply added in v0.6.0 

func (r *Rewriter) Apply() (newSource []byte, edits []InputEdit, err error)

Apply sorts edits, validates no overlaps, applies them, and returns the new source bytes plus InputEdit records for incremental reparsing. Returns an error if edits overlap, use reversed ranges, or extend beyond the source.

func (*Rewriter) ApplyToTree added in v0.6.0 

func (r *Rewriter) ApplyToTree(tree *Tree) ([]byte, error)

ApplyToTree is a convenience that calls Apply(), then tree.Edit() for each edit, returning the new source ready for ParseIncremental.

func (*Rewriter) Delete added in v0.6.0 

func (r *Rewriter) Delete(node *Node)

Delete removes the source text covered by node.

func (*Rewriter) InsertAfter added in v0.6.0 

func (r *Rewriter) InsertAfter(node *Node, text []byte)

InsertAfter inserts text immediately after node.

func (*Rewriter) InsertBefore added in v0.6.0 

func (r *Rewriter) InsertBefore(node *Node, text []byte)

InsertBefore inserts text immediately before node.

func (*Rewriter) Replace added in v0.6.0 

func (r *Rewriter) Replace(node *Node, newText []byte)

Replace replaces the source text covered by node with newText.

func (*Rewriter) ReplaceRange added in v0.6.0 

func (r *Rewriter) ReplaceRange(startByte, endByte uint32, newText []byte)

ReplaceRange replaces bytes in [startByte, endByte) with newText.

type StateID 

type StateID uint32

StateID is a parser state index. uint32 supports grammars with >65K states (e.g. COBOL with 67K states from 1071 rules).

type StatelessExternalScanner added in v0.45.0 

type StatelessExternalScanner interface {
	ExternalScanner
	ExternalScannerIsStateless() bool
}

StatelessExternalScanner is implemented by external scanners that carry no serialized state across tokens. Their Scan decision is a pure function of the byte stream at the lexer position and the valid-symbol set, and the valid-symbol set is itself a pure function of the LR parser state. For such a scanner the state at any boundary equals the state a fresh parse holds there, so the scanner-quiescence proof obligation (campaign O(edit) workstream W4) is discharged at every boundary. Go's automatic-semicolon scanner is the reference example; see external_scanner_quiescence.go for the proof.

A scanner implements this only when it can meet every quiescence obligation. The classifier reads the marker as a proof, so an incorrect true is silent incremental corruption.

type Symbol 

type Symbol uint16

Symbol is a grammar symbol ID (terminal or nonterminal).

type SymbolMetadata 

type SymbolMetadata struct {
	Name               string
	Visible            bool
	Named              bool
	Supertype          bool
	GeneratedRepeatAux bool
}

SymbolMetadata holds display information about a symbol.

type Tag 

type Tag struct {
	Kind      string // e.g. "definition.function", "reference.call"
	Name      string // the captured symbol text
	Range     Range  // full span of the tagged node
	NameRange Range  // span of the @name capture
}

Tag represents a tagged symbol in source code, extracted by a Tagger. Kind follows tree-sitter convention: "definition.function", "reference.call", etc. Name is the captured symbol text (e.g., the function name).

type Tagger 

type Tagger struct {
	// contains filtered or unexported fields
}

Tagger extracts symbol definitions and references from source code using tree-sitter tags queries. It is the tagging counterpart to Highlighter.

Tags queries use a convention where captures follow the pattern:

  • @name captures the symbol name (e.g., function identifier)
  • @definition.X or @reference.X captures the kind

Example query: 

(function_declaration name: (identifier) @name) @definition.function
(call_expression function: (identifier) @name) @reference.call

func NewTagger 

func NewTagger(lang *Language, tagsQuery string, opts ...TaggerOption) (*Tagger, error)

NewTagger creates a Tagger for the given language and tags query.

func (*Tagger) Tag 

func (tg *Tagger) Tag(source []byte) []Tag

Tag parses source and returns all tags.

func (*Tagger) TagIncremental 

func (tg *Tagger) TagIncremental(source []byte, oldTree *Tree) ([]Tag, *Tree)

TagIncremental re-tags source after edits to oldTree. Returns the tags and the new tree for subsequent incremental calls.

func (*Tagger) TagIncrementalStrict added in v0.37.0 

func (tg *Tagger) TagIncrementalStrict(source []byte, oldTree *Tree) ([]Tag, *Tree, error)

TagIncrementalStrict is like TagIncremental, but reports ErrParseStoppedEarly and skips query execution when parsing is stopped by a timeout, cancellation, token-source EOF, or parser safety limit. The partial tree is returned so callers can release it or use it for diagnostics.

func (*Tagger) TagIncrementalUTF16 added in v0.16.0 

func (tg *Tagger) TagIncrementalUTF16(source []uint16, oldTree *Tree) ([]UTF16Tag, *Tree)

TagIncrementalUTF16 re-tags UTF-16 source after edits to oldTree. Call oldTree.EditUTF16 before calling this.

func (*Tagger) TagIncrementalUTF16Bytes added in v0.16.0 

func (tg *Tagger) TagIncrementalUTF16Bytes(source []byte, oldTree *Tree, order UTF16ByteOrder) ([]UTF16Tag, *Tree, error)

TagIncrementalUTF16Bytes is like TagIncrementalUTF16 for endian-specific UTF-16 bytes.

func (*Tagger) TagTree 

func (tg *Tagger) TagTree(tree *Tree) []Tag

TagTree extracts tags from an already-parsed tree.

func (*Tagger) TagTreeUTF16 added in v0.16.0 

func (tg *Tagger) TagTreeUTF16(tree *Tree) []UTF16Tag

TagTreeUTF16 extracts tags from an already-parsed UTF-16 tree.

func (*Tagger) TagUTF16 added in v0.16.0 

func (tg *Tagger) TagUTF16(source []uint16) []UTF16Tag

TagUTF16 parses UTF-16 source and returns all tags with UTF-16 ranges.

func (*Tagger) TagUTF16Bytes added in v0.16.0 

func (tg *Tagger) TagUTF16Bytes(source []byte, order UTF16ByteOrder) ([]UTF16Tag, error)

TagUTF16Bytes is like TagUTF16 for endian-specific UTF-16 bytes.

type TaggerOption 

type TaggerOption func(*Tagger)

TaggerOption configures a Tagger.

func WithTaggerTimeoutMicros added in v0.37.0 

func WithTaggerTimeoutMicros(timeoutMicros uint64) TaggerOption

WithTaggerTimeoutMicros bounds every full and incremental parse performed by the tagger. A value of zero disables timeout checks.

func WithTaggerTokenSourceFactory 

func WithTaggerTokenSourceFactory(factory func(source []byte) TokenSource) TaggerOption

WithTaggerTokenSourceFactory sets a factory function that creates a TokenSource for each Tag call.

type Token 

type Token struct {
	Symbol     Symbol
	Text       string
	StartByte  uint32
	EndByte    uint32
	StartPoint Point
	EndPoint   Point
	Missing    bool
	// NoLookahead marks a synthetic EOF used to force EOF-table reductions
	// without consuming input, matching tree-sitter's lex_state = -1.
	NoLookahead bool
	// ExternalScannerToken marks tokens produced by an external scanner.
	// ExternalScannerStartByte is the byte offset where that scanner call
	// began, before scanner-side skip advances moved StartByte forward.
	ExternalScannerToken     bool
	ExternalScannerStartByte uint32
}

Token is a lexed token with position info.

type TokenSource 

type TokenSource interface {
	// Next returns the next token. It should skip whitespace and comments
	// as appropriate for the language. Returns a zero-Symbol token at EOF.
	Next() Token
}

TokenSource provides tokens to the parser. This interface abstracts over different lexer implementations: the built-in DFA lexer (for hand-built grammars) or custom bridges like GoTokenSource (for real grammars where we can't extract the C lexer DFA).

type TokenSourceFactory added in v0.16.0 

type TokenSourceFactory func(source []byte) (TokenSource, error)

TokenSourceFactory builds a token source for parser source bytes.

type TokenSourceRebuilder added in v0.7.0 

type TokenSourceRebuilder interface {
	RebuildTokenSource(source []byte, lang *Language) (TokenSource, error)
}

TokenSourceRebuilder is an optional extension for token sources that can build a fresh equivalent token source for another source buffer. Result normalization uses this to reparse isolated fragments with the same lexer backend as the original parse.

type Tree 

type Tree struct {
	// contains filtered or unexported fields
}

Tree holds a complete syntax tree along with its source text and language. Tree is safe for concurrent reads after construction. Edit and Release are not safe for concurrent use.

func NewTree 

func NewTree(root *Node, source []byte, lang *Language) *Tree

NewTree creates a new Tree.

func (*Tree) ArenaBreakdown added in v0.18.0 

func (t *Tree) ArenaBreakdown() (ArenaBreakdown, bool)

ArenaBreakdown returns optional arena/materialization attribution captured when EnableArenaBreakdown(true) was set before parsing.

func (*Tree) ChangedRanges added in v0.6.0 

func (t *Tree) ChangedRanges() []Range

ChangedRanges converts this tree's recorded edits into changed source ranges. Overlapping ranges are coalesced.

func (*Tree) Copy added in v0.7.0 

func (t *Tree) Copy() *Tree

Copy returns an independent copy of this tree.

The copied tree has distinct node objects, so subsequent Tree.Edit calls on either tree do not mutate the other's spans/dirty bits. Source bytes and language pointer are shared (read-only).

func (*Tree) DOT added in v0.7.0 

func (t *Tree) DOT(lang *Language) string

DOT returns a DOT graph representation of this tree.

func (*Tree) DescendantForUTF16Range added in v0.16.0 

func (t *Tree) DescendantForUTF16Range(startCodeUnit, endCodeUnit uint32) *Node

DescendantForUTF16Range returns the smallest descendant that fully contains the given UTF-16 code-unit range, or nil when no such descendant exists.

func (*Tree) Edit 

func (t *Tree) Edit(edit InputEdit)

Edit records an edit on this tree. Call this before ParseIncremental to inform the parser which regions changed. The edit adjusts byte offsets and marks overlapping nodes as dirty so the incremental parser knows what to re-parse.

func (*Tree) EditUTF16 added in v0.16.0 

func (t *Tree) EditUTF16(edit UTF16Edit, newSource []uint16) bool

EditUTF16 records a UTF-16 code-unit edit on a UTF-16 tree.

newSource is the full source after the edit; it is used to derive the internal UTF-8 endpoint for NewEndCodeUnit.

func (*Tree) Edits 

func (t *Tree) Edits() []InputEdit

Edits returns the pending edits recorded on this tree.

func (*Tree) EnclosingDefinition added in v0.20.6 

func (t *Tree) EnclosingDefinition(byteOffset uint32) (DefinitionSpan, bool)

EnclosingDefinition returns the nearest definition node that contains byteOffset.

func (*Tree) InputEditForUTF16 added in v0.16.0 

func (t *Tree) InputEditForUTF16(edit UTF16Edit, newSource []uint16) (InputEdit, bool)

InputEditForUTF16 converts a UTF-16 code-unit edit into the parser's internal UTF-8 byte-coordinate edit. The tree must have been produced by ParseUTF16.

func (*Tree) Language 

func (t *Tree) Language() *Language

Language returns the language used to parse this tree.

func (*Tree) NamedDescendantForUTF16Range added in v0.16.0 

func (t *Tree) NamedDescendantForUTF16Range(startCodeUnit, endCodeUnit uint32) *Node

NamedDescendantForUTF16Range returns the smallest named descendant that fully contains the given UTF-16 code-unit range, or nil when no such descendant exists.

func (*Tree) NamedNodeAtByte added in v0.20.6 

func (t *Tree) NamedNodeAtByte(byteOffset uint32) *Node

NamedNodeAtByte returns the smallest named root descendant that contains byteOffset.

func (*Tree) NodeAtByte added in v0.20.6 

func (t *Tree) NodeAtByte(byteOffset uint32) *Node

NodeAtByte returns the smallest root descendant that contains byteOffset.

func (*Tree) ParseRuntime added in v0.6.0 

func (t *Tree) ParseRuntime() ParseRuntime

ParseRuntime returns parser-loop diagnostics captured when this tree was built.

func (*Tree) ParseStopReason added in v0.6.0 

func (t *Tree) ParseStopReason() ParseStopReason

ParseStopReason reports why parsing terminated.

func (*Tree) ParseStoppedEarly added in v0.6.0 

func (t *Tree) ParseStoppedEarly() bool

ParseStoppedEarly reports whether parsing hit an early-stop condition.

func (*Tree) Release 

func (t *Tree) Release()

Release decrements arena references held by this tree. After Release, the tree should be treated as invalid and not reused.

func (*Tree) RootNode 

func (t *Tree) RootNode() *Node

RootNode returns the tree's root node.

func (*Tree) RootNodeWithOffset added in v0.7.0 

func (t *Tree) RootNodeWithOffset(offsetBytes uint32, offsetExtent Point) *Node

RootNodeWithOffset returns a copy of the root node with all spans shifted by the provided byte and point offsets.

This mirrors tree-sitter C's root-node-with-offset behavior for callers that need to embed a parsed tree at a larger document offset.

func (*Tree) Source 

func (t *Tree) Source() []byte

Source returns the original source text.

func (*Tree) SourceEncoding added in v0.16.0 

func (t *Tree) SourceEncoding() InputEncoding

SourceEncoding returns the encoding used by the caller that produced this tree.

For UTF-16 parses, Source still returns the parser's canonical UTF-8 copy. Use SourceUTF16 and UTF16RangeForNode when caller-facing UTF-16 coordinates are needed.

func (*Tree) SourceUTF16 added in v0.16.0 

func (t *Tree) SourceUTF16() []uint16

SourceUTF16 returns the original UTF-16 source for trees produced by ParseUTF16. It returns nil for ordinary UTF-8 parses.

func (*Tree) UTF8ByteForUTF16Offset added in v0.16.0 

func (t *Tree) UTF8ByteForUTF16Offset(offset uint32) (uint32, bool)

UTF8ByteForUTF16Offset converts a UTF-16 code-unit offset to the parser's canonical UTF-8 byte offset for trees produced by ParseUTF16.

func (*Tree) UTF16OffsetForByte added in v0.16.0 

func (t *Tree) UTF16OffsetForByte(offset uint32) (uint32, bool)

UTF16OffsetForByte converts a parser UTF-8 byte offset to a UTF-16 code-unit offset for trees produced by ParseUTF16.

func (*Tree) UTF16PointForByte added in v0.16.0 

func (t *Tree) UTF16PointForByte(offset uint32) (Point, bool)

UTF16PointForByte converts a parser UTF-8 byte offset to a UTF-16 point.

func (*Tree) UTF16RangeForByteRange added in v0.16.0 

func (t *Tree) UTF16RangeForByteRange(startByte, endByte uint32) (UTF16Range, bool)

UTF16RangeForByteRange converts a canonical UTF-8 byte range into UTF-16 code-unit coordinates.

func (*Tree) UTF16RangeForNode added in v0.16.0 

func (t *Tree) UTF16RangeForNode(n *Node) (UTF16Range, bool)

UTF16RangeForNode returns a node range in UTF-16 code-unit coordinates.

func (*Tree) UTF16RangeForRange added in v0.16.0 

func (t *Tree) UTF16RangeForRange(r Range) (UTF16Range, bool)

UTF16RangeForRange converts a canonical UTF-8 Range into UTF-16 code-unit coordinates.

func (*Tree) UTF16SourceForNode added in v0.16.0 

func (t *Tree) UTF16SourceForNode(n *Node) ([]uint16, bool)

UTF16SourceForNode returns the original UTF-16 code units covered by n.

func (*Tree) WriteDOT added in v0.7.0 

func (t *Tree) WriteDOT(w io.Writer, lang *Language) error

WriteDOT writes a DOT graph representation of this tree to w.

type TreeCursor added in v0.6.0 

type TreeCursor struct {
	// contains filtered or unexported fields
}

TreeCursor provides stateful, O(1) tree navigation. It maintains a stack of (node, childIndex) frames enabling efficient parent, child, and sibling movement without scanning.

The cursor holds pointers to Nodes. If the underlying Tree is released, edited, or replaced via incremental reparse, the cursor should be recreated.

func NewTreeCursor added in v0.6.0 

func NewTreeCursor(node *Node, tree *Tree) *TreeCursor

NewTreeCursor creates a cursor starting at the given node. The optional tree reference enables field name resolution and text extraction.

func NewTreeCursorFromTree added in v0.6.0 

func NewTreeCursorFromTree(tree *Tree) *TreeCursor

NewTreeCursorFromTree creates a cursor starting at the tree's root node.

func (*TreeCursor) Copy added in v0.6.0 

func (c *TreeCursor) Copy() *TreeCursor

Copy returns an independent copy of the cursor. The copy shares the same tree reference but has its own navigation stack.

func (*TreeCursor) CurrentFieldID added in v0.6.0 

func (c *TreeCursor) CurrentFieldID() FieldID

CurrentFieldID returns the field ID of the current node within its parent. Returns 0 if the cursor is at the root or the node has no field assignment.

func (*TreeCursor) CurrentFieldName added in v0.6.0 

func (c *TreeCursor) CurrentFieldName() string

CurrentFieldName returns the field name of the current node within its parent. Returns "" if no tree is associated, the cursor is at the root, or the node has no field assignment.

func (*TreeCursor) CurrentNode added in v0.6.0 

func (c *TreeCursor) CurrentNode() *Node

CurrentNode returns the node the cursor is currently pointing to.

func (*TreeCursor) CurrentNodeIsNamed added in v0.6.0 

func (c *TreeCursor) CurrentNodeIsNamed() bool

CurrentNodeIsNamed returns whether the current node is a named node.

func (*TreeCursor) CurrentNodeText added in v0.6.0 

func (c *TreeCursor) CurrentNodeText() string

CurrentNodeText returns the source text of the current node. Requires a tree with source to be associated.

func (*TreeCursor) CurrentNodeType added in v0.6.0 

func (c *TreeCursor) CurrentNodeType() string

CurrentNodeType returns the type name of the current node. Requires a tree with a language to be associated.

func (*TreeCursor) Depth added in v0.6.0 

func (c *TreeCursor) Depth() int

Depth returns the cursor's current depth (0 at the root).

func (*TreeCursor) GotoChildByFieldID added in v0.6.0 

func (c *TreeCursor) GotoChildByFieldID(fid FieldID) bool

GotoChildByFieldID moves the cursor to the first child with the given field ID. Returns false if no child has that field.

func (*TreeCursor) GotoChildByFieldName added in v0.6.0 

func (c *TreeCursor) GotoChildByFieldName(name string) bool

GotoChildByFieldName moves the cursor to the first child with the given field name. Returns false if the tree has no language, the field name is unknown, or no child has that field.

func (*TreeCursor) GotoFirstChild added in v0.6.0 

func (c *TreeCursor) GotoFirstChild() bool

GotoFirstChild moves the cursor to the first child of the current node. Returns false if the current node has no children.

func (*TreeCursor) GotoFirstChildForByte added in v0.6.0 

func (c *TreeCursor) GotoFirstChildForByte(targetByte uint32) int64

GotoFirstChildForByte moves the cursor to the first child whose byte range contains targetByte (i.e., first child where endByte > targetByte). Returns the child index, or -1 when no child contains the byte.

func (*TreeCursor) GotoFirstChildForPoint added in v0.6.0 

func (c *TreeCursor) GotoFirstChildForPoint(targetPoint Point) int64

GotoFirstChildForPoint moves the cursor to the first child whose point range contains targetPoint (i.e., first child where endPoint > targetPoint). Returns the child index, or -1 when no child contains the point.

func (*TreeCursor) GotoFirstNamedChild added in v0.6.0 

func (c *TreeCursor) GotoFirstNamedChild() bool

GotoFirstNamedChild moves the cursor to the first named child of the current node, skipping anonymous nodes. Returns false if no named child exists.

func (*TreeCursor) GotoLastChild added in v0.6.0 

func (c *TreeCursor) GotoLastChild() bool

GotoLastChild moves the cursor to the last child of the current node. Returns false if the current node has no children.

func (*TreeCursor) GotoLastNamedChild added in v0.6.0 

func (c *TreeCursor) GotoLastNamedChild() bool

GotoLastNamedChild moves the cursor to the last named child of the current node, skipping anonymous nodes. Returns false if no named child exists.

func (*TreeCursor) GotoNextNamedSibling added in v0.6.0 

func (c *TreeCursor) GotoNextNamedSibling() bool

GotoNextNamedSibling moves the cursor to the next named sibling, skipping anonymous nodes. Returns false if no named sibling follows.

func (*TreeCursor) GotoNextSibling added in v0.6.0 

func (c *TreeCursor) GotoNextSibling() bool

GotoNextSibling moves the cursor to the next sibling. Returns false if the cursor is at the root or the last sibling.

func (*TreeCursor) GotoParent added in v0.6.0 

func (c *TreeCursor) GotoParent() bool

GotoParent moves the cursor to the parent of the current node. Returns false if the cursor is at the root.

func (*TreeCursor) GotoPrevNamedSibling added in v0.6.0 

func (c *TreeCursor) GotoPrevNamedSibling() bool

GotoPrevNamedSibling moves the cursor to the previous named sibling, skipping anonymous nodes. Returns false if no named sibling precedes.

func (*TreeCursor) GotoPrevSibling added in v0.6.0 

func (c *TreeCursor) GotoPrevSibling() bool

GotoPrevSibling moves the cursor to the previous sibling. Returns false if the cursor is at the root or the first sibling.

func (*TreeCursor) Reset added in v0.6.0 

func (c *TreeCursor) Reset(node *Node)

Reset resets the cursor to a new root node, clearing the navigation stack.

func (*TreeCursor) ResetTree added in v0.6.0 

func (c *TreeCursor) ResetTree(tree *Tree)

ResetTree resets the cursor to the root of a new tree.

type UTF16ByteOrder added in v0.16.0 

type UTF16ByteOrder uint8

UTF16ByteOrder identifies the byte order used by a UTF-16 byte source. 

const (
	UTF16LittleEndian UTF16ByteOrder = iota
	UTF16BigEndian
)

func (UTF16ByteOrder) String added in v0.16.0 

func (o UTF16ByteOrder) String() string

type UTF16Edit added in v0.16.0 

type UTF16Edit struct {
	StartCodeUnit  uint32
	OldEndCodeUnit uint32
	NewEndCodeUnit uint32
}

UTF16Edit describes a source edit in UTF-16 code-unit offsets.

type UTF16HighlightRange added in v0.16.0 

type UTF16HighlightRange struct {
	StartCodeUnit uint32
	EndCodeUnit   uint32
	StartPoint    Point
	EndPoint      Point
	Capture       string
	PatternIndex  int
}

UTF16HighlightRange is a styled source range in UTF-16 code-unit coordinates.

type UTF16Injection added in v0.16.0 

type UTF16Injection struct {
	// Language is the detected language name (e.g., "javascript").
	Language string
	// Tree is the parse tree for this region, or nil if the language
	// was not registered.
	Tree *Tree
	// Ranges are the source ranges this tree covers in UTF-16 code units.
	Ranges []UTF16Range
	// Node is the parent tree node that triggered the injection.
	Node *Node
}

UTF16Injection is a single embedded language region with ranges in UTF-16 code-unit coordinates.

type UTF16InjectionResult added in v0.16.0 

type UTF16InjectionResult struct {
	// Tree is the parent language's parse tree.
	Tree *Tree
	// Injections contains child language parse results, ordered by position.
	Injections []UTF16Injection
	// contains filtered or unexported fields
}

UTF16InjectionResult holds parse results for a UTF-16 multi-language document. Injection ranges are expressed in UTF-16 code units.

type UTF16Range added in v0.16.0 

type UTF16Range struct {
	StartCodeUnit uint32
	EndCodeUnit   uint32
	StartPoint    Point
	EndPoint      Point
}

UTF16Range is a source range in UTF-16 code units.

StartPoint and EndPoint use UTF-16 code-unit columns, matching the coordinate system used by many editors and LSP clients.

type UTF16Tag added in v0.16.0 

type UTF16Tag struct {
	Kind      string
	Name      string
	Range     UTF16Range
	NameRange UTF16Range
}

UTF16Tag represents a tagged symbol with ranges in UTF-16 code-unit coordinates.

type UnaryWrapperFlatteningRule added in v0.48.0 

type UnaryWrapperFlatteningRule struct {
	PublicParent        Symbol
	Wrapper             Symbol
	Leaf                Symbol
	WrapperPreGotoState StateID
}

UnaryWrapperFlatteningRule identifies one exact public-parent, wrapper, leaf, and parser-state chain that native materialization must flatten. Runtime profiles attach these rules only to certified grammar artifacts.

type WalkAction 

type WalkAction int

WalkAction controls the tree walk behavior. 

const (
	// WalkContinue continues the walk to children and siblings.
	WalkContinue WalkAction = iota
	// WalkSkipChildren skips the current node's children but continues to siblings.
	WalkSkipChildren
	// WalkStop terminates the walk entirely.
	WalkStop
)

Source Files

View all Source files

Directories

Path Synopsis
cgo_harness module
cmd
benchgate command
benchmatrix command
gen_linguist command
Command gen_linguist generates grammars/linguist_gen.go by matching gotreesitter grammar names to GitHub Linguist's languages.yml.
 Command gen_linguist generates grammars/linguist_gen.go by matching gotreesitter grammar names to GitHub Linguist's languages.yml.
gen_subset_blob_embeds command
Command gen_subset_blob_embeds generates the per-language z_subset_blob_embed_<lang>.go files that power embedded grammar_subset builds (issue #88: per-language compile-time grammar selection).
 Command gen_subset_blob_embeds generates the per-language z_subset_blob_embed_<lang>.go files that power embedded grammar_subset builds (issue #88: per-language compile-time grammar selection).
grammar_update_guard command
Command grammar_update_guard checks lock-update reports for scanner-facing changes that require hand-written scanner review before grammar blobs move.
 Command grammar_update_guard checks lock-update reports for scanner-facing changes that require hand-written scanner review before grammar blobs move.
grammar_updater command
Command grammar_updater refreshes pinned grammar commits in grammars/languages.lock and emits a machine-readable update report.
 Command grammar_updater refreshes pinned grammar commits in grammars/languages.lock and emits a machine-readable update report.
grammarblobprobe command
Command grammarblobprobe is a minimal binary that blank-imports the grammars package so that whatever grammar blobs are embedded by the active build tags are linked into the binary.
 Command grammarblobprobe is a minimal binary that blank-imports the grammars package so that whatever grammar blobs are embedded by the active build tags are linked into the binary.
grammargen command
Command grammargen generates tree-sitter parser artifacts from grammar definitions.
 Command grammargen generates tree-sitter parser artifacts from grammar definitions.
harnessgate command
parity_report command
pgo_repdriver command
Command pgo_repdriver is a representative multi-grammar parsing workload used to (a) collect a CPU profile for Go build-time PGO and (b) produce a stable digest of parsed trees for before/after correctness comparisons.
 Command pgo_repdriver is a representative multi-grammar parsing workload used to (a) collect a CPU profile for Go build-time PGO and (b) produce a stable digest of parsed trees for before/after correctness comparisons.
releasegate command
ts2go command
Command ts2go reads a tree-sitter generated parser.c file and outputs a Go source file containing a function that returns a populated *gotreesitter.Language with all extracted parse tables.
 Command ts2go reads a tree-sitter generated parser.c file and outputs a Go source file containing a function that returns a populated *gotreesitter.Language with all extracted parse tables.
tsquery command
Command tsquery generates type-safe Go code from tree-sitter .scm query files.
 Command tsquery generates type-safe Go code from tree-sitter .scm query files.
wasmassets command
Command wasmassets builds the persistent gotreesitter WASM runtime and emits a self-contained browser asset bundle for one registered language.
 Command wasmassets builds the persistent gotreesitter WASM runtime and emits a self-contained browser asset bundle for one registered language.
Package corpuscheck parses the upstream tree-sitter corpus test format.
 Package corpuscheck parses the upstream tree-sitter corpus test format.
cmd/corpuscheck command
Command corpuscheck runs gotreesitter against upstream tree-sitter corpora.
 Command corpuscheck runs gotreesitter against upstream tree-sitter corpora.
Package grammargen implements a pure-Go grammar generator for gotreesitter.
 Package grammargen implements a pure-Go grammar generator for gotreesitter.
grammarjs
Package grammarjs wires grammargen's grammar.js importer to gotreesitter's embedded JavaScript grammar.
 Package grammarjs wires grammargen's grammar.js importer to gotreesitter's embedded JavaScript grammar.
c_sharp precise ExternalLexStates — DEFAULT since the 2026-07 cliff campaign (previously staged behind -tags csharp_precise_els).
 c_sharp precise ExternalLexStates — DEFAULT since the 2026-07 cliff campaign (previously staged behind -tags csharp_precise_els).
Package grep provides structural code search, match, and rewrite using tree-sitter parse trees.
 Package grep provides structural code search, match, and rewrite using tree-sitter parse trees.
internal
benchfixtures
Package benchfixtures provides immutable, content-addressed benchmark inputs.
 Package benchfixtures provides immutable, content-addressed benchmark inputs.
parsercorephase0
Package parsercorephase0 contains a diagnostic-only parser-core prototype.
 Package parsercorephase0 contains a diagnostic-only parser-core prototype.
Package taproot is the common front-end harness shared by M31 DSLs that use the gotreesitter runtime.
 Package taproot is the common front-end harness shared by M31 DSLs that use the gotreesitter runtime.
diag
Package diag provides a generic structured diagnostic type and a source-quoting renderer.
 Package diag provides a generic structured diagnostic type and a source-quoting renderer.
walk
Package walk is the grammar-free core of taproot: load a tree-sitter Language from a pre-generated blob and navigate the CST with a Walker.
 Package walk is the grammar-free core of taproot: load a tree-sitter Language from a pre-generated blob and navigate the CST with a Walker.
wasm
grammargen command
runtime command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.