Help us shape the future of Go! Take the 2026 Go Developer Survey

treesitter

package
v0.0.0-...-ea53456 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 10, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Overview

Package treesitter is the bundled structural provider of Section 11.3: a tree-sitter pass over each supported source file that publishes declarations, containing scopes, signatures, imports and exports, syntax references and call sites, test declarations and attached documentation at precision syntax, and, from the same worker walk, the file-local dependence families control_depends_on, data_flows_to, reads and writes between the file's parameters and locals at precision static_analysis (docs/providers-treesitter.md, Dependence facts). Parsing runs in isolated worker subprocesses (package worker) started through the shared process runner; this package is the parent side, which streams pinned bytes to a worker, validates every framed fact it answers with against those bytes, resolves identities through the unit's resolver and emits facts through the sink. It never links the grammars itself and never holds a repository-wide AST or source cache: the unit of work is one file, and its bytes live only for that unit.

Memory is taken from each file's observed need (ADR-0012 decision 5): a worker holds its reported base on the process's reservation ledger, and each file reserves its predicted increment before it is dispatched -- the learned p99 of need per source byte for its repository, language, grammar fingerprint and size class, or the structural prior for the first file of that key. The need a worker measures covers the parse, the extraction and the dependence lowering and analysis of every callable, which all run in the worker's one walk over the file, so the model learns all three. The overrun target of at most 2% of files per class after the first generation is not claimed: overruns are counted per class and disclosed in Stats, and every file that overruns still runs.

Index

Constants

View Source
const ScopePrefix = "file:"

ScopePrefix is the unit scope this provider indexes: one file, as "file:"+path. The coordinator plans one unit per supported file.

Variables

This section is empty.

Functions

This section is empty.

Types

type Options

type Options struct {
	// Languages restricts the supported set (config tree_sitter.languages);
	// empty means every pinned language.
	Languages []string
	// MaxWorkers is one per CPU (config.ParserWorkers): the most parser subprocesses
	// alive at once.
	MaxWorkers int
	// MaxParseFileBytes is workspace.max_parse_file_bytes, the only size
	// policy on a file: a larger one is reported unavailable, never streamed.
	// Zero is unlimited. A file of any admitted size is streamed to the worker
	// in transport-unit frames (wire.ChunkBytes).
	MaxParseFileBytes config.Limit
	// MaxCalleeReferences is tree_sitter.max_callee_references: how many
	// distinct cross-file callee names one file may mint nodes for. Unlimited
	// by default; past a user-set bound the call is counted into the file's
	// dropped count and the file reports partial.
	MaxCalleeReferences config.Limit
	// MaxEvidencePerFact is the effective per-fact evidence clip: the operator's
	// index.max_evidence_per_fact, or the model's record ceiling when they set
	// none. Zero selects the ceiling. Occurrences past it are counted and
	// disclosed, never dropped in silence.
	MaxEvidencePerFact int
	// Rederive is handed the parser workers' residency, their part of the
	// product's own, just before each file's increment is reserved; the
	// ledger's re-derivation, which runs before every admission, adds the
	// last figure handed over back to the allocation it derives. It is
	// required.
	Rederive func(workerResidentBytes int64)
	// Admission is the process's one reservation ledger. Each worker holds
	// its base on it from before it is started until the runner has reaped
	// it, and each file holds its predicted increment from before it is
	// dispatched until its Done, so parser workers are admitted against the
	// same allocation, in the same queue, as every other heavy child. It is required: a pool with a running total of its own beside
	// the ledger is the oversubscription the ledger exists to prevent.
	Admission *admission.Ledger
	// Worker is the worker executable; Runner starts it; WorkDir is the
	// absolute private directory it runs in.
	Worker  WorkerCommand
	Runner  *process.Runner
	WorkDir string
}

Options configure the provider. Nothing here has a default of its own: the two config.Limit fields are unlimited at zero, as config's defaults are; MaxEvidencePerFact selects the model's ceiling at zero; and the worker count, the allocation's re-derivation, the ledger, the runner, the worker and its directory come from the composition root and are required. There is no parse timeout -- a worker is ended only by a progress-based hang detector (see pool.go).

type Overrun

type Overrun struct {
	Language          string `json:"language"`
	SizeClass         int    `json:"size_class"`
	Files             uint64 `json:"files"`
	LargestDriftBytes int64  `json:"largest_drift_bytes"`
	ReservedBytes     int64  `json:"reserved_bytes"`
	NeedBytes         int64  `json:"need_bytes"`
}

Overrun is one class's files whose observed need exceeded the increment they were admitted with (ADR-0012 decision 5). Each such file ran and succeeded; the drift is need minus reservation, and ReservedBytes and NeedBytes are those of the file with the largest.

type Probe

type Probe struct {
	Language     string
	Declarations int
	Imports      int
	References   int
	SyntaxErrors bool
	Truncated    bool
}

Probe is the outcome of a diagnostic parse: what the worker extracted, without storage, identities or the resolver.

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider is the treesitter provider. It is safe for concurrent use; Close stops every worker.

func New

func New(o Options) (*Provider, error)

New validates options and builds the provider. Nothing is started until the first unit.

func (*Provider) Close

func (p *Provider) Close()

Close stops every worker and waits for the runner to reap each one.

func (*Provider) Descriptor

func (p *Provider) Descriptor() model.ProviderDescriptor

Descriptor reports identity and the scheduling contract. Version folds the binding module, every pinned grammar version and ABI and the query packs (lang.Version), so a grammar or query change invalidates every unit.

func (*Provider) Detect

Detect reports whether the worker executable can be started. The grammars are linked into that executable, so nothing in the repository decides availability; an absent or non-executable worker is unavailable, not a failure.

func (*Provider) IndexUnit

IndexUnit indexes the one file the unit's scope key names. The run always reports succeeded when facts were produced or the file was honestly skipped, with one capability row per descriptor capability for the file's scope. The structure row says fresh, partial (syntax errors, a query that exceeded its match limit, or a bound the builder disclosed) or unavailable (over max_parse_file_bytes, wider than the parser can address, not UTF-8, or not a supported language); the four dependence rows are unavailable with it, and otherwise follow builder.capabilities, which never fails the unit; an unhealthy worker is replaced and the parse retried once; anything else fails the unit. A header is parsed with the grammar its snapshot's census chooses and falls back once to the other (grammarOf); its facts are in the grammar of the parse the worker kept, and the choice is disclosed on its row.

func (*Provider) LanguageOf

func (p *Provider) LanguageOf(fv model.FileVersion) (lang.Language, bool)

LanguageOf reports whether the provider parses a manifest row, and the language it parses it with first in a repository with no C or C++ translation unit. Whether a row is parsed never depends on the repository, so the answer is the planner's eligibility gate; a header's grammar does, so every parse asks grammarOf with its snapshot's census instead.

func (*Provider) OpenStage

func (p *Provider) OpenStage(ctx context.Context) (closeStage func() model.StageFigures)

OpenStage opens one parse stage that holds the pool across every unit indexed before closeStage is called. IndexUnit keeps its own bracket, so a caller that never opens a stage still has the pool held for each unit it indexes, and drained when the last of them ends. closeStage leaves the stage once, however often it is called, under the run the stage was opened in, so no caller can close another run's total. It answers what the pool did while the stage was open, measured after the leave so a drained worker's processor time is in it, and every later call answers the same figures.

func (*Provider) ParseProbe

func (p *Provider) ParseProbe(ctx context.Context, relPath string, src []byte) (Probe, error)

ParseProbe parses src as the file at relPath through the real worker path and reports the extraction counts. It is the diagnostic surface behind doctor-style checks and the resource plateau benchmark: the same pool, framing, validation and retry as IndexUnit, with no unit or sink -- and, as enterStage records, no stage of its own.

func (*Provider) Stats

func (p *Provider) Stats() Stats

Stats is the aggregate parent-plus-worker resource view.

type Stats

type Stats struct {
	// Processes is every worker process the runner has not yet reaped: busy,
	// idle and shutting down alike. It never exceeds Options.MaxWorkers.
	Processes int `json:"processes"`
	// IdleWorkers is the reusable subset of Processes, and BusyWorkers the
	// rest: parsing for a caller, or on their way out. The two are reported
	// separately because one number cannot say whether the pool is saturated
	// with work or merely holding warm processes.
	IdleWorkers    int    `json:"idle_workers"`
	BusyWorkers    int    `json:"busy_workers"`
	WorkersStarted uint64 `json:"workers_started"`
	WorkersExited  uint64 `json:"workers_exited"`
	Parses         uint64 `json:"parses"`
	Retries        uint64 `json:"retries"`
	// MaxParsesInFlight is the most files that held an increment at once.
	MaxParsesInFlight int `json:"max_parses_in_flight"`
	// StageWallMS is the wall time the parse stage was open, from its first
	// enter to its last leave, summed over every stage so far.
	StageWallMS int64 `json:"stage_wall_ms"`
	// WorkerCPUMS sums the processor time of every reaped worker, and is -1
	// when any of them was not measured.
	WorkerCPUMS int64 `json:"worker_cpu_ms"`
	// UnmeasuredFiles are the files whose worker reported no need, from which
	// nothing was learned, and NeedModelFailures the learned models that could
	// not be read or encoded, whose files were reserved at the prior or not
	// recorded.
	UnmeasuredFiles   uint64 `json:"unmeasured_files"`
	NeedModelFailures uint64 `json:"need_model_failures"`
	// Overruns are the classes in which a file needed more than its
	// increment, ordered by language and size class.
	Overruns []Overrun `json:"overruns,omitempty"`
	// PriorShortLanguages are the languages, in order, in which a file
	// reserved at the structural prior needed more than it, and FilesRunAlone
	// the later files of those languages that were reserved at the prior and
	// so ran alone among the parses.
	PriorShortLanguages []string `json:"prior_short_languages,omitempty"`
	FilesRunAlone       uint64   `json:"files_run_alone"`
	// WorkerRSSBytes sums the base each live worker last reported, and is -1
	// when any live worker has reported none.
	WorkerRSSBytes int64 `json:"worker_rss_bytes"`
	// ParentRSSBytes is this process's resident set.
	ParentRSSBytes int64 `json:"parent_rss_bytes"`
	// WorkerPIDs are the live workers, for an external check that they exit.
	WorkerPIDs []int `json:"worker_pids,omitempty"`
}

Stats is the aggregate parent-plus-worker resource view of Section 22: process and idle worker counts, lifetime counters and resident memory. A value that cannot be measured is -1, never zero.

type WorkerCommand

type WorkerCommand struct {
	Path string
	Args []string
}

WorkerCommand is how a parser worker is started: the absolute path of the executable and its literal arguments. In production it is this binary with the hidden wire.Subcommand argument; tests point it at their own test binary, whose TestMain dispatches to worker.Main.

Directories

Path Synopsis
Package flow is the per-function dependence core: a language-neutral control-flow graph, its dominance relations, control dependence and SSA def-use.
Package flow is the per-function dependence core: a language-neutral control-flow graph, its dominance relations, control dependence and SSA def-use.
Package lang is the pinned language registry of the tree-sitter provider: the nine mandatory grammars of Section 11.3, the exact module version, grammar ABI and grammar metadata each one was reviewed at, the file extensions each grammar declares, and the embedded structural query pack.
Package lang is the pinned language registry of the tree-sitter provider: the nine mandatory grammars of Section 11.3, the exact module version, grammar ABI and grammar metadata each one was reviewed at, the file extensions each grammar declares, and the embedded structural query pack.
Package wire is the framed protocol between the tree-sitter provider and its parser worker subprocess.
Package wire is the framed protocol between the tree-sitter provider and its parser worker subprocess.
Package worker is the parser side of the tree-sitter provider: a subprocess that owns every native object (parser, tree, query, query cursor) and never touches storage, identities or the repository.
Package worker is the parser side of the tree-sitter provider: a subprocess that owns every native object (parser, tree, query, query cursor) and never touches storage, identities or the repository.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL