Documentation
¶
Overview ¶
Package treesitter is the bundled structural provider of Section 11.3: a tree-sitter pass over each supported source file that publishes declarations, containing scopes, signatures, imports and exports, syntax references and call sites, test declarations and attached documentation at precision syntax, and, from the same worker walk, the file-local dependence families control_depends_on, data_flows_to, reads and writes between the file's parameters and locals at precision static_analysis (docs/providers-treesitter.md, Dependence facts). Parsing runs in isolated worker subprocesses (package worker) started through the shared process runner; this package is the parent side, which streams pinned bytes to a worker, validates every framed fact it answers with against those bytes, resolves identities through the unit's resolver and emits facts through the sink. It never links the grammars itself and never holds a repository-wide AST or source cache: the unit of work is one file, and its bytes live only for that unit.
Memory is taken from each file's observed need (ADR-0012 decision 5): a worker holds its reported base on the process's reservation ledger, and each file reserves its predicted increment before it is dispatched -- the learned p99 of need per source byte for its repository, language, grammar fingerprint and size class, or the structural prior for the first file of that key. The need a worker measures covers the parse, the extraction and the dependence lowering and analysis of every callable, which all run in the worker's one walk over the file, so the model learns all three. The overrun target of at most 2% of files per class after the first generation is not claimed: overruns are counted per class and disclosed in Stats, and every file that overruns still runs.
Index ¶
- Constants
- type Options
- type Overrun
- type Probe
- type Provider
- func (p *Provider) Close()
- func (p *Provider) Descriptor() model.ProviderDescriptor
- func (p *Provider) Detect(_ context.Context, _ workspace.Root, _ workspace.Policy) (provider.Detection, error)
- func (p *Provider) IndexUnit(ctx context.Context, req provider.UnitRequest, sink provider.Sink) (model.ProviderResult, error)
- func (p *Provider) LanguageOf(fv model.FileVersion) (lang.Language, bool)
- func (p *Provider) OpenStage(ctx context.Context) (closeStage func() model.StageFigures)
- func (p *Provider) ParseProbe(ctx context.Context, relPath string, src []byte) (Probe, error)
- func (p *Provider) Stats() Stats
- type Stats
- type WorkerCommand
Constants ¶
const ScopePrefix = "file:"
ScopePrefix is the unit scope this provider indexes: one file, as "file:"+path. The coordinator plans one unit per supported file.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
// Languages restricts the supported set (config tree_sitter.languages);
// empty means every pinned language.
Languages []string
// MaxWorkers is one per CPU (config.ParserWorkers): the most parser subprocesses
// alive at once.
MaxWorkers int
// MaxParseFileBytes is workspace.max_parse_file_bytes, the only size
// policy on a file: a larger one is reported unavailable, never streamed.
// Zero is unlimited. A file of any admitted size is streamed to the worker
// in transport-unit frames (wire.ChunkBytes).
MaxParseFileBytes config.Limit
// MaxCalleeReferences is tree_sitter.max_callee_references: how many
// distinct cross-file callee names one file may mint nodes for. Unlimited
// by default; past a user-set bound the call is counted into the file's
// dropped count and the file reports partial.
MaxCalleeReferences config.Limit
// MaxEvidencePerFact is the effective per-fact evidence clip: the operator's
// index.max_evidence_per_fact, or the model's record ceiling when they set
// none. Zero selects the ceiling. Occurrences past it are counted and
// disclosed, never dropped in silence.
MaxEvidencePerFact int
// Rederive is handed the parser workers' residency, their part of the
// product's own, just before each file's increment is reserved; the
// ledger's re-derivation, which runs before every admission, adds the
// last figure handed over back to the allocation it derives. It is
// required.
Rederive func(workerResidentBytes int64)
// Admission is the process's one reservation ledger. Each worker holds
// its base on it from before it is started until the runner has reaped
// it, and each file holds its predicted increment from before it is
// dispatched until its Done, so parser workers are admitted against the
// same allocation, in the same queue, as every other heavy child. It is required: a pool with a running total of its own beside
// the ledger is the oversubscription the ledger exists to prevent.
Admission *admission.Ledger
// Worker is the worker executable; Runner starts it; WorkDir is the
// absolute private directory it runs in.
Worker WorkerCommand
Runner *process.Runner
WorkDir string
}
Options configure the provider. Nothing here has a default of its own: the two config.Limit fields are unlimited at zero, as config's defaults are; MaxEvidencePerFact selects the model's ceiling at zero; and the worker count, the allocation's re-derivation, the ledger, the runner, the worker and its directory come from the composition root and are required. There is no parse timeout -- a worker is ended only by a progress-based hang detector (see pool.go).
type Overrun ¶
type Overrun struct {
Language string `json:"language"`
SizeClass int `json:"size_class"`
Files uint64 `json:"files"`
LargestDriftBytes int64 `json:"largest_drift_bytes"`
ReservedBytes int64 `json:"reserved_bytes"`
NeedBytes int64 `json:"need_bytes"`
}
Overrun is one class's files whose observed need exceeded the increment they were admitted with (ADR-0012 decision 5). Each such file ran and succeeded; the drift is need minus reservation, and ReservedBytes and NeedBytes are those of the file with the largest.
type Probe ¶
type Probe struct {
Language string
Declarations int
Imports int
References int
SyntaxErrors bool
Truncated bool
}
Probe is the outcome of a diagnostic parse: what the worker extracted, without storage, identities or the resolver.
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider is the treesitter provider. It is safe for concurrent use; Close stops every worker.
func (*Provider) Close ¶
func (p *Provider) Close()
Close stops every worker and waits for the runner to reap each one.
func (*Provider) Descriptor ¶
func (p *Provider) Descriptor() model.ProviderDescriptor
Descriptor reports identity and the scheduling contract. Version folds the binding module, every pinned grammar version and ABI and the query packs (lang.Version), so a grammar or query change invalidates every unit.
func (*Provider) Detect ¶
func (p *Provider) Detect(_ context.Context, _ workspace.Root, _ workspace.Policy) (provider.Detection, error)
Detect reports whether the worker executable can be started. The grammars are linked into that executable, so nothing in the repository decides availability; an absent or non-executable worker is unavailable, not a failure.
func (*Provider) IndexUnit ¶
func (p *Provider) IndexUnit(ctx context.Context, req provider.UnitRequest, sink provider.Sink) (model.ProviderResult, error)
IndexUnit indexes the one file the unit's scope key names. The run always reports succeeded when facts were produced or the file was honestly skipped, with one capability row per descriptor capability for the file's scope. The structure row says fresh, partial (syntax errors, a query that exceeded its match limit, or a bound the builder disclosed) or unavailable (over max_parse_file_bytes, wider than the parser can address, not UTF-8, or not a supported language); the four dependence rows are unavailable with it, and otherwise follow builder.capabilities, which never fails the unit; an unhealthy worker is replaced and the parse retried once; anything else fails the unit. A header is parsed with the grammar its snapshot's census chooses and falls back once to the other (grammarOf); its facts are in the grammar of the parse the worker kept, and the choice is disclosed on its row.
func (*Provider) LanguageOf ¶
LanguageOf reports whether the provider parses a manifest row, and the language it parses it with first in a repository with no C or C++ translation unit. Whether a row is parsed never depends on the repository, so the answer is the planner's eligibility gate; a header's grammar does, so every parse asks grammarOf with its snapshot's census instead.
func (*Provider) OpenStage ¶
func (p *Provider) OpenStage(ctx context.Context) (closeStage func() model.StageFigures)
OpenStage opens one parse stage that holds the pool across every unit indexed before closeStage is called. IndexUnit keeps its own bracket, so a caller that never opens a stage still has the pool held for each unit it indexes, and drained when the last of them ends. closeStage leaves the stage once, however often it is called, under the run the stage was opened in, so no caller can close another run's total. It answers what the pool did while the stage was open, measured after the leave so a drained worker's processor time is in it, and every later call answers the same figures.
func (*Provider) ParseProbe ¶
ParseProbe parses src as the file at relPath through the real worker path and reports the extraction counts. It is the diagnostic surface behind doctor-style checks and the resource plateau benchmark: the same pool, framing, validation and retry as IndexUnit, with no unit or sink -- and, as enterStage records, no stage of its own.
type Stats ¶
type Stats struct {
// Processes is every worker process the runner has not yet reaped: busy,
// idle and shutting down alike. It never exceeds Options.MaxWorkers.
Processes int `json:"processes"`
// IdleWorkers is the reusable subset of Processes, and BusyWorkers the
// rest: parsing for a caller, or on their way out. The two are reported
// separately because one number cannot say whether the pool is saturated
// with work or merely holding warm processes.
IdleWorkers int `json:"idle_workers"`
BusyWorkers int `json:"busy_workers"`
WorkersStarted uint64 `json:"workers_started"`
WorkersExited uint64 `json:"workers_exited"`
Parses uint64 `json:"parses"`
Retries uint64 `json:"retries"`
// MaxParsesInFlight is the most files that held an increment at once.
MaxParsesInFlight int `json:"max_parses_in_flight"`
// StageWallMS is the wall time the parse stage was open, from its first
// enter to its last leave, summed over every stage so far.
StageWallMS int64 `json:"stage_wall_ms"`
// WorkerCPUMS sums the processor time of every reaped worker, and is -1
// when any of them was not measured.
WorkerCPUMS int64 `json:"worker_cpu_ms"`
// UnmeasuredFiles are the files whose worker reported no need, from which
// nothing was learned, and NeedModelFailures the learned models that could
// not be read or encoded, whose files were reserved at the prior or not
// recorded.
UnmeasuredFiles uint64 `json:"unmeasured_files"`
NeedModelFailures uint64 `json:"need_model_failures"`
// Overruns are the classes in which a file needed more than its
// increment, ordered by language and size class.
Overruns []Overrun `json:"overruns,omitempty"`
// PriorShortLanguages are the languages, in order, in which a file
// reserved at the structural prior needed more than it, and FilesRunAlone
// the later files of those languages that were reserved at the prior and
// so ran alone among the parses.
PriorShortLanguages []string `json:"prior_short_languages,omitempty"`
FilesRunAlone uint64 `json:"files_run_alone"`
// WorkerRSSBytes sums the base each live worker last reported, and is -1
// when any live worker has reported none.
WorkerRSSBytes int64 `json:"worker_rss_bytes"`
// ParentRSSBytes is this process's resident set.
ParentRSSBytes int64 `json:"parent_rss_bytes"`
// WorkerPIDs are the live workers, for an external check that they exit.
WorkerPIDs []int `json:"worker_pids,omitempty"`
}
Stats is the aggregate parent-plus-worker resource view of Section 22: process and idle worker counts, lifetime counters and resident memory. A value that cannot be measured is -1, never zero.
type WorkerCommand ¶
WorkerCommand is how a parser worker is started: the absolute path of the executable and its literal arguments. In production it is this binary with the hidden wire.Subcommand argument; tests point it at their own test binary, whose TestMain dispatches to worker.Main.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package flow is the per-function dependence core: a language-neutral control-flow graph, its dominance relations, control dependence and SSA def-use.
|
Package flow is the per-function dependence core: a language-neutral control-flow graph, its dominance relations, control dependence and SSA def-use. |
|
Package lang is the pinned language registry of the tree-sitter provider: the nine mandatory grammars of Section 11.3, the exact module version, grammar ABI and grammar metadata each one was reviewed at, the file extensions each grammar declares, and the embedded structural query pack.
|
Package lang is the pinned language registry of the tree-sitter provider: the nine mandatory grammars of Section 11.3, the exact module version, grammar ABI and grammar metadata each one was reviewed at, the file extensions each grammar declares, and the embedded structural query pack. |
|
Package wire is the framed protocol between the tree-sitter provider and its parser worker subprocess.
|
Package wire is the framed protocol between the tree-sitter provider and its parser worker subprocess. |
|
Package worker is the parser side of the tree-sitter provider: a subprocess that owns every native object (parser, tree, query, query cursor) and never touches storage, identities or the repository.
|
Package worker is the parser side of the tree-sitter provider: a subprocess that owns every native object (parser, tree, query, query cursor) and never touches storage, identities or the repository. |