indexer

package
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 8, 2026 License: MIT Imports: 38 Imported by: 0

Documentation

Overview

Package indexer implements the two-pass Go indexing pipeline: Pass 0 (deterministic file discovery, this file's neighbor discover.go), Pass 1 (parallel per-file extraction), and Pass 2 (sequential cross-file resolution), writing through internal/graphstore.

Package indexer's resolve.go implements Pass 2 of the two-pass indexing pipeline (D-04): build a global symbol index over every Pass-1 result, resolve calls/imports/embeds/contains references into ground-truth edges, deterministically collapse duplicate edges (D-05), and commit the whole resolved graph through exactly one batched GraphStore.Writer (D-04a).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Extract

func Extract(files []DiscoveredFile, limit int) ([]goextract.FileResult, error)

Extract runs Pass 1 of the two-pass indexing pipeline (D-04): a bounded pool of persistent workers walks every discovered file and produces one goextract.FileResult per file. Each worker owns a small per-language parser cache (map[string]parser.Parser, lazily populated, closed at worker exit) instead of a single parser for its whole lifetime — the Pitfall-1 fix required the moment a single Extract() call spans more than one language: a worker that claims a Java file then a TypeScript file must be able to swap grammars without reconstructing a parser per file.

limit caps the number of workers (and therefore the parser-cache count); limit <= 0 defaults to runtime.NumCPU().

Results are written to a pre-allocated slice at each file's own index — never appended in completion order — so the returned slice is always in the SAME order as files, regardless of which worker happened to claim which file or when it finished (the first line of defense for D-01a determinism; RESEARCH Pattern 2).

A per-file extraction failure (e.g. parser.ErrSourceTooLarge) is recorded on that file's FileResult.Err and does NOT abort the rest of the batch (RESEARCH Pitfall 4, threat T-02-03) — Extract itself returns a non-nil error only for a condition that makes the whole batch meaningless, such as a worker failing to construct a language's parser, an unregistered language, or an unreadable file.

func RegisteredLanguageIDs

func RegisteredLanguageIDs() []string

RegisteredLanguageIDs returns every language ID currently registered in the LanguageSpec registry (D-01), sorted ascending. This is the D-11 capability matrix's source of truth for "which languages must the human-/machine-readable coverage matrix cover" — internal/indexer/capability/matrix_test.go consumes this to assert the matrix descriptor covers exactly the registered languages, no missing, no extra.

func Resolve

func Resolve(store graphstore.GraphStore, results []goextract.FileResult, modulePath string) (int, error)

Resolve runs the whole of Pass 2: builds the global symbol index, resolves every file's Unresolved references into edges, deterministically collapses duplicates, and commits the resolved graph through exactly one batched GraphStore.Writer (D-04a). It returns the count of references that could not be resolved (surfaced later via --verbose, never silently dropped — D-06a).

func ShouldSkipDir

func ShouldSkipDir(name string) bool

ShouldSkipDir reports whether a directory named name should be excluded from traversal — Discover's own WalkDir callback and, per Phase 4 D-04, the native filesystem watcher's recursive-add loop both call this exact predicate so the two never silently diverge on which paths they cover. vendor/ and any dot-prefixed directory (.git, .codegraph, etc.) are excluded.

Types

type DiscoveredFile

type DiscoveredFile struct {
	// AbsPath is the absolute path on disk, for reading bytes.
	AbsPath string
	// RelPath is the slash-normalized path relative to the repo root, the
	// file_path stored on Node/File records.
	RelPath string
	// ImportPath is this file's cross-file symbol-index key, computed by
	// its language's LanguageSpec.ModuleKey (D-03/Pitfall 2). For Go this
	// is the module path joined with the file's relative directory (""
	// relative directory means the module root package) — byte-identical
	// to the pre-Phase-5 behavior. A file whose language has no resolvable
	// project descriptor still gets a value here (its LanguageSpec's
	// path-based fallback), never a dropped file.
	ImportPath string
	// Language is the registered LanguageSpec.ID this file's extension
	// resolved to ("go", "java", "python", ...) — the key extract.go's
	// worker pool uses to select the correct parser + extractor per file
	// (Pitfall 1).
	Language string

	// MtimeUnixNs and SizeBytes are the file's on-disk stat info at
	// discovery time (Phase 4 D-01a) — carried through Extract into the
	// committed File record so Sync's stat pre-filter has something cheap
	// to compare against on the next invocation, without hashing every
	// file every sync.
	MtimeUnixNs int64
	SizeBytes   int64
}

DiscoveredFile is one discovered source file of any registered language, the shape Pass 1 (extract) and Pass 2 (resolve) consume.

func Discover

func Discover(root string) ([]DiscoveredFile, string, error)

Discover walks root and returns every file whose extension is claimed by a registered LanguageSpec (D-03), sorted by RelPath in ascending byte order. This stable order is determinism's first line of defense: the same input tree always yields the same output order, regardless of filesystem walk order.

vendor/ directories and any dot-prefixed directory (.git, .codegraph, etc.) are skipped entirely (ShouldSkipDir, shared verbatim with the Phase-4 watcher). A candidate file is included iff its extension is registered in the extension->language registry (languages.go); an unsupported extension (.md, .json, ...) is never returned. Go source files additionally require go/build.Context.MatchFile to report they belong to the default build context (GOOS/GOARCH, build tags) — the same primitive the go toolchain itself uses — gated to Language=="go" only, since no other language in the registry has a build-tag concept.

After the walk, each language actually present is given exactly one chance to resolve its repo-root project descriptor (go.mod, pom.xml, *.csproj, ...) via LanguageSpec.Descriptor. A descriptor that is absent, malformed, or simply not implemented for that language does NOT fail Discover (D-03/T-05-Manifest) — LanguageSpec.ModuleKey is called with a nil descriptor and is required to degrade to a path-based identity rather than dropping the file. This is the one behavioral relaxation from the pre-Phase-5 contract: a root with no go.mod (and only Go files) used to be a hard Discover error; it now succeeds with Go's own nil-descriptor fallback (languages_go.go).

Discover's second return value remains the repo's Go module path specifically (as resolved by the "go" LanguageSpec's own descriptor, if any) — every existing caller (Sync, Resolve, symbolindex.go) consumes this Go-specific value unchanged; it is "" when no go.mod was found.

type LanguageSpec

type LanguageSpec struct {
	// ID is the language's stable registry key ("go", "java", "csharp",
	// "python", "typescript", ...).
	ID string

	// Extensions are the file extensions (including the leading ".") this
	// language claims during discovery.
	Extensions []string

	// NewParser constructs a fresh parser.Parser for this language, routed
	// through the parser.Parser seam (and, for the CGo backend, the
	// MaxSourceBytes ceiling in newCGoParser/Parse — Security Domain V5).
	NewParser func() (parser.Parser, error)

	// Extract walks one file's already-parsed syntax tree into a
	// goextract.FileResult. moduleKey is this language's cross-file
	// symbol-index key for the file (Go's importPath is the first
	// instance of this concept; other languages compute a structurally
	// different key via ModuleKey below).
	Extract func(p parser.Parser, moduleKey, relPath string, src []byte) (goextract.FileResult, error)

	// ModuleKey computes this language's cross-file symbol-index key for a
	// discovered file, given the repo's resolved ProjectDescriptor (which
	// may be nil if this language's descriptor could not be resolved — a
	// file whose language has no descriptor still gets extracted with
	// path-based identity per D-03, so implementations must tolerate a
	// nil descriptor rather than panicking).
	ModuleKey func(descriptor ProjectDescriptor, relPath string) string

	// Descriptor parses this language's manifest (go.mod, pom.xml,
	// *.csproj, pyproject.toml, package.json+tsconfig.json, ...) once per
	// repo root to resolve module/namespace identity for the whole repo.
	Descriptor func(root string) (ProjectDescriptor, error)
}

LanguageSpec is the single source of truth for one language's parser+extractor+ModuleKey selection (D-01). Every subsequent wave (discovery, extract, resolve, per-language extractors, dispatch, routing) reads this registry — nothing else can land before it.

type Options

type Options struct {
	// Workers bounds Pass 1's extraction worker pool. <= 0 defaults to
	// runtime.NumCPU() — Extract's own default (D-04), applied here too so
	// callers observe the same behavior whether they pass Options{} or
	// Options{Workers: runtime.NumCPU()} explicitly.
	Workers int

	// Verbose and Quiet are carried through for the CLI's summary output
	// (Stats already reports Unresolved/Skipped counts regardless of
	// these flags); Run itself does no logging.
	Verbose bool
	Quiet   bool
}

Options configures one Run invocation.

type ProjectDescriptor

type ProjectDescriptor interface {
	// ModulePath returns the descriptor's resolved base identity — a Go
	// module path today; a Java/C# root package/namespace, a Python
	// project root, or a TS/JS resolved package name for later languages.
	ModulePath() string
}

ProjectDescriptor resolves a repo's per-language module/namespace identity (go.mod, pom.xml, *.csproj, package.json+tsconfig.json, pyproject.toml, ...), parsed once per repo root (D-03). Go's existing go.mod path resolution (languages_go.go) is the first implementation of this hook; Wave 2 (discover.go's generalization) adds the remaining per-language descriptor parsers behind this same interface.

type Stats

type Stats struct {
	Files      int
	Nodes      int
	Edges      int
	Unresolved int
	Skipped    int
	Duration   time.Duration

	// FilesReparsed, FilesPruned, NodesRemoved, EdgesRemoved, and
	// DependentsRecomputed are Sync-only summary counts (Phase 4 D-01b) —
	// zero-valued for a from-scratch Run. FilesReparsed is the size of the
	// Extract() batch (added ∪ modified ∪ dependent); FilesPruned counts
	// modified+deleted files whose subgraph was pruned via the x/ index;
	// NodesRemoved/EdgesRemoved count the individual n/e records
	// point-deleted during that prune; DependentsRecomputed counts files
	// re-extracted purely because a symbol they referenced was pruned
	// (RESEARCH Pitfall 2), not because their own content changed.
	FilesReparsed        int
	FilesPruned          int
	NodesRemoved         int
	EdgesRemoved         int
	DependentsRecomputed int
}

Stats summarizes one Run invocation: how many files were discovered, how many nodes/edges landed in the committed graph, how many cross-file references could not be resolved (D-06a — never silently dropped), how many files were skipped outright (parser.ErrSourceTooLarge or a read failure, RESEARCH Pitfall 4), and how long the whole run took.

func Run

func Run(repoRoot, storeDir string, opts Options) (Stats, error)

Run executes the full from-scratch indexing pipeline (D-04, D-01a): Discover walks repoRoot for every source file whose extension is claimed by a registered LanguageSpec (Phase 5 D-03 — Go was the only such language through Phase 4), Extract runs Pass 1 (parallel, bounded worker pool, selecting a parser+extractor per file's own Language — Phase 5 Pitfall 1) over them, and Resolve runs Pass 2 (the single coordinated writer) against the GraphStore at storeDir, committing the resolved graph and stamping Meta.

The store is opened exactly once and Closed on every return path — success or failure — mirroring pebble_store.Open's Close-once lifecycle discipline (T-02-11): a failure partway through Pass 2 never leaves an open engine handle or lock behind.

func Sync

func Sync(repoRoot, storeDir string, opts Options) (Stats, error)

Sync performs an incremental update of the graph at storeDir against the current on-disk state of repoRoot (INDX-03): only content-hash-changed files, plus their direct call-graph dependents, are reparsed; changed/deleted files' subgraphs are pruned via the x/ file-owned secondary index (D-02); and the whole prune+write lands in exactly ONE atomic Writer commit (D-01b — never per-symbol). One routine, three callers: `codegraph sync`, the debounced daemon cycle, and MCP-reconnect reconcile — see 04-RESEARCH.md Pattern 1 for the algorithm this implements.

Sync mirrors Run's open-once/Close-on-every-path lifecycle (D-04), but performs a store-seeded incremental resolve instead of a from-scratch one: the global symbol index is seeded from the store's own committed nodes (newSymbolIndexFromStore), not just the reparse batch, so a reference into an unchanged file still resolves (RESEARCH Pitfall 1).

Directories

Path Synopsis
Package capability is the D-11 language capability descriptor: the machine-readable half of the "language capability matrix" (the other half is the human-readable docs/LANGUAGE-CAPABILITY-MATRIX.md, which this package's own consistency test — matrix_test.go — keeps mirrored to this file exactly, coverage value for coverage value, gap for gap).
Package capability is the D-11 language capability descriptor: the machine-readable half of the "language capability matrix" (the other half is the human-readable docs/LANGUAGE-CAPABILITY-MATRIX.md, which this package's own consistency test — matrix_test.go — keeps mirrored to this file exactly, coverage value for coverage value, gap for gap).
Package csharpextract walks a C# file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/javaextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package csharpextract walks a C# file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/javaextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package dispatch synthesizes Go's implicit structural interface satisfaction into explicit "implements" edges (RES-02, Phase 5 Pattern 3): a struct whose method set (by name+arity, D-06's bounded-matching discipline) is a superset of an interface's own method-spec set is synthesized as implementing that interface.
Package dispatch synthesizes Go's implicit structural interface satisfaction into explicit "implements" edges (RES-02, Phase 5 Pattern 3): a struct whose method set (by name+arity, D-06's bounded-matching discipline) is a superset of an interface's own method-spec set is synthesized as implementing that interface.
Package goextract implements Pass 1's Go-specific tree-walk (LANG-01): mapping a parsed Go file's tree-sitter concrete syntax tree onto the codegraph node/edge vocabulary (D-06) — function/method/struct/ interface/type_alias/constant/variable nodes, intra-file contains edges, and unresolved cross-file references (calls, imports, struct/ interface embedding) for Pass 2 (resolve, a later plan) to settle.
Package goextract implements Pass 1's Go-specific tree-walk (LANG-01): mapping a parsed Go file's tree-sitter concrete syntax tree onto the codegraph node/edge vocabulary (D-06) — function/method/struct/ interface/type_alias/constant/variable nodes, intra-file contains edges, and unresolved cross-file references (calls, imports, struct/ interface embedding) for Pass 2 (resolve, a later plan) to settle.
Package javaextract walks a Java file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package javaextract walks a Java file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
mainstream
cextract
Package cextract walks a C or C++ file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package cextract walks a C or C++ file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
kotlinextract
Package kotlinextract walks a Kotlin file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/ IntraEdge/UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package kotlinextract walks a Kotlin file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/ IntraEdge/UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
phpextract
Package phpextract walks a PHP file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package phpextract walks a PHP file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
rubyextract
Package rubyextract walks a Ruby file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package rubyextract walks a Ruby file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
rustextract
Package rustextract walks a Rust file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package rustextract walks a Rust file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
swiftextract
Package swiftextract walks a Swift file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package swiftextract walks a Swift file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary.
Package nodeid computes the deterministic content-hashed identifier assigned to every extracted symbol (D-02/D-02a).
Package nodeid computes the deterministic content-hashed identifier assigned to every extracted symbol (D-02/D-02a).
Package pyextract walks a Python file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary — the same discipline javaextract/csharpextract already follow.
Package pyextract walks a Python file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/ UnresolvedRef, goextract.Kind*/RefKind*), mirroring internal/indexer/goextract's shape (D-01) rather than redefining its own copy of that vocabulary — the same discipline javaextract/csharpextract already follow.
Package routes implements LANG-07's per-framework route-detector registry (D-08): a small, opt-in-per-detected-dependency (D-09) set of AST-based detectors — Gin (Go), Spring (Java), ASP.NET (C#), Django/Flask/FastAPI (Python), Express/NestJS (TypeScript/JavaScript) — each scanning one already-parsed file's syntax tree for that framework's route-declaration shape and resolving the route's handler to an already-known function/method node id.
Package routes implements LANG-07's per-framework route-detector registry (D-08): a small, opt-in-per-detected-dependency (D-09) set of AST-based detectors — Gin (Go), Spring (Java), ASP.NET (C#), Django/Flask/FastAPI (Python), Express/NestJS (TypeScript/JavaScript) — each scanning one already-parsed file's syntax tree for that framework's route-declaration shape and resolving the route's handler to an already-known function/method node id.
Package tsextract walks a TypeScript, TSX, or JavaScript file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/UnresolvedRef, goextract.Kind*/RefKind*) — ONE extractor serving all three grammars registered by languages_typescript.go ("typescript" .ts, "tsx" .tsx, "javascript" .js/.jsx/.mjs/.cjs), mirroring javaextract's/csharpextract's/ pyextract's shape (D-01) rather than defining three near-duplicate packages.
Package tsextract walks a TypeScript, TSX, or JavaScript file's tree-sitter syntax tree into the shared codegraph vocabulary (goextract.FileResult/ExtractedNode/IntraEdge/UnresolvedRef, goextract.Kind*/RefKind*) — ONE extractor serving all three grammars registered by languages_typescript.go ("typescript" .ts, "tsx" .tsx, "javascript" .js/.jsx/.mjs/.cjs), mirroring javaextract's/csharpextract's/ pyextract's shape (D-01) rather than defining three near-duplicate packages.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL