Documentation
¶
Overview ¶
Package indexer is the indexing pipeline that walks repositories, parses sources, and produces fingerprints for the store.
Index ¶
- func BuildSubsystems(ctx context.Context, s *store.Store) (int, error)
- func BuiltinHelpers(lang string) map[string]struct{}
- func CalleeKey(lang, qualifier, name string) string
- func CloneHash(node *sitter.Node, source []byte) string
- func CloneHashOf(ctx context.Context, lang string, snippet []byte) string
- func DisplayHelper(key string) string
- func ErrorClassNoun(lang, class string) string
- func ErrorClassRank(lang, class string) (int, bool)
- func FamilyApplies(lang, family string, fp types.Fingerprint) bool
- func FamilyMember(lang, family, key string) string
- func FinalizeVocabulary(ctx context.Context, s *store.Store) (int, error)
- func IsDistinctBuiltin(key string) bool
- func IsGeneratedPath(path string) bool
- func IsTestFile(repoRoot, path string) bool
- func IsTestScope(scope string) bool
- func LanguageFor(path string) string
- func MinePatterns(ctx context.Context, s *store.Store) (int, error)
- func Parse(ctx context.Context, repoRoot string, in <-chan string, workers int) (<-chan types.FunctionRecord, <-chan types.VocabEntry, <-chan types.FileRecord)
- func PropagateIO(ctx context.Context, s *store.Store) (int, error)
- func ScopeDir(scope string) string
- func ScopeFor(repoRoot, path string) string
- func Walk(ctx context.Context, repoRoot string) (<-chan string, error)
- func Write(ctx context.Context, s *store.Store, functions <-chan types.FunctionRecord, ...) (int, error)
- type Analysis
- type FunctionCalls
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func BuildSubsystems ¶
BuildSubsystems aggregates per-directory statistics from the functions and fingerprints tables and populates the subsystems table.
func BuiltinHelpers ¶
BuiltinHelpers returns the builtin callee names the miner strips for a language, so other packages can strip them the same way.
func CalleeKey ¶
CalleeKey is the identity of a callee in mined claims: the qualifier and name joined with the language's path separator, or the bare name when the call could not be qualified.
func CloneHash ¶
CloneHash hashes a function as a token sequence in which every identifier is replaced by the index of its first occurrence and every other token keeps its text, so two functions that differ only by renaming hash alike while a changed literal — a different SQL string, a different message — tells them apart. It is the Type-2 clone test behind duplicate detection.
func CloneHashOf ¶
CloneHashOf parses snippet — one function's source, as the index stores it — as lang and returns the CloneHash of the first function in it, or "" when none parses.
func DisplayHelper ¶
DisplayHelper shortens a callee key for prose: the last path element of an import path, so github.com/pkg/errors.Wrap reads as errors.Wrap and github.com/go-chi/chi/v5.NewRouter as chi.NewRouter.
func ErrorClassNoun ¶
ErrorClassNoun describes an error-handling class for prose.
func ErrorClassRank ¶
ErrorClassRank returns the rank of an error-handling class for a language (lower is better) and whether the class is known.
func FamilyApplies ¶
func FamilyApplies(lang, family string, fp types.Fingerprint) bool
FamilyApplies reports whether a family's choice is one a function with fingerprint fp makes at all, so check skips the functions mining skipped.
func FamilyMember ¶
FamilyMember returns the label of the member of family that key belongs to for the given language, or "" when key is in no member. It is the read side the check uses to match a function's calls against a family_dominance claim.
func FinalizeVocabulary ¶
FinalizeVocabulary adds function-derived vocabulary from the functions table and updates usage counts for all entries. The vocabulary table must be cleared before the parse pipeline runs; this function only appends function entries and updates counts.
func IsDistinctBuiltin ¶
IsDistinctBuiltin reports whether key is one of the builtins whose absence from a scope check enforces without the repository opting in: the debugging and panicking calls repositories ban.
func IsGeneratedPath ¶
IsGeneratedPath reports whether path's name marks machine-generated code.
func IsTestFile ¶
IsTestFile reports whether path is a test file by its language's naming convention or by living under a test directory inside repoRoot.
func IsTestScope ¶
IsTestScope reports whether scope is a test partition.
func LanguageFor ¶
LanguageFor returns the language name for a file path based on its extension, or "" if the extension is not supported.
func MinePatterns ¶
MinePatterns analyzes indexed data and populates the patterns table with statistical claims about coding style in the repository.
func Parse ¶
func Parse(ctx context.Context, repoRoot string, in <-chan string, workers int) (<-chan types.FunctionRecord, <-chan types.VocabEntry, <-chan types.FileRecord)
Parse consumes file paths from in, parses each file, extracts named function definitions, vocabulary entries (types, constants, macros), and file-level facts, and emits them to the returned channels. All three close when in closes or ctx is cancelled. repoRoot scopes test files. workers sets the parser pool size (each worker owns its tree-sitter parser); values below 1 are raised to 1.
func PropagateIO ¶
PropagateIO walks the call graph and updates does_io for functions that transitively call I/O functions. Returns the number of functions flipped.
func ScopeFor ¶
ScopeFor returns the mining scope of a source file: its directory, with a "#test" suffix when the file is a test file.
func Walk ¶
Walk traverses repoRoot and emits paths to source files in supported languages. The returned channel closes when the walk completes or the context is cancelled. Files that cannot be read or stat'd are skipped silently. Hard-coded ignores (dot-directories, node_modules, vendor, target, dist, build, __pycache__, venv, coverage) take precedence over .gitignore matching; .gitignore files at any depth in the repo add project-specific exclusions on top. The function returns an error only if the walk cannot begin (e.g., root does not exist or is not a directory). The channel is only valid when the returned error is nil.
func Write ¶
func Write(ctx context.Context, s *store.Store, functions <-chan types.FunctionRecord, vocab <-chan types.VocabEntry, files <-chan types.FileRecord) (int, error)
Write consumes function records, vocabulary entries, and file records from their channels and persists them to the store in batched transactions. Returns the total number of function records written and the first error that aborted any write loop.
Types ¶
type Analysis ¶
type Analysis struct {
File types.FileRecord
Generated bool
IgnoreFile bool // an opentuple:ignore-file directive in the file's first lines
Functions []FunctionCalls
}
Analysis is everything the check command learns about one source file without the index: its file-level facts, whether it is generated, and its functions with their callee sets.
func AnalyzeSource ¶
func AnalyzeSource(ctx context.Context, repoRoot, filePath string, source []byte) (*Analysis, error)
AnalyzeSource parses source as the language implied by filePath, exactly as the indexer would, with language builtins removed from the callee sets as the miner removes them. It returns nil, nil when the extension maps to no supported language.
type FunctionCalls ¶
type FunctionCalls struct {
Record types.FunctionRecord
Direct map[string]struct{}
All map[string]struct{}
Undeferred map[string]struct{} // callees reached by at least one call outside defer/finally
Ignores map[string]bool // rule types an opentuple:ignore directive on the function waives; "*" for all
CloneHash string // see CloneHash
}
FunctionCalls is one named function with its record and callee sets, keyed by CalleeKey. Direct holds the callees the indexer attributes to the function itself — calls inside nested function literals belong to the literal, as in extractCalls — so it matches the evidence the miner counted. All adds the callees reached inside nested function literals, for the lenient side of a pairing test: a deferred closure that unlocks still unlocks.
Source Files
¶
- analyze.go
- errors.go
- facts.go
- families_config.go
- fingerprint.go
- fingerprint_config.go
- imports.go
- io_propagation.go
- languages.go
- miner.go
- miner_config.go
- miner_conventions.go
- miner_families.go
- miner_files.go
- miner_helpers.go
- miner_reference.go
- miner_wrappers.go
- parser.go
- parser_bindings.go
- scope.go
- subsystems.go
- vocabulary.go
- walker.go
- writer.go