ingest

package
v0.22.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 15, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var MaxIngestFileSize int64 = 100 << 20 // 100 MB

MaxIngestFileSize is the largest file we'll read into memory during ingestion or schema inference. Files above this are silently skipped. Set to 0 to disable the size limit. Configurable via --max-file-size.

Functions

func FlattenASTDB added in v0.17.0 

func FlattenASTDB(db *sql.DB, sourceLike string, limit int) ([]any, error)

FlattenASTDB streams FCA records from a leyline-produced `_ast` database, mirroring the record shape FlattenAST builds from an in-process tree-sitter parse: one record per `_ast` row (leyline stores only named nodes — the same population FlattenAST walks) with 

{"type": node_kind, "has_<field>": true, "field_<field>_type": <child raw_kind>}

for every node_child entry carrying a tree-sitter field name. node_child includes anonymous children too (e.g. the `func` keyword, the `+` operator); fieldless ones carry a NULL field and are skipped, while anonymous children WITH a field (e.g. binary_expression's `operator`) are recorded with their raw_kind — exactly matching FlattenAST, whose child.Type() for an anonymous token is the token itself.

The parent_hash→child join is content-level: identical subtrees share a node_hash, so each `_ast` occurrence of a duplicated subtree re-joins the same children and produces its own record — matching FlattenAST emitting one record per occurrence.

sourceLike filters `_ast.source_id` with SQL LIKE (empty = all sources); limit bounds the number of RECORDS returned (0 = unlimited). Ordering is deterministic document order (source, then position, pre-order on ties).

Caveat: language-specific enrichment (lang.Language.EnrichNode — currently only HCL/Terraform's enrichHCLNode, which is sitter-typed) is NOT replicated here. HCL inference via `_ast` therefore lacks that enrichment until it is ported; Go and every other language are unaffected.

func GetGitHints added in v0.2.0 

func GetGitHints() map[string]string

GetGitHints returns the default inference hints for Git repositories.

func LoadFileIndex added in v0.6.1 

func LoadFileIndex(dbPath string) (map[string]FileIndexEntry, error)

LoadFileIndex reads the file_index table from an existing index database. Returns a map of path → (modTime, size) for incremental comparison.

func LoadGitCommits added in v0.2.0 

func LoadGitCommits(repoPath string) ([]any, error)

LoadGitCommits loads all commits from a repository using git log.

func LoadGitignore added in v0.6.3 

func LoadGitignore(rootDir string) *gitignoreMatcher

LoadGitignore reads .gitignore from rootDir and discovers nested .gitignore files in the tree. Returns nil if no .gitignore exists at all.

func LoadIndexCoverage added in v0.8.0 

func LoadIndexCoverage(dbPath string) (map[string][]CoverageEntry, error)

LoadIndexCoverage reads the _index_coverage table from an existing index database. Returns a map keyed by source_id whose values are the per-producer entries that touched that source. Returns (nil, nil) when the table doesn't exist — the caller treats that as "no coverage info; assume any absence is unknown."

func LoadSQLite 

func LoadSQLite(dbPath string) ([]any, error)

LoadSQLite opens a SQLite database, reads all records from the results table, parses each JSON record, and returns them as a slice. Kept for backward compatibility with tests; prefer StreamSQLite for large datasets.

func ParseSize added in v0.5.6 

func ParseSize(s string) (int64, error)

ParseSize parses a human-readable size string (e.g. "100MB", "1GB", "0"). Returns bytes. Supported suffixes: KB, MB, GB (case-insensitive).

func RegisterASTCallPatterns added in v0.8.0 

func RegisterASTCallPatterns(langName string, patterns []CallPattern)

RegisterASTCallPatterns registers per-language structured call patterns for the batched ExtractCalls / ExtractQualifiedCalls fast path.

func RegisterASTContextKinds added in v0.8.0 

func RegisterASTContextKinds(langName string, kinds []string)

RegisterASTContextKinds registers the top-level node kinds whose source bytes constitute the context blob for the given language.

func RegisterASTFileLevelRefPatterns added in v0.10.0 

func RegisterASTFileLevelRefPatterns(langName string, patterns []CallPattern)

RegisterASTFileLevelRefPatterns registers the per-language file-level ref capture shapes for ExtractFileLevelRefs.

func RegisterAddressRefQuery added in v0.6.2 

func RegisterAddressRefQuery(langName, scheme, query string)

RegisterAddressRefQuery registers an address-aware ref extraction query for a specific language. The query must capture values as @ref. When matched, captured strings are unquoted (if quoted) and prefixed with scheme + ":" before being emitted as ref tokens.

Multiple queries can be registered per language by calling this function multiple times; entries are appended.

func RenderTemplate 

func RenderTemplate(tmpl string, values map[string]any) (string, error)

RenderTemplate delegates to internal/template.Render. Kept as a public alias for backward compatibility with existing callers.

func RenderTemplateWithFuncs added in v0.6.2 

func RenderTemplateWithFuncs(tmpl string, values map[string]any, extraFuncs template.FuncMap, cache *sync.Map) (string, error)

RenderTemplateWithFuncs delegates to internal/template.RenderWithFuncs.

func SchemaUsesTreeSitter added in v0.2.0 

func SchemaUsesTreeSitter(schema *api.Topology) bool

SchemaUsesTreeSitter returns true if the schema's selectors are tree-sitter S-expressions rather than JSONPath. S-expressions always start with '('.

func ShouldSkipDir added in v0.6.0 

func ShouldSkipDir(base string) bool

ShouldSkipDir returns true for hidden dirs and common build artifact directories.

func ShouldSkipFile added in v0.5.6 

func ShouldSkipFile(path string, size int64) bool

ShouldSkipFile returns true if the file should not be ingested. Checks extension blocklist, size limit, and binary content.

func StreamSQLite 

func StreamSQLite(dbPath string, fn func(recordID string, record any) error) error

StreamSQLite iterates over all records in a SQLite database, calling fn for each one. Only one parsed record is alive at a time, keeping memory usage constant.

func StreamSQLiteRaw 

func StreamSQLiteRaw(dbPath string, fn func(id, raw string) error) error

StreamSQLiteRaw iterates over all records yielding raw (id, json) strings without parsing. Used by the parallel ingestion pipeline where workers handle JSON parsing on their own goroutines.

func TuneReadConnForBuild added in v0.18.0 

func TuneReadConnForBuild(db *sql.DB)

TuneReadConnForBuild applies aggressive read tuning that is ONLY safe when the caller exclusively owns db — i.e. a one-shot `mache build` over a private temp _ast db that ley-line already closed. It MUST NOT be called on a served/mounted or otherwise shared handle: SetMaxOpenConns(1) clobbers the SQLiteGraph's own pool and locking_mode=EXCLUSIVE holds a POSIX file lock for the connection's life, blocking every other reader/writer of that file (mache-010123).

What it buys (mache-4f3840, whole-repo build): EXCLUSIVE eliminates the per-statement fcntl F_SETLK lock/unlock dance (~81% of CPU before the index fix cut query count); mmap turns page reads into memory-mapped accesses instead of pread syscalls; the large cache keeps the working set resident. Best-effort — a failed pragma degrades to slower-but-correct.

Types

type ASTRoot added in v0.6.7 

type ASTRoot struct {
	DB           *sql.DB
	SourceID     string // which source file (key into _source)
	ParentPrefix string // scope queries to children under this prefix
}

ASTRoot is the root context for ASTWalker queries. It scopes queries to a subtree of the AST via the parentPrefix.

type ASTScope added in v0.18.0 

type ASTScope interface {
	// ASTSourceID returns the real `_ast`/`_source` key for this match's file.
	ASTSourceID() string
	// ASTScopeID returns the `_ast` scope node id (ctx.ParentPrefix) this
	// construct's calls are constrained to.
	ASTScopeID() string
}

ASTScope is implemented by matches backed by a pre-parsed `_ast` table (astMatch) to expose the two identifiers a serve-time scoped+qualified callee query needs: the real `_ast`/`_source` key for the file, and the scope node id under which calls should be constrained (the same SourceID/ParentPrefix that ScopeCalls itself uses).

The graph node id assigned to a construct (e.g. "cmd/functions/evalOrAbs") is NEITHER of these — it's a schema-rendered path, not an _ast key. Feeding it as source_id/scope id produces zero `_ast` rows every time (bead mache-fd9982 root cause: find_callees was silently broken on the serve/mount path). The engine persists these two identifiers onto the construct's graph node (Properties "ast_source_id"/"ast_scope_id") so GetCallees can recover them without re-deriving anything from the node id.

type ASTWalker added in v0.6.7 

type ASTWalker struct {
	// contains filtered or unexported fields
}

ASTWalker implements Walker by querying _ast and nodes tables produced by ley-line's ll-open/ts crate. This eliminates the CGO dependency on tree-sitter Go bindings — the AST was already parsed by Rust and stored in SQLite. Mache reads it via sqlite3_deserialize (zero-copy).

See ADR-014 for the design rationale.

func NewASTWalker added in v0.6.7 

func NewASTWalker(db *sql.DB) *ASTWalker

NewASTWalker creates a walker backed by a SQLite database containing ley-line's _ast, _source, and nodes tables.

It does NOT mutate the connection: the walker may run against a shared, long-lived, SERVED database (serve/mount wire it onto a SQLiteGraph via pickCallExtractor), where changing the pool size or holding a file lock for the daemon's lifetime is harmful (mache-010123). Read-perf tuning that is only safe when mache exclusively OWNS the db (a one-shot build's temp _ast db) lives in TuneReadConnForBuild, which the build path opts into. The big projection speedup — the per-file in-memory node index (mache-4f3840) — is connection-count-agnostic and applies here regardless.

func (*ASTWalker) Close added in v0.6.7 

func (w *ASTWalker) Close()

Close is a no-op — the ASTWalker doesn't own the database connection.

func (*ASTWalker) EnsureIndexes added in v0.6.8 

func (w *ASTWalker) EnsureIndexes() error

EnsureIndexes creates compound indexes on the _ast table for query performance. Call once after opening the DB, before concurrent queries. Transforms findNodesByKind from O(N) full table scan to O(K) index lookup. Returns an error if the index cannot be created (e.g., no _ast table, read-only DB, or connection pool exhausted).

func (*ASTWalker) ExtractAddressRefs added in v0.6.7 

func (w *ASTWalker) ExtractAddressRefs(sourceID, langName string) ([]string, error)

ExtractAddressRefs runs all registered address ref queries for sourceID and language by querying the _ast table (whole file). sourceID is leyline's root-relative _source.id (for example "sub/nested.go"), not an arbitrary OS path. Returns deduplicated, scheme-prefixed tokens such as "env:DATABASE_URL".

func (*ASTWalker) ExtractAddressRefsScoped added in v0.10.0 

func (w *ASTWalker) ExtractAddressRefsScoped(sourceID, scopeID, langName string) ([]string, error)

ExtractAddressRefsScoped is the per-construct equivalent: address refs whose matched node lives under scopeID's id-path. Mirrors SitterWalker's per-scope ExtractAddressRefs(scopeNode, ...). sourceID is the _source key.

It filters the whole-file address-ref set (computed once and cached per file) by node-id prefix in Go rather than re-running the generic Query per construct. The old per-construct path re-scanned every call_expression in every function — 84% of a whole-repo projection's CPU (mache-4f3840). The prefix filter (scopeID+"/") is byte-identical to the SQL `n.id LIKE scopeID || '/%'` the per-construct Query used, so the projected refs are unchanged.

func (*ASTWalker) ExtractCalls added in v0.8.0 

func (w *ASTWalker) ExtractCalls(sourcePath, langName string) ([]string, error)

ExtractCalls returns deduplicated function-call tokens for the given source file, evaluating every registered pattern over the file's node section — one statement to load it, none per pattern or per scope.

Returns (nil, nil) when the language has no registered patterns — the caller treats that as "no calls" and falls through to other paths.

func (*ASTWalker) ExtractCallsScoped added in v0.10.0 

func (w *ASTWalker) ExtractCallsScoped(sourceID, scopeID, langName string) ([]string, error)

ExtractCallsScoped is the per-construct equivalent of ExtractCalls: it returns only call tokens whose leaf node lives under scopeID's id-path — matching SitterWalker's per-scope ExtractCalls(scopeNode, ...). sourceID is the _source key (filepath.Base of the file). Empty scopeID would match the whole file; callers pass a real construct id.

func (*ASTWalker) ExtractContext added in v0.8.0 

func (w *ASTWalker) ExtractContext(sourceID, langName string) ([]byte, error)

ExtractContext returns the concatenated source text of the top-level context nodes for the given file (e.g. Go's import/const/var/type declarations). Used by schema context fields. Mirrors SitterWalker.ExtractContext but slices the file's section: the node kinds from its index, the bytes from its _source row.

Returns (nil, nil) when no context kinds are registered for the language, and (nil, err) when the file's source cannot be read.

sourceID is the _source/_ast key (the path relative to the ingest root, as ley-line produces it) — NOT a filesystem path. Callers that hold a path use Engine.sourceIDFor to derive it (mache-30edfa).

func (*ASTWalker) ExtractFileLevelRefs added in v0.10.0 

func (w *ASTWalker) ExtractFileLevelRefs(sourceID, langName string) ([]string, error)

ExtractFileLevelRefs returns deduplicated identifier tokens captured by the language's file-level ref patterns across the WHOLE file. Mirrors SitterWalker.ExtractFileLevelRefs (the _file_level: sentinel feed that dead_code reads) but queries _ast/nodes via SQL instead of CGO tree-sitter. sourceID is the _source key (filepath.Base of the file). Returns (nil, nil) when no patterns are registered for the language.

func (*ASTWalker) ExtractGoImports added in v0.7.0 

func (w *ASTWalker) ExtractGoImports(sourceID string) (map[string]string, error)

ExtractGoImports reads Go import aliases from the _imports table (produced by ley-line-open's `leyline parse`). Returns alias → path map for qualified call resolution (e.g., auth.Validate → github.com/foo/auth). Mirrors SitterWalker.ExtractGoImports but uses SQL instead of CGO tree-sitter.

Older .dbs produced before LLO grew _imports have no such table; that is probed once per walker (not once per file), and resolves to no imports.

func (*ASTWalker) ExtractQualifiedCalls added in v0.8.0 

func (w *ASTWalker) ExtractQualifiedCalls(sourcePath, langName string) ([]graph.QualifiedCall, error)

ExtractQualifiedCalls returns call tokens with optional package qualifiers. Patterns whose QualifierKind is non-empty produce QualifiedCall with both fields; bare patterns produce QualifiedCall with empty Qualifier.

func (*ASTWalker) ExtractQualifiedCallsScoped added in v0.18.0 

func (w *ASTWalker) ExtractQualifiedCallsScoped(sourceID, scopeID, langName string) ([]graph.QualifiedCall, error)

ExtractQualifiedCallsScoped is the per-construct equivalent of ExtractQualifiedCalls: it returns qualified call tokens whose leaf node lives under scopeID's id-path, instead of the whole file. sourceID is the real `_ast`/`_source` key (e.g. "agent.go"), NOT a graph node id — feeding a graph node id here (e.g. "cmd/functions/evalOrAbs") matches zero `_ast` rows, which was the root cause of find_callees silently returning nothing on the serve/mount path (bead mache-fd9982). Callers recover the correct (sourceID, scopeID) pair from the construct's graph node Properties ("ast_source_id"/"ast_scope_id"), persisted at projection time by engine_walk.go via the ASTScope interface. Empty scopeID matches the whole file, mirroring ExtractQualifiedCalls.

func (*ASTWalker) InvalidateSource added in v0.18.0 

func (w *ASTWalker) InvalidateSource(sourceID string)

InvalidateSource drops every per-file cache entry for sourceID so a subsequent query re-reads that file from the db. The mount/serve watcher calls this when a source file changes (mache-018eee) — without it, ReIngestFile would re-project from the walker's immortal caches and never see the edit.

It also bounds cache growth: the per-file caches (indexCache/addrRefCache/callTokenCache) otherwise accumulate O(repo) node rows + source bytes for the walker's lifetime. A serve/mount daemon evicts per-file on change here (mache-024e9c), and the engine evicts each file as soon as its projection is written (processSourceFileResult) — a one-shot build walker is short-lived, but for that lifetime it held every file's full index at once, which was the build's peak RSS, not a rounding error (mache-95a33d).

func (*ASTWalker) Query added in v0.6.7 

func (w *ASTWalker) Query(root any, selector string) ([]Match, error)

Query implements Walker. The selector is a tree-sitter S-expression pattern, evaluated against the file's in-memory section of the nodes⋈_ast tables.

Supports the common pattern: (node_kind field: (child_kind) @capture) @scope, plus simple #eq? / #match? predicates over captured text. Field labels constrain like tree-sitter's: a labelled step reaches only the child under that field, an unlabelled step any child of the kind (mache-91d903).

type CallExtractor added in v0.10.0 

type CallExtractor interface {
	// ScopeCalls returns deduplicated call tokens + typed address-refs found
	// within this match's scope. Per-construct, not whole-file.
	ScopeCalls() []string
}

CallExtractor is implemented by matches that can extract the call tokens and address-refs WITHIN their @scope (per construct) for the callers/callees refs index. Since ADR-0012 step 4 removed in-process tree-sitter, astMatch is the sole implementer — it runs scope-prefixed SQL over _ast — and parentAwareMatch forwards. The interface is retained so the engine stays walker-agnostic. Correctness is covered by `task test:ast`.

type CallPattern added in v0.8.0 

type CallPattern struct {
	OuterKind     string   // e.g. "call_expression"
	Ancestors     []string // intermediate kinds between outer and leaf
	LeafKind      string   // node kind whose record is the call name (@call)
	QualifierKind string   // optional sibling-leaf kind for the package qualifier (@pkg); empty when not qualified
	// RequirePriorSibling, when true, restricts the match to leaves whose
	// immediate ancestor (Ancestors[0], the direct child of OuterKind) has
	// an EARLIER same-kind sibling under OuterKind. This reproduces
	// tree-sitter's positional matching for value-position captures like
	//   (keyed_element (literal_element) (literal_element (identifier) @x))
	// where only the SECOND literal_element (the value) is captured, not
	// the first (the key). Requires len(Ancestors) >= 1.
	RequirePriorSibling bool
}

CallPattern describes one shape of function call for a language. The walker evaluates these over the file's in-memory node section (one ancestor-chain check per candidate leaf) — no SQL per pattern, and none of the N-queries-per-file pattern of the generic selector path.

Examples: 

Go bare:       OuterKind=call_expression, LeafKind=identifier
Go qualified:  OuterKind=call_expression, Ancestors=[selector_expression],
               LeafKind=field_identifier, QualifierKind=identifier

type CoverageEntry added in v0.8.0 

type CoverageEntry struct {
	Producer  string    // 'tree-sitter' | 'lsp' | future
	Fidelity  string    // 'mention' | 'binding' | 'reachability'
	IndexedAt time.Time // when the indexer wrote the row
	Complete  bool      // indexer claims full coverage of source_id at this fidelity
}

CoverageEntry is one (source_id, producer) row from _index_coverage. A consumer that needs to distinguish "no binding row exists for this source" from "no producer at the binding level ever indexed this source" joins against this — see ADR-0013 wedge case 1.

type DocScope added in v0.10.0 

type DocScope interface {
	// ScopeSource returns the file's source bytes (for line/text computation).
	ScopeSource() []byte
	// DocRange returns the doc-extended start, the scope start, and the scope
	// end byte offsets. docStart == scopeStart when there are no contiguous
	// comment siblings. ok is false when the match has no @scope (e.g. a "$"
	// grouping match).
	DocRange() (docStart, scopeStart, scopeEnd uint32, ok bool)
}

DocScope is implemented by matches that can locate their @scope node's byte range, extended backward over contiguous preceding comment siblings (doc comments). The engine reads the `location` node property — and doc-comment text — through this interface so it stays walker-agnostic. astMatch (queries the _ast comment rows) is the sole implementer since in-process tree-sitter was removed; correctness is covered by `task test:ast`.

type Engine 

type Engine struct {
	Schema           *api.Topology
	Store            IngestionTarget
	RootPath         string // absolute path to the root of the ingestion
	RespectGitignore bool   // when true, skip files matching .gitignore patterns (default: true)
	// contains filtered or unexported fields
}

Engine drives the ingestion process.

func NewEngine 

func NewEngine(schema *api.Topology, store IngestionTarget) *Engine

func (*Engine) DiagramFuncMap added in v0.6.2 

func (e *Engine) DiagramFuncMap() template.FuncMap

DiagramFuncMap returns a template.FuncMap containing the {{diagram "name"}} function. The returned FuncMap is built once via sync.Once and reused; the closure inside captures the Engine and lazily initializes community data on first call. Safe for concurrent use.

func (*Engine) Gitignore added in v0.6.3 

func (e *Engine) Gitignore() GitignoreMatcher

Gitignore returns the gitignore matcher loaded during Ingest, or nil if none was loaded. Pass this to WithGitignore when creating a Watcher so the watcher skips the same directories the engine does.

func (*Engine) Ingest 

func (e *Engine) Ingest(path string) error

Ingest processes a file or directory. Safe to call multiple times — internal dedup state is reset on each call.

func (*Engine) IngestRecords added in v0.2.0 

func (e *Engine) IngestRecords(records []any) error

IngestRecords processes in-memory records (e.g. from Git).

func (*Engine) PrintRoutingSummary added in v0.2.0 

func (e *Engine) PrintRoutingSummary()

PrintRoutingSummary outputs a summary of files routed to _project_files/.

func (*Engine) ReIngestFile added in v0.6.0 

func (e *Engine) ReIngestFile(path string) error

ReIngestFile re-ingests a single file, preserving the existing RootPath. Used by the live graph refresher to update stale nodes without a full walk. After re-ingestion, the store's file mtime is updated.

func (*Engine) RenderContentTemplate added in v0.6.2 

func (e *Engine) RenderContentTemplate(tmpl string, values map[string]any) (string, error)

RenderContentTemplate renders a content template with the standard mache functions plus the Engine's diagram function. This is the method that processNode and collectNodes should use for file content rendering.

func (*Engine) SetASTWalker added in v0.6.7 

func (e *Engine) SetASTWalker(w *ASTWalker)

SetASTWalker configures the engine's SQL-backed ASTWalker — the sole walker for source (tree-sitter S-expression) schemas since ADR-0012 step 4 removed in-process CGO tree-sitter. The ASTWalker queries pre-parsed _ast/_source tables from a ley-line .db, so no source parsing happens in mache.

func (*Engine) SetFileIndex added in v0.6.1 

func (e *Engine) SetFileIndex(index map[string]FileIndexEntry)

SetFileIndex sets a cached file index for incremental re-ingestion. Files matching (path, mtime, size) will be skipped during ingestion.

type FileIndexEntry added in v0.6.1 

type FileIndexEntry struct {
	ModTime time.Time
	Size    int64
	// ClaimedIDs are the construct node IDs this file owned in the previous
	// build. An incremental pass SKIPS an unchanged file, so nothing
	// re-claims its IDs — and claimConstructID starts from an empty map on
	// every Ingest, so a CHANGED file rendering the same construct name would
	// take the bare ID the skipped file still holds, overwriting it
	// (mache-7a7919). Seeding from here is what keeps the assignment the
	// same as a full ingest's.
	ClaimedIDs []string
}

FileIndexEntry stores cached file metadata for incremental comparison.

type FileMeta added in v0.10.0 

type FileMeta interface {
	// Lang is the source language ("go", "python", ...), or "" if unknown.
	Lang() string
	// PackageName is the file's package/namespace (Go package, etc.), or ""
	// when the language has no such concept or it can't be determined.
	PackageName() string
}

FileMeta is an optional interface a Match implements to expose per-file metadata the engine records as node Properties (lang, pkg). Reading it through this interface lets the engine stay walker-agnostic. Since ADR-0012 step 4 removed in-process tree-sitter, astMatch (SQL over _ast) is the sole implementer; correctness is covered by `task test:ast`.

type GitignoreMatcher added in v0.6.3 

type GitignoreMatcher interface {
	Match(rel string, isDir bool) bool
}

GitignoreMatcher matches paths against .gitignore-style rules.

type IngestionTarget 

type IngestionTarget interface {
	graph.Graph
	AddNode(n *graph.Node)
	AddRoot(n *graph.Node)
	AddRef(token, nodeID string) error
	AddDef(token, dirID string) error
	DeleteFileNodes(filePath string)
	// DeleteNodes removes specific nodes by ID. Needed because a construct
	// DIRECTORY carries no file association — only its source leaf does — so
	// DeleteFileNodes can never reach one (mache-399c25).
	DeleteNodes(ids []string)
	// ForgetFile drops a path from whatever index records "this build
	// projected this file", so a deleted file stops being treated as
	// unchanged (mache-31abc0).
	ForgetFile(path string)
	AddFileChildren(parent *graph.Node, files []*graph.Node)
}

IngestionTarget combines Graph reading with writing capabilities.

type JsonWalker 

type JsonWalker struct{}

JsonWalker implements Walker for JSON-like data.

func NewJsonWalker 

func NewJsonWalker() *JsonWalker

func (*JsonWalker) Query 

func (w *JsonWalker) Query(root any, selector string) ([]Match, error)

Query implements Walker.

type Match 

type Match interface {
	// Values returns the captured values.
	// For Tree-sitter, these are the named captures from the query (e.g., "res.type" -> "aws_s3_bucket").
	// For JSONPath, if the match is an object, its fields are returned as values.
	// If the match is a primitive, it might be returned under a default key (e.g., "value").
	Values() map[string]any

	// Context returns the underlying object/node to be used as the root for child queries.
	// For JSONPath, this is the matched object.
	// For Tree-sitter, this is the node captured as @scope (or similar convention).
	Context() any
}

Match represents a single result from a query. It provides a map of values that can be used to render path templates.

type OriginProvider 

type OriginProvider interface {
	CaptureOrigin(name string) (startByte, endByte uint32, ok bool)
}

OriginProvider is an optional interface that Match implementations can satisfy to expose source byte ranges for write-back. Type-asserted in engine, not required by JSON walker.

type SQLiteWriter added in v0.2.0 

type SQLiteWriter struct {
	// contains filtered or unexported fields
}

SQLiteWriter implements IngestionTarget for the new high-performance schema.

func NewSQLiteWriter added in v0.2.0 

func NewSQLiteWriter(dbPath string) (*SQLiteWriter, error)

NewSQLiteWriter creates a new writer and initializes the schema.

func (*SQLiteWriter) Act added in v0.5.0 

func (w *SQLiteWriter) Act(id, action, payload string) (*graph.ActionResult, error)

func (*SQLiteWriter) AddDef added in v0.2.0 

func (w *SQLiteWriter) AddDef(token, dirID string) error

func (*SQLiteWriter) AddFileChildren added in v0.6.8 

func (w *SQLiteWriter) AddFileChildren(parent *graph.Node, files []*graph.Node)

func (*SQLiteWriter) AddNode added in v0.2.0 

func (w *SQLiteWriter) AddNode(n *graph.Node)

AddNode writes a node to the database.

func (*SQLiteWriter) AddRef added in v0.2.0 

func (w *SQLiteWriter) AddRef(token, nodeID string) error

func (*SQLiteWriter) AddRoot added in v0.2.0 

func (w *SQLiteWriter) AddRoot(n *graph.Node)

func (*SQLiteWriter) Close added in v0.2.0 

func (w *SQLiteWriter) Close() error

func (*SQLiteWriter) DeleteFileNodes added in v0.2.0 

func (w *SQLiteWriter) DeleteFileNodes(filePath string)

DeleteFileNodes removes every node that originated from filePath, with its refs and defs. Every statement is index-driven (idx_source_file on the subquery, idx_refs_node / idx_defs_node on the outer delete), so the cost is proportional to the file's own rows, not to the size of the graph.

func (*SQLiteWriter) DeleteNodes added in v0.22.0 

func (w *SQLiteWriter) DeleteNodes(ids []string)

func (*SQLiteWriter) ForgetFile added in v0.22.0 

func (w *SQLiteWriter) ForgetFile(path string)

DeleteNodes removes the named nodes with their refs and defs, whatever file they came from.

DeleteFileNodes keys on source_file, which a CONSTRUCT DIRECTORY does not carry — only its source leaf does — so it cannot reach one (mache-399c25). One statement per id rather than a generated IN-list: the caller passes the construct IDs of a single re-ingested file, so the count is small and bounded by that file's constructs, and a fixed statement keeps the query plan stable the way deleteFileNodesSQL does. ForgetFile drops path's file_index row, so a later build does not treat a file that is gone as merely unchanged. Without it the path stays "known" forever and every subsequent build repeats the same decision about it (mache-31abc0).

func (*SQLiteWriter) GetCallees added in v0.2.0 

func (w *SQLiteWriter) GetCallees(id string) ([]*graph.Node, error)

func (*SQLiteWriter) GetCallers added in v0.2.0 

func (w *SQLiteWriter) GetCallers(token string) ([]*graph.Node, error)

func (*SQLiteWriter) GetNode added in v0.2.0 

func (w *SQLiteWriter) GetNode(id string) (*graph.Node, error)

func (*SQLiteWriter) Invalidate added in v0.2.0 

func (w *SQLiteWriter) Invalidate(id string)

func (*SQLiteWriter) ListChildStats added in v0.6.8 

func (w *SQLiteWriter) ListChildStats(id string) ([]graph.NodeStat, error)

func (*SQLiteWriter) ListChildren added in v0.2.0 

func (w *SQLiteWriter) ListChildren(id string) ([]string, error)

ListChildren answers from the parent_id column, the same way NodesTableReader does at read time.

This returned `nil, nil // Not used during ingest` for a long time, and that stub read like a CONSTRAINT — you cannot ask for children while still writing them. It is not one. Rows INSERTed in an open transaction are readable inside that same transaction (probed: 3 inserted, 2 children visible before commit), and this writer already holds the tx and already writes parent_id on every node.

The stub had a cost. Because neither accessor answered, engine code that needed a node's children read GetNode().Children instead — which MemoryStore populates and every SQLite-backed store leaves nil. A dedup guard branched on exactly that, was live on one backend and dead on the other, and silently dropped 540 constructs (mache-c725e9). Making this method honest is what lets children have ONE accessor that is correct everywhere (mache-e3d9bb).

Note the tx caveat: file nodes are buffered by bufferingTarget for a later ReplaceFileNodes swap, so they are not in the table yet. bufferingTarget overrides this method to union its buffer in; callers that go through it get the complete answer.

func (*SQLiteWriter) ReadContent added in v0.2.0 

func (w *SQLiteWriter) ReadContent(id string, buf []byte, offset int64) (int, error)

func (*SQLiteWriter) RecordFile added in v0.6.1 

func (w *SQLiteWriter) RecordFile(path string, modTime time.Time, size int64)

RecordFile stores file metadata for incremental re-ingestion. On subsequent mounts, files with matching (path, mod_time, size) are skipped.

func (*SQLiteWriter) RecordIndexCoverage added in v0.8.0 

func (w *SQLiteWriter) RecordIndexCoverage(sourceID, producer, fidelity string, indexedAt time.Time, complete bool)

RecordIndexCoverage stores a coverage row for a (source_id, producer) pair, per ADR-0013 Step 2. The fidelity argument is one of "mention", "binding", or "reachability"; complete=true means the indexer claims full coverage of source_id at that fidelity. A re-index replaces the prior row for the same (source_id, producer) — different producers keep distinct rows.

type Walker 

type Walker interface {
	// Query executes a selector (query) against the given root node and returns a list of matches.
	// The root node can be a *sitter.Node (for code) or a generic Go object (for data).
	Query(root any, selector string) ([]Match, error)
}

Walker abstracts over JSONPath (Data) and Tree-sitter (Code). It provides a unified way to query a tree-like structure and extract values for path templating.

func SelectWalker added in v0.6.7 

func SelectWalker(db *sql.DB) (Walker, error)

SelectWalker inspects a SQLite database and returns the ASTWalker when the database has an `_ast` table (produced by ley-line's ll-open/ts). Since ADR-0012 step 4 removed in-process CGO tree-sitter, a database WITHOUT an `_ast` table is an error — there is no fallback walker. Callers must parse source through ley-line first (see runBuildViaLeylineSchema / autoInvokeLeylineParse).

type Watcher added in v0.6.1 

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher monitors a directory tree for file changes and invokes callbacks when source files are created/modified or deleted. It debounces rapid changes so that a burst of writes to the same file produces a single callback after a quiet period.

func NewWatcher added in v0.6.1 

func NewWatcher(rootDir string, onChange, onDelete func(path string), opts ...WatcherOption) (*Watcher, error)

NewWatcher creates a file watcher on rootDir. onChange is called for created/modified files; onDelete is called for removed files. Both callbacks receive the absolute file path. Hidden files, .git directories, and non-source extensions are ignored.

func (*Watcher) Stop added in v0.6.1 

func (w *Watcher) Stop()

Stop shuts down the watcher. Safe to call concurrently and multiple times.

type WatcherOption added in v0.6.1 

type WatcherOption func(*Watcher)

WatcherOption configures a Watcher.

func WithDebounce added in v0.6.1 

func WithDebounce(d time.Duration) WatcherOption

WithDebounce sets the quiet period before a change callback fires. Defaults to 100ms.

func WithGitignore added in v0.6.3 

func WithGitignore(gi GitignoreMatcher) WatcherOption

WithGitignore configures the watcher to skip directories matching gitignore rules. This prevents watching build artifact directories (target/, dist/, node_modules/) that would otherwise consume thousands of kqueue FDs on macOS.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.