ingest

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 13, 2026 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FlattenAST added in v0.1.1

func FlattenAST(root *sitter.Node) []any

FlattenAST walks the tree and returns a list of records for FCA analysis.

func LoadSQLite

func LoadSQLite(dbPath string) ([]any, error)

LoadSQLite opens a SQLite database, reads all records from the results table, parses each JSON record, and returns them as a slice. Kept for backward compatibility with tests; prefer StreamSQLite for large datasets.

func RenderTemplate

func RenderTemplate(tmpl string, values map[string]any) (string, error)

RenderTemplate renders a Go text/template with the standard mache template functions. Parsed templates are cached — repeated calls with the same template string skip parsing.

func StreamSQLite

func StreamSQLite(dbPath string, fn func(recordID string, record any) error) error

StreamSQLite iterates over all records in a SQLite database, calling fn for each one. Only one parsed record is alive at a time, keeping memory usage constant.

func StreamSQLiteRaw

func StreamSQLiteRaw(dbPath string, fn func(id, raw string) error) error

StreamSQLiteRaw iterates over all records yielding raw (id, json) strings without parsing. Used by the parallel ingestion pipeline where workers handle JSON parsing on their own goroutines.

Types

type Engine

type Engine struct {
	Schema   *api.Topology
	Store    IngestionTarget
	RootPath string // absolute path to the root of the ingestion
	// contains filtered or unexported fields
}

Engine drives the ingestion process.

func NewEngine

func NewEngine(schema *api.Topology, store IngestionTarget) *Engine

func (*Engine) Ingest

func (e *Engine) Ingest(path string) error

Ingest processes a file or directory.

type IngestionTarget

type IngestionTarget interface {
	graph.Graph
	AddNode(n *graph.Node)
	AddRoot(n *graph.Node)
	AddRef(token, nodeID string) error
}

IngestionTarget combines Graph reading with writing capabilities.

type JsonWalker

type JsonWalker struct{}

JsonWalker implements Walker for JSON-like data.

func NewJsonWalker

func NewJsonWalker() *JsonWalker

func (*JsonWalker) Query

func (w *JsonWalker) Query(root any, selector string) ([]Match, error)

Query implements Walker.

type Match

type Match interface {
	// Values returns the captured values.
	// For Tree-sitter, these are the named captures from the query (e.g., "res.type" -> "aws_s3_bucket").
	// For JSONPath, if the match is an object, its fields are returned as values.
	// If the match is a primitive, it might be returned under a default key (e.g., "value").
	Values() map[string]any

	// Context returns the underlying object/node to be used as the root for child queries.
	// For JSONPath, this is the matched object.
	// For Tree-sitter, this is the node captured as @scope (or similar convention).
	Context() any
}

Match represents a single result from a query. It provides a map of values that can be used to render path templates.

type OriginProvider

type OriginProvider interface {
	CaptureOrigin(name string) (startByte, endByte uint32, ok bool)
}

OriginProvider is an optional interface that Match implementations can satisfy to expose source byte ranges for write-back. Type-asserted in engine, not required by JSON walker.

type SQLiteResolver

type SQLiteResolver struct {
	// contains filtered or unexported fields
}

SQLiteResolver resolves ContentRef entries by fetching records from SQLite and re-rendering their content templates.

func NewSQLiteResolver

func NewSQLiteResolver() *SQLiteResolver

func (*SQLiteResolver) Close

func (r *SQLiteResolver) Close()

Close closes all open database connections.

func (*SQLiteResolver) Resolve

func (r *SQLiteResolver) Resolve(ref *graph.ContentRef) ([]byte, error)

Resolve fetches a record from SQLite and renders its content template.

type SitterRoot

type SitterRoot struct {
	Node   *sitter.Node
	Source []byte
	Lang   *sitter.Language
}

SitterRoot encapsulates the necessary context for querying a Tree-sitter tree. It includes the root node, the source code (for extracting content), and the language (for compiling the query).

type SitterWalker

type SitterWalker struct {
	// contains filtered or unexported fields
}

SitterWalker implements Walker for Tree-sitter parsed code.

func NewSitterWalker

func NewSitterWalker() *SitterWalker

func (*SitterWalker) ExtractCalls

func (w *SitterWalker) ExtractCalls(root *sitter.Node, source []byte, lang *sitter.Language) ([]string, error)

ExtractCalls finds all function calls in the given node using a predefined query. The compiled query is cached per language to avoid recompilation on every call.

func (*SitterWalker) Query

func (w *SitterWalker) Query(root any, selector string) ([]Match, error)

Query implements Walker.

type Walker

type Walker interface {
	// Query executes a selector (query) against the given root node and returns a list of matches.
	// The root node can be a *sitter.Node (for code) or a generic Go object (for data).
	Query(root any, selector string) ([]Match, error)
}

Walker abstracts over JSONPath (Data) and Tree-sitter (Code). It provides a unified way to query a tree-like structure and extract values for path templating.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL