Documentation
¶
Overview ¶
Package engine orchestrates a dfetch query: parse the SQL, resolve each referenced schema to a connector, fetch and load each table into a per-request local SQLite database (pushing down as much of the query as is safe), then resolve the original query against it.
Index ¶
- type DB
- type Engine
- func (e *Engine) DescribeTable(ctx context.Context, schema, table string) (source.TableSchema, error)
- func (e *Engine) ListTables(ctx context.Context, schema, filter string) ([]string, error)
- func (e *Engine) Run(ctx context.Context, query string) (*Result, error)
- func (e *Engine) RunWithParams(ctx context.Context, query string, params map[string]any) (*Result, error)
- func (e *Engine) SchemaSummaries(ctx context.Context) []SchemaSummary
- type OpenDBFunc
- type Option
- type Result
- type SchemaSummary
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type DB ¶
type DB interface {
// Attach makes schema available so a schema-qualified table (e.g.
// github.issues) can be created under it. Called once per referenced schema
// before any CreateTable; must be idempotent.
Attach(ctx context.Context, schema string) error
// CreateTable creates an empty table matching ts under the attached schema.
CreateTable(ctx context.Context, schema string, ts source.TableSchema) error
// Insert loads one chunk of rows into a previously created table. Each row's
// values are ordered to match cols. The engine serializes Insert calls.
Insert(ctx context.Context, schema, table string, cols []string, rows [][]any) error
// Query runs the original SQL against the loaded tables, binding args (e.g.
// sql.Named values for :name parameters), and returns the column names plus
// the rows in column order.
Query(ctx context.Context, query string, args ...any) (columns []string, rows [][]any, err error)
// Close releases the database and any files backing it. Called once per run.
Close() error
}
DB is the per-request local SQL database the engine loads sources into and resolves the final query against. localdb.DB is the default implementation; supply another via WithDB to control how the underlying database files are created and managed (e.g. in-memory, a fixed path, or a shared cache).
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine resolves dfetch queries against configured connectors, keyed by the SQL schema they serve (e.g. "github").
func New ¶
New builds an Engine from the given options. Options apply in order, and the last registration of a schema name wins — so registering the default set first and a config's sources after lets config override a built-in schema.
New carries no default connectors: the connectors package provides dfetch's built-in set (connectors.DefaultOptions). An engine with no connectors is valid; every query then fails with "no connector for schema". Typed sources (WithSources/WithConfig) are built at New time via the WithRegistry registry.
func (*Engine) DescribeTable ¶
func (e *Engine) DescribeTable(ctx context.Context, schema, table string) (source.TableSchema, error)
DescribeTable returns the column schema of one table, resolving it on demand for dynamic connectors (SchemaDescriber) and from Tables() otherwise.
func (*Engine) ListTables ¶
ListTables returns the table names served under schema, filtered by a case-insensitive substring. A dynamic connector (TableLister) lists on demand; a static one lists from Tables().
func (*Engine) RunWithParams ¶
func (e *Engine) RunWithParams(ctx context.Context, query string, params map[string]any) (*Result, error)
RunWithParams executes the full pipeline for a SQL query (SQLite syntax), binding params as named SQLite parameters (referenced as :name in the SQL). A nil or empty params map runs the query with no bound parameters.
The params are also handed to the push-down planner: the planner resolves a bind-parameter RHS (e.g. `service_name = :service`) to its value so the filter can be pushed to the connector, while the final query keeps the :name bind for SQLite. Without this, connectors that require a filter value at fetch time (jaeger.spans needs service_name or trace_id, github.pulls needs owner/repo) would never see the value, since a bind is opaque until SQLite executes.
func (*Engine) SchemaSummaries ¶
func (e *Engine) SchemaSummaries(ctx context.Context) []SchemaSummary
SchemaSummaries returns one summary per connector schema (sorted), for the top-level `dfetch tables` view. A dynamic source's count comes from listing its table names; if that fails (e.g. the source is unreachable) the count is -1 rather than failing the whole listing.
type OpenDBFunc ¶
OpenDBFunc creates the per-request DB. The engine calls it once per Run and closes the returned DB when the run completes.
type Option ¶
type Option func(*settings)
Option configures the Engine built by New.
func WithConfig ¶
WithConfig registers every source declared in cfg, equivalent to WithSources(cfg.Sources...). A nil cfg is a no-op.
func WithConnector ¶
WithConnector registers a caller-built connector under a schema name, so its tables resolve as <name>.<table>. This is how a program plugs in its own Connector implementation.
func WithDB ¶
func WithDB(open OpenDBFunc) Option
WithDB sets how the per-request local database is created. The default opens localdb's temp-file SQLite database.
func WithRegistry ¶
WithRegistry merges reg's factories into the registry used to build WithSources/WithConfig entries; on a type name both define, the later WithRegistry wins. Without any WithRegistry the registry is empty, so every typed source fails at New with "unknown connector type". The connectors package provides DefaultRegistry() with every built-in type — appending WithRegistry(myReg) after connectors.DefaultOptions() adds or overrides types without losing the defaults.
func WithSources ¶
func WithSources(sources ...config.SourceConfig) Option
WithSources declares config-style sources; New builds each via the registry (see WithRegistry) from its Type and Params and registers it under its Name.
type Result ¶
Result holds the columns and rows produced by a resolved query, plus any non-fatal warnings gathered while fetching (e.g. a connector truncated at a cap, so the result may be incomplete).
func (*Result) Project ¶
Project returns a copy of the result narrowed to cols, in the given order. An empty cols list returns the result unchanged (all columns). It errors if a requested column is not present, listing the columns that are available, so a stale saved-query projection fails loudly rather than silently dropping data.
type SchemaSummary ¶
type SchemaSummary struct {
Schema string
TableCount int // number of tables, or -1 when a dynamic source couldn't be listed
Dynamic bool // true when the connector lists/describes tables on demand
}
SchemaSummary is one schema's entry in the top-level `dfetch tables` listing.
Source Files