Documentation
¶
Overview ¶
Package driver is the plugin registry. It maps a source kind or sink type to the factory that opens it, so the orchestration consumes only the source and sink contracts and never a concrete implementation. Concrete drivers register themselves from init() via a blank import or dynamically via LoadPlugin; the zero Registry is safe to use and lazy-initializes its maps.
Index ¶
- Constants
- func CapsForKind(kind string) (source.Capabilities, error)
- func LoadPlugin(path string, wrap PluginWrap) (err error)
- func OpenSink(ctx context.Context, s *spec.Spec) (sink.Sink, error)
- func OpenSinkConfig(ctx context.Context, cfg sink.Config) (sink.Sink, error)
- func OpenSource(s *spec.Spec, rt source.Runtime) (source.Source, error)
- func RegisterSink(scheme string, factory SinkFactory) error
- func RegisterSource(kind string, caps source.Capabilities, factory SourceFactory) error
- func SinkConfig(s *spec.Spec) sink.Config
- func SinkTypeExists(scheme string) bool
- func ValidateParallelism(kind string, maxParallelChunks int) error
- type PluginWrap
- type SinkFactory
- type SourceFactory
Constants ¶
const ( OptWarehouse = "warehouse" OptClientID = "client_id" OptClientSecret = "client_secret" OptScope = "scope" OptCommitMode = "commit_mode" OptTargetFileSize = "target_file_size" OptEvolveSchema = "evolve_schema" )
Sink option keys — the neutral-config contract between SinkConfig and each sink factory. Named constants so a typo is a compile error, not a silent "" read. client_secret is a credential: never log the Options map.
const DefaultSinkType = "iceberg+rest"
DefaultSinkType is the sink type used when a config does not declare one. It is the only sink type today; a future non-Iceberg sink registers its own type and the spec names it explicitly.
Variables ¶
This section is empty.
Functions ¶
func CapsForKind ¶
func CapsForKind(kind string) (source.Capabilities, error)
CapsForKind returns a source kind's capabilities without a full spec. Used by admission validation, which must check resource ceilings before boot.
func LoadPlugin ¶
func LoadPlugin(path string, wrap PluginWrap) (err error)
LoadPlugin opens a Go plugin (.so) and calls its exported Init function. The plugin must export:
func Init() error
Init is responsible for calling RegisterSource/RegisterSink exactly as a built-in driver would. If wrap is non-nil, LoadPlugin then replaces every source/sink kind the plugin newly registered with wrap's wrapped form — see PluginWrap. A nil wrap registers the plugin's factories unwrapped (test-only; production callers always pass a wrap so a .so's driver is actually served over Arrow Flight, never called in-process by contract).
Rules: same Go version, same dependency graph, Linux/macOS only.
func OpenSinkConfig ¶
OpenSinkConfig resolves and instantiates a sink from a neutral config.
func OpenSource ¶
OpenSource resolves and instantiates a source for a spec's source kind.
func RegisterSink ¶
func RegisterSink(scheme string, factory SinkFactory) error
RegisterSink registers a sink type (iceberg+rest, delta, …). Called from init() in each sink package. A duplicate or empty type is an error (see RegisterSource).
func RegisterSource ¶
func RegisterSource(kind string, caps source.Capabilities, factory SourceFactory) error
RegisterSource registers a source kind (mysql, postgres, kafka) with its static capabilities. Called from init() in each source package.
A duplicate registration is an error: the last-write-wins behavior would let a plugin silently hijack a builtin kind and redirect every pipeline that names it. An empty kind is an error too. Callers in init() panic on error — a duplicate at boot is a programming error, not a runtime choice.
func SinkConfig ¶
SinkConfig renders a spec's sink section into the neutral config.
func SinkTypeExists ¶
SinkTypeExists reports whether a sink type is registered. An empty type resolves to the default sink. Used by admission validation.
func ValidateParallelism ¶
ValidateParallelism rejects a parallel-chunk setting above the ceiling the source driver declares: a shared MySQL with max_connections=100 does not tolerate the same snapshot concurrency as a dedicated Postgres. A ceiling of 0 means the driver has no opinion.
Types ¶
type PluginWrap ¶
type PluginWrap interface {
WrapSource(path string, real SourceFactory) SourceFactory
WrapSink(path string, real SinkFactory) SinkFactory
}
PluginWrap turns a .so's raw registered factories into the actually-served form. LoadPlugin calls it (when non-nil) on every source/sink kind a plugin's Init newly registered, so a .so author's RegisterSource/ RegisterSink code never has to know or care what happens after — today that means wrapping the real driver in an in-process Arrow Flight server and handing back a Flight client adapter (internal/plugin/flightserver + internal/plugin/client), so the .so speaks the SAME contract a subprocess plugin speaks. This package stays free of that machinery on purpose (TestContractsArePluginSafe): driver is a contract package an external plugin module imports directly, and must not transitively import engine internals. The concrete PluginWrap implementation lives in internal/plugin/flightwrap; cmd/* wires it in.
type SinkFactory ¶
SinkFactory opens a sink from a neutral config.
Source Files