Directories
¶
| Path | Synopsis |
|---|---|
|
Package builtin blank-imports the built-in source and sink drivers so their init() functions register with the driver registry.
|
Package builtin blank-imports the built-in source and sink drivers so their init() functions register with the driver registry. |
|
Package coordinator drives the source side of the split pipeline: it owns the replication reader and the DBLog snapshot, serves the control plane (Session/Assignment) and the Arrow Flight data plane, and streams change batches to the connected workers.
|
Package coordinator drives the source side of the split pipeline: it owns the replication reader and the DBLog snapshot, serves the control plane (Session/Assignment) and the Arrow Flight data plane, and streams change batches to the connected workers. |
|
Package dashboard serves the coordinator's embedded monitoring UI and its JSON API.
|
Package dashboard serves the coordinator's embedded monitoring UI and its JSON API. |
|
dataplanetest
Package dataplanetest holds the batch fixtures the engine's tests build their inputs from.
|
Package dataplanetest holds the batch fixtures the engine's tests build their inputs from. |
|
Package enrich is the table-level enrichment stage: a columnar broadcast hash join against small reference tables (CR-069).
|
Package enrich is the table-level enrichment stage: a columnar broadcast hash join against small reference tables (CR-069). |
|
Package errors decides whether a source failure is worth retrying.
|
Package errors decides whether a source failure is worth retrying. |
|
Package eventlog implements the per-run-id JSONL event log in S3 — one event per lifecycle cycle or commit, never per row.
|
Package eventlog implements the per-run-id JSONL event log in S3 — one event per lifecycle cycle or commit, never per row. |
|
Package faultinject is test instrumentation for the commit-boundary crash matrix (issue #354): named points in the commit path where a test can make the process die at exactly that step, so each crash window is reproducible instead of a matter of timing.
|
Package faultinject is test instrumentation for the commit-boundary crash matrix (issue #354): named points in the commit path where a test can make the process die at exactly that step, so each crash window is reproducible instead of a matter of timing. |
|
Package gate is the shared DBLog window gate used by both the collapsed runner and the distributed coordinator (design: one orchestration for collapsed and distributed mode, issue #404).
|
Package gate is the shared DBLog window gate used by both the collapsed runner and the distributed coordinator (design: one orchestration for collapsed and distributed mode, issue #404). |
|
Package grpctls builds mutual-TLS gRPC credentials for the coordinator (server) and worker (client) control plane.
|
Package grpctls builds mutual-TLS gRPC credentials for the coordinator (server) and worker (client) control plane. |
|
Package historyserver serves a read-only, request/response API over terminated pipeline runs, backed by the durable eventlog trail in S3.
|
Package historyserver serves a read-only, request/response API over terminated pipeline runs, backed by the durable eventlog trail in S3. |
|
Package logging builds the process-wide slog logger from CLI flags, so every binary in the tree configures its handler and level the same way.
|
Package logging builds the process-wide slog logger from CLI flags, so every binary in the tree configures its handler and level the same way. |
|
Package maintenance schedules the engine's table-maintenance operations.
|
Package maintenance schedules the engine's table-maintenance operations. |
|
Package memlimit gives the Go runtime the container's memory limit.
|
Package memlimit gives the Go runtime the container's memory limit. |
|
Package observability exposes the lean Prometheus metrics (§13.1) and the live status endpoint (§13.4): a small set of series, no per-event logging.
|
Package observability exposes the lean Prometheus metrics (§13.1) and the live status endpoint (§13.4): a small set of series, no per-event logging. |
|
Package operator reconciles CDCPipeline CRs into a coordinator StatefulSet: one coordinator per pipeline, owning the workers it creates.
|
Package operator reconciles CDCPipeline CRs into a coordinator StatefulSet: one coordinator per pipeline, owning the workers it creates. |
|
Package partition orders primary-key tuples and locates the partition range that owns a key.
|
Package partition orders primary-key tuples and locates the partition range that owns a key. |
|
Package pipeline defines the stage abstraction that bridges external plugin processes to the runner's source/sink interfaces.
|
Package pipeline defines the stage abstraction that bridges external plugin processes to the runner's source/sink interfaces. |
|
Package plan performs the boot introspection both orchestration modes share: resolve every spec table through the source, apply the cast policy, resolve the sink schema, and expand the enrich reference columns — including the real names a wildcard select only discovers by running its query — ONCE, producing a single Plan the collapsed runner (internal/runner) and the distributed coordinator (internal/coordinator) each consume.
|
Package plan performs the boot introspection both orchestration modes share: resolve every spec table through the source, apply the cast policy, resolve the sink schema, and expand the enrich reference columns — including the real names a wildcard select only discovers by running its query — ONCE, producing a single Plan the collapsed runner (internal/runner) and the distributed coordinator (internal/coordinator) each consume. |
|
Package plugin implements adapters that wrap the Flight client into the standard source.Source and sink.Sink interfaces.
|
Package plugin implements adapters that wrap the Flight client into the standard source.Source and sink.Sink interfaces. |
|
flightserver
Package flightserver serves the plugin Arrow Flight contract (internal/ plugin/contract) in-process, backed by a real source.Source or sink.Sink.
|
Package flightserver serves the plugin Arrow Flight contract (internal/ plugin/contract) in-process, backed by a real source.Source or sink.Sink. |
|
flightwrap
Package flightwrap implements driver.PluginWrap: it turns a .so's raw registered source/sink factories into factories that serve the real driver over an in-process Arrow Flight server (internal/plugin/ flightserver) and hand back a Flight client adapter (internal/plugin) — the SAME contract a subprocess plugin speaks.
|
Package flightwrap implements driver.PluginWrap: it turns a .so's raw registered source/sink factories into factories that serve the real driver over an in-process Arrow Flight server (internal/plugin/ flightserver) and hand back a Flight client adapter (internal/plugin) — the SAME contract a subprocess plugin speaks. |
|
proc
Package proc manages external plugin processes: spawn, readiness, log pumping, and lifecycle.
|
Package proc manages external plugin processes: spawn, readiness, log pumping, and lifecycle. |
|
Package resume resolves where one pipeline resumes from and which tables it must snapshot or adopt — the shared boot step of the collapsed runner (internal/runner) and the distributed coordinator (internal/coordinator).
|
Package resume resolves where one pipeline resumes from and which tables it must snapshot or adopt — the shared boot step of the collapsed runner (internal/runner) and the distributed coordinator (internal/coordinator). |
|
Package routing decides which partition owner holds a primary key, and only that.
|
Package routing decides which partition owner holds a primary key, and only that. |
|
Package rowchange defines the change event that flows from source decoding to sink writing: one row-level operation with its before/after images, primary key, and source position.
|
Package rowchange defines the change event that flows from source decoding to sink writing: one row-level operation with its before/after images, primary key, and source position. |
|
Package runner wires the collapsed process: one binary runs the source reader, the DBLog snapshot, and the worker in a single process (local mode).
|
Package runner wires the collapsed process: one binary runs the source reader, the DBLog snapshot, and the worker in a single process (local mode). |
|
sink
|
|
|
clickhouse
Package clickhouse implements the sink contract on ClickHouse.
|
Package clickhouse implements the sink contract on ClickHouse. |
|
couchbase
Package couchbase implements the sink contract on Couchbase: key-value document writes where upsert-by-key IS the native operation — no equality deletes, no eventual merges.
|
Package couchbase implements the sink contract on Couchbase: key-value document writes where upsert-by-key IS the native operation — no equality deletes, no eventual merges. |
|
iceberg
Package iceberg wraps the Apache Iceberg REST catalog and the write paths exercised by the spike: create/load, append, and the data+equality-delete transaction that mirrors upsert.
|
Package iceberg wraps the Apache Iceberg REST catalog and the write paths exercised by the spike: create/load, append, and the data+equality-delete transaction that mirrors upsert. |
|
rowmeta
Package rowmeta resolves a row's metadata columns (op, commit_ts, position, …) for the row-based sinks, ClickHouse and Couchbase, so both write the same value for the same key (issue #403).
|
Package rowmeta resolves a row's metadata columns (op, commit_ts, position, …) for the row-based sinks, ClickHouse and Couchbase, so both write the same value for the same key (issue #403). |
|
Package snapshot holds the source-agnostic DBLog snapshot orchestrator: chunking by primary key, low/high watermarks, and the caught-up proof that closes each window — never a timer.
|
Package snapshot holds the source-agnostic DBLog snapshot orchestrator: chunking by primary key, low/high watermarks, and the caught-up proof that closes each window — never a timer. |
|
source
|
|
|
filterexpr
Package filterexpr renders the shared spec.Filter AST into an expr-lang expression.
|
Package filterexpr renders the shared spec.Filter AST into an expr-lang expression. |
|
kafka
Direct-to-Arrow Kafka consumption (#733): the decoder writes each record's fields straight into a transport.RowEncoder built from the target table's canonical schema, so no per-record rowchange.Change is materialized and the source puller never re-encodes one.
|
Direct-to-Arrow Kafka consumption (#733): the decoder writes each record's fields straight into a transport.RowEncoder built from the target table's canonical schema, so no per-record rowchange.Change is materialized and the source puller never re-encodes one. |
|
kafka/decoder
Columnar decoding: the direct-to-Arrow sibling of Decoder (#733).
|
Columnar decoding: the direct-to-Arrow sibling of Decoder (#733). |
|
mysql
Package source.mysql implements the MySQL replication source on top of go-mysql/canal: a single binlog reader that decodes row events into rowchange.Change, positions them at their transaction GTID, and exposes the synced and master positions for the DBLog watermark logic.
|
Package source.mysql implements the MySQL replication source on top of go-mysql/canal: a single binlog reader that decodes row events into rowchange.Change, positions them at their transaction GTID, and exposes the synced and master positions for the DBLog watermark logic. |
|
mysql/charsetgen
command
Command charsetgen turns the .tsv tables in this directory into charset_tables.go, the single-byte decoding tables for the 7 character sets that have no decoder in golang.org/x/text (see issue #113).
|
Command charsetgen turns the .tsv tables in this directory into charset_tables.go, the single-byte decoding tables for the 7 character sets that have no decoder in golang.org/x/text (see issue #113). |
|
postgres
Package source.postgres implements the PostgreSQL replication source: a single logical-decoding reader (pgoutput) that decodes row changes into rowchange.Change, positions them at their commit LSN, and exposes the synced and confirmed positions for the DBLog watermark logic.
|
Package source.postgres implements the PostgreSQL replication source: a single logical-decoding reader (pgoutput) that decodes row changes into rowchange.Change, positions them at their commit LSN, and exposes the synced and confirmed positions for the DBLog watermark logic. |
|
Package sourcepull adapts a push-based change channel into the pull-based source.Reader.Next surface, encoding buffered changes into wire batches.
|
Package sourcepull adapts a push-based change channel into the pull-based source.Reader.Next surface, encoding buffered changes into wire batches. |
|
Package supervisor manages external plugin subprocesses with restart backoff, circuit breaker protection, and graceful shutdown.
|
Package supervisor manages external plugin subprocesses with restart backoff, circuit breaker protection, and graceful shutdown. |
|
Flight data-plane codec: change batches travel as Arrow IPC records with a BatchMeta proto in the FlightData app_metadata.
|
Flight data-plane codec: change batches travel as Arrow IPC records with a BatchMeta proto in the FlightData app_metadata. |
|
Package version exposes build metadata injected via ldflags (see Makefile).
|
Package version exposes build metadata injected via ldflags (see Makefile). |
|
Package worker implements the worker plane: per-table batchers that accumulate changes and flush them collapsed to a committer, with commits strictly serialized per table.
|
Package worker implements the worker plane: per-table batchers that accumulate changes and flush them collapsed to a committer, with commits strictly serialized per table. |
Click to show internal directories.
Click to hide internal directories.