tiflow

module

v0.0.0-...-145e2ed Latest Latest Go to latest Published: Dec 25, 2025 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/pingcap/tiflow

Links

Open Source Insights

README ¶

TiFlow

GitHub release (latest SemVer) GitHub Release Date GitHub go.mod Go version

Introduction

TiFlow is a unified data replication platform for TiDB that consists of two main components: TiDB Data Migration (DM) and TiCDC.

DM enables full data migration and incremental data replication from MySQL or MariaDB to TiDB.
TiCDC replicates change data to various downstream systems, such as MySQL protocol-compatible databases and Kafka.

Note:

The TiCDC in this repository is only used for versions v8.5.x and lower (old architecture). A TiCDC with new architecture is hosted in github.com/pingcap/ticdc, starting from v8.5.4.

For more details, see DM README and TiCDC README.

You can also check out the DeepWiki documentation for more information.

License

TiFlow is under the Apache 2.0 license. See the LICENSE file for details.

Directories ¶

Path	Synopsis
cdc
api
api/middleware
api/owner
api/status
api/v1
api/v2 Package v2 is a generated GoMock package.	Package v2 is a generated GoMock package.
async
capture
capture/mock Package mock_capture is a generated GoMock package.	Package mock_capture is a generated GoMock package.
entry
entry/schema
kv
kv/regionlock
kv/sharedconn
metrics
model
model/codec
model/codec/v1
owner
owner/mock Package mock_owner is a generated GoMock package.	Package mock_owner is a generated GoMock package.
processor
processor/memquota
processor/mock Package mock_processor is a generated GoMock package.	Package mock_processor is a generated GoMock package.
processor/sinkmanager
processor/sourcemanager
processor/sourcemanager/sorter
processor/sourcemanager/sorter/factory
processor/sourcemanager/sorter/memory Package memory is an in-memory table based EventSortEngine implementation.	Package memory is an in-memory table based EventSortEngine implementation.
processor/sourcemanager/sorter/mock Package mock_sorter is a generated GoMock package.	Package mock_sorter is a generated GoMock package.
processor/sourcemanager/sorter/pebble Package pebble is an pebble-based EventSortEngine implementation with such properties:	Package pebble is an pebble-based EventSortEngine implementation with such properties:
processor/sourcemanager/sorter/pebble/encoding
processor/tablepb
puller
puller/frontier
puller/memorysorter Package memorysorter is an in-memory event sorter implementation.	Package memorysorter is an in-memory event sorter implementation.
redo Package redo provide a redo log for cdc.	Package redo provide a redo log for cdc.
redo/common
redo/reader
redo/writer
redo/writer/blackhole
redo/writer/factory
redo/writer/file
redo/writer/memory
scheduler
scheduler/internal
scheduler/internal/v3
scheduler/internal/v3/agent
scheduler/internal/v3/compat
scheduler/internal/v3/keyspan
scheduler/internal/v3/member
scheduler/internal/v3/replication
scheduler/internal/v3/scheduler
scheduler/internal/v3/transport
scheduler/schedulepb
server
sink/ddlsink
sink/ddlsink/blackhole
sink/ddlsink/cloudstorage
sink/ddlsink/factory
sink/ddlsink/mq
sink/ddlsink/mq/ddlproducer
sink/ddlsink/mysql
sink/dmlsink
sink/dmlsink/blackhole
sink/dmlsink/cloudstorage
sink/dmlsink/factory
sink/dmlsink/mq
sink/dmlsink/mq/dispatcher
sink/dmlsink/mq/dispatcher/partition
sink/dmlsink/mq/dispatcher/topic
sink/dmlsink/mq/dmlproducer
sink/dmlsink/mq/manager
sink/dmlsink/mq/transformer
sink/dmlsink/mq/transformer/columnselector
sink/dmlsink/txn
sink/dmlsink/txn/mysql
sink/metrics
sink/metrics/cloudstorage
sink/metrics/mq
sink/metrics/tablesink
sink/metrics/txn
sink/tablesink
sink/tablesink/state
sink/util
sink/validator
syncpointstore
vars
cmd
cdc command
dm-ctl command
dm-master command
dm-syncer command
dm-worker command
filter-helper command
kafka-consumer command
oauth2-server command
pulsar-consumer command
storage-consumer command
tiflow command
dm
_utils/terror_gen command
chaos/cases command
checker
common
config
config/dbconfig
config/security
ctl
ctl/common
ctl/master
debug-tools/binlog-event-blackhole command
dumpling
loader
master
master/metrics
master/scheduler
master/shardddl
master/workerrpc
openapi Package openapi provides primitives to interact with the openapi HTTP API.	Package openapi provides primitives to interact with the openapi HTTP API.
openapi/fixtures
pb Package pb is a reverse proxy.	Package pb is a reverse proxy.
pbmock Package pbmock is a generated GoMock package.	Package pbmock is a generated GoMock package.
pkg/backoff
pkg/binlog
pkg/binlog/common
pkg/binlog/event
pkg/binlog/reader
pkg/checker
pkg/conn
pkg/context
pkg/cputil
pkg/dumpling
pkg/election
pkg/encrypt
pkg/etcdutil
pkg/func-rollback
pkg/gtid
pkg/ha
pkg/helper
pkg/log
pkg/parser
pkg/printinit
pkg/retry
pkg/schema
pkg/shardddl/optimism
pkg/shardddl/pessimism
pkg/storage
pkg/streamer
pkg/terror
pkg/upgrade
pkg/utils
pkg/v1dbschema
pkg/v1workermeta
relay
simulator/config Package config is the configuration definitions used by the simulator.	Package config is the configuration definitions used by the simulator.
simulator/mcp Package mcp defines the Modification Candidate Pool (MCP).	Package mcp defines the Modification Candidate Pool (MCP).
simulator/sqlgen Package sqlgen is the logic for generating different kinds of SQL statements.	Package sqlgen is the logic for generating different kinds of SQL statements.
syncer
syncer/binlogstream Package binlogstream is used by syncer to read binlog.	Package binlogstream is used by syncer to read binlog.
syncer/dbconn
syncer/metrics
syncer/online-ddl-tools
syncer/safe-mode
syncer/shardddl
syncer/sharding-meta
tests/_dmctl_tools command
tests/utils
ui this file implement all of the APIs of the DataMigration service.	this file implement all of the APIs of the DataMigration service.
unit
worker
engine
chaos/cases command
chaos/cases/dm
enginepb Package enginepb is a reverse proxy.	Package enginepb is a reverse proxy.
enginepb/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
executor
executor/cvs
executor/dm
executor/fakejob
executor/server
executor/server/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
executor/worker
executor/worker/internal
framework
framework/config
framework/internal/errors
framework/internal/eventloop
framework/internal/master
framework/internal/worker
framework/logutil
framework/metadata
framework/model
framework/registry
framework/statusutil
framework/taskutil
internal/pkg/discovery
jobmaster/cvsjob
jobmaster/dm
jobmaster/dm/bootstrap
jobmaster/dm/checkpoint
jobmaster/dm/config
jobmaster/dm/metadata
jobmaster/dm/openapi Package openapi provides primitives to interact with the openapi HTTP API.	Package openapi provides primitives to interact with the openapi HTTP API.
jobmaster/dm/runtime
jobmaster/example
jobmaster/fakejob
model
pkg/adapter
pkg/client Package client is a generated GoMock package.	Package client is a generated GoMock package.
pkg/client/internal
pkg/client/internal/endpoint
pkg/clock
pkg/cmd
pkg/cmd/cli
pkg/cmd/executor
pkg/cmd/master
pkg/cmd/util
pkg/cmd/version
pkg/containers
pkg/context
pkg/ctxmu
pkg/dbutil
pkg/deps
pkg/dm
pkg/dm/ticker
pkg/errctx
pkg/externalresource/broker
pkg/externalresource/integration_test
pkg/externalresource/internal
pkg/externalresource/internal/bucket
pkg/externalresource/internal/local
pkg/externalresource/manager
pkg/externalresource/model
pkg/fakejob
pkg/httputil
pkg/httputil/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
pkg/meta
pkg/meta/internal
pkg/meta/internal/etcdkv
pkg/meta/internal/etcdkv/model
pkg/meta/internal/etcdkv/namespace
pkg/meta/internal/mockkv
pkg/meta/internal/sqlkv
pkg/meta/internal/sqlkv/model
pkg/meta/internal/sqlkv/namespace
pkg/meta/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
pkg/meta/model
pkg/notifier
pkg/openapi
pkg/orm
pkg/orm/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
pkg/orm/model
pkg/p2p
pkg/promutil
pkg/promutil/test command
pkg/quota
pkg/rpcutil
pkg/rpcutil/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
pkg/tenant
servermaster
servermaster/jobop
servermaster/jobop/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
servermaster/scheduler
servermaster/scheduler/model
servermaster/serverutil
test/e2e
test/mock
pkg
api/internal/rest
api/v2
api/v2/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
applier
binlog-filter
causality
causality/internal
causality/tests
chann Package chann providesa a unified channel package.	Package chann providesa a unified channel package.
chdelay
check
cmd
cmd/cli
cmd/context
cmd/factory
cmd/factory/mock Package mock_factory is a generated GoMock package.	Package mock_factory is a generated GoMock package.
cmd/redo
cmd/server
cmd/util
cmd/version
column-mapping
compression
config
config/outdated
container/queue
container/sortmap
ddl
diff
election
election/mock Package mock is a generated GoMock package.	Package mock is a generated GoMock package.
errors
errorutil
etcd
etcd/mock Package mock_etcd is a generated GoMock package.	Package mock_etcd is a generated GoMock package.
filter
flags
fsutil
hash
httputil
importer
integrity
label
leakutil
logutil
migrate
notify
orchestrator Package orchestrator mainly implements a ETCD worker.	Package orchestrator mainly implements a ETCD worker.
orchestrator/util
p2p
p2p/internal
pdutil
quotes
redo
retry
security
sink
sink/cloudstorage
sink/codec
sink/codec/avro
sink/codec/builder
sink/codec/canal
sink/codec/common
sink/codec/craft
sink/codec/csv
sink/codec/debezium
sink/codec/internal
sink/codec/maxwell
sink/codec/open
sink/codec/simple
sink/codec/simple/mock Package mock_simple is a generated GoMock package.	Package mock_simple is a generated GoMock package.
sink/codec/utils
sink/kafka
sink/kafka/claimcheck
sink/kafka/v2
sink/kafka/v2/mock Package mock_v2 is a generated GoMock package.	Package mock_v2 is a generated GoMock package.
sink/mysql
sink/observer
sink/pulsar
spanz
sqlmodel
tcpserver
txnutil
txnutil/gc Package gc privodes TiDB GC related utilities.	Package gc privodes TiDB GC related utilities.
types
upstream
util
util/seahash The file is copied from blainsmith.com/go/seahash@v1.2.1.	The file is copied from blainsmith.com/go/seahash@v1.2.1.
uuid
version
workerpool
proto
benchmark
canal
p2p
sync_diff_inspector
checkpoints
chunk
config
diff
progress
report
source
source/common
splitter
utils
tests
integration_tests/api_v2 command
integration_tests/bank command
integration_tests/cdc command
integration_tests/cdc/dailytest
integration_tests/ddl_wait command
integration_tests/ddl_with_exists command
integration_tests/default_value command
integration_tests/http_proxies command
integration_tests/many_pk_or_uk command
integration_tests/move_table command This is a program that drives the CDC cluster to move a table	This is a program that drives the CDC cluster to move a table
integration_tests/multi_source command
integration_tests/resolve_lock command
integration_tests/util
mq_protocol_tests command
mq_protocol_tests/cases
mq_protocol_tests/framework
mq_protocol_tests/framework/avro
mq_protocol_tests/framework/canal
mq_protocol_tests/framework/mysql
utils/checksum_checker command
utils/gen_kafka_big_messages command

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL