Data Migration (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process.
The Data Migration tool includes three components: DM-master, DM-worker, and dmctl.
DM-master manages and schedules the operation of data replication tasks.
DM-worker executes specific data replication tasks.
For details about DM-worker, see DM-worker Introduction.
dmctl is the command line tool used to control the DM cluster.
This section describes the data replication feature provided by Data Migration in detail.
The black and white lists filtering rule of the upstream database instances is similar to MySQL replication-rules-db/tables, which can be used to filter or only replicate all operations of some databases or some tables.
Binlog event filtering is a more fine-grained filtering rule than the black and white lists filtering rule at the schema and table levels. You can use statements like
TRUNCATE TABLE to specify the Binlog events of the database(s) or table(s) that you need to replicate or filter out.
Column mapping is used to resolve the conflicts occurred when the sharding auto-increment primary key IDs are merged for sharded tables. The value of the auto-increment primary key ID can be modified according to the instance-id, which is configured by the user, and the schema/table ID.
DM supports merging the original sharded instances and tables into TiDB, with some restrictions.
Currently, TiDB is not compatible with all the DDL statements that MySQL supports. See the DDL statements supported by TiDB.
DM reports an error when it encounters an incompatible DDL statement. To solve this error, you need to manually handle it using dmctl, either skipping this DDL statement or replacing it with a specified DDL statement(s). For details, see Skip or replace abnormal SQL statements.