The TiDB ecosystem has a wealth of tools for data migration, backup & restore for users with different use cases to choose from.
This guide is specifically designed to help you better understand these tools and therefore make an informed decision while choosing these tools to support your business.
TiDB Lightning (Lightning) is a tool used for the fast full import of large amounts of data into a TiDB cluster. Currently, TiDB Lightning supports reading SQL dump exported via Mydumper or CSV data source.
TiDB Lightning supports two back ends: “Importer” and “TiDB”. It determines how tidb-lightning delivers data into the target cluster. The two back ends are as follows:
Importerback end. When using
Importeras the back end, the cluster cannot provide normal services during the import process. It is used for a large amount of data importing (TB).
TiDBback end (just work as Loader). It is much slower than
Importerback end model. But the cluster could serve the application during the import process. It is used to handle tens/hundreds of GB data.
The following are the basics of TiDB Lightning:
BR (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/Loader/Lightning, BR is more suitable for scenarios of huge data volume.
The following are the basics of BR:
TiDB Loader is to be deprecated and replaced with Lightning.
TiDB Loader is a lightweight full-data importing tool for TiDB. It reads the output file of Mydumper and loads the data into TiDB.
The following are the basics of Loader:
Syncer is a tool used to import data incrementally. It acts as a MySQL slave to read binlog from MySQL/MariaDB master and replicate the binlog to the downstream. It is recommended to use TiDB Data Migration to replace Syncer.
The following are the basics of Syncer:
TiDB Data Migration (DM) is an integrated data replication task management platform that supports the full data migration and the incremental data migration from MySQL/MariaDB into TiDB. It can help to reduce the operations cost and simplify the troubleshooting process.
For the full data migration, it uses an embedded Loader and an embedded Mydumper. For the incremental data migration, it uses Syncer as its kernel.
The following are the basics of DM:
Mydumper is a tool to create a logical full backup for TiDB.
The following are the basics of Mydumper:
BR (Backup & Restore) is a command-line tool for distributed backup and restoration of the TiDB cluster data. Compared with Mydumper/loader, BR is more suitable for scenarios of huge data volume.
The following are the basics of BR:
TiDB Binlog is a system that collects binlog for TiDB clusters and provides tools for near real-time sync and backup.
The following are the basics of TiDB Binlog:
CDC (Change Data Capture) is a system that collects changelog for key value pairs in TiKV and outputs to downstream systems in row changed order.
For the recommended tools for other TiDB versions, see Recommended tools for TiDB versions.
TiDB 3.0 is the recommended version and is also the most widely adopted version. In addition, TiDB 3.1 GA and 4.0 GA will be released this year.
For TiDB 3.0 versions, this section covers how to migrate data from MySQL to TiDB, between TiDB clusters, and from TiDB to MySQL for each version, as well as how to back up and restore data.
If the MySQL data volume is in TBs:
If the MySQL data volume is in GBs:
You can use TiDB Binlog to replicate data between TiDB clusters. You can also use TiDB Binlog to replicate data to the downstream MySQL cluster.
tidbbackend for full data restore