TiDB-Lightning is a tool used for fast full import of large amounts of data into a TiDB cluster. Currently, TiDB-Lightning supports reading SQL dump exported via mydumper. You can use it in the following two scenarios:
The TiDB-Lightning tool set consists of two components:
tidb-lightning (the “front end”) reads the SQL dump and imports the database structure into the TiDB cluster, and also transforms the data into Key-Value (KV) pairs and sends them to
tikv-importer (the “back end”) combines and sorts the KV pairs and then imports these sorted pairs as a whole into the TiKV cluster.
The complete import process is as follows:
tidb-lightning switches the TiKV cluster to “import mode”, which optimizes the cluster for writing and disables automatic compaction.
tidb-lightning creates the skeleton of all tables from the data source.
For each table,
tikv-importer via gRPC to create an engine file to store KV pairs.
tidb-lightning then reads the SQL dump in parallel, transforms the data into KV pairs according to the TiDB rules, and sends them to
tikv-importer’s engine files.
Once a full table of KV pairs are received,
tikv-importer divides and schedules these data and imports them into the target TiKV cluster.
tidb-lightning then performs a checksum comparison between the local data source and those calculated from the cluster, to ensure there is no data corruption in the process.
After all tables are imported,
tidb-lightning performs a global compaction on the TiKV cluster, and tells TiDB to
ANALYZE all imported tables, to prepare for optimal query planning.
tidb-lightning switches the TiKV cluster back to “normal mode”, so the cluster resumes normal services.
What’s on this page