TiDB Lightning supports two backends: Importer and TiDB. It determines how
tidb-lightning delivers data into the target cluster.
The Importer-backend (default) requires
tidb-lightning to first encode the SQL or CSV data into KV pairs, and relies on the external
tikv-importer program to sort these KV pairs and ingest directly into the TiKV nodes.
The TiDB-backend requires
tidb-lightning to encode these data into SQL
INSERT statements, and has these statements executed directly on the TiDB node.
|Speed||Fast (~300 GB/hr)||Slow (~50 GB/hr)|
|ACID respected while importing||No||Yes|
|Target tables||Must be empty||Can be populated|
When using the TiDB-backend, you no longer need
tikv-importer. Compared with the standard deployment procedure, the TiDB-backend deployment has the following two differences:
tikv-importercan all be skipped.
The speed of TiDB Lightning using TiDB-backend is limited by the SQL processing speed of TiDB. Therefore, even a lower-end machine may max out the possible performance. The recommended hardware configuration is:
[importer_server] section in
inventory.ini can be left blank.
... [importer_server] # keep empty [lightning_server] 192.168.20.10 ...
tikv_importer_port setting in
group_vars/all.yml is ignored, and the file
group_vars/importer_server.yml does not need to be changed. But you need to edit
conf/tidb-lightning.yml and change the
backend setting to
... tikv_importer: backend: "tidb" # <-- change this ...
Bootstrap and deploy the cluster as usual.
Mount the data source for TiDB Lightning as usual.
tidb-lightning as usual.
You do not need to download and configure
tidb-lightning, add the following lines into the configuration file:
[tikv-importer] backend = "tidb"
or supplying the
--backend tidb arguments when executing
The TiDB-backend supports importing to an already-populated table. However, the new data might cause a unique key conflict with the old data. You can control how to resolve the conflict by using this task configuration.
[tikv-importer] backend = "tidb" on-duplicate = "replace" # or "error" or "ignore"
|Setting||Behavior on conflict||Equivalent SQL statement|
|replace||New entries replace old ones||
|ignore||Keep old entries and ignore new ones||