This document introduces the common error scenarios in using Loader or TiDB Data Migration (DM) during the full data import process, and also provides their reasons and solutions.
Try adjusting the `max_allowed_packet` variable
The following error is reported during full data import:
packet for query is too large. Try adjusting the 'max_allowed_packet' variable
max_allowed_packet
quotas. If any of the max_allowed_packet
quotas is violated, the client receives a corresponding error message. Currently, the latest version of Syncer, Loader, DM, and TiDB Server all have a default max_allowed_packet
quota of 64M
.
dump sqls
files.For the above reasons, it is recommended to use the -s, --statement-size
option which Mydumper offers to control the size of Insert Statement
: Attempted size of INSERT statement in bytes, default 1000000
.
According to the default configuration of --statement-size
, the size of Insert Statement
that Mydumper generates is as close as 1M
. This default configuration ensures that this error will not occur in most cases.
Sometimes the following WARN
log appears during the dump process. The WARN
log itself does not affect the dump process but indicates that the dumped table might be a wide table.
Row bigger than statement_size for xxx
If a single row of a wide table exceeds 64M, you need to modify and enable the following two configurations:
set @@global.max_allowed_packet=134217728
(134217728 = 128M
) in TiDB Server.max-allowed-packet=128M
to the Loader configuration file or DB configuration in DM task configuration file, and then restart the progress or task.