Porting TiDB to ARM64 for Greater Flexibility
The Intel x86-64 is the most widely used processor in server architectures. However, ARM, previously known as Advanced RISC Machine, is attracting more and more attention, and many companies have migrated their products from x86-64 to ARM.
At PingCAP, we also want to run TiDB on ARM. This article describes how we compiled and benchmarked TiDB on the Amazon Web Services (AWS) ARM64 platform.
The TiDB platform is comprised of three main components: the TiDB server, the Placement Driver (PD) server, and the TiKV server. To compile and build these components, we did the following:
Installed supporting software that is compatible with ARM64.
Before we compiled, TiKV, we used yum to install the following dependencies:
sudo yum install -y gcc gcc-c++ wget git zlib-devel wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo rpm -ivh epel-release-latest-7.noarch.rpm sudo yum install -y epel-release sudo yum install -y cmake3 gcc gcc-c++ wget git zlib-devel sudo ln -s /usr/bin/cmake3 /usr/bin/cmake
During the TiDB and PD builds, we discovered compatibility issues with a TiDB vendor. We upgraded the vendor software and solved the problem.
By the way, there is a script which can help you build TiDB on ARM64. Enjoy it.
After we built our TiDB version for ARM64, we benchmarked it to compare its performance with the same configuration on x86-64.
We used Yahoo! Cloud Serving Benchmark (YCSB) for our benchmarking software.
We used two machines for our test setup: one ARM64 machine with 16 cores and 32 GB RAM, and one x86-64 machine with 16 cores and 32 GB RAM. On each machine, we started one PD instance and one TiKV instance with default configurations.
Note: We only use the go-ycsb Raw mode to benchmark TiKV, not TiDB.
We used 100 threads, loaded 1 million rows of data, and executed 1 million operations. The benchmark results were as follows:
|x86-64 (operations/sec)||ARM64 (operations/sec)|
|Workload A||Update: 14,244.5|
|Workload B||Update: 4,819.0|
|Workload C||Read: 113,663.1||Read: 49,262.0|
As you can see, the performance on ARM64 is much worse than that on x86-64. So we need to do some optimizations. From Optimizing Network Intensive Workloads on Amazon EC2 A1 Instances, we know we can do something to handle the network more efficiently. This article provides a script that makes CPU 0 and 8 handle network interruptions, and evenly distribute received packets to all CPUs. But unfortunately, after we applied this script and did the benchmark again, we ran
mpstat and found that CPU 0 and 8 were full.
So we did a little change, to distribute received packets to all the CPUs except 0 and 8:
sudo sh -c 'echo fe > /sys/class/net/eth0/queues/rx-0/rps_cpus' sudo sh -c 'echo fe00 > /sys/class/net/eth0/queues/rx-1/rps_cpus'
The result was better on ARM64, but CPU 0 and 8 were still the bottleneck. Specifically, the bottleneck was caused by the network interrupt handling. We'll try to optimize it later.
You can see that CPUs run more evenly on x86-64 than on ARM64.
This article showed how we built TiDB and performed some benchmark optimizations on ARM. Because of its low power and low cost, I believe ARM can be used more and more widely in servers. Maybe this is the future in China, too.