This document describes how to deploy a TiDB cluster on Alibaba Cloud Kubernetes with your laptop (Linux or macOS) for development or testing.
The access key used must be granted permissions to control resources.
kubectl >= 1.12
helm >= 2.9.1 and <= 2.11.0
jq >= 1.6
To deploy a TiDB cluster, make sure you have the following privileges:
The default setup creates:
A managed ACK (Alibaba Cloud Kubernetes) cluster with the following ECS instance worker nodes:
ecs.i2.xlargeinstances for PD
ecs.i2.2xlargeinstances for TiKV
In addition, the monitoring node mounts a 500GB cloud disk as data volume. All the instances except ACK mandatory workers span in multiple available zones to provide cross-AZ high availability.
The auto-scaling group ensures the desired number of healthy instances, so the cluster can auto-recover from node failure or even available zone failure.
Configure the target region and credential (you can also set these variables in
terraform command prompt):
export TF_VAR_ALICLOUD_REGION=<YOUR_REGION> && \ export TF_VAR_ALICLOUD_ACCESS_KEY=<YOUR_ACCESS_KEY> && \ export TF_VAR_ALICLOUD_SECRET_KEY=<YOUR_SECRET_KEY>
variables.tf file contains default settings of variables used for deploying the cluster, you can change it or use
-var option to override a specific variable to fit your need.
Use the following commands to set up the cluster.
Get the code from Github:
git clone --depth=1 https://github.com/pingcap/tidb-operator && \ cd tidb-operator/deploy/aliyun
Apply the configs, note that you must answer “yes” to
terraform apply to continue:
If you get an error while running
terraform apply, fix the error (for example, lack of permission) according to the description and run
terraform apply again.
terraform apply takes 5 to 10 minutes to create the whole stack, once complete, basic cluster information is printed:
Apply complete! Resources: 3 added, 0 changed, 1 destroyed. Outputs: bastion_ip = 188.8.131.52 bastion_key_file = /root/tidb-operator/deploy/aliyun/credentials/tidb-cluster-bastion-key.pem cluster_id = ca57c6071f31f458da66965ceddd1c31b kubeconfig_file = /root/tidb-operator/deploy/aliyun/.terraform/modules/a2078f76522ae433133fc16e24bd21ae/kubeconfig_tidb-cluster monitor_endpoint = 184.108.40.206:3000 region = cn-hangzhou tidb_port = 4000 tidb_slb_ip = 192.168.5.53 tidb_version = v3.0.0-rc.1 vpc_id = vpc-bp16wcbu0xhbg833fymmc worker_key_file = /root/tidb-operator/deploy/aliyun/credentials/tidb-cluster-node-key.pem
You can use the
terraform outputcommand to get the output again.
You can then interact with the ACK cluster using
tidb-cluster by default):
You can connect the TiDB cluster via the bastion instance, all necessary information are in the output printed after installation is finished (replace the
<> parts with values from the output):
ssh -i credentials/<cluster_name>-bastion-key.pem root@<bastion_ip>
mysql -h <tidb_slb_ip> -P <tidb_port> -u root
<monitor_endpoint> to view the grafana dashboards. You can find this information in the output of installation.
The initial login credentials are:
It is strongly recommended to set
variables.tffor security if you already have a VPN connecting to your VPC or plan to setup one.
To upgrade TiDB cluster, modify
tidb_version variable to a higher version in
variables.tf and run
This may take a while to complete, watch the process using command:
kubectl get pods --namespace tidb -o wide --watch
To scale the TiDB cluster, modify
tidb_count to your desired numbers, and then run
By default, the terraform script will create a new VPC. You can use an existing VPC by setting
vpc_id to use an existing VPC. Note that kubernetes node will only be created in available zones that has vswitch existed when using existing VPC.
An ecs instance is also created by default as bastion machine to connect to the created TiDB cluster, because the TiDB service is only exposed to intranet. The bastion instance has mysql-cli and sysbench installed that helps you use and test TiDB.
If you don’t have to access TiDB from internet, you could disable the creation of bastion instance by setting
create_bastion to false in
The worker node instance types are also configurable, there are two ways to configure that:
Because the Alibaba Cloud offers different instance types in different region, it is recommended to specify the capacity instead of certain type. You can configure these in the
variables.tf, note that instance type overrides capacity configurations.
There is an exception for PD and TiKV instances, because PD and TiKV required local SSD, so you cannot specify instance types for them. Instead, you can choose the type family among
ecs.i2g, which has one or more local NVMe SSD, and select a certain type in the type family by specifying
For more customization options, please refer to
It may take some while to finish destroying the cluster.
Alibaba cloud terraform provider does not handle kubernetes creation error properly, which causes an error when destroying. In that case, you can remove the kubernetes resource from the local state manually and proceed to destroy the rest resources:
terraform state list
terraform state rm module.ack.alicloud_cs_managed_kubernetes.k8s
You have to manually delete the cloud disk used by monitoring node in Aliyun’s console after destroying if you don’t need it anymore.
You cannot change pod cidr, service cidr and worker instance types once the cluster is created.