TiDB and TiKV Blog

See TiDB Deployment Tutorials for:

Quickly Find Rust Program Bottlenecks Online Using a Go Tool

It can be hard to find Rust programs' performance bottlenecks online. By integrating pprof-rs in TiKV, we can use the Go tool pprof to visualize TiKV's profiling data. This helps analyze the program's performance online.

Run Your First Chaos Experiment in 10 Minutes

In this 10-minute tutorial, we will help you to quickly get started with Chaos Engineering and run your first chaos experiment with Chaos Mesh.

Choosing the Right Database for Your Applications

With so many databases on the market, choosing the right fit for your app is not an easy task. As a leading e-commerce platform in Southeast Asia and Taiwan, Shopee has an effective strategy for database selection. Learn from their experience.

TiDB Usability Challenge - Dare to Dream Bigger

TiDB Challenge Program is a series community program to bring TiDB to a new level in terms of stability, performance, and usability. With amazing improvements in performance in season 1 last year, we are now pleased to announce that season 2 is ready for you.

TiDB Celebrates 400 Contributors

The TiDB repository reaches the 400th contributor milestone. All our honors and achievements belong to the TiDB community, our beloved contributors, and distinguished committers.

Early Impressions of Go from a Rust Programmer

Nick Cameron is a long-time Rust programmer who has recently started using Go. In this post, he talks about his early impressions of Go. Read this post to learn more.

How We Reduced Multi-region Read Latency and Network Traffic by 50%

High read latency and network traffic are common issues for a multi-region architecture. At TiDB Hackathon 2019, a team won 2nd place by reducing multi-region read latency and network traffic by 50%. Read this post to learn how they did it.

Remote Work - Part 3

It's the third post in a series on remote work. In this post, Nick Cameron talks a bit about some of the practicalities, specifically around work and life balance.

Remote Work - Part 2

Nick Cameron has nearly ten-year experience of remote work. In this post, he discusses communication in remote work.

Remote Work - Part 1

Nick has been working remotely for nearly ten years. In this post, he introduces why you might benefit from remote work and some of the things that make remote work successful.

Doubling System Read Throughput with Only 26 Lines of Code

The Follower Read feature lets any follower replica in a Region serve a read request under the premise of strongly consistent reads. It reduces the load on the Raft leader and improves the read throughput of the TiDB cluster. Read this post to learn more.

The Rust Compilation Model Calamity

In this first episode of the Rust Compile time series, Brian Anderson, one of Rust's original authors, shares with you his researches and experiences with Rust compile times, using the TiKV project as a case study.

Chaos Mesh - Your Chaos Engineering Solution for System Resiliency on Kubernetes

Recently, PingCAP open-sourced Chaos Mesh, a Chaos Engineering platform that features all-around fault injection methods for complex Kubernetes systems. Find out how this testing tool can make your applications more reliable.

How TiKV Reads and Writes

This post details how TiKV, as a distributed database, stores the data contained in a write request and how it retrieves the corresponding data with consistency guaranteed.

AutoTiKV: TiKV Tuning Made Easy by Machine Learning

AutoTiKV is a machine-learning-based tuning tool that helps decrease tuning costs and make life easier for DBAs. This post shows AutoTiKV's design, its machine learning model, and the automatic tuning workflow.

Squashed Bugs, Served Hot and Fresh with Failure Rate Heatmaps

At TiDB Hackathon 2019, a team won the third prize by building a bot that helps quickly locate bugs in the code. Read this post to get more details.

How We Achieved 40 K QPS and 20+ Billion Rows of Data in a Single Scale-out Cluster

As the business developed, standalone MySQL couldn't meet JD Cloud's OSS metadata storage requirements. This post introduces how TiKV empowered JD Cloud to manage huge amounts of OSS metadata with a simple and horizontally scalable architecture.

NetEase Games: Why We Chose TiDB over Other MySQL-based and NewSQL Storage Solutions

As business boomed, NetEase Games met bottlenecks in database scalability and data isolation. This post introduces why they chose TiDB over other MySQL-based and NewSQL storage solutions to solve their pain points.

New Structure, New Roles - TiDB Community Upgrade!

PingCAP has upgraded and adjusted the TiDB community organization structure, with more community roles and community organizations introduced to better stimulate the vitality of the community and maintain a positive and healthy community environment.

10x Performance Improvement for Expression Evaluation Made Possible by Vectorized Execution and the Community

We vectorized 360+ built-in functions along with the TiDB community. Vectorized execution has greatly improved the expression evaluation performance. Some functions even achieved 10x performance. Here is our vectorization story.

Building a Large-scale Distributed Storage System Based on Raft

This post introduces the PingCAP team's firsthand experience in designing a large-scale distributed storage system based on the Raft consensus algorithm.

How We Compiled a Golang Database in the Browser Using WebAssembly

We compiled a Golang database (TiDB) into an in-browser database using WebAssembly (Wasm). This post introduces why and how we built an in-browser database.

TiDB in the Browser: Running a Golang Database on WebAssembly

This post introduces how you can run TiDB directly in a web browser, how it is possible, and what the limitations are.

Delivering Real-time Analytics and True HTAP by Combining Columnstore and Rowstore

TiDB is an HTAP database that targets both OLTP and OLAP scenarios. TiFlash is its extended analytical engine. This post introduces how TiFlash fuels TiDB to become a true HTAP database that lets users perform real-time analytics.

INSERT INTO tidb.hackathon_2019 VALUES ("Hack", "Fun", "TiDB Ecosystem")

This post introduces the top 3 winners at TiDB Hackathon 2019 and the cozy and friendly environment of this meeting.

Safety First! Common Safety Pitfalls in Distributed Databases Found by Jepsen Tests

This post introduces why Jepsen is important to the distributed database industry, how it works, common issues found, how we use Jepsen with TiDB, and a short summary of all problems Jepsen has found.

How TSQ Becomes a Performance Bottleneck for TiKV in AWS ARM Environment

Explore into Linux kernel to find out how TSQ becomes a performance bottleneck for TiKV in an AWS ARM environment.

Lesson Learned from Queries over 1.3 Trillion Rows of Data Within Milliseconds of Response Time at Zhihu.com

As the business boomed and massive data in applications accrued each month, Zhihu faced severe challenges in scaling the backend system. This post introduces how Zhihu managed to keep milliseconds of query response time over a large amount of data and how TiDB, an open source MySQL-compatible HTAP database, empowered Zhihu to get real-time insights into data.

Porting TiDB to ARM64 for Greater Flexibility

This article describes how PingCAP compiled and benchmarked TiDB on the Amazon Web Services (AWS) ARM64 platform.

Migrating the TiKV Rust Client from Futures 0.1 to 0.3

This post introduces Nick's experience in migrating the TiKV Rust client from Futures 0.1 to 0.3.

TiDB Operator 1.0 GA: Database Cluster Deployment and Management Made Easy with Kubernetes

Deploying transactional databases like TiDB in Kubernetes for production has always been challenging. Now, with TiDB Operator 1.0, running TiDB database clusters has never been easier.

Why Benchmarking Distributed Databases Is So Hard

Benchmarks are hard to get right, and many articles touting benchmarks are actually benchmarketing, showcasing skewed outcomes to sell products. This post introduces some of the motivations for benchmarking and the common tools, and discusses a few things to keep in mind when benchmarking.

The KOST Stack: An Open-Source Cloud-Native Hybrid Database Stack

A new infrastructure pattern is emerging called the KOST stack (Kubernetes, Operator, Spark, TiDB). This blog post introduces each component of the KOST stack and cloud-native HTAP in the wild.

TiDB 3.0 Reaches GA, Delivers Stability at Scale and Performance Boost

TiDB 3.0 is released! This blog post introduces some highlights of TiDB 3.0, including major features focused on stability, significant performance improvements in Sysbench and TPC-C benchmarks, a newly introduced component, and important features and improvements.

Distributed Systems Training in Go and Rust

PingCAP has made a talent plan for training and/or evaluating students, new employees, and new contributors to TiDB and TiKV. This blog post gives a brief overview of training courses for this plan on writing distributed systems in Go and Rust.

Best Practices for Developing Applications with TiDB

This article shows how to efficiently develop high-quality applications with TiDB, an open-source NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads and can serve as a scale-out MySQL database without manual sharding.

TiDB Passes Jepsen Test for Snapshot Isolation and Single-Key Linearizability

TiDB's first official Jepsen Test report is published. This post introduces some additional context to the test results and PingCAP's thoughts on what's next.

Percona Live Austin -- Summary and Reflection

In this post, Kevin Xu shares the PingCAP team's experience and activities at Percona Live in Austin, including several technical sharing sessions and boosting friendship with people inside and outside PingCAP.

Tutorial: Use TiDB DM to Migrate and Replicate Data from MySQL, MariaDB & Amazon Aurora

Learn to migrate and replicate data from MySQL, MariaDB and Amazon Aurora using TiDB Data Migration.

Design and Implementation of Golang Failpoints

This post introduces design and implementation of Golang Failpoint.

TiDB Binlog Tutorial

Learn to deploy TiDB Binlog with a simple TiDB cluster.

PingCAP Open-Sources TiDB Binlog to Reduce Migration Cost

Our team at PingCAP is excited to announce that we are open-sourcing our own binlog implementation, TiDB Binlog. TiDB Binlog is a tool we’ve been developing in-house (until now) to collect binary log data from TiDB server and provide real-time data backup and replication.

Try to Fix Two Linux Kernel Bugs While Testing TiDB Operator in K8s

This post introduces how the EE team at PingCAP tackled two Linux kernel bugs while testing TiDB Operator in K8s. PingCAP engineers also hope the K8s community, RHEL and CentOS can help fix these bugs thoroughly in the near future.

PingCAPers at the First RustCon Asia!

PingCAP has been an active member in the Rust community and one of the first production users of Rust in building TiKV. As part of being a co-organizer for the first RustCon Asia, PingCAP also contributed several topics in both keynote sessions and workshops.

Titan: A RocksDB Plugin to Reduce Write Amplification

As a RocksDB plugin for key-value separation, inspired by WiscKey, Titan is available for preview in TiDB 3.0. This post introduces the design and implementation of Titan.

Powering the Xiaomi Mobile Lifestyle with TiDB

As sales of Xiaomi smartphones continue to climb and the MIUI user base continues to grow, the Database Administration team at Xiaomi was having an increasingly hard time managing their MySQL database infrastructure until they adopted TiDB, an open source distributed hybrid transactional and analytical processing database created and supported by PingCAP. Now they also have plans to migrate additional workloads to TiDB in the future.

Introduction to Analytics Queries for the MySQL DBA

This post introduces some simple use cases of analytics queries where a MySQL DBA can expand their repertoire and answer some basic business questions by writing SQL queries with window functions.

BookMyShow.com: More Uptime, 30% Less Operational Cost with TiDB

With TiDB, BookMyShow has experienced increased uptime and availability, since TiDB automatically partitions and distributes data across the cluster. Meanwhile, operational and maintenance cost has been reduced by 30%. No engineer needs to be fully dedicated to database operations anymore. This post will introduce why BookMyShow chose TiDB over Greenplum by illustrating how TiDB solved their pain points.

TiDB Certification (Beta) is Ready

The official TiDB Certification is now available in beta and an online exam is ready for you to sign up.

Shopping on Shopee, the TiDB Way

As our business boom, our team faced severe challenges in scaling our backend system to meet the demand until we found TiDB, a MySQL-compatible NewSQL hybrid transactional and analytical processing (HTAP) database, built and supported by PingCAP. Now we can provide better service and experience for our users without worrying about our database capacity.

FOSDEM 2019 Recap: Global Technology, Local Community

In this post, Kevin Xu shares the PingCAP team's experience and activities at FOSDEM 2019, including three talks they delivered at the conference -- the first one about TiDB and its architecture, horizontal scalability and MySQL compatibility, the second about improving development posture in Rust based on the work building TiKV, and the last about Rust implementation of Prometheus.

TiDB Tools (III): TiDB Data Migration Architecture Design and Implementation Principles

TiDB Data Migration is an integrated data transfer and replication management platform that supports full data migration or incremental data replication from MySQL or MariaDB instances into a TiDB cluster. This post introduces its architecture design and implementation principles.

TiDB Tools (II): Introducing TiDB Lightning

TiDB Lightning is an open source TiDB ecosystem tool that supports high speed full-import of a large SQL dump into a TiDB cluster. This post introduces its architecture and future improvements on the roadmap.

TiDB Tools (I): TiDB Binlog Architecture Evolution and Implementation Principles

TiDB Binlog is a tool used to collect the logical changes made to a TiDB cluster and provide incremental backup and replication. This post introduces its architecture evolution and implementation principles.

TiDB 3.0 Beta: Stability at Scale

This post introduces the key new features of TiDB 3.0 Beta.

PingCAP 2018 Year in Review

Thank you for a great 2018!

5 Key Differences Between MySQL and TiDB for Scaling in the Cloud

This post introduces the top five key differences between TiDB and MySQL.

TiDB, the Key to a Better Life for Meituan-Dianping’s 290 Million Monthly Users

As our business grew quickly, we were overwhelmed trying to tackle the mounting data until we found TiDB, a MySQL-compatible NewSQL hybrid transactional and analytical processing (HTAP) database, built and supported by PingCAP. Now we can harness our data with more confidence than ever before and provide better services for our users to enjoy a better life.

TiDB 2.1 GA: Battle-Tested to Handle an Unpredictable World

PingCAP, a leading distributed database company that created the popular MySQL-compatible cloud-native NewSQL database TiDB, announces that TiDB 2.1 is ready for General Availability.

How TiKV Uses "Lease Read" to Guarantee High Performances, Strong Consistency and Linearizability

This post discusses Raft Log Read, `ReadIndex` Read, and Lease Read, and why TiKV adopts the Lease Read approach.

TiDB: Architecture and Use Cases of A Cloud-Native NewSQL Database

This post provides a macro-level overview of TiDB and is the main reference content for readers to orient and dig deeper into other TiDB subjects.

Announcing TiDB Cloud, Managed as a Service and in the Marketplace

This post announces that TiDB Cloud is now available for public preview.

Always Fun, Always On: How TiDB Helps iQiyi Deliver Streaming Videos

As our business grew exponentially, we were overwhelmed trying to handle the mounting data until we found TiDB, a MySQL-compatible NewSQL hybrid transactional and analytical processing (HTAP) database, built and supported by PingCAP. Now we no longer worry about data volume and can bring high-quality entertainment services to our users with more confidence than before.

Launching TiDB Academy, First Course-“Distributed Database with TiDB for MySQL DBAs”

Today, we are excited to launch TiDB Academy, a series of technical training courses and certifications on TiDB and distributed databases in general, taught by our senior technical team.

PingCAP Raises $50 Million in Series C Round, Sets Eyes on Global Expansion, Cross-Cloud Offering, and More Core Technology Investment

PingCAP, a leading distributed database company that created the popular cloud-native NewSQL database TiDB, announces a $50 million Series C funding round led by FOSUN and Morningside Venture Capital. All previous investors—China Growth Capital, Yunqi Partners, Matrix Partners China, and others—have also participated in this round. PingCAP plans to use this new capital to expand the TiDB ecosystem globally, build cross-cloud product offering, and invest in innovation of its core technology.

TiDB Reaches the 200 Contributors Milestone

TiDB recently added its 200th contributor. As CEO and co-founder of PingCAP who began building TiDB three years ago, I would like to thank the entire TiDB community for helping us reach this important milestone!

9 Why's to Ask When Evaluating a Distributed Database

Since PingCAP started building TiDB, it has encountered countless challenges, pitfalls, and critical design choices. Over the past three years, the PingCAP team has answered many questions from the users. Ed Huang, the CTO of PingCAP, has summarized these questions into “9 why’s” that every engineer may ask when looking at a distributed database and gives his answers to these questions in this post to help make decision-making of engineers a bit easier.

Managing the Surging Data Volume of a Fast-Growing Marketplace with TiDB

With our fast-growing business, the surging data volume posed a serious challenge to our backend system and put the operations team under great pressure. How to tackle these challenges became a thorny problem until we found TiDB, a MySQL compatible distributed hybrid transactional and analytical processing (HTAP) database, built and supported by PingCAP. Finally, we do not have to worry about scaling databases and can focus on building better applications for our users.

Landing Your First Rust Pull Request in TiKV

This guide is intended to show how you can land your first Pull Request (PR) in Rust to contribute to TiKV in less than 30 minutes. But before we do that, here’s some helpful background.

TiSpark: More Data Insights, Less ETL

The motivation behind building TiSpark was to enable real-time analytics on TiDB without the delay and challenges of ETL. Extract, transform, and load (ETL)--a process to extract data from operational databases, transform that data, then load it into a database designed to supporting analytics--has been one of the most complex, tedious, error-prone, and therefore disliked tasks for many data engineers. However, it was a necessary evil to make data useful, because there hasn’t been good solutions on the market to render ETL obsolete--until now.

How To Spin Up an HTAP Database in 5 Minutes with TiDB + TiSpark

In this 5-minute tutorial for beginners, we will show you how to spin up a standard TiDB cluster using Docker Compose on your local computer, so you can get a taste of its hybrid power, before using it for work or your own project in production.

Implement Raft in Rust

As an open-source distributed scalable HTAP database, TiDB uses the Raft Consensus Algorithm in its distributed transactional key-value storage engine, TiKV, to ensure data consistency, auto-failover, and fault tolerance. TiDB has thus far been used by more than 200 companies in their production environments in a wide range of industries, from e-commerce and food delivery, to fintech, media, gaming, and travel.

TiDB 2.0 is Ready - Faster, Smarter, and Battle-Tested

TiDB 2.0 is released! We absorbed insights and feedbacks from our customers, listened to requests and issues from our community, and reflected internally on our ultimate vision of building a distributed hybrid transactional and analytical processing database that scales itself, heals itself, and lives in the cloud.

From Chaos to Order -- Tools and Techniques for Testing TiDB, A Distributed NewSQL Database

As an open source distributed NewSQL Hybrid Transactional/Analytical Processing (HTAP) database, TiDB contains the most important asset of our customers--their data. One of the fundamental and foremost requirements of our system is to be fault-tolerant. But how do you ensure fault tolerance in a distributed database? This article covers the top fault injection tools and techniques in Chaos Engineering, as well as how to execute Chaos practices in TiDB.

Blitzscaling the Largest Dockless Bikesharing Platform with TiDB’s Help

Mobike has been using the TiDB database in the production environment since early 2017. Now they have deployed TiDB in multiple clusters with close to 100 nodes, handling dozens of TBs of data for different application scenarios. This post will provide a deep dive on why Mobike chose TiDB over MySQL and its sharding solutions by illustrating how TiDB solves their pain points.

How to do Performance Tuning on TiDB, A Distributed NewSQL Database

Doing performance tuning on distributed systems is no joking matter. It’s much more complicated than on a single node server, and bottlenecks can pop up anywhere, from system resources in a single node or subcomponent, to cooperation between nodes, to even network bandwidth. Performance tuning is a practice that aims to find these bottlenecks and address them, in order to reveal more bottlenecks and address them as well, until the system reaches an optimal performance level. In this article, I will share some best practices on how to tune "write" operations in TiDB to achieve maximum performance.

Bringing TiKV to Rust Devroom at FOSDEM 2018

At the crack of dawn on February 1, I landed in Brussels, Belgium, for the first time in my life. The goal of my trip wasn’t to taste the local cuisine, tour world-famous museums, or grab a pint of the local brew. It was to deliver a talk three days later at FOSDEM 2018 Rust Devroom about our experience at PingCAP using Rust to build TiKV, a distributed transactional Key-Value storage engine.

TiDB DevCon 2018 Recap - News, Latest Development, and Roadmap

On January 20th, 2018, more than 200 coders, hackers, and techies streamed into Garage Café, a chic coffee shop in the heart of Beijing’s techhub, Zhongguancun. They were there to be part of TiDB DevCon 2018, a technology party for the developers, by the developers!

2017 Reflection and Gratitude

Thank you all, our beloved contributors, customers, and partners, for an amazing 2017! Hello, 2018!

Tick or Tock? Keeping Time and Order in Distributed Databases

At re:Invent 2017, Amazon Web Services (AWS) announced Amazon Time Sync Service which is a highly accurate and reliable time reference that is natively accessible from Amazon EC2 instances. It is much like the Google TrueTime which was published in 2012. Why do Google and AWS both want to make efforts to provide global time service? Is there any inspiration for building distributed database? This topic is important to think about.

PingCAP Plants its Seed in Silicon Valley

PingCAP, a cutting-edge distributed Hybrid Transactional/Analytical Processing (HTAP) database company, is excited to announce the opening of its Silicon Valley office, located at the GSV Labs in Redwood City, California.

A TiKV Source Code Walkthrough – Raft Optimization

Paxos or Raft is frequently used to ensure data consistency in the distributed computing area. But Paxos is known for its complexity and is rather difficult to understand while Raft is very simple. Therefore, a lot of emerging databases tend to use Raft as the consensus algorithm at its bottom layer. TiKV is no exception.

PingCAP Launches TiDB 1.0

TiDB is compatible with MySQL, strong consistent and highly available.

Scale the Relational Database with NewSQL

This is the speech Li SHEN gave at the 3rd NEXTCON.

Why did we choose Rust over Golang or C/C++ to develop TiKV?

Every developer has his/her favorite programming language. For the TiKV team members, it's Rust.

RocksDB in TiKV

This is the speech Siddon Tang gave at the RocksDB meetup on August 28, 2017.

Futures and gRPC in Rust

This is the speech Siddon Tang gave at Bay Area Rust Meetup August 2017.

How We Found a Data Corruption Bug in RocksDB

Data was corrupted. A cluster panicked. The crime scene was compromised. What happened? Detective Huang went all lengths to locate the criminal and solved it once and for all.

When TiDB Meets Jepsen

What happens when TiDB meets Jepsen?

The Design and Implementation of Multi-raft

The goal of TiKV is to support 100 TB+ data and it is impossible for one Raft group to make it, we need to use multiple Raft groups, which is called Multi-raft.

How TiDB tackles fast data growth and complex queries for yuanfudao.com

This document is a case study that details the reasons why yuanfudao.com chose TiDB as its backend database solution to tackle their fast data growth and complex queries.

A TiKV Source Code Walkthrough - Raft in TiKV

TiKV uses the Raft algorithm to implement the strong consistency of data in a distributed environment. This blog introduces the details how Raft is implemented.

TiDB Best Practices

This article summarizes some best practices in using TiDB, mainly including SQL usage, OLAP/OLTP optimization techniques and especially TiDB's exclusive optimization switches.

TiDB Internal (III) - Scheduling

This is the third one of three blogs to introduce TiDB internal.

TiDB Internal (II) - Computing

This is the second one of three blogs to introduce TiDB internal.

TiDB Internal (I) - Data Storage

This is the first one of three blogs to introduce TiDB internal.

Refactoring the Built-in Functions in TiDB

In order to accelerate expression evaluation, we recently refactored its framework. This tutorial will show you how to use the new computational framework to rewrite or add a built-in function in TiDB.

Rust in TiKV

This is the speech Siddon Tang gave at the 1st Rust Meetup in Beijing on April 16, 2017.

A Brief Introduction of TiDB

This is the speech Edward Huang gave at Percona Live Open Source Database Conference 2017.

Migration from MySQL to TiDB to handle tens of millions of rows of data per day

This document is a use case that details the performance of MySQL and TiDB with tens of millions of rows of data per day.

About the TiDB Source Code

The target audience of this document is the contributors in the TiDB community. The document aims to help them understand the TiDB project. It covers the system architecture, the code structure, and the execution process.

Adding Built-in Functions

TiDB code is updated and the procedure of adding built-in functions is greatly simplified. This document describes how to add built-in functions to TiDB.

Subquery Optimization in TiDB

Subquery optimization, especially rewriting the correlated subquery, is a very difficult part in SQL query optimization. To be compatible with MySQL, TiDB enables users to write subqueries anywhere they want. For those subqueries that are not correlated, which are also called uncorrelated subqueries, TiDB evaluates in advance; for those correlated subqueries, TiDB removes the correlations as much as possible. For example, TiDB can rewrite a correlated subquery to `SemiJoin`. This article is focused on introducing the correlated subquery optimization methods in TiDB.

MVCC in TiKV

This document gives an overview of MVCC implementation in TiKV.

Travelling Back in Time and Reclaiming the Lost Treasures

This document introduces the History Read feature in TiDB.

A Deep Dive into TiKV

This document introduces how TiKV works as a Key-Value database.

How we build TiDB

This is the speech Max Liu gave at Percona Live Open Source Database Conference 2016.

- Show More Articles -