Today, our team at PingCAP is excited to announce that we are open-sourcing our own binlog implementation, TiDB Binlog. TiDB Binlog is a tool we've been developing in-house (until now) to collect binary log data from TiDB server and provide real-time data backup and replication.
It's similar in functionality to MySQL master-slave replication, but the main difference is that because TiDB is a distributed database, the binlog generated by each TiDB instance needs to be merged and sorted according to the time of the transaction commit before being consumed downstream.
Many of our users have been using TiDB Binlog in-production for incremental backups, to replicate data from one TiDB cluster to another (often across multiple data centers or availability zones), or to send TiDB updates through Kafka to any given downstream platform. It's particularly useful if you are a MySQL or MariaDB user thinking about trying out TiDB, but would like to minimize the risk of migration and testing. TiDB Binlog enables application traffic to TiDB to be pushed to a downstream MySQL or MariaDB instance/cluster. This ensures that even if the migration to TiDB encounters problems, you can easily revert the application back to MySQL or MariaDB.
(For more detailed information on the architectural design and implementation of TiDB Binlog, read this deep-dive blog post.)
TiDB Binlog has become a popular tool among our users, as TiDB itself continues to gain popularity. Whether it's replicating data between two data centers to provide disaster recovery, or evaluating TiDB in a more risk-free manner, TiDB Binlog is an essential part of that experience. In order to deliver better testing and in-production experience to more users, we believe open-sourcing it is the best approach. As a company that lives and breathes the open-source ethos, we firmly believe that more openness in general is always better than less.
To help you “kick the tires” now, we've put together this tutorial to help you deploy and get a feel for how TiDB Binlog operates.
We look forward to your issues, PRs, and contributions. As always, we appreciate everything our open-source community has done to make TiDB what it is today.