What’s Next for SQL @ Scale? TiDB’s Vision to Evolve Growing Businesses

Scaling database systems remains one of the biggest challenges facing growing companies modernizing IT infrastructure. Max Liu, Co-Founder and CEO of PingCAP, discussed these critical scaling challenges during his opening keynote, “What’s Next for SQL @ Scale,” at HTAP Summit 2024. Drawing from real-world examples and industry insights, Liu explored how TiDB addresses the limitations of traditional databases, such as MySQL, and explained why scaling is a key factor for business success. 

This blog is a quick summary of the core aspects of his talk. It highlights the evolution of database scaling, TiDB’s unique capabilities, and the increasing role of AI in this space.

Database Scaling in the Real World: The Need for Change

Liu kicked off his presentation by addressing the stark realities rocking companies as they manage increasing amounts of data. He mentioned that TiDB, an open source, distributed SQL database, now manages more than 50,000 clusters. He also highlighted a customer use case where a single TiDB cluster handles 1.2 petabytes (PBs) of data and 600,000 tables. This kind of scale demonstrates TiDB’s capacity to handle rapidly growing workloads. It also serves as a reminder that managing large data volumes requires careful strategy and technological infrastructure.

Many companies struggle when they reach a certain threshold of scale. Initially, databases perform well with small datasets. However, as companies grow and handle more traffic, the strain on the system becomes noticeable. Liu pointed out that while SQL works effortlessly with small-scale datasets, it begins to falter when scaling out to hundreds of terabytes (TBs) or more. This is where traditional databases hit a wall—and TiDB’s distributed SQL architecture steps in to provide much-needed horizontal scalability without compromising performance.

Identifying the Real Bottleneck: It’s Not the Code, It’s the Scale

Liu emphasized that, often, scaling challenges arise even when no changes have been made to the underlying code. As systems grow, they must adapt to more users, more transactions, and increased complexity, all of which put stress on traditional databases. He described this phenomenon as a “shift in scale” — what worked perfectly at one point starts to break down when the system scales up. Scaling, as Liu explained, doesn’t just mean handling more data. It also involves ensuring high performance across multiple dimensions, including query performance, response times, and data consistency.

To solve these challenges, Liu stressed the importance of moving away from manual sharding, which many traditional databases rely on to scale. Manual sharding introduces significant complexity into the system and often leads to operational inefficiencies. TiDB’s distributed architecture, by contrast, allows companies to scale out and in seamlessly without the need for manual intervention.

TiDB’s Strengths: Simplifying Database Scaling and Ensuring Performance

Liu outlined several key advantages of TiDB’s architecture, all of which allow it to address the limitations of traditional databases:

1. Horizontal Scalability: With TiDB, companies can easily add nodes to their clusters to scale horizontally. This means that businesses can grow their infrastructure as needed without having to worry about the limitations of vertical scaling. These limitations include hitting hardware limits or experiencing degraded performance.

2. ACID Transactions Across Distributed Nodes: One of TiDB’s standout features is its ability to support ACID (Atomicity, Consistency, Isolation, Durability) transactions across a distributed architecture. Many databases sacrifice consistency for performance when scaling out. However, TiDB maintains transactional integrity, no matter how large the dataset grows.

3. Zero Downtime Scaling: A core tenet of TiDB’s design is the ability to scale without taking the system offline. This is a crucial feature for businesses operating in 24/7 environments where even a few minutes of downtime could lead to significant losses.

4. Unified SQL Interface: While TiDB operates in a distributed environment, it retains the familiar SQL interface. This makes it easy for developers and engineers to work with the system. This reduces the learning curve and allows for faster deployment.

AI’s Role in the Future of Database Scaling

Liu didn’t just focus on the present. He also discussed the future of databases, particularly related to artificial intelligence (AI). As companies increasingly integrate AI into their systems, the need for scalable, high-performance databases becomes even more pressing. AI models generate vast amounts of data, and this data must be processed, analyzed, and stored efficiently. 

Liu argued that AI accelerates the process of hitting scalability issues. Since AI workloads are so data-intensive, companies are more likely to encounter scalability challenges earlier in their growth. TiDB, with its distributed architecture and ability to handle PBs of data, is well-suited for the growing demand in AI-driven industries.

Conclusion

As Liu stated, TiDB allows companies to follow their intuition when working with SQL, regardless of the scale they are dealing with. Whether managing GBs or PBs, TiDB removes the barriers traditionally associated with scaling. It lets businesses focus on what matters most: innovation and growth.

The success of companies like Databricks, Pinterest, and Plaid, all of whom use TiDB to power their data-intensive applications, highlights the real-world impact of our database technology. As the demand for scalable databases continues to grow, TiDB stands out as a powerful solution. It’s capable of handling the complexities of modern business applications without sacrificing performance or reliability.

Ensuring database scalability is no longer optional—it’s essential. TiDB’s approach to distributed databases provides a robust foundation for companies looking to scale their operations without getting bogged down by the complexities of traditional database management.

Want to scale your data beyond traditional databases while minimizing complexity? Register to watch Liu’s entire keynote for additional insights on how to get started. Happy viewing!