KNN3 Consolidated Its Web3 Data Stack with TiDB for Up to 30% Cost Savings and Optimized Performance

KNN3 Network is a pioneering Web3 & AI company that develops next-generation businesses and applications, particularly for developers working with multiple blockchains.  Founded just two years ago in 2021, KNN3 has rapidly expanded, introducing innovative product lines tailored to the evolving needs of the Web3 community:

• Typography AI: A platform that helps users learn about web3 and connect with web3 applications.
• TopScore: A platform that calculates users’ credits and reputation on different web3 platforms.
• K.Transformer: A platform that helps developers build data query workflows.

Among them, Typography AI, since its launch in 2022, has attracted over 260,000 registered users and maintained 2,000 to 3,000 daily active users.

The Problem: Growth Challenges with the Original Solution                                  

The diagram below shows the architecture of KNN3’s original database solution. It was a complex combination of multiple databases, including:

 Figure 1. KNN3’s Original Database Solution

• ClickHouse is the primary database to store raw data subscribed from various blockchain networks. 
• PostgreSQL is the secondary database to sync and store processed and converted data from ClickHouse.
• Neo4j is the graph database delivering graph results.
• Amazon S3 and MongoDB –  handle periodic analytical queries.

While building the Typography AI and TopScore, KNN3 encountered several growth challenges:

•  Mixed task of analytical task processing: With a fast-growing user base and business, KNN3 needed to deal with a mix of analytical tasks. On the one hand, they have computation-intensive analytical processing (AP) tasks that require long processing times, such as weekly on-chain scoring services for TopScore. On the other, they need to handle time-sensitive transactional processing (TP) tasks that require real-time feedback, such as graph query service. When these workloads came with significant traffic influx, they posed a challenge to the previous data stack.
• Cost-efficiency with the previous data infrastructure
  • The previous solution was too complicated and costly for KNN3 backend engineers to operate and maintain. 
  • KNN3 had to ensure flexibility in its database schema to accommodate customizable technical requirements from enterprise partners. This resulted in high service costs for customizable enterprise requirements.
• DevOps difficulties – As a Web3 + AI startup, KNN3 always needed to integrate new database technologies without disrupting its existing infrastructure. This presented a challenge in migration and integration in terms of DevOps.  

With the above challenges and the exploding  Web3 data across blockchains, KNN3 needed:

• A more scalable database that can handle Analytical and Transactional workloads 
• The ability to handle vast throughputs of streaming data in realtime. 
• A cost-effective solution that balances the cost, performance, and future flexibility of a database system. 

The Solution: Replacing the Old Infrastructure with TiDB 

After several months of research and testing, the KNN3 team finally integrated TiDB, the distributed SQL database, into their data infrastructure. In their current storage architecture, most KNN3 storage components have been replaced by TiDB Dedicated and TiDB Serverless, the fully managed Database as a Service (DBaaS) offerings of TiDB. While TiDB Serverless is beneficial for experimental and early-stage startups, TiDB Dedicated is an excellent fit for more predictable workloads.

Figure 2. Simplified Data Architecture with TiDB

Raw blockchain data is dumped into ClickHouse and then synced directly to TiDB for real-time transactional and analytical processing. This shortens the data processing flow, improves performance, and decreases cost. 

Shortened Data Flow with TiDB

Why TiDB?

KNN3 chose TiDB for the following reasons:

• Mixed workload processing capabilities: TiDB’s Hybrid Transactional/Analytical Processing (HTAP) architecture allowed KNN3 to handle transactional data processing and analytical data processing simultaneously, reducing DevOps costs and architectural redesigning efforts. This also provides flexibility for future business expansion. 
• Consolidation of workloads with a fully managed solution: With TiDB Dedicated and TiDB Serverless, KNN3 consolidated tasks previously spread across multiple databases (PostgreSQL, MongoDB, Neo4j), significantly easing backend and DevOps workloads. The fully managed nature of TiDB Dedicated and TiDB Serverless in the future relieves the data team from the maintenance burden.
• Scalability and cost efficiency: TiDB Serverless offers automatical scalability and consumption-based pricing, crucial for experimental or early-stage features in a startup company like KNN3. 
• Easy migration and integration:  TiDB Serverless’s developer-friendly nature and MySQL compatibility make it easy to migrate from PostgreSQL and an ideal choice for KNN3’s diverse needs.
• Excellent support from PingCAP. The solid technical support and training provided by the PingCAP team during the migration were beneficial.

The Benefits Brought by TiDB 

• Scalability and cost-efficiency:  The automatical scalability and pay-as-you-go pricing of TiDB Serverless cater to the expanding project needs and evolving business requirements of KNN3. This has brought a 30% cost decrease for early development stages and experimental projects.
• Enhanced user experience: TiDB’s integration allows for more efficient data processing,  improving user experience within all KNN3’s platforms. 
• Operational efficiency:  The consolidation of databases by introducing TiDB Dedicated and TiDB Serverless significantly streamlined KNN3’s operational processes. 
• A future-ready solution: The above benefits and features of TiDB satisfy KNN3’s growing data storage requirements while ensuring a future-ready solution.

Looking to the future

Using TiDB Dedicated and TiDB Serverless, KNN3 simplified its storage architecture, lowered its storage costs, and optimized its service performance. In the future, KNN3 expects to replace the remaining storage components with the TiDB solution.