KNN3 Network is a pioneering Web3 company that provides a one-stop Web3 user-centric DataFi solution for decentralized apps (d/Apps) and smart contracts. It allows d/Apps and smart contracts to interact with cross-platform user-centric data in multiple algo-friendly ways.
Urgent storage requirements
As the Web3 data is exploding every second on every blockchain, it is not easy for KNN3 to subscribe and consolidate the data from various blockchain networks. They need:
- A more scalable database.
- Hybrid Analytical and Transaction Processing capability. Analytical queries are increasing but some data is too large to be consumed transactionally. So a database that can deal with hybrid workloads is required.
- The ability to handle huge throughputs of streaming data in real time.
- A cost-effective solution. KNN3 is a Web3 startup. They need to balance the cost, performance, and future flexibility of a database system.
Pain points with the previous database solution
The diagram below shows the architecture of KNN3’s previous database solution. It was a complex combination of multiple databases including ClickHouse, PostgreSQL, Neo4j, Amazon S3, and MongoDB.
KNN3 Network’s previous database architecture
This solution had the following roles:
- ClickHouse was the primary database to store raw data subscribed from across various blockchain networks.
- PostgreSQL was the secondary database to sync and store processed and transformed data from ClickHouse.
- Neo4j was a database for delivering graph results.
- Amazon S3 and MongoDB handled periodic analytical queries.
This solution worked for a while. But as the blockchain data size spiked and the business grew, this architecture suffered.
- It was too complicated for KNN3 backend engineers to operate and maintain—especially for a small engineering team with four people.
- It could not handle huge throughput of streaming data in real time.
- It could not provide real-time data insights to end users because Amazon S3 backend is slow in data writes.
- It was costly, with too many storage components combined. The maintenance cost was also high.
Replacing the old infrastructure with TiDB Cloud
After several months of research and testing, the KNN3 team finally chose TiDB Cloud. In their current storage architecture, a majority of KNN3 storage components have been replaced by TiDB Cloud.
TiDB Cloud has greatly simplified the storage architecture
Raw blockchain data is dumped into ClickHouse and then synced directly to TiDB for both real-time transactional and analytical processing. This shortens the data processing flow, improves performance, and decreases cost.
Shortened data flow
The benefits brought by TiDB Cloud
A simpler and easy-to-use storage architecture
Currently, TiDB Cloud has replaced at least four to five components of KNN3’s legacy database system, and shortened the data processing flow. Now, it only takes two engineers (at most) to operate and maintain the TiDB solution.
Hybrid transactional and analytical capabilities
TiDB Cloud has a two-in-one architecture, or an HTAP architecture. Transactional and analytical workloads do not have to be separately processed by different databases anymore. One TiDB system can handle both workloads at the same time.
Much lower costs
TiDB Cloud replaced most of KNN3’s storage components, which cut their cost significantly. In addition, engineers now maintain fewer databases, so the maintenance cost is lower too.
Seamless, horizontal, transparent, and automatic scalability
TiDB Cloud can scale out elastically and easily based on changes in real-time read and write traffic. This perfectly fits KNN3’s business needs because the Web3 industry is booming. When their business expands, they don’t have to redesign their storage architecture.
Cloud-native design for future integration
TiDB is a cloud-native database, which makes it simple and easy to integrate with the ecosystem. This paves the way for future expansion and cooperation with various partners.
Try TiDB Cloud Free Now!
Get the power of a cloud-native, distributed SQL database built for real-time
analytics in a fully managed service.
Looking to the future
By using TiDB Cloud, KNN3 simplified its storage architecture, lowered its storage costs, and optimized its service performance. In the future, KNN3 expects to rely on TiDB Cloud even more and to totally replace the rest of their remaining storage components with TiDB Cloud.
This article is created based on a talk given by Thomas Yu at the HTAP Summit 2022.
