How to Choose the Right Database for Your NFT Marketplace

The NFT market is booming

An NFT, or non-fungible token, is a blockchain-based digital asset. Unlike Bitcoin and Ether, two world leading cryptocurrencies that are interchangeable and fungible, an NFT represents the unique ownership of a digital item such as a painting, a song, a video, and even a tweet. It cannot be copied, substituted, or subdivided, but it can be held, transferred, and traded. 

The recent years have witnessed the booming of the NFT market. In 2021 alone, the NFT trading volume hit more than $17 billion, a 21,000% increase over 2020. Tens of thousands of artists have produced and traded NFT artworks. NFT marketplaces such as Opensea and Coinbase that offer platforms for NFT transactions are springing up like mushrooms after a rain. 

The traditional data architecture of an NFT marketplace

An NFT marketplace consists of multiple components including digital wallets, smart contracts, databases, and blockchains. One of the most crucial components is the database. It stores, processes, and analyzes massive amounts of blockchain data. If you are a new participant in the Web3 or NFT ecosystem and plan to build an NFT platform yourself, how will you design your data architecture and choose your database?

Traditional data architectures need at least the following elements: 

• A blockchain subscriber
  Because there are huge amounts of NFT transactions happening per second on multiple blockchains such as Ethereum, Solana, Polygon, and BNB Chain, you’ll need a blockchain subscriber to collect all those NFT transactions. 

• A message queue (MQ) system like Kafka and an MQ reader
  A message queue system like Kafka helps you receive and channel the NFT transaction data you collect from blockchains. Then, an MQ reader consumes the messages asynchronously in the message queue and bulk inserts them into an analytical database. 

• An OLAP database
  An Online Analytical Processing (OLAP) database such as the combination of Apache Hive and Apache Spark helps you analyze the NFT transactions and respond to analytical queries.

• An OLTP database
  An Online Transactional Processing (OTLP) database like MySQL stores the analytical results returned by the OLAP database and handles the requests from the web server of your NFT platform.

• A web server
  You’ll also need a web server to run your NFT platform. 

Traditional data architecture for an NFT marketplace 

Pain points you’ll probably encounter 

When you implement, run, and operate the traditional data system for your NFT marketplace, you’ll probably meet the following pain points. 

• You can hardly get real-time NFT deals. Every second, numerous NFT items are created and traded on each blockchain. Your NFT marketplace must be agile enough to get the most up-to-date information in the NFT market. The faster your platform reacts, the more accurate NFT deals your users can get. But the existing data architecture has a long data pipeline with a complex technology stack, which undermines the agility of your platform. 
• The size of the NFT data stored in blockchains is enormous and is growing continuously. If you store all existing NFT transactions from mainstream blockchains along with their metadata, you’ll need at least three terabytes of storage space, even after the data is compressed. Although most data warehouses can hold such huge amounts of data, most OLTP databases struggle to do so. 
• The technology stack is too complex. If you choose the traditional data architecture, you’ll need a big data system (let’s say Apache Hive) to import the historical NFT data from blockchains to your NFT marketplace storage system. You also need to build an extra program to subscribe to real-time transaction data separately from blockchains or the NFT API, write it to a message queue, and then batch write it to the big data system. What’s more? You need an OLTP database to store the analytical results from Apache Spark. Not only is this expensive, but it also undermines your platform agility. 

Why TiDB? 

TiDB is an open-source, distributed, MySQL compatible database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is also horizontally scalable, strongly consistent, and highly availabilable. 

TiDB can solve the pain points of NFT marketplaces because it can: 

• Handle both real-time writes and analytics. TiDB stores data in both row and column format, and the data is strongly consistent. TiDB can analyze real-time data while accepting high-workload writes. It can calculate analytical information such as NFT rankings in real time.
• Scale out easily. TiDB can horizontally scale out on demand to adapt to your rapidly growing volume of the blockchain data. 
• Simplify the technology stack. TiDB can replace the Apache Kafka and Hive parts of the traditional data architecture, accept high QPS writes, and store large amounts of data. It is also a perfect replacement of MySQL, which can directly query the analytical results in real time and display them on the NFT marketplace. With TiDB, you can quickly develop and release your minimum viable products.
• Support JSON format. The most common format for NFT metadata is JSON. TiDB fully supports the JSON format, so it can fetch and store the blockchain data directly without predefining the metadata schema.  

The new data architecture with TiDB for an NFT marketplace 

In addition to the on-premises version of TiDB, you can also experience the same capability with more flexibility on TiDB Serverless, a fully managed Database-as-a-Service (DBaas) for auto-scaling workloads built with TiDB.

TiDB Cloud architecture and how TiDB Cloud supports applications

How a leading NFT participant uses TiDB 

One of TiDB adopters is a leading participant in the NFT market. They track NFT transactions on dozens of public chains and help NFT users discover investment opportunities with real-time NFT information. 

They previously used a data warehouse to store all their NFT transaction data, made daily batch analytics, and then wrote the analytical results to an OLTP database. With this data solution, their data latency was as long as one day. Since they migrated data to TiDB, the data latency was reduced from one day to seconds. Their data volume has tripled to almost 20 terabytes. Thanks to its elastic scalability, TiDB could easily support their huge data increase. 

In addition, by adopting TiDB, they get a much more simplified data architecture, which greatly reduces their engineering, operation, and maintenance costs. 

Try our NFT demo now!

We made a simple NFT demo—-TiDB Cloud NFT Quickstart with JavaScript and TiDB Cloud. Before you start your own NFT products, you can try this demo first and learn what TiDB Cloud can do and how it performs.

This demo can: 

• Scan historical NFT transactions on Ethereum and track real-time transactions. 
• Browse the hottest items in the NFT market. 

An NFT demo: TiDB Cloud NFT Quickstart 

If you are interested in the demo source code, you can find it here on GitHub. 

Keep in contact with us!

You’re welcome to contact our sales staff if you are interested in TiDB, or you can spin up a TiDB Serverless cluster free with only a few clicks. You can also join our community on Slack and TiDB Internals to share your thoughts with us. Don’t forget to follow us on Twitter, LinkedIn, or GitHub for the latest information. 

Keep reading: 
How to Efficiently Choose the Right Database for Your Applications
Technical Paths to HTAP: GreenPlum and TiDB as Examples
How to Process and Analyze Billions of Rows of Data in a Single Table with TiDB on AWS

Edited by: Fendy Feng, Tom Dewan