How Data Engineering is Powering Pinterest’s Global Platform

The following post is an excerpt from Technology Magazine’s feature article, How data engineering is powering Pinterest’s global platform.

Pinterest had long recognized the need to streamline and optimize its data storage system.

Using HBase, the image sharing and social media platform was carrying a large footprint, with more than 50 clusters and a total data size of no less than one petabyte (PB). 

That’s where PingCAP came in.

PingCAP launched in 2015 when three seasoned data infrastructure engineers grew tired with the way databases scaled while working at leading tech companies.

Seeing no capable solutions on the market, they built one from scratch themselves.

TiDB, their flagship project, is an advanced, open-source distributed SQL database for powering modern applications with elastic scaling, real-time analytics and continuous access to data. Under the hood, it can be used for large-scale applications reaching several-hundred terabytes. 

What was Pinterest’s data engineering team trying to achieve?

The objective for Pinterest was to try and address the limitations of its existing HBase system, which was mainly being used for user-facing applications.

“What Pinterest’s storage and caching team wanted was to find their next-generation, unifying storage system,” explains Liquan Pei, Principal Technologist at PingCAP. “As a NoSQL database, HBase offers a very simple key value interface, but the business logistics are complex. 

“When Pinterest wanted to add some new features, they had to constantly build additional layers on top of HBase, just to try and avoid its limitations. Multiple components make up this system, which incurs a very high maintenance workload.”

With those motivations in mind, Pinterest evaluated more than 15 database solutions – including in-house and open-source – and eventually settled on TiDB back in 2020. Pei says the reason for Pinterest choosing PingCAP was two-fold: TiDB’s outstanding technical capabilities and PingCAP’s high-quality enterprise support. 

By nature, TiDB offers SQL and ACID transactions, giving Pinterest the opportunity to consolidate their tech stack, enhance development velocity, and reduce costs. 

From a support perspective, PingCAP offers real-time response and ensures Pinterest understands how to get the best out of TiDB to meet their personal requirements. 

PingCAP brings lasting benefits to Pinterest data operations

The expectation is that TiDB will bring a host of benefits to Pinterest’s day-to-day operations – both immediately and over the coming years.  

When carrying out the project, PingCAP evaluated Pinterest’s secondary index services system, IXIA, by implementing the safe functionality provided by IXIA on TiDB. Straight away, PingCAP’s technicians achieved 80% cost reduction, strong consistency and better latency compared to the previous system. 

“Because of TiDB’s capabilities, we were able to reduce the system from six components to only one, greatly reducing the maintenance burden,” adds Pei.  

In the long run, TiDB’s expressiveness and scalability should also help Pinterest’s IT teams from a practical perspective. 

“As a company, I think the team will be able to move faster; they won’t have to stay up all hours of the night handling incidents,” continues Pei. “People from Pinterest will enjoy peace of mind because TiDB handles a lot of work and they can focus on more impactful work instead.”

Click here to read the entire Technology Magazine article.