Scaling Beyond Limits: How EasyPost Future-Proofed Its Infrastructure with Distributed SQL

When legacy infrastructure hits its limits, innovation isn’t just a choice — it’s a necessity. For EasyPost, a leader in logistics technology, scaling up meant more than just more hardware. It meant a fundamental shift in their approach to data. 

In this customer story, we explore how EasyPost’s infrastructure team, led by Head of Infrastructure Adam Firester, navigated a complex scaling challenge and found their answer in TiDB, a powerful distributed SQL database.

The Scaling Up Challenge: A 32-Terabyte Problem

EasyPost, the all-in-one shipping API platform that connects e-commerce companies with major carriers like USPS, UPS, and FedEx, has grown substantially since its founding in 2012. At the center of this growth was a legacy MySQL 5.7 monolith — now a 32-terabyte (TB) beast with serious limitations.

“We’d been talking about sharding for over a decade,” said Firester. “But nothing ever tackled the core issue: Shard zero just kept growing. We were hitting the max local storage on our IBM bare metal servers. Backups would run over each other. DDL changes took hours. We were reaching the end of our runway.”

The company needed more than incremental fixes. They needed a transformation.

Evaluating the Options

EasyPost’s team considered the classic trio of solutions for scaling up its data infrastructure:

1. Sharding: Complex and ineffective against their primary bottleneck. Sharding is a database architecture pattern that involves splitting a database into multiple smaller databases, or shards. While sharding can be an effective way to scale a database, it is also a complex solution that can introduce a number of challenges. In EasyPost’s case, sharding was ineffective because it did not address the core issue of shard zero’s continuous growth.
   
2. Outsourcing the problem: Ideal in theory, but not aligned with their current ops. Outsourcing the problem to a cloud provider could have been a potential solution, but it was not aligned with EasyPost’s current operations.
   
3. Replatforming to a new database: Risky, but potentially the most impactful. Replatforming to a new database would be a risky move with the possibility of substantial code refactoring, but it was also the solution with the most potential impact.

After an extensive review of options across these categories, one solution stood out: TiDB.

“We chose TiDB because it aligned with both our current constraints and future ambitions,” Firester explained. “We could run it on bare metal now, but it’s cloud-ready. That gave us optionality.”

Scaling Up: Why TiDB?

Several features made TiDB the clear front-runner:

• Horizontal scalability: EasyPost needed a solution that could grow with them. TiDB is a horizontally scalable database, meaning it can be easily scaled by adding more nodes to the cluster. This was important for EasyPost because they needed a solution that could handle their growing data volume.
• MySQL compatibility: Allowed for easier migration and avoided the need for a major upgrade to MySQL 8. TiDB is compatible with the MySQL protocol, which means that EasyPost could easily migrate their data from their legacy MySQL database to TiDB without making any major changes to their application code.
• Time-to-Live (TTL): Critical for aging out old data from the oversized shard zero. TiDB supports Time-to-Live (TTL), which allows data to be automatically deleted after a certain period of time. This was important for EasyPost because they needed to age out old data from their database to free up space.
• Support for ACID transactions and hybrid workloads: Enabled EasyPost to handle online transaction processing (OLTP) and online analytical processing (OLAP) from the same system. TiDB also supports strong ACID transactions. Both were important for EasyPost as they required a database that could handle both workload types with strong data integrity.
• Strong commercial support from PingCAP: This is essential for navigating the unknowns of production readiness. PingCAP is the company behind TiDB, and it provides strong commercial support. This was important for EasyPost just in case they encountered any problems during the migration or operation of their TiDB cluster.

The Proof of Concept

Rather than risking disruption in production, the team set up a parallel TiDB cluster: 18 TiKV nodes, 3 PD nodes, and 3 TiDB nodes with ProxySQL in front, mirroring their MySQL environment.

Figure 1. EasyPost’s original solution architecture for the PoC containing TiDB.

Their goal was simple: replicate real-world conditions, then stress test it. ProxySQL mirrored reads at first, then full read/write transactions. By tapping into ProxySQL logs, they could analyze top query performance and identify opportunities for tuning.

The POC wasn’t without its bumps. The TiDB nodes initially suffered from out-of-memory crashes due to disabled garbage collection by default. “Once we enabled Go’s memory limits and tuned query timeouts, we stabilized things,” said Firester. “Most of the early issues had more to do with our own technical debt than the platform itself.”

PingCAP’s responsiveness was another big win. “We reported three bugs. Two had fixes within a week. That’s lightning-fast in vendor time,” Firester added.

Scaling Up: Unexpected Lessons and Wins

The journey surfaced some hidden landmines and surprising performance gains.

• Hotspot Issues: TiDB’s default use of auto-increment keys introduced latency. Removing the auto-increment attribute from key tables improved P80 insert times by 90%.
  
• Missing Indexes: Just like in MySQL, missing indexes crippled performance. Fixing them brought dramatic gains.
  
• Query Plan Surprises: The optimizer didn’t always choose the best execution plan, sometimes opting for full table scans. Using the plan replayer helped, though it required patience.
  
• Partition Performance: MySQL-style partitioning didn’t carry over well. Dropping partitions and reloading data slashed latency.
  
• TTL Missteps: Dumping years of data in one go backfired. Now, the team plans to purge iteratively: Seven years back, then five, then three.
  

Still, the wins far outweighed the pains. “Some DDL changes that used to take hours? Instant now,” Firester emphasized. “I’ve never seen anything that fast.”

Putting TiDB Through the Holiday Gauntlet

One of the boldest moves was keeping the TiDB cluster live through EasyPost’s busiest time of the year: The holiday peak season.

“We mirrored traffic through ProxySQL during peak season as a trial-by-fire. We were ready to pull the ripcord if things went south. But we didn’t have to. It stood up the whole time — rock solid,” Firester said.

Based on peak load analysis, they scaled TiDB compute nodes 3x for holiday traffic but later realized backend storage needed less capacity than anticipated. TiDB proved not only stable but efficient.

Still to Come: Active-Active Ambitions

EasyPost isn’t done. With the POC successful and production rollout in view, the next steps include:

• Addressing locking differences: MySQL uses pessimistic locking to prevent duplicate shipment charges, something TiDB handles differently.
• Initial deployment in active-passive mode: This mirrors the current MySQL setup and buys time for confidence-building.
• Active-active deployment (long-term goal): TiDB supports this model, enabling a resilient, highly available future state.

Advice for Others

Firester shared a few key lessons for teams embarking on a similar journey:

• Understand the knobs: Spend the time learning how to tune TiDB.
  
• Analyze your queries: What worked in MySQL may need rethinking.
  
• Be ready to refactor: Hardware sizing, indexing strategies, partitioning — all may change.
  
• Don’t install monitoring on functional nodes: Isolate Prometheus and others from production workloads.
  
• Purge data gradually: Especially when using TTL — don’t bite off too much at once.
  
• Lean on PingCAP: “They’ve been great partners,” Firester said. “Responsive, invested, knowledgeable.”

Conclusion

EasyPost’s move toward TiDB isn’t just a story about migrating databases — it’s a blueprint for modernizing infrastructure in a high-growth, high-demand environment. From monolithic pain to scalable promise, their journey highlights what’s possible when a team combines practical engineering with forward-thinking architecture.

As EasyPost gears up for full production deployment, they’ve already proven something crucial: With the right platform and partner, even a 32 TB legacy system can be transformed into a flexible, modern data foundation.

Want to learn how your company can get started scaling up TiDB the right way? Book a tech talk with one of our distributed SQL experts.