Ping An Technology Boosts Performance and Cuts Costs by Migrating from Oracle to TiDB 

Ping An Technology is a wholly-owned subsidiary of Ping An Group dedicated to driving technological transformation within the company. It leverages cutting-edge technologies such as artificial intelligence (AI), blockchain, cloud computing, and intelligent cognition to transform Ping An Group into a technology-driven financial conglomerate. As the core of the group’s high-tech operations and tech incubator, Ping An Technology develops and manages critical platforms and services for the company.

Its primary focus areas include finance, healthcare, and smart city solutions, applying internationally certified technological capabilities to real-world business scenarios.

The Challenges: Overcoming Obstacles to Scalability, Performance, and Affordability

Ping An Technology relied heavily on Oracle databases for critical systems such as the Unified Payment System and the Workflow System. Despite Oracle’s robust capabilities, Ping An encountered several critical challenges as the demands on their data infrastructure grew:

• Scalability Limitations: Ping An Technology’s data volume rapidly expanded, and Oracle’s vertical scaling model became increasingly inefficient. Expanding storage or compute resources required costly and time-consuming hardware upgrades, which led to operational bottlenecks and limited the company’s ability to scale flexibly.
• High Operational Costs: Oracle’s licensing and maintenance costs have become a significant financial burden. As data volumes grew, the need for frequent hardware upgrades, expensive storage solutions, and complex management tools led to spiraling costs, negatively impacting the overall cost-efficiency of their operations.
• Performance Degradation: The increasing volume of transactions in Ping An’s Unified Payment System led to noticeable performance degradation. High-latency issues emerged during peak processing periods, impacting the speed and efficiency of financial transactions. This degradation also affected their Workflow System, where delays in processing critical business tasks became more frequent.
• Limited Flexibility: Ping An’s existing Oracle-based architecture lacked the flexibility to handle rapidly changing data models, distributed environments, and real-time analytics. The rigid data schema and reliance on vertical scaling reduced the company’s agility in responding to evolving business requirements.

Original Data Architecture

The original data architecture at Ping An Technology was structured around a traditional relational database system, primarily based on Oracle. This architecture consisted of several critical components:

1. Database Management System: The core of the architecture was the Oracle Database, which facilitated data storage, management, and retrieval. This system was well-suited for transactional processes and supported complex queries essential for financial services.
2. Data Warehousing: A separate data warehouse was implemented to consolidate data from various sources across the organization. This allowed for analytical processing and reporting, enabling Ping An to make informed decisions based on historical data analysis.
3. Application Layer: The application layer included multiple services and platforms developed to meet specific business needs. These applications interacted with the Oracle database for real-time data access, ensuring operational efficiency in customer-facing services.
4. Integration Components: The architecture featured integration tools to connect disparate systems and ensure data flow between applications. These tools played a vital role in synchronizing data and maintaining consistency across the enterprise.
5. Data Security and Compliance: Stringent security measures were in place to protect sensitive customer data following regulatory requirements. These included access controls, encryption, and regular audits to ensure compliance with industry standards.
6. Scalability Challenges: While the original architecture had served the organization well initially, it began facing challenges related to scalability. As data volumes increased, the ability of the Oracle system to efficiently handle high transaction loads and complex queries became a concern.
7. Cost and Maintenance: Maintaining the Oracle database and associated infrastructure required significant resources, both in terms of financial investment and skilled personnel. This highlighted a need for a more cost-effective solution that could offer greater flexibility and ease of management.

This structured approach to data management laid the foundation for Ping An Technology’s operations. However, the growing demands for scalability, real-time data processing, and cost-effectiveness prompted the exploration of more advanced database solutions, ultimately leading to the implementation of TiDB.

The Solution: UbiSQL Built on Top of TiDB 

Ping An Technology introduced UbiSQL as a strategic solution aimed at “de-Oracleization” and enhancing the overall data management framework within the organization. Built on the robust architecture of TiDB, an open source, distributed SQL database, UbiSQL was tailored to meet the diverse and dynamic needs of Ping An’s operations across finance, healthcare, and smart city initiatives. 

The following sections elaborate on the comprehensive features and capabilities of UbiSQL:

1. Distributed Architecture

UbiSQL is designed with a distributed database architecture, allowing seamless horizontal scaling. This is crucial for accommodating the increasing data volumes generated by Ping An’s extensive business operations. Key aspects of this architecture include:

• Data Sharding: Data is partitioned and distributed across multiple nodes, which optimizes performance and enhances query efficiency by parallelizing data access. Each shard can operate independently, enabling faster read and write operations.
• Load Balancing: The system automatically distributes incoming requests across available nodes to prevent any single node from becoming a bottleneck. This load-balancing capability ensures consistent performance even during peak usage periods.

2. Compatibility with SQL Standards

UbiSQL maintains high compatibility with SQL, allowing it to integrate seamlessly with existing applications initially designed for Oracle. This compatibility means that developers and data analysts can continue to use familiar SQL syntax, reducing the learning curve and minimizing the need for extensive rewrites of existing codebases.

• Advanced SQL Features: UbiSQL supports a wide range of SQL features, including joins, transactions, and complex queries, allowing users to leverage their existing skills and tools effectively.

3. Real-Time Data Processing

UbiSQL enables real-time data processing, which is essential for Ping An’s operations and requires immediate insights and analytics. Key capabilities include:

• Real-Time Analytics: The architecture allows for continuous data ingestion and analysis, enabling businesses to derive insights and make decisions based on the most current data available.
• Event-Driven Architecture: UbiSQL supports an event-driven approach, allowing businesses to respond to data changes in real-time, enhancing operational agility.

4. High Availability and Fault Tolerance

To ensure business continuity, UbiSQL incorporates several features designed to enhance availability and fault tolerance:

• Data Replication: Data is replicated across multiple nodes, ensuring that an alternate copy of the data is readily available in the event of a failure. This redundancy minimizes the risk of data loss and downtime.
• Automatic Failover: In the event of a node failure, UbiSQL automatically redirects requests to other operational nodes without user intervention, ensuring that services remain uninterrupted.

5. Enhanced Data Security

Recognizing the sensitivity of financial and healthcare data, UbiSQL prioritizes security through multiple layers of protection:

• Access Controls: Granular user permissions are implemented to control access to sensitive data, ensuring that only authorized personnel can perform specific actions on the database.
• Data Encryption: Both data at rest and in transit are encrypted, safeguarding sensitive information against unauthorized access and breaches.

6. Integrated Management Tools

UbiSQL comes equipped with integrated management and monitoring tools that facilitate effective database administration:

• Performance Monitoring: Built-in tools allow administrators to monitor system performance in real time, enabling proactive identification and resolution of potential issues before they impact operations.
• Automated Backups: Regular automated backups ensure that data can be restored quickly in case of corruption or loss, further enhancing data reliability and safety.

7. Customizability and Extensibility

UbiSQL is designed to be adaptable, allowing for customization based on specific business needs. This flexibility is vital for Ping An, which operates across multiple sectors with varying requirements:

• Plugin Architecture: The system supports a plugin architecture, which enables the addition of custom modules and features as business needs evolve.
• APIs for Integration: UbiSQL provides APIs allowing easy integration with other systems and services, facilitating data exchange and collaboration across different business units.

De-Oracleization in Action

Ping An Technology’s migration from Oracle to UbiSQL involved a systematic approach focusing on high availability and data integrity. UbiSQL’s architecture, utilizing F5 load balancing and Raft protocol for strong consistency, was distributed across three data centers, with asynchronous replication to a remote cluster for disaster recovery.

Key preparatory steps included a thorough assessment of the Oracle environment, covering database load, application compatibility, and synchronization methods. Post-assessment, a database selection process weighed RDBMS, open-source, and distributed options, followed by an application decoupling and data synchronization design.

Migration was labor-intensive, requiring 10 developers over two months to convert 20,000 lines of Oracle code into Java. Testing phases included full regression, performance, and stress testing. Traffic replication and replay methods facilitated seamless validation by simulating production traffic in UbiSQL, ensuring application reliability. Data consistency was rigorously checked using Ludbgate, which supported full and incremental data comparisons, ensuring successful transition and robust system integrity in production.

Tangible Outcomes and Operational Enhancements Achieved through TiDB Implementation

Implementing TiDB through UbiSQL has yielded several tangible results for Ping An Group, significantly enhancing the performance and operational efficiency of its core business systems. The following achievements highlight the impact of the migration:

• Improved Performance Metrics: After migrating to UbiSQL, performance improved for COUNT statements, with only a minimal performance decrease (under 10%) for other queries and inserts, which met application standards. 
• Successful Migration of Critical Systems: The migration process enabled the seamless transition of key systems, including the Unified Payment System, the CF2 Customer Relationship System, and the Workflow System. These systems are essential for managing financial transactions and customer interactions, and their successful migration underscores the reliability of the new architecture.
• Enhanced Data Synchronization: The implementation facilitated strong consistency in data synchronization across three data centers, utilizing UbiSQL’s robust architecture. This architecture supports business continuity and disaster recovery, ensuring operations remain uninterrupted even in adverse situations.
• Increased Testing Efficiency: The integration of traffic replication and replay allowed for comprehensive application and database architecture validation, significantly reducing the risks associated with deploying new technology. This proactive approach ensured that the migrated systems could handle production-grade workloads effectively.
• Strategic Cost Savings: The methodologies developed during the migration, particularly traffic replication for stress testing, have established a framework that can be reused in future de-Oracleization projects. Strategic cost savings were achieved by storing cold data on SATA disks and using traffic replication to lower deployment risks and reuse resources for future de-Oracleization projects.

Conclusion

The migration from Oracle to UbiSQL within Ping An Technology marks a significant milestone in modernizing its database infrastructure. Adopting TiDB successfully addressed its critical challenges, enhancing performance, scalability, and flexibility across its core systems. The meticulous migration process, coupled with robust traffic replication and data comparison strategies, ensured a smooth transition and fortified their operational efficiency.

The results speak volumes: improved response times, enhanced data synchronization, and an overall increase in system reliability have positioned Ping An Technology to meet the evolving demands of the financial sector. This transformation illustrates how innovative database solutions can substantially improve data management and business processes.

As organizations navigate the complexities of digital transformation, TiDB stands out as a powerful ally. Experience TiDB’s transformational impact on your database management and unlock new possibilities for growth and efficiency in your operations. Reach out today to discover how TiDB can elevate your organization’s data strategy!