Transforming Vivo’s Database and Storage Architecture with TiDB for Enhanced Scalability, Efficiency, and Security

Author: Xiao Bo, Director of R&D, Vivo Storage Technology Team

Faced with challenges surrounding high-performance storage, operational efficiency, and data security, Vivo sought a solution to support both structured and unstructured data at scale. TiDB’s implementation allowed Vivo to achieve remarkable improvements across critical business operations.

Key successes included: 

• Improved Performance: The integration of TiDB’s disk-based KV storage solution, with its compatibility with the Redis protocol, reduced data access latency to P99 ≤ 10ms, enabling faster and more efficient data handling.
• Operational Efficiency: TiDB’s automated failover and horizontal scalability enabled seamless scaling and data migration, reducing operational overhead and ensuring uninterrupted business access.
• Cost-Effective Storage: The object storage service provided with secure, large-capacity storage at a reduced cost while enabling real-time access to terabytes of structured and unstructured data.
• Data Security and Consistency: TiDB ensured zero data loss during power failures, implemented robust data backup and rollback capabilities, and delivered superior data security through encryption and consistent management practices.

What is Vivo? 

Vivo, founded in 2009, is one of the world’s leading smartphone manufacturers and a prominent player in the global consumer electronics industry. With a footprint spanning over 100 countries, Vivo ranks among the top smartphone brands, known for its innovation in mobile technology, sleek designs, and strong focus on user experience. The company caters to millions of customers globally, with a robust presence in major markets such as China, India, and Southeast Asia. Vivo’s product portfolio extends beyond smartphones to include accessories, software, and smart devices.

Competition among manufacturers has intensified as the smartphone industry experiences rapid growth, driven by rising digital consumption and technological advancements. Industry players like Vivo constantly innovate to provide more sophisticated devices and services, ensuring a superior customer experience. 

The company processes billions of transactions and massive amounts of data daily. This high volume of data, combined with user demand for real-time responses, makes data management a critical part of Vivo’s operations. Vivo’s end users expect seamless performance, fast updates, and uninterrupted services, making efficient database management essential for the company to maintain its competitive edge. The implementation of TiDB has been instrumental in supporting Vivo’s data-heavy infrastructure, ensuring that it continues to meet consumer expectations for high performance, reliability, and customer satisfaction.

Through its adoption of TiDB, Vivo has not only streamlined its data management processes but also enhanced system scalability and efficiency, contributing to the overall success of its business operations. This case study demonstrates how deploying cutting-edge technology in a highly competitive and fast-paced industry can drive operational excellence, improve service delivery, and maintain customer satisfaction at scale.

The Challenges: Navigating a Precarious Database Landscape

Vivo’s original data architecture posed several significant challenges that jeopardized the efficiency and reliability of its database services:

1. Precarious Availability: The availability of online databases was precarious, often impacted by inefficient SQL queries and human errors. This unreliability led to frequent outages, disrupting user experiences and affecting customer satisfaction.
2. Inefficient Change Management: Implementing changes was labor-intensive and inefficient, as it relied on manual operations via command-line interfaces. This approach not only elevated database usage costs but also introduced risks associated with human error, particularly in complex business scenarios.
3. Scalability Constraints: While the database instances expanded significantly, the underlying infrastructure struggled to scale in tandem. The rapid data volume and user base growth exacerbated existing performance issues, leading to latency and slower application response times.
4. Insufficient Security Protocols: Vivo’s database architecture lacked comprehensive security standards, particularly in areas such as data classification, password management, and account permissions. This vulnerability heightened the risk of data breaches and compromised sensitive information.
5. Operational Overhead: As the volume of data grew, the operational overhead associated with database management increased. The challenges posed by many database instances per DBA added to the complexity, making it difficult to ensure consistent performance and reliability.

Original Data Architecture

In late 2018, Vivo’s data architecture was characterized by a complex and precarious structure that struggled to keep up with the rapid growth of its mobile internet user base, which had exceeded 220 million. The architecture consisted of several critical components and challenges:

1. Server Infrastructure: Vivo operated over 4,000 database servers, but the architecture lacked the scalability and robustness to handle the increasing load. The servers were often overwhelmed by the volume of requests from various applications, leading to performance bottlenecks.
2. Database Instances: The number of database instances surpassed 50,000, with each instance required to manage multiple applications and user interactions. The sheer scale of these instances added complexity to the overall management and maintenance processes.
3. Inefficient Query Execution: A significant portion of the performance issues stemmed from inefficient SQL queries, which frequently caused slowdowns and impacted availability. These queries often lacked optimization, leading to increased resource consumption and longer execution times.
4. Human Errors and Maintenance: The reliance on command-line terminals for changes and maintenance operations contributed to frequent human errors. Without a user-friendly interface or comprehensive platform support, DBAs faced challenges managing the database effectively.
5. Lack of Security Standards: Security measures were insufficient, and no established protocols for data classification, grading, password management, or account permissions existed. This left the architecture vulnerable to potential data breaches and compliance violations.
6. Rapid Expansion: The growth of database instances and the increasing deployment density on single machines further strained the architecture. The workload per Database Administrator (DBA) rose significantly, with each DBA managing 14.9 times more database instances than before, leading to inefficiencies and potential oversight in maintenance.

The Solution: Strategic Implementation of TiDB to Address Vivo’s Database Challenges

Implementation Process 

Implementing TiDB at Vivo was a critical step toward modernizing its database architecture. This process required careful planning, meticulous execution, and ongoing optimization to ensure the new system effectively met Vivo’s business needs. 

1. Pre-Implementation Planning

a. Requirement Analysis

• Stakeholder Engagement: Vivo initiated the process by engaging key stakeholders, including database administrators, developers, and business leaders, to identify specific requirements and challenges faced with the existing database infrastructure.
• Needs Assessment: A thorough assessment was conducted to understand the volume of data, transaction types, and performance benchmarks necessary for applications, particularly those experiencing rapid growth in user engagement.

b. Architecture Design

• System Architecture: The architecture was designed to leverage TiDB’s distributed capabilities, ensuring optimal performance and scalability. This included selecting the appropriate number of TiDB and TiKV nodes based on projected growth.
• Data Flow Mapping: A detailed mapping of data flows was created to visualize how data would be accessed, processed, and stored within the new system. This helped identify potential bottlenecks and optimize the overall architecture.

2. Migration Strategy Development

a. Phased Migration Approach

• Incremental Migration: Vivo adopted a phased migration strategy, allowing specific application components to transition to TiDB one at a time. This minimized risk and allowed for continuous monitoring and adjustments.
• Data Backup: To ensure data safety and integrity, comprehensive backups of existing data were performed before the migration process commenced.

b. Tool Selection

• Migration Tools: Vivo selected specialized migration tools, such as TiDB’s Data Migration (DM) tool, which facilitated efficient data transfer from legacy database systems to TiDB.
• Validation Tools: Automated validation tools were also chosen to ensure data was accurately transferred and consistent throughout the migration process.

3. Data Migration Execution

a. Initial Data Transfer

• Bulk Data Load: The first phase involved bulk loading historical data into TiDB. This was executed during off-peak hours to reduce the impact on ongoing operations.
• Data Validation: After each bulk load, data validation checks were performed to confirm that the data in TiDB matched the source database, ensuring no data loss or corruption occurred during the transfer.

b. Live Data Synchronization

• Change Data Capture (CDC): Vivo implemented a CDC approach to synchronize live data between the legacy systems and TiDB. This allowed for continuous data updates during migration, ensuring no disruption to ongoing operations or user experiences.
• Monitoring Synchronization: Real-time monitoring dashboards were set up to track the synchronization process, alerting the team to any discrepancies or issues that arose.

4. Integration with Existing Systems

a. Application Integration

• APIs and Connectors: Vivo developed necessary APIs and connectors to integrate TiDB with existing applications, ensuring that all application components could communicate effectively with the new database system.
• Code Refactoring: Some application code was refactored to align with TiDB’s SQL syntax and functionalities, leveraging its MySQL compatibility to minimize changes to the existing codebase.

b. Testing and Validation

• Functional Testing: Comprehensive testing, including unit tests, integration tests, and end-to-end testing, was conducted to ensure that applications functioned as expected with TiDB.
• Performance Benchmarking: Performance benchmarking was executed to compare query execution times, transaction throughput, and overall system responsiveness between the legacy database and TiDB.

5. Performance Optimization

a. Query Optimization

• Analyzing Query Performance: Vivo focused on optimizing SQL queries to leverage TiDB’s strengths after migration. The team utilized TiDB’s built-in performance analysis tools to identify slow queries and optimize them accordingly.
• Index Management: Careful index management enhanced query performance, balancing the trade-offs between write performance and read efficiency.

b. Configuration Tuning

• TiDB Configuration: The team fine-tuned TiDB configurations to optimize resource allocation based on workload patterns. This included adjusting memory settings, connection limits, and scheduling parameters.
• Load Testing: Load testing was performed to simulate peak usage scenarios, allowing Vivo to proactively identify performance issues and address them before they impacted users.

6. Training and Knowledge Transfer

a. Team Training

• Workshops and Documentation: Vivo conducted training sessions and seminars for its IT staff, equipping them with the knowledge to manage and maintain the TiDB environment effectively.
• Knowledge Base Creation: A centralized knowledge base was created to document best practices, troubleshooting tips, and operational guidelines specific to TiDB.

b. Continuous Learning

• Feedback Loop: Establishing a feedback loop among team members allowed continuous learning and sharing of insights gained during the migration and optimization processes.

7. Post-Implementation Monitoring and Support

a. Ongoing Monitoring

• Performance Dashboards: Continuous monitoring was implemented to track key performance metrics, including response times, transaction throughput, and resource utilization.
• Alerting Systems: Real-time alerting systems were set up to notify the IT team of any performance degradation, enabling proactive issue resolution.

b. Regular Review and Iteration

• Periodic Reviews: Regular review meetings were scheduled to assess the system’s performance and identify areas for further improvement. This iterative approach ensures that Vivo can continuously optimize its TiDB deployment.

Critical Components of the New Architecture

Vivo’s new architecture, characterized by integrating TiDB, Redis, and a robust object storage service, addresses the company’s initial challenges and positions it for continued growth and success. The architecture enhances performance and scalability and ensures that Vivo can meet the demands of its expanding user base while maintaining high levels of customer satisfaction.

TiDB (Distributed SQL Database)

• Horizontal Scalability: TiDB allows Vivo to scale out seamlessly as user demand grows, supporting an expanding database with minimal downtime.
• Strong Consistency: Utilizes a distributed consensus protocol to ensure that all transactions are consistent across nodes, enhancing data reliability.
• Real-Time Analytics: Offers real-time query capabilities, enabling Vivo to make data-driven decisions swiftly.

Disk-Based KV Storage Solution

• Compatibility with Redis Protocol: Built on TiKV, this solution supports Redis commands, allowing Vivo to utilize Redis features while maintaining high performance.
• Persistent Large-Scale Storage: Ensures efficient storage management, enabling Vivo to handle vast amounts of data generated by its applications.

Redis (In-Memory Data Store)

• High-Performance Caching Layer: Caches frequently accessed data, significantly reducing latency and improving application response times.
• Real-Time Data Processing: Facilitates instant data access and manipulation, ideal for services requiring quick updates and interactions.
• Enhanced Scalability: Supports horizontal scaling to accommodate increasing user loads while maintaining performance.
• Data Persistence: Provides options for data persistence through RDB and AOF strategies, ensuring data recovery and consistency.
• Pub/Sub Messaging: Enables real-time messaging features, enhancing user engagement through notifications and live updates.

Object Storage Service

• Large-Capacity, Low-Cost Storage: Supports extensive data storage needs, particularly for media and application data, at a reduced cost.
• Support for Various Services: Facilitates file system services and video processing, essential for Vivo’s multimedia applications.

Operational Efficiency Enhancements

• DevOps Practices: Implements DevOps methodologies to streamline development and operational processes, improving overall efficiency.
• Data Security Management: Integrates advanced security features to ensure data protection, compliance, and user privacy.

The Results: Measurable Impact Post-TiDB Implementation

Implementing TiDB and the new architecture at Vivo yielded significant tangible results and achievements that positively impacted the company’s performance. The following vital outcomes were observed:

1. Improved Database Performance
   • Enhanced Availability: The introduction of TiDB increased the overall availability of Vivo’s online database environment, mitigating previous issues caused by inefficient SQL queries and human errors.
   • Efficient Operations: Maintenance and changes to the database became more efficient as the platform supported automated processes rather than relying on command-line terminals, reducing overhead and costs associated with complex business scenarios.
2. Scalability and Management
   • Increased Instance Capacity: Vivo’s database instances grew substantially, reflecting the system’s ability to scale to meet the demands of an expanding user base, which surpassed 270 million online users by 2021.
   • High Transaction Volumes: The architecture’s ability to handle increased transaction volumes was critical, as daily active users for various applications exceeded 10 million, and the scale of database and storage products expanded to over 4,000 servers and more than 50,000 database instances.
3. Cost Efficiency
   • Reduction in Operational Costs: Improved database management and operational efficiencies reduced Vivo’s overall database usage costs significantly, translating to better financial performance.
4. Enhanced Security Standards
   • Improved Data Security: Implementing standardized data classification, grading, password management, and account permissions strengthened Vivo’s data security measures, addressing previous insufficiencies.
5. Increased Customer Satisfaction
   • Positive Impact on User Experience: Vivo’s applications’ enhanced performance and reliability improved customer satisfaction, as evidenced by the increase in daily and monthly active users across various internet applications.

Future Plans

Vivo aims to further enhance its database and storage capabilities through strategic initiatives. One of the primary focuses will be on enhancing the storage service matrix. The company plans to continue developing TiDB’s storage solutions to improve overall performance and efficiency while integrating advanced features to cater to evolving business needs. This commitment to continuous improvement will help Vivo maintain a competitive edge in the rapidly changing technology landscape.

In addition to enhancing storage solutions, Vivo is also focused on expanding its Software as a Service (SaaS) offerings. By providing scalable SaaS solutions, Vivo aims to accommodate growing user demands and develop new services that leverage TiDB’s capabilities to deliver better customer experiences. This expansion aligns with the increasing reliance on cloud-based solutions and the need for flexibility in service delivery.

Moreover, Vivo invests in basic data services to ensure seamless integration and accessibility across various applications. The company is exploring autonomous storage services that utilize intelligent algorithms for improved data management, further enhancing efficiency and reducing operational costs.

Data security is another critical area of focus for Vivo’s future plans. The company is committed to strengthening data security protocols to protect sensitive information and ensure compliance with industry standards. Implementing comprehensive data governance frameworks will help manage data classification and permissions effectively, addressing growing concerns around data privacy.

Finally, Vivo plans to integrate TiDB into a hybrid cloud architecture, enhancing flexibility and scalability. This integration aims to provide a standardized user experience across various platforms and services, simplify operations, and improve overall performance. Through these initiatives, Vivo is dedicated to maximizing TiDB’s potential and ensuring its database and storage platform remain at the forefront of innovation and efficiency in the industry.

Conclusion 

The successful implementation of TiDB has marked a significant turning point for Vivo, enabling the company to address critical challenges and achieve remarkable advancements in its database and storage architecture. With a solid foundation built on scalability, efficiency, and security, Vivo is poised to significantly enhance its operational capabilities and customer satisfaction. The improvements in performance metrics, coupled with the strategic focus on expanding service offerings and data security, showcase how TiDB has become an integral part of Vivo’s growth trajectory.

As Vivo continues to evolve and adapt to the dynamic technology landscape, the partnership with TiDB will facilitate ongoing innovation and excellence in service delivery. The company’s future plans, including enhancements to its storage service matrix, expansion of SaaS offerings, and the adoption of hybrid cloud architecture, are a testament to TiDB’s transformational impact.

We invite you to experience TiDB’s transformational impact on your organization. Discover how TiDB can streamline your database operations, enhance performance, and unlock new growth opportunities.