Integrating Vector Search into TiDB Serverless for AI Applications

We’re thrilled to share that TiDB Serverless, our fully managed Database-as-a-Service offering, is introducing a built-in vector search to the MySQL landscape. With this support, you will be able to develop your AI applications using TiDB Serverless without requiring a new database or additional technical stacks.

Join the waitlist for the private beta at tidb.cloud/ai.

Understanding Vector Search 

Unlike traditional keyword-based searches, vector search operates on the principle of understanding the meaning and context of data. It converts complex data – like text, images, or audio – into numerical vector embeddings. These embeddings enable the database to perform searches based on semantic understanding rather than just exact word matches. This approach is crucial for AI applications dealing with vast amounts of unstructured data, where precision and context are key.

Vector search isn’t just about smarter data interpretation; it’s also about performance and scalability. It optimizes query efficiency, enabling quicker and more accurate searches across large, complex datasets. By adding vector search to TiDB Serverless, we’re enhancing its capability to handle AI and machine learning workloads efficiently. This makes it a powerful tool for developers in the MySQL+AI ecosystem.

Why Do We Need Vector Search in MySQL?

As AI continues to reshape industries, the demand for databases capable of handling complex, high-dimensional data surges. Traditional MySQL databases, while robust and popular, encounter limitations when dealing with AI applications:

• Traditional MySQL systems generally do not handle unstructured data prevalent in AI applications, such as text, images, and audio. This leads to inefficiencies and bottlenecks in data processing and querying.
• MySQL’s architecture lacks native capabilities for deep semantic understanding, which is vital for accurately interpreting and utilizing AI-generated data.

By adding vector search capability into MySQL systems, developers can manage vector embeddings alongside structured data. This greatly simplifies the handling of diverse data types crucial in AI. The ability to perform semantic-rich searches enhances MySQL’s functionality, allowing it to delve beyond basic keyword matching.

Benefits of Built-in Vector Search in TiDB Serverless

2023 witnessed a pivotal shift in the realm of AI applications, with an increasing demand for scalable, flexible, and cost-effective solutions to manage high-dimensional data. While specialized vector databases have emerged, there’s a growing realization that vector search should be an integral feature of existing databases, not a separate entity.

With built-in vector search, TiDB Serverless is leading this transformation by combining traditional database functionality with the advanced capabilities of vector search for the MySQL ecosystem. This strategic approach merges the strengths of general-purpose databases with advanced vector search capabilities, offering the best of both worlds. Key benefits include:

• Scalable to AI Demands: TiDB Serverless offers elastic scalability for the dynamic and unpredictable data requirements of AI applications. This guarantees both efficiency and economical operation. Its combination of Hybrid Transactional/Analytical Processing (HTAP) and Serverless architecture supports real-time, extensive data processing, essential for large language models and AI applications.
• MySQL and vector all in one: The build-in approach eliminates the need for separate databases for vector and operational data, thus avoiding data redundancy. Store vector embeddings directly alongside your MySQL data, simplifying your data architecture with the straightforwardness of SQL.
• Join vector data with ease:  Developers can leverage the familiar SQL environment to effortlessly join, index, and query both operational and vector data. This capability enables advanced semantic searches, combining the power of vector search with the reliability and ease of MySQL.

• A vast array of use cases: TiDB Serverless with vector search powers a wide range of applications, from Retrieval Augmented Generation (RAG) to semantic searches. The system integrates smoothly with leading AI platforms and tools like OpenAI, Hugging Face, LangChain, and LlamaIndex, broadening its applicability across various AI-driven scenarios.

How Are We Making This Happen?

TiDB Serverless’s robust architecture, designed for horizontal scaling and distributed computing, is already primed for handling workloads of AI applications. By innovatively introducing the vector data type into our distinctive storage engines and implementing similarity search indexes and algorithms like Hierarchical Navigable Small World (HNSW), we have enabled efficient storage, indexing, and retrieval of vector data in TiDB Serverless. This architectural advancement optimizes table scans and computation, allowing for the direct joining of vector data with traditional data types through SQL. 

The result? Seamless AI integration within your MySQL workflow. You can store and retrieve vector data through your favorite MySQL tools and leverage powerful AI capabilities without leaving your comfort zone.

Shaping the Future of AI Applications Together 

This journey with TiDB Serverless and vector search isn’t just about pushing the limits of technology. It’s about redefining how data fuels AI innovation. We’re not simply building a product, but crafting a seamless bridge between your familiar MySQL environment and the boundless possibilities of the AI age.

Ready to pioneer the future with us? Join the waitlist to try out the private beta at https://tidb.cloud/ai. You’ll also receive the latest updates and be among the first to experience the power of TiDB with vector search.