Introduction to Milvus: The Foundation of GenAI
From creating realistic virtual assistants to automating complex content generation tasks, GenAI adoption is accelerating across industries. Crafting human-like text, generating stunning visuals, or creating immersive audio experiences, GenAI is rapidly redefining how we interact with technology across sectors like media, healthcare, finance, education and e-commerce.
For those eager to harness the potential of GenAI, understanding its foundation is key. Let’s understand the backbone for building advanced GenAI applications.
The Rise of vector databases
With the explosion of unstructured data, traditional databases are no longer sufficient to meet the needs of data storage and retrieval.
Modern AI applications like RAG need to work with unstructured data like text, images, videos, and audio, which are inherently complex and require specialized handling.
Unlike structured data that fits neatly into rows and columns, unstructured data must be processed and analysed in its raw form. This is where Milvus vector databases come into play.
With Milvus integrated into IBM watsonx.data, users can now leverage advanced vector search capabilities with enterprise-grade scalability, reliability, and security.
What is Milvus?
Milvus is an open-source purpose-built vector database that stores and manages high-dimensional vectors—mathematical representations of unstructured data that captures the core characteristics or context of the data. By leveraging advanced indexing techniques and distributed architecture, Milvus enables efficient similarity searches across massive datasets.
Decoding the jargons
Note: If you are already aware of basic concepts like vectors, similarity search, ANN etc you can skip to next section.
What’s a Vector?
A vector is like a fingerprint for data. It’s an array of float numbers that represents the essence or meaning of that data.
How are these generated?
We use machine learning models or neural networks to convert unstructured data (like images, audio, text) into feature vectors or embeddings.
Why do vectors matter?
Consider the text “exploring”. When we generate vector embeddings for this text using a particular model say with ‘ibm/slate-125m-english-rtrvr-v2’ model from IBM watsonx.ai, it will generate a vector representation in such a way, that it will be closer/similar to the vector representations of words like “finding”, “searching”, ”learning” but very different from words like “flower”, “pizza” or any words that does not relate to the word “exploring”.
Let us consider we have generated a 3-dimensional vector (spanning on x, y and z axis/direction/dimension) for the word “exploring”, e.g. [2.087537, 25.71689, 13.2392].
On a 3-dimensional space this would look like:
Here each colored dot represents a vector in a 3-dimensional space.
The similar colored (similar in characteristics) are placed close to each other. So for example: the yellowish dots could be representing the words “exploring”, “finding”, “searching”,” learning” while the purple ones could be “food”, “pizza”, “cuisine”, etc.
In reality, these vectors span to much higher dimensions like 384 dimensions,768 dimensions, 1024 dimensions etc, which cannot be visualized on a 2D plane or 3D space.
How does Milvus help you find things?
Milvus uses multiple different algorithms to calculate the distance between various unstructured data and returns most similar items.
For example:
- If you give Milvus a picture of a cat, it will search for vectors that captures features like "furry," "four-legged," and "pointy ears."
- If you give it a sentence about summer vacations, it will search for vectors capturing concepts like "beach," "sun," and "relaxation."
Now, all these vectors are stored in Milvus’ database.
Milvus doesn’t just look for the exact words in the query text instead, it compares the vector of your query with the vectors of all the stored data and finds the ones that are most similar in meaning. This is called vector similarity search. This way of searching is also referred as Approximate Nearest Neighbour(ANN) search, which is a probabilistic way of searching.
How does Milvus work in the background?
Let’s simplify it even more:
1. Data Ingestion: First, you send your data (like pictures or text) to Milvus, and it turns that data into vectors using APIs or SDKs.
2. Storage: These vectors are organized and stored in a way that makes them easy to search.
- Indexing: Milvus uses special techniques to build an index (analogous to address book) for the vectors so it can quickly find similar ones.
- Search: When you ask for something, Milvus first converts that query text into vector embeddings then compares your query embeddings/vector to the stored vectors and gives you the closest matches. The match is driven my similarity metrics that calculates the distance between the two.
In short, Milvus is like a genius librarian who can understand the meaning behind data, organize it, and help you find what you’re looking for—even when the data is messy or unstructured. It’s a powerful tool for anyone building AI applications, even if you’re new to this world!
Applications of Milvus
· Similarity Search: Find similar documents based on sematic meaning, find visually similar images.
- RAG Pipelines: Empower chatbots, virtual assistants, and search engines to provide contextually relevant answers by combining vector search with large language models.
- GenAI: Enhance generative models by feeding them the most relevant data retrieved through vector search.
- Recommendation Systems: Deliver personalized content by finding similar users or items.
While Milvus stands out as a cutting-edge solution, questions often arise about how to leverage Milvus’ powerful features without compromising data integrity or risking misuse.
IBM watsonx.data has seamlessly integrated Milvus into its robust data offerings, available both on IBM Cloud and on-premises, so that organizations can confidently adopt this technology and unlock its full potential.
Why choose IBM watsonx.data Milvus?
- AI-Centric: Built to support AI and machine learning workflows, Milvus integrates seamlessly with IBM watsonx.ai and other popular frameworks for generating embeddings
- Scalability: watsonx.data Milvus offers different T shirt sizes (starter, small, medium and large) for different workloads to optimise cost and resources.
- High Performance: Milvus ensures low-latency search and retrieval, even with billions of vectors.
- Flexibility: Support for multiple embedding models, similarity metrics and index types
- Reliability: Built-in data replication and backup mechanisms
- Security: We have implemented enhanced security measures at multiple levels, along with vulnerability fixes, to ensure the integrity and reliability of the data.
- Ease of Use: Rich client SDKs and simple API interface
In our upcoming blog, we will cover how to get started with Milvus and more. If you would like to explore any specific topics drop a comment or suggestions.
Next Steps:
#watsonx.data