watsonx.data

This document is designed to guide developers in selecting similarity metrics and In-Memory indexes when using the Milvus vector database. It provides a concise overview of key concepts and parameters, encourages iterative testing with your dataset, and invites users to explore the resource links for an in-depth understanding of the topics discussed.

For floating-point embeddings, Milvus provides three metric types:

1. Euclidean Distance (L2):

   • Measures the straight-line distance between two points in space.
   • Suitable for raw distance-based comparisons.

2. Inner Product (IP):

   • Measures similarity based on vector projections.
   • Effective when vector magnitudes influence similarity.

3. Cosine Similarity (COSINE):

   • Compares the angle between two vectors.
   • Ideal for normalized vectors.
   • Default metric in Milvus.

Pro Tip: Use COSINE for normalized vectors unless your use case benefits specifically from another metric.

Milvus supports several index types, each optimized for specific scenarios:

• Overview: Compares the query vector with every vector in the dataset.
• Scenario: Best for datasets with around 100K rows requiring perfect accuracy.
• Trade-offs: Slower query times and higher resource usage.

• Overview: Graph-based index offering high recall and speed. Faster than any of the suggested indexes.

• Scenario: Preferred for datasets >2GB where accuracy and speed are critical.

• Trade-offs: High memory usage and slower index building times.

• Key Parameters:

  • Index Parameters

    • M: Number of edges per element during graph creation.
      • Suggested value: 16.
    • efConstruction: Neighbors considered during graph construction.
      • Suggested value: 200.

  • Search Parameters

    • efSearch: Neighbors considered during queries.
      • Suggested value: 50.

• NB: efSearch should always be greater than the number of nearest neighbors you want to retrieve.

• Overview: Speeds up queries by organizing vectors into clusters and calculating the distance between the target input vector and the cluster centers. Based on the configured number of clusters to search, the system performs similarity comparisons only within the most relevant cluster(s), returning results more efficiently.

• Scenario: Preferred for scenarios where a mid accuracy and speed are needed.

• Key Parameters:

  • Index Parameters

    • nlist: Number of clusters formed during indexing.

  • Search Parameters

    • nprobe: Number of clusters to search during a query.

• Determine the size of each field in your collection schema based on their data types (e.g., float32, int64, strings, etc.).
• Sum up the sizes of all fields to get the total size of a single entity/row.
• Convert this size into megabytes (MB).

• Divide the segment size by the average entity size (in MB):

  n = 512 / (average entity size in MB)
  

• Use the following rule of thumb:

  nlist = 4 × √n
  

  Where n is the total number of entities in a segment.

• Calculate using:

  nprobe = nlist / 16
  

• NB: nprobe should either be less than or equal to nlist

• Overview: Involves Scalar Quantization of vectors to 8-bit representation. Less memory-intensive than any of the suggested indexes, with reduced accuracy.
• Scenario: Suitable for scenarios involving massive datasets requiring fast approximate searches.
• Parameters: Same as IVF_FLAT (nlist, nprobe). Refer the Index Parameter Calculation Guide section of IVF_FLAT to get suggested values for nlist and nprobe.
• NB: There are lesser memory-intensive indexes like IVF_PQ and SCANN. But they come with further reduced accuracy.

1. Iterative Optimization:

   • Benchmark indexes systematically using representative datasets.

   • Compare performance trade-offs between different strategies.

2. Performance Tracking:

   • Log key metrics: query latency, vector collection size, search scope, and index parameters.

   • Use data to drive indexing decisions.

3. Methodical Approach:

   • Start with FLAT index as a baseline.

   • Progressively explore more complex indexing techniques.

4. Continuous Profiling:

   • Regularly benchmark and adapt index strategies.

   • Create repeatable optimization processes.

• Similarity Metrics Documentation
• Index Types Documentation
• Visual Guide to Picking a Vector Index
• Performance FAQ
• Product FAQ
• Accelerating Similarity Search on Really Big Data with Vector Indexing
• Understanding HNSW Graph-Based Indexing

Note: This guide is a starting point for selecting similarity metrics and indexes in Milvus. For optimal results, always test with your dataset and refine parameters iteratively.