What's the best way to get a random vector from a Milvus collection

[mkotsalainen]

Hi! I'm trying to get a random vector from our Milvus dataset. I'm not sure what the best approach is but I've tried searching with random vectors, but doesn't work so well.

I'm doing this but I keep getting duplicate results which seems unlikely because the collection contains 2 million entries. Can anybody point me in the right direction? Thanks!

def get_random_ids(collection_name, n=1, dim=768, normalize=True):
    """
        Retrieve random N IDs of all indexed documents
        Params:
          - collection_name (str): name of the collection
          - n (int): how many items to retrieve
        Returns:
          - (list[str]): list of n IDs
    """
    vectors = [random_vector(dim=dim, normalize=normalize) for _ in range(n)]
    res = get_collection(collection_name).search(
        anns_field='vec',
        data=vectors,
        param=search_param(n),
        limit=1,
        output_fields=['id'],
    )
    return [hit.ids[0] for hit in res]
    
def random_vector(dim=768, normalize=True):
    vec = np.random.random(dim)
    if normalize:
        vec = normalize_vector(vec)
    return vec
    ```

[yhmo]

What kind of index?
What's the value of the "search_param(n)"?

[yhmo]

Per your description, seems you are trying to "get a random entity by a random vector from a Milvus collection".

I wrote a script to test this case. This script creates a collection with 1M entities of 768-dim vectors, and generates 10 random vectors to search.

import random
import numpy as np

from pymilvus import (
    connections,
    FieldSchema, CollectionSchema, DataType,
    Collection,
    utility,
)


HOST = '127.0.0.1'
PORT = '19530'

if __name__ == '__main__':
    try:
        connections.connect(host="localhost", port="19530")
        print("connected Successfully")
    except Exception as e:
        print(f"error occurred: {e}")


    for name in utility.list_collections():
        print(f"collection {name} has alias {utility.list_aliases(name)}")

    collection_name = "test"
    dim=768

    if utility.has_collection(collection_name):
        utility.drop_collection(collection_name)
    print(collection_name)
    field_1 = FieldSchema(
        name="face_id",
        dtype=DataType.VARCHAR,
        max_length=65535,
        is_primary=True,
    )
    field_2 = FieldSchema(
        name="face_vector",
        dtype = DataType.FLOAT_VECTOR,
        dim=dim
    )
    schema = CollectionSchema(fields=[field_1, field_2])
    collection = Collection(name=collection_name, schema=schema)
    print(f"Collection {collection}")

    count = 10000
    for i in range(100):
        print(i)
        data = [
            [f"id_{k+count*i}" for k in range(count)], # id
            [[random.random() for _ in range(dim)] for _ in range(count)],
        ]
        tt = collection.upsert(data)

    collection.flush()
    print("row count:", collection.num_entities)

    index_params = {
        'metric_type': 'L2',
        'index_type': "IVF_FLAT",
        'params': {"nlist": 128}
    }
    collection.create_index(field_name="face_vector", index_params=index_params)
    print("index done")
    collection.load()
    print("loaded")


    def random_vector(dim):
        vec = np.random.random(dim)
        return vec

    search_params = {
        "metric_type": "L2",
        "params": {"nprobe": 12},
    }

    print("search...")
    vectors = [random_vector(dim=dim) for _ in range(10)]
    results = collection.search(
        data=vectors,
        anns_field="face_vector",
        param=search_params,
        output_fields=["face_id"],
        limit=1,
    )

    for i, result in enumerate(results):
        print("\nSearch result for {}th vector: ".format(i))
        for j, res in enumerate(result):
            print("Top {}: {}".format(j, res))

The results are random:

Search result for 0th vector: 
Top 0: id: id_471764, distance: 103.51850128173828, entity: {'face_id': 'id_471764'}

Search result for 1th vector: 
Top 0: id: id_996574, distance: 108.94523620605469, entity: {'face_id': 'id_996574'}

Search result for 2th vector: 
Top 0: id: id_985151, distance: 99.88804626464844, entity: {'face_id': 'id_985151'}

Search result for 3th vector: 
Top 0: id: id_991748, distance: 101.92803955078125, entity: {'face_id': 'id_991748'}

Search result for 4th vector: 
Top 0: id: id_281506, distance: 108.33908081054688, entity: {'face_id': 'id_281506'}

Search result for 5th vector: 
Top 0: id: id_299321, distance: 104.22210693359375, entity: {'face_id': 'id_299321'}

Search result for 6th vector: 
Top 0: id: id_288190, distance: 104.24483489990234, entity: {'face_id': 'id_288190'}

Search result for 7th vector: 
Top 0: id: id_43245, distance: 102.99598693847656, entity: {'face_id': 'id_43245'}

Search result for 8th vector: 
Top 0: id: id_577547, distance: 101.66751098632812, entity: {'face_id': 'id_577547'}

Search result for 9th vector: 
Top 0: id: id_866334, distance: 104.88261413574219, entity: {'face_id': 'id_866334'}

[mkotsalainen]

Thank you @yhmo your approach is the same as mine. I realised that I changed my random vector to have negative components the results I got back were better. Like so:

def random_vector(dim=768, normalize=True):
    v = np.random.random(dim) * 2 - 1
    if normalize:
        v /= v.sum()
    return v

I'm closing this. I think this is probably the best we can do.

[xiaofan-luan]

Hi! I'm trying to get a random vector from our Milvus dataset. I'm not sure what the best approach is but I've tried searching with random vectors, but doesn't work so well.

I'm doing this but I keep getting duplicate results which seems unlikely because the collection contains 2 million entries. Can anybody point me in the right direction? Thanks!

def get_random_ids(collection_name, n=1, dim=768, normalize=True):
    """
        Retrieve random N IDs of all indexed documents
        Params:
          - collection_name (str): name of the collection
          - n (int): how many items to retrieve
        Returns:
          - (list[str]): list of n IDs
    """
    vectors = [random_vector(dim=dim, normalize=normalize) for _ in range(n)]
    res = get_collection(collection_name).search(
        anns_field='vec',
        data=vectors,
        param=search_param(n),
        limit=1,
        output_fields=['id'],
    )
    return [hit.ids[0] for hit in res]
    
def random_vector(dim=768, normalize=True):
    vec = np.random.random(dim)
    if normalize:
        vec = normalize_vector(vec)
    return vec
    ```

One way to sample random data is to implement a random function in the filter.

But unluckily it is not supported in milvus right now.

Anyone interested please take it. thanks