for index all the rows are not getting used

[vihariazure]

hi team
below is the my dataframe size
(1000000, 12)
i have created the index
when i seen the logs(describe index ) i noticed for index creation not all the data is used

{'m': '16',
'ef_construct': '100',
'index_type': 'HNSW',
'metric_type': 'L2',
'field_name': 'product_embeddings',
'index_name': 'first_index',
'total_rows': 987000,
'indexed_rows': 758000,
'pending_index_rows': 964000,
'state': 'Finished'}

may i know what is the root cause

[yhmo]

In milvus, each segment has an independent index. Index nodes build index for each segment separately. Index tasks are executed asynchronously.
Data nodes compact small segments to large segments. For example, seg_A and seg_B are compacted to seg_C. Index nodes build index for seg_A and seg_B, then seg_C is generated, index nodes will build index for seg_C. So the indexed_rows + pending_index_rows might be larger than total_rows.

'total_rows': 987000, ------------------------- the collection has 987000 rows (maybe some other rows are not flushed, still in pulsar not consumed by data node, after a while, the left rows will be flushed to new segment)
'indexed_rows': 758000, -------------------- index nodes already build index for 758000 rows, some rows might belong to small segments that will be compacted
'pending_index_rows': 964000,------------ there are some index tasks are in queue not executed, 964000 rows will be indexed in these tasks

[vihariazure]

thanks for the reply the point i want to convey is
i have 1m data points
i added i million data point as a batch to create index
so in the index also total_rows should be 1m right . they why it is 987000 only
similarly for indexed_rows, pending_index_rows as well
the main goal is what ever data points (1m) should be utilized to create a index. i know for unsealed segments milvus uses brut force method, but it reduces the accuracy

do i need to wait for sometime , at the background the process will happen and all the points are used for index creation. in this case what is the ideal time it take

do i need to change any parameter so that what are data insertion happens, immediately all data points are used for index creation
so that pending_index_rows is zero

[yhmo]

"so in the index also total_rows should be 1m right . they why it is 987000 only" ---------------------- Because the insert request is executed by two steps:

When you insert some data by insert():

1. The data is received by proxy node, proxy node splits the data into pieces by shards_num(which is defined in collection schema, default is 1), and sends the pieces(as messages) into Pulsar/Kafka. The insert() returns success to the client once the data is persisted into Pulsar/Kafka.
2. For each growing segment, two "data channels" are created, one is between querynode and Pulsar/Kafka, the other is between datanode and Pulsar/Kafka. Querynode and datanode subscribe messages asynchronously from Pulsar/Kafka, the data pieces are consumed by querynode and datanode.

The step 2 is delayed after step 1 by hundreds of milliseconds(sometimes might delay a few seconds). The "total_rows" is the rows that already consumed by data node. So, the "total_rows" is less than 1m in a short time.

[yhmo]

To wait index done, use this method:

utility.wait_for_index_building_complete(COLLECTION_NAME)

[xiaofan-luan]

hi team below is the my dataframe size (1000000, 12) i have created the index when i seen the logs(describe index ) i noticed for index creation not all the data is used

{'m': '16', 'ef_construct': '100', 'index_type': 'HNSW', 'metric_type': 'L2', 'field_name': 'product_embeddings', 'index_name': 'first_index', 'total_rows': 987000, 'indexed_rows': 758000, 'pending_index_rows': 964000, 'state': 'Finished'}

may i know what is the root cause

could you do count and check if you have 1M rows?
could it happen that you have some duplicated PK ?