[Bug]: [json-inverted] CreateIndex hangs when creating the same index repeatedly after inserting data

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: harbor.milvus.io/milvus/milvus:JsDove-optimization_json-72e0417-20241225
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

cluster with dev-image: harbor.milvus.io/milvus/milvus:JsDove-optimization_json-72e0417-20241225

client

1. create collection with 3 fields: id(int64 & pk) + vector + json

{'collection_name': 'laion_json_1', 'auto_id': False, 'num_shards': 1, 'description': '', 'fields': [{'field_id': 100, 'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'params': {}, 'is_primary': True}, {'field_id': 101, 'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'field_id': 102, 'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>, 'params': {}}], 'aliases': [], 'collection_id': 454845922362524415, 'consistency_level': 2, 'properties': {}, 'num_partitions': 1, 'enable_dynamic_field': False}

2. create HNSW index for vector fields
3. insert 10million entities -> flush
4. create index again -> hangs (client didn't receive response for several hours)

[2024-12-25 04:55:45,906 -  INFO - fouram]: [Base] Index params of laion_json_1:[{'float_vector': {'index_type': 'HNSW', 'metric_type': 'COSINE', 'params': {'M': 30, 'efConstruction': 200}}}] (base.py:491)
[2024-12-25 04:55:45,906 -  INFO - fouram]: [Base] Start build index of HNSW for field:float_vector collection:laion_json_1, params:{'index_type': 'HNSW', 'metric_type': 'COSINE', 'params': {'M': 30, 'efConstruction': 200}}, kwargs:{} (base.py:472)
[2024-12-25 04:55:45,907 - DEBUG - fouram]: (api_request)  : [Index] args: [<Collection>: 
-------------
<name>: laion_json_1
<description>: 
<schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>}], 'enable_dynamic_field': False}
, 'float_vector', {'index_type': 'HNSW', 'metric_type': 'COSINE', 'params': {'M': 30, 'efConstruction': 200}}], kwargs: {}, [requestId: 807b4138-c27c-11ef-a2a6-a6b4a1f06ead] (api_request.py:77)

server logs

Expected Behavior

No response

Steps To Reproduce

argo workflow: zong-json-index-1

Milvus Log

pod:

json-inverted-op-97-1369-etcd-0                                   1/1     Running                  0                6h11m   10.104.21.106   4am-node24   <none>           <none>
json-inverted-op-97-1369-etcd-1                                   1/1     Running                  0                6h11m   10.104.33.220   4am-node36   <none>           <none>
json-inverted-op-97-1369-etcd-2                                   1/1     Running                  0                6h11m   10.104.23.84    4am-node27   <none>           <none>
json-inverted-op-97-1369-milvus-datanode-56b8d4b7bf-g8r2k         1/1     Running                  0                6h7m    10.104.15.251   4am-node20   <none>           <none>
json-inverted-op-97-1369-milvus-indexnode-7dd96c757-6d8z8         1/1     Running                  0                6h7m    10.104.14.43    4am-node18   <none>           <none>
json-inverted-op-97-1369-milvus-indexnode-7dd96c757-b9g8z         1/1     Running                  0                6h7m    10.104.13.144   4am-node16   <none>           <none>
json-inverted-op-97-1369-milvus-indexnode-7dd96c757-pdsv4         1/1     Running                  0                6h7m    10.104.15.250   4am-node20   <none>           <none>
json-inverted-op-97-1369-milvus-mixcoord-564c6ffd7c-v8plr         1/1     Running                  0                6h7m    10.104.15.252   4am-node20   <none>           <none>
json-inverted-op-97-1369-milvus-proxy-744995f869-fws7x            1/1     Running                  0                6h7m    10.104.30.218   4am-node38   <none>           <none>
json-inverted-op-97-1369-milvus-querynode-0-f696dfb79-pkg4t       1/1     Running                  0                6h7m    10.104.15.253   4am-node20   <none>           <none>
json-inverted-op-97-1369-milvus-querynode-0-f696dfb79-z89lq       1/1     Running                  0                62m     10.104.33.235   4am-node36   <none>           <none>
json-inverted-op-97-1369-minio-0                                  1/1     Running                  0                6h11m   10.104.21.107   4am-node24   <none>           <none>
json-inverted-op-97-1369-minio-1                                  1/1     Running                  0                6h11m   10.104.33.222   4am-node36   <none>           <none>
json-inverted-op-97-1369-minio-2                                  1/1     Running                  0                6h11m   10.104.23.85    4am-node27   <none>           <none>
json-inverted-op-97-1369-minio-3                                  1/1     Running                  0                6h11m   10.104.19.246   4am-node28   <none>           <none>
json-inverted-op-97-1369-pulsar-bookie-0                          1/1     Running                  0                6h11m   10.104.33.223   4am-node36   <none>           <none>
json-inverted-op-97-1369-pulsar-bookie-1                          1/1     Running                  0                6h11m   10.104.21.110   4am-node24   <none>           <none>
json-inverted-op-97-1369-pulsar-bookie-2                          1/1     Running                  0                6h11m   10.104.23.89    4am-node27   <none>           <none>
json-inverted-op-97-1369-pulsar-bookie-init-glq7l                 0/1     Completed                0                6h11m   10.104.33.215   4am-node36   <none>           <none>
json-inverted-op-97-1369-pulsar-broker-0                          1/1     Running                  0                6h11m   10.104.13.131   4am-node16   <none>           <none>
json-inverted-op-97-1369-pulsar-proxy-0                           1/1     Running                  0                6h11m   10.104.21.103   4am-node24   <none>           <none>
json-inverted-op-97-1369-pulsar-pulsar-init-hpb6k                 0/1     Completed                0                6h11m   10.104.6.91     4am-node13   <none>           <none>
json-inverted-op-97-1369-pulsar-recovery-0                        1/1     Running                  0                6h11m   10.104.13.134   4am-node16   <none>           <none>
json-inverted-op-97-1369-pulsar-zookeeper-0                       1/1     Running                  0                6h11m   10.104.23.86    4am-node27   <none>           <none>
json-inverted-op-97-1369-pulsar-zookeeper-1                       1/1     Running                  0                6h10m   10.104.33.234   4am-node36   <none>           <none>
json-inverted-op-97-1369-pulsar-zookeeper-2                       1/1     Running                  0                6h10m   10.104.18.227   4am-node25   <none>           <none>

Anything else?

No response

[xiaofan-luan]

/assign @xiaocai2333
might need a atomicity check here?

[xiaocai2333]

/assign @xiaocai2333 might need a atomicity check here?

This has already been discussed offline with @JsDove : the issue arises because JSONKeyTask does not ensure execution on sorted segment. This results in the sort stats task being skipped for the segment, which in turn blocks index building.
@JsDove is working on fixing it.