/assign @xiaocai2333
/unassign

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.5-20241220-8cbe8da9-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: multi-vector-corn-1-1734789600 test case name: test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
multi-vector-corn-1-1734789600-1-etcd-0                           1/1     Running            0               3h7m    10.104.30.34    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-etcd-1                           1/1     Running            0               3h7m    10.104.26.159   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-etcd-2                           1/1     Running            0               3h7m    10.104.33.235   4am-node36   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-datanode-bbf7967962zh94   1/1     Running            2 (3h7m ago)    3h7m    10.104.21.190   4am-node24   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8fdh99j   1/1     Running            2 (3h7m ago)    3h7m    10.104.21.188   4am-node24   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8ffh8wd   1/1     Running            2 (3h7m ago)    3h7m    10.104.17.5     4am-node23   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8frxxvx   1/1     Running            2 (3h7m ago)    3h7m    10.104.25.134   4am-node30   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-indexnode-55966f8fs2jrg   1/1     Running            2 (3h7m ago)    3h7m    10.104.27.116   4am-node31   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-mixcoord-99656c8f86bzxr   1/1     Running            2 (3h7m ago)    3h7m    10.104.17.4     4am-node23   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-proxy-757456df88-zstwf    1/1     Running            2 (3h7m ago)    3h7m    10.104.21.189   4am-node24   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-querynode-64c79698g5jxm   1/1     Running            2 (3h7m ago)    3h7m    10.104.15.143   4am-node20   <none>           <none>
multi-vector-corn-1-1734789600-1-milvus-querynode-64c79698kqx69   1/1     Running            2 (3h7m ago)    3h7m    10.104.20.58    4am-node22   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-0                          1/1     Running            0               3h7m    10.104.30.35    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-1                          1/1     Running            0               3h7m    10.104.26.158   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-2                          1/1     Running            0               3h7m    10.104.33.232   4am-node36   <none>           <none>
multi-vector-corn-1-1734789600-1-minio-3                          1/1     Running            0               3h7m    10.104.32.134   4am-node39   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-0                1/1     Running            0               3h7m    10.104.30.37    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-1                1/1     Running            0               3h7m    10.104.33.237   4am-node36   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-2                1/1     Running            0               3h7m    10.104.26.164   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-bookie-init-b64fh       0/1     Completed          0               3h7m    10.104.6.74     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-broker-0                1/1     Running            0               3h7m    10.104.6.76     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-broker-1                1/1     Running            0               3h7m    10.104.14.4     4am-node18   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-proxy-0                 1/1     Running            0               3h7m    10.104.6.71     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-proxy-1                 1/1     Running            0               3h7m    10.104.9.51     4am-node14   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-pulsar-init-l62fs       0/1     Completed          0               3h7m    10.104.6.72     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-recovery-0              1/1     Running            0               3h7m    10.104.6.77     4am-node13   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-zookeeper-0             1/1     Running            0               3h7m    10.104.30.33    4am-node38   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-zookeeper-1             1/1     Running            0               3h7m    10.104.26.160   4am-node32   <none>           <none>
multi-vector-corn-1-1734789600-1-pulsarv3-zookeeper-2             1/1     Running            0               3h7m    10.104.33.236   4am-node36   <none>           <none>

{pod=~"multi-vector-corn-1-1734789600-1-milvus-.*"} |~ "492bd40ad8b73896695b8ee7d7cd1e89"

client log:

, 'scene_test_partition_hybrid_search_WMV5LDzd', ''], kwargs: {}, [requestId: c0749536-bfa8-11ef-b10b-7693a8272652] (api_request.py:77)
[2024-12-21 14:35:25,111 - DEBUG - fouram]: (api_response) : [Partition] {"name":"scene_test_partition_hybrid_search_WMV5LDzd","collection_name":"fouram_hsF7bpBM","description":""}, [requestId: c0749536-bfa8-11ef-b10b-7693a8272652] (api_request.py:44)
[2024-12-21 14:35:25,112 - DEBUG - fouram]: [Base] Create partition scene_test_partition_hybrid_search_WMV5LDzd of collection(fouram_hsF7bpBM) (base.py:821)
[2024-12-21 14:35:25,247 - DEBUG - fouram]: (api_request)  : [Collection.insert] args: <Collection.insert fields: 7, length: 3000, content: [ [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'list'>, dtype<>` [0.19579262613213944, 0. ... ], [ `type<class 'list'>, dtype<>` [0.85696574318921, 0.722 ... ], [ `type<class 'list'>, dtype<>` [0.6981304524714663, 0.5 ... ], [ `type<class 'list'>, dtype<>` [0.4047851630760757, 0.0 ... ], [ `type<class 'int'>, dtype<>` 0 ... ], [ `type<class 'str'>, dtype<>` 0 ... ] ]>, ['scene_test_partition_hybrid_search_WMV5LDzd'], kwargs: {'timeout': 600}, [requestId: d0ee73be-bfa8-11ef-b10b-7693a8272652] (api_request.py:77)
[2024-12-21 14:36:11,341 - DEBUG - fouram]: [Base] Start flush partition scene_test_partition_hybrid_search_WMV5LDzd, kwargs: {'timeout': 600} (base.py:836)
[2024-12-21 14:36:31,580 - DEBUG - fouram]: [Base] Partition scene_test_partition_hybrid_search_WMV5LDzd num entities: (3000) (base.py:832)
[2024-12-21 14:36:31,745 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:int64_1, index_params:{'index_type': 'INVERTED'} (base.py:1126)
[2024-12-21 14:36:32,259 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:varchar_1, index_params:{'index_type': 'INVERTED'} (base.py:1126)
[2024-12-21 14:36:33,259 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector, index_params:{'params': {'nlist': 1024}, 'index_type': 'IVF_FLAT', 'metric_type': 'L2'} (base.py:1126)
[2024-12-21 14:36:33,776 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector_1, index_params:{'metric_type': 'L2', 'params': {'M': 8, 'efConstruction': 200}, 'index_type': 'HNSW'} (base.py:1126)
[2024-12-21 14:36:34,770 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector_2, index_params:{'index_type': 'DISKANN', 'metric_type': 'IP', 'params': {}} (base.py:1126)
[2024-12-21 14:36:35,286 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:float_vector_3, index_params:{'metric_type': 'L2', 'params': {'nlist': 2048}, 'index_type': 'IVF_SQ8'} (base.py:1126)
[2024-12-21 14:37:04,042 - DEBUG - fouram]: [Base] Partition:scene_test_partition_hybrid_search_WMV5LDzd start building index, field_name:id, index_params:{} (base.py:1126)
[2024-12-21 14:37:58,791 - DEBUG - fouram]: [Base] Start load partition scene_test_partition_hybrid_search_WMV5LDzd, replica_number:1, kwargs:{'timeout': 600} (base.py:842)
[2024-12-21 14:39:45,129 - DEBUG - fouram]: [Base] Params of partition:scene_test_partition_hybrid_search_WMV5LDzd hybrid_search: reqs:[{'anns_field': 'float_vector', 'param': {'metric_type': 'L2', 'params': {'nprobe': 128}}, 'limit': 100, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_1', 'param': {'metric_type': 'L2', 'params': {'ef': 64}}, 'limit': 10, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_2', 'param': {'metric_type': 'IP', 'params': {'search_list': 32}}, 'limit': 30, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_3', 'param': {'metric_type': 'L2', 'params': {'nprobe': 16}}, 'limit': 400, 'expr': None, 'nq': 1}], rerank:{'strategy': 'rrf', 'params': {'k': 60}}, limit:1, timeout:600, kwargs:{'check_task': 'check_response'} (base.py:871)
[2024-12-21 14:39:51,338 - DEBUG - fouram]: [Base] Start release partition scene_test_partition_hybrid_search_WMV5LDzd (base.py:848)
[2024-12-21 14:41:06,795 - DEBUG - fouram]: [Base] Params of partition:scene_test_partition_hybrid_search_WMV5LDzd hybrid_search: reqs:[{'anns_field': 'float_vector', 'param': {'metric_type': 'L2', 'params': {'nprobe': 128}}, 'limit': 100, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_1', 'param': {'metric_type': 'L2', 'params': {'ef': 64}}, 'limit': 10, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_2', 'param': {'metric_type': 'IP', 'params': {'search_list': 32}}, 'limit': 30, 'expr': None, 'nq': 1}, {'anns_field': 'float_vector_3', 'param': {'metric_type': 'L2', 'params': {'nprobe': 16}}, 'limit': 400, 'expr': None, 'nq': 1}], rerank:{'strategy': 'rrf', 'params': {'k': 60}}, limit:1, timeout:600, kwargs:{'check_task': 'check_error_response', 'check_items': {'code': 65535, 'message': 'not loaded'}} (base.py:871)
[2024-12-21 14:41:10,411 - DEBUG - fouram]: (api_request)  : [Partition.drop] args: [], kwargs: {'timeout': 600}, [requestId: 9eaa535e-bfa9-11ef-b10b-7693a8272652] (api_request.py:77)
[2024-12-21 14:41:42,516 - ERROR - fouram]: (api_response) : [Partition.drop] <MilvusException: (code=65535, message=segment was removed from the loading map early)>, [requestId: 9eaa535e-bfa9-11ef-b10b-7693a8272652] (api_request.py:57)

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `DQL & DML(partition)`
            verify concurrent DQL & DML(partition) scenario,
            which has 4 vector fields(IVF_FLAT, HNSW, DISKANN, IVF_SQ8) and scalar fields: `int64_1`, `varchar_1`

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim,
                'float_vector_1': 128dim,
                'float_vector_2': 128dim,
                'float_vector_3': 128dim,
                scalar field: int64_1, varchar_1
            2. build indexes:
                IVF_FLAT: 'float_vector'
                HNSW: 'float_vector_1',
                DISKANN: 'float_vector_2'
                IVF_SQ8: 'float_vector_3'
                INVERTED: 'int64_1', 'varchar_1'
                default scalar index: 'id'
            3. insert 1 million data into 10 partitions
            4. flush collection
            5. build indexes again using the same params
            6. load collection
                replica: 1
            7. concurrent request:
                - scene_test_partition_hybrid_search
                    (partition: create->insert->flush->index again->load->hybrid_search->release->hybrid_search failed->drop)
                - search
                - hybrid_search
                - query

Milvus Log

No response

Anything else?

test result:

[2024-12-21 17:21:18,861 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     hybrid_search                                                                   3255     0(0.00%) |   3914      17   37035   2400 |    0.30        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     query                                                                            400     0(0.00%) |   4785      62   65199    490 |    0.04        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     scene_test_partition_hybrid_search                                               393     1(0.25%) | 317768   65544  669267 310000 |    0.04        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: grpc     search                                                                          3145     0(0.00%) |  23646    2548   52879  23000 |    0.29        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]:          Aggregated                                                                      7193     1(0.01%) |  29738      17  669267  14000 |    0.67        0.00 (stats.py:789)
[2024-12-21 17:21:18,861 -  INFO - fouram]:  (stats.py:790)
[2024-12-21 17:21:18,866 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'cluster',
            'config_name': 'cluster_2c8m',
            'config': {'queryNode': {'resources': {'limits': {'cpu': '32.0', 'memory': '32Gi'}, 'requests': {'cpu': '17.0', 'memory': '17Gi'}}, 'replicas': 2},
                       'indexNode': {'resources': {'limits': {'cpu': '8.0', 'memory': '8Gi'}, 'requests': {'cpu': '5.0', 'memory': '5Gi'}}, 'replicas': 4},
                       'dataNode': {'resources': {'limits': {'cpu': '2.0', 'memory': '8Gi'}, 'requests': {'cpu': '2.0', 'memory': '5Gi'}}},
                       'cluster': {'enabled': True},
                       'pulsarv3': {},
                       'kafka': {},
                       'minio': {'metrics': {'podMonitor': {'enabled': True}}},
                       'etcd': {'metrics': {'enabled': True, 'podMonitor': {'enabled': True}}},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus', 'tag': '2.5-20241220-8cbe8da9-amd64'}}},
            'host': 'multi-vector-corn-1-1734789600-1-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_hybrid_search_locust_dql_dml_partition_hybrid_search_cluster',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'scalars_index': {'id': {}, 'int64_1': {'index_type': 'INVERTED'}, 'varchar_1': {'index_type': 'INVERTED'}},
                                                    'vectors_index': {'float_vector_1': {'index_type': 'HNSW',
                                                                                         'index_param': {'M': 8, 'efConstruction': 200},
                                                                                         'metric_type': 'L2'},
                                                                      'float_vector_2': {'index_type': 'DISKANN', 'index_param': {}, 'metric_type': 'IP'},
                                                                      'float_vector_3': {'index_type': 'IVF_SQ8',
                                                                                         'index_param': {'nlist': 2048},
                                                                                         'metric_type': 'L2'}},
                                                    'scalars_params': {'float_vector_1': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_2': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}},
                                                                       'float_vector_3': {'params': {'dim': 128}, 'other_params': {'dataset': 'sift'}}},
                                                    'extra_partitions': {'partitions': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                        'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                        'partition_9'],
                                                                         'data_repeated': False},
                                                    'dataset_name': 'sift',
                                                    'dataset_size': 1000000,
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['float_vector_1', 'float_vector_2', 'float_vector_3', 'int64_1', 'varchar_1'],
                                                       'shards_num': 2},
                                 'resource_groups_params': {'reset': False},
                                 'database_user_params': {'reset_rbac': False, 'reset_db': False},
                                 'index_params': {'index_type': 'IVF_FLAT', 'index_param': {'nlist': 1024}},
                                 'concurrent_params': {'concurrent_number': 20, 'during_time': '3h', 'interval': 20, 'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'scene_test_partition_hybrid_search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 1,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10},
                                                                           {'search_param': {'search_list': 32}, 'anns_field': 'float_vector_2', 'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'RRFRanker': []},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'hybrid_search_counts': 1,
                                                                  'data_size': 3000,
                                                                  'ni': 3000}},
                                                      {'type': 'search',
                                                       'weight': 8,
                                                       'params': {'nq': 1000,
                                                                  'top_k': 1,
                                                                  'search_param': {'nprobe': 1000},
                                                                  'expr': 'int64_1 >= 0',
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'output_fields': None,
                                                                  'ignore_growing': False,
                                                                  'group_by_field': None,
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'hybrid_search',
                                                       'weight': 8,
                                                       'params': {'nq': 1,
                                                                  'top_k': 100,
                                                                  'reqs': [{'search_param': {'nprobe': 128}, 'anns_field': 'float_vector', 'top_k': 100},
                                                                           {'search_param': {'ef': 64}, 'anns_field': 'float_vector_1', 'top_k': 10},
                                                                           {'search_param': {'search_list': 32}, 'anns_field': 'float_vector_2', 'top_k': 30},
                                                                           {'search_param': {'nprobe': 16}, 'anns_field': 'float_vector_3', 'top_k': 400}],
                                                                  'rerank': {'WeightedRanker': [0.85, 0.95, 0.51, 0.32]},
                                                                  'output_fields': ['*'],
                                                                  'ignore_growing': False,
                                                                  'guarantee_timestamp': None,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'timeout': 600,
                                                                  'random_data': True,
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'ids': None,
                                                                  'expr': 'int64_1 > -1 && ',
                                                                  'output_fields': ['*'],
                                                                  'offset': None,
                                                                  'limit': None,
                                                                  'ignore_growing': False,
                                                                  'partition_names': ['_default', 'partition_1', 'partition_2', 'partition_3', 'partition_4',
                                                                                      'partition_5', 'partition_6', 'partition_7', 'partition_8',
                                                                                      'partition_9'],
                                                                  'timeout': 600,
                                                                  'consistency_level': None,
                                                                  'random_data': True,
                                                                  'random_count': 20,
                                                                  'random_range': [0, 100000],
                                                                  'field_name': 'id',
                                                                  'field_type': 'int64',
                                                                  'custom_expr': None,
                                                                  'custom_range': [0, 1],
                                                                  'check_task': 'check_response',
                                                                  'check_items': None}}]},
            'run_id': 2024122104547868,
            'datetime': '2024-12-21 14:14:14.362982',
            'client_version': '2.5.0'},
 'result': {'test_result': {'index': {'RT': 22.1728,
                                      'float_vector_1': {'RT': 0.5159},
                                      'float_vector_2': {'RT': 1.5223},
                                      'float_vector_3': {'RT': 0.5149},
                                      'id': {'RT': 0.5167},
                                      'int64_1': {'RT': 0.5163},
                                      'varchar_1': {'RT': 0.515}},
                            'insert': {'total_time': 121.5605, 'VPS': 8227.1488, 'batch_time': 1.2156, 'batch': 10000.0},
                            'flush': {'RT': 2.5249},
                            'load': {'RT': 2.2435},
                            'Locust': {'Aggregated': {'Requests': 7193,
                                                      'Fails': 1,
                                                      'RPS': 0.67,
                                                      'fail_s': 0.0,
                                                      'RT_max': 669267.21,
                                                      'RT_avg': 29738.27,
                                                      'TP50': 14000.0,
                                                      'TP99': 399000.0},
                                       'hybrid_search': {'Requests': 3255,
                                                         'Fails': 0,
                                                         'RPS': 0.3,
                                                         'fail_s': 0.0,
                                                         'RT_max': 37035.08,
                                                         'RT_avg': 3914.48,
                                                         'TP50': 2400.0,
                                                         'TP99': 21000.0},
                                       'query': {'Requests': 400,
                                                 'Fails': 0,
                                                 'RPS': 0.04,
                                                 'fail_s': 0.0,
                                                 'RT_max': 65199.15,
                                                 'RT_avg': 4785.13,
                                                 'TP50': 500.0,
                                                 'TP99': 48000.0},
                                       'scene_test_partition_hybrid_search': {'Requests': 393,
                                                                              'Fails': 1,
                                                                              'RPS': 0.04,
                                                                              'fail_s': 0.0,
                                                                              'RT_max': 669267.21,
                                                                              'RT_avg': 317768.67,
                                                                              'TP50': 310000.0,
                                                                              'TP99': 615000.0},
                                       'search': {'Requests': 3145,
                                                  'Fails': 0,
                                                  'RPS': 0.29,
                                                  'fail_s': 0.0,
                                                  'RT_max': 52879.95,
                                                  'RT_avg': 23646.62,
                                                  'TP50': 23000.0,
                                                  'TP99': 42000.0}}}}}

guess this is not bug.
Drop parttion and load happens at the same time.

/assign @wangting0128
please check your test scripts