Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][retrieve] Milvus standalone OOM after writing about 471M data serially #38792

Open
1 task done
wangting0128 opened this issue Dec 26, 2024 · 0 comments
Open
1 task done

[Bug]: [benchmark][retrieve] Milvus standalone OOM after writing about 471M data serially #38792

wangting0128 opened this issue Dec 26, 2024 · 0 comments
Assignees
@sunby @yanliang567
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@wangting0128
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:sunby-no_unlink-ceed4f7-20241225
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc124
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-qsssq

server:

NAME                                                              READY   STATUS                   RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
wt-test-unlink-5-etcd-0                                           1/1     Running                  0                110m    10.104.16.180   4am-node21   <none>           <none>
wt-test-unlink-5-milvus-standalone-5f56b78dd9-29xlt               0/1     CrashLoopBackOff         21 (2m21s ago)   110m    10.104.15.76    4am-node20   <none>           <none>
wt-test-unlink-5-minio-7f65646cc5-75c7h                           1/1     Running                  0                110m    10.104.30.222   4am-node38   <none>           <none>
截屏2024-12-26 20 00 53 截屏2024-12-26 20 01 25

client log:

[2024-12-26 10:12:45,560 -  INFO - fouram]: [Base] Connection params: {'alias': 'default', 'host': 'wt-test-unlink-5-milvus.qa-milvus.svc.cluster.local', 'port': '19530', 'uri': '', 'secure': False, 'user': '', 'password': '', 'token': '', 'db_name': ''} (base.py:240)
[2024-12-26 10:12:45,575 -  INFO - fouram]: [Base] Create collection scene_float_1 (base.py:273)
[2024-12-26 10:12:45,590 -  INFO - fouram]: [Base] Collection schema: 
{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'binary_vector', 'description': '', 'type': <DataType.BINARY_VECTOR: 100>, 'params': {'dim': 8}},
            {'name': 'float_1', 'description': '', 'type': <DataType.FLOAT: 10>}],
 'enable_dynamic_field': False} (base.py:329)
[2024-12-26 10:12:45,590 -  INFO - fouram]: [CommonCases] Prepare collection scene_float_1 done. (common_cases.py:77)
[2024-12-26 10:12:45,593 -  INFO - fouram]: [Base] Collection:scene_float_1 is not building index (base.py:494)
[2024-12-26 10:12:45,593 -  INFO - fouram]: [Base] Start release collection scene_float_1 (base.py:324)
[2024-12-26 10:12:45,597 -  INFO - fouram]: [Base] Clean all index done. (base.py:518)
[2024-12-26 10:12:45,597 -  INFO - fouram]: [Base] Start build index of BIN_FLAT for field:binary_vector collection:scene_float_1, params:{'index_type': 'BIN_FLAT', 'metric_type': 'JACCARD', 'params': {}}, kwargs:{} (base.py:472)
[2024-12-26 10:12:46,106 -  INFO - fouram]: [Time] Index run in 0.5094s (api_request.py:49)
[2024-12-26 10:12:46,106 -  INFO - fouram]: [CommonCases] RT of build index BIN_FLAT: 0.5094s (common_cases.py:168)
[2024-12-26 10:12:46,106 -  INFO - fouram]: [CommonCases] Prepare index BIN_FLAT done. (common_cases.py:170)
[2024-12-26 10:12:46,107 -  INFO - fouram]: [CommonCases] No scalar and vector fields need to be indexed. (common_cases.py:189)
[2024-12-26 10:12:46,108 -  INFO - fouram]: [Base] Index params of scene_float_1:[{'binary_vector': {'index_type': 'BIN_FLAT', 'metric_type': 'JACCARD', 'params': {}}}] (base.py:491)
[2024-12-26 10:12:46,108 -  INFO - fouram]: [Base] Start inserting 8200000000 vectors to collection scene_float_1 (base.py:386)
[2024-12-26 10:12:59,511 -  INFO - fouram]: [Base] Start inserting, ids: 0 - 1999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:13:00,984 -  INFO - fouram]: [Time] Collection.insert run in 1.4722s (api_request.py:49)
[2024-12-26 10:13:00,987 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 0 (base.py:535)
[2024-12-26 10:13:14,117 -  INFO - fouram]: [Base] Start inserting, ids: 2000000 - 3999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:13:15,559 -  INFO - fouram]: [Time] Collection.insert run in 1.4408s (api_request.py:49)
[2024-12-26 10:13:15,562 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 2000000 (base.py:535)
[2024-12-26 10:13:29,218 -  INFO - fouram]: [Base] Start inserting, ids: 4000000 - 5999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:13:30,763 -  INFO - fouram]: [Time] Collection.insert run in 1.5433s (api_request.py:49)
[2024-12-26 10:13:30,766 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 4000000 (base.py:535)
[2024-12-26 10:13:44,060 -  INFO - fouram]: [Base] Start inserting, ids: 6000000 - 7999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:13:45,542 -  INFO - fouram]: [Time] Collection.insert run in 1.4813s (api_request.py:49)
[2024-12-26 10:13:45,544 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 6000000 (base.py:535)
[2024-12-26 10:13:58,833 -  INFO - fouram]: [Base] Start inserting, ids: 8000000 - 9999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:14:00,282 -  INFO - fouram]: [Time] Collection.insert run in 1.4481s (api_request.py:49)
[2024-12-26 10:14:00,284 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 8000000 (base.py:535)
[2024-12-26 10:14:13,647 -  INFO - fouram]: [Base] Start inserting, ids: 10000000 - 11999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:14:15,059 -  INFO - fouram]: [Time] Collection.insert run in 1.41s (api_request.py:49)
[2024-12-26 10:14:15,061 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 10000000 (base.py:535)
[2024-12-26 10:14:28,566 -  INFO - fouram]: [Base] Start inserting, ids: 12000000 - 13999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:14:30,219 -  INFO - fouram]: [Time] Collection.insert run in 1.652s (api_request.py:49)
[2024-12-26 10:14:30,221 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 12000000 (base.py:535)
[2024-12-26 10:14:44,275 -  INFO - fouram]: [Base] Start inserting, ids: 14000000 - 15999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:14:45,782 -  INFO - fouram]: [Time] Collection.insert run in 1.5062s (api_request.py:49)
[2024-12-26 10:14:45,785 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 14000000 (base.py:535)
[2024-12-26 10:14:59,686 -  INFO - fouram]: [Base] Start inserting, ids: 16000000 - 17999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:15:01,164 -  INFO - fouram]: [Time] Collection.insert run in 1.4776s (api_request.py:49)
[2024-12-26 10:15:01,166 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 16000000 (base.py:535)
[2024-12-26 10:15:15,267 -  INFO - fouram]: [Base] Start inserting, ids: 18000000 - 19999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:15:16,763 -  INFO - fouram]: [Time] Collection.insert run in 1.4951s (api_request.py:49)
[2024-12-26 10:15:16,765 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 18000000 (base.py:535)
[2024-12-26 10:15:31,091 -  INFO - fouram]: [Base] Start inserting, ids: 20000000 - 21999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:15:32,576 -  INFO - fouram]: [Time] Collection.insert run in 1.4838s (api_request.py:49)
[2024-12-26 10:15:32,578 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 20000000 (base.py:535)
[2024-12-26 10:15:46,359 -  INFO - fouram]: [Base] Start inserting, ids: 22000000 - 23999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:15:47,868 -  INFO - fouram]: [Time] Collection.insert run in 1.5084s (api_request.py:49)
[2024-12-26 10:15:47,870 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 22000000 (base.py:535)
[2024-12-26 10:16:01,503 -  INFO - fouram]: [Base] Start inserting, ids: 24000000 - 25999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:16:02,893 -  INFO - fouram]: [Time] Collection.insert run in 1.3895s (api_request.py:49)
[2024-12-26 10:16:02,896 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 24000000 (base.py:535)
[2024-12-26 10:16:16,373 -  INFO - fouram]: [Base] Start inserting, ids: 26000000 - 27999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:16:17,845 -  INFO - fouram]: [Time] Collection.insert run in 1.4715s (api_request.py:49)
[2024-12-26 10:16:17,848 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 26000000 (base.py:535)
[2024-12-26 10:16:31,150 -  INFO - fouram]: [Base] Start inserting, ids: 28000000 - 29999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:16:32,593 -  INFO - fouram]: [Time] Collection.insert run in 1.4425s (api_request.py:49)
[2024-12-26 10:16:32,595 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 28000000 (base.py:535)
[2024-12-26 10:16:45,947 -  INFO - fouram]: [Base] Start inserting, ids: 30000000 - 31999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:16:47,444 -  INFO - fouram]: [Time] Collection.insert run in 1.4962s (api_request.py:49)
[2024-12-26 10:16:47,446 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 30000000 (base.py:535)
[2024-12-26 10:17:00,850 -  INFO - fouram]: [Base] Start inserting, ids: 32000000 - 33999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:17:02,371 -  INFO - fouram]: [Time] Collection.insert run in 1.5209s (api_request.py:49)
[2024-12-26 10:17:02,373 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 32000000 (base.py:535)
[2024-12-26 10:17:15,814 -  INFO - fouram]: [Base] Start inserting, ids: 34000000 - 35999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:17:17,834 -  INFO - fouram]: [Time] Collection.insert run in 2.0195s (api_request.py:49)
[2024-12-26 10:17:17,898 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 34000000 (base.py:535)
[2024-12-26 10:17:31,145 -  INFO - fouram]: [Base] Start inserting, ids: 36000000 - 37999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:17:34,001 -  INFO - fouram]: [Time] Collection.insert run in 2.8549s (api_request.py:49)
[2024-12-26 10:17:34,154 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 36000000 (base.py:535)
[2024-12-26 10:17:47,893 -  INFO - fouram]: [Base] Start inserting, ids: 38000000 - 39999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:17:51,097 -  INFO - fouram]: [Time] Collection.insert run in 3.2037s (api_request.py:49)
[2024-12-26 10:17:51,109 -  INFO - fouram]: [Base] Number of vectors in the collection(scene_float_1): 38000000 (base.py:535)
[2024-12-26 10:18:04,541 -  INFO - fouram]: [Base] Start inserting, ids: 40000000 - 41999999, data size: 8,200,000,000 (base.py:366)
[2024-12-26 10:18:08,467 -  INFO - fouram]: [batch_insert] retry:4, cost: 0.27s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:09,403 -  INFO - fouram]: [batch_insert] retry:5, cost: 0.81s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:10,873 -  INFO - fouram]: [batch_insert] retry:6, cost: 2.43s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:14,019 -  INFO - fouram]: [batch_insert] retry:7, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:17,744 -  INFO - fouram]: [batch_insert] retry:8, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:21,394 -  INFO - fouram]: [batch_insert] retry:9, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:25,095 -  INFO - fouram]: [batch_insert] retry:10, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:28,762 -  INFO - fouram]: [batch_insert] retry:11, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:32,426 -  INFO - fouram]: [batch_insert] retry:12, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:36,082 -  INFO - fouram]: [batch_insert] retry:13, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:39,744 -  INFO - fouram]: [batch_insert] retry:14, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:43,465 -  INFO - fouram]: [batch_insert] retry:15, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:47,147 -  INFO - fouram]: [batch_insert] retry:16, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:50,867 -  INFO - fouram]: [batch_insert] retry:17, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:54,553 -  INFO - fouram]: [batch_insert] retry:18, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:18:58,264 -  INFO - fouram]: [batch_insert] retry:19, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:01,989 -  INFO - fouram]: [batch_insert] retry:20, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:05,662 -  INFO - fouram]: [batch_insert] retry:21, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:09,340 -  INFO - fouram]: [batch_insert] retry:22, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:13,015 -  INFO - fouram]: [batch_insert] retry:23, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:16,687 -  INFO - fouram]: [batch_insert] retry:24, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:20,356 -  INFO - fouram]: [batch_insert] retry:25, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:24,047 -  INFO - fouram]: [batch_insert] retry:26, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:27,837 -  INFO - fouram]: [batch_insert] retry:27, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:31,653 -  INFO - fouram]: [batch_insert] retry:28, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:35,471 -  INFO - fouram]: [batch_insert] retry:29, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:39,227 -  INFO - fouram]: [batch_insert] retry:30, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:42,911 -  INFO - fouram]: [batch_insert] retry:31, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:46,657 -  INFO - fouram]: [batch_insert] retry:32, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:50,408 -  INFO - fouram]: [batch_insert] retry:33, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:54,170 -  INFO - fouram]: [batch_insert] retry:34, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:19:57,842 -  INFO - fouram]: [batch_insert] retry:35, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:01,528 -  INFO - fouram]: [batch_insert] retry:36, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:05,202 -  INFO - fouram]: [batch_insert] retry:37, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:08,894 -  INFO - fouram]: [batch_insert] retry:38, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:12,586 -  INFO - fouram]: [batch_insert] retry:39, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:16,235 -  INFO - fouram]: [batch_insert] retry:40, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:19,903 -  INFO - fouram]: [batch_insert] retry:41, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:23,546 -  INFO - fouram]: [batch_insert] retry:42, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:27,188 -  INFO - fouram]: [batch_insert] retry:43, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:30,837 -  INFO - fouram]: [batch_insert] retry:44, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:34,495 -  INFO - fouram]: [batch_insert] retry:45, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:38,152 -  INFO - fouram]: [batch_insert] retry:46, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:41,803 -  INFO - fouram]: [batch_insert] retry:47, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:45,508 -  INFO - fouram]: [batch_insert] retry:48, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:49,151 -  INFO - fouram]: [batch_insert] retry:49, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:52,794 -  INFO - fouram]: [batch_insert] retry:50, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:20:56,464 -  INFO - fouram]: [batch_insert] retry:51, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:00,113 -  INFO - fouram]: [batch_insert] retry:52, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:03,765 -  INFO - fouram]: [batch_insert] retry:53, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:07,410 -  INFO - fouram]: [batch_insert] retry:54, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:11,118 -  INFO - fouram]: [batch_insert] retry:55, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:14,772 -  INFO - fouram]: [batch_insert] retry:56, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:18,412 -  INFO - fouram]: [batch_insert] retry:57, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:22,127 -  INFO - fouram]: [batch_insert] retry:58, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:25,773 -  INFO - fouram]: [batch_insert] retry:59, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:29,420 -  INFO - fouram]: [batch_insert] retry:60, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:33,154 -  INFO - fouram]: [batch_insert] retry:61, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:36,811 -  INFO - fouram]: [batch_insert] retry:62, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:40,506 -  INFO - fouram]: [batch_insert] retry:63, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:44,161 -  INFO - fouram]: [batch_insert] retry:64, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:47,812 -  INFO - fouram]: [batch_insert] retry:65, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:51,475 -  INFO - fouram]: [batch_insert] retry:66, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:55,133 -  INFO - fouram]: [batch_insert] retry:67, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:21:58,793 -  INFO - fouram]: [batch_insert] retry:68, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:02,463 -  INFO - fouram]: [batch_insert] retry:69, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:06,160 -  INFO - fouram]: [batch_insert] retry:70, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:09,980 -  INFO - fouram]: [batch_insert] retry:71, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:13,697 -  INFO - fouram]: [batch_insert] retry:72, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:17,507 -  INFO - fouram]: [batch_insert] retry:73, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:21,186 -  INFO - fouram]: [batch_insert] retry:74, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:24,872 -  INFO - fouram]: [batch_insert] retry:75, cost: 3.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)> (decorators.py:100)
[2024-12-26 10:22:28,579 - ERROR - fouram]: RPC error: [batch_insert], <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-12-26T10:22:28.578161535+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)"}"
>>, message=Retry run out of 75 retry times, message=failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111))>, <Time:{'RPC start': '2024-12-26 10:18:04.541858', 'RPC error': '2024-12-26 10:22:28.579088'}> (decorators.py:140)
[2024-12-26 10:22:28,581 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-12-26T10:22:28.578161535+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)"}"
>>, message=Retry run out of 75 retry times, message=failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111))>, [requestId: b19e7784-c372-11ef-bac7-86980e7afab4] (api_request.py:57)
[2024-12-26 10:22:28,581 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=<bound method _MultiThreadedRendezvous.code of <_MultiThreadedRendezvous of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-12-26T10:22:28.578161535+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111)"}"
>>, message=Retry run out of 75 retry times, message=failed to connect to all addresses; last error: UNKNOWN: ipv4:10.255.34.185:19530: Failed to connect to remote host: connect: Connection refused (111))> (func_check.py:106)

Expected Behavior

No response

Steps To Reproduce

1. create a collection with fiels: 'id'(INT64 primary key), 'binary_vector'(8dim), 'float_1'
2. build BIN_FLAT index on field binary_vector
3. insert data <- milvus OOM when inserting 32m data

Milvus Log

etcd:
birdwather.txt

Anything else?

server config:

extraConfigFiles:
  user.yaml: |
    dataCoord:
      segment:
        sealProportion: 1
    indexCoord:
      scheduler:
        interval: 1
    queryNode:
      cache:
        warmup: sync
      mmap:
        vectorField: true
        vectorIndex: true
        scalarField: true
        scalarIndex: true
standalone:
  resources:
    limits:
      cpu: '2.0'
      memory: 16Gi
    requests:
      cpu: '1.0'
      memory: 10Gi
  nodeSelector:
    node-role/nvme: 'true'

client config:

{
     "dataset_params": {
          "metric_type": "JACCARD",
          "vector_field_name": "binary_vector",
          "dim": 8,
          "dataset_name": "local",
          "dataset_size": "8200m",
          "ni_per": 2000000
     },
     "collection_params": {
          "shards_num": 1,
          "other_fields": [
               "float_1"
          ],
          "collection_name": "scene_float_1"
     },
     "index_params": {
          "index_type": "BIN_FLAT",
          "index_param": {}
     },
     "concurrent_params": {
          "concurrent_number": [
               10,
               100
          ],
          "during_time": "30m",
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "expr": "",
                    "output_fields": [
                         "float_1"
                    ],
                    "limit": 10,
                    "timeout": 600,
                    "random_data": true,
                    "random_count": 10,
                    "random_range": [
                         0,
                         8200000000
                    ],
                    "field_name": "id",
                    "field_type": "int64"
               }
          }
     ]
}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunby sunby

@yanliang567 yanliang567

Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sunby @wangting0128 @yanliang567