datanode崩溃问题 #38198

silear · 2024-12-04T02:50:20Z

silear
Dec 4, 2024

您好，我在集群上部署milvus2.4.14，平稳运行了一段时间。做了数据量上限的测试，插入了1000w数据，datanode崩溃，重启服务，datanode反复重启，报oomkilled，调大了datanode的内存，现在mixcoord、proxy、datanode均无法启动，日志如下：

Welcome to use Milvus!
Version: v2.4.14
Built: Tue Oct 29 09:50:17 UTC 2024

GoVersion: go version go1.21.11 linux/amd64

open pid file: /run/milvus/datanode.pid
lock pid file: /run/milvus/datanode.pid
[2024/12/04 02:19:12.317 +00:00] [INFO] [roles/roles.go:333] ["starting running Milvus components"]
[2024/12/04 02:19:12.317 +00:00] [INFO] [roles/roles.go:182] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/12/04 02:19:12.325 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{"UseEmbed":false,"EnableAuth":false,"UserName":"","PassWord":"","UseSSL":false,"Endpoints":["t-milvus-etcd-0.t-milvus-etcd-headless.default.svc.cluster.local:2379","t-milvus-etcd-1.t-milvus-etcd-headless.default.svc.cluster.local:2379","t-milvus-etcd-2.t-milvus-etcd-headless.default.svc.cluster.local:2379"],"KeyPrefix":"by-dev","CertFile":"/path/to/etcd-client.pem","KeyFile":"/path/to/etcd-client-key.pem","CaCertFile":"/path/to/ca.pem","MinVersion":"1.3","RefreshInterval":5000000000}"]
[2024/12/04 02:19:12.325 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[t-milvus-etcd-0.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-1.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-2.t-milvus-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/12/04 02:19:12.325 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/12/04 02:19:12.328 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[t-milvus-etcd-0.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-1.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-2.t-milvus-etcd-headless.default.svc.cluster.local:2379]"]
[2024/12/04 02:19:12.329 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
[2024/12/04 02:19:12.330 +00:00] [INFO] [paramtable/component_param.go:4243] ["DeployModeEnv is not set, use default"] [default=0.5]
[2024/12/04 02:19:12.330 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/12/04 02:19:12.331 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2024/12/04 02:19:12.331 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2024/12/04 02:19:12.331 +00:00] [INFO] [roles/roles.go:282] [setupPrometheusHTTPServer]
[2024/12/04 02:19:12.331 +00:00] [INFO] [http/server.go:160] ["management listen"] [addr=:9091]
[2024/12/04 02:19:12.331 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=7730941132]
[2024/12/04 02:19:12.331 +00:00] [INFO] [datanode/service.go:102] ["DataNode listen on"] [address="[::]:21124"] [port=21124]
[2024/12/04 02:19:12.332 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[t-milvus-etcd-0.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-1.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-2.t-milvus-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/12/04 02:19:12.334 +00:00] [INFO] [datanode/service.go:253] ["DataNode address"] [address=12.11.0.120:21124]
[2024/12/04 02:19:12.334 +00:00] [INFO] [datanode/service.go:254] ["DataNode serverID"] [serverID=0]
[2024/12/04 02:19:12.435 +00:00] [INFO] [datanode/service.go:263] ["initializing RootCoord client for DataNode"]
[2024/12/04 02:19:12.435 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[t-milvus-etcd-0.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-1.t-milvus-etcd-headless.default.svc.cluster.local:2379,t-milvus-etcd-2.t-milvus-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/12/04 02:19:12.442 +00:00] [WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:12.442 +00:00] [WARN] [grpcclient/client.go:488] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:12.443 +00:00] [INFO] [grpcclient/client.go:238] ["previous client closed"] [role=rootcoord] [addr=]
[2024/12/04 02:19:12.444 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:12.645 +00:00] [WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:12.645 +00:00] [WARN] [grpcclient/client.go:488] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:12.645 +00:00] [INFO] [grpcclient/client.go:238] ["previous client closed"] [role=rootcoord] [addr=]
[2024/12/04 02:19:13.047 +00:00] [WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:13.047 +00:00] [WARN] [grpcclient/client.go:488] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:13.047 +00:00] [INFO] [grpcclient/client.go:238] ["previous client closed"] [role=rootcoord] [addr=]
[2024/12/04 02:19:13.849 +00:00] [WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:13.849 +00:00] [WARN] [grpcclient/client.go:488] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:13.849 +00:00] [INFO] [grpcclient/client.go:238] ["previous client closed"] [role=rootcoord] [addr=]
[2024/12/04 02:19:15.451 +00:00] [WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:15.451 +00:00] [WARN] [grpcclient/client.go:488] ["start to reset connection because of specific reasons"] [client_role=rootcoord] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:15.451 +00:00] [INFO] [grpcclient/client.go:238] ["previous client closed"] [role=rootcoord] [addr=]
[2024/12/04 02:19:15.452 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=4] [error="rpc error: code = Unknown desc = node not match[expectedNodeID=1045][actualNodeID=1048]"]
[2024/12/04 02:19:18.653 +00:00] [INFO] [componentutil/componentutil.go:61] ["WaitForComponentStates success"] ["current state"=Healthy]
[2024/12/04 02:19:18.653 +00:00] [INFO] [datanode/service.go:274] ["RootCoord client is ready for DataNode"]
[2024/12/04 02:19:18.655 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:18.655 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:18.655 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentInitOrHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:71\n | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n | \t/workspace/source/internal/distributed/datanode/service.go:289\n | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n | \t/workspace/source/internal/distributed/datanode/service.go:184\n | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n | \t/workspace/source/cmd/components/data_node.go:59\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:126\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:18.655 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:18.656 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:18.656 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetComponentStates\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:121\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentInitOrHealthy[...]\n | \t/workspace/source/internal/util/componentutil/componentutil.go:71\n | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).init\n | \t/workspace/source/internal/distributed/datanode/service.go:289\n | github.com/milvus-io/milvus/internal/distributed/datanode.(*Server).Run\n | \t/workspace/source/internal/distributed/datanode/service.go:184\n | github.com/milvus-io/milvus/cmd/components.(*DataNode).Run\n | \t/workspace/source/cmd/components/data_node.go:59\n | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n | \t/workspace/source/cmd/roles/roles.go:126\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

yhmo · 2024-12-04T03:31:49Z

yhmo
Dec 4, 2024
Collaborator

有没有mixcoord的日志？

11 replies

silear Dec 4, 2024
Author

[2024/12/04 02:19:15.551 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:15.551 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:15.552 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:15.552 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:15.619 +00:00] [WARN] [grpcclient/client.go:404] ["call received grpc error"] [clientRole=rootcoord] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2024/12/04 02:19:15.620 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=4] [error="rpc error: code = Canceled desc = grpc: the client connection is closing"]
[2024/12/04 02:19:15.620 +00:00] [WARN] [grpcclient/client.go:556] ["ClientBase Call grpc call get error"] [role=rootcoord] [address=12.11.0.122:53100] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/rootcoord/client/client.go:639 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n/workspace/source/internal/datacoord/broker/coordinator_broker.go:123 github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n/workspace/source/internal/datacoord/meta.go:234 github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n/workspace/source/internal/datacoord/server.go:671 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/datacoord/server.go:670 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:555\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n | \t/workspace/source/internal/datacoord/meta.go:234\n | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n | \t/workspace/source/internal/datacoord/server.go:671\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n | \t/workspace/source/internal/datacoord/server.go:670\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/rootcoord/client/client.go:639 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n | /workspace/source/internal/datacoord/broker/coordinator_broker.go:123 github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n | /workspace/source/internal/datacoord/meta.go:234 github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n | /workspace/source/internal/datacoord/server.go:671 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n | /workspace/source/internal/datacoord/server.go:670 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2024/12/04 02:19:15.620 +00:00] [WARN] [broker/coordinator_broker.go:127] ["failed to ListDatabases"] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/rootcoord/client/client.go:639 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n/workspace/source/internal/datacoord/broker/coordinator_broker.go:123 github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n/workspace/source/internal/datacoord/meta.go:234 github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n/workspace/source/internal/datacoord/server.go:671 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/datacoord/server.go:670 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:555\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n | \t/workspace/source/internal/datacoord/meta.go:234\n | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n | \t/workspace/source/internal/datacoord/server.go:671\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n | \t/workspace/source/internal/datacoord/server.go:670\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/rootcoord/client/client.go:639 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n | /workspace/source/internal/datacoord/broker/coordinator_broker.go:123 github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n | /workspace/source/internal/datacoord/meta.go:234 github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n | /workspace/source/internal/datacoord/server.go:671 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n | /workspace/source/internal/datacoord/server.go:670 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
[2024/12/04 02:19:15.620 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/rootcoord/client/client.go:639 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n/workspace/source/internal/datacoord/broker/coordinator_broker.go:123 github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n/workspace/source/internal/datacoord/meta.go:234 github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n/workspace/source/internal/datacoord/server.go:671 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n/workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n/workspace/source/internal/datacoord/server.go:670 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: rpc error: code = Canceled desc = grpc: the client connection is closing"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = Canceled desc = grpc: the client connection is closing\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:555\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n | \t/workspace/source/internal/datacoord/meta.go:234\n | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n | \t/workspace/source/internal/datacoord/server.go:671\n | github.com/milvus-io/milvus/pkg/util/retry.Do\n | \t/workspace/source/pkg/util/retry/retry.go:44\n | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n | \t/workspace/source/internal/datacoord/server.go:670\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/rootcoord/client/client.go:639 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n | /workspace/source/internal/datacoord/broker/coordinator_broker.go:123 github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n | /workspace/source/internal/datacoord/meta.go:234 github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n | /workspace/source/internal/datacoord/server.go:671 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n | /workspace/source/pkg/util/retry/retry.go:44 github.com/milvus-io/milvus/pkg/util/retry.Do\n | /workspace/source/internal/datacoord/server.go:670 github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) rpc error: code = Canceled desc = grpc: the client connection is closing\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]

silear Dec 4, 2024
Author

[2024/12/04 02:19:15.990 +00:00] [WARN] [rootcoord/root_coord.go:1179] ["failed to describe collection"] [collectionName=] [dbName=] [id=454078395778423185] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:15.990 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395778423185] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:15.990 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:16.191 +00:00] [WARN] [rootcoord/root_coord.go:1179] ["failed to describe collection"] [collectionName=] [dbName=] [id=454078395778423185] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:16.191 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395778423185] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:16.591 +00:00] [WARN] [rootcoord/root_coord.go:1179] ["failed to describe collection"] [collectionName=] [dbName=] [id=454078395778423185] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:16.591 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395778423185] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:16.622 +00:00] [INFO] [rootcoord/root_coord.go:931] ["received request to list databases"] [msgID=0]
[2024/12/04 02:19:16.622 +00:00] [INFO] [rootcoord/root_coord.go:957] ["done to list databases"] [msgID=0] ["num of databases"=2]
[2024/12/04 02:19:16.623 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395769758799]
[2024/12/04 02:19:16.623 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395769758799]
[2024/12/04 02:19:16.624 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395774072047]
[2024/12/04 02:19:16.624 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395774072047]
[2024/12/04 02:19:16.625 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395775210007]
[2024/12/04 02:19:16.625 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395775210007]
[2024/12/04 02:19:16.625 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772995367]
[2024/12/04 02:19:16.625 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772995367]
[2024/12/04 02:19:16.625 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772792066]
[2024/12/04 02:19:16.625 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772792066]
[2024/12/04 02:19:16.626 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772996696]
[2024/12/04 02:19:16.626 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772996696]
[2024/12/04 02:19:16.626 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395778195899]
[2024/12/04 02:19:16.626 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395778195899]
[2024/12/04 02:19:16.627 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773859706]
[2024/12/04 02:19:16.627 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773859706]
[2024/12/04 02:19:16.627 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395776042153]
[2024/12/04 02:19:16.627 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395776042153]
[2024/12/04 02:19:16.627 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395775411974]
[2024/12/04 02:19:16.627 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395775411974]
[2024/12/04 02:19:16.628 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772793212]
[2024/12/04 02:19:16.628 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772793212]
[2024/12/04 02:19:16.628 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772792245]
[2024/12/04 02:19:16.628 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772792245]
[2024/12/04 02:19:16.629 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772590544]
[2024/12/04 02:19:16.629 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772590544]
[2024/12/04 02:19:16.629 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772794406]
[2024/12/04 02:19:16.629 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772794406]
[2024/12/04 02:19:16.630 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773405418]
[2024/12/04 02:19:16.630 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773405418]
[2024/12/04 02:19:16.630 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395774786756]
[2024/12/04 02:19:16.630 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395774786756]
[2024/12/04 02:19:16.631 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395777537020]
[2024/12/04 02:19:16.631 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395777537020]
[2024/12/04 02:19:16.631 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772589291]
[2024/12/04 02:19:16.631 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772589291]
[2024/12/04 02:19:16.631 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772995597]
[2024/12/04 02:19:16.631 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772995597]
[2024/12/04 02:19:16.632 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395774994421]
[2024/12/04 02:19:16.632 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395774994421]
[2024/12/04 02:19:16.632 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395776246422]
[2024/12/04 02:19:16.632 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395776246422]
[2024/12/04 02:19:16.633 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773199422]
[2024/12/04 02:19:16.633 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773199422]
[2024/12/04 02:19:16.633 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773616177]
[2024/12/04 02:19:16.633 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773616177]
[2024/12/04 02:19:16.634 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772793470]
[2024/12/04 02:19:16.634 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772793470]
[2024/12/04 02:19:16.634 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773407806]
[2024/12/04 02:19:16.634 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773407806]
[2024/12/04 02:19:16.635 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773615758]
[2024/12/04 02:19:16.635 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773615758]
[2024/12/04 02:19:16.635 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395774367538]
[2024/12/04 02:19:16.635 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395774367538]
[2024/12/04 02:19:16.636 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395772791242]
[2024/12/04 02:19:16.636 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395772791242]
[2024/12/04 02:19:16.636 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395773858691]
[2024/12/04 02:19:16.636 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395773858691]
[2024/12/04 02:19:16.637 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395777110525]
[2024/12/04 02:19:16.637 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395777110525]
[2024/12/04 02:19:16.637 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395776463396]
[2024/12/04 02:19:16.637 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395776463396]
[2024/12/04 02:19:16.638 +00:00] [INFO] [datacoord/meta.go:272] ["meta update: add collection"] [collectionID=454078395777748011]
[2024/12/04 02:19:16.638 +00:00] [INFO] [datacoord/meta.go:277] ["meta update: add collection - complete"] [collectionID=454078395777748011]

silear Dec 4, 2024
Author

[2024/12/04 02:19:17.138 +00:00] [WARN] [proxyutil/proxy_client_manager.go:288] ["proxy client is empty, GetMetrics will not send to any client"]
[2024/12/04 02:19:17.139 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:17.139 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:17.139 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:17.139 +00:00] [WARN] [grpcclient/client.go:453] ["fail to get grpc client"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:17.139 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querycoord] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:272\n | github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:239\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n | \t/workspace/source/internal/rootcoord/quota_center.go:393\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:17.139 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:402\n | github.com/milvus-io/milvus/internal/rootcoord.getDataCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:257\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func2\n | \t/workspace/source/internal/rootcoord/quota_center.go:438\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:17.140 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:17.140 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:17.140 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:272\n | github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:239\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n | \t/workspace/source/internal/rootcoord/quota_center.go:393\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:17.140 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:17.140 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:17.140 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=0] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:402\n | github.com/milvus-io/milvus/internal/rootcoord.getDataCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:257\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func2\n | \t/workspace/source/internal/rootcoord/quota_center.go:438\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:17.151 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querycoord] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).ReleaseCollection\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:164\n | github.com/milvus-io/milvus/internal/rootcoord.(*ServerBroker).ReleaseCollection\n | \t/workspace/source/internal/rootcoord/broker.go:76\n | github.com/milvus-io/milvus/internal/rootcoord.(*releaseCollectionStep).Execute\n | \t/workspace/source/internal/rootcoord/step.go:293\n | github.com/milvus-io/milvus/internal/rootcoord.(*stepStack).Execute\n | \t/workspace/source/internal/rootcoord/step_executor.go:59\n | github.com/milvus-io/milvus/internal/rootcoord.(*bgStepExecutor).process.func1\n | \t/workspace/source/internal/rootcoord/step_executor.go:201\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

silear Dec 4, 2024
Author

[2024/12/04 02:19:21.994 +00:00] [WARN] [datacoord/handler.go:444] ["datacoord ServerHandler GetCollection finally failed"] [collectionID=454078395778423185] [error="collection not found[collection=454078395778423185]"]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:313] ["NEITHER segment position or channel start position are found, setting channel seek position to collection start position"] [channel=by-dev-rootcoord-dml_1_454078395778423185v3] [posTs=0] [posTime=1970/01/01 00:00:00.000 +00:00]
[2024/12/04 02:19:21.994 +00:00] [WARN] [datacoord/handler.go:363] ["get channel checkpoint failed, channelCPMeta and earliestSegDMLPos and collStartPos are all invalid"] [channel=by-dev-rootcoord-dml_1_454078395778423185v3]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395781565817] [channel=by-dev-rootcoord-dml_2_454078395781565817v0] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_2_454078395781565817v0] [posTs=454330655271813121] [posTime=2024/12/02 10:05:08.355 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395794954214] [channel=by-dev-rootcoord-dml_15_454078395794954214v3] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_15_454078395794954214v3] [posTs=454263305743368193] [posTime=2024/11/29 10:43:10.293 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395780848142] [channel=by-dev-rootcoord-dml_6_454078395780848142v1] [numOfSegments=0]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_6_454078395780848142v1] [posTs=454263305846652930] [posTime=2024/11/29 10:43:10.687 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395800510482] [channel=by-dev-rootcoord-dml_9_454078395800510482v1] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_9_454078395800510482v1] [posTs=454284471801479169] [posTime=2024/11/30 09:08:52.397 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395796012830] [channel=by-dev-rootcoord-dml_0_454078395796012830v2] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_454078395796012830v2] [posTs=454263305323937794] [posTime=2024/11/29 10:43:08.693 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395782027786] [channel=by-dev-rootcoord-dml_12_454078395782027786v1] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_12_454078395782027786v1] [posTs=454338099755089921] [posTime=2024/12/02 17:58:26.805 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395789143157] [channel=by-dev-rootcoord-dml_9_454078395789143157v1] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_9_454078395789143157v1] [posTs=454263305650044930] [posTime=2024/11/29 10:43:09.937 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395771651366] [channel=by-dev-rootcoord-dml_1_454078395771651366v2] [numOfSegments=1]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:341] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_1_454078395771651366v2] [posTs=454332622321483778] [posTime=2024/12/02 12:10:12.054 +00:00]
[2024/12/04 02:19:21.994 +00:00] [INFO] [datacoord/handler.go:60] [GetDataVChanPositions] [collectionID=454078395790450304] [channel=by-dev-rootcoord-dml_6_454078395790450304v1] [numOfSegments=1]

silear Dec 4, 2024
Author

[2024/12/04 02:19:21.995 +00:00] [WARN] [rootcoord/root_coord.go:1179] ["failed to describe collection"] [collectionName=] [dbName=] [id=454078395800104160] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:21.995 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395800104160] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:21.995 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:22.196 +00:00] [WARN] [rootcoord/root_coord.go:1179] ["failed to describe collection"] [collectionName=] [dbName=] [id=454078395800104160] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:22.196 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395800104160] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:22.597 +00:00] [WARN] [rootcoord/root_coord.go:1179] ["failed to describe collection"] [collectionName=] [dbName=] [id=454078395800104160] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:22.597 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395800104160] [error="collection not found[collection=454078395800104160]"]
[2024/12/04 02:19:23.348 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=querycoord] [error="empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:272\n | github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:239\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n | \t/workspace/source/internal/rootcoord/quota_center.go:393\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:23.348 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=datacoord] [error="empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:402\n | github.com/milvus-io/milvus/internal/rootcoord.getDataCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:257\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func2\n | \t/workspace/source/internal/rootcoord/quota_center.go:438\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/12/04 02:19:23.349 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:23.349 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=querycoord] [error="find no available querycoord, check querycoord state"]
[2024/12/04 02:19:23.349 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:23.350 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=datacoord] [error="find no available datacoord, check datacoord state"]
[2024/12/04 02:19:23.349 +00:00] [WARN] [grpcclient/client.go:556] ["ClientBase Call grpc call get error"] [role=querycoord] [address=] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/querycoord/client/client.go:100 github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/querycoord/client/client.go:272 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n/workspace/source/internal/rootcoord/util.go:239 github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n/workspace/source/internal/rootcoord/quota_center.go:393 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: empty grpc client: find no available querycoord, check querycoord state"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:555\n | [...repeated from below...]\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/querycoord/client/client.go:100 github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/querycoord/client/client.go:272 github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n | /workspace/source/internal/rootcoord/util.go:239 github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n | /workspace/source/internal/rootcoord/quota_center.go:393 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 golang.org/x/sync/errgroup.(*Group).Go.func1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/querycoord/client/client.go:272\n | github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:239\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n | \t/workspace/source/internal/rootcoord/quota_center.go:393\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) empty grpc client\nWraps: (5) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString"]
[2024/12/04 02:19:23.350 +00:00] [WARN] [grpcclient/client.go:556] ["ClientBase Call grpc call get error"] [role=datacoord] [address=] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/datacoord/client/client.go:402 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n/workspace/source/internal/rootcoord/util.go:257 github.com/milvus-io/milvus/internal/rootcoord.getDataCoordMetrics\n/workspace/source/internal/rootcoord/quota_center.go:438 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func2\n/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 golang.org/x/sync/errgroup.(*Group).Go.func1\n/usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit: empty grpc client: find no available datacoord, check datacoord state"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: empty grpc client: find no available datacoord, check datacoord state\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:555\n | [...repeated from below...]\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/datacoord/client/client.go:402 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n | /workspace/source/internal/rootcoord/util.go:257 github.com/milvus-io/milvus/internal/rootcoord.getDataCoordMetrics\n | /workspace/source/internal/rootcoord/quota_center.go:438 github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func2\n | /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78 golang.org/x/sync/errgroup.(*Group).Go.func1\n | /usr/local/go/src/runtime/asm_amd64.s:1650 runtime.goexit\nWraps: (3) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n | \t/workspace/source/internal/util/grpcclient/client.go:473\n | github.com/milvus-io/milvus/pkg/util/retry.Handle\n | \t/workspace/source/pkg/util/retry/retry.go:104\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n | \t/workspace/source/internal/util/grpcclient/client.go:466\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:553\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).GetMetrics\n | \t/workspace/source/internal/distributed/datacoord/client/client.go:402\n | github.com/milvus-io/milvus/internal/rootcoord.getDataCoordMetrics\n | \t/workspace/source/internal/rootcoord/util.go:257\n | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func2\n | \t/workspace/source/internal/rootcoord/quota_center.go:438\n | golang.org/x/sync/errgroup.(*Group).Go.func1\n | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (4) empty grpc client\nWraps: (5) find no available datacoord, check datacoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *errors.errorString"]

yhmo · 2024-12-04T04:22:33Z

yhmo
Dec 4, 2024
Collaborator

像是querycoord在初始化的过程中崩了，你用这个脚本把集群的日志都导出来发给我们看看，最好是用文件的形式attach上来
https://github.com/milvus-io/milvus/tree/master/deployments/export-log

0 replies

xiaofan-luan · 2024-12-04T04:41:08Z

xiaofan-luan
Dec 4, 2024
Maintainer

[2024/12/04 02:19:21.995 +00:00] [WARN] [datacoord/handler.go:439] ["failed to load collection from rootcoord"] [collectionID=454078395800104160] [error="collection not found[collection=454078395800104160]"]

there seems to be collecton information missing.
What operation you do after your restart? Any manual operation on etcd data or minio data?

0 replies

xiaofan-luan · 2024-12-04T04:42:25Z

xiaofan-luan
Dec 4, 2024
Maintainer

@silear
you need to track your log about 454078395800104160. Is it deleted? we need coordinator log for the 1-2 days to traceback.

Right now, I fixed could be clean all the meta on etcd for collection 454078395800104160 using birdwatcher

30 replies

silear Dec 13, 2024
Author

正常情况下，如果flush执行完毕，是不是datanode的cpu利用率会降下来？flush执行大概多长时间？与数据量成正比吗？我发现在插入过程中，cpu升起来，我停止插入，cpu很久都不会降下来，是flush没有执行完吗，还是什么原因

yhmo Dec 13, 2024
Collaborator

flush由内部逻辑决定，默认growing segment如果达到100M左右就触发flush，growing segment如果存在时间超过10分钟，就触发flush。flush仅仅是写磁盘，没什么计算量的，磁盘快就flush快，最多也就秒级。我测试下来datanode的cpu利用率保持在10%以下，datanode真没什么计算的事情。
你可以间化流程，不建索引，不load，只insert

xiaofan-luan Dec 13, 2024
Maintainer

正常情况下，如果flush执行完毕，是不是datanode的cpu利用率会降下来？flush执行大概多长时间？与数据量成正比吗？我发现在插入过程中，cpu升起来，我停止插入，cpu很久都不会降下来，是flush没有执行完吗，还是什么原因

确定是datanode的cpu么，还是index node？indexnode构建索引时间比较长，不降下来是正常的。如果datanode不降下来，大概说明是消费积压了，从流里面消费不过来

silear Dec 13, 2024
Author

确定是datanode，消费积压原因是什么？与我的kafka规格有关吗？要如何解决呢

xiaofan-luan Dec 13, 2024
Maintainer

方便用pprof 连接9091，分析下cpu开销来源么？

xiaofan-luan · 2024-12-09T12:05:50Z

xiaofan-luan
Dec 9, 2024
Maintainer

1c4g通常情况不够，尤其是indexnode，建议至少2c8g。

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

datanode崩溃问题 #38198

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 41 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

datanode崩溃问题 #38198

silear Dec 4, 2024

Replies: 5 comments · 41 replies

yhmo Dec 4, 2024 Collaborator

silear Dec 4, 2024 Author

silear Dec 4, 2024 Author

silear Dec 4, 2024 Author

silear Dec 4, 2024 Author

silear Dec 4, 2024 Author

yhmo Dec 4, 2024 Collaborator

xiaofan-luan Dec 4, 2024 Maintainer

xiaofan-luan Dec 4, 2024 Maintainer

silear Dec 13, 2024 Author

yhmo Dec 13, 2024 Collaborator

xiaofan-luan Dec 13, 2024 Maintainer

silear Dec 13, 2024 Author

xiaofan-luan Dec 13, 2024 Maintainer

xiaofan-luan Dec 9, 2024 Maintainer

silear
Dec 4, 2024

Replies: 5 comments 41 replies

yhmo
Dec 4, 2024
Collaborator

silear Dec 4, 2024
Author

silear Dec 4, 2024
Author

silear Dec 4, 2024
Author

silear Dec 4, 2024
Author

silear Dec 4, 2024
Author

yhmo
Dec 4, 2024
Collaborator

xiaofan-luan
Dec 4, 2024
Maintainer

xiaofan-luan
Dec 4, 2024
Maintainer

silear Dec 13, 2024
Author

yhmo Dec 13, 2024
Collaborator

xiaofan-luan Dec 13, 2024
Maintainer

silear Dec 13, 2024
Author

xiaofan-luan Dec 13, 2024
Maintainer

xiaofan-luan
Dec 9, 2024
Maintainer