Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abnormal bandwidth usage of nats cluster service [v2.10.22 || v2.10.23] #6238

Open
SucceedGetRich opened this issue Dec 10, 2024 · 4 comments
Labels
defect Suspected defect such as a bug or regression

Comments

@SucceedGetRich
Copy link

Observed behavior

I used AWS servers, using servers in different countries. There were servers in four countries in total, three servers in each country, and a total of 12 servers to build the nats cluster.The number of copies is 3, the message volume is not large, but the bandwidth usage is abnormal.

clipboard_image_1733815894601
clipboard_image_1733824781486

Expected behavior

There are more than thirty streams in total, and there are about twenty consumers every month. If twelve clusters are used as a cluster, the upstream and downstream bandwidth should not be so high.

Server and client version

Using the latest nats-server version

Host environment

No response

Steps to reproduce

No response

@SucceedGetRich SucceedGetRich added the defect Suspected defect such as a bug or regression label Dec 10, 2024
@wallyqs
Copy link
Member

wallyqs commented Dec 11, 2024

Is the topology 12 servers clustered together with routes over different regions? Do you see any readloop processing errors or Slow Consumer errors in the logs? Have you tried modifying the max_outstanding_catchup setting from JetStream to something lower max_outstanding_catchup=32mb?

@wallyqs wallyqs changed the title Abnormal bandwidth usage of nats cluster service Abnormal bandwidth usage of nats cluster service [v2.10.22, v2.10.23] Dec 11, 2024
@wallyqs wallyqs changed the title Abnormal bandwidth usage of nats cluster service [v2.10.22, v2.10.23] Abnormal bandwidth usage of nats cluster service [v2.10.22 || v2.10.23] Dec 11, 2024
@SucceedGetRich
Copy link
Author

拓扑结构中 12 台服务器是否通过不同区域的路由聚集在一起?您是否在日志中看到任何 readloop 处理错误或 Slow Consumer 错误?您是否尝试过将设置max_outstanding_catchup从 JetStream 修改为更低的值max_outstanding_catchup=32mb

Can a nats cluster specify a machine as a cluster?

@SucceedGetRich
Copy link
Author

拓扑结构中 12 台服务器是否通过不同区域的路由聚集?您是否在日志中存在任何 readloop 处理错误或慢速消费者错误?您是否尝试过设置max_outstanding_catchup从 JetStream 修改互连的值max_outstanding_catchup=32mb

max_outstanding_catchup defaults to 32mb

@wallyqs
Copy link
Member

wallyqs commented Dec 11, 2024

it might be helpful to share the output of nats server ls and nats server report js using the system account

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Development

No branches or pull requests

2 participants