DataCoord memory keep increasing

[byeongjokim]

datacoord memory keep increasing and pod died

Could you tell me why this issue occurs and how to prevent it?

• milvus: 2.2.13

[yhmo]

How much memory is assigned to the datacoord?

[byeongjokim]

@yhmo
it was 1Gi.

[yhmo]

Assign more memory(2GB) to this datacoord, the memory usage should less than a certain value. If it still OOM with 2GB memory, then it could be a memory leak bug. So far as I know, we didn't get any memory leak issues for datacoord.

[byeongjokim]

@yhmo
The memory usage of datacoord(memory: 2GB) continues to grow.
Could you check that there is no problem with memory leak in milvus v2.2.13?

[xiaofan-luan]

THis could definitely improved
We will test in house on this situation also.
so far our suggestion to give more memory for datacoord, for instance - 4g

[byeongjokim]

@xiaofan-luan

There is a misunderstanding.
The current datacoord memory is 4Gi...

After the test, please share the results asap.
thank you.

[xiaofan-luan]

From the flame graph you shared, we only saw 1GB memory usage?

[xiaofan-luan]

when you say 4GB, did it mean only datacoord?
how large is your pod limit and request?

[byeongjokim]

@xiaofan-luan

limit: 4GB

There is the other question..

i set active-standby for ha (replicas=2).
However there is 3 cases.
Assume that, there is A, B two datacoord pods.

1. An active A pod died with oom error caused by above memory problem. -> B pod is still in standby mode.
2. I delete active A pod using kubectl command (kubectl delete pod --force ). -> B pod is changed to active mode.
3. I generate oom error on active A pod using stress-ng command (stress-ng --brk 2 --stack 2 --bigheap 2). -> B pod is changed to active mode.

the cases 2, 3 mean the active-standby works,
however the case 1 mean the active-standby doesnot work.

are there any differences between three cases?

in our guess...
case 1: The A pod died and again lived, in between the period of liveness metric for active-standby.