Bug: Increased latencies on store-gatway after updating to 2.14.2

[ecktom]

What is the bug?

Hi guys, we've recently upgraded Mimir from 2.13.0 > 2.14.2 and noticed an sharp increase of latencies for the the store-gateway on the read path which we cannot make any reasoning of.

Store-gateway average went up from 50ms -> 1s / p99 went up from 1s -> 25s

We're not having set any special configurations for the store-gateway. Did someone experience some similar issues?

How to reproduce it?

What did you think would happen?

What was your environment?

• Helm Chart 5.5.1
• Mimir 2.14.2
• Microservices mode

Any additional context to share?

No response

[narqo]

Could you expand on your environment: which object storage do you run on (s3, gcs, something on-prem)? Could you identify any particular object storage operations, where the latency increased (the "Mimir / Reads" dashboard should have the "Block object storage (stote-gateway)" set of panels)? Does the caching still work as expected? Anything particular that the "Mimir / Reads resources" dashboard shows?

Also, do you happen to have traces that could reveal the source of latency?