mimir-distributor in restart loop #8926

mrclrchtr · 2024-08-07T11:27:40Z

mrclrchtr
Aug 7, 2024

Hi,

my mimir-distributor is in a restart loop and I don't know why exactly and what to do.

so I would be happy for every advice!

Some logs:

ts=2024-08-07T11:19:31.140502276Z caller=main.go:225 level=info msg="Starting application" version="(version=2.13.0, branch=HEAD, revision=4775ec1)"
ts=2024-08-07T11:19:31.14618636Z caller=server.go:352 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2024-08-07T11:19:31.155833355Z caller=memberlist_client.go:435 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-distributor-74d5b44c7-kfmpx-3585e879
ts=2024-08-07T11:19:31.161005435Z caller=memberlist_client.go:541 level=info msg="memberlist fast-join starting" nodes_found=8 to_join=4
ts=2024-08-07T11:19:31.162504647Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2024-08-07T11:19:31.162602008Z caller=module_service.go:82 level=info msg=starting module=active-groups-cleanup-service
ts=2024-08-07T11:19:31.162712889Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2024-08-07T11:19:31.162699849Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2024-08-07T11:19:31.162753529Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2024-08-07T11:19:31.162761529Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2024-08-07T11:19:31.162787169Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2024-08-07T11:19:31.162807129Z caller=module_service.go:82 level=info msg=starting module=server
ts=2024-08-07T11:19:31.163018811Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2024-08-07T11:19:31.163045491Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2024-08-07T11:19:31.163312253Z caller=module_service.go:82 level=info msg=starting module=ingester-ring
ts=2024-08-07T11:19:31.170034666Z caller=memberlist_client.go:561 level=info msg="memberlist fast-join finished" joined_nodes=4 elapsed_time=11.051286ms
ts=2024-08-07T11:19:31.170094786Z caller=memberlist_client.go:573 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.monitoring.svc.xxx
ts=2024-08-07T11:19:31.171040034Z caller=module_service.go:82 level=info msg=starting module=distributor-service
ts=2024-08-07T11:19:31.171332636Z caller=basic_lifecycler.go:295 level=info msg="instance found in the ring" instance=mimir-distributor-74d5b44c7-kfmpx ring=distributor state=ACTIVE tokens=1 registered_at="2024-08-07 11:18:18 +0000 UTC"
ts=2024-08-07T11:19:31.171756919Z caller=distributor.go:628 level=info msg="waiting until distributor is ACTIVE in the ring"
ts=2024-08-07T11:19:31.17183328Z caller=mimir.go:890 level=info msg="Application started"
ts=2024-08-07T11:19:31.197738682Z caller=memberlist_client.go:580 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=8 elapsed_time=27.631496ms
ts=2024-08-07T11:19:31.264712483Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.20.252:7946"
ts=2024-08-07T11:19:31.267774587Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.21.212:7946"
ts=2024-08-07T11:19:31.271557737Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.19.190:7946"
ts=2024-08-07T11:19:32.97964348Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.19.190:7946"
ts=2024-08-07T11:19:34.981915555Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.20.249:7946"
ts=2024-08-07T11:19:34.982935723Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.20.133:7946"
ts=2024-08-07T11:19:34.983955931Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-eb98afdf' from=10.0.20.252:7946"
ts=2024-08-07T11:19:38.312109014Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.20.249:7946"
ts=2024-08-07T11:19:39.260806163Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.19.201:7946"
ts=2024-08-07T11:19:40.314392729Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.20.133:7946"
ts=2024-08-07T11:19:40.31577218Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.21.14:7946"
ts=2024-08-07T11:19:40.316929109Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.19.190:7946"
ts=2024-08-07T11:19:41.263802164Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.21.212:7946"
ts=2024-08-07T11:19:41.264343409Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.20.252:7946"
ts=2024-08-07T11:19:41.26578066Z caller=log.go:245 level=warn msg="Got ping for unexpected node 'mimir-distributor-74d5b44c7-kfmpx-0670df0b' from=10.0.20.249:7946"

ts=2024-08-07T11:20:54.09909693Z caller=main.go:225 level=info msg="Starting application" version="(version=2.13.0, branch=HEAD, revision=4775ec1)"
ts=2024-08-07T11:20:54.102856919Z caller=server.go:352 level=info msg="server listening on addresses" http=[::]:8080 grpc=[::]:9095
ts=2024-08-07T11:20:54.1106069Z caller=memberlist_client.go:435 level=info msg="Using memberlist cluster label and node name" cluster_label= node=mimir-distributor-74d5b44c7-kfmpx-40e12355
ts=2024-08-07T11:20:54.114899133Z caller=module_service.go:82 level=info msg=starting module=sanity-check
ts=2024-08-07T11:20:54.115012774Z caller=sanity_check.go:32 level=info msg="Checking directories read/write access"
ts=2024-08-07T11:20:54.115029774Z caller=sanity_check.go:37 level=info msg="Directories read/write access successfully checked"
ts=2024-08-07T11:20:54.115034494Z caller=sanity_check.go:39 level=info msg="Checking object storage config"
ts=2024-08-07T11:20:54.115004254Z caller=module_service.go:82 level=info msg=starting module=activity-tracker
ts=2024-08-07T11:20:54.115008934Z caller=module_service.go:82 level=info msg=starting module=active-groups-cleanup-service
ts=2024-08-07T11:20:54.115049854Z caller=sanity_check.go:44 level=info msg="Object storage config successfully checked"
ts=2024-08-07T11:20:54.115064415Z caller=module_service.go:82 level=info msg=starting module=server
ts=2024-08-07T11:20:54.115170095Z caller=module_service.go:82 level=info msg=starting module=runtime-config
ts=2024-08-07T11:20:54.115162255Z caller=module_service.go:82 level=info msg=starting module=memberlist-kv
ts=2024-08-07T11:20:54.115149295Z caller=memberlist_client.go:541 level=info msg="memberlist fast-join starting" nodes_found=8 to_join=4
ts=2024-08-07T11:20:54.115593099Z caller=module_service.go:82 level=info msg=starting module=ingester-ring
ts=2024-08-07T11:20:54.126955027Z caller=memberlist_client.go:561 level=info msg="memberlist fast-join finished" joined_nodes=4 elapsed_time=13.755227ms
ts=2024-08-07T11:20:54.127043148Z caller=memberlist_client.go:573 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.monitoring.svc.xxx
ts=2024-08-07T11:20:54.127700313Z caller=module_service.go:82 level=info msg=starting module=distributor-service
ts=2024-08-07T11:20:54.127897275Z caller=basic_lifecycler.go:295 level=info msg="instance found in the ring" instance=mimir-distributor-74d5b44c7-kfmpx ring=distributor state=ACTIVE tokens=1 registered_at="2024-08-07 11:18:18 +0000 UTC"
ts=2024-08-07T11:20:54.128041636Z caller=distributor.go:628 level=info msg="waiting until distributor is ACTIVE in the ring"
ts=2024-08-07T11:20:54.128105396Z caller=mimir.go:890 level=info msg="Application started"
ts=2024-08-07T11:20:54.154041158Z caller=memberlist_client.go:580 level=info phase=startup msg="joining memberlist cluster succeeded" reached_nodes=8 elapsed_time=26.98537ms

No special distributor configs:

global:
  clusterDomain: xxx
  extraEnvFrom:
    - secretRef:
        name: mimir-s3-credentials

minio:
  enabled: false

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          xxx
    blocks_storage:
      s3:
        bucket_name: mimir-blocks
    alertmanager_storage:
      s3:
        bucket_name: mimir-alertmanager
    ruler_storage:
      s3:
        bucket_name: mimir-ruler
    limits:
      ingestion_rate: 100000
      ingestion_burst_size: 1000000
      max_global_series_per_user: 1000000

ruler:
  enabled: false

alertmanager:
  enabled: false

ingester:
  zoneAwareReplication:
    enabled: false
  persistentVolume:
    storageClass: longhorn-no-replica
    size: 6Gi
  resources:
    requests:
      cpu: 133m
      memory: 1816Mi
    limits:
      memory: 1816Mi

compactor:
  persistentVolume:
    storageClass: longhorn-no-replica
    size: 6Gi
  resources:
    requests:
      cpu: 10m
      memory: 151Mi
    limits:
      memory: 151Mi

distributor:
  resources:
    requests:
      cpu: 166m
      memory: 147Mi
    limits:
      memory: 147Mi

querier:
  resources:
    requests:
      cpu: 16m
      memory: 53Mi
    limits:
      memory: 53Mi

query_scheduler:
  resources:
    requests:
      cpu: 10m
      memory: 31Mi
    limits:
      memory: 31Mi

query_frontend:
  resources:
    requests:
      cpu: 10m
      memory: 39Mi
    limits:
      memory: 39Mi

store_gateway:
  zoneAwareReplication:
    enabled: false
  persistentVolume:
    storageClass: longhorn-no-replica
    size: 2Gi
  resources:
    requests:
      cpu: 14m
      memory: 38Mi
    limits:
      memory: 38Mi

metaMonitoring:
  dashboards:
    enabled: true
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true
    mimirAlerts: true
    mimirRules: true

rollout_operator:
  resources:
    requests:
      cpu: 10m
      memory: 19Mi
    limits:
      memory: 19Mi

overrides_exporter:
  resources:
    requests:
      cpu: 10m
      memory: 31Mi
    limits:
      memory: 31Mi

nginx:
  nginxConfig:
    # increase the value to to fix `context deadline exceeded` / `server returned HTTP status 502 Bad Gateway`
    # https://github.com/grafana/mimir/issues/8293
    serverSnippet: |
      client_max_body_size 10m;
  resources:
    requests:
      cpu: 10m
      memory: 20Mi
    limits:
      memory: 20Mi

Thank you very much

Answered by mrclrchtr

Aug 7, 2024

Ok.. it was OOMKilled.. my fault.

View full answer

mrclrchtr · 2024-08-07T13:03:18Z

mrclrchtr
Aug 7, 2024
Author

Ok.. it was OOMKilled.. my fault.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mimir-distributor in restart loop #8926

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

mimir-distributor in restart loop #8926

mrclrchtr Aug 7, 2024

Replies: 1 comment

mrclrchtr Aug 7, 2024 Author

mrclrchtr
Aug 7, 2024

mrclrchtr
Aug 7, 2024
Author