Good morning,
Our mimir ingester is breaking all the pods when the error mentioned below occurs. This is very bad because we have 5 pods and the error happens in just one, all the others break and we are left with no metrics.
Could anyone help us?
Error:
"“invalid service state: Failed, expected: Running, failure: opening existing TSDBs: unable to open TSDB for user anonymous: failed to open TSDB: /data/tsdb/anonymous: /data/tsdb/anonymous/chunks_head/000477: invalid magic number 0"”
installation:
repository = “Grafana Community Kubernetes Helm Charts | helm-charts”
chart = “mimir-distributed”
version = “3.0.0”
GKE
Config:
limits:
max_global_series_per_user: 10000000
max_global_series_per_metric: 10000000
ingestion_rate: 2000000
ingestion_burst_size: 20000000
max_fetched_chunks_per_query: 0
query_sharding_max_sharded_queries: 512
max_query_parallelism: 100
ruler_max_rules_per_rule_group: 30
out_of_order_time_window: 5760m