Prometheus remote-write: err-mimir-sample-out-of-order rejections

Grafana Alloy
, , ,
1

I’m getting repeated err-mimir-sample-out-of-order rejections when pushing Prometheus metrics via remote-write to Grafana Mimir. Out-of-order sample errors are preventing some metrics from being ingested.

ts=2025-05-22T23:28:55.00276654Z caller=push.go:236 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=400 err="send data to ingesters: failed pushing to ingester mimir-ingester-zone-c-0: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-05-22T23:28:49.92Z and is from series thanos_objstore_bucket_operation_transferred_bytes_bucket{cluster=\"mimir\", component=\"usage-stats\", container=\"store-gateway\", endpoint=\"http-metrics\", instance=\"10.42.2.67:8080\", job=\"mimir/store-gateway\", le=\"131072.0\", namespace=\"mimir\", operation=\"upload\", pod=\"mimir-store-gateway-zone-c-0\", service=\"mimir-store-gateway-zone-c\"} (sampled 1/10)" insight=true
ts=2025-05-22T23:29:52.077337864Z caller=push.go:236 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=400 err="send data to ingesters: failed pushing to ingester mimir-ingester-zone-a-0: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-05-22T23:29:49.92Z and is from series activity_tracker_failed_total{cluster=\"mimir\", container=\"store-gateway\", endpoint=\"http-metrics\", instance=\"10.42.2.67:8080\", job=\"mimir/store-gateway\", namespace=\"mimir\", pod=\"mimir-store-gateway-zone-c-0\", reason=\"empty_activity\", service=\"mimir-store-gateway-zone-c\"} (sampled 1/10)" insight=true

These two logs are being repeated.

I’m running both Mimir (helm chart https://grafana.github.io/helm-charts/mimir-distributed chart version 5.7.0, app version 2.16.0) and Alloy (helm chart https://grafana.github.io/helm-charts/alloy chart version 1.0.3, app version 1.8.3) on K3s version (v1.32.1+k3s1) with a self hosted MinIO as the backend for storage.

I’m trying to scrape mimirs servicemonitors using alloy with this config:

  configMap:
    create: true
    content: |-
      prometheus.remote_write "mimir" {
        endpoint {
          url     = "http://mimir-distributor.mimir.svc.cluster.local:8080/api/v1/push"
          
          headers = {
            "X-Scope-OrgID" = "anonymous",
          }
        }
      }

      prometheus.operator.servicemonitors "mimir_smon" {
        forward_to = [prometheus.remote_write.mimir.receiver]

        namespaces = ["mimir"]
      }

I’m enabling the serviceMonitors in the mimir values.yaml with

metaMonitoring:
  dashboards:
    enabled: true
    namespace: grafana
    annotations:
      k8s-sidecar-target-directory: /tmp/dashboards/Mimir
    labels:
      grafana_dashboard: "1"

  # ServiceMonitor configuration for monitoring Kubernetes Services with Prometheus Operator and/or Grafana Agent
  serviceMonitor:
    enabled: true
    clusterLabel: ""
    namespace: null
    namespaceSelector: null
    annotations: {}
    labels: {}

Ive tried adding instance labels and tried scraping the pods themselves directly with no luck. I can provide more logs or more config if needed.

Thank you.