# Error forwarding metrics with Alloy to Mimir **URL:** \ **Category:** Grafana Alloy\ **Created:** [October 26, 2024, 5:12pm UTC](https://community.grafana.com/t/error-forwarding-metrics-with-alloy-to-mimir/135136 "2024-10-26T17:12:54Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![mitchmurphy](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/mitchmurphy/32/47103_2.png) [@mitchmurphy](https://community.grafana.com/u/mitchmurphy)\ **Post date:** [October 26, 2024, 5:12pm UTC](https://community.grafana.com/t/error-forwarding-metrics-with-alloy-to-mimir/135136/1 "2024-10-26T17:12:54Z")
I am having issues forwarding metrics to Mimir using Alloy. My setup is as follows: - RKE2 v1.31 - Grafana LGTM 2.1.0 - Grafana Alloy 0.9.2 - kube-state-metrics - cilium v1.16.2 - kube-proxy replacement - ingress controller - l2 announcement - gateway API - clustermesh (type LoadBalancer) - OpenEBS - prometheus CRDs For the LGTM stack a few minor modificiations were made such as changing the dnsService for rke2. In Alloy I enabled varlog and dockercontainers mounts for collecting logs (loki is working fine), other than that I did not make any changes. As a new user I can not upload files, so here is the [gist](https://gist.github.com/mkm29/537e956669c38101d90e08033cf605f0) of my alloy config. These are the errors I am getting, not sure what to make of these: ```auto ts=2024-10-26T17:08:07.813508529Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.services scrape_pool=prometheus.scrape.services target=http://lgtm-distributed-tempo-query-frontend.monitoring.svc:9095/metrics err="Get \"http://lgtm-distributed-tempo-query-frontend.monitoring.svc:9095/metrics\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x00\\x00\\f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x00\\x00@\\x00\\x00\\x03\\x00\\x00\\x00d\"" ts=2024-10-26T17:08:07.841161424Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.pods scrape_pool=prometheus.scrape.pods target=http://192.168.1.120:80/metrics err="Get \"http://192.168.1.120:80/metrics\": dial tcp 192.168.1.120:80: connect: connection refused" ts=2024-10-26T17:08:07.880070607Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.services scrape_pool=prometheus.scrape.services target=http://lgtm-distributed-mimir-compactor.monitoring.svc:9095/metrics err="Get \"http://lgtm-distributed-mimir-compactor.monitoring.svc:9095/metrics\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x00\\x00\\f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x00\\x00@\\x00\\x00\\x03\\x00\\x00\\x00d\"" ts=2024-10-26T17:08:08.187541573Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.services scrape_pool=prometheus.scrape.services target=http://lgtm-distributed-tempo-distributor.monitoring.svc:9095/metrics err="Get \"http://lgtm-distributed-tempo-distributor.monitoring.svc:9095/metrics\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x00\\x00\\f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x00\\x00@\\x00\\x00\\x03\\x00\\x00\\x00d\"" ts=2024-10-26T17:08:08.340802804Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.services scrape_pool=prometheus.scrape.services target=http://lgtm-distributed-loki-querier-headless.monitoring.svc:9095/metrics err="Get \"http://lgtm-distributed-loki-querier-headless.monitoring.svc:9095/metrics\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x00\\x00\\f\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x00\\x00@\\x00\\x00\\x03\\x00\\x00\\x00d\"" ts=2024-10-26T17:08:08.363253285Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.pods scrape_pool=prometheus.scrape.pods target=http://10.0.0.144:80/metrics err="Get \"http://10.0.0.144:80/metrics\": dial tcp 10.0.0.144:80: connect: connection refused" ts=2024-10-26T17:08:08.451460337Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.pods scrape_pool=prometheus.scrape.pods target=http://10.0.0.144:80/metrics err="Get \"http://10.0.0.144:80/metrics\": dial tcp 10.0.0.144:80: connect: connection refused" ts=2024-10-26T17:08:08.699649589Z level=debug msg="Scrape failed" component_path=/ component_id=prometheus.scrape.pods scrape_pool=prometheus.scrape.pods target=http://192.168.1.121:80/metrics err="Get \"http://192.168.1.121:80/metrics\": dial tcp 192.168.1.121:80: connect: connection refused" ``` ---
**Author:** ![yosiasz](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/yosiasz/32/45847_2.png) [@yosiasz](https://community.grafana.com/u/yosiasz)\ **Post date:** [October 26, 2024, 8:39pm UTC](https://community.grafana.com/t/error-forwarding-metrics-with-alloy-to-mimir/135136/2 "2024-10-26T20:39:29Z")
Is it writing to Loki at all? Permission issues with minor? ---
**Author:** ![mitchmurphy](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/mitchmurphy/32/47103_2.png) [@mitchmurphy](https://community.grafana.com/u/mitchmurphy)\ **Post date:** [October 26, 2024, 10:35pm UTC](https://community.grafana.com/t/error-forwarding-metrics-with-alloy-to-mimir/135136/3 "2024-10-26T22:35:32Z")
I do see Loki logs when I explore the Loki datasource in Grafana, and I am seeing a small number of chunks in Minio (the default LGTM helm chart sets the console keys to grafana-mimir/supersecret). On this installation I did not create separate keys, I did this in another installation where I am using Ceph/Rook. ---
**Author:** ![mitchmurphy](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/mitchmurphy/32/47103_2.png) [@mitchmurphy](https://community.grafana.com/u/mitchmurphy)\ **Post date:** [October 27, 2024, 1:18am UTC](https://community.grafana.com/t/error-forwarding-metrics-with-alloy-to-mimir/135136/4 "2024-10-27T01:18:03Z")
the mimir distributor pod rejects every sample with `err-mimir-sample-out-of-order` error, however the nodes all have their time synced with crony and I added this to my values so not sure why this is happening… ```auto structuredConfig: limits: max_global_series_per_user: 20000000 ingestion_rate: 200000 ingestion_burst_size: 1000000 out_of_order_time_window: 5m ```