All of a sudden all exports from alloy are unauthenticated

Grafana Alloy
1

All of a sudden all requests from alloy are unauthenticated. No changes were made during the night.

Logs:

ts=2025-05-28T17:21:58.712095839Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs responded with HTTP Status Code 401" dropped_items=1
ts=2025-05-28T17:22:08.744884902Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs responded with HTTP Status Code 401" dropped_items=1
ts=2025-05-28T17:22:35.234700598Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics responded with HTTP Status Code 401" dropped_items=23
ts=2025-05-28T17:22:36.03677991Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics responded with HTTP Status Code 401" dropped_items=2
ts=2025-05-28T17:22:38.440836888Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics responded with HTTP Status Code 401" dropped_items=102
ts=2025-05-28T17:22:38.860125571Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs responded with HTTP Status Code 401" dropped_items=1
ts=2025-05-28T17:22:40.263322185Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/logs responded with HTTP Status Code 401" dropped_items=3
ts=2025-05-28T17:23:35.233373937Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics responded with HTTP Status Code 401" dropped_items=23
ts=2025-05-28T17:23:36.037715951Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics responded with HTTP Status Code 401" dropped_items=2
ts=2025-05-28T17:23:38.447215743Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.grafana_cloud error="not retryable error: Permanent error: rpc error: code = Unauthenticated desc = error exporting items, request to https://otlp-gateway-prod-eu-west-2.grafana.net/otlp/v1/metrics responded with HTTP Status Code 401" dropped_items=108

Started happening today at around 2am. Can see there has been no ingestion since then as well. API Tokens are still in place with no expiration, Tried replacing the API token with no effect.

image
image2468×702 55.7 KB

2

Feels like a grafana cloud issue perhaps? Have you tried reaching out to Grafana Cloud support?

3

@tonyswumac Indeed, I reached out to support. In the initial information gathering phase, hopefully they get to it soon. I’m on a free plan so that would delay support :sweat_smile:

4

Hello,
I think I am experiencing the same issue as you have.
Has there any progress been made on this?

5

I wasn’t patient enough. I deleted my stack and recreated it, that resolved the issue and I was able to restore service instantly.