OTEL metrics' resources attributes gets dropped when exported to Grafana cloud

Grafana Cloud
1

I use OTEL http exporter to export metrics data to Grafana Cloud.
I noticed that resource attributes are ignored when ingesting metrics data into prometheus by Grafana cloud.
I wonder if it is possible to include resource attribute in time series label?

2

That’s because Prometheus doesn’t have native support of resource attributes. But it will have in the future.

Workaround: use prometheusremotewrite exporter with

resource_to_telemetry_conversion: 
 enabled: true
3

Hi team,

We are block by this issue.
Is there any workaround?
Do you know when will Prometheus solve this problem natively?

Other Observability backend like Honeycomb, Dyantrace, Signoz and New Relic are already supporting it.

Thank you for the help and support :slight_smile:

Antonio from Cisco ThousandEyes.

4

Grafana Mimir is long-term storage for Prometheus, so I believe Prometheus must sort it first - there is no point to have vendor specific implementation for this. Try to find some Github issue for that.

I would say workaround is simple - just use some processor and “move” all resource attributes to attributes before exporting to Mimir/Grafana Cloud.

5

We’re also running into this problem, using the OTEL .NET library, and sending logs, traces and metrics to the OTEL ingestion endpoint for Grafana Cloud.

Resource attributes are not being preserved for metrics, which means we can’t distinguish which deployment environment the metrics are coming from.

We’re not using a collector, and it’s also likely not necessary / an option for the environment, and as we’re just going through the OTEL endpoint it’s not clear how we could resolve this.

6

I faced the same issue recently, but in my case I used grafana-alloy (previously agent) and found out how to solve it there:

otelcol.exporter.prometheus "to_prometheus" {
  forward_to = [prometheus.remote_write.default.receiver]
  resource_to_telemetry_conversion = true
}

the resource_to_telemetry_conversion is the key

7

I haven’t tested it myself as we’re using the Prometheus remote write method, but it also seems this may be fixed by enabling the experimental Mimir config flag -distributor.promote-otel-resource-attributes on your Grafana Cloud instance.

I only found out about it by chancing upon the discussion at Idea: Support Prometheus' `keep_identifying_resource_attributes` on Mimir's OTLP endpoint · Issue #10855 · grafana/mimir · GitHub .