Ship metrics to Azure Monitor Workspace (Managed Prometheus)

Grafana Alloy
1

Hi all,

Completely green with Alloy so bear with me please.

From an on-prem cluster I’m trying to ship Prometheus metrics to a Managed Prometheus instance we have in Azure, following the documentation here and here.

This brought me to the configuration below:

prometheus.remote_write "staging" {
  endpoint {
    name = "amw-prometheus"
    url = "https://amw-prometheus-xxxx.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-89xxxx650b374xxxx1exxxx7xxxxac4f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24"
    proxy_from_environment = true
    azuread {
      cloud = "AzurePublic"
      managed_identity {
        client_id = "8exxxxb6-xxxx-xxxx-xxxx-6d75xxxxa4f7"
      }
    }
  }    
}

prometheus.operator.servicemonitors "services" {
  namespaces = ["cattle-monitoring-system"]
  forward_to = [prometheus.remote_write.staging.receiver]
}

This results to this in the Alloy log:

alloy ts=2024-11-25T15:24:15.027912477Z level=warn msg="Failed to send batch, retrying" component_path=/ component_id=prometheus.remote_write.staging subcomponent=rw remote_name=amw-sbx-mgmt-prometheus url="https://amw-prometheus-xxxx.westeurope-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-89xxxx650b374xxxx1exxxx7xxxxac4f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24\": Failed to get access token: ManagedIdentityCredential: ManagedIdentityCredential: context deadline exceeded"

I believe this azuread block only works when Alloy is on the AKS cluster itself (to be able to use the correct client_id and get the correct privileges).

So my question is, which config block can I use to authenticate from an on-prem Kubernetes cluster to this managed Prometheus instance in Azure? I think none of the documented blocks (basic_auth, authorization, oauth2) will work? But I hope I’m wrong :slight_smile: