Metric Gap for Azure VM when using alloy

Grafana Alloy
1

I have 0 experience using grafana and its agent.
The goal is to read matrices from Azure monitor and push it to Grafana cloud using alloy.
I have configured for several resource types all looks good under grafana metrics dashboard. But only for the resource type “Microsoft.Compute/virtualMachines” CPU and MEM metrics has gap in it randomly. What can be the issue, the same resource has proper value when seen under azure portal. Below us my configuration
And I dont want to use Azure Monitor Data source. Using alloy is the requirement.

prometheus.exporter.azure "virtual_machines" {
    subscriptions = ["SUBSCRIPTIONHERE"]
    resource_type = "Microsoft.Compute/virtualMachines"
    metrics       = [
           "Available Memory Percentage",
           "Percentage CPU",
         ]
        metric_aggregations = [
        "Average",
        "Maximum",
        "Minimum",
        ]
    regions       = ["westeurope"]
}

prometheus.scrape "integrations_azure_exporter_virtual_machines" {
    targets    = prometheus.exporter.azure.virtual_machines.targets
    forward_to = [prometheus.remote_write.metrics_hosted_prometheus.receiver]
    scrape_interval = "60s"
    job_name   = "integrations/azure_exporter_virtual_machines"
}

image
image786×227 7.97 KB

What am I missing here. Thanks

2

Try:

  1. Use a Windows view and narrow your metrics to one series and one of the gap, and see if you may be getting metrics that are outside of interval window
  2. Change your dashboard interval to 2 minutes and see if that helps

With scrape interval of 60s, you could run into scenarios like this:

metric1 - 00:55 (minute 0, 55 sec)
metric2 - 02:05 (minute 2, 05 sec

The two metrics are 70 seconds apart, but if your graph interval is 1 minute you’ll see a gap on minute 1. #1 should tell you if this is the case.

If you are indeed missing metrics, I’d check Alloy logs (maybe enable debug) and see if it says anything interesting.