Metric Gap for Azure VM when using alloy

Grafana Alloy
1

I have 0 experience using grafana and its agent.
The goal is to read matrices from Azure monitor and push it to Grafana cloud using alloy.
I have configured for several resource types all looks good under grafana metrics dashboard. But only for the resource type “Microsoft.Compute/virtualMachines” CPU and MEM metrics has gap in it randomly. What can be the issue, the same resource has proper value when seen under azure portal. Below us my configuration
And I dont want to use Azure Monitor Data source. Using alloy is the requirement.

prometheus.exporter.azure "virtual_machines" {
    subscriptions = ["SUBSCRIPTIONHERE"]
    resource_type = "Microsoft.Compute/virtualMachines"
    metrics       = [
           "Available Memory Percentage",
           "Percentage CPU",
         ]
        metric_aggregations = [
        "Average",
        "Maximum",
        "Minimum",
        ]
    regions       = ["westeurope"]
}

prometheus.scrape "integrations_azure_exporter_virtual_machines" {
    targets    = prometheus.exporter.azure.virtual_machines.targets
    forward_to = [prometheus.remote_write.metrics_hosted_prometheus.receiver]
    scrape_interval = "60s"
    job_name   = "integrations/azure_exporter_virtual_machines"
}

image
image786×227 7.97 KB

What am I missing here. Thanks

2

Try:

  1. Use a Windows view and narrow your metrics to one series and one of the gap, and see if you may be getting metrics that are outside of interval window
  2. Change your dashboard interval to 2 minutes and see if that helps

With scrape interval of 60s, you could run into scenarios like this:

metric1 - 00:55 (minute 0, 55 sec)
metric2 - 02:05 (minute 2, 05 sec

The two metrics are 70 seconds apart, but if your graph interval is 1 minute you’ll see a gap on minute 1. #1 should tell you if this is the case.

If you are indeed missing metrics, I’d check Alloy logs (maybe enable debug) and see if it says anything interesting.

3

Thank you for the input.

I’ve tried the suggestions you provided. However, enabling debug mode doesn’t give me much information about the data gaps.

image
image2552×938 222 KB

As you can see in the screenshot, some resources are consistently sending data every minute, while others have gaps of more than 10 minutes. Is there anything I can check in Azure? It’s worth noting that I am receiving data from other resources of the same type.

Also mentioning that I don’t have access to use Alloy UI as its a restricted enviornment.

Thanks again for your support.

4

You could try to use another agent to verify Alloy. Perhaps node-exporter, have both agents running and have both send metrics to your backend and see what happens.

5

Thank you for your inputs. This issue was fixed with a PR in alloy repo.

Now the Alloy can query Azure with interval and timespan independently instead of interval = timespan