Azure Monitor Queries are much slower than Azure Portal

Grafana Azure Monitor
1

I’ve noticed that when I run a query in Grafana using the built in grafana-azure-monitor-datasource plugin, a query takes will take about 5 times longer than if I copy the exact same query out of Grafana (using Inspect > Query) and into the Azure Portal (Web UI) targeting my Log Analytics Workspace.

This is true for both simple/narrow and complex/big queries. It’s particularly a problem with big queries because they time out in Grafana (but run fine in Azure Portal).

Is this the expected & normal behavior or does it indicate some configuration problem on our end? In other words, does Azure apply some deprioritization or rate limiting to external queries that they don’t to things running from within the Portal?

  • What Grafana version and what operating system are you using?

  • What are you trying to achieve?
    Trying to make Azure queries run at a similar speed as they do in the Azure Portal.

  • How are you trying to achieve it?
    Comparing the Total request time for the same query between Grafana and Azure Portal

  • What happened?
    Grafana is about 5x slower than Azure Portal across many tested queries.

  • What did you expect to happen?
    I would hope for similar response times from Azure as they provide in their WebUI

  • Can you copy/paste the configuration(s) that you are having problems with?
    N/A

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No errors other than timeouts if our query takes too long from Grafana.

  • Did you follow any online instructions? If so, what is the URL?
    N/A

2

Also, to be clear, there’s not some kind of query caching in play here as far as I can tell: Portal is just faster even if I don’t run the query in Grafana first. Same when running the same query multiple times.

3

Update: we were able to fix this to be in parity with the Portal by using a different Azure resource URI that pointed directly at the Log Analytics Workspace rather than a larger scope.

Eg we replaced:

/subscriptions/1234-a4bb-460b-8139-25845132135

With

/subscriptions/1af234-822a-4b67-ab70-511155/resourceGroups/Monitoring-NonProd-East/providers/Microsoft.OperationalInsights/workspaces/Monitoring-NonProd-East

4

My service principal has the role ‘monitoring reader’ on the log analytics resource (which is monitoring a k8s cluster), but I still have 5-10 times slower querying time than if I do it in the Azure Portal. Any ideas? I haven’t found any caching capabilities in Grafana when querying Azure Monitor. I am running a self-hosted Grafana in our k8s cluster.