Problems getting metrics from Azure in Grafana

Hi there,

within Azure there is the service principal with the roles “Monitoring Reader” and “Reader” assigned.

Therefor the resource group “elkprodresourcegroup” within the subscription “IT Commerce PROD” is accessable.

This resource group has 3017 records, at least 6 of type “SQL database” and 5 of type “SQL server” which are of interest right now.

Using Grafana 7.5.5 or Grafana 8.3.1 together with Azure Monitor as data source leads to the same error behavior.

So I choose:

  • Service: “Metrics”
  • Subscription: “IT Commerce PROD”
  • Resource group: “elkprodresourcegroup”
  • Namespace (only Grafana 7.5.5): “Microsoft.Cache/Redis”
  • Resource type (only Grafana 8.3.1): “Azure Cache for Redis”

From my perspective it seems entirely wrong that Namespace or Resource type only shows “Azure Cache for Redis”.

I’d expect that more types are shown here, like all the types of the 3017 records, for example “Storage account”, “SQL database”, “SQL server”, “Kubernetes service”, “Key vault”…

image2021-12-9_11-48-19

Using commandline after logging in with that service principal I am able to get some results for:
az monitor metrics list-namespaces --resource /subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/providers/Microsoft.Sql/servers/elkproddbsrvprocessdatadeltaprod -o json

This prooves that the service principal has enough rights to access this.

image2021-12-9_13-22-53

I just want within Grafana to be able to work on the metrics of the SQL database “elkproddbprocessdatadeltaprod”.

The web ui shows metrics like e.g. “CPU percentage”:

Best regards,
Marco Schmitz

Hi Marco, these SQL DB metrics should indeed be available. Can you share the version of the Azure Monitor plugin in use with a screenshot of your query and query inspector?

1 Like

Hi kaprince,
thanks for responding and trying to help.

Regarding the plugin versions for Grafana 8.3.1 and Grafana 7.5.5 (same plugin versions for both Grafana distributions) using grafana-cli plugins list-remote:

  • id: grafana-azure-data-explorer-datasource version: 3.6.1
  • id: grafana-azure-monitor-datasource version: 0.3.1

Using http://********/plugins/grafana-azure-monitor-datasource?page=overview I get 0.3.0 (same plugin versions for both Grafana distributions):

The query is:
{"queryType":"Azure Monitor","azureMonitor":{"timeGrain":"auto","resourceGroup":"elkprodresourcegroup","dimensionFilters":[]},"subscription":"2ee14c5f-****-****-****-************"}

And the screenshot:

All the best,
Marco Schmitz

Hello there,

so it took me some time to get used to the Azure REST API.

Right now I am successfully requesting “SQL database (not working in Grafana)” as well as “Azure Cache for Redis (the only one working in Grafana)” from Azure REST API.

But it only works if I use different api-version parameters.

For example for “SQL database (not working in Grafana)” I need “api-version=2021-08-01-preview”.
https://management.azure.com/subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/providers/Microsoft.Sql/servers/elkproddbsrvprocessdatadeltaprod/databases/elkproddbprocessdatadeltaprod?api-version=2021-08-01-preview

But for “Azure Cache for Redis (the only one working in Grafana)” I need “api-version=2021-06-01”.
https://management.azure.com/subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/providers/Microsoft.Cache/Redis/elkprodrediscachedeltaprod?api-version=2021-06-01

Using some wrong api-version like “api-version=2021-04-01” I get an error with an response containing JSON naming the api-versions I have to use.

So I started to think if Grafana is using the wrong api-version, one that fits for “Azure Cache for Redis (the only one working in Grafana)” but not for “SQL database (not working in Grafana)”.

But on https://github.com/grafana/grafana/blob/main/public/app/plugins/datasource/grafana-azure-monitor-datasource/azure_monitor/azure_monitor_datasource.ts#L25 I found:

apiVersion = '2018-01-01';
apiPreviewVersion = '2017-12-01-preview';

Using these version I get the error with that response containing JSON for both resources.

So right now I am confused how Grafana solves this?

Can I configure the api-version within Azure for some specific resources like the SQL database in order to gain more compatibility ?

All the best,
Marco Schmitz

Hi there,

in parallel I am in contact with Azure support. And I have to correct myself, because the apiVersion was a dead end. To be honest I was using the wrong URLs, so here comes the update.

The structure for a GET request regarding the metrics is:
https://management.azure.com/{{resourceUri}}/providers/Microsoft.Insights/metrics?api-version=2018-01-01

SQL database (not working in Grafana):
https://management.azure.com/subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/providers/Microsoft.Sql/servers/elkproddbsrvprocessdatadeltaprod/databases/elkproddbprocessdatadeltaprod/providers/Microsoft.Insights/metrics?api-version=2018-01-01

Azure Cache for Redis (the only one working in Grafana):
https://management.azure.com/subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/providers/Microsoft.Cache/Redis/elkprodrediscachedeltaprod/providers/Microsoft.Insights/metrics?api-version=2018-01-01

Using Postman I was able to receive metrics from both resources.

And as you can see, the apiVersion is the same as it is used in the Grafana sourcecode.

But something is still causing these problems so that I cannot get my metrics…

Best regards,
Marco Schmitz

Hi all,

I tried to find out what request is fired when choosing the resource group within Grafana.

So this is my request (within Firefox using Grafana v8.3.1 locally):
GET http://grafana.staged-by-discourse.com/api/datasources/1/resources/azuremonitor/subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/resources?api-version=2018-01-01

Using Postman leads to the same result:
https://management.azure.com/subscriptions/2ee14c5f-****-****-****-************/resourceGroups/elkprodresourcegroup/resources?api-version=2018-01-01

The response contains 999 JSON Objects and a nextLink.

Since this resource group has 3017 records I think there is kind of a paging going on here.

Within the first 999 JSON Objects there is no “SQL”.

In fact I was able to use the nextLink URLs and there are 4 pages alltogether. “SQL” is to be found on the 4th page.

So I think the “bug” within Grafana is about not using this paging…