Metric request error - Dashboard with AWS Cloudwatch as Datasource

[luispuente]

Hi!

We are using grafana as our main observability platform to monitor AWS resources in 2 accounts, using CloudWatch as our data source.

Right now we are working with 2 distinct grafana instances, each allocated in each AWS account via AWS Managed Grafana.

These accounts and grafana instances are labeled as Dev & Prod, so I’ll be using that as short way to refer to both accounts and grafana instances hosted in the respective accounts.

Now, as one does, we first created a bunch of dashboards using templates in Dev, and they all work fine, no issues there. Most of the templates were from monitoring artist and the grafana github.

But, when moving into replicating these dashboards (from the exact same templates) in Prod, some of them are not working, showing some metric request errors and not retrieving anything when creating variables. The biggest one being for Elastic Load Balancer.

The thing is, this template works perfectly fine in Dev, it retrieves and shows all the data with no issue. We’ve checked our IAM permissions and roles, we’ve checked cloudwatch itself and it all seems fine.

Now I’ll attach some images to paint the picture of what we are seeing. Since I can’t embed more than 5 pictures, I’ll leave a link to what we are seeing in each grafana instance for the ELB Template.

Dev Grafana Instance:https://files.fm/u/uc5vfcdw4

Prod Grafana Instance:https://files.fm/u/ep9qt9yxn

Would really appreciate the help. If you need some more detailed info please let me know.

Thanks in advance!

L.

[jangaraj]

Please show your PRD/DEV datasource config.

[luispuente]

DEV:

PROD:

PROD error from the 1st panel of ELB template

status:400
statusText:"Bad Request"
data:Object
results:Object
config:Object
method:"POST"
url:"api/ds/query"
data:Object
headers:Object
retry:0
hideFromInspector:false
message:"metric request error: \"InvalidParameter: 2 validation error(s) found.\\n- minimum field size of 1, GetMetricDataInput.MetricDataQueries[0].MetricStat.Metric.Dimensions[0].Value.\\n- minimum field size of 1, GetMetricDataInput.MetricDataQueries[1].MetricStat.Metric.Dimensions[0].Value.\\n\""

I don’t know if having different authentication methods affects in some way, because in PROD the metrics for other services are working fine, that includes EC2, RDS, Billing, cloudwatch logs, amongst others.

Also, the name spaces that are lacking are not needed thus far in PROD, so we haven’t add them yet.

We are only having issues with the whole ELB dashboard.

[jangaraj]

Advance AWS users use different regions for PROD and DEV env. Do you really keep PROD and DEV env in eu-west-1 region?

[luispuente]

Yes, we are. And it’s not up to me to decide that.

So far we are using completely different root accounts in the same region, so the different accounts are not related in any way. We are also using cloudwatch cross-account observability and it seems to be working fine in the S3 dashboard.

I don’t know how that is related to the issue we are having.

[jangaraj]

Dunno It is AMG, so ask your AWS support.