We are using grafana as our main observability platform to monitor AWS resources in 2 accounts, using CloudWatch as our data source.
Right now we are working with 2 distinct grafana instances, each allocated in each AWS account via AWS Managed Grafana.
These accounts and grafana instances are labeled as Dev & Prod, so I’ll be using that as short way to refer to both accounts and grafana instances hosted in the respective accounts.
Now, as one does, we first created a bunch of dashboards using templates in Dev, and they all work fine, no issues there. Most of the templates were from monitoring artist and the grafana github.
But, when moving into replicating these dashboards (from the exact same templates) in Prod, some of them are not working, showing some metric request errors and not retrieving anything when creating variables. The biggest one being for Elastic Load Balancer.
The thing is, this template works perfectly fine in Dev, it retrieves and shows all the data with no issue. We’ve checked our IAM permissions and roles, we’ve checked cloudwatch itself and it all seems fine.
Now I’ll attach some images to paint the picture of what we are seeing. Since I can’t embed more than 5 pictures, I’ll leave a link to what we are seeing in each grafana instance for the ELB Template.
Dev Grafana Instance:https://files.fm/u/uc5vfcdw4
Prod Grafana Instance:https://files.fm/u/ep9qt9yxn
Would really appreciate the help. If you need some more detailed info please let me know.
Thanks in advance!