I have Grafana running on EKS behind an nginx ingress controller. When I try to import the AWS ELB Application LoadBalancer Dashboard https://grafana.com/grafana/dashboards/650, the dashboard is fetched and I am asked to choose the the correct datasource, region and ALB name. The ALB names populate correctly, so the datasource is working fine. However, when I choose the ALB, all hell breaks loose and I get bombarded with a ton of 403s.
But when I remove the variables from the dashboard and hardcode the correct values in json, the dashboard works perfectly!
Also the EFS Dashboard https://grafana.com/grafana/dashboards/653 works right out of the box without any issues.
I do not see 403 errors on my application logs. I only see 400 errors on queries like the one below.
t=2020-08-26T18:54:34+0000 lvl=info msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=POST path=/api/tsdb/query status=400 remote_addr=redacted time_ms=695 size=1222 referer=“https://grafana.redacted.com/d/10p1yUHGz/aws-elb-application-load-balancer?orgId=1”
I was initially looking at nginx but how would other dashboard work just fine it was an nginx misconfig? Any help is appreciated! Thank you in advance!