403 Forbidden when importing only some dashboards

Grafana Configuration
1

I have Grafana running on EKS behind an nginx ingress controller. When I try to import the AWS ELB Application LoadBalancer Dashboard AWS ELB Application Load Balancer | Grafana Labs, the dashboard is fetched and I am asked to choose the the correct datasource, region and ALB name. The ALB names populate correctly, so the datasource is working fine. However, when I choose the ALB, all hell breaks loose and I get bombarded with a ton of 403s.

Screen Shot 2020-08-26 at 1.54.55 PM
Screen Shot 2020-08-26 at 1.54.55 PM801×418 204 KB

But when I remove the variables from the dashboard and hardcode the correct values in json, the dashboard works perfectly!

Screen Shot 2020-08-26 at 1.56.51 PM
Screen Shot 2020-08-26 at 1.56.51 PM1009×438 50.4 KB

Also the EFS Dashboard AWS EFS | Grafana Labs works right out of the box without any issues.

I do not see 403 errors on my application logs. I only see 400 errors on queries like the one below.

t=2020-08-26T18:54:34+0000 lvl=info msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=POST path=/api/tsdb/query status=400 remote_addr=redacted time_ms=695 size=1222 referer=“https://grafana.redacted.com/d/10p1yUHGz/aws-elb-application-load-balancer?orgId=1”

I was initially looking at nginx but how would other dashboard work just fine it was an nginx misconfig? Any help is appreciated! Thank you in advance!

Screen Shot 2020-08-26 at 2.05.57 PM
Screen Shot 2020-08-26 at 2.05.57 PM435×524 19.5 KB

2

/api/annotations query is failing (default Grafana annotations are part of some dashboards). You should to debug why and what returns 403 response.

3

Thank you @jangaraj , I was using an old version of the kiwigrid/k8s-sidecar container. I bumped is up to 0.1.178 and it fixed the issue. I am not sure what the root cause it however.

I also bumped up grafana to 7.1.1

4

Actually I am seeing the error again when saving the dashboard. I will post details shortly.

5

I think I found my problem, looks like a WAF blocking it.

6

I solved my issue. It was a WAF rule blocking all requests when it found a sensitive keyword in the name of the load balancer. I had a loadbalancer named random-soarsext-alb (that I was trying to choose in the dashboard and save) and the organization WAF was blocking it because there was a rule to block all requests that match the string “sex” :rofl: