400 Bad Request behind nginx ingress controller

Grafana Configuration
, ,
1

Grafana: 8.1.1
Helm: v3.7.0

I came across topic entitled “how-to-configure-grafana-behind-reverse-proxy-ingress-nginx-controller” while searching for help. I too find myself in a similar boat when trying to access https://MyDomain/ through an Nginx Ingress Controller in a K8S cluster. Difference being I am recieving 400 Bad Request instead of 502 Bad Gateway.

I updated my grafana.ini with the domain name as described in this how-to tutorials:run-grafana-behind-a-proxy, but with no change in behavior. .

Here is content from grafana.ini:

[server]
cert_file = /etc/secrets/tls.crt
cert_key = /etc/secrets/tls.key
domain = grafana.ccp.local
protocol = https
root_url = https://grafana.ccp.local:3000/
router_logging = true

I also tried using ‘%(protocol)s://%(domain)s:%(http_port)s/’ as the root_url, but still no help.

The above link refers to configuration in the nginx ingress controller, but the implication is that it is “to proxy Grafana Live WebSocket connections”. As far as I know I am not utilizing Grafana Live Websocket connections, so is the nginx configuration still needed?

Since I am getting a 400 Bad Request and not 502 Bad Gateway, I was hoping there may be a different solution.

Please, any help would be greatly appreciated.

2

can you please provide a link to the tutorial that you used? What is an official tutorial from Grafana? :+1:

3

I would have put it in the first post, but being a new member, it restricted the number of links I could include.

I do not know of an “official” tutorial.

4

so have you installed and configured Nginx inside your kubernetes cluster? You will need to set that up as a reverse proxy first

5

I am running ingress-nginx-controller for the reverse proxy in a minikube k8s cluster. I have configured the domain name in the Grafana configuration as described in the above tutorial, though I am not using the sub path.

Blockquotespec:
rules:

  • host: grafana dot local dot com
    http:
    paths:
    • backend:
      service:
      name: grafana
      port:
      number: 80
      path: /
      pathType: Prefix

Blockquote

The tutorial has further instruction for configuring NGINX which implies modification of an existing configuration. The tutorial does not provide guidance on how to fit it in an existing configuration. I have tried creating a custom template that is applied via a configMap. My trying to fit in the suggested configuration content resulted in either duplicate configuration lines (errors) or configuration lines that could be potentially included in all servers behind the proxy (its a template). Now I am trying to use snippets via annotations in the ingress resource. For example:

Blockquote

kind: Ingress
metadata:
annotations:
kubernetesdotio/ingressdotclass: nginx
metadothelmdotsh/release-name: grafana
metadothelmdotsh/release-namespace: default
nginxdotorg/location-snippets: |
add_header my-test-header test-value;
nginxdotorg/server-snippets: |
location /api/live {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection “Upgrade”
proxy_set_header Host $http_host;
proxy_pass http://grafana.local.com:3000/;
{

Blockquote

However, I do not see this reflected in nginx.conf when the services are restarted. This is at present a mystery.

Note: my use of the literal “dot” is to satisfy my restrictions on using more than two links.

6

ah ok. IMO, running this setup on minikube + localhost could be the cause here. Have you tried setting this up on using a kubernetes management service from a cloud provider? Or even a VM running minikube that you could try and access externally? GKE and DigitalOcean K8s are my two go-tos for throwaway clusters. Also, here are some resources I found on using the nginx-ingress with minikube. Maybe they help w your troubleshooting?