My graphs only work if i browse with firefox

Grafana Configuration
1

Howdy,
I installed grafana on a vps within a docker container and successfully added an elasticsearch datasource and proceeded to create some pretty graphs. All good.
Then one colleague tells me it just does not work. At all. Turns out he is using chromium and I am using firefox. So, since firefox is all good, I start chromium and check the console for errors and behold, i get forbidden request errors and I have no idea why and how. (same problem on edge)

Failed to load resource: the server responded with a status of 403 (Forbidden)
metrics_panel_ctrl.ts:171 Panel data error: Object

My data source is configured as is:
URL : http://my.local.ip.address:9200 (which is an actual ssh port forward to my elasticsearch server)
Acess: Proxy

No credentials, no auth, no tls verification, es is only locally accessible and I wanted to keep it simple since i’m just getting used to graphing it.

From edge the error looks a bit different but it’s basically the same thing:

HTTP403: FORBIDDEN - The server understood the request, but is refusing to fulfill it.
(XHR)POST - https://<MY_GRAFANA_INSTANCE>/api/datasources/proxy/1/_msearch

Needless to say, From firefox the console is mostly empty except for this warning

window.controllers/Controllers is deprecated. Do not use it for UA detection.

Anybody know why and how to fix this problem ?

thanks ^^

2

Hi,

Please check the grafana server log for more information and see if you can see any related errors there when you get your described problems.

If you had access direct I would have guessed this was a CORS problem between Grafana and Elasticsearch, but since you’re using proxy mode it can’t be that.

Marcus

Marcus

3

In the logs all i get is this (logging level set to debug)
per each panel on my dashboard i see this :

t=2018-03-07T16:14:47+0200 lvl=info msg=“Request Completed” logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/1/_msearch status=403 remote_addr=10.1.1.100 time_ms=154 size=0 referer=“http://10.1.1.102:3000/d/jEVLWZgmk/dashboard?orgId=1”

so i get a 403… firefox works just fine . chrome, edge, internet explorer, opera all dont work

4

Updated grafana to latest. Still happening. Any help :frowning: ?

5

What version of Firefox?

6

firefox 59.0.1 (64-bit)
waterfox 56.0.4.1 (64-bit)
All foxes work. problem is with chrome/chromium/edge/ie/opera

7

Okay. Please include details of one of your panels/queries that fails in chrome. Use the query inspector and include full request/response here.

So can you access your elasticsearch instance directly in the browser using chrome for example? If not, that’s where your problem is.

Marcus

8

Chrome:

{
“data”: “”,
“status”: 403,
“statusText”: “Forbidden”,
“xhrStatus”: “complete”,
“request”: {
“method”: “POST”,
“url”: “api/datasources/proxy/1/_msearch”,
“data”: “{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"logstash-","max_concurrent_shard_requests":256}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1522060448384","lte":"1522060748384","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":""}}]}},"aggs":{"4":{"terms":{"field":"host.keyword","size":10,"order":{"_count":"desc"},"min_doc_count":1},"aggs":{"2":{"date_histogram":{"interval":"5m","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":"1522060448384","max":"1522060748384"},"format":"epoch_millis"},"aggs":{}}}}}}\n”
}
}

Same on edge/ie etc.
In Firefox:

{
“xhrStatus”: “complete”,
“request”: {
“method”: “POST”,
“url”: “api/datasources/proxy/1/_msearch”,
“data”: “{"search_type":"query_then_fetch","ignore_unavailable":true,"index":"logstash-","max_concurrent_shard_requests":256}\n{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"gte":"1522060545290","lte":"1522060845290","format":"epoch_millis"}}},{"query_string":{"analyze_wildcard":true,"query":""}}]}},"aggs":{"4":{"terms":{"field":"host.keyword","size":10,"order":{"_count":"desc"},"min_doc_count":1},"aggs":{"2":{"date_histogram":{"interval":"5m","field":"@timestamp","min_doc_count":0,"extended_bounds":{"min":"1522060545290","max":"1522060845290"},"format":"epoch_millis"},"aggs":{}}}}}}\n”
},
“response”: {
“responses”: [
{
“took”: 405,
“timed_out”: false,
“_shards”: {
“total”: 1329,
“successful”: 1329,
“skipped”: 1325,
“failed”: 0
},
“hits”: {
“total”: 210343,
“max_score”: 0,
“hits”:
},
“aggregations”: {
“4”: {
“doc_count_error_upper_bound”: 0,
“sum_other_doc_count”: 0,
“buckets”: [
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 58303
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 9528
}
]
},
“key”: “fr06”,
“doc_count”: 67831
},
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 41420
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 5369
}
]
},
“key”: “ro03”,
“doc_count”: 46789
},
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 32380
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 5229
}
]
},
“key”: “fr04”,
“doc_count”: 37609
},
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 23041
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 3774
}
]
},
“key”: “us02”,
“doc_count”: 26815
},
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 18194
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 3140
}
]
},
“key”: “fr03”,
“doc_count”: 21334
},
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 4666
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 687
}
]
},
“key”: “fr09”,
“doc_count”: 5353
},
{
“2”: {
“buckets”: [
{
“key_as_string”: “1522060500000”,
“key”: 1522060500000,
“doc_count”: 4037
},
{
“key_as_string”: “1522060800000”,
“key”: 1522060800000,
“doc_count”: 575
}
]
},
“key”: “fr08”,
“doc_count”: 4612
}
]
}
},
“status”: 200
}
]
}
}

Trying to access es directly from browser ? No of course not. it’s 127.0.0.1:9200 bound serverside.
How come firefox works then ?
Acessing my site https://my.grafana.public.url:9200 gives expected error. Why would i leave es exposed to public anyway ?

9

This is all a bit not so clear to me, sorry. But

  • Are you running elasticsearch and grafana on different servers, what’s their ips (or similar)?
  • What does your elasticsearch configuration look like?
  • How have you configured elasticsearch datasource in Grafana, please include screenshot?
  • Do you run any reverse proxy like nginx or apache?

Marcus

10

es and grafana run have their own boxes
es has 9 nodes, 1 master. up for more than a year, kibana works, not an ES issue.
grafana is on another machine.
grafana has ssh port forwarding so that it perceives es as listening on it’s loopback interface 127.0.0.1:9200 (easier than vpn/ipsec and all that and fast enough for now)

Screenshot-2018-3-26%20Grafana%20-%20TPOT
Screenshot-2018-3-26%20Grafana%20-%20TPOT980×974 47.4 KB

grafana listens to 127.0.0.1:3000
grafana is served by nginx using this config
cat /etc/nginx/sites-enabled/MY.SERVER.FQDN.conf

server {
listen 80;

server_name "MY.SERVER.FQDN";
return 301 https://$server_name$request_uri;

}

server {
listen 443;
listen [::]:443;
server_name MY.SERVER.FQDN;
access_log /var/log/nginx/MY.SERVER.FQDN-ssl-access.log;
error_log /var/log/nginx/MY.SERVER.FQDN-ssl-error.log error;
ssl on;
ssl_certificate /etc/letsencrypt/live/MY.SERVER.FQDN/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/MY.SERVER.FQDN/privkey.pem;
ssl_dhparam /etc/ssl/private/dh.pem;
location / {
proxy_pass http://127.0.0.1:3000;
proxy_set_header X-Real-IP $remote_addr;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection ‘upgrade’;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}

}

11

Might be related to this issue where someone describes how Chrome adds an Origin header that Firefox does not:

1 Like
12

Umm… ok but how does this relate ?
I don’t understand why should i change my Es config ? my grafana is served on a public address. there is no way I’m ok to allowing inbound internet access to my es cluster. grafana is the only one that needs it in this scenario, and the only public information available on the internet must be served by nginx and nginx alone. I really dont see how is this an es issue since it works so well in ff and absolutely fails in any other browser. kibana works flawlessly in any browser btw, but i want grafana for my own reasons.

13

adding
proxy_set_header Origin "";
inside the location / {} block in nginx is a valid workaround. this is weird. can’t this be fixed grafana-side ? maybe in a future update ?

14

Sounds like an ES bug to me.

Would not (updating the CORS settings) as described in the link above be both easier and more secure?