How to change the way grafana performs query to ElasticSearch (content-type issue)?

Hi,

We are facing an issue while we want to connect Grafana to an ElasticSearch database.

Here is our situation : Grafana is connected to ElasticSearch through a gateway Knox.
Knox, for reasons we don’t understand yet, removes the “\n” characters and so, when Grafana performs an “_msearch” query (each query separated by “\n”), it gives an HTTP 400…

We tried with to query ElasticSearch with “curl” through Knox like that :

curl -i -H “Content-Type: application/json” -XPOST “http://…” => does not work
curl -i -H “Content-Type: application/octet-stream” -XPOST “http://…” => it works !

So, with a “Content-Type: application/octet-stream” we success to query ElasticSearch through Knox.
But, Grafana does not work this way : it produces HTTP query with “Content-Type: application/json”.

Thus, we attempt to modify Grafana’s source code but even if we modify this file :

replacing “application/json” by “application/octet-stream”, recompiling the whole application, Grafana still query ElasticSearch with “application/json”… :frowning:

So, could you help us to solve this issue and make possible Grafana querying with “content-type:application/octet-stream” instead of “content-type:application/json” ?

Thanks :slight_smile:

How did you configure your ES datasource and where do you see “Content-Type: application/json” header?

Indeed, I forgot to tell that we success in configuring the ES datasource : when we “save & test” it shows that everything is ok. Behind this test, I presume that Grafana performs a simple ES query in only one line.

Things get complicated when we design a dashboard. We choose a datasource (our ES data source) and then, we get the error. After researches, we found that Grafana performs an “_msearch” query that means that the query is structured like that (just a generic example) :

GET _msearch/
{“index” : “index1”}
{“query” : {“match_all” : {}}}
{“index” : “index2”}
{“query” : {“match_all” : {}}}
etc.

As you can see, carriage returns are very important, if we had done it in one line, ES would had returned an error.

With Firefox in debug mode, we see the “_msearch” query and its header “content-type:application/json” (which would mean that the query is performed from the front-end), and the in error response of ElasticSearch that we are able to reproduce outside Grafana as well.

The knox gateway removing “\n”, it brokes the query format. the workaround from knox side is to performs query with “application/octet-stream”, but Grafana does not work this way that’s why we attempt to modify Grafana’s code.

I hope my explanations will help.

Thanks.

Your ES query is performed from the front-end, but you have edited the backend (Go) part. Switch your ES data source Acces to Server (Default) option. Then hopefully your custom hack in the Golang code will fix your issue.

thanks we will try to do this, and make you a feedback soon

Clarification: The backend (go) part is only currently used in alerting scenarios for evaluating alert rules in Grafana backend, otherwise only frontend code is used. Access mode should not matter given you’re not using alerting, howewer Server access mode goes through Grafana backend proxy where’s Browser access mode does not - please verify/test the different and see if there are any bug somewhere.

Grafana send request with content type “application/json”, when it send multisearch request with multiple json documents, knox remove “/n”. I solved this problem by modifying the DispatchElasticserach classe of Knox by rewriting the request send from Grafana to be send to ES with a content type “application/x-ndjson”.