Query Elasticsearch from Grafana

Grafana Elasticsearch
1

Hello,

I am trying to visualize some data in Grafana that I get from Elasicsearch.

I was able to add the data source and I am able to visualize simple things like the count time series for an index.

What I am not able to do is:

  1. Group by successfully. I am trying to group by with “Terms” and selecting a keyword field (a keyword text field with name structure .keyword).
    As soon as I do this, I get “No data” on my Grafana panel.

I also tried to copy the query from Query Inspector > Object > request > data (removing the first json with “search_type” and “index”, and taking the second, which contains the actual query in Query DSL for Elasticsearch) and to use it to make a GET request to elasticsearch.
When I do this, it works and returns the aggregated values. Unfortunately I am not able to edit the query in Grafana, nor to understand fully the additional fields that I find when opening the Query Inspector, nor to understand why specifying terms in the Group by UI gives me no data.

  1. Understand what are the current capabilities of querying Elasticsearch through Grafana. I tried to input something with Lucene syntax in the open text field “Query” and I get some values. But I am not able to put there queries written in Query DSL (jsons) there. And I am struggling to find a decent guide with some examples that can help me understanding how to use this.
    For example I am not able to answer simple questions like: “Is it possible to aggregate/group by using the Query field?”, “Are the Query field and the Metric + Group by fields 2 orthogonal ways to query data? Or can/should they be used together to structure a query?”.

Thank you in advance for your help.

[I am running Grafana 6.7.3 and Elasticsearch 7.7]

1 Like
2

Hi,

  1. Grouping by Terms should work, indeed with using the .keyword version of the field. (it is necessary to make elasticsearch and grafana talk nicely to eachother)
    When working with elasticsearch setting the Metric field to ‘Logs’ or ‘Raw Document’ is necessary to actually parse the elasticsearch fields.

  2. Grafana doesn’t support the Elasticsearch Query DSL. Only lucene queries.
    So it’s indeed not possible to group your data with the query field.
    Grouping and aggregations will have to be done in the grafana gui.

3

Thank you.

Just a further question on the second point. What are you referring to exactly when suggesting that “grouping and aggregations will have to be done in the grafana gui”? Do you suggest to tweak the datasources to get something already aggregated?

4

You can use the group-by functions in the query panel.

image
image846×204 12.2 KB

And further process your data by using the Transform panel.

Maybe “doesn’t support” was poorly worded. Grafana will build up the query-dsl based on what you enter in the query panel with the ‘Metric’ and ‘Group by’ options.

The above screenshot will result in this query-dsl being fired against elasticsearch:

{
    "size": 0,
        "query": {
            "bool": {
                "filter": [
                    {
                        "range": {
                            "@timestamp": {
                                "gte": 1590118795217,
                                "lte": 1590140395217,
                                "format": "epoch_millis"
                            }
                        }
                    },
                    {
                        "query_string": {
                            "analyze_wildcard": true,
                            "query": "*"
                        }
                    }
                ]
            }
        },
        "aggs": {
            "3": {
                "terms": {
                    "field": "application.keyword",
                    "size": 10,
                    "order": {
                        "_term": "desc"
                    },
                    "min_doc_count": 0
                },
                "aggs": {
                    "2": {
                        "date_histogram": {
                            "interval": "15s",
                            "field": "@timestamp",
                            "min_doc_count": 0,
                            "extended_bounds": {
                                "min": 1590118795217,
                                "max": 1590140395217
                            },
                            "format": "epoch_millis"
                        },
                        "aggs": {}
                    }
                }
            }
        }
    }