ElasticSearch Template Variable with nested fields

Grafana
1

Hi,

I try to find a way to display all “fields” from nested fields in elasticsearch as template variable. I know the doc (http://docs.grafana.org/features/datasources/elasticsearch/#templating) but I dont think it is possible to get what I want.

I use Icinga2 with the elasticsearch writer module. I get my checkresults in elasticsearch and now I want to display them in grafana with most flexibilty for the user, with template variables.

This is an example of one document:

{
  "_index": "icinga2-2018.08.15",
  "_type": "doc",
  "_id": "zJI7PmUBXvkFreDxTo0y",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-08-15T17:36:42.152+0200",
    "check_command": "load",
    "check_result.check_source": "*******.com",
    "check_result.command": [
      "/usr/lib/nagios/plugins/check_load",
      "-c",
      "10,6,4",
      "-r",
      "-w",
      "5,4,3"
    ],
    "check_result.execution_end": "2018-08-15T17:36:42.152+0200",
    "check_result.execution_start": "2018-08-15T17:36:42.148+0200",
    "check_result.execution_time": 0.003974199295043945,
    "check_result.exit_status": 0,
    "check_result.latency": 0,
    "check_result.output": "OK - load average: 0.21, 0.22, 0.24",
    "check_result.perfdata.load1.crit": 10,
    "check_result.perfdata.load1.value": 0.211,
    "check_result.perfdata.load1.warn": 5,
    "check_result.perfdata.load15.crit": 4,
    "check_result.perfdata.load15.value": 0.24,
    "check_result.perfdata.load15.warn": 3,
    "check_result.perfdata.load5.crit": 6,
    "check_result.perfdata.load5.value": 0.221,
    "check_result.perfdata.load5.warn": 4,
    "check_result.schedule_end": "2018-08-15T17:36:42.152+0200",
    "check_result.schedule_start": "2018-08-15T17:36:42.152+0200",
    "check_result.state": 0,
    "check_result.vars_after": {
      "attempt": 1,
      "reachable": true,
      "state": 0,
      "state_type": 1
    },
    "check_result.vars_before": {
      "attempt": 1,
      "reachable": true,
      "state": 0,
      "state_type": 1
    },
    "current_check_attempt": 1,
    "host": "*****.com",
    "last_hard_state": 0,
    "last_state": 0,
    "max_check_attempts": 2,
    "reachable": true,
    "service": "CPU Check Percentage",
    "state": 0,
    "state_type": 1,
    "timestamp": "2018-08-15T17:36:42.152+0200",
    "type": "icinga2.event.checkresult"
  },
  "fields": {
    "@timestamp": [
      "2018-08-15T15:36:42.152Z"
    ],
    "check_result.execution_start": [
      "2018-08-15T15:36:42.148Z"
    ],
    "check_result.execution_end": [
      "2018-08-15T15:36:42.152Z"
    ],
    "check_result.schedule_end": [
      "2018-08-15T15:36:42.152Z"
    ],
    "check_result.schedule_start": [
      "2018-08-15T15:36:42.152Z"
    ],
    "timestamp": [
      "2018-08-15T15:36:42.152Z"
    ]
  },
  "highlight": {
    "service": [
      "@kibana-highlighted-field@CPU@/kibana-highlighted-field@ Check Percentage"
    ],
    "host": [
      "@kibana-highlighted-field@*****@/kibana-highlighted-field@.@kibana-highlighted-field@******.com@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1534347402152
  ]
}

Extracting the Hosts and Services as template value was easy. But what about the checkresults with performance data?
What would the template query look like if I want to get “load1”, “load5” and “load15” as template value?
In Graphite it would look like "check_result.perfdata.* " to get all the possible perfdata results in a template variable.
I tried the “field” and “type” query option for template, but both seems not to be the right because its neighter a field nor a type.

A workaround would be a static template variable with all the possible perfrdata fields, but I would like to avoid it because of obvious reasons.

I’m surprised I can’t find any example on the internet for the combination of Icinga2 + ES Writer + Grafana, because It would be an awesome combination :slight_smile:

Thanks for your help.
Patrick

1 Like
2

Hi again,

I found a workaround but it is not a 100% solution. Would love to hear your opinion.
I was able to get all the fields with following template query:
Query: {"find":"fields"}
With the Regex input I filtered just for the perfdata fields:
Regex: /perfdata.(.*).value/

This looks almost like the thing I needed, but the query was not filtered by any other template variable like Host or Service. So I tried to add the query function:
{"find":"fields", "query":"host:$host"}
But in the network tab I can see there is just the “_mapping” request to the elastic search. Its seems like the feature works only for the “find”: “terms” query :frowning:

So right now I have every perfdata field from the elasticsearch in my new template variable. It would be cool to have only the available fields in relation to the selected Host and/or service.
Would be a github issue the better way to address my issue?

Thanks and regards
Patrick

1 Like
3

Did you find a solution for this
Please help.