I’ve configured Elasticsearch 7.12.0 as a data source in Grafana 7.5.2.
I can successfully build a dashboard, which shows the data from Elasticsearch that I expect to see. However, when I attempt to configure an alert, no data is returned.
The issue appears to be that the query created for the alert doesn’t populate the value for date_histogram.field. I.e., when I test the alert, the following query is generated:
{"ignore_unavailable":true,"index":"filebeat-*","search_type":"query_then_fetch"}
{"aggs":{"2":{"date_histogram":{"field":"","interval":"1m","min_doc_count":0,"extended_bounds":{"min":"1623161857376","max":"1623162457376"},"format":"epoch_millis"}}},"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":"1623161857376","lte":"1623162457376"}}},{"query_string":{"analyze_wildcard":true,"query":"event.dataset:\"foo.bar\""}}]}},"size":0}
As you can see, date_histogram.field: "" is blank. When I run this query from the command line, I get the following data back:
{
"took" : 54,
"responses" : [
{
"took" : 38,
"timed_out" : false,
"_shards" : {
"total" : 200,
"successful" : 200,
"skipped" : 150,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 740,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"2" : {
"buckets" : [
{
"key_as_string" : "1623161280000",
"key" : 1623161280000,
"doc_count" : 0
},
{
"key_as_string" : "1623161340000",
"key" : 1623161340000,
"doc_count" : 0
},
{
"key_as_string" : "1623161400000",
"key" : 1623161400000,
"doc_count" : 0
},
{
"key_as_string" : "1623161460000",
"key" : 1623161460000,
"doc_count" : 0
},
{
"key_as_string" : "1623161520000",
"key" : 1623161520000,
"doc_count" : 0
},
{
"key_as_string" : "1623161580000",
"key" : 1623161580000,
"doc_count" : 0
},
{
"key_as_string" : "1623161640000",
"key" : 1623161640000,
"doc_count" : 0
},
{
"key_as_string" : "1623161700000",
"key" : 1623161700000,
"doc_count" : 0
},
{
"key_as_string" : "1623161760000",
"key" : 1623161760000,
"doc_count" : 0
},
{
"key_as_string" : "1623161820000",
"key" : 1623161820000,
"doc_count" : 0
},
{
"key_as_string" : "1623161880000",
"key" : 1623161880000,
"doc_count" : 0
}
]
}
},
"status" : 200
}
]
}
The total number of hits shows the expected value, the individual buckets show doc_count: 0. However, if I set date_histogram.field: "@timestamp" while executing the query from the command line as follows, it segregates the count into buckets, as expected.
{"ignore_unavailable":true,"index":"filebeat-*","search_type":"query_then_fetch"}
{"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"1m","min_doc_count":0,"extended_bounds":{"min":"1623161336265","max":"1623161936265"},"format":"epoch_millis"}}},"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":"1623161336265","lte":"1623161936265"}}},{"query_string":{"analyze_wildcard":true,"query":"event.dataset:\"foo.bar\""}}]}},"size":0}