Trying out the ElasticSearch Alerting in 5.2.x

Grafana Elasticsearch
1

Hi All,

I’m trying out the elasticsearch alerting today and I seem to get very odd results.

I’m doing a simple query (count grouped by date histogram).

My alert is simple “for last 5 minutes, if count is higher than 5 then alert”.

However, testing the rule reveals a count of 1501.

That’s odd given the index itself has only 29 documents at the moment.

I’m unsure how it’s produced this result but expanding the query in the test rule does seem to reveal an array of 1501 results.

Anyone have any ideas why I’d be getting results like this?

I’ve attempted to use Elasticsearch 6.3 and 5.6 and tried changing the datasource version between 5.x and 5.6+ for 5.6 whilst leaving it on 5.6+ for 6.3 - Needless to say it made no difference, the results were the same.

Many thanks,

Jase

2

What is the ‘When’ clause for your alert? Can you provide more data - like the Data Source, Query Inspector, and Query (all under the Metric tab), the specific Alert settings along with at least a portion of the Test results from the Alert?

We have Elasticsearch Alerts working here, but it took a little while to figure out the nuances of calculating the desired results.

3

@davewaters - Thanks for your reply.

{
  "xhrStatus": "complete",
  "request": {
    "method": "POST",
    "url": "api/datasources/proxy/1/_msearch",
    "data": "{\"search_type\":\"query_then_fetch\",\"ignore_unavailable\":true,\"index\":\"jason\",\"max_concurrent_shard_requests\":256}\n{\"size\":0,\"query\":{\"bool\":{\"filter\":[{\"range\":{\"@timestamp\":{\"gte\":\"1530631264806\",\"lte\":\"1530632164806\",\"format\":\"epoch_millis\"}}},{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}]}},\"aggs\":{\"2\":{\"date_histogram\":{\"interval\":\"10s\",\"field\":\"@timestamp\",\"min_doc_count\":0,\"extended_bounds\":{\"min\":\"1530631264806\",\"max\":\"1530632164806\"},\"format\":\"epoch_millis\"},\"aggs\":{}}}}\n"
  },
  "response": {
    "responses": [
      {
        "took": 48,
        "timed_out": false,
        "_shards": {
          "total": 5,
          "successful": 5,
          "skipped": 0,
          "failed": 0
        },
        "hits": {
          "total": 740000,
          "max_score": 0,
          "hits": []
        },
        "aggregations": {
          "2": {
            "buckets": [
              {
                "key_as_string": "1530631260000",
                "key": 1530631260000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631270000",
                "key": 1530631270000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631280000",
                "key": 1530631280000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631290000",
                "key": 1530631290000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631300000",
                "key": 1530631300000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631310000",
                "key": 1530631310000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631320000",
                "key": 1530631320000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631330000",
                "key": 1530631330000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631340000",
                "key": 1530631340000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631350000",
                "key": 1530631350000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631360000",
                "key": 1530631360000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631370000",
                "key": 1530631370000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631380000",
                "key": 1530631380000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631390000",
                "key": 1530631390000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631400000",
                "key": 1530631400000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631410000",
                "key": 1530631410000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631420000",
                "key": 1530631420000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631430000",
                "key": 1530631430000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631440000",
                "key": 1530631440000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631450000",
                "key": 1530631450000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631460000",
                "key": 1530631460000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631470000",
                "key": 1530631470000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631480000",
                "key": 1530631480000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631490000",
                "key": 1530631490000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631500000",
                "key": 1530631500000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631510000",
                "key": 1530631510000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631520000",
                "key": 1530631520000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631530000",
                "key": 1530631530000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631540000",
                "key": 1530631540000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631550000",
                "key": 1530631550000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631560000",
                "key": 1530631560000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631570000",
                "key": 1530631570000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631580000",
                "key": 1530631580000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631590000",
                "key": 1530631590000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631600000",
                "key": 1530631600000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631610000",
                "key": 1530631610000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631620000",
                "key": 1530631620000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631630000",
                "key": 1530631630000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631640000",
                "key": 1530631640000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631650000",
                "key": 1530631650000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631660000",
                "key": 1530631660000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631670000",
                "key": 1530631670000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631680000",
                "key": 1530631680000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631690000",
                "key": 1530631690000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631700000",
                "key": 1530631700000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631710000",
                "key": 1530631710000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631720000",
                "key": 1530631720000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631730000",
                "key": 1530631730000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631740000",
                "key": 1530631740000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631750000",
                "key": 1530631750000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631760000",
                "key": 1530631760000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631770000",
                "key": 1530631770000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631780000",
                "key": 1530631780000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631790000",
                "key": 1530631790000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631800000",
                "key": 1530631800000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631810000",
                "key": 1530631810000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631820000",
                "key": 1530631820000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631830000",
                "key": 1530631830000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631840000",
                "key": 1530631840000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631850000",
                "key": 1530631850000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631860000",
                "key": 1530631860000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631870000",
                "key": 1530631870000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631880000",
                "key": 1530631880000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631890000",
                "key": 1530631890000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631900000",
                "key": 1530631900000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631910000",
                "key": 1530631910000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631920000",
                "key": 1530631920000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631930000",
                "key": 1530631930000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631940000",
                "key": 1530631940000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631950000",
                "key": 1530631950000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631960000",
                "key": 1530631960000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631970000",
                "key": 1530631970000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631980000",
                "key": 1530631980000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530631990000",
                "key": 1530631990000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632000000",
                "key": 1530632000000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632010000",
                "key": 1530632010000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632020000",
                "key": 1530632020000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632030000",
                "key": 1530632030000,
                "doc_count": 60000
              },
              {
                "key_as_string": "1530632040000",
                "key": 1530632040000,
                "doc_count": 60000
              },
              {
                "key_as_string": "1530632050000",
                "key": 1530632050000,
                "doc_count": 80000
              },
              {
                "key_as_string": "1530632060000",
                "key": 1530632060000,
                "doc_count": 120000
              },
              {
                "key_as_string": "1530632070000",
                "key": 1530632070000,
                "doc_count": 120000
              },
              {
                "key_as_string": "1530632080000",
                "key": 1530632080000,
                "doc_count": 100000
              },
              {
                "key_as_string": "1530632090000",
                "key": 1530632090000,
                "doc_count": 100000
              },
              {
                "key_as_string": "1530632100000",
                "key": 1530632100000,
                "doc_count": 100000
              },
              {
                "key_as_string": "1530632110000",
                "key": 1530632110000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632120000",
                "key": 1530632120000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632130000",
                "key": 1530632130000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632140000",
                "key": 1530632140000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632150000",
                "key": 1530632150000,
                "doc_count": 0
              },
              {
                "key_as_string": "1530632160000",
                "key": 1530632160000,
                "doc_count": 0
              }
            ]
          }
        },
        "status": 200
      }
    ]
  }
}

Datasource is elasticsearch.

Query as mentioned:

I’m doing a simple query (count grouped by date histogram).

image
image1188×549 31.6 KB

Alert rule:

My alert is simple “for last 5 minutes, if count is higher than 5 then alert”.

image
image600×635 35.9 KB

Given the count shown clearly on the charts in the first screen grab, the second screen grab says “Value: 1501.00” - Not sure where this figure has come about.

Thanks,

Jase

4

Please try and set an explicit interval in query date histogram

1 Like
5

@mefraimsson - Setting the explicit interval seems to adjust the number.
The only pattern I can see is the number seems to represent the amount of points plotted, not the actual Y Axis value.

Edit: Looks like the problem is the alert function I’m using.
Effectively I’m doing a ‘count of a count’, which I guess does mean the plotted points count as opposed to count of elasticsearch documents in the query.
Adjusting to say, ‘sum()’ produces the expected results.
Or even using ‘last()’ if I wanted a more instant result.

1 Like
6

@mefraimsson - Elastic search alerting not working with Templating in V 5.2.1. I am getting error “Template variables are not supported in alert queries”. Can someone help me with this?

7

Template variables is not supported in alerting, see https://github.com/grafana/grafana/issues/6557