Alerting - no data

Grafana Elasticsearch
1

Hi, daniellee. I install grafana 5.2.1, linked to datasource with Elasticsearch and create Graph and shown liner data. Query is too simple
| check.Status:passing

“{“search_type”:“query_then_fetch”,“ignore_unavailable”:true,“index”:[“consulbeat-isu–2018.07.02”]} {“size”:0,“query”:{“bool”:{“filter”:[{“range”:{”@timestamp":{“gte”:“1530522963384”,“lte”:“1530526563386”,“format”:“epoch_millis”}}},{“query_string”:{“analyze_wildcard”:true,“query”:“check.Status:passing AND !(service.ID:consul)”}}]}},“aggs”:{“3”:{“terms”:{“field”:“service.ID.keyword”,“size”:500,“order”:{“_term”:“desc”},“min_doc_count”:1},“aggs”:{“2”:{“date_histogram”:{“interval”:“10s”,“field”:“@timestamp”,“min_doc_count”:0,“extended_bounds”:{“min”:“1530522963384”,“max”:“1530526563386”},“format”:“epoch_millis”},“aggs”:{}}}}}} "

shows liner graph with value 2. if some service is down, value will be 1
Created Alert:
when avg () of query (A, 20s, now) is below 1,9
with rools:

  1. nodata : alerting
  2. error execution : alerting

create fail service, which have value = 1 on graph

08
081171×380 21.6 KB

add notification on email

try to “Test Rule” but no result

firing:false
state:“alerting”
conditionEvals:“false = false”
timeMs:“8.986ms”
logs:Array[1]
0:Object
message:“Condition: Eval: false, Query Returned No Series (reduced to null/no value)”
data:null

i also trying nightly build “grafana-5.3.0-22954pre1.windows-amd64.zip”, but have no result too

2

Hi,

Please try avg () of query (A, 20s, now-10s or look further back from now. Since you’re using an interval of 10s there’s a probability that there won’t be any data at the end of the time range.

Marcus

3

i try:

  1. (A, 20s, now-10s) - as you recomended. no results
  2. (A, 20s, now-1h) - test. no results
  3. (A, 1h, now-5h) - test. no results

firing:false
state:“alerting”
conditionEvals:“false = false”
timeMs:“5.012ms”
logs:Array[1]
0:Object
message:“Condition: Eval: false, Query Returned No Series (reduced to null/no value)”
data:null

on file log last lines is

t=2018-07-02T15:40:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:40:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:41:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:41:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:41:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:42:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:42:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:42:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:43:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:43:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:43:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:44:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:44:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:44:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:45:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:45:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:45:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:46:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:46:18+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:46:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:46:24+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:46:34+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:46:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:46:47+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:47:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:47:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:47:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:48:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:48:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T15:48:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:49:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T15:49:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T15:49:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data

4

Try a different interval in query, like 1m instead of 10s. Are you sure you’re getting valid data in your graph? If you only show points instead of lines?

Marcus

5

i change interval in Metrics to 1m and get new graph :

28
28853×821 70.1 KB

6

And if you change to points instead of lines under the display tab? Please use the query inspector tab to inspect the returned data is as you expect.

7
  1. liner ==> point Monosnap
  2. request of 1.1 is 1.2 in DropMeFiles – free one-click file sharing service

“key”: “beta_screening_app3” has value == 6 when interval is 1m. For interval 10s value will be 1. And alert must be firing a some message

8

Does it make any difference if you remove the trim edges setting? Doesn’t seem to be included in your request attached.

9

i removed edged. Trying with interval = 1m, Alert: avg() of query (A, 5m, now-1m) is below 10. no results.

firing:false
state:“alerting”
conditionEvals:“false = false”
timeMs:“8.002ms”
logs:Array[1]
0:Object
message:“Condition: Eval: false, Query Returned No Series (reduced to null/no value)”
data:null

query inspector:

{“search_type”:“query_then_fetch”,“ignore_unavailable”:true,“index”:[“consulbeat-isu–2018.07.02”]} {“size”:0,“query”:{“bool”:{“filter”:[{“range”:{“@timestamp”:{“gte”:“1530536682490”,“lte”:“1530537582490”,“format”:“epoch_millis”}}},{“query_string”:{“analyze_wildcard”:true,“query”:“check.Status:passing AND !(service.ID:consul)”}}]}},“aggs”:{“3”:{“terms”:{“field”:“service.ID.keyword”,“size”:500,“order”:{“_term”:“desc”},“min_doc_count”:1},“aggs”:{“2”:{“date_histogram”:{“interval”:“1m”,“field”:“@timestamp”,“min_doc_count”:0,“extended_bounds”:{“min”:“1530536682490”,“max”:“1530537582490”},“format”:“epoch_millis”},“aggs”:{}}}}}}

logs:

t=2018-07-02T16:15:55+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:15:57+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:15:58+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:16:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T16:16:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T16:16:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:17:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T16:17:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T16:17:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:18:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T16:18:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T16:18:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:19:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T16:19:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:19:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T16:19:36+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T16:20:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T16:20:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T16:20:40+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting

10

One odd thing that brought my attention is that your index has a weird/non-standard dash (-).

Compare
consulbeat-isu–2018.07.02
with
consulbeat-isu-2018.07.02

You see that your dash is longer in the first one. Could you please try and update your datasource settings and specify index using the dash at the bottom?

Marcus

11

my data source writing with 2 dashed, and index has 2 dashes too

{
“consulbeat-isu–2018.06.27”: {
“settings”: {
“index”: {
“refresh_interval”: “5s”,
“number_of_shards”: “1”,
“provided_name”: “consulbeat-isu–2018.06.27”,
“creation_date”: “1530131188612”,
“number_of_replicas”: “1”,
“uuid”: “SXPcmlAeRJG77RH0IPIEbw”,
“version”: {
“created”: “5060099”
}
}
}
}
}

i must enter more shorting name in index, isn’t it?

12

Hmm okay. Lets try and enable debug logging and see what information you get when alert rules are evaluated.

[log]
filters = tsdb.elasticsearch.client:debug
13

t=2018-07-02T17:06:27+0300 lvl=info msg=“HTTP Server Listen” logger=http.server address=0.0.0.0:3005 protocol=http subUrl= socket=
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Creating new client” logger=tsdb.elasticsearch.client version=5 timeField=@timestamp indices=logs-isu-2018.07.02
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Executing multisearch” logger=tsdb.elasticsearch.client search requests=1
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Encoding batch requests to json” logger=tsdb.elasticsearch.client batch requests=1
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Encoded batch requests to json” logger=tsdb.elasticsearch.client took=998.5µs
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Executing request” logger=tsdb.elasticsearch.client url=http://serv-es:9200/_msearch method=POST
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Executed request” logger=tsdb.elasticsearch.client took=6.0035ms
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Received multisearch response” logger=tsdb.elasticsearch.client code=400 status=“400 Bad Request” content-length=-1
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Decoding multisearch json response” logger=tsdb.elasticsearch.client
t=2018-07-02T17:07:01+0300 lvl=dbug msg=“Decoded multisearch json response” logger=tsdb.elasticsearch.client took=999.9µs
t=2018-07-02T17:07:01+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=1 name=“Response times alert” changing state to=no_data
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Creating new client” logger=tsdb.elasticsearch.client version=5 timeField=@timestamp indices=logs-isu-2018.07.02
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Creating new client” logger=tsdb.elasticsearch.client version=5 timeField=@timestamp indices=consulbeat-isu–2018.07.02
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Executing multisearch” logger=tsdb.elasticsearch.client search requests=1
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Executing multisearch” logger=tsdb.elasticsearch.client search requests=1
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Encoding batch requests to json” logger=tsdb.elasticsearch.client batch requests=1
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Encoding batch requests to json” logger=tsdb.elasticsearch.client batch requests=1
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Encoded batch requests to json” logger=tsdb.elasticsearch.client took=0s
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Encoded batch requests to json” logger=tsdb.elasticsearch.client took=994.7µs
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Executing request” logger=tsdb.elasticsearch.client url=http://serv-es:9200/_msearch method=POST
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Executing request” logger=tsdb.elasticsearch.client url=http://serv-es:9200/_msearch method=POST
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Executed request” logger=tsdb.elasticsearch.client took=3.1522ms
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Executed request” logger=tsdb.elasticsearch.client took=2.1532ms
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Received multisearch response” logger=tsdb.elasticsearch.client code=400 status=“400 Bad Request” content-length=-1
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Received multisearch response” logger=tsdb.elasticsearch.client code=400 status=“400 Bad Request” content-length=-1
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Decoding multisearch json response” logger=tsdb.elasticsearch.client
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Decoding multisearch json response” logger=tsdb.elasticsearch.client
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Decoded multisearch json response” logger=tsdb.elasticsearch.client took=8.0019ms
t=2018-07-02T17:07:20+0300 lvl=dbug msg=“Decoded multisearch json response” logger=tsdb.elasticsearch.client took=3.9884ms
t=2018-07-02T17:07:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=2 name=“Errors alert” changing state to=no_data
t=2018-07-02T17:07:20+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=3 name=“Healthcheck alert” changing state to=alerting
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Creating new client” logger=tsdb.elasticsearch.client version=5 timeField=@timestamp indices=consulbeat-isu–2018.07.02
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Executing multisearch” logger=tsdb.elasticsearch.client search requests=1
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Encoding batch requests to json” logger=tsdb.elasticsearch.client batch requests=1
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Encoded batch requests to json” logger=tsdb.elasticsearch.client took=1.0013ms
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Executing request” logger=tsdb.elasticsearch.client url=http://serv-es:9200/_msearch method=POST
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Executed request” logger=tsdb.elasticsearch.client took=3.2063ms
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Received multisearch response” logger=tsdb.elasticsearch.client code=400 status=“400 Bad Request” content-length=-1
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Decoding multisearch json response” logger=tsdb.elasticsearch.client
t=2018-07-02T17:07:27+0300 lvl=dbug msg=“Decoded multisearch json response” logger=tsdb.elasticsearch.client took=8.9975ms
t=2018-07-02T17:07:27+0300 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=0 name=“Healthcheck alert” changing state to=alerting

14

What is the result from the query inspector? The Alert rules says it isn’t getting any data - I’m curious what the Query Inspector shows the actual results to be (is data being returned at all?)

15

That doesn’t look right. Could you please share your datasource index configuration?

Marcus

16

sorry, it’s second datasource

17

i change range for graph to last 5 m
result from query inpector
https://dropmefiles.com/KTugD

18

Sorry didn’t see that. But elasticsearch returns a 400 bad request. Wonder if the index with “long dash” maybe needs to be url encoded or something. Have to dig into that.

The query inspector is only for the queries executed from the UI. Those queries are executed in the browser where alerting queries are executed on the server.

Marcus

19

ok, i’ll try from other index with template logs-isu-YYYY.MM.DD. it’s correct index name?

20

Yes please try that instead. If that’s not working there must be a bug with parsing Grafana query to ES query. And you’re using elasticsearch version 5 - correct?

Marcus